A position-sensitive illumination consistency rendering method and device

Abstract

The present invention discloses a position-sensitive illumination consistency rendering method, which first produces a data set, and then constructs and trains a basic illumination consistency rendering model based on soft contrast learning; the trained neural network model receives pictures that need to be rendered with illumination consistency, and outputs the pictures after completing the harmony processing. The method of the present invention successfully trains the generator model through position-aware segmentation and soft contrast learning to enable it to have position-sensitive image coordination capabilities, which can effectively solve the problem that the synthesized image may look unnatural and disharmonious after a realistic foreground object is inserted into another background image. In addition, the present invention achieves local style consistency through the method of soft contrast learning, and at the same time achieves global style alignment and content retention through a comprehensive content optimization strategy, and finally achieves the natural harmony of the synthesized image, thereby achieving the purpose of image coordination.

Description

Technical Field

[0001] The present invention belongs to the field of illumination consistency rendering based on deep learning technology, and specifically relates to a position-sensitive illumination consistency rendering method and device. Background Art

[0002] Image synthesis is an important operation in image processing. After inserting a realistic foreground object into another background image, the synthesized image may look unnatural and disharmonious. The purpose of illumination consistency rendering is to transfer the style of the background to the foreground object, which is a challenging task because there is a large domain gap between the foreground and the background. The current mainstream methods (PHDNet, ProPIH) generally choose to align the foreground style with the overall style of the background, but the style differences in different areas of the image may be large. This method does not take into account the local style, which leads to local disharmony and unnaturalness. Therefore, the present invention proposes a position-sensitive illumination consistency rendering method. Summary of the Invention

[0003] To address the shortcomings of the existing technology, the present invention provides a method and device for position-sensitive illumination consistency rendering. The specific technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a position-sensitive lighting consistency rendering method, comprising the following steps:

[0005] First, create a dataset and build the image dataset required for network training.

[0006] Then a basic lighting consistency rendering model based on soft contrast learning is constructed.

[0007] (1) Theoretical modeling of basic lighting consistency rendering model based on soft contrast learning.

[0008] (2) Construct a training network for illumination consistency rendering based on soft contrast learning.

[0009] (3) Training is performed to generate a network model with position-sensitive image coordination effects, that is, a basic lighting consistency rendering model based on soft contrast learning is obtained.

[0010] The trained neural network model receives images that need to be rendered for lighting consistency, completes the harmonization process, and outputs the images.

[0011] In one possible implementation, the public datasets COCO and WikiArt were downloaded during dataset creation. Images from the WikiArt dataset were used as background images, and the instance segmentation masks provided by the COCO dataset were used to extract photographic foreground objects from the COCO dataset. The foreground objects were randomly inserted into the background images using instance segmentation masks to form a composite image. .

[0012] In one possible implementation, the theoretical modeling process includes:

[0013] A generator including a style transfer module takes the composite image, background image, and foreground mask as input and outputs a harmonious image. Secondly, a discriminator is proposed to guide the generator to produce more realistic and harmonious images.

[0014] The generator is trained by continuously optimizing the loss. The total loss of the generator includes: local soft contrast style loss for achieving local style consistency, global style loss for comprehensive content optimization, adversarial loss, structural similarity loss and detail similarity loss.

[0015] In one possible implementation, a training network for illumination-consistent rendering based on soft contrast learning includes a generator and a discriminator. The generator consists of an encoder, a style transfer module, and a decoder. The encoder uses a pretrained VGG-19 network, and the decoder design mirrors the encoder architecture by using upsampling operations. The style transfer module captures the style information of the background and applies it to the foreground object. The discriminator assesses whether there are discordant areas in the harmonized image.

[0016] In one possible implementation, the discriminator includes a light spatial encoder and a lightweight autoencoder. Specifically, six downsampling blocks are applied in the light spatial encoder, where each encoding block contains a convolutional layer with a kernel size of 4 and a stride of 2, followed by a batch normalization layer and an activation function layer; the lightweight autoencoder includes two downsampling blocks and two upsampling blocks; each downsampling block in the lightweight autoencoder sequentially contains a convolutional layer with a kernel size of 3 and a stride of 1, a batch normalization layer, and an activation function layer; and each upsampling block has the same structure as the downsampling block of the lightweight autoencoder, except for a different activation function; the downsampling blocks use LeakyReLU activation while the upsampling blocks use ReLU activation.

[0017] In one possible implementation, the training process is supervised by soft contrastive learning. First, a position-aware segmentation module is used to obtain candidate patches to be reconciled from the composite image. Soft contrastive learning is then used to transform the style of the foreground in each patch to achieve local consistency.

[0018] To preserve the details of the transferred foreground, a content-preserving process (i.e., comprehensive content optimization strategy) is proposed, which consists of two parts: global style alignment and content preservation.

[0019] Global style alignment uses adversarial loss to initially align the styles of the foreground and background regions, and further enhances the style alignment effect through distribution-based style alignment loss (i.e., overall style loss):

[0020] The content is preserved by using structural similarity loss to maintain the consistency of the structure before and after the transformation, and by using detail similarity loss based on contrastive learning to better preserve details.

[0021] In one possible implementation, the process of performing harmonization processing on the trained neural network model includes:

[0022] First, the trained light consistency rendering network model weights are loaded and the model parameters are updated. Second, the synthesized image to be harmonized is passed as input data to the generator network model. The input data passes through the generator to obtain the harmonized output image.

[0023] In a second aspect, an embodiment of the present application provides a position-sensitive illumination consistency rendering device, comprising the following modules:

[0024] The dataset creation module is used to create the image dataset required for network training.

[0025] The model building and training module is used to build a basic lighting consistency rendering model based on soft contrast learning, including:

[0026] (1) Theoretical modeling of basic lighting consistency rendering model based on soft contrast learning.

[0027] (2) Construct a training network for illumination consistency rendering based on soft contrast learning.

[0028] (3) Training is performed to generate a network model with position-sensitive image coordination effects, that is, a basic lighting consistency rendering model based on soft contrast learning is obtained.

[0029] The image harmony module uses the neural network model trained in the model building and training module to receive images that need to be rendered for lighting consistency, and outputs the images after completing the harmony processing.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

[0031] The memory is used to store computer programs;

[0032] The processor is used to implement any lighting consistency rendering method described in this application when executing the program stored in the memory.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any of the lighting consistency rendering methods described in the present application.

[0034] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any one of the lighting consistency rendering methods described in the present application.

[0035] The beneficial effects of the present invention are as follows:

[0036] Our method successfully trains a generator model through position-aware segmentation and soft contrast learning, enabling it to achieve position-sensitive illumination-consistent rendering. This effectively addresses the problem of unnatural and dissonant synthesized images when realistic foreground objects are inserted into a background image. Furthermore, our method achieves local style consistency through soft contrast learning, while simultaneously implementing a comprehensive content optimization strategy to achieve global style alignment and content preservation, ultimately achieving a natural and harmonious synthesized image, thus achieving illumination-consistent rendering. Furthermore, user studies demonstrate that our method produces more pleasingly harmonious images. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0038] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention.

[0039] Figure 2 This is a structural diagram of the generator model according to an embodiment of the present invention.

[0040] Figure 3 This is a comparison chart of the experimental results of the embodiments of the present invention.

[0041] Figure 4 This is a diagram showing the results of a user study of an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0043] This paper proposes a soft-contrast learning-based approach to achieve local style consistency. First, adversarial learning is used to bridge the domain gap between foreground and background feature maps. Soft-contrast learning is then used to achieve position-sensitive illumination-consistent rendering. A comprehensive content optimization strategy is also designed to achieve global style alignment and content preservation, ultimately achieving position-sensitive adaptive illumination-consistent rendering. The generator consists of an encoder, a style transfer module, and a decoder. The encoder uses a pre-trained VGG-19 network, while the decoder design mirrors the encoder architecture by replacing downsampling with upsampling. The style transfer module captures background style information and applies it to foreground objects. The discriminator assesses whether there are discordant regions in the harmonized image. The discriminator enhances the generator through adversarial learning, making the generated improved foreground feature map indistinguishable from the background feature map, resulting in a more natural-looking composite image. To achieve position-sensitive illumination-consistent rendering, the paper first uses position-aware segmentation to obtain candidate patches to be harmonized from the composite image. Soft-contrast learning is then used to transform the foreground style in each patch to achieve local consistency.

[0044] This approach effectively addresses the problem of unnatural and discordant composite images resulting from inserting realistic foreground objects into a background image. Furthermore, the present invention achieves local style consistency through soft contrast learning, while simultaneously implementing a comprehensive content optimization strategy to achieve global style alignment and content preservation. Ultimately, this approach results in a natural and harmonious composite image, ultimately achieving lighting-consistent rendering.

[0045] In one possible implementation, Figure 1 As shown, the method of the present invention specifically comprises the following steps:

[0046] Step 1: Dataset creation.

[0047] Establish the image dataset required for network training in step 2. Download the public datasets COCO and WikiArt datasets. The COCO dataset is a large-scale dataset containing 123,287 images, which has instance segmentation annotations for objects in 80 categories. The Wikiart dataset is a large digital art dataset containing 81,444 images from 27 different styles. In the present invention, images from the WikiaArt dataset are used as background images, and the instance segmentation masks provided by the COCO dataset are used to extract photographic foreground objects from the COCO dataset. Randomly use instance segmentation masks to insert foreground objects into the background image to form a synthetic picture .

[0048] Step 2: Build a basic lighting consistency rendering model based on soft contrast learning.

[0049] Step 2.1: Theoretical modeling of basic lighting consistency rendering model based on soft contrast learning.

[0050] Use the generator including the style transfer module to synthesize the image , background image and foreground mask M as input and output a harmonious image. Secondly, a discriminator is proposed to guide the generator to produce more realistic and harmonious images.

[0051] The theoretical model of the generator is expressed as follows:

[0052]

[0053] in represents a composite image, represents the background image, the foreground mask M represents the area to be coordinated, and G represents the generator network.

[0054] The generator is trained by continuously optimizing the loss. The total loss of the generator can be expressed as:

[0055]

[0056] in It is a local soft contrast style loss used to achieve local style consistency. , , and They are overall style loss, adversarial loss, structural similarity loss and detail similarity loss for comprehensive content optimization. ( =1,...,5) is a hyperparameter, which is set to 0.5, 3, 6, 4, 10 in the present invention.

[0057] Step 2.2: Build a training network for lighting consistency rendering based on soft contrast learning.

[0058] The network model consists of a generator and a discriminator. The generator uses a synthetic image ( ), background image ( ) and foreground mask (M) as input. Figure 2 As shown in Figure 2, the generator consists of an encoder, a style transfer module, and a decoder. The encoder Em contains the first 31 layers of the pre-trained VGG-19 (up to ReLU4_1), and the decoder structure is symmetrical to the encoder. When training the network, the encoder Em adds skip connections on ReLU1_1, ReLU2_1, and Relu3_1 to preserve the content details in the low-level feature maps. First, the encoder Em is trained from the background image. and composite images The L=4 layers of feature maps are extracted from the four encoder layers ReLU1_1, ReLU2_1, ReLU3_1 and ReLU4_1, respectively, to obtain the background image features { } and synthetic image features { }. For layer, the feature map and with resized foreground mask are input together into the style conversion module (AdaIN layer), which adjusts Statistics of the foreground area, making it The statistical information of The decoder is a symmetrical structure of the encoder, which converts the coordinated encoder features As the input of the decoder or connected with the decoder features through a skip connection to obtain the output of the final generator .

[0059] The input of the discriminator is the harmonious image The discriminator consists of a light spatial encoder Ds and a lightweight autoencoder Da, built on top of a downsampling (DS) block and an upsampling (US) block. Specifically, six DS blocks are applied within the light spatial encoder Ds, where each DS block contains a convolutional layer with a kernel size of 4 and a stride of 2, followed by a batch normalization (BN) layer and a LeakyReLU activation. Da is a small autoencoder consisting of two DS blocks and two US blocks. Each DS block in Da sequentially contains a Conv layer with a kernel size of 3 and a stride of 1, a BN layer, and a LeakyReLU activation. Each US block has the same structure as the DS block in Da, except for the ReLU activation. The downsampling DS block uses LeakyReLU activation while the upsampling US block uses ReLU activation. In this way, the discriminator can identify whether there are discordant parts in a harmonious image.

[0060] Step 2.3: Train the training network for illumination consistency rendering based on soft contrast learning to form a network model that produces illumination consistency rendering effects with position sensitivity, that is, obtain a basic illumination consistency rendering model based on soft contrast learning.

[0061] In order to train the generator network to achieve position-sensitive lighting consistency rendering, the training process needs to be supervised by soft contrast learning. First, the position-aware segmentation module is used to synthesize the image. The candidate patches that need to be coordinated are obtained from , and then the style of the foreground in each patch is transformed using soft contrast learning to achieve local consistency. Specifically, the position-aware segmentation module first synthesizes the image Divide into k non-overlapping patches of size p*p, these patches are named { , ,... ,... }, similarly, the background image ( ) and foreground mask (M) are split into { , ,... ,... }and{ , ,... ,... }.

[0062] Will exist The part in is used as an indicator to describe the consistency of the style of the i-th patch. This indicator is obtained by To obtain, it is expressed as:

[0063]

[0064] in ∈[0,1] represents the style consistency degree of the i-th patch. The larger the value, the better the consistency.

[0065] Taking into account Larger Patches have good style consistency, which means they provide less information and cannot effectively help us learn good style transformations to reduce style inconsistency. Therefore, this paper only selects relatively small The patch is used as a candidate patch for subsequent processing, so the candidate patch is obtained:

[0066]

[0067] in is a preset threshold value, which is 0.7 in the embodiment. Represents a candidate patch set. The same method is used to obtain the candidate patch set .

[0068] The traditional contrast loss assigns equal weights to all negative samples. However, this setting is somewhat unreasonable because different The negative patches of clearly show different importances. Although this different importance may be perceived when the network has learned to distinguish style differences, it is still challenging in the early stages of training. Considering Contains the degree of style consistency, introducing It is suitable to assist contrastive learning as a soft label. ∈ As negative samples, ∈ As a positive sample for soft contrast learning, the formula is expressed as:

[0069]

[0070] in It is a local soft contrast style loss; According to formula (4), The corresponding patch extracted from By patch The deep features extracted by the encoder ReLU4_1 are then obtained through the projection network. N represents the number of candidate patches in each set. Here τ and are two hyperparameters, τ is the temperature coefficient set to 0.03, is the total number of negative samples, set to 1024. The negative sample is a queue of size 1024, which is slowly updated by momentum, and N negative samples are updated for each picture.

[0071] The aforementioned patch-based soft contrast loss learns local style information by maximizing the mutual information between corresponding patches in the input and output images. However, this patch-based process can overemphasize local style consistency while neglecting global style alignment between foreground and background, potentially introducing local style inconsistencies, especially in backgrounds with uneven style distribution. Furthermore, since no ground-truth labels are provided during training, preserving foreground details in the transfer is also challenging. Therefore, this paper proposes a content-preserving process (i.e., a comprehensive content optimization strategy) consisting of two components: global style alignment and content preservation.

[0072] Global style alignment aims to eliminate potential local style inconsistencies by globally aligning the style of the foreground with the background. First, we use the adversarial loss To preliminarily align the styles of the foreground and background areas, the expression is:

[0073]

[0074] Where D represents the discriminator. Next, we introduce another distribution-based style alignment loss (overall style loss) To further enhance the style alignment effect:

[0075]

[0076] in represents the downsampled foreground mask M. represents the output of the encoder ReLU layer i-1, represents the average value, Represents standard deviation.

[0077] Content preservation aims to keep the details of the transferred foreground area consistent with the original content. To this end, a structural similarity loss is first introduced to maintain the consistency of the structure before and after the transformation. The expression is:

[0078]

[0079] in Indicates that the feature tensor is extracted by the encoder ReLU4_1. In addition, a detail similarity loss based on contrastive learning is further formulated to better preserve details.

[0080] Specifically, for each and , first randomly select a background, denoted as , then use with Same foreground generation .Will and Input generator to get harmonized output Then, use the bounding box to crop , get the anchor point The detail similarity loss can be expressed as:

[0081]

[0082] in represents a negative sample. τ is a temperature coefficient set to 0.03, is the total number of negative samples, set to 1024. The negative sample is a queue of size 1024, which is slowly updated by momentum, and 1 negative sample is updated for each picture.

[0083] Network structure and implementation details. Training is performed using the Adam optimizer with β1=0 and β2=0.9. By default, the learning rate for the discriminator is 0.0004 and the learning rate for the generator is 0.0001. During training and testing, the input images are resized to 256×256.

[0084] Step 3: The trained neural network model receives the image that needs to be rendered for lighting consistency, completes the harmony processing, and outputs the image.

[0085] First, load the weights of the lighting consistency rendering network model trained in step 2 and update the parameters in the model. Then, pass the synthesized image to be harmonized as input data into the generator network model. The input data passes through the generator to obtain the harmonized output image.

[0086] The present application also provides a position-sensitive illumination consistency rendering device, including the following modules:

[0087] The dataset creation module is used to create the image dataset required for network training.

[0088] The model building and training module is used to build a basic lighting consistency rendering model based on soft contrast learning, including:

[0089] (1) Theoretical modeling of basic lighting consistency rendering model based on soft contrast learning.

[0090] (2) Construct a training network for illumination consistency rendering based on soft contrast learning.

[0091] (3) Training is performed to generate a network model with position-sensitive image coordination effects, that is, a basic lighting consistency rendering model based on soft contrast learning is obtained.

[0092] The image harmony module uses the neural network model trained in the model building and training module to receive images that need to be rendered for lighting consistency, and outputs the images after completing the harmony processing.

[0093] In one possible implementation, the dataset creation module uses images from the WikiaArt dataset as background images and extracts photographic foreground objects from the COCO dataset using instance segmentation masks provided by the COCO dataset. The foreground objects are randomly inserted into the background images using the instance segmentation masks to form a composite image.

[0094] In one possible implementation, the process of theoretical modeling of the model construction and training module includes:

[0095] A generator including a style transfer module takes the composite image, background image, and foreground mask as input and outputs a harmonious image. Secondly, a discriminator is proposed to guide the generator to produce more realistic and harmonious images.

[0096] The generator is trained by continuously optimizing the loss. The total loss of the generator includes: local soft contrast style loss for achieving local style consistency, global style loss for comprehensive content optimization, adversarial loss, structural similarity loss and detail similarity loss.

[0097] In one possible implementation, the model building and training module's training network for illumination-consistent rendering based on soft contrast learning includes a generator and a discriminator. The generator consists of an encoder, a style transfer module, and a decoder. The encoder uses a pre-trained VGG-19 network, and the decoder's design mirrors the encoder's architecture by using upsampling operations. The style transfer module captures background style information and applies it to foreground objects. The discriminator assesses whether there are discordant areas in the harmonized image.

[0098] In one possible implementation, the discriminator includes a light spatial encoder Ds and a lightweight autoencoder Da. Specifically, six DS blocks are applied within the light spatial encoder Ds, where each DS block contains a convolutional layer with a kernel size of 4 and a stride of 2, followed by a batch normalization (BN) layer and a LeakyReLU activation. Da is a small autoencoder consisting of two DS blocks and two US blocks. Each DS block in Da sequentially contains a Conv layer with a kernel size of 3 and a stride of 1, a BN layer, and a LeakyReLU activation. Each US block has the same structure as the DS block in Da, except for the ReLU activation. The downsampling DS block uses LeakyReLU activation while the upsampling US block uses ReLU activation.

[0099] In one possible implementation, the model building and training module is supervised by soft contrastive learning. First, a position-aware segmentation module is used to obtain candidate patches to be reconciled from the composite image. Then, soft contrastive learning is used to transform the style of the foreground in each patch to achieve local consistency.

[0100] To preserve the details of the transferred foreground, a content-preserving process (i.e., comprehensive content optimization strategy) is proposed, which consists of two parts: global style alignment and content preservation.

[0101] Global style alignment uses adversarial loss to initially align the styles of the foreground and background regions, and further enhances the style alignment effect through distribution-based style alignment loss (i.e., overall style loss):

[0102] The content is preserved by using structural similarity loss to maintain the consistency of the structure before and after the transformation, and by using detail similarity loss based on contrastive learning to better preserve details.

[0103] In a possible implementation, the image harmonization module performs harmonization processing, including:

[0104] First, the weights of the lighting consistency rendering network model trained in the model building and training module are loaded and the model parameters are updated. Second, the synthesized image to be harmonized is passed as input data to the generator network model. The input data passes through the generator to obtain the harmonized output image.

[0105] Experimental comparison: The proposed method successfully achieves position-sensitive illumination consistency rendering, while the existing advanced methods (PHDNet, ProPIH) do not have the ability to perceive style based on position. The results are as follows Figure 3 As shown. In addition, we also conducted a user study to objectively and quantitatively evaluate the 12 comparison methods. We first randomly selected background images, foreground images and corresponding masks, generated 100 synthetic images, and obtained their harmonious results through all comparison methods. Therefore, we obtained 100 groups of harmonious images, each group containing 12 harmonious images of the same synthetic image generated by different methods. For each group of images, 66 different combinations can be obtained to form a pair of different images. Therefore, we can get 6,600 pairs of harmonious images for user research. We organized 15 participants to score these 6,600 pairs of harmonious images and ranked each pair of images according to visual quality. Then, we used the Bradley-Terry (BT) model (Bradley and Terry, 1952) to calculate the user study scores of the comparison methods, and the results are shown as follows. Figure 4 shown.

[0106] The embodiment of the present application further provides an electronic device, comprising a processor and a memory;

[0107] The memory is used to store computer programs;

[0108] The processor is used to implement any method described in this application when executing the program stored in the memory.

[0109] In a possible implementation, the electronic device of the embodiment of the present application further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.

[0110] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0111] The communication interface is used for communication between the above electronic device and other devices.

[0112] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0113] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0114] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any method described in the present application is implemented.

[0115] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the methods described in the present application.

[0116] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0117] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0118] Each embodiment in this specification is described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other.

[0119] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A position-sensitive illumination consistency rendering method, characterized in that: The following steps are involved: First, create a data set and establish the image data set required for network training; Then build a basic lighting consistency rendering model based on soft contrast learning: (1) Theoretical modeling of basic illumination consistency rendering model based on soft contrast learning; (2) Constructing a training network for illumination consistency rendering based on soft contrast learning; (3) Training to generate a network model with position-sensitive image coordination effects, that is, obtaining a basic lighting consistency rendering model based on soft contrast learning; The trained neural network model receives the image that needs to be rendered for lighting consistency, completes the harmony processing, and outputs the image; The training process is supervised by soft contrast learning. First, a position-aware segmentation module is used to obtain candidate patches that need to be coordinated from the composite image. Then, soft contrast learning is used to transform the style of the foreground in each patch to achieve local consistency. To preserve the details of the transferred foreground, a content-preserving process is proposed, which consists of two parts: global style alignment and content preservation; Global style alignment uses adversarial loss to initially align the styles of foreground and background areas, and further enhances the style alignment effect through distribution-based style alignment loss: The content is preserved by using structural similarity loss to maintain the consistency of the structure before and after the transformation, and by using detail similarity loss based on contrastive learning to better preserve details.

2. The position-sensitive illumination consistency rendering method according to claim 1, characterized in that: When making the dataset, images from the WikiaArt dataset were used as background images, and the instance segmentation masks provided by the COCO dataset were used to extract photographic foreground objects from the COCO dataset; the foreground objects were randomly inserted into the background images using instance segmentation masks to form synthetic images.

3. The position-sensitive illumination consistency rendering method according to claim 1, characterized in that: The theoretical modeling process includes: A generator including a style transfer module takes a composite image, a background image, and a foreground mask as input and outputs a harmonious image. Secondly, a discriminator is proposed to guide the generator to produce more realistic and harmonious images. The generator is trained by continuously optimizing the loss. The total loss of the generator includes: local soft contrast style loss for achieving local style consistency, global style loss for comprehensive content optimization, adversarial loss, structural similarity loss and detail similarity loss.

4. The position-sensitive illumination consistency rendering method according to claim 1, characterized in that: A training network for illumination consistency rendering based on soft contrast learning, including a generator and a discriminator; The generator consists of an encoder, a style transfer module, and a decoder; the encoder uses a pre-trained VGG-19 network, and the decoder is designed to reflect the encoder architecture by using upsampling operations; The style transfer module is used to capture the style information of the background and apply it to the foreground object; the discriminator is used to evaluate whether there are discordant regions in the harmonized image.

5. The position-sensitive illumination consistency rendering method according to claim 4, characterized in that: The discriminator includes a light spatial encoder and a lightweight autoencoder; specifically, six downsampling blocks are applied in the light spatial encoder, where each downsampling block contains a convolution layer with a kernel size of 4 and a stride of 2, followed by a batch normalization layer and an activation function layer; the lightweight autoencoder includes two downsampling blocks and two upsampling blocks; each downsampling block in the lightweight autoencoder contains a convolution layer with a kernel size of 3 and a stride of 1, a batch normalization layer and an activation function layer in sequence; and each upsampling block has the same structure as the downsampling block, except for the activation function; the downsampling block uses LeakyReLU activation while the upsampling block uses ReLU activation.

6. The position-sensitive illumination consistency rendering method according to claim 1, characterized in that: The process of harmonizing the trained neural network model includes: First, the trained light consistency rendering network model weights are loaded and the parameters in the model are updated. Second, the synthesized image to be harmonized is passed as input data into the generator network model, and the input data is passed through the generator to obtain the harmonized output image.

7. A position-sensitive illumination consistency rendering device, characterized in that: Includes the following modules: Dataset creation module, used to create image datasets required for network training; The model building and training module is used to build a basic lighting consistency rendering model based on soft contrast learning, including: (1) Theoretical modeling of basic illumination consistency rendering model based on soft contrast learning; (2) Constructing a training network for illumination consistency rendering based on soft contrast learning; (3) Training to generate a network model with position-sensitive image coordination effects, that is, obtaining a basic lighting consistency rendering model based on soft contrast learning; The image harmonization module uses the neural network model trained in the model building and training module to receive the image that needs to be rendered for lighting consistency, and outputs the image after completing the harmonization process; The model building and training process of the training module is supervised by soft contrast learning. First, the position-aware segmentation module is used to obtain candidate patches that need to be coordinated from the composite image. Then, soft contrast learning is used to transform the style of the foreground in each patch to achieve local consistency. To preserve the details of the transferred foreground, a content-preserving process is proposed, which consists of two parts: global style alignment and content preservation; Global style alignment uses adversarial loss to initially align the styles of foreground and background areas, and further enhances the style alignment effect through distribution-based style alignment loss: The content is preserved by using structural similarity loss to maintain the consistency of the structure before and after the transformation, and by using detail similarity loss based on contrastive learning to better preserve details.

8. An electronic device, characterized in that: including processor and memory; The memory is used to store computer programs; The processor is configured to implement the lighting consistency rendering method according to any one of claims 1 to 6 when executing the program stored in the memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the lighting consistency rendering method according to any one of claims 1 to 6 is implemented.

Images