Single picture relighting method, system, terminal and storage medium based on prior knowledge

By using a convolutional neural network based on prior knowledge, the problem of long rendering time, large data volume, and slow rendering of traditional relighting technology is solved. It achieves fast rendering of single image relighting with low data requirements, making it suitable for mobile devices.

CN113409186BActive Publication Date: 2024-11-22SHANGHAI TECH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110731039.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-30
Publication Date
2024-11-22
Estimated Expiration
2041-06-30

AI Technical Summary

Technical Problem

Existing relighting technology requires pre-capturing all light sources in the device, which is time-consuming, generates a lot of data, and has slow rendering, making it difficult to apply effectively on mobile devices.

Method used

A convolutional neural network based on prior knowledge is used to obtain the light and shadow of objects through pre-training with a dome light field. The encoder and decoder are used to extract image features and replace ambient light to achieve relighting of a single image.

Benefits of technology

It eliminates the need to pre-capture all the light sources, offers fast rendering speeds, consumes minimal data, enables real-time relighting under existing hardware conditions, and requires only tens of MB of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113409186B_ABST
    Figure CN113409186B_ABST
Patent Text Reader

Abstract

The present application provides a single image relighting method, system, terminal and storage medium based on prior knowledge, including: given an object image for relighting and a panoramic image as ambient light; inputting the object image and the panoramic image into a convolutional neural network based on prior knowledge; extracting the high-dimensional features of the object in the object image and the ambient light through the encoder of the convolutional neural network; replacing the extracted ambient light with the given ambient light, and inputting the image as ambient light and the extracted high-dimensional features into the decoder in the convolutional neural network, and arranging jump connections in the encoder and decoder to obtain the relighted image. The present invention has no pre-requirements, and there is no need to pre-shoot all optical fibers in the dome light field; the rendering is fast, and the convolutional neural network can achieve real-time under existing hardware conditions; compared with the several GB required for the dome light field to relight a single object, the space occupied by the present invention is only tens of MB.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of single image relighting, and in particular to a single image relighting method, system, terminal and storage medium based on prior knowledge. Background Art

[0002] Relighting technology usually refers to modifying the light and shadow of a specific object to the result under the influence of a specified environment map. The current relighting technology is mainly the dome light field (Light Field Stage). Traditional relighting technology requires all light to be captured in the device in advance, which has problems such as time-consuming, large amount of data, slow rendering, etc., and its performance on mobile terminals is poor. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a method for solving the technical problems of the traditional heavy lighting technology in the prior art, such as many pre-processing steps, slow rendering, and large amount of data.

[0004] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a single image relighting method based on prior knowledge, including: giving an object image for relighting and a panoramic image as ambient light; inputting the object image and the panoramic image into a convolutional neural network based on prior knowledge; extracting high-dimensional features of the object in the object image and the ambient light through the encoder of the convolutional neural network; replacing the extracted ambient light with the given ambient light, and inputting the image as ambient light and the extracted high-dimensional features into the decoder in the convolutional neural network, and arranging jump connections in the encoder and the decoder to obtain the relighted image.

[0005] In some embodiments of the first aspect of the present application, the training process of the convolutional neural network based on prior knowledge includes: acquiring multiple images of the photographed object under different lighting conditions based on the dome light field; and using the multiple images as training sets to train the convolutional neural network.

[0006] In some embodiments of the first aspect of the present application, the method also includes using a multi-scale progressive training method to train the convolutional neural network, which includes first using a 64×64 size image to input the convolutional neural network for training, and after convergence, doubling the size of the output image each time and then inputting it into the convolutional neural network for training until the image size reaches 512×512.

[0007] In some embodiments of the first aspect of the present application, the process of extracting high-dimensional features of objects in object images and the ambient light in which they are located includes: the encoder encodes a 512×512 input image into 512 layers of 16×16 features through several layers of convolutional layers, activation layers, and pooling layers; wherein the middle 3 layers of the 512 layers of 16×16 features are set to the ambient light in which the input image is located.

[0008] In some embodiments of the first aspect of the present application, the method for obtaining the relighted image includes: the decoder in the pre-trained convolutional neural network decodes 512 layers of 16×16 features and corresponding jump connections into a 512×512 output image through several layers of upsampling layers, convolution layers and activation layers.

[0009] In some embodiments of the first aspect of the present application, the convolutional neural network also sets a loss function; the loss function includes any one or more combinations of the following: the logarithmic 1-norm distance between the ambient light output from the input image through the encoder and the real ambient light; the 1-norm distance and multi-scale structural similarity between the output image directly output through the decoder without replacing the ambient light and the input image; the 1-norm distance and multi-scale structural similarity between the output image output through the decoder after replacing the ambient light and the real image; the generative adversarial network loss function of the output image output through the decoder after replacing the ambient light and the real image; the cross entropy loss between the output segmentation and the real segmentation; the 1-norm distance and multi-scale structural similarity of the two frames of output images output through the decoder after replacing the ambient light of two adjacent frames with video continuity after corresponding optical flow transformation.

[0010] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a single-image relighting system based on prior knowledge, including: an image input module, used to give an object image for relighting and a panoramic image as ambient light; input the object image and the panoramic image into a convolutional neural network based on prior knowledge; a feature extraction module, used to extract high-dimensional features of objects in the object image and the ambient light through the encoder of the convolutional neural network; a relighting module, used to replace the extracted ambient light with the given ambient light, and input the image as ambient light and the extracted high-dimensional features into the decoder in the convolutional neural network, and arrange jump connections in the encoder and the decoder to obtain the relighted image.

[0011] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the single image relighting method based on prior knowledge is implemented.

[0012] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides an electronic terminal, including: a processor and a memory; the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal executes the single image relighting method based on prior knowledge.

[0013] As described above, the single image relighting method, system, terminal and storage medium based on prior knowledge of the present application have the following beneficial effects:

[0014] 1) There is no pre-requisite, the steps are simple, and only a picture of a given object is needed to complete it, without the need to pre-photograph all optical fibers in the dome light field;

[0015] 2) Fast rendering speed, convolutional neural network can achieve real-time under existing hardware conditions;

[0016] 3) It occupies less data. Compared with the several GB required for re-lighting a single object by a dome light field, the space occupied by the present invention is only tens of MB. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flowchart of a single image relighting method based on prior knowledge in one embodiment of the present application.

[0018] Figure 2 Shown is a structural schematic diagram of a single image re-lighting system based on prior knowledge in one embodiment of the present application.

[0019] Figure 3 Shown is a schematic diagram of the structure of an electronic terminal in one embodiment of the present application. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0022] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0023] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "comprising" and "including" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. It should be further understood that the terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition occur only when a combination of elements, functions, or operations are inherently mutually exclusive in some manner.

[0024] The present invention uses the dome light field to complete the pre-training of the convolutional neural network, giving the convolutional neural network prior knowledge of the light and shadow of a certain type of object. Afterwards, the present invention can quickly re-light other objects of the same type, effectively solving the problem that the traditional re-lighting technology needs to capture all the light in the device in advance, which is time-consuming, data-intensive, and slow to render. In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention are further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the invention.

[0025] like Figure 1 As shown, a flow chart of a single image relighting method based on prior knowledge in one embodiment of the present invention is shown. It should be noted that the single image relighting method in the present invention can be applied to computer devices, such as desktop computers, laptops, tablet computers, smart phones, smart bracelets, smart watches, smart helmets, etc.; it can also be applied to servers, and the servers can be arranged on one or more physical servers according to various factors such as functions and loads, and can also be composed of distributed or centralized server clusters.

[0026] In this embodiment, the single image relighting method based on prior knowledge mainly includes steps S11 to S14, and each step will be further explained below.

[0027] Step S11: Given an object image for relighting and a panoramic image as ambient light. It should be noted that the ambient light refers to various types of light used for lighting in daily life, such as artificial light in life or interior scenes in drama creation. Ambient light can be used to create various types of light, such as cloudy, sunny, day, night, dusk, dawn, etc. Common ambient light such as sunlight, light generated by light sources such as lamps, and light reflected from glass curtain walls of buildings are also a type of ambient light.

[0028] Step S12: inputting the object image and the panoramic image into a convolutional neural network based on prior knowledge.

[0029] The convolutional neural network is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is constructed by imitating the visual perception mechanism of organisms and can perform supervised learning and unsupervised learning. Generally speaking, the structure of a convolutional neural network includes an input layer, a hidden layer, and an output layer.

[0030] The input layer can process multi-dimensional data. The more common ones are: the input layer of a one-dimensional convolutional neural network receives a one-dimensional or two-dimensional array (a one-dimensional array is usually time or spectrum sampling, and a two-dimensional array may contain multiple channels), the input layer of a two-dimensional convolutional neural network receives two-dimensional or three-dimensional data, and the input layer of a three-dimensional convolutional neural network receives a four-dimensional array.

[0031] Hidden layers include three common structures: convolutional layers, pooling layers, and fully connected layers. The function of the convolutional layer is to extract features from the input data. It contains multiple convolution kernels. Each element of the convolution kernel corresponds to a weight coefficient and a bias, similar to a neuron in a feedforward neural network. The convolutional layer contains an activation function (such as the ReLU function) to help express complex features. The function of the pooling layer is that after the convolutional layer extracts features, the feature map output by the pooling layer will be passed to the pooling layer for feature selection and information filtering. The pooling layer contains a preset pooling function, which replaces the result of a single point in the feature map with the feature map statistics of its adjacent area. The fully connected layer is located at the last part of the hidden layer of the convolutional neural network and only transmits signals to other fully connected layers. Its function is to nonlinearly combine the extracted features to obtain the output, so as to use the existing high-order features to achieve the learning goal.

[0032] The upstream of the output layer is usually the fully connected layer at the end of the hidden layer, which uses a logistic function or a normalized exponential function (softmax function) to output the classification label; in object recognition problems, the output layer can be designed to output the center coordinates, size, and classification of the object; in image semantic segmentation, the output layer directly outputs the classification result of each pixel.

[0033] The training of convolutional neural networks mainly consists of the following two stages: the first stage is the stage of data propagation from low level to high level, that is, the forward propagation stage; the second stage is the stage of propagating the error from high level to low level when the result obtained by the forward propagation does not meet the expectation, that is, the back propagation stage. The specific training steps include: 1. Initialize the network weights; 2. The input data is forward propagated through the convolution layer, downsampling layer, and fully connected layer to obtain the output value; 3. Calculate the error between the output value of the network and the target value; 4. When the error is greater than our expected value, the error is transmitted back to the network, and the error of the fully connected layer, downsampling layer, and convolution layer is obtained in turn; the error of each layer can be understood as the total error of the network; 5. If the error is greater than the expected value, return to the above step 2 to continue training; 6. If the total error is equal to or less than the expected value, the training ends.

[0034] In this embodiment, the training process of the convolutional neural network based on prior knowledge includes: acquiring multiple images of the photographed object based on the dome light field, and using the multiple images as training sets to train the convolutional neural network. The dome light field can be a spherical lighting shooting system containing about 150 256-level controllable LED light sources, each LED light source has 6 high-brightness LED lamp beads, these lamp beads are divided into two groups for independent control, and each group is installed with different polarizers; the light source is evenly distributed on an aluminum alloy frame sphere with a diameter of about 3 meters, all lights change brightness at a frequency of up to 1000Hz, and are synchronized with a high-speed camera and thirty industrial cameras at the millisecond level. Every time all lights change brightness, the high-speed camera takes a picture. In this way, the performance of the photographed object under different lighting conditions can be collected, and these performance data can be used as training data for training the convolutional neural network.

[0035] Preferably, in the training process of the convolutional neural network based on prior knowledge, the convolutional neural network is trained in advance using data captured by the dome light field on a certain type of object, so that the convolutional neural network acquires prior knowledge of light and shadow for the certain type of object.

[0036] Preferably, in the training process of the convolutional neural network based on prior knowledge, a multi-scale progressive training method is used, for example, the network input and output images of size 64×64 are first trained, and after convergence, the output image size is doubled each time for training until it reaches a size of 512×512.

[0037] Step S13: extracting high-dimensional features of the object and the ambient light in the object image through the encoder of the convolutional neural network.

[0038] For example, the pre-trained convolutional neural network includes an encoder and a decoder. The encoder encodes a 512×512 input image into 512 layers of 16×16 features through several layers of convolutional layers, activation layers, and pooling layers. The three layers in the middle of the 512 layers of 16×16 features are artificially defined as the ambient light of the input image, that is, a 16×16 ambient light expansion map, so as to complete the subsequent replacement of the ambient light.

[0039] It should be noted that skip connection is usually used in residual networks. Its function is to solve the problem of gradient explosion and gradient vanishing during training in relatively deep networks. In general neural networks, it is necessary to pass through the I+1 layer to reach the I+2 layer, while in the residual block, a[I] is directly passed through the I+2 layer, so that a[l+2]=g(z[l+2]+a[l]), that is, in addition to being related to the z of the I+2 layer, the a[l+2] activation unit is also related to the a of the I layer. This is the skip connection.

[0040] Step S14: Replace the extracted ambient light with the given ambient light, and input the picture of the ambient light and the extracted high-dimensional features into the decoder in the convolutional neural network, and arrange jump connections in the encoder and decoder to obtain a re-lit picture.

[0041] The decoder in the pre-trained convolutional neural network decodes 512 layers of 16×16 features and corresponding jump connections into a 512×512 output image through several layers of upsampling layers, convolution layers, and activation layers. The three layers in the middle of the 512 layers of 16×16 features are artificially defined as the ambient light of the input image, that is, the 16×16 ambient light expansion map, so as to complete the subsequent replacement of the ambient light. In the face relighting task, in addition to the three layers of 512×512 representing the color of the output image, the network is also required to output two layers to separate the skin and hair of the face, so as to strengthen the network's understanding of the difference between the two and prevent jitter.

[0042] In some examples, the convolutional neural network also sets a loss function, which is used to evaluate the degree to which the predicted value of the model is different from the true value. The better the loss function, the better the performance of the model. In the convolutional neural network in this embodiment, the loss function that can be constructed includes but is not limited to the following: the logarithmic 1-norm distance between the ambient light output from the input image through the encoder and the real ambient light; the 1-norm distance and multi-scale structural similarity between the output image directly output through the decoder without replacing the ambient light and the input image; the 1-norm distance and multi-scale structural similarity between the output image output through the decoder after replacing the ambient light and the real image; the generative adversarial network loss function between the output image output through the decoder after replacing the ambient light and the real image; the cross entropy loss between the output segmentation and the real segmentation; the 1-norm distance and multi-scale structural similarity between the two adjacent frames with video continuity after replacing the ambient light and the two frames of output images output through the decoder after the corresponding optical flow transformation.

[0043] like Figure 2 , which shows a schematic diagram of the structure of a single image relighting system based on prior knowledge in one embodiment of the present invention. The single image relighting system 200 of this embodiment includes an image input module 201 , a feature extraction module 202 and a relighting module 203 .

[0044] The image input module 201 is used to provide an object image for relighting and a panoramic image as ambient light; and input the object image and the panoramic image into a convolutional neural network based on prior knowledge.

[0045] It should be noted that the convolutional neural network is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is constructed by imitating the visual perception mechanism of organisms and can perform supervised learning and unsupervised learning. Generally speaking, the structure of a convolutional neural network includes an input layer, a hidden layer, and an output layer. The structure of a convolutional neural network has been described in detail in the previous article and will not be repeated here.

[0046] The feature extraction module 202 is used to extract high-dimensional features of objects and the ambient light in the object image through the encoder of the convolutional neural network.

[0047] For example, the pre-trained convolutional neural network includes an encoder and a decoder. The encoder encodes a 512×512 input image into 512 layers of 16×16 features through several layers of convolutional layers, activation layers, and pooling layers. The three layers in the middle of the 512 layers of 16×16 features are artificially defined as the ambient light of the input image, that is, a 16×16 ambient light expansion map, so as to complete the subsequent replacement of the ambient light.

[0048] It should be noted that skip connection is usually used in residual networks. Its function is to solve the problem of gradient explosion and gradient vanishing during training in relatively deep networks. In general neural networks, it is necessary to pass through the I+1 layer to reach the I+2 layer, while in the residual block, a[I] is directly passed through the I+2 layer, so that a[l+2]=g(z[l+2]+a[l]), that is, in addition to being related to the z of the I+2 layer, the a[l+2] activation unit is also related to the a of the I layer. This is the skip connection.

[0049] The relighting module 203 is used to replace the extracted ambient light with the given ambient light, and input the picture of the ambient light and the extracted high-dimensional features into the decoder in the convolutional neural network, and arrange jump connections in the encoder and decoder to obtain the relighting picture.

[0050] The decoder in the trained convolutional neural network decodes 512 layers of 16×16 features and corresponding jump connections into a 512×512 output image through several layers of upsampling layers, convolution layers, and activation layers. The 3 layers in the middle of the 512 layers of 16×16 features are artificially defined as the ambient light of the input image, that is, the 16×16 ambient light expansion map, so as to complete the subsequent replacement of the ambient light. In the face relighting task, in addition to the 3 layers of 512×512 representing the color of the output image, the network is also required to output 2 layers to separate the skin and hair of the face, so as to strengthen the network's understanding of the difference between the two and prevent jitter.

[0051] In some examples, the system 200 also includes a loss function module 204, which is used to evaluate the degree to which the predicted value of the model is different from the true value. The better the loss function, the better the performance of the model. In the convolutional neural network in this embodiment, the loss function that can be constructed includes but is not limited to the following: the logarithmic 1-norm distance between the ambient light output from the input image through the encoder and the real ambient light; the 1-norm distance and multi-scale structural similarity between the output image directly output through the decoder without replacing the ambient light and the input image; the 1-norm distance and multi-scale structural similarity between the output image output through the decoder after replacing the ambient light and the real image; the generative adversarial network loss function between the output image output through the decoder after replacing the ambient light and the real image; the cross entropy loss between the output segmentation and the real segmentation; the 1-norm distance and multi-scale structural similarity between the two adjacent frames with video continuity after replacing the ambient light and the two frames of output images output through the decoder after corresponding optical flow transformation.

[0052] It should be understood that the division of the modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0053] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0054] like Figure 3 As shown, a schematic diagram of the structure of an electronic terminal in an embodiment of the present invention is shown. The electronic terminal provided in this example includes: a processor 31, a memory 32, and a communicator 33; the memory 32 is connected to the processor 31 and the communicator 33 through a system bus and completes communication with each other, the memory 32 is used to store computer programs, the communicator 33 is used to communicate with other devices, and the processor 31 is used to run the computer program, so that the electronic terminal executes each step of the single image relighting method based on prior knowledge as described above.

[0055] The system bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage.

[0056] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0057] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the single image relighting method based on prior knowledge.

[0058] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0059] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0060] In summary, the present application provides a single image relighting method, system, terminal and storage medium based on prior knowledge, 1) no pre-requirements, simple steps, only one image of a given object is needed to complete, no need to pre-shoot all optical fibers in the dome light field; 2) fast rendering speed, convolutional neural network can achieve real-time under existing hardware conditions; 3) less data is occupied, compared with the several GB required for the dome light field to relight a single object, the space occupied by the present invention is only tens of MB. Therefore, the present application effectively overcomes various shortcomings in the prior art and has a high industrial utilization value.

[0061] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A single image relighting method based on prior knowledge, characterized in that: include: Given an object image for relighting and a panoramic image as ambient light; Inputting the object image and the panoramic image into a convolutional neural network based on prior knowledge; The training process of the convolutional neural network based on prior knowledge includes: acquiring multiple images of the photographed object under different lighting conditions based on the dome light field; using the multiple images as training sets to train the convolutional neural network; using a multi-scale progressive training method to train the convolutional neural network, which includes first using a 64×64 size image to input the convolutional neural network for training, and after convergence, doubling the size of the output image each time and then inputting it into the convolutional neural network for training until the image size reaches 512×512; Extracting high-dimensional features of objects and ambient light in the object image through the encoder of the convolutional neural network; The extracted ambient light is replaced with the given ambient light, and the picture of the ambient light and the extracted high-dimensional features are input into the decoder in the convolutional neural network, and skip connections are arranged in the encoder and the decoder to obtain a re-lit picture; the re-lit picture is obtained in the following manner: the decoder in the pre-trained convolutional neural network decodes 512 layers of 16×16 features and corresponding skip connections into an output image of 512×512 through several layers of upsampling layers, convolution layers and activation layers; The convolutional neural network also sets a loss function; the loss function includes any one or more combinations of the following: the logarithmic 1-norm distance between the ambient light output from the input image through the encoder and the real ambient light; the 1-norm distance and multi-scale structural similarity between the output image directly output through the decoder without replacing the ambient light and the input image; the 1-norm distance and multi-scale structural similarity between the output image output through the decoder after replacing the ambient light and the real image; the generative adversarial network loss function between the output image output through the decoder after replacing the ambient light and the real image; the cross entropy loss between the output segmentation and the real segmentation; the 1-norm distance and multi-scale structural similarity between the two adjacent frames with video continuity after replacing the ambient light and the two frames of output images output through the decoder after the corresponding optical flow transformation.

2. The single image relighting method based on prior knowledge according to claim 1, characterized in that: The process of extracting high-dimensional features of objects in object images and the ambient light in which they are located includes: the encoder encodes a 512×512 input image into 512 layers of 16×16 features through several layers of convolutional layers, activation layers, and pooling layers; wherein the middle 3 layers of the 512 layers of 16×16 features are set as the ambient light in which the input image is located.

3. A single image relighting system based on prior knowledge, characterized in that: include: An image input module is used to provide an object image for relighting and a panoramic image as ambient light; Inputting the object image and the panoramic image into a convolutional neural network based on prior knowledge; The training process of the convolutional neural network based on prior knowledge includes: acquiring multiple images of the photographed object under different lighting conditions based on the dome light field; using the multiple images as training sets to train the convolutional neural network; using a multi-scale progressive training method to train the convolutional neural network, which includes first using a 64×64 size image to input the convolutional neural network for training, and after convergence, doubling the size of the output image each time and then inputting it into the convolutional neural network for training until the image size reaches 512×512; A feature extraction module, used to extract high-dimensional features of objects and ambient light in the object image through the encoder of the convolutional neural network; A relighting module is used to replace the extracted ambient light with the given ambient light, and input the picture of the ambient light and the extracted high-dimensional features into the decoder in the convolutional neural network, and arrange jump connections in the encoder and the decoder to obtain the relighted picture; the relighted picture is obtained in the following manner: the decoder in the pre-trained convolutional neural network decodes 512 layers of 16×16 features and corresponding jump connections into a 512×512 output image through several layers of upsampling layers, convolution layers and activation layers; A loss function module is used to set a loss function for the convolutional neural network; wherein the loss function includes any one or more combinations of the following: the logarithmic 1-norm distance between the ambient light output from the input image through the encoder and the real ambient light; the 1-norm distance and multi-scale structural similarity between the output image directly output through the decoder without replacing the ambient light and the input image; the 1-norm distance and multi-scale structural similarity between the output image output through the decoder after replacing the ambient light and the real image; the generative adversarial network loss function of the output image output through the decoder after replacing the ambient light and the real image; the cross entropy loss of the output segmentation and the real segmentation; the 1-norm distance and multi-scale structural similarity of the two frames of output images output through the decoder after replacing the ambient light of two adjacent frames with video continuity after corresponding optical flow transformation.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the single image relighting method based on prior knowledge as described in any one of claims 1 to 2 is implemented.

5. An electronic terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the terminal executes the single image relighting method based on prior knowledge as claimed in any one of claims 1 to 2.