Direct and indirect lighting separation method based on event camera guidance

Abstract

The method for separating direct and indirect illumination guided by an event camera belongs to the field of illumination separation methods, and includes: shooting event signals triggered by shadow changes, estimating initial grayscale separation results, optimizing initial grayscale separation results, and color compensation of illumination separation results. The present invention only requires shooting one photo and one piece of event data in the scene, which shortens the data shooting time and can be applied to dynamic scenes. The event triggering model of the event camera enables it to discretely record the continuous change information in the scene illumination changes, while achieving high temporal resolution and reducing data throughput. The present invention successfully integrates the images output by the event camera and the traditional machine vision camera, and can reconstruct high-quality color direct and indirect illumination separation results from a single image and an event signal triggered by illumination changes. At the same time, a deep learning method is used to effectively combine scene images and event signals, and has a better separation effect in high-frequency indirect illumination scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of illumination separation methods, and in particular relates to a direct illumination and indirect illumination separation method based on event camera guidance. Background Art

[0002] When a scene is illuminated by a light source, the radiance of each point consists of two parts: direct illumination and indirect illumination. The direct illumination component represents the part of the illumination that enters the camera after only one reflection from the light source to the surface of the object. At the same time, the illumination interacts with the environment in a complex way, leading to phenomena such as mutual reflection, scattering, and diffusion, which produce indirect illumination. The separation of direct and global illumination components can provide material properties of the scene and reveal information about complex shape-material-lighting interactions. For example, eliminating the influence of indirect illumination can improve the performance of shape recovery techniques such as photometric stereo measurement and structured light scanning. Such techniques often assume that there is only direct illumination and ignore the influence of indirect illumination.

[0003] How to accurately separate direct and indirect illumination components is a challenging problem in computer vision. The direct illumination component travels the shortest distance among all the light paths reaching the pixel. The different arrival times of the direct and indirect illumination components allow them to be distinguished in time sequence. Ultra-high temporal resolution cameras such as time-of-flight (ToF) cameras and single-photon sensors have been used to distinguish the arrival time differences between direct and indirect illumination components on a picosecond scale. Although these methods have good separation effects, they have extremely high requirements on the temporal resolution of the sensor. Such sensors are often expensive, which limits the scope of application of such methods. In addition, although such sensors have high temporal resolution, it is difficult to take into account spatial resolution. In order to obtain complete spatiotemporal three-dimensional data, multiple shots are often required, which makes the data shooting time longer and can only be applied to static and repeatable scenes.

[0004] Given the inherent limitations of temporal separation methods, some methods have been proposed for separation based on traditional images using the different characteristics of direct and indirect illumination. Image-based separation methods are usually implemented using high-frequency illumination, using devices such as projectors to project high-spatial-frequency light and dark patterns onto the scene, and locally encoding direct and indirect illumination. High-frequency spatial illumination methods require taking multiple photos, which takes a long time to shoot, limiting the application of this method to static scenes. In recent years, some single-image illumination separation methods using deep learning have emerged. Due to the lack of physical constraints, single-image illumination separation methods are an obvious ill-posed problem, and such methods often have limited generalization performance.

[0005] Event cameras are a new type of neuromorphic sensor that detects brightness changes and triggers an event signal when its logarithmic change exceeds a threshold. Thanks to their high temporal resolution, event cameras have been applied to physics-based vision tasks, providing solutions that strike a good balance between speed and accuracy, such as polarization shape recovery, photometric stereo measurement, and structured light scanning. Several properties of event cameras make them suitable for separating direct and indirect illumination components. First, borrowing the principle of using high-frequency illumination for separation, only the minimum and maximum brightness values ​​of each pixel are needed to calculate the direct and indirect illumination components. The signal triggering model of event cameras enables them to effectively capture the entire radiance change process with significantly reduced data throughput. Second, the high temporal resolution of event cameras can significantly reduce data capture time, making them applicable to dynamic scenes. Third, the high dynamic range of temporal cameras can alleviate the overexposure problem, especially in areas with high specular reflections.

[0006] The document "Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination, TOG 2006" discloses a method for separating direct and indirect illumination. The method uses a controllable light source such as a projector to project high-frequency patterns of illumination such as a checkerboard in the scene. The direct illumination component in the dark area is 0; when the frequency of the illumination is greater than the frequency of the indirect illumination, the indirect illumination component depends on the ratio of the shadow area to the entire scene. By shooting images under different patterns, this characteristic can be used to solve the direct and indirect illumination components. The method disclosed in the above document requires shooting multiple images corresponding to different projection patterns. The data shooting time is long and the method is limited to be used only in static scenes. At the same time, the method assumes that the indirect illumination is relatively smooth, and the separation effect will be wrong in the presence of high-frequency indirect illumination.

[0007] The document "Decomposing Global Light Transport using Time of Flight Imaging, CVPR 2012" discloses a method for separating illumination components. This method uses a high-speed time-of-flight camera to obtain a high-temporal-resolution brightness temporal variation map for each pixel, which encodes the complex interaction between the incident light and the scene geometry with spatially varying material properties. The brightness variation on each pixel is analyzed, and the light transport is decomposed into its direct, subsurface scattering and interreflection components. The method disclosed in the above document requires the use of a sensor with ultra-high temporal resolution, and the equipment is expensive and requires relatively cumbersome equipment to be built. At the same time, the time required for this method to capture data is extremely long, usually more than 1 hour, and it is difficult to apply to downstream tasks.

[0008] The document "A Data-Driven Approach for Direct and Global Component Separation from a Single Image, ACCV 2018" discloses a method for separating direct and indirect illumination. This method applies a deep learning method to directly separate direct and indirect illumination from a single image. First, the method in the document "Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination, TOG 2006" is used to collect a data set in an indoor controllable scene and obtain a pseudo-separation truth value, and then a generative adversarial network model is built for training to obtain the final result. Although the method disclosed in the above document only requires one image to achieve direct and indirect illumination separation, due to the lack of physical constraint information, the natural ill-posedness of the task of direct and indirect illumination separation of a single image leads to that this method can only predict the relationship between object materials and scene illumination propagation through prior knowledge, and the generalization is poor, giving unstable and unreliable results in many cases. Summary of the invention

[0009] In order to solve the technical problems existing in the prior art, the present invention provides a method for separating direct illumination and indirect illumination based on event camera guidance.

[0010] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0011] The present invention provides a method for separating direct illumination and indirect illumination based on event camera guidance, which mainly includes the following steps:

[0012] Step 1: Use an event camera to capture event signals triggered by shadow changes, and use a traditional machine vision camera to capture a single initial image of the scene;

[0013] Step 2: Given an initial image and a corresponding event signal, the event trigger model of the event camera reconstructs the instantaneous image corresponding to any moment in the event signal recording time period, and reconstructs a set of image sequences from the event signal and the initial image by updating the event one by one. The maximum and minimum values ​​of each pixel position are calculated from the image sequence; the grayscale initial direct illumination and indirect illumination separation results are obtained according to the relationship between the direct illumination component and the indirect illumination component;

[0014] Step 3: Convert the discrete event signal into an event frame, use the event frame, the grayscale initial direct illumination and indirect illumination separation results, and the scene image brightness channel as the input of the separation optimization network, and predict the grayscale direct illumination and indirect illumination separation results after denoising optimization;

[0015] Step 4: Use the color compensation network to take the grayscale direct and indirect illumination separation results and the scene image chromaticity channel as input to predict the color direct and indirect illumination separation results.

[0016] Furthermore, in step one, a light source blocker is placed in front of the light source, and the light source blocker is shaken quickly so that the shadow created by the light source blocker sweeps across the entire scene; or, a high-frequency light and dark pattern is directly projected when the light source is controllable.

[0017] Furthermore, the light source shielding object is a linear shielding object and a mesh shielding object.

[0018] Furthermore, in step 2, the formula used by the event trigger model of the event camera to reconstruct the instantaneous image corresponding to any moment in the event signal recording time period is:

[0019]

[0020] Where T0 represents the initial time, represents the initial image; T1 represents any target moment within the event signal recording period, represents the corresponding instantaneous image; θ represents the event trigger threshold; an event signal e is represented by a quaternion (x, y, t, σ), where (x, y) is the pixel position of the event trigger, t represents the event trigger timestamp, and σ represents the polarity indicating whether the brightness increases or decreases; it is assumed here that N e event signal, denoted as The image sequence within the event recording time period is reconstructed by the above formula Where N is the number of reconstructed images.

[0021] Furthermore, in step 2, the relationship between the direct illumination component and the indirect illumination component is:

[0022] I max =I d +I g ,I min =βI g ,

[0023] Where β represents the proportion of the area directly illuminated by the light source, which depends on the area blocked by the shadow; I d is the direct illumination component; I g is the indirect lighting component; I maxand I min Respectively represent the image sequence The maximum and minimum values ​​at each pixel position calculated in .

[0024] Furthermore, the separation optimization network uses the U-Net network structure as the basic network architecture, adds a separation correction module at each level of the decoder, and uses the spatial attention mechanism to optimize the initial grayscale separation result; the separation correction module is used for a given input feature F∈R H×W×C , first aggregate its channel information through average pooling and maximum pooling to obtain and They are then concatenated and processed through a convolutional layer with a kernel size of 11×11 to obtain the spatial attention map A s ∈R H×W×1 .

[0025] Furthermore, the color compensation network uses the U-Net network structure as the basic network architecture, adds a color correction module at each level of the decoder, and uses the transposed self-attention mechanism to perform color correction to obtain the final separation result; the color correction module is designed based on the transposed self-attention mechanism.

[0026] Furthermore, when the separation optimization network and the color compensation network are trained, their loss function is the weighted sum of the square loss function and the perceptual loss function between the illumination separation image and the pseudo-true value separation image, and the weights of each layer are continuously updated by the back propagation algorithm; the separation optimization network and the color compensation network are trained separately, the separation optimization network is trained first, its parameters are fixed, and then the color compensation network is trained.

[0027] Furthermore, during training, the separation optimization network and the color compensation network randomly crop the data into small areas of 256×256 pixels and perform spatial-level data enhancement, while adding pixel-level data enhancement operations.

[0028] The beneficial effects of the present invention are:

[0029] (1) The method for separating direct illumination and indirect illumination based on event camera guidance provided by the present invention only requires taking a photo and a piece of event data in the scene, which greatly shortens the data capture time and reaches a level close to that of a single-image method, and can be applied to dynamic scenes.

[0030] (2) In the present invention, the event triggering model of the event camera enables it to discretely record the continuous change information in the scene illumination change. Therefore, compared with multiple images under different projection patterns, the event records more complete scene direct and indirect illumination coding information while reducing the data throughput.

[0031] (3) The present invention successfully integrates the images output by the event camera and the traditional camera, and can reconstruct high-quality color direct and indirect illumination separation results from a single image and event signals triggered by illumination changes.

[0032] (4) The present invention uses a deep learning method to effectively combine scene images and event signals, and has a better separation effect in high-frequency indirect lighting scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flow chart of a method for separating direct illumination and indirect illumination based on event camera guidance provided by the present invention.

[0034] Figure 2 The block diagram of the structure of the hybrid camera system. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0036] The present invention provides a method for separating direct illumination and indirect illumination based on event camera guidance, which mainly includes the following steps: shooting event signals triggered by shadow changes, estimating initial grayscale separation results, optimizing initial grayscale separation results and color compensation of illumination separation results.

[0037] See also Figure 1 To illustrate, the present invention provides a method for separating direct illumination and indirect illumination based on event camera guidance, and its specific operation process is as follows:

[0038] Step 1: photograph the event signal triggered by the shadow change;

[0039] The photographer places a light blocker in front of the light source and quickly shakes the light blocker so that the shadow cast by the light blocker sweeps across the entire scene. The event camera is used to record this process and the event signal ε triggered by the shadow change is recorded. In conjunction with this, a single initial image of the scene is taken with a traditional machine vision camera, recorded as

[0040] The direct illumination and indirect illumination separation method based on event camera guidance proposed in the present invention is applicable to single light source scenes. High-frequency shadows are projected in the scene using light source obstructions, and the process of shadows sweeping across the entire scene is recorded using an event camera. The light source obstructions tested by the present invention include linear obstructions (sticks, etc.) and mesh obstructions (wire fences, etc.), and other types of obstructions that can project high-frequency shadows can be used. For static scenes, the movement speed of the light source obstruction is not limited, and the present invention can be applied to achieve event-guided illumination separation. The present invention reduces the data capture time through the characteristic of the event camera to quickly record changes in light intensity, so that the present invention can be applied to dynamic scenes. For dynamic scenes, the movement speed of the light source obstruction is required to be significantly faster than the scene movement speed. At this time, the method proposed in the present invention can be applied to the dynamic scene.

[0041] As a preferred implementation, the above-mentioned method of using light source obstructions to encode direct lighting and indirect lighting can be replaced by the following method: when the light source is controllable, high-frequency light and dark patterns can be directly projected, and event cameras are also used to record changes in the projection pattern.

[0042] Step 2: Estimation of initial grayscale separation results;

[0043] The model is triggered by events from the event camera, given an initial image And the corresponding event signal, the event signal records the instantaneous image corresponding to any moment T1 within the time period They can be reconstructed using the following formula:

[0044]

[0045] Where T0 represents the initial time, represents the initial image; T1 represents any target moment within the event signal recording period, represents the corresponding instantaneous image; θ represents the event trigger threshold; an event signal e is represented by a quaternion (x, y, t, σ), where (x, y) is the pixel position of the event trigger, t represents the event trigger timestamp, and σ represents the polarity indicating whether the brightness increases or decreases; it is assumed here that N e event signal, denoted as

[0046] By updating event by event, we can obtain the event signal ε and the initial image Reconstruct an image sequence Where N is the number of reconstructed images. From this image sequence, the maximum and minimum values ​​of each pixel position can be calculated, which are denoted by I and max and I minIn the case of high-frequency shadows, the direct illumination component of the area covered by the shadow is 0, and the indirect illumination component depends on the ratio of the area directly illuminated by the light source to the entire scene. The direct illumination component and the indirect illumination component satisfy the following formula:

[0047] I max =I d +I g ,I min =βI g

[0048] Where β represents the area ratio directly illuminated by the light source. If a linear occluder such as a stick is used, β = 1; I d is the direct illumination component; I g is the indirect illumination component. From the above formula, we can get the initial direct illumination and indirect illumination separation results, but because the event camera only records grayscale information, the initial illumination separation result obtained at this time is also grayscale, recorded as and

[0049] Step 3: Optimize the initial grayscale separation results;

[0050] Due to the threshold of the event camera's temporal and spatial changes and the quantization error in the event trigger model, there is relatively serious noise in the grayscale initial separation result. At the same time, when there is a high-frequency indirect illumination component in the scene, the initial illumination separation result will have problems such as inaccurate separation. Therefore, the present invention converts the discrete event signal into an event frame, and uses the event frame, the grayscale initial direct illumination and indirect illumination separation results, and the scene image brightness channel as the input of the separation optimization network to obtain the grayscale direct illumination and indirect illumination separation results after denoising optimization, which are recorded as and

[0051] Among them, the separation optimization network used takes the U-Net network structure as the basic network architecture, adds a separation correction module at each layer scale of the decoder, and uses the spatial attention mechanism to optimize the initial grayscale separation result.

[0052] Among them, the separation correction module used is designed as follows:

[0053] For a given input feature F∈R H×W×C , first aggregate its channel information through average pooling and maximum pooling to obtain and They are then concatenated and processed through a convolutional layer with a kernel size of 11×11 to obtain the spatial attention map A s ∈R H×W×1 .

[0054] As a preferred implementation, the above-mentioned U-Net network structure can be replaced with other network structures, such as using Transformer as the basic network architecture, or removing specially designed modules in the original U-Net network structure.

[0055] Step 4: Color compensation of illumination separation results;

[0056] Through step 3, we get a high-quality grayscale separation result. Since the event signal lacks color information, step 4 will extract color information from the chromaticity channel of the scene image and restore the color information based on the grayscale separation result. Therefore, we use the color compensation network to take the grayscale direct and indirect illumination separation results and the scene image chromaticity channel as input to predict the color direct and indirect illumination separation results.

[0057] Among them, the color compensation network used takes the U-Net network structure as the basic network architecture, adds a color correction module at each level of the decoder, and uses the transposed self-attention mechanism to perform color correction to obtain the final separation result.

[0058] Among them, the color correction module used can be designed based on the transposed self-attention mechanism.

[0059] As a preferred implementation, the above-mentioned U-Net network structure can be replaced with other network structures, such as using Transformer as the basic network architecture, or removing specially designed modules in the original U-Net network structure.

[0060] In the present invention, both the optimization of the initial grayscale separation result in step 3 and the color compensation of the illumination separation result in step 4 are implemented through a neural network. Specifically, in step 3, a separation optimization network is used to obtain the grayscale direct illumination and indirect illumination separation results after denoising optimization; in step 4, a color compensation network is used to predict the color direct illumination and indirect illumination separation results. Therefore, the present invention uses self-collected indoor scene data to train the above networks, and the specific training process is as follows:

[0061] 1. Indoor scene data collection;

[0062] a) Collection equipment: The data collection equipment includes a projector, several types of light source occluders, and a hybrid camera system, such as Figure 2As shown, the hybrid camera system consists of a traditional machine vision camera 1 and an event camera 2, which are coaxially calibrated through a beam splitter 3. The beam splitter 3 splits the incident light into two beams, which enter the field of view of the two camera sensors synchronously. This hybrid camera system is used to capture traditional images and event signals simultaneously, and finally the separation of direct illumination and indirect illumination is achieved through a direct illumination and indirect illumination separation process 4 guided by the event camera.

[0063] Among them, the machine vision traditional camera 1 can be specifically used in the model HIKVISION MV-CA050-

[0064] The lens model of the 12UC traditional camera is a 35mm lens, but it is not limited to this.

[0065] Specifically, the event camera 2 may use a camera of model PROPHESEE GEN4.1, and its lens model is a 35mm lens, but is not limited thereto.

[0066] The beam splitter 3 can be implemented by using a beam splitter of model Thorlabs CCM1-BS013.

[0067] b) Building indoor scenes: The captured scenes include a wide range of everyday objects that are carefully selected to cover different global illumination effects such as inter-reflections and subsurface scattering.

[0068] c) Acquisition of pseudo-truth value: A high-frequency checkerboard light and dark pattern is projected onto the scene using a projector, and the checkerboard pattern is slightly offset. A total of 25 images are taken, and the direct and indirect illumination separation results are calculated using the existing multi-image method as the pseudo-truth value.

[0069] d) Data collection: The projector projects a white background pattern, and a light source shield is prepared in front of the light source, which is shaken at different speeds to make its shadow sweep across the scene. In this step, the traditional image frame video and the corresponding event signal are recorded simultaneously.

[0070] 2. Training of neural networks;

[0071] a) The present invention uses two networks, namely, a separation optimization network and a color compensation network, which are respectively used for the two steps of "optimization of initial grayscale separation results" and "color compensation of illumination separation results". Both network structures use U-Net as the basic architecture, which is an encoder-decoder structure containing a skip connection. First, a high-dimensional feature vector with a small spatial dimension and a large feature dimension is extracted from the image and event data through a convolution layer and fused, and then the obtained high-dimensional feature vector is expanded into a new image with a large spatial dimension and a small feature dimension by using a deconvolution layer, and fused with each scale feature in the encoder. A separation correction module and a color correction module are added to each layer of the decoder of the separation optimization network and the color compensation network to improve its performance under this task.

[0072] b) The loss function of the two network training is the weighted sum of the square loss function (L2 Loss, SquaredErrorLoss) between the illumination separation image and the pseudo-true value separation image and the perceptual loss function (Perceptual Loss), where the weights of the two are 100 and 0.5 respectively, and then the weights of each layer are continuously updated by the back propagation algorithm. The two networks are trained separately, first train the separation optimization network, fix its parameters, and then train the color compensation network.

[0073] c) During training, the data is randomly cropped into small areas of 256×256 pixels and spatial data enhancements such as horizontal flipping and rotation are applied. Pixel-level data enhancements such as random reorganization of color channels and brightness scaling are also added. The separation optimization network and the color compensation network are trained for 100 cycles respectively. The initial learning rate is set to 3×10 -4 , and adopted the cosine annealing learning rate scheduling strategy.

[0074] The present invention preferably uses Python language for programming, implements the neural network based on the Pytorch framework, and runs, trains and tests on a single NVIDIA GeForce RTX 3090 GPU.

[0075] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for separating direct and indirect illumination based on event camera guidance, characterized in that: The following steps are involved: Step 1: Use an event camera to capture event signals triggered by shadow changes, and use a traditional machine vision camera to capture a single initial image of the scene; Step 2: Given an initial image and a corresponding event signal, use the event trigger model of the event camera to reconstruct the instantaneous image corresponding to any moment in the event signal recording time period, reconstruct a set of image sequences from the event signal and the initial image by updating the event one by one, and calculate the maximum and minimum values ​​of each pixel position from the image sequence; According to the relationship between the direct illumination component and the indirect illumination component, the grayscale initial direct illumination and indirect illumination separation results are obtained; Step 3: Convert the discrete event signal into an event frame, use the event frame, the grayscale initial direct illumination and indirect illumination separation results, and the scene image brightness channel as the input of the separation optimization network, and predict the grayscale direct illumination and indirect illumination separation results after denoising optimization; Step 4: Use the color compensation network to take the grayscale direct and indirect illumination separation results and the scene image chromaticity channel as input to predict the color direct and indirect illumination separation results.

2. The method for separating direct illumination and indirect illumination based on event camera guidance according to claim 1, characterized in that: In step one, a light source blocker is placed in front of the light source, and the light source blocker is shaken quickly so that the shadow created by the light source blocker sweeps across the entire scene; or, a high-frequency light and dark pattern is directly projected when the light source is controllable.

3. The method for separating direct illumination and indirect illumination based on event camera guidance according to claim 2, characterized in that: The light source shielding object is a linear shielding object and a mesh shielding object.

4. The method for separating direct illumination and indirect illumination based on event camera guidance according to claim 1, characterized in that: In step 2, the formula used by the event trigger model of the event camera to reconstruct the instantaneous image corresponding to any moment in the event signal recording time period is: Where T0 represents the initial time, represents the initial image; T1 represents any target moment within the event signal recording period, represents the corresponding instantaneous image; θ represents the event trigger threshold; an event signal e is represented by a quaternion (x, y, t, σ), where (x, y) is the pixel position of the event trigger, t represents the event trigger timestamp, and σ represents the polarity indicating whether the brightness increases or decreases; it is assumed here that N e event signal, denoted as The image sequence within the event recording time period is reconstructed by the above formula Where N is the number of reconstructed images.

5. The method for separating direct illumination and indirect illumination based on event camera guidance according to claim 1, characterized in that: In step 2, the relationship between the direct illumination component and the indirect illumination component is: I max =I d +I g ,I min =βI g Where β represents the proportion of the area directly illuminated by the light source, which depends on the area blocked by the shadow; I d is the direct illumination component; I g is the indirect lighting component; I max and I min Respectively represent the image sequence The maximum and minimum values ​​at each pixel position calculated in .

6. The method for separating direct illumination and indirect illumination based on event camera guidance according to claim 1, characterized in that: The separation optimization network uses the U-Net network structure as the basic network architecture, adds a separation correction module at each layer scale of the decoder, and uses the spatial attention mechanism to optimize the initial grayscale separation result; The separation correction module is given by the input feature F∈R H×W×C , first aggregate its channel information through average pooling and maximum pooling to obtain and They are then concatenated and processed through a convolutional layer with a kernel size of 11×11 to obtain the spatial attention map A s ∈R H×W×1 .

7. The method for separating direct illumination and indirect illumination based on event camera guidance according to claim 1, characterized in that: The color compensation network uses the U-Net network structure as the basic network architecture, adds a color correction module at each level of the decoder, and uses the transposed self-attention mechanism to perform color correction to obtain the final separation result; The color correction module is designed based on the transposed self-attention mechanism.

8. The method for separating direct illumination and indirect illumination based on event camera guidance according to claim 1, characterized in that: When the separation optimization network and the color compensation network are trained, the loss function is the weighted sum of the square loss function between the illumination separation image and the pseudo-true value separation image and the perceptual loss function, and the weights of each layer are continuously updated by the back propagation algorithm; The separation optimization network and the color compensation network are trained separately. The separation optimization network is trained first, and its parameters are fixed before training the color compensation network.

9. The method for separating direct illumination and indirect illumination based on event camera guidance according to claim 8, characterized in that: When training the separation optimization network and the color compensation network, the data are randomly cropped into small areas of 256×256 pixels and spatial-level data enhancement is performed, while pixel-level data enhancement operations are added.

Images