A reflection separation method and system for light field images

Abstract

The present invention provides a reflection separation method and system for a light field image, which is used to solve the problem of inaccurate reflection separation of light field images in the prior art. The method obtains multi-scale spatial angle features based on light field images; after performing a refocusing operation to obtain a refocused image, a preliminary background image and a preliminary reflection image are generated respectively, and further convolution is performed to obtain a background edge map, a background refocused image, a reflection edge map, and a reflection refocused image; then the background and reflection spatial attention weights are generated based on a spatial attention mechanism map; the multi-scale spatial angle features and the refocusing features are combined to obtain background reconstruction features and reflection reconstruction features, and further generate reflection channel attention weights, adjust the reconstruction features according to the two weights, and finally use a U-type network to reconstruct the separated background image and reflection image respectively. The present invention automatically restores the corresponding background image and reflection image through a light field image with reflection, thereby improving the qualitative and quantitative analysis effects of the image.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a reflection separation method and system for a light field image. Background Art

[0002] The light field camera adds a microlens array between the main lens and the imaging sensor. The microlens array is a two-dimensional array composed of many microlens units, which is conjugate with the main lens and the imaging sensor. Therefore, the position and direction of a beam of light can be determined by the microlens plane and the imaging sensor plane, so that the light field camera can simultaneously record the intensity, position and direction of the light. Compared with traditional cameras, it has the characteristics of taking pictures first and then focusing, which has great advantages in multi-view information collection.

[0003] Reflection is a common image interference. When taking photos through transparent surfaces such as glass, objects on the camera side will also be captured, which is not only visually unsightly, but also affects the performance of some other computer vision tasks, such as target detection and target tracking. The light field images taken by light field cameras also have this problem. Therefore, it is generally necessary to separate the image through a separation method to obtain the actual scene image.

[0004] In the existing technology, the reflection separation method of a single image usually needs to assume that the reflection layer is relatively weak or blurred. These assumptions severely limit the use of these methods in actual complex scenes. At the same time, the reflection separation method based on the assumed conditions is often calculated in minutes, and the processing speed is slow. Summary of the invention

[0005] The purpose of an embodiment of the present invention is to propose a reflection separation method for a light field image, obtain multi-scale spatial angle features through multi-scale spatial angle convolution, obtain a preliminary background image and a reflection image through a refocused image obtained through a refocusing operation, further generate a background edge map, a background refocusing feature, a reflection edge map, and a reflection refocusing feature, and then generate background space attention weights and background channel attention weights through the background edge map and the background refocusing feature, and adjust the background reconstruction feature; generate reflection space attention weights and reflection channel attention weights through the reflection edge map and the reflection refocusing feature, and adjust the reflection reconstruction feature, reconstruct the background image and the reflection image according to the adjusted background reconstruction feature and reflection reconstruction feature, respectively, and automatically restore the corresponding background image and reflection image through the light field image with reflection, thereby improving the qualitative and quantitative analysis effects of the image.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a reflection separation method for a light field image, the reflection separation method comprising the following steps:

[0008] Step S1, obtaining an original light field image, performing multi-scale spatial angle convolution, and obtaining multi-scale spatial angle features;

[0009] Step S2, obtaining a refocused image by performing a refocusing operation on the original light field image;

[0010] Step S3, performing a dynamic convolution operation on the refocused image to generate a preliminary background image and a preliminary reflection image respectively;

[0011] Step S4, performing convolution on the preliminary background image and the preliminary reflection image respectively to obtain a background edge map, a background refocusing feature, a reflection edge map, and a reflection refocusing feature;

[0012] Step S5, based on the spatial attention mechanism, using the background edge map to generate a background spatial attention weight, and using the reflection edge map to generate a reflection spatial attention weight;

[0013] Step S6, combining the multi-scale spatial angle feature and the background refocusing feature to obtain a background reconstruction feature, combining the multi-scale spatial angle feature and the reflection refocusing feature to obtain a reflection reconstruction feature;

[0014] Step S7, based on the channel attention mechanism, using the background reconstruction feature to generate a background channel attention weight, and using the reflection reconstruction feature to generate a reflection channel attention weight;

[0015] Step S8, adjusting the background reconstruction feature using the background space attention weight and the background channel attention weight to obtain the adjusted background reconstruction feature; adjusting the reflection reconstruction feature using the reflection space attention weight and the reflection channel attention weight to obtain the adjusted reflection reconstruction feature;

[0016] Step S9, using the U-type network to reconstruct the background image from the adjusted background reconstruction features, and using the U-type network to reconstruct the reflection image from the adjusted reflection reconstruction features.

[0017] As a preferred embodiment of the present invention, the step S1 multi-scale spatial angle convolution specifically comprises the following steps:

[0018] Use spatial 2D convolution to downsample once and twice to obtain images with 1 / 2 and 1 / 4 original resolution respectively;

[0019] Perform 4 spatial 2D convolutions and 4 angular 2D convolutions on the original, 1 / 2 original resolution, and 1 / 4 original resolution images respectively to obtain features of different scales;

[0020] The features of different scales are fused using spatial 2D convolution to obtain multi-scale spatial angle features.

[0021] As a preferred embodiment of the present invention, the formula for the refocusing operation in step S2 is:

[0022]

[0023] In formula (1), d represents the value of the refocusing operation, L represents the light field image, u, v represent the angle dimension, x, y represent the spatial dimension, u c ,v c Represents the angular coordinates of the central viewing angle, u∈[0,U], v∈[0,V], U and V are two constants, I d (x, y) represents the refocused image after the refocusing operation using numerical values.

[0024] As a preferred embodiment of the present invention, the step S3 dynamic convolution specifically comprises the following steps:

[0025] The refocused image is input into two different sets of 3D convolutions to generate background dynamic convolution kernel and reflection dynamic convolution kernel respectively;

[0026] generating a preliminary background image from the refocused image using a background dynamic convolution kernel;

[0027] A preliminary reflection image is generated from the refocused image using a reflection dynamic convolution kernel.

[0028] As a preferred embodiment of the present invention, the operation of generating a preliminary background image and a preliminary reflection image by dynamic convolution in step S3 can be expressed by the following formula:

[0029] B inter (x,y)=∑ d I(d,x,y)*W B (d,x,y) (2)

[0030] R inter (x,y)=∑ d I(d,x,y)*W R (d,x,y) (3)

[0031] In equations (2) and (3), d represents the value of the refocusing operation, x and y represent the spatial dimensions, I(d, x, y) represents all refocused images, and B inter (x, y) represents the preliminary background image, WB (d,x,y) represents the background dynamic convolution kernel, R inter (x, y) represents the preliminary reflection image, W R (d,x,y) represents the reflected dynamic convolution kernel.

[0032] As a preferred embodiment of the present invention, the step S5 is based on the spatial attention mechanism, using the background edge map to generate the background spatial attention weight, and using the reflection edge map to generate the reflection spatial attention weight, and specifically includes the following steps:

[0033] Input the background edge map into the spatial 2D convolution, change its channel to 1, and obtain the background spatial attention weight;

[0034] The reflected edge map is input into the spatial 2D convolution, its channel is changed to 1, and the reflected spatial attention weight is obtained.

[0035] As a preferred embodiment of the present invention, the step S7 is based on a channel attention mechanism, using the background reconstruction feature to generate a background channel attention weight, and using the reflection reconstruction feature to generate a reflection channel attention weight, and specifically includes the following steps:

[0036] Perform global maximum pooling and global average pooling operations on the background reconstruction features, change their spatial resolution to 1*1, and obtain background maximum pooling features and background average pooling features;

[0037] Input the background maximum pooling feature and the background average pooling feature into the fully connected layer to obtain the background channel attention weight;

[0038] Perform global maximum pooling and global average pooling operations on the reflection reconstruction features, change their spatial resolution to 1*1, and obtain reflection maximum pooling features and reflection average pooling features;

[0039] The reflection maximum pooling feature and the reflection average pooling feature are input into the fully connected layer to obtain the reflection channel attention weight.

[0040] As a preferred embodiment of the present invention, the step S8 obtains the adjusted background reconstruction feature and reflection reconstruction feature through the following formula:

[0041]

[0042]

[0043] In formulas (4) and (5), x and y represent the spatial dimensions, and CA B (x,y) represents the background channel attention weight, SA B (x,y) represents the background spatial attention weight, FB (x,y) represents the background reconstruction feature, Represents the adjusted background reconstruction features; CA R (x,y) represents the reflection channel attention weight, SA R (x,y) represents the reflection spatial attention weight, F R (x,y) represents the reflection reconstruction feature, represents the adjusted reflection reconstruction feature.

[0044] As a preferred embodiment of the present invention, the structure of the U-type network in step S9 specifically includes a coding layer, a decoding layer and a jump layer structure; wherein,

[0045] The encoding layer consists of 3 groups of modules including spatial 2D convolution;

[0046] The decoding layer consists of 3 groups of modules including spatial 2D convolution and transposed convolution;

[0047] The skip layer structure is used to connect the encoding layer and the decoding layer.

[0048] In a second aspect, an embodiment of the present invention further provides a reflection separation system for a light field image, the system comprising: a light field image acquisition module, a multi-scale spatial angle feature generation module, a refocusing module, a preliminary separation image generation module, an edge map generation module, a spatial attention weight generation module, a refocusing feature generation module, a reconstruction feature generation module, a channel attention weight generation module, a reconstruction feature adjustment module, and an image separation module; wherein,

[0049] The light field image acquisition module is used to acquire the original light field image;

[0050] The multi-scale spatial angle feature generation module is used to perform multi-scale spatial angle convolution on the original light field image, obtain multi-scale spatial angle features, and send them to the reconstruction feature generation module;

[0051] The refocusing module is used to perform a refocusing operation on the original light field image to obtain a refocused image;

[0052] The preliminary separation image generation module is used to perform a dynamic convolution operation on the refocused image to generate a preliminary background image and a preliminary reflection image respectively;

[0053] The edge map generation module is used to perform convolution on the preliminary background image and the preliminary reflection image respectively to obtain a background edge map and a reflection edge map;

[0054] The spatial attention weight generation module is used to generate background spatial attention weights using the background edge map and reflective spatial attention weights using the reflective edge map based on the spatial attention mechanism;

[0055] The refocusing feature generation module is used to perform convolution on the preliminary background image and the preliminary reflection image respectively to obtain background refocusing features and reflection refocusing features;

[0056] The reconstruction feature generation module is used to merge the multi-scale spatial angle feature and the background refocusing feature to obtain the background reconstruction feature, and merge the multi-scale spatial angle feature and the reflection refocusing feature to obtain the reflection reconstruction feature;

[0057] The channel attention weight generation module is used to generate background channel attention weights using background reconstruction features and to generate reflection channel attention weights using reflection reconstruction features;

[0058] The reconstruction feature adjustment module is used to adjust the background reconstruction feature using the background space attention weight and the background channel attention weight to obtain the adjusted background reconstruction feature; and to adjust the reflection reconstruction feature using the reflection space attention weight and the reflection channel attention weight to obtain the adjusted reflection reconstruction feature;

[0059] The image separation module is used to reconstruct the background image from the adjusted background reconstruction features using a U-shaped network, and to reconstruct the reflection image from the adjusted reflection reconstruction features using a U-shaped network.

[0060] The present invention has the following beneficial effects:

[0061] The reflection separation method of the light field image of the embodiment of the present invention automatically separates the background image and the reflection image by using the light field image with reflection, so as to obtain a clearer and independent background image and reflection image in a real scene, and the image has richer restored details, and the processing speed is also greatly improved. The embodiment of the present invention can be applied to image preprocessing of various common computer vision tasks, such as target detection, target tracking, three-dimensional reconstruction, etc., and improves the image processing effect without increasing the image processing time, and improves the qualitative and quantitative analysis effect of the image. For example, the separated background image can be used for more accurate target detection, target tracking, and three-dimensional reconstruction, and the separated reflection image can be used for the detection of cases in the field of criminal investigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0063] Figure 1It is a flow chart of a reflection separation method of a light field image according to an embodiment of the present invention;

[0064] Figure 2 FIG. 1 is a diagram showing an example of generating a refocused image in an embodiment of the present invention;

[0065] Figure 3 Shown is a schematic diagram of reflection separation of a light field image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] Below, by referring to exemplary embodiments, technical problems, technical solutions and advantages of the present invention are explained in detail. The exemplary embodiments described below are only used to explain the present invention and cannot be interpreted as limiting the present invention. It can be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the field to which the present invention belongs. It should also be understood that those terms such as those defined in general dictionaries should be understood to have a meaning consistent with the meaning in the context of the prior art, and unless defined here, will not be interpreted with an idealized or overly formal meaning.

[0067] The present invention provides a reflection separation method for a light field image, which obtains a light field image with reflection and separates a background image and a reflection image. The reflection separation method uses the rich angular spatial information of the light field image to automatically and accurately separate the background image and the reflection image, wherein the reflection image refers to an interference image formed by reflection. Compared with the existing technology, the method has the advantages of excellent separation effect and fast processing speed.

[0068] The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0069] This embodiment provides a reflection separation method for a light field image, which separates an input light field image with reflection and outputs a background image and a reflection image of the central viewing angle.

[0070] Figure 1 FIG. 1 is a flow chart of the reflection separation method of the light field image in this embodiment. Figure 1 and Figure 3 As shown, the reflection separation method of the light field image described in this embodiment includes the following steps:

[0071] Step S1, obtaining an original light field image, performing multi-scale spatial angle convolution, and obtaining multi-scale spatial angle features.

[0072] In this step, multi-scale spatial angular convolution is performed on the light field image, specifically including: first, a spatial 2D convolution with a convolution kernel size of 3*3, a step size of 2, and a padding of 1 is used for downsampling, and is used once and twice respectively to obtain images with 1 / 2 original resolution and 1 / 4 original resolution; then, the original, 1 / 2 original resolution, and 1 / 4 original resolution images are subjected to 4 spatial 2D convolutions and 4 angular 2D convolutions respectively to obtain features of 3 different scales, where the convolution kernel size of the 4 spatial 2D convolutions and the first two angular 2D convolutions is 3*3, the step size is 1, and the padding is 1, and the convolution kernel size of the last two angular 2D convolutions is 3*3, the step size is 1, and the padding is 0. The spatial convolution is to convolve the above image in the x, y dimensions, and the angular convolution is to convolve the above image in the u, v dimensions. In this embodiment, the convolution operation can be expressed by the following formula:

[0073]

[0074] In formula (1), is a convolution operation, W represents a weight matrix, B represents a bias matrix, X is an input, Y is an output, and h is an activation function. Preferably, in this embodiment, all activation functions use the ReLU function, which can be expressed as:

[0075] h(x)=max(0,x) (2)

[0076] Afterwards, features of different scales are fused into multi-scale spatial angle features by using a spatial 2D convolution. The scale here is the resolution; the convolution kernel size of the spatial 2D convolution is 3*3, the step size is 1, and the padding is 1.

[0077] It should be noted that 2D convolution includes spatial 2D convolution and angular 2D convolution. When there are no restrictions on the two convolution methods, the description of 2D convolution is adopted to perform spatial 2D convolution and / or angular 2D convolution as needed.

[0078] Step S2, obtaining a refocused image by performing a refocusing operation on the original light field image.

[0079] In this step, the refocusing operation can be expressed as:

[0080]

[0081] In formula (3), d represents the value of the refocusing operation, L represents the light field image, u, v represent the angle dimension, x, y represent the spatial dimension, u c ,v c Represents the angular coordinates of the central viewing angle, u∈[0,U], v∈[0,V], U and V are two constants, I d(x, y) represents the refocused image after the refocusing operation using numerical values. Figure 2 This is an example of how a refocused image is generated.

[0082] After the above refocusing operation, the acquired refocused images are generally multiple. Preferably, in this embodiment, d∈[-2,2], the step size is 0.4, and a total of 11 refocused images are generated.

[0083] Step S3, performing a dynamic convolution operation on the refocused image to generate a preliminary background image and a preliminary reflected image respectively.

[0084] In this step, the refocused image is subjected to two different dynamic convolutions, which are realized by five series-connected 3D convolutions to generate a background dynamic convolution kernel and a reflection dynamic convolution kernel, respectively. The background dynamic convolution kernel is used to generate a preliminary background image from the refocused image, and the reflection dynamic convolution kernel is used to generate a preliminary reflection image from the refocused image. The specific formula is as follows:

[0085] B inter (x,y)=∑ d I(d,x,y)*W B (d,x,y) (4)

[0086] R inter (x,y)=∑ d I(d,x,y)*W R (d,x,y) (5)

[0087] In equations (4) and (5), d represents the value of the refocusing operation, x and y represent the spatial dimensions, I(d, x, y) represents all refocused images, and B inter (x, y) represents the preliminary background image, W B (d,x,y) represents the background dynamic convolution kernel, R inter (x, y) represents the preliminary reflection image, W R (d,x,y) represents the reflected dynamic convolution kernel.

[0088] Step S4, convolving the preliminary background image and the preliminary reflection image respectively to obtain a background edge map, a background refocusing feature, a reflection edge map, and a reflection refocusing feature.

[0089] In this step, the preliminary background image is subjected to three spatial 2D convolutions to obtain a background edge map and background refocusing features, and the preliminary reflection image is subjected to three spatial 2D convolutions to obtain a reflection edge map and reflection refocusing features. In order to better obtain the edge map, the L1 loss of the edge map is introduced for supervision.

[0090] Preferably, the convolution kernel size of the spatial 2D convolution in this step is 3*3, the step size is 1, and the padding is 1.

[0091] Step S5, based on the spatial attention mechanism, using the background edge map to generate background spatial attention weights, and using the reflection edge map to generate reflection spatial attention weights.

[0092] In this step, the background edge map is input into a spatial 2D convolution, its channel is changed to 1, and the background spatial attention weight is obtained; at the same time, the reflection edge map is input into a spatial 2D convolution, its channel is changed to 1, and the reflection spatial attention weight is obtained.

[0093] Preferably, the convolution kernel size of the spatial 2D convolution in this step is 3*3, the step size is 1, and the padding is 1.

[0094] Step S6, merging the multi-scale spatial angle feature and the background refocusing feature to obtain a background reconstruction feature, and merging the multi-scale spatial angle feature and the reflection refocusing feature to obtain a reflection reconstruction feature.

[0095] In this step, the multi-scale spatial angle features and the background refocusing features are input into a spatial 2D convolution to obtain the background reconstruction features; at the same time, the multi-scale spatial angle features and the reflection refocusing features are input into another spatial 2D convolution to obtain the reflection reconstruction features.

[0096] Preferably, the convolution kernel size of the spatial 2D convolution in this step is 3*3, the step size is 1, and the padding is 1.

[0097] Step S7, based on the channel attention mechanism, using the background reconstruction feature to generate the background channel attention weight, and using the reflection reconstruction feature to generate the reflection channel attention weight.

[0098] In this step, the background reconstruction feature is subjected to global maximum pooling operation and global average pooling operation, and its spatial resolution becomes 1*1, and the background maximum pooling feature and the background average pooling feature are obtained, and the background maximum pooling feature and the background average pooling feature are input into the fully connected layer to obtain the background channel attention weight; similarly, the reflection reconstruction feature is subjected to global maximum pooling operation and global average pooling operation, and its spatial resolution becomes 1*1, and the reflection maximum pooling feature and the reflection average pooling feature are obtained, and the reflection maximum pooling feature and the reflection average pooling feature are input into the fully connected layer to obtain the reflection channel attention weight.

[0099] Step S8, using the background space attention weight and the background channel attention weight to adjust the background reconstruction feature, and obtain the adjusted background reconstruction feature; using the reflection space attention weight and the reflection channel attention weight to adjust the reflection reconstruction feature, and obtain the adjusted reflection reconstruction feature.

[0100] In this step, the adjusted reconstruction features are obtained according to formulas (6) and (7):

[0101]

[0102]

[0103] In formulas (4) and (5), x and y represent the spatial dimensions, and CA B (x,y) represents the background channel attention weight,

[0104] SA B (x,y) represents the background spatial attention weight, F B (x,y) represents the background reconstruction feature, Represents the adjusted background reconstruction features; CA R (x,y) represents the reflection channel attention weight, SA R (x,y) represents the reflection spatial attention weight, F R (x,y) represents the reflection reconstruction feature, represents the adjusted reflection reconstruction feature.

[0105] Step S9, using the U-type network to reconstruct the background image from the adjusted background reconstruction features, and using the U-type network to reconstruct the reflection image from the adjusted reflection reconstruction features.

[0106] In this step, the background reconstruction features and the reflection reconstruction features are respectively input into two different U-type networks. The U-type network includes a coding layer, a decoding layer and a skip layer connection. The coding layer is composed of 3 groups of modules including spatial 2D convolutions, where the spatial 2D convolutions include two step sizes of 1 and 2; the decoding layer is composed of 3 groups of modules including spatial 2D convolutions and transposed convolutions; the skip layer structure is used to connect the coding layer and the decoding layer. L1 loss supervision is added to the background image and the reflection image during training; finally, the background image and the reflection image are obtained from the U-type network. When the background image and the reflection image are obtained using the U-type network, the coding layer performs 2 downsampling operations (spatial 2D convolutions with a convolution kernel size of 3*3, a step size of 2, and a padding of 1) and 3 spatial 2D convolutions (convolution kernel size of 3*3, a step size of 1, and a padding of 1); the decoding layer performs 2 upsampling operations (transposed convolutions) and 3 spatial 2D convolutions (convolution kernel size of 3*3, a step size of 1, and a padding of 1).

[0107] It can be seen from the above that the reflection separation method of the light field image in the embodiment of the present invention can automatically obtain a background image and a reflection image only through an original light field image with reflection; the separation effect is good, the processing speed is fast, and the robustness to real complex scenes is high. The obtained background image and reflection image are highly independent and rich in details, and can be widely used in common computer vision tasks to improve their processing performance, such as: target detection, target tracking, three-dimensional reconstruction, etc. For example, the separated background image can be used for more accurate target detection and tracking, and the separated reflection image can be used for the detection of cases in the field of criminal investigation.

[0108] Based on the same idea, an embodiment of the present invention also provides a reflection separation system for light field images, which includes: a light field image acquisition module, a multi-scale spatial angle feature generation module, a refocusing module, a preliminary separation image generation module, an edge map generation module, a spatial attention weight generation module, a refocusing feature generation module, a reconstruction feature generation module, a channel attention weight generation module, a reconstruction feature adjustment module, and an image separation module.

[0109] Wherein, the light field image acquisition module is used to acquire the original light field image;

[0110] The multi-scale spatial angle feature generation module is used to perform multi-scale spatial angle convolution on the original light field image, obtain multi-scale spatial angle features, and send them to the reconstruction feature generation module;

[0111] The refocusing module is used to perform a refocusing operation on the original light field image to obtain a refocused image;

[0112] The preliminary separation image generation module is used to perform a dynamic convolution operation on the refocused image to generate a preliminary background image and a preliminary reflection image respectively;

[0113] The edge map generation module is used to perform convolution on the preliminary background image and the preliminary reflection image respectively to obtain a background edge map and a reflection edge map;

[0114] The spatial attention weight generation module is used to generate background spatial attention weights using the background edge map and reflective spatial attention weights using the reflective edge map based on the spatial attention mechanism;

[0115] The refocusing feature generation module is used to perform convolution on the preliminary background image and the preliminary reflection image respectively to obtain background refocusing features and reflection refocusing features;

[0116] The reconstruction feature generation module is used to merge the multi-scale spatial angle feature and the background refocusing feature to obtain the background reconstruction feature, and merge the multi-scale spatial angle feature and the reflection refocusing feature to obtain the reflection reconstruction feature;

[0117] The channel attention weight generation module is used to generate background channel attention weights using background reconstruction features and to generate reflection channel attention weights using reflection reconstruction features;

[0118] The reconstruction feature adjustment module is used to adjust the background reconstruction feature using the background space attention weight and the background channel attention weight to obtain the adjusted background reconstruction feature; and to adjust the reflection reconstruction feature using the reflection space attention weight and the reflection channel attention weight to obtain the adjusted reflection reconstruction feature;

[0119] The image separation module is used to reconstruct the background image from the adjusted background reconstruction features using a U-shaped network, and to reconstruct the reflection image from the adjusted reflection reconstruction features using a U-shaped network.

[0120] In this embodiment, each module is implemented by a processor, and a memory is appropriately added when storage is required. Among them, the processor can be but is not limited to a microprocessor MPU, a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, etc. The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0121] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0122] It should also be noted that the reflection separation system of the light field image described in this embodiment corresponds to the reflection separation method, and the description and limitation of the method are also applicable to the system, which will not be repeated here.

[0123] The above is a preferred embodiment of the present invention. It should be noted that the present invention is not limited to the exemplary embodiments disclosed above. The essence of the description is only to help those skilled in the relevant field to comprehensively understand the specific details of the present invention. For those skilled in the art, without departing from the principles of the present invention, several improvements and modifications made within the technical scope disclosed by the present invention, and changes or replacements that can be easily thought of, should be covered within the protection scope of the present invention.

Claims

1. A reflection separation method for a light field image, characterized in that: The reflection separation method comprises the following steps: Step S1, obtaining an original light field image, performing multi-scale spatial angle convolution, and obtaining multi-scale spatial angle features; Step S2, obtaining a refocused image by performing a refocusing operation on the original light field image; Step S3, performing a dynamic convolution operation on the refocused image to generate a preliminary background image and a preliminary reflection image respectively; Step S4, performing convolution on the preliminary background image and the preliminary reflection image respectively to obtain a background edge map, a background refocusing feature, a reflection edge map, and a reflection refocusing feature; Step S5, based on the spatial attention mechanism, using the background edge map to generate a background spatial attention weight, and using the reflection edge map to generate a reflection spatial attention weight; Step S6, combining the multi-scale spatial angle feature and the background refocusing feature to obtain a background reconstruction feature, combining the multi-scale spatial angle feature and the reflection refocusing feature to obtain a reflection reconstruction feature; Step S7, based on the channel attention mechanism, using the background reconstruction feature to generate a background channel attention weight, and using the reflection reconstruction feature to generate a reflection channel attention weight; Step S8, adjusting the background reconstruction feature using the background space attention weight and the background channel attention weight to obtain the adjusted background reconstruction feature; adjusting the reflection reconstruction feature using the reflection space attention weight and the reflection channel attention weight to obtain the adjusted reflection reconstruction feature; Step S9, using the U-type network to reconstruct the background image from the adjusted background reconstruction features, and using the U-type network to reconstruct the reflection image from the adjusted reflection reconstruction features.

2. The light field image reflection separation method according to claim 1, characterized in that: The step S1 multi-scale spatial angle convolution specifically comprises the following steps: Use spatial 2D convolution to downsample once and twice to obtain images with 1 / 2 and 1 / 4 original resolution respectively; Perform 4 spatial 2D convolutions and 4 angular 2D convolutions on the original, 1 / 2 original resolution, and 1 / 4 original resolution images respectively to obtain features of different scales; The features of different scales are fused using spatial 2D convolution to obtain multi-scale spatial angle features.

3. The light field image reflection separation method according to claim 1, characterized in that: The formula for the refocusing operation in step S2 is: In formula (1), d represents the value of the refocusing operation, L represents the light field image, u, v represent the angle dimension, x, y represent the spatial dimension, and u c , v c Represents the angular coordinates of the central viewing angle, u∈[0,U], v∈[0,V], U and V are two constants, I d (x, y) represents the refocused image after the refocusing operation using numerical values.

4. The light field image reflection separation method according to claim 1, characterized in that: The step S3 dynamic convolution specifically comprises the following steps: The refocused image is input into two different sets of 3D convolutions to generate background dynamic convolution kernel and reflection dynamic convolution kernel respectively; generating a preliminary background image from the refocused image using a background dynamic convolution kernel; A preliminary reflection image is generated from the refocused image using a reflection dynamic convolution kernel.

5. The light field image reflection separation method according to claim 4, characterized in that: The step S3 uses dynamic convolution to generate a preliminary background image and a preliminary reflection image, which is expressed by the following formula: B inter (x,y)=∑ d I(d,x,y)*W B (d,x,y) (2) R inter (x,y)=∑ d I(d,x,y)*W R (d,x,y) (3) In equations (2) and (3), d represents the value of the refocusing operation, x and y represent the spatial dimensions, I(d, x, y) represents all refocused images, and B inter (x, y) represents the preliminary background image, W B (d, x, y) represents the background dynamic convolution kernel, R inter (x, y) represents the preliminary reflection image, W R (d, x, y) represents the reflected dynamic convolution kernel.

6. The light field image reflection separation method according to claim 1, characterized in that: The step S5 is based on the spatial attention mechanism, uses the background edge map to generate the background spatial attention weight, and uses the reflection edge map to generate the reflection spatial attention weight, and specifically includes the following steps: Input the background edge map into the spatial 2D convolution, change its channel to 1, and obtain the background spatial attention weight; The reflected edge map is input into the spatial 2D convolution, its channel is changed to 1, and the reflected spatial attention weight is obtained.

7. The light field image reflection separation method according to claim 1, characterized in that: The step S7 is based on the channel attention mechanism, uses the background reconstruction feature to generate the background channel attention weight, and uses the reflection reconstruction feature to generate the reflection channel attention weight, and specifically includes the following steps: Perform global maximum pooling and global average pooling operations on the background reconstruction features, change their spatial resolution to 1*1, and obtain background maximum pooling features and background average pooling features; Input the background maximum pooling feature and the background average pooling feature into the fully connected layer to obtain the background channel attention weight; Perform global maximum pooling and global average pooling operations on the reflection reconstruction features, change their spatial resolution to 1*1, and obtain reflection maximum pooling features and reflection average pooling features; The reflection maximum pooling feature and the reflection average pooling feature are input into the fully connected layer to obtain the reflection channel attention weight.

8. The light field image reflection separation method according to claim 1, characterized in that: The step S8 obtains the adjusted background reconstruction features and reflection reconstruction features through the following formula: In formulas (4) and (5), x and y represent the spatial dimensions, and CA B (x, y) represents the background channel attention weight, SA B (x, y) represents the background spatial attention weight, F B (x, y) represents the background reconstruction feature, Represents the adjusted background reconstruction features; CA R (x, y) represents the reflection channel attention weight, SA R (x, y) represents the reflection spatial attention weight, F R (x, y) represents the reflection reconstruction feature, represents the adjusted reflection reconstruction feature.

9. The light field image reflection separation method according to claim 1, characterized in that: The structure of the U-type network in step S9 specifically includes a coding layer, a decoding layer and a jump layer structure; wherein, The encoding layer consists of 3 groups of modules including spatial 2D convolutions; The decoding layer consists of 3 groups of modules including spatial 2D convolution and transposed convolution; The skip layer structure is used to connect the encoding layer and the decoding layer.

10. A reflection separation system for a light field image, characterized in that: The system comprises: a light field image acquisition module, a multi-scale spatial angle feature generation module, a refocusing module, a preliminary separation image generation module, an edge map generation module, a spatial attention weight generation module, a refocusing feature generation module, a reconstruction feature generation module, a channel attention weight generation module, a reconstruction feature adjustment module, and an image separation module; wherein, The light field image acquisition module is used to acquire the original light field image; The multi-scale spatial angle feature generation module is used to perform multi-scale spatial angle convolution on the original light field image, obtain multi-scale spatial angle features, and send them to the reconstruction feature generation module; The refocusing module is used to perform a refocusing operation on the original light field image to obtain a refocused image; The preliminary separation image generation module is used to perform a dynamic convolution operation on the refocused image to generate a preliminary background image and a preliminary reflection image respectively; The edge map generation module is used to perform convolution on the preliminary background image and the preliminary reflection image respectively to obtain a background edge map and a reflection edge map; The spatial attention weight generation module is used to generate background spatial attention weights using the background edge map and reflective spatial attention weights using the reflective edge map based on the spatial attention mechanism; The refocusing feature generation module is used to perform convolution on the preliminary background image and the preliminary reflection image respectively to obtain background refocusing features and reflection refocusing features; The reconstruction feature generation module is used to merge the multi-scale spatial angle feature and the background refocusing feature to obtain the background reconstruction feature, and merge the multi-scale spatial angle feature and the reflection refocusing feature to obtain the reflection reconstruction feature; The channel attention weight generation module is used to generate background channel attention weights using background reconstruction features and to generate reflection channel attention weights using reflection reconstruction features; The reconstruction feature adjustment module is used to adjust the background reconstruction feature using the background space attention weight and the background channel attention weight to obtain the adjusted background reconstruction feature; and to adjust the reflection reconstruction feature using the reflection space attention weight and the reflection channel attention weight to obtain the adjusted reflection reconstruction feature; The image separation module is used to reconstruct the background image from the adjusted background reconstruction features using a U-shaped network, and to reconstruct the reflection image from the adjusted reflection reconstruction features using a U-shaped network.

Images