A method and system for extracting target depth image and multispectral image

Abstract

This invention relates to a method and system for extracting target depth images and multispectral images. The method includes: Step S1: acquiring a multispectral light field image of the target using a multispectral light field camera, and segmenting the multispectral light field image into multiple sub-images; Step S2: extracting effective pixels from each sub-image of the multispectral light field image; Step S3: selecting effective pixels at the same position in each sub-image, and re-stitching the effective pixels at the same position to form several sub-aperture images; Step S4: selecting effective pixels from different spectral channels from the several sub-aperture images to form a multispectral image of the target; simultaneously selecting effective pixels of the same spectral channel from the several sub-aperture image sequences, and calculating the depth image of the target based on the effective pixels of the same spectral channel. This invention can effectively extract multispectral images and depth images of a target.

Description

Technical Field

[0001] The present invention relates to the technical field of spectral light field imaging, and in particular to a method and system for extracting target depth images and multi-spectral images. Background Art

[0002] The natural world is colorful, ever-changing, and contains a wealth of information. This information can be represented by a nine-dimensional full-light function, including the three-dimensional spatial coordinates of light, propagation angle, wavelength, polarization state, and dynamic changes of light over time. However, traditional optical imaging methods can only capture the intensity of light radiation on a two-dimensional plane, and lose optical information in other dimensions. With the development of science and technology, how to break through the limitations of the imaging system and obtain higher-dimensional information has become one of the goals of the optical imaging field.

[0003] In 2005, Ren Ng and others from Stanford University successfully developed the first handheld light field camera Lytro by modifying a SLR camera, marking the true transition of light field imaging technology from the laboratory to practical use. Unlike traditional optical imaging technology, light field imaging, as a computational imaging technology, can image three-dimensional objects, while recording the two-dimensional spatial information of the target and the angle of light propagation, and then reconstruct the depth information of the target through a later data reconstruction algorithm. However, this data reconstruction algorithm cannot extract the spectral information of the target, and it processes the light field image in the frequency domain. The calculation process is relatively complicated, which is not conducive to practical application.

[0004] The multispectral light field camera is a combination of light field imaging technology and filter spectroscopic technology. It overcomes the drawback of traditional light field cameras that they can only obtain the three-dimensional spatial information of the target within a single exposure time. By introducing a filter array to obtain additional spectral information of the target, it can achieve simultaneous acquisition of the three-dimensional spatial information and spectral information of the target scene, and has good application prospects in precision agriculture, biomedicine, cultural relics protection and other fields.

[0005] So far, the existing reported multispectral light field camera data reconstruction algorithms focus on the research of multispectral image reconstruction, such as the multimodal camera based on light field camera in the relevant literature, the hyperspectral plenoptic camera in patent US11115573B2, and the multispectral light field camera in patent CN103234527A, which do not involve the reconstruction of depth images. However, the multispectral light field camera is an imaging system that can simultaneously collect five-dimensional light field information of the target, and the five-dimensional light field information includes two-dimensional spatial information, propagation angle information and spectral information. How to make full use of the five-dimensional light field information obtained by the optical lens and simultaneously invert the target multispectral image and depth image has become an urgent problem to be solved. Summary of the invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the disadvantage that the multispectral light field camera in the prior art cannot simultaneously invert the target multispectral image and the depth image.

[0007] In order to solve the above technical problems, the present invention provides a method for extracting a target depth image and a multispectral image, comprising:

[0008] Step S1: collecting a multispectral light field image of a target, and dividing the multispectral light field image into a plurality of sub-images;

[0009] Step S2: extracting effective pixels in each sub-image of the multispectral light field image;

[0010] Step S3: Select valid pixels at the same position in each sub-image, and reassemble the valid pixels at the same position to form a plurality of sub-aperture images;

[0011] Step S4: selecting effective pixels of different spectral channels from a plurality of sub-aperture images to form a target multispectral image;

[0012] At the same time, effective pixels of the same spectral channel are selected from a plurality of sub-aperture image sequences, and the target depth image is obtained by calculation based on the effective pixels of the same spectral channel.

[0013] In one embodiment of the present invention, the method for obtaining the target depth image based on effective pixel calculations of the same spectral channel in step S4 is: obtaining the target depth image based on effective pixel calculations of the same spectral channel using a digital refocusing algorithm.

[0014] In one embodiment of the present invention, the digital refocusing algorithm is constructed based on the original focus depth of the multispectral light field camera and the focus depth of the multispectral light field camera after refocusing.

[0015] In one embodiment of the present invention, the target depth image is obtained by calculating the effective pixels of the same spectral channel using a digital refocusing algorithm, and the formula is:

[0016]

[0017] Among them, I λ is the target depth image, L F is the effective pixel of the collected multispectral channel, F is the original focus depth of the multispectral light field camera, F' is the focus depth of the multispectral light field camera after refocusing, u, v, s, t, λ are the propagation angle coordinates, two-dimensional space coordinates and wavelength of the light respectively.

[0018] In one embodiment of the present invention, the method of step S2 comprises:

[0019] Calculating the centroid of each sub-image in the multispectral light field image;

[0020] Taking the centroid as the center, effective pixels in each sub-image are extracted through an interpolation resampling method.

[0021] In one embodiment of the present invention, before step S1, the step further includes: calibrating the multispectral light field camera.

[0022] In order to solve the above technical problems, the present invention provides a target depth image and multispectral image extraction system, comprising:

[0023] An acquisition module is used to acquire a multispectral light field image of a target through a multispectral light field camera, and to divide the multispectral light field image into a plurality of sub-images;

[0024] Extraction module: used for extracting effective pixels in each sub-image of the multispectral light field image;

[0025] Selection module: used to select valid pixels at the same position in each sub-image, and reassemble the valid pixels at the same position to form several sub-aperture images;

[0026] Construction module: used for selecting effective pixels of different spectral channels from a plurality of sub-aperture images to form a target multispectral image;

[0027] At the same time, it is used to select effective pixels of the same spectral channel from a plurality of sub-aperture image sequences, and calculate the target depth image according to the effective pixels of the same spectral channel.

[0028] In one embodiment of the present invention, the multispectral light field camera comprises:

[0029] A main lens, wherein the aperture diaphragm of the main lens is provided with a filter array, and the filter array is used for filtering light;

[0030] A microlens array is arranged at the image plane of the main lens, and is used to perform image segmentation on the image formed on the microlens array plane after the target passes through the main lens provided with a filter array;

[0031] The detector is arranged at the focal plane of the microlens array and is used for receiving and displaying the optical signal from the microlens array.

[0032] In one embodiment of the present invention, the microlens array is coupled to a detector.

[0033] In one embodiment of the present invention, the filter array includes a plurality of sub-filters, and the sub-filters are narrow-band filters or full-color band filters.

[0034] The above technical solution of the present invention has the following advantages compared with the prior art:

[0035] In order to make full use of the five-dimensional light field data collected by the optical lens (multispectral light field camera), the present invention adopts a new spectral light field data reconstruction algorithm, and inverts the target multispectral image and depth image from it, overcoming the drawbacks of the data reconstruction algorithm in the prior art that can only restore the multispectral image but cannot obtain the depth image, or the depth image calculation method is too complicated;

[0036] Under different demand backgrounds, the multi-spectral light field camera of the present invention can replace the main lens and the filter array to adapt to different application occasions;

[0037] The microlens array in the present invention is directly coupled to the detector, which avoids the problem that the detector chip may be damaged by removing the detector protection glass, greatly reduces the difficulty of coupling the microlens array and the detector, and is conducive to the mass production of cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0039] Figure 1 is a flow chart of the method of the present invention;

[0040] Figure 2 is a schematic diagram of the structure of a multi-spectral light field camera in an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of the structure of a filter array according to an embodiment of the present invention;

[0042] Figure 4 is a valid pixel map of a multispectral light field image in an embodiment of the present invention;

[0043] Figure 5 is a schematic diagram of the principle of extracting a sub-aperture image in an embodiment of the present invention;

[0044] Figure 6 2 is a schematic diagram of a transmittance curve of a sub-filter in an embodiment of the present invention;

[0045] Figure 7 is a schematic structural diagram of a microlens array in an embodiment of the present invention;

[0046] Figure 8 Schematic diagram of the structure of a microlens unit in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0048] Embodiment 1

[0049] Reference Figure 1 As shown, the present invention relates to a method for extracting a target depth image and a multispectral image, comprising:

[0050] Step S1: collecting a multispectral light field image of a target, and dividing the multispectral light field image into a plurality of sub-images;

[0051] Step S2: extracting effective pixels in each sub-image of the multispectral light field image;

[0052] Step S3: Select valid pixels at the same position in each sub-image, and reassemble the valid pixels at the same position to form a plurality of sub-aperture images;

[0053] Step S4: selecting effective pixels of different spectral channels from a plurality of the sub-aperture images to form the target multispectral image;

[0054] At the same time, effective pixels of the same spectral channel are selected from a plurality of sub-aperture image sequences, and the target depth image is obtained by calculation based on the effective pixels of the same spectral channel.

[0055] In order to make full use of the five-dimensional light field data collected by the optical lens, the present invention adopts a new spectral light field data reconstruction algorithm, and inverts the target multispectral image and depth image therefrom, which are introduced in detail below.

[0056] Furthermore, the method for obtaining the target depth image based on effective pixel calculations of the same spectral channel in step S4 is: obtaining the target depth image based on effective pixel calculations of the same spectral channel using a digital refocusing algorithm.

[0057] Furthermore, the digital refocusing algorithm is constructed based on the original focus depth of the multispectral light field camera and the focus depth after the multispectral light field camera is refocused.

[0058] Furthermore, the target depth image is obtained by using a digital refocusing algorithm based on effective pixel calculations of the same spectral channel, and the formula is:

[0059]

[0060] Among them, I λ is the target depth image, L Fis the effective pixel of the collected multispectral channel, F is the original focus depth of the multispectral light field camera, F' is the focus depth of the multispectral light field camera after refocusing, u, v, s, t, λ are the propagation angle coordinates, two-dimensional spatial coordinates and wavelength (i.e., five-dimensional light field data) of the light respectively.

[0061] Furthermore, the method of step S2 includes:

[0062] Calculating the centroid of each sub-image in the multispectral light field image;

[0063] Taking the centroid as the center, effective pixels in each sub-image are extracted through an interpolation resampling method.

[0064] Furthermore, before step S1, the method further includes: calibrating the multispectral light field camera.

[0065] Embodiment 2

[0066] This embodiment provides a target depth image and multispectral image extraction system, including:

[0067] An acquisition module is used to acquire a multispectral light field image of a target through a multispectral light field camera, and to divide the multispectral light field image into a plurality of sub-images;

[0068] Extraction module: used for extracting effective pixels in each sub-image of the multispectral light field image;

[0069] Selection module: used to select valid pixels at the same position in each sub-image, and reassemble the valid pixels at the same position to form several sub-aperture images;

[0070] Construction module: used for selecting effective pixels of different spectral channels from a plurality of sub-aperture images to form a target multispectral image;

[0071] At the same time, it is used to select effective pixels of the same spectral channel from a plurality of sub-aperture image sequences, and calculate the target depth image according to the effective pixels of the same spectral channel.

[0072] See also Figure 2 The multi-spectral light field camera (optical system) used in this embodiment is mainly composed of four parts: a main lens 1, a filter array 2, a microlens array 3 and a detector 4. Among them, the filter array 2 is located at the aperture stop of the main lens 1, and the detector 4 is located at the back focal plane of the microlens array 3. When the main lens 1 images the target, the light 5 emitted by the target is filtered by the filter array 2 and then incident on the microlens array 3. The microlens array 3 is located at the image plane of the main lens 1, and divides the target image plane into multiple sub-images. These sub-images are then projected onto the detector 4 via the microlens array 3, and are received and displayed by the detector 4.

[0073] It is worth mentioning that the microlens array 3 and the detector 4 in this embodiment are mechanically coupled, which avoids the problem that the detector chip may be damaged by removing the protective glass of the detector 4, greatly reduces the difficulty of coupling the microlens array 3 and the detector 4, and is conducive to the mass production of multi-spectral light field cameras.

[0074] See also Figure 3 , the number of sub-filters of the filter array 2 ranges from 9 to 10000. In this embodiment, the number of spectral channels of the filter array 2 is set to 9. Among them, the first 8 sub-filters are narrow-band filters, and the first 8 sub-filters correspond to 8 characteristic bands (blue-violet M: 405nm, blue B: 522nm, green G: 553nm, yellow Y: 573nm, red R: 673nm, near infrared I1: 723nm, near infrared I2: 760nm, near infrared I3: 900nm), and the spectral resolution is 2nm to 20nm. The 9th sub-filter P is a full-color band.

[0075] See also Figure 6 In this embodiment, the passband shape of the transmittance curve of the red R sub-filter of the filter array 2 is close to a rectangle, the spectral resolution is 10nm, and the peak transmittance is close to 0.9.

[0076] See also Figure 7 , the microlens array 3 includes a plurality of microlens units 3.1.

[0077] The material of the microlens array 3 in this embodiment is fused quartz JCSI, and 99%@400-900nm anti-reflection film is coated on both sides.

[0078] Furthermore, the micro lens unit 3.1 may be circular or rectangular in shape, and may be arranged in a rectangular or regular hexagonal arrangement. Figure 8 In this embodiment, the micro lens units 3.1 are square in shape and arranged in a rectangular manner.

[0079] In order to obtain the target depth image and the multispectral image, in the above embodiment 1, the multispectral light field camera needs to be calibrated and preprocessed first, and the specific steps are as follows:

[0080] (1) Accurately measure the transmittance of each sub-filter in the filter array 2 and the quantum efficiency curve of the detector 4 to obtain the true spectral radiation brightness of the target;

[0081] (2) Aim the multispectral light field camera at a standard white board to obtain a white image, and use the white image to determine the image plane position of the detector 4 corresponding to each microlens unit 3.1;

[0082] (3) Based on the results of the above two steps of calibration preprocessing, a mapping relationship between the collected white image and the five-dimensional light field data is established.

[0083] In the above technical solution, after the multispectral light field camera is calibrated, the multispectral light field camera is aimed at the object to be measured to collect the multispectral light field image of the target, and the spectral light field data reconstruction algorithm is applied to the obtained target original image. The specific algorithm steps are as follows:

[0084] (1) Calculate the centroid of each sub-image in the multispectral light field image, and extract the valid pixels in the sub-image by interpolating and resampling the adjacent four pixels with the centroid as the center. Figure 4 The effective pixel diagram of the multispectral light field image is shown, and its size is consistent with the size of the multispectral light field image collected by the multispectral light field camera, and Figure 4 The sub-images in (4 in total) are essentially valid pixel maps corresponding to the sub-images;

[0085] (2) Select valid pixels at the same position in each sub-image and reassemble them to form a series of sub-aperture images; see Figure 5 , by extracting valid pixels at the same position from each sub-image and rejoining these valid pixels, sub-aperture images of the target at different viewing angles can be obtained.

[0086] (3) Effective pixels of different spectral channels are selected from several sub-aperture images to form the target multispectral image.

[0087] (4) Select effective pixels of the same spectral channel from several sub-aperture images, use their spectral invariance to extract the parallax in each sub-aperture image, and apply the digital refocusing algorithm to obtain the target depth image. The specific expression of the digital refocusing algorithm is as follows:

[0088]

[0089] Among them, I λ is the depth image after refocusing, L F is the effective pixel of the collected multispectral channel, F is the original focus depth of the multispectral light field camera, F' is the focus depth of the multispectral light field camera after refocusing, u, v, s, t, λ are the propagation angle coordinates, two-dimensional spatial coordinates and wavelength (i.e., five-dimensional light field data) of the light respectively.

[0090] To summarize, the present embodiment selects effective pixels of different spectral channels from a plurality of sub-aperture images to form a target multispectral image; and selects effective pixels of the same spectral channel from a plurality of sub-aperture images, utilizes their spectral invariance, extracts the parallax in each sub-aperture image, and applies a digital refocusing algorithm to obtain a target depth image. The digital refocusing algorithm adopted in the present embodiment is simple, reliable, easy to implement, and applicable to a variety of practical scenarios.

[0091] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A method for extracting a target depth image and a multispectral image, characterized in that: include: Step S1: collecting a multispectral light field image of a target by a multispectral light field camera, and dividing the multispectral light field image into a plurality of sub-images; Step S2: extracting effective pixels in each sub-image of the multispectral light field image; The method of step S2 comprises: Calculating the centroid of each sub-image in the multispectral light field image; Extracting effective pixels in each sub-image by an interpolation resampling method with the centroid as the center; Step S3: Select valid pixels at the same position in each sub-image, and reassemble the valid pixels at the same position to form a plurality of sub-aperture images; Step S4: selecting effective pixels of different spectral channels from each of the sub-aperture images to form a target multispectral image; At the same time, effective pixels of the same spectral channel are selected from each of the sub-aperture image sequences, and a target depth image is calculated based on the effective pixels of the same spectral channel; The method for obtaining the target depth image based on the effective pixels of the same spectral channel in step S4 is: using a digital refocusing algorithm to obtain the target depth image based on the effective pixels of the same spectral channel, the formula is: Among them, I λ is the target depth image, L F is the effective pixel of the collected multispectral channel, F is the original focus depth of the multispectral light field camera, F' is the focus depth of the multispectral light field camera after refocusing, u, v, s, t, λ are the propagation angle coordinates, two-dimensional space coordinates and wavelength of the light respectively.

2. The method for extracting target depth image and multispectral image according to claim 1, characterized in that: The digital refocusing algorithm is constructed based on the original focus depth of the multispectral light field camera and the focus depth after the multispectral light field camera is refocused.

3. The method for extracting target depth image and multispectral image according to claim 1, characterized in that: Before step S1, the method further includes: calibrating the multispectral light field camera.

4. A target depth image and multispectral image extraction system, characterized in that: include: An acquisition module is used to acquire a multispectral light field image of a target through a multispectral light field camera, and to divide the multispectral light field image into a plurality of sub-images; Extraction module: used for extracting effective pixels in each sub-image of the multispectral light field image; The extraction module comprises: Calculating the centroid of each sub-image in the multispectral light field image; Extracting effective pixels in each sub-image by an interpolation resampling method with the centroid as the center; Selection module: used to select valid pixels at the same position in each sub-image, and reassemble the valid pixels at the same position to form several sub-aperture images; Construction module: used for selecting effective pixels of different spectral channels from each of the sub-aperture images to form a target multispectral image; At the same time, it is used to select effective pixels of the same spectral channel from each of the sub-aperture image sequences, and calculate the target depth image according to the effective pixels of the same spectral channel; The construction module calculates the target depth image based on the effective pixels of the same spectral channel, including: using a digital refocusing algorithm to calculate the target depth image based on the effective pixels of the same spectral channel, the formula is: Among them, I λ is the target depth image, L F is the effective pixel of the collected multispectral channel, F is the original focus depth of the multispectral light field camera, F' is the focus depth of the multispectral light field camera after refocusing, u, v, s, t, λ are the propagation angle coordinates, two-dimensional space coordinates and wavelength of the light respectively.

5. The target depth image and multispectral image extraction system according to claim 4, characterized in that: The multispectral light field camera comprises: A main lens (1), wherein the aperture diaphragm of the main lens (1) is provided with a filter array (2), and the filter array (2) is used for filtering light; A microlens array (3) is arranged at the image plane of the main lens (1) and is used to perform image segmentation on an image formed on the plane of the microlens array (3) after the target passes through the main lens (1) provided with a filter array (2); The detector (4) is arranged at the focal plane of the microlens array (3) and is used for receiving and displaying the light signal from the microlens array (3).

6. The target depth image and multispectral image extraction system according to claim 5, characterized in that: The microlens array (3) is coupled to a detector (4).

7. The target depth image and multispectral image extraction system according to claim 5, characterized in that: The filter array (2) comprises a plurality of sub-filters, and the sub-filters are narrow-band filters or full-color band filters.

Images