Multispectral demosaicing method based on variable lens

Abstract

The present invention belongs to the field of optical imaging technology, and in particular, relates to a multispectral demosaicing method based on a variable lens. It comprises: S1: building a demosaicing device, the demosaicing device comprising a variable lens module and a spectral camera arranged in sequence according to the direction of the light path; S2: after aligning the variable lens module with the target to be measured, adjusting the variable lens module so that the target to be measured is imaged on the spectral camera at least once, and a multispectral mosaic image corresponding to the number of imaging times is obtained; S3: according to the quantitative relationship between the number of imaging times in step S2 and the number of filtering channels in a single filtering cycle of the spectral camera, the demosaicing processing of the multispectral mosaic image is realized based on different demosaicing algorithms. The present invention can increase the sampling rate of real data and reduce the loss of pattern texture information. In an ideal case, a high-resolution multispectral image can be directly obtained without using an algorithm.

Description

Technical Field

[0001] The invention belongs to the technical field of optical imaging, and in particular relates to a multi-spectral demosaicing method based on a variable lens. Background Art

[0002] Compared with color images, multispectral images contain richer spectral information and have advantages in practical tasks such as food safety inspection, land cover classification and target tracking. Due to the structure of the sensor, traditional multispectral imagers require multiple exposures or scans and are not suitable for capturing multispectral and spatial information of moving objects in dynamic scenes.

[0003] There are many types of spectral cameras on the market, which can also be called video-level spectral cameras or mosaic spectral cameras. Their spectral methods include multi-spectral filter array spectral, material dyeing spectral, colloidal quantum dot spectral, metasurface spectral and micro-nano manufacturing technology spectral. Spectral cameras designed and developed using the above technologies can all be called spectral cameras. The difference is that spectral cameras based on multi-spectral filter arrays can directly obtain narrowband spectral information, while spectral cameras based on material dyeing and other spectral methods need to be calculated to obtain a single-channel spectral image. Snapshot spectral cameras obtain spectral information of the target to be measured through a single exposure. The spectral information obtained in each pixel is distributed according to a fixed period. Therefore, the spectral information in this image is mosaic.

[0004] The spectral camera can obtain the multispectral information of the target from a single two-dimensional image with one exposure, but it loses the spatial information of the target. The high-resolution multispectral image obtained is Figure 2 As shown in the figure, the spectral and spatial information of each spectrum segment of the high-resolution multispectral image obtained at this time is incomplete, and it is a mosaic image. In order to obtain a high-resolution multispectral image, it is necessary to demosaic and reconstruct the obtained multispectral image. Figure 2 As shown, in order to obtain spectral information and image information at the same time, the original multispectral image needs to be divided into multiple single channel components (when the channel period is 3×3, the number of divisions is 9). Each single channel image is a downsampling result of the original multispectral image. For the single channel component image of the original multispectral image, only one pixel value in a single period is the true value for the channel, and there are too few valid pixels. If the method based on the Bayer filter array 1 is used to interpolate and reconstruct the sparse image, the mosaic effect of the obtained image is serious, and the edge processing effect of the object in the image is poor.

[0005] At present, the methods for de-mosaicing multispectral images obtained by spectral cameras are all implemented at the software algorithm level, which has the following problems: the super-resolution interpolation algorithm based on the Bayer filter array cannot be directly applied to spectral cameras. Among them, the principle of obtaining color images based on the Bayer filter array is as follows: Figure 1 As shown, at the position of 2×2 pixels on the imaging chip space, three filter units, red, green, and blue, are set, and the three filter units transmit red, green, and blue light respectively, and the position of 2×2 pixels is used as a filter unit, namely, RGB unit 2. The Bayer filter array 1 is composed of multiple RGB units 2. When imaging, it is possible to obtain the red, green, and blue images of the material in one image at the same time, and then split the images of the red, green, and blue bands to obtain mosaic images of their respective bands. The sampling rate of the red band is 25%, the sampling rate of the green band is 50%, and the sampling rate of the blue band is 25%. The color image is processed using a simple Bayer interpolation algorithm to achieve de-mosaic reconstruction of the image. Spectral cameras all obtain two-dimensional mosaic multi-spectral images through single exposure (taking the filter period 3×3 as an example), as shown Figure 3 As shown in the figure, in a single filtering period of the spectral camera, only one pixel has a true value for the channel, and the actual sampling rate is 11% (if the filtering period is 4×4, the actual sampling rate is 6.25%, and when the filtering period is 5×5, the actual sampling rate is 4%). That is, the fewer the effective pixels, the greater the requirements for the reconstruction algorithm. At the same time, the zipper effect will inevitably occur when spectrally reconstructing the edge of the image, resulting in blurred image, and complex texture and high-frequency information such as spectrum are difficult to recover. Summary of the invention

[0006] In view of this, the present invention aims to provide a multispectral demosaicing method based on a variable lens to solve the problem that the existing technology will inevitably produce a zipper effect when performing spectral reconstruction on the edge of the image, resulting in blurred images and difficult recovery of high-frequency information such as complex textures and spectra. The present invention can increase the sampling rate of real data and reduce the loss of pattern texture information. Under ideal circumstances, high-resolution multispectral images can be directly acquired without using algorithms.

[0007] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0008] A multispectral demosaicing method based on a variable lens specifically comprises the following steps:

[0009] S1: Building a demosaicing device, which includes a variable lens module and a spectral camera arranged in sequence according to the direction of the light path;

[0010] S2: After aligning the variable lens module with the target to be measured, adjusting the variable lens module so that the target to be measured is imaged at least once on the spectral camera, and obtaining a multispectral mosaic image corresponding to the number of imaging times;

[0011] S3: According to the quantitative relationship between the number of imaging times in step S2 and the number of filtering channels in a single filtering cycle of the spectral camera, demosaicing of the multispectral mosaic image is implemented based on different demosaicing methods.

[0012] Further, in step S1, the variable lens module includes a transmission mirror and a two-dimensional motor, and the transmission mirror is connected to the two-dimensional motor;

[0013] Taking the geometric center of the transmission mirror as the origin, a two-dimensional rectangular coordinate system is established on the transmission surface of the transmission mirror;

[0014] The two-dimensional motor controls the transmission mirror to rotate along the X-axis of the two-dimensional rectangular coordinate system or along the Y-axis of the two-dimensional rectangular coordinate system.

[0015] Furthermore, the filtering period of the spectral camera is n×n; each filtering period includes m×m filtering channels; and the number of filtering periods occupied by the mapping area of ​​the target to be measured on the spectral camera is (n-1)×(n-1).

[0016] Furthermore, in step S2, at least one imaging corresponds to different pixels of the imaging target surface of the spectral camera.

[0017] Furthermore, step S3 specifically includes the following steps:

[0018] S31: if the number of imaging in step S2 is the same as the number of filtering channels in a single filtering cycle of the spectral camera, the first demosaicing method is used to perform spectral splitting and merging on all multispectral mosaic images to achieve demosaicing of the multispectral mosaic images and obtain a multispectral image of the target to be measured, otherwise, step S32 is executed;

[0019] S32: Using the second de-mosaicing method to perform spectral splitting and merging on all multispectral mosaic images, realize de-mosaicing of the multispectral mosaic images, and obtain a multispectral image of the target to be measured.

[0020] Furthermore, step S31 specifically includes the following steps:

[0021] S311: split all multispectral mosaic images into single-spectrum images with a single filter channel;

[0022] S312: Spectrally merge the single-spectrum images with the same filtering channel information to achieve de-mosaicing of the multi-spectral mosaic image, and obtain a multi-spectral image of the target to be measured.

[0023] Further, step S32 specifically includes the following steps:

[0024] S321: split all multispectral mosaic images into single-spectrum images with a single filtering channel;

[0025] S322: construct a reconstruction module, input all single-spectrum images into the reconstruction module in sequence for demosaicing operation, and obtain a multi-spectral image of the target to be measured.

[0026] Furthermore, the reconstruction module includes N cascaded image reconstruction submodules, N=round(mL / m)+m, round represents rounding up, L represents the number of sampling times, and 1≤L<9, and m represents the square root of the number of filtering channels in a single cycle, that is, the number N of image reconstruction submodules is positively correlated with the number of sampling times and the number of channels in a single filtering cycle;

[0027] The image reconstruction submodule includes a first 1×1 convolutional layer, a second 1×1 convolutional layer, a first 3×3 convolutional layer, a second 3×3 convolutional layer and a pixel-level attention submodule. The single-spectrum image is processed by the first 1×1 convolutional layer, the first 3×3 convolutional layer, the pixel-level attention submodule and the second 1×1 convolutional layer in sequence to obtain a feature map A1. The single-spectrum image is input into the second 3×3 convolutional layer for processing to obtain a feature map A2. The feature maps A1, A2 and the single-spectrum image are added to obtain an output feature map B1 of the image reconstruction submodule.

[0028] The output feature map of the current image reconstruction submodule is used as the input feature of the next image reconstruction submodule, and the above operation is repeated until the output feature map BN of the Nth image reconstruction submodule is obtained. The set of feature maps BN corresponding to each single spectrum image is used as the multispectral image of the target to be measured.

[0029] Furthermore, the pixel-level attention submodule includes a first fully connected layer, a second fully connected layer, a ReLU activation function and a Sigmoid activation function. The feature map C1 is input into the pixel-level attention submodule, and the feature map C1 is processed by the first fully connected layer, the ReLU activation function, the second fully connected layer and the Sigmoid activation function in sequence to obtain the feature map C2; the effective pixel value of each pixel point of the single-spectrum image corresponding to the feature map C2 is obtained; the feature map C2 is multiplied by the corresponding element of the effective pixel value of each pixel point to obtain the output feature map C3 of the pixel-level attention submodule.

[0030] Furthermore, the method for solving the effective pixel value of each pixel point of the single-spectrum image is: obtain the pixel value of the pixel point to be solved, and the pixel values ​​of each pixel point adjacent to the pixel point to be solved; and take the average pixel value of the pixel point to be solved and each pixel point adjacent to the pixel point to be solved as the effective pixel value of the pixel point to be solved.

[0031] Compared with the prior art, the invention can achieve the following beneficial effects:

[0032] The multispectral de-mosaicing method based on a variable lens created by the present invention is applicable to various types of snapshot spectral cameras. In order to solve the problem of low image sampling rate output by snapshot spectral cameras, the present invention uses a variable lens module to periodically map the image of the same target to different spectral periodic positions on the detector, so that each spectral channel can obtain more spatial position information, increase the sampling rate of real data, and reduce the loss of pattern texture information. In an ideal case, a high-resolution multispectral image can be directly obtained without using an algorithm; when conditions do not support it, the number of sampling times can be reduced, and at a higher sampling rate, the difficulty of the super-resolution algorithm can be reduced to obtain a high-resolution multispectral image; if the requirement is to quickly obtain a high-resolution multispectral image of a target to be measured, in an extreme case, a single-exposure image is input into the designed de-mosaicing algorithm based on deep learning to obtain a high-resolution multispectral image of the target to be measured. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:

[0034] Figure 1 A schematic diagram of obtaining a color image based on a Bayer filter array according to the present invention;

[0035] Figure 2 It is a schematic diagram of the imaging technology based on multi-spectral filter array of the present invention;

[0036] Figure 3 It is the first channel image of single imaging based on multi-spectral filter array of the present invention;

[0037] Figure 4 A schematic flow chart of a multi-spectral demosaicing method based on a variable lens according to an embodiment of the present invention;

[0038] Figure 5 A schematic diagram of the structure of a variable lens module according to an embodiment of the present invention;

[0039] Figure 6 A schematic diagram of two degrees of freedom of the transmission mirror described in an embodiment of the present invention;

[0040] Figure 7 A schematic diagram of the change in direction of refracted light after the state of the transmission mirror is changed according to an embodiment of the present invention;

[0041] Figure 8 A schematic diagram of the network structure of the reconstruction module according to an embodiment of the present invention;

[0042] Fig. 9 A schematic diagram of the network structure of the pixel-level attention submodule described in the embodiment of the present invention;

[0043] Fig.10 A schematic diagram of obtaining different spectra at a certain point by changing the state of a variable lens module as described in an embodiment of the present invention;

[0044] FIG. 11 ( a ) is a schematic diagram of mapping an image A′ onto a filtering channel in sequence along a row direction with a vertex of a filtering period as an origin according to an embodiment of the present invention;

[0045] FIG. 11 ( b ) is a schematic diagram of mapping an image A′ onto the filter channels in a column direction with the vertex of the filter period as the origin according to an embodiment of the present invention;

[0046] Fig.12 A schematic diagram of the corresponding relationship between point A and surrounding points and the filtering channel according to an embodiment of the present invention;

[0047] Fig.13 A schematic diagram of spectral data output by a snapshot spectral camera according to an embodiment of the present invention after a single exposure;

[0048] Fig.14 A mosaic multispectral image of a target to be measured obtained by changing the state of a variable lens module nine times as described in an embodiment of the present invention;

[0049] Fig.15 A schematic diagram of spectrum splitting and merging of mosaic multi-spectral images according to an embodiment of the present invention;

[0050] Fig.16 A schematic diagram of splitting and merging spectral information of the first filtering channel described in an embodiment of the present invention;

[0051] Fig.17 A schematic diagram of completing the spectral splitting and merging of 9 mosaic multi-spectral images according to an embodiment of the present invention;

[0052] Fig.18 The present invention creates an embodiment of obtaining 9 groups of mosaic multi-spectral images of the target to be measured by changing the state of the transmission mirror;

[0053] Fig.19 This is a multispectral image obtained by changing the state of the transmission mirror five times as described in the embodiment of the present invention.

[0054] Description of reference numerals:

[0055] 1. Bayer filter array; 2. RGB unit; 3. Color sensor chip; 4. Snapshot spectral camera imaging chip; 5. Incident light; 6. Variable lens module; 6-1. Two-dimensional motor; 6-2. Transmission mirror; 7. Spectral camera; 8. First state; 9. Second state; 10. Refracted light. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0057] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0058] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0059] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0060] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0061] like Figure 4 As shown, a multi-spectral demosaicing method based on a variable lens specifically includes the following steps:

[0062] S1: Building a demosaicing device, which includes a variable lens module 6 and a spectral camera 7 arranged in sequence according to the direction of the light path;

[0063] S2: After the variable lens module 6 is aligned with the target to be measured, the variable lens module 6 is adjusted so that the target to be measured is imaged at least once on the spectral camera 7, and a multispectral mosaic image corresponding to the number of imaging times is obtained;

[0064] The imaging that is performed at least once corresponds to different pixels on the imaging target surface of the spectral camera 7 .

[0065] S3: According to the quantitative relationship between the number of imaging times in step S2 and the number of filtering channels in a single filtering cycle of the spectral camera 7, demosaicing of the multispectral mosaic image is implemented based on different demosaicing methods.

[0066] The filtering period of the spectral camera 7 is n×n; each filtering period includes m×m filtering channels; the number of filtering periods occupied by the mapping area of ​​the target to be measured on the spectral camera 7 is (n-1)×(n-1).

[0067] S31: If the number of imaging in step S2 is the same as the number of filtering channels in a single filtering cycle of the spectral camera 7, the first de-mosaicing method is used to perform spectral splitting and merging on all multispectral mosaic images to achieve de-mosaicing of the multispectral mosaic images and obtain a multispectral image of the target to be measured, otherwise, step S32 is executed;

[0068] S311: split all multispectral mosaic images into single-spectrum images with a single filter channel;

[0069] S312: Spectrally merge the single-spectrum images with the same filtering channel information to achieve de-mosaicing of the multi-spectral mosaic image, and obtain a multi-spectral image of the target to be measured.

[0070] S32: Using the second de-mosaicing method to perform spectral splitting and merging on all multispectral mosaic images, realize de-mosaicing of the multispectral mosaic images, and obtain a multispectral image of the target to be measured.

[0071] S321: split all multispectral mosaic images into single-spectrum images with a single filtering channel;

[0072] S322: construct a reconstruction module, input all single-spectrum images into the reconstruction module in sequence for demosaicing operation, and obtain a multi-spectral image of the target to be measured.

[0073] The de-mosaicing device provided by the present invention is as follows Figure 5 As shown, it includes a variable lens module 6 and a spectral camera 7 which are arranged in sequence according to the direction of the light path. The variable lens module 6 includes a transmission mirror 6-2 and a two-dimensional motor 6-1. The transmission mirror 6-2 is connected to the two-dimensional motor 6-1.

[0074] like Figure 6 As shown, a two-dimensional rectangular coordinate system is established on the transmission surface of the transmission mirror 6-2 with the geometric center of the transmission mirror 6-2 as the origin; the two-dimensional motor 6-1 controls the transmission mirror 6-2 to rotate along the X-axis or the Y-axis of the two-dimensional rectangular coordinate system.

[0075] The transmission mirror 6-2 has two degrees of freedom, rotating around the X-axis and the Y-axis. The two-dimensional motor 6-1 changes the state of the transmission mirror 6-2 by changing the slight rotation of the transmission mirror 6-2 in the directions of the two degrees of freedom, such as Figure 7 As shown, when the state of the incident light 5 remains unchanged, if the transmission mirror 6-2 changes from the first state 8 to the second state 9, the angle of the refracted light 10 will change.

[0076] like Figure 8 As shown, the reconstruction module includes N cascaded image reconstruction submodules, N=round(mL / m)+m, round means rounding up, L means the number of sampling times, and 1≤L<9, m means the square root of the number of filtering channels in a single cycle, that is, the number of image reconstruction submodules N is positively correlated with the number of sampling times and the number of channels in a single filtering cycle;

[0077] The image reconstruction submodule includes a first 1×1 convolutional layer, a second 1×1 convolutional layer, a first 3×3 convolutional layer, a second 3×3 convolutional layer and a pixel-level attention submodule. The single-spectrum image is processed by the first 1×1 convolutional layer, the first 3×3 convolutional layer, the pixel-level attention submodule and the second 1×1 convolutional layer in sequence to obtain a feature map A1. The single-spectrum image is input into the second 3×3 convolutional layer for processing to obtain a feature map A2. The feature maps A1, A2 and the single-spectrum image are added to obtain an output feature map B1 of the image reconstruction submodule.

[0078] The output feature map of the current image reconstruction submodule is used as the input feature of the next image reconstruction submodule, and the above operation is repeated until the output feature map BN of the Nth image reconstruction submodule is obtained. The set of feature maps BN corresponding to each single spectrum image is used as the multispectral image of the target to be measured.

[0079] like Fig. 9As shown, the pixel-level attention submodule includes a first fully connected layer, a second fully connected layer, a ReLU activation function and a Sigmoid activation function. The feature map C1 is input into the pixel-level attention submodule, and the feature map C1 is processed by the first fully connected layer, the ReLU activation function, the second fully connected layer and the Sigmoid activation function in sequence to obtain the feature map C2; the effective pixel value of each pixel point of the single spectrum image corresponding to the feature map C2 is obtained; the feature map C2 is multiplied by the corresponding element of the effective pixel value of each pixel point to obtain the output feature map C3 of the pixel-level attention submodule.

[0080] The method for solving the effective pixel value of each pixel of the single-spectrum image is as follows: obtain the pixel value of the pixel to be solved and the pixel values ​​of each pixel adjacent to the pixel to be solved; take the average pixel value of the pixel to be solved and each pixel adjacent to the pixel to be solved as the effective pixel value of the pixel to be solved (assuming that the pixel to be solved is located in the upper left corner of the single-spectrum image, then the number of pixels adjacent to it is three. At this time, the pixel values ​​of the pixel to be solved and the three adjacent pixels are averaged to obtain the effective pixel value of the pixel to be solved; assuming that the pixel to be solved is located in the center of the single-spectrum image, then the number of pixels adjacent to it is eight. At this time, the pixel values ​​of the pixel to be solved and the eight adjacent pixels are averaged to obtain the effective pixel value of the pixel to be solved).

[0081] like Fig.10 As shown in the figure, assuming that the filter period of the spectral camera 7 is 3×3, the light emitted from a certain point of the target to be measured in the space passes through the variable lens module 6, and its image is mapped to the position of the first filter channel in a certain filter period of the spectral camera 7. At the same time, the spectral camera 7 is controlled to be exposed to collect image information, that is, the imaging area of ​​the spectral camera 7 corresponding to the position of the first filter channel in the current filter period obtains the spectral information of the point at the wavelength corresponding to the first band. Similarly, for a certain point A in the space, when the state of its incident light 5 remains unchanged, by changing the state of the variable lens module 6, its image A' is respectively mapped to the 9 filter channels of the same filter period. After the spectral camera 7 is exposed accordingly, the 9 spectrum band information of the point is obtained. For the spectral camera 7 with a filter period of 4×4 or 5×5, it is necessary to adjust the state of the variable lens module 6 so that its image A' is correspondingly mapped to 16 or 25 filter channels of the same filter period, and the spectral camera 7 is controlled to be exposed at the same time to obtain the 16 spectrum band information or 25 spectrum band information of point A.

[0082] The method for obtaining the multispectral information of point A is: using the variable lens module 6 to map its image A' to all the filter channels of a certain filter period of the spectral camera 7, so as to obtain its multispectral information, which is independent of the order of mapping to each filter channel. Therefore, the mapping order can be arbitrarily selected to complete the acquisition of the multispectral information of point A. In order to obtain the multispectral information of point A more quickly and accurately, the mapping order needs to be determined.

[0083] Take the spectral camera 7 with a filtering period of 3×3 as an example.

[0084] As shown in Figure 11 (a) and Figure 11 (b), the vertex of a single filtering cycle is used as the initial position, and mapping is performed in sequence in rows or columns. As shown in Figure 11 (a), the vertex of the upper left corner of the filtering cycle is used as the initial position, and mapping is performed in the row direction. First, the image A' of point A is mapped to the position of the first filtering channel of the current filtering cycle, and the spectral camera 7 is controlled to be exposed. The exposure time is t1, and the spectral camera 7 outputs the spectral information of point A in the first filtering channel; then, by adjusting the state of the transmission mirror 6-2 by adjusting the two-dimensional motor 6-1, the image A' of point A is moved from the first filtering channel to the second filtering channel, and the moving time is recorded as t2, and the spectral camera 7 is controlled to be exposed. At this time, the moving time t2 is also the waiting time required by the spectral camera 7 between two adjacent exposure times, and the spectral camera 7 outputs the spectral information of point A in the second filtering channel; repeat the above steps until the spectral information of the 9 spectrum segments of point A is obtained. The mapping method of Figure 11 (b) is the same as that of Figure 11 (a). Theoretically, the time required to obtain the multispectral information of point A using this mapping method is T2=9t1+8t2.

[0085] like Fig.12 As shown, each point on the target to be measured only obtains the spectral information of a certain filter channel in the filter cycle. At this time, the image output by the spectral camera 7 is a mosaic multi-spectral image, and the spectral information of each pixel in the image strictly corresponds to the spectral information corresponding to the filter channel.

[0086] like Fig.13 As shown, if you want to obtain the spectral information corresponding to all the filtering channels of all the points on the target C in a single filtering cycle, that is, to obtain the multi-spectral information of the target C, you need to change the state of the transmission mirror 6-2 through the two-dimensional motor 6-1 according to the mapping method proposed above, and obtain the multi-spectral information of point A. At this time, the points in the neighborhood of point A also obtain multi-spectral information. Similarly, all the points on the target C obtain multi-spectral information, such as Fig.14 As shown, the obtained multiple mosaic multispectral images are spectrally split and merged, so that each image is a spectral image of the same spectrum segment of the target C.

[0087] like Fig.15 As shown, taking a mosaic multispectral image as an example, before performing spectral splitting and merging, it is necessary to generate multiple blank images of the same size as the mosaic multispectral image, and the number of blank images is the same as the number of filter channels contained in a single filter period of the spectral camera 7, that is, when the single filter period of the spectral camera 7 is 3×3, 4×4 or 5×5, the number of blank images that need to be generated is 9, 16 or 25, respectively, which are respectively called the 1st spectral segment, the 2nd spectral segment, ..., the 25th spectral segment; then, the spectral information of the pixel points corresponding to a certain filter channel in all filter periods of the mosaic multispectral image is taken out and stored in the corresponding position of the corresponding spectral segment. Taking the first filter channel as an example, in the mosaic multispectral image, the spectral information of the pixel points corresponding to the first filter channel of all filter periods is taken out, and the position information of each pixel point is recorded. According to the position information of each pixel point, the corresponding spectral information is stored in the first spectral segment, as shown in FIG. Fig.16 As shown, the splitting and merging of the spectral information corresponding to the first filter channel of all filter cycles in the current image are completed, and then the splitting and merging of the spectral information corresponding to the second filter channel to the ninth filter channel are performed in sequence according to the above steps to obtain a multispectral image of the target to be measured.

[0088] The following is a detailed introduction to the two demosaicing methods:

[0089] The first de-mosaicing method: when the number of times the state of the variable lens module 6 is changed is equal to the number of all filter channels in a single filter period of the spectral camera 7, that is, the variable lens module 6 needs to be used to directly obtain a high-resolution multispectral image of the target to be measured. Taking the spectral camera 7 with a filter period of 3×3 as an example, first, adjust the transmission mirror 6-2 so that the image of the target to be measured is completely mapped on the spectral camera 7; then, by changing the state of the transmission mirror 6-2, 9 mosaic multispectral images of the target to be measured are obtained; finally, the mosaic multispectral image is spectrally split and merged, as shown in FIG. Fig.17 Based on this, the acquisition of multispectral images is realized, and high-resolution multispectral images of the target to be measured can be obtained without using any algorithm, such as Fig.18 As shown, the spectral data processing is to perform spectral splitting and spectral merging on the mosaic multi-spectral image output by the spectral camera 7, and output the multi-spectral image.

[0090] The second de-mosaicing method: To improve the sampling speed of multispectral images and ensure that the current minimum sampling rate is close to the minimum sampling rate of color images (the minimum sampling rate of color images is 25%), the speed of image acquisition by the spectral camera 7 is improved by reducing the adjustment of the variable lens module 6. For example, the filter period of the spectral camera 7 is 3×3, and the target to be measured is acquired 5 times. Fig.19As shown, a sparse mosaic multispectral image is obtained at this time, and the spatial information is incomplete. The present invention demosaices and reconstructs the multispectral image by constructing a reconstruction module.

[0091] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0092] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multispectral demosaicing method based on a variable lens, characterized in that: The specific steps include: S1: Building a demosaicing device, wherein the demosaicing device includes a variable lens module and a spectral camera arranged in sequence according to the direction of the light path; S2: after aligning the variable lens module with the target to be measured, adjusting the variable lens module so that the target to be measured is imaged on the spectral camera at least once, and obtaining a multispectral mosaic image corresponding to the number of imaging times; S3: according to the quantitative relationship between the number of imaging times in step S2 and the number of filtering channels in a single filtering cycle of the spectral camera, a demosaicing process is implemented on the multispectral mosaic image based on different demosaicing methods; S31: if the number of imaging in step S2 is the same as the number of filtering channels in a single filtering cycle of the spectral camera, the first demosaicing method is used to perform spectral splitting and merging on all multispectral mosaic images to achieve demosaicing of the multispectral mosaic images and obtain a multispectral image of the target to be measured, otherwise, step S32 is executed; S311: split all multispectral mosaic images into single-spectrum images with a single filter channel; S312: spectrally merging the single-spectrum images with the same filtering channel information to achieve demosaicing of the multispectral mosaic image, thereby obtaining a multispectral image of the target to be measured; S32: using the second demosaicing method to perform spectral splitting and merging on all the multispectral mosaic images, so as to achieve demosaicing of the multispectral mosaic images and obtain a multispectral image of the target to be measured; S321: split all multispectral mosaic images into single-spectrum images with a single filtering channel; S322: construct a reconstruction module, input all single-spectrum images into the reconstruction module in sequence for demosaicing operation, and obtain a multi-spectral image of the target to be measured.

2. The multispectral demosaicing method based on a variable lens according to claim 1, characterized in that: In step S1, the variable lens module includes a transmission mirror and a two-dimensional motor, and the transmission mirror is connected to the two-dimensional motor; Taking the geometric center of the transmission mirror as the origin, a two-dimensional rectangular coordinate system is established on the transmission surface of the transmission mirror; The two-dimensional motor controls the transmission mirror to rotate along the X-axis of the two-dimensional rectangular coordinate system or along the Y-axis of the two-dimensional rectangular coordinate system.

3. The multispectral demosaicing method based on a variable lens according to claim 1, characterized in that: The filtering period of the spectral camera is n×n; each filtering period includes m×m filtering channels; the number of filtering periods occupied by the mapping area of ​​the target to be measured on the spectral camera is (n-1)×(n-1).

4. The multispectral demosaicing method based on a variable lens according to claim 1, characterized in that: In step S2, at least one imaging corresponds to different pixels of the imaging target surface of the spectral camera.

5. The multispectral demosaicing method based on a variable lens according to claim 1, characterized in that: The reconstruction module includes N cascaded image reconstruction submodules, where N=round(mL / m)+m, where round means rounding up, L means the number of sampling times, and 1≤L<9, and m means the square root of the number of filtering channels in a single cycle; The image reconstruction submodule includes a first 1×1 convolutional layer, a second 1×1 convolutional layer, a first 3×3 convolutional layer, a second 3×3 convolutional layer and a pixel-level attention submodule. The single-spectrum image is processed by the first 1×1 convolutional layer, the first 3×3 convolutional layer, the pixel-level attention submodule and the second 1×1 convolutional layer in sequence to obtain a feature map A1. The single-spectrum image is input into the second 3×3 convolutional layer for processing to obtain a feature map A2. The feature map A1, the feature map A2 and the single-spectrum image are added to obtain an output feature map B1 of the image reconstruction submodule. The output feature map of the current image reconstruction submodule is used as the input feature of the next image reconstruction submodule, and the above operation is repeated until the output feature map BN of the Nth image reconstruction submodule is obtained. The set of feature maps BN corresponding to each single spectrum image is used as the multispectral image of the target to be measured.

6. The multispectral demosaicing method based on a variable lens according to claim 5, characterized in that: The pixel-level attention submodule includes a first fully connected layer, a second fully connected layer, a ReLU activation function and a Sigmoid activation function. The feature map C1 is input into the pixel-level attention submodule, and the feature map C1 is processed by the first fully connected layer, the ReLU activation function, the second fully connected layer and the Sigmoid activation function in sequence to obtain the feature map C2; the effective pixel value of each pixel point of the single spectrum image corresponding to the feature map C2 is obtained; the feature map C2 is multiplied by the corresponding element of the effective pixel value of each pixel point to obtain the output feature map C3 of the pixel-level attention submodule.

7. The multi-spectral demosaicing method based on a variable lens according to claim 6, characterized in that: The method for solving the effective pixel value of each pixel point of the single-spectrum image is as follows: obtain the pixel value of the pixel point to be solved and the pixel values ​​of each pixel point adjacent to the pixel point to be solved; and take the average pixel value of the pixel point to be solved and each pixel point adjacent to the pixel point to be solved as the effective pixel value of the pixel point to be solved.

Images