Multispectral image demosaicing method based on galvanometer

Abstract

The present invention belongs to the field of optical imaging technology, and in particular, relates to a multispectral image demosaicing method based on a galvanometer. The method comprises: S1: building a demosaicing device, the demosaicing device comprising a lens, a galvanometer module and a spectral camera arranged in sequence according to the direction of the optical path; S2: aiming the lens at a target to be measured, adjusting the galvanometer module, so that the target to be measured is imaged on the spectral camera at least once, and a multispectral mosaic image corresponding to the number of imaging times is obtained; S3: according to the quantitative relationship between the number of imaging times in step S2 and the number of spectral channels in a single filtering cycle of the spectral camera, the demosaicing processing of the multispectral mosaic image is realized based on different demosaicing methods. While obtaining the multispectral information of the target to be measured, the present invention can obtain more spatial position information thereof, and by increasing the sampling rate of the target to be measured, the loss of pattern texture information is reduced, and fast and accurate multispectral imaging is realized.

Description

Technical Field

[0001] The invention belongs to the technical field of optical imaging, and in particular relates to a multi-spectral image de-mosaicing method based on a galvanometer. Background Art

[0002] Compared with color images, multispectral images contain richer spectral information and have advantages in practical tasks such as food safety inspection, land cover classification and target tracking. Due to the structure of the sensor, traditional multispectral imagers require multiple exposures or scans and are not suitable for capturing multispectral and spatial information of moving objects in dynamic scenes.

[0003] There are many types of spectral cameras on the market, which can also be called video-level spectral cameras or mosaic spectral cameras. Their spectral methods include multi-spectral filter array spectral, material dyeing spectral, colloidal quantum dot spectral, metasurface spectral and micro-nano manufacturing technology spectral. Spectral cameras designed and developed using the above technologies can all be called spectral cameras. The difference is that spectral cameras based on multi-spectral filter arrays can directly obtain narrowband spectral information, while spectral cameras based on material dyeing and other spectral methods need to be calculated to obtain a single-channel spectral image. Snapshot spectral cameras obtain spectral information of the target to be measured through a single exposure. The spectral information obtained in each pixel is distributed according to a fixed period. Therefore, the spectral information in this image is mosaic.

[0004] The spectral camera can obtain the multispectral information of the target from a single two-dimensional image with one exposure, but it loses the spatial information of the target. The high-resolution multispectral image obtained is Figure 2 As shown in the figure, the spectral and spatial information of each spectrum segment of the high-resolution multispectral image obtained at this time is incomplete, and it is a mosaic image. In order to obtain a high-resolution multispectral image, it is necessary to demosaic and reconstruct the obtained multispectral image. Figure 2 As shown, in order to obtain spectral information and image information at the same time, the original multispectral image needs to be divided into multiple single channel components (when the channel period is 3×3, the number of divisions is 9). Each single channel image is a downsampling result of the original multispectral image. For the single channel component image of the original multispectral image, only one pixel value in a single period is the true value for the channel, and there are too few valid pixels. If the method based on the Bayer filter array 1 is used to interpolate and reconstruct the sparse image, the mosaic effect of the obtained image is serious, and the edge processing effect of the object in the image is poor.

[0005] At present, the methods for de-mosaicing multispectral images obtained by spectral cameras are all implemented at the software algorithm level, which has the following problems: the super-resolution interpolation algorithm based on the Bayer filter array cannot be directly applied to spectral cameras. Among them, the principle of obtaining color images based on the Bayer filter array is as follows: Figure 1 As shown, at the position of 2×2 pixels on the imaging chip space, three filter units, red, green, and blue, are set, and the three filter units transmit red, green, and blue light respectively, and the position of 2×2 pixels is used as a filter unit, namely, RGB unit 2. The Bayer filter array 1 is composed of multiple RGB units 2. When imaging, it is possible to obtain the red, green, and blue images of the material in one image at the same time, and then split the images of the red, green, and blue bands to obtain mosaic images of their respective bands. The sampling rate of the red band is 25%, the sampling rate of the green band is 50%, and the sampling rate of the blue band is 25%. The color image is processed using a simple Bayer interpolation algorithm to achieve de-mosaic reconstruction of the image. Spectral cameras all obtain two-dimensional mosaic multi-spectral images through single exposure (taking the filter period 3×3 as an example), as shown Figure 3 As shown in the figure, in a single filtering period of the spectral camera, only one pixel has a true value for the channel, and the actual sampling rate is 11% (if the filtering period is 4×4, the actual sampling rate is 6.25%, and when the filtering period is 5×5, the actual sampling rate is 4%). That is, the fewer the effective pixels, the greater the requirements for the reconstruction algorithm. At the same time, the zipper effect will inevitably occur when spectrally reconstructing the edge of the image, resulting in blurred image, and complex texture and high-frequency information such as spectrum are difficult to recover. Summary of the invention

[0006] In view of this, the invention aims to provide a multispectral image demosaicing method based on a galvanometer to solve the problem that the existing technology will inevitably produce a zipper effect when reconstructing the spectrum of the image edge, resulting in blurred images, and complex textures and high-frequency information such as spectra are difficult to restore. To achieve the above purpose, the technical solution created by the invention is implemented as follows:

[0007] A multispectral image demosaicing method based on a galvanometer comprises the following steps:

[0008] S1: Build a demosaicing device, which includes a lens, a galvanometer module and a spectral camera arranged in sequence according to the direction of the light path;

[0009] S2: Aim the lens at the target to be measured, adjust the galvanometer module, so that the target to be measured is imaged at least once on the spectral camera, and obtain a multispectral mosaic image corresponding to the number of imaging times;

[0010] S3: According to the quantitative relationship between the number of imaging times in step S2 and the number of spectral channels in a single filtering cycle of the spectral camera, demosaicing of the multispectral mosaic image is implemented based on different demosaicing methods.

[0011] Further, in step S1, the galvanometer module includes a reflector and a two-dimensional motor, and the reflector is connected to the two-dimensional motor;

[0012] Taking the geometric center of the reflector as the origin, a two-dimensional rectangular coordinate system is established on the reflective surface of the reflector;

[0013] The two-dimensional motor controls the reflecting mirror to rotate along the X-axis of the two-dimensional rectangular coordinate system or along the Y-axis of the two-dimensional rectangular coordinate system.

[0014] Furthermore, the filtering period of the spectral camera is n×n; each filtering period includes m×m filtering channels; and the number of filtering periods occupied by the mapping area of ​​the target to be measured on the spectral camera is (n-1)×(n-1).

[0015] Furthermore, in step S2, at least one imaging corresponds to different pixels of the imaging target surface of the spectral camera.

[0016] Furthermore, in step S3, the number of imaging times in step S2 is compared with the number of spectral channels in a single filtering cycle of the spectral camera:

[0017] If the number of imaging times in step S2 is the same as the number of spectral channels in a single filtering cycle of the spectral camera, the first demosaicing method is used to perform spectral splitting and merging on all multispectral mosaic images, so as to achieve demosaicing of the multispectral mosaic images and obtain a multispectral image of the target to be measured;

[0018] If the number of imaging times in step S2 is different from the number of spectral channels in a single filtering cycle of the spectral camera, and the number of imaging times in step S2 is greater than or equal to one quarter of the number of spectral channels in a single filtering cycle of the spectral camera, the second demosaicing method is used to perform spectral splitting and merging on all multispectral mosaic images to achieve demosaicing of the multispectral mosaic images, and obtain a multispectral image of the target to be measured;

[0019] If the number of imaging times in step S2 is greater than one and less than one quarter of the number of spectral channels in a single filtering cycle of the spectral camera, all multispectral mosaic images are spectrally split and merged to obtain a sparse multispectral image, and the third demosaicing method is used to demosaic the sparse multispectral image to obtain a multispectral image of the target to be measured.

[0020] Furthermore, the steps of the first demosaicing method specifically include:

[0021] All multispectral mosaic images were split into single-spectral images with a single filtering channel;

[0022] The single-spectral images with the same filtering channel information are spectrally merged to achieve demosaicing of the multispectral mosaic image and obtain the multispectral image of the target to be measured.

[0023] Furthermore, the steps of the second demosaicing method specifically include:

[0024] All multispectral mosaic images were split into single-spectral images with a single filtering channel;

[0025] The single spectrum images with the same filtering channel information are spectrally merged to obtain a sparse single spectrum image;

[0026] Constructing a pixel position matrix based on a set of pixel positions of missing spectral information of each sparse single-spectrum image;

[0027] Perform edge expansion on each sparse single-spectrum image, and the number of expanded rows and columns is (m-1);

[0028] The size of the interpolation operator is set to 2m-1, and the interpolation operator is used to perform weighted summation on the grayscale value of each pixel located at the center of the interpolation operator in each sparse single spectrum image, so as to realize image reconstruction of each sparse single spectrum image, and obtain a corresponding reconstructed single spectrum image;

[0029] After restoring the size of each reconstructed single spectrum image, the reconstructed spectral information is extracted from each reconstructed single spectrum image using the pixel position matrix. The pixel positions where the reconstructed spectral information of each reconstructed single spectrum image is located correspond one to one with the pixel positions of the missing spectral information of each sparse single spectrum image. The extracted spectral information is used to fill each sparse single spectrum image to realize the demosaicing of the multispectral mosaic image and obtain the multispectral image of the target to be measured.

[0030] Furthermore, the steps of the third demosaicing method specifically include:

[0031] All multispectral mosaic images were split into single-spectral images with a single filtering channel;

[0032] The single spectrum images with the same filtering channel information are spectrally merged to obtain a sparse single spectrum image;

[0033] Constructing a pixel position matrix based on a set of pixel positions of missing spectral information of each sparse single-spectrum image;

[0034] Perform edge expansion on each sparse single-spectrum image, and the number of expanded rows and columns is (m-1);

[0035] The size of the interpolation operator is set to 2m-1, and the interpolation operator is used to perform weighted summation on the grayscale value of each pixel located at the center of the interpolation operator in each sparse single spectrum image, so as to realize image reconstruction of each sparse single spectrum image, and obtain a corresponding reconstructed single spectrum image;

[0036] After restoring the size of each reconstructed single spectrum image, the reconstructed spectrum information is extracted from each reconstructed single spectrum image using the pixel position matrix, the pixel position where the reconstructed spectrum information of each reconstructed single spectrum image is located corresponds to the pixel position of the missing spectrum information of each sparse single spectrum image one by one, and each sparse single spectrum image is filled with the extracted spectrum information to obtain an initial demosaiced image accordingly;

[0037] A reconstruction module is constructed, and all the initial demosaiced images are sequentially input into the reconstruction module for demosaicing operation to obtain a multispectral image of the target to be measured.

[0038] Further, the reconstruction module includes N cascaded image reconstruction submodules, and N=2m-1;

[0039] The image reconstruction submodule includes a first depthwise separable convolution module, a second depthwise separable convolution module and a third depthwise separable convolution module. The initial demosaiced image is input into the image reconstruction module. The initial demosaiced image is processed by the first depthwise separable convolution module and the second depthwise separable convolution module in sequence to obtain a feature map A1. The initial demosaiced image is input into the third depthwise separable convolution module for processing to obtain a feature map A2. The feature maps A1 and A2 are added to obtain an output feature map B1 of the image reconstruction submodule.

[0040] The output feature map of the current image reconstruction submodule is used as the input feature of the next image reconstruction submodule, and the above operation is repeated until the output feature map B of the Nth image reconstruction submodule is obtained. N , the feature map B N The sparse multispectral image corresponding to the current initial demosaiced image is added to complete the demosaicing operation of the multispectral image and obtain the multispectral image of the target to be measured.

[0041] Compared with the prior art, the invention can achieve the following beneficial effects:

[0042] The present invention creates a multispectral image de-mosaicing method based on a galvanometer, which is applicable to various types of snapshot spectral cameras. In order to solve the problem of low image sampling rate output by snapshot spectral cameras, the present invention uses a controllable precision reflection component (galvanometer module) to periodically map the image of the same target to different spectral period positions on the detector, so that each spectral channel can obtain more spatial position information, increase the sampling rate of real data, and reduce the loss of pattern texture information. In an ideal case, a high-resolution multispectral image can be directly obtained without using an algorithm; when conditions do not support it, the number of sampling times can be reduced, and at a higher sampling rate, the difficulty of the interpolation super-resolution algorithm can be reduced to obtain a high-resolution multispectral image; if the requirement is to quickly obtain a high-resolution multispectral image of the target to be measured, in an extreme case, a single-exposure image is input into the designed de-mosaicing algorithm based on deep learning to obtain a high-resolution multispectral image of the target to be measured. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:

[0044] Figure 1 A schematic diagram of obtaining a color image based on a Bayer filter array according to the present invention;

[0045] Figure 2 It is a schematic diagram of the imaging technology based on multi-spectral filter array of the present invention;

[0046] Figure 3 It is the first channel image of single imaging based on multi-spectral filter array of the present invention;

[0047] Figure 4 A schematic diagram of the process of the multi-spectral image demosaicing method based on a galvanometer according to an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of the structure of the galvanometer module according to an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of two degrees of freedom of the reflector described in the embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the change in direction of reflected light after the state of the reflector is changed according to an embodiment of the present invention;

[0051] Figure 8 A schematic diagram of obtaining different spectra at a certain point by changing the state of the galvanometer module as described in an embodiment of the present invention;

[0052] Fig. 9 A schematic diagram of mapping an image A' on a filter channel in sequence in a row direction according to an embodiment of the present invention;

[0053] Fig.10 A schematic diagram of mapping an image A' on a filter channel in sequence in a column direction according to an embodiment of the present invention;

[0054] Fig.11 A schematic diagram of mapping the image A' on the filter channel in a clockwise or counterclockwise direction in sequence according to an embodiment of the present invention;

[0055] Fig.12 A schematic diagram of the corresponding relationship between point A and surrounding points and the filtering channel according to an embodiment of the present invention;

[0056] Fig.13 A schematic diagram of spectral data output by a snapshot spectral camera according to an embodiment of the present invention after a single exposure;

[0057] Fig.14 The mosaic multi-spectral image of the target to be measured is obtained by changing the state of the galvanometer module 9 times as described in the embodiment of the present invention;

[0058] Fig.15 A schematic diagram of spectrum splitting and merging of mosaic multi-spectral images according to an embodiment of the present invention;

[0059] Fig.16 A schematic diagram of splitting and merging spectral information of the first filtering channel described in an embodiment of the present invention;

[0060] Fig.17 A schematic diagram of completing the spectral splitting and merging of 9 mosaic multi-spectral images according to an embodiment of the present invention;

[0061] Fig.18 The present invention creates an embodiment of obtaining 9 groups of mosaic multi-spectral images of the target to be measured by changing the state of the galvanometer;

[0062] Fig.19 The multispectral image obtained by changing the galvanometer state five times as described in the embodiment of the present invention;

[0063] Fig. 20 A schematic diagram of an interpolation operator according to an embodiment of the present invention;

[0064] Fig.21 A schematic diagram of reconstructing an image edge using an interpolation operator according to an embodiment of the present invention;

[0065] Fig. 22 A schematic diagram of image edge expansion according to an embodiment of the present invention;

[0066] Fig.23 A schematic diagram of cutting out a target image from a reconstructed image according to an embodiment of the present invention;

[0067] Fig.24 A schematic diagram of the minimum sampling when the filtering period is 3×3 according to an embodiment of the present invention;

[0068] Fig.25 A schematic diagram of the structure of a deep learning-based de-mosaicing algorithm described in an embodiment of the present invention.

[0069] Description of reference numerals:

[0070] 1. Bayer filter array; 2. RGB unit; 3. Color sensor chip; 4. Snapshot spectral camera imaging chip; 5. Lens; 6. Vibration mirror module; 6-1. Two-dimensional motor; 6-2. Reflector; 7. Spectral camera; 8. First state; 9. Second state; 10. Incident light; 11. Reflected light. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0072] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0073] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0074] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0075] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0076] like Figure 4 As shown, the present invention proposes a multispectral image de-mosaicing method based on a galvanometer, which specifically includes the following steps:

[0077] S1: Build a de-mosaicing device, which includes a lens 5, a galvanometer module 6 and a spectral camera 7 which are sequentially arranged along the direction of the light path.

[0078] S2: Aim the lens 5 at the target to be measured, and adjust the galvanometer module 6 so that the target to be measured is imaged on the spectral camera 7 at least once, and obtain a multispectral mosaic image corresponding to the number of imaging times.

[0079] In step S2 , at least one imaging corresponds to different pixels of the imaging target surface of the spectral camera 7 .

[0080] S3: According to the quantitative relationship between the number of imaging times in step S2 and the number of spectral channels in a single filtering cycle of the spectral camera 7, demosaicing of the multispectral mosaic image is implemented based on different demosaicing methods.

[0081] In step S3, the number of imaging times in step S2 is compared with the number of spectral channels in a single filtering cycle of the spectral camera:

[0082] If the number of imaging times in step S2 is the same as the number of spectral channels of light in a single filtering cycle of the spectral camera 7 (i.e., m×m), the first demosaicing method is used to perform spectral splitting and merging on all multispectral mosaic images to achieve demosaicing of the multispectral mosaic images and obtain a multispectral image of the target to be measured (a collection of all merged single-spectral images).

[0083] The steps of the first demosaicing method specifically include:

[0084] All multispectral mosaic images were split into single-spectral images with a single filtering channel;

[0085] The single-spectral images with the same filtering channel information are spectrally merged to achieve demosaicing of the multispectral mosaic image and obtain the multispectral image of the target to be measured.

[0086] If the number of imaging times in step S2 is different from the number of spectral channels of light in a single filtering cycle of the spectral camera 7, and the number of imaging times in step S2 is greater than or equal to one quarter of the number of spectral channels in a single filtering cycle of the spectral camera 7, the second demosaicing method is used to perform spectral splitting and merging on all multispectral mosaic images to achieve demosaicing of the multispectral mosaic images and obtain a multispectral image of the target to be measured.

[0087] The steps of the second demosaicing method specifically include:

[0088] All multispectral mosaic images were split into single-spectral images with a single filtering channel;

[0089] The single spectrum images with the same filtering channel information are spectrally merged to obtain a sparse single spectrum image;

[0090] Constructing a pixel position matrix based on a set of pixel positions of missing spectral information of each sparse single-spectrum image;

[0091] Perform edge expansion on each sparse single-spectrum image, and the number of expanded rows and columns is (m-1);

[0092] The size of the interpolation operator is set to 2m-1, and the interpolation operator is used to perform weighted summation on the grayscale value of each pixel located at the center of the interpolation operator in each sparse single spectrum image, so as to realize image reconstruction of each sparse single spectrum image, and obtain a corresponding reconstructed single spectrum image;

[0093] After restoring the size of each reconstructed single spectrum image, the reconstructed spectral information is extracted from each reconstructed single spectrum image using the pixel position matrix. The pixel positions where the reconstructed spectral information of each reconstructed single spectrum image is located correspond one to one with the pixel positions of the missing spectral information of each sparse single spectrum image. The extracted spectral information is used to fill each sparse single spectrum image to realize the demosaicing of the multispectral mosaic image and obtain the multispectral image of the target to be measured.

[0094] If the number of imaging times in step S2 is greater than one and less than one quarter of the number of spectral channels in a single filtering cycle of the spectral camera, all multispectral mosaic images are spectrally split and merged to obtain a sparse multispectral image, and the third demosaicing method is used to demosaic the sparse multispectral image to obtain a multispectral image of the target to be measured.

[0095] The steps of the third demosaicing method specifically include:

[0096] All multispectral mosaic images were split into single-spectral images with a single filtering channel;

[0097] The single spectrum images with the same filtering channel information are spectrally merged to obtain a sparse single spectrum image;

[0098] Constructing a pixel position matrix based on a set of pixel positions of missing spectral information of each sparse single-spectrum image;

[0099] Perform edge expansion on each sparse single-spectrum image, and the number of expanded rows and columns is (m-1);

[0100] The size of the interpolation operator is set to 2m-1, and the interpolation operator is used to perform weighted summation on the grayscale value of each pixel located at the center of the interpolation operator in each sparse single spectrum image, so as to realize image reconstruction of each sparse single spectrum image, and obtain a corresponding reconstructed single spectrum image;

[0101] After restoring the size of each reconstructed single spectrum image, the reconstructed spectrum information is extracted from each reconstructed single spectrum image using the pixel position matrix, the pixel position where the reconstructed spectrum information of each reconstructed single spectrum image is located corresponds to the pixel position of the missing spectrum information of each sparse single spectrum image one by one, and each sparse single spectrum image is filled with the extracted spectrum information to obtain an initial demosaiced image accordingly;

[0102] A reconstruction module is constructed, and all the initial demosaiced images are sequentially input into the reconstruction module for demosaicing operation to obtain a multispectral image of the target to be measured.

[0103] The reconstruction module includes N cascaded image reconstruction submodules, where N=2m-1;

[0104] The image reconstruction submodule includes a first depthwise separable convolution module, a second depthwise separable convolution module and a third depthwise separable convolution module. The initial demosaiced image is input into the image reconstruction module. The initial demosaiced image is processed by the first depthwise separable convolution module and the second depthwise separable convolution module in sequence to obtain a feature map A1. The initial demosaiced image is input into the third depthwise separable convolution module for processing to obtain a feature map A2. The feature maps A1 and A2 are added to obtain an output feature map B1 of the image reconstruction submodule.

[0105] The output feature map of the current image reconstruction submodule is used as the input feature of the next image reconstruction submodule, and the above operation is repeated until the output feature map B of the Nth image reconstruction submodule is obtained. N , the feature map B N The sparse multispectral image corresponding to the current initial demosaiced image is added to complete the demosaicing operation of the multispectral image and obtain the multispectral image of the target to be measured.

[0106] The de-mosaicing device provided by the present invention is as follows Figure 5 As shown, it includes a lens 5, a galvanometer module 6 and a spectral camera 7 which are arranged in sequence according to the direction of the optical path, wherein the lens 5 is used to form a clear image of the scene to be photographed on the focal plane. The galvanometer module 6 is used to image the target to be measured in different areas of the spectral camera 7, and the spectral camera 7 is used to image the target to be measured.

[0107] The de-mosaicing device further includes a spectral data processing module, which is connected to the spectral camera 7 and converts the mosaic multi-spectral image into a multi-spectral image.

[0108] The galvanometer module 6 includes a reflector 6-2 and a two-dimensional motor 6-1, and the reflector 6-2 is connected to the two-dimensional motor 6-1, that is, the galvanometer module 6 is equivalent to a reflector 6-2 with controllable posture. The function of the reflector 6-2 is to change the propagation direction of the light, and the function of the two-dimensional motor 6-1 is to control the posture of the reflector 6-2. Figure 6 As shown, a two-dimensional rectangular coordinate system is established on the reflection surface of the reflector 6-2 with the geometric center of the reflector 6-2 as the origin; the two-dimensional motor 6-1 controls the reflector 6-2 to rotate along the X-axis of the two-dimensional rectangular coordinate system or along the Y-axis of the two-dimensional rectangular coordinate system.

[0109] Here, the X-axis and the Y-axis are parallel to the adjacent sides of the reflector 6-2, respectively. The reflector 6-2 has two degrees of freedom of rotation around the X-axis and the Y-axis. The two-dimensional motor 6-1 changes the state of the reflector 6-2 by changing the slight rotation of the reflector 6-2 in the directions of the two degrees of freedom, such as Figure 7 As shown, when the state of the incident light 10 remains unchanged, if the reflector 6-2 changes from the first state 8 to the second state 9, the angle of the reflected light 11 will change.

[0110] The galvanometer module 6 is installed between the lens 5 and the spectral camera 7. At the same time, the installation position of the spectral camera 7 is changed, for example, from being installed directly behind the lens 5 to being installed on the side of the lens 5 (it can be any one of the left side, right side, upper side or lower side, and the specific installation position is determined by the state of the galvanometer module 6). When the galvanometer is in the initial position, the relative relationship between the lens 5 and the spectral camera 7 can be restored, and the images obtained by the spectral camera 7 in the two states are in a mirror relationship when the galvanometer module 6 is added and when the galvanometer module 6 is not added. By controlling the movement of the two-dimensional motor 6-1, the reflector 6-2 is slightly rotated, and the pixel position of the image at a certain position in the space projected onto the spectral camera 7 will change compared to before the rotating reflector 6-2. Therefore, the posture of the reflector 6-2 can be changed by adjusting the two-dimensional motor 6-1. Finally, the image of the object will be imaged in different areas of the spectral camera 7.

[0111] The state of the reflector 6-2 is quickly changed by the two-dimensional motor 6-1, so that the image of the target to be measured changes within the filtering period of the spectral camera 7, that is, the image of the same point of the target to be measured in space is mapped on different filtering channels of the spectral camera 7, thereby realizing the acquisition of multi-spectral information of the point. According to the number of acquired images, different methods can be selected to obtain the multi-spectral image of the target to be measured. If the number of acquired images is equal to the number of channels in a single filtering period of the spectral camera 7, the image can be directly physically super-resolved to obtain a real multi-spectral image of the target; if the number of acquired images is less than the number of channels in a single filtering period of the spectral camera 7, but the sampling rate of a single band is greater than or equal to 25% of the minimum sampling rate of the color camera, the image can be demosaiced and reconstructed according to the designed interpolation algorithm to obtain a multi-spectral image of the target to be measured; if there are strict requirements for the imaging speed, when the sampling rate of a single band is less than 25% of the minimum sampling rate of the color camera or only one sampling is performed, an algorithm can be designed according to the low sampling rate to demosaic and reconstruct the image with the low sampling rate to obtain a multi-spectral image with high spatial resolution estimated by the algorithm.

[0112] like Figure 8 As shown, assuming that the filter period of the spectral camera 7 is 3×3, the light emitted from a certain point of the target to be measured in the space passes through the lens 5 and the galvanometer module 6 in sequence, and its image is mapped to the position of the first filter channel in a certain filter period of the spectral camera 7. At the same time, the spectral camera 7 is controlled to be exposed and the image information is collected, that is, the imaging area of ​​the spectral camera 7 corresponding to the position of the first filter channel in the current filter period obtains the spectral information of the point at the wavelength corresponding to the first band. Similarly, for a certain point A in the space, when the state of its incident light 10 remains unchanged, by changing the state of the galvanometer module 6, its image A' is respectively mapped to the 9 filter channels of the same filter period. After the spectral camera 7 is exposed accordingly, the 9 spectrum band information of the point is obtained. For the spectral camera 7 with a filter period of 4×4 or 5×5, it is necessary to adjust the state of the galvanometer module 6 so that its image A' is correspondingly mapped to 16 or 25 filter channels of the same filter period, and the spectral camera 7 is controlled to be exposed at the same time to obtain the 16 spectrum band information or 25 spectrum band information of point A.

[0113] The method for obtaining the multi-spectral information of point A is: using the galvanometer module 6 to map its image A' to all the filter channels of a certain filter period of the spectral camera 7, the multi-spectral information can be obtained, which is independent of the order of mapping to each filter channel. Therefore, the mapping order can be arbitrarily selected to complete the acquisition of the multi-spectral information of point A. In order to obtain the multi-spectral information of point A more quickly and accurately, the mapping order needs to be determined.

[0114] Take the spectral camera 7 with a filtering period of 3×3 as an example.

[0115] The first method is to map all filter channels in the filter period in sequence in the row direction or column direction.

[0116] Assume that the mapping order is from the 1st filter channel to the 9th filter channel. First, the method of mapping in the row direction is introduced, such as Fig. 9 As shown, the arrows in the figure indicate that by changing the state of the galvanometer module 6, the image A' of the point A is mapped in sequence in the row direction of a certain filtering period.

[0117] First, the image A' of point A is mapped to the position of the first filter channel in the current filter cycle. At this time, the spectral camera 7 is controlled to expose, and the exposure time is t1 to obtain the spectral information of point A in the first filter channel; then, by adjusting the two-dimensional motor 6-1 to change the state of the reflector 6-2, the image A' of point A is moved from the position of the first filter channel to the position of the second filter channel, and the moving time t2 is recorded, and then the spectral camera 7 is controlled to expose. At this time, the moving time t2 is also the waiting time required by the spectral camera 7 between two adjacent exposure times. The spectral camera 7 outputs the spectral information of point A in the second filter channel; the image A' of point A is moved from the second filter channel to the position of the third filter channel, and the above steps are performed. In the first three steps, the spectral camera 7 needs to wait for the same time between each two adjacent exposures, which is t2. At this time. The acquisition of the spectral information corresponding to the first row of filter channels of point A in the current filter cycle is completed.

[0118] Adjust the two-dimensional motor 6-1, change the state of the reflector 6-2, and move the image A' of point A from the third filter channel to the fourth filter channel. Compared with the movement between the first filter channel and the third filter channel, the required movement time is longer. The movement time at this time is recorded as t3. At this time, t3>t2, and the spectral camera 7 is controlled to expose. The movement time at this time is the time that the spectral camera 7 needs to wait. Next, according to the above operation, the image A' of point A is sequentially mapped in the second row of the filter channel of the current filter cycle, and the spectral information corresponding to the second row of the filter channel of point A in the current filter cycle is obtained. Similarly, the spectral information corresponding to the third row of the filter channel of point A in the current filter cycle is obtained. At this time, the spectral information of the 9 spectrum segments of point A is obtained.

[0119] like Fig.10As shown, the principle of mapping in the column direction to obtain the multi-spectral information of point A is the same as that of mapping in the row direction to obtain the multi-spectral information of point A. Theoretically, the time required to obtain the multi-spectral information of point A is T1=9t1+6t2+2t3. However, taking the row direction as an example, when the image A' of point A is moved from the 3rd filter channel to the 4th filter channel, or from the 6th filter channel to the 7th filter channel, the angle of rotation of the reflector 6-2 controlled by the two-dimensional motor 6-1 is large, which may cause the image A' of point A to be unable to be correctly and completely mapped to the spatial position of the 4th filter channel or the 7th filter channel, and there is a risk of inaccurate spectral information being obtained.

[0120] The second mapping method is to use the center of the filtering period as the origin, and map the image A' of point A to each filtering channel within a filtering period in a clockwise or counterclockwise manner, such as Fig.11 shown.

[0121] Taking counterclockwise as an example, first, the image A' of point A is mapped to the position of the 5th filter channel of the current filter cycle, and the spectral camera 7 is controlled to be exposed, and the exposure time is t1. The spectral camera 7 outputs the spectral information of point A in the 5th filter channel; then, by adjusting the state of the reflector 6-2 by adjusting the two-dimensional motor 6-1, the image A' of point A is moved from the 5th filter channel to the 4th filter channel, and the moving time is recorded as t2, and the spectral camera 7 is controlled to be exposed. At this time, the moving time t2 is also the waiting time required between two adjacent exposure times of the spectral camera 7, and the spectral camera 7 outputs the spectral information of point A in the 2nd filter channel; repeat the above steps until the spectral information of the 9 spectrum segments of point A is obtained. In theory, the time required to obtain the multi-spectral information of point A using the second mapping method is T3=9t1+8t2.

[0122] Comparing the second mapping method with the first mapping method, the total time difference between the two methods is δT=T1-T2=2(t3-t2). Since t3>t2, δT>0, the first method takes longer time. At the same time, in the second mapping method, the image A' of point A moves the same distance in two consecutive times, making the two-dimensional motor 6-1 more stable when controlling the mirror 6-2 to rotate with two degrees of freedom. Therefore, the second mapping method is better than the first mapping method. Fig.12As shown, if point A obtains the spectral information of the first filter channel of a certain filter period, then the spectral information of the filter channels within the 8 neighborhoods centered on the first filter channel will be obtained. Similarly, for a point B within the neighborhood of point A, when it obtains the spectral information of a certain filter channel, other points within the neighborhood of point B will obtain the spectral information corresponding to the filter channel within the neighborhood centered on the current filter channel. By analogy, the spectral information of all points of the target C to be measured can be obtained with a single exposure of the spectral camera 7. However, at this time, the spectral information of all points of the target C to be measured is not the same, but presents the same periodic spectral distribution as the filter array of the spectral camera 7 (assuming that the image of target C is mapped to the snapshot spectral camera in size of 6×6, which is the same size as the filter array composed of 2×2 filter periods of the spectral camera 7 in physical space), as shown Fig.13 As shown, each point on the target C only obtains the spectral information of a certain filter channel in the filter period. At this time, the image output by the spectral camera 7 is a mosaic multi-spectral image, and the spectral information of each pixel in the image strictly corresponds to the spectral information corresponding to the filter channel.

[0123] like Fig.14 As shown, if you want to obtain the spectral information corresponding to all filtering channels of all points on target C within a single filtering cycle, that is, to obtain the multi-spectral information of target C, you need to map the image A' of a point A on target C according to the second mapping method proposed above, and change the state of the reflector 6-2 through the two-dimensional motor 6-1 to obtain the multi-spectral information of point A. At this time, the points within the neighborhood of point A also obtain multi-spectral information. Similarly, all points on target C obtain multi-spectral information, and then perform spectral splitting and merging on the obtained multiple mosaic multi-spectral images, so that each image is a spectral image of the same spectral segment of target C.

[0124] like Fig.15 As shown, taking a mosaic multispectral image as an example, before performing spectral splitting and merging, it is necessary to generate multiple blank images of the same size as the mosaic multispectral image, and the number of blank images is the same as the number of filter channels contained in a single filter period of the spectral camera 7, that is, when the single filter period of the spectral camera 7 is 3×3, 4×4 or 5×5, the number of blank images that need to be generated is 9, 16 or 25, respectively, which are respectively called the 1st spectral segment, the 2nd spectral segment, ..., the 25th spectral segment; then, the spectral information of the pixel points corresponding to a certain filter channel in all filter periods of the mosaic multispectral image is taken out and stored in the corresponding position of the corresponding spectral segment. Taking the first filter channel as an example, in the mosaic multispectral image, the spectral information of the pixel points corresponding to the first filter channel of all filter periods is taken out, and the position information of each pixel point is recorded. According to the position information of each pixel point, the corresponding spectral information is stored in the first spectral segment, as shown in FIG. Fig.16As shown, the splitting and merging of the spectral information corresponding to the first filter channel of all filter cycles in the current image are completed, and then the splitting and merging of the spectral information corresponding to the second filter channel to the ninth filter channel are performed in sequence according to the above steps to obtain a multispectral image of the target to be measured.

[0125] The following is a detailed introduction to the three demosaicing methods:

[0126] The first de-mosaicing method: when the number of times the state of the galvanometer module 6 is changed is equal to the number of all filter channels in a single filter period of the spectral camera 7, that is, the galvanometer needs to be used to directly obtain a high-resolution multispectral image of the target to be measured, taking the spectral camera 7 with a filter period of 3×3 as an example, first, adjust the reflector 6-2 and the lens 5 so that the image of the target to be measured is completely mapped on the spectral camera 7; then, by changing the state of the galvanometer module 6, 9 mosaic multispectral images of the target to be measured are obtained; finally, the mosaic multispectral image is spectrally split and merged, as shown in FIG. Fig.17 Based on this, the acquisition of multispectral images is realized, and high-resolution multispectral images of the target to be measured can be obtained without using any algorithm, such as Fig.18 As shown, the spectral data processing is to perform spectral splitting and spectral merging on the mosaic multi-spectral image output by the spectral camera 7, and output the multi-spectral image.

[0127] The second de-mosaicing method: To improve the sampling speed of multi-spectral images and ensure that the current minimum sampling rate is close to the minimum sampling rate of color images (the minimum sampling rate of color images is 25%), the speed of image acquisition by the spectral camera 7 is improved by reducing the adjustment of the galvanometer module 6. For example, the filter period of the spectral camera 7 is 3×3, and the target to be measured is acquired 5 times. Fig.19 As shown, a sparse mosaic multispectral image is obtained at this time, and the spatial information is incomplete. The present invention de-mosaic and reconstructs the multispectral image by designing an interpolation operator. The designed interpolation operator is as follows: Fig. 20 As shown, the size of the interpolation operator is related to the size of the filter period of the spectral camera 7. Specifically, the operator size k is k=2m-1, and sqrt represents a square root operation, that is, when the filter period is 3×3, 4×4 or 5×5, the operator size is 5, 7 or 9 respectively.

[0128] The design idea of ​​the interpolation operator is that in an image of a certain spectral band, the energy obtained by a certain pixel and the energy obtained by its surrounding neighboring pixels present a two-dimensional Gaussian distribution, that is, the current pixel obtains the most energy, and the neighboring pixels close to the current pixel obtain higher energy, but not higher than the energy of the current pixel. Therefore, the energy value of the current pixel can be obtained by weighted summing the energy of the neighboring pixels through the designed interpolation operator.

[0129] The design process of the interpolation operator is as follows:

[0130] This theoretical basis is obtained by weighted summing of the energy of a certain pixel by the pixels in its neighborhood. The size of the interpolation operator designed in the present invention is related to the size of the filtering period of the spectral camera 7 and is an odd number. At the same time, any row and column of the interpolation operator conforms to the one-dimensional Gaussian distribution. For ease of understanding, the positive directions of the x-coordinate (horizontally to the right of the image) and the y-coordinate (vertically upward direction of the image) of the three-dimensional rectangular coordinate system are set on the plane of the image; the means of the two dimensions x and y are both set to 0. The energy distribution of the pixels in the neighborhood of a certain pixel in the spectral camera 7 is theoretically uniform. Therefore, the standard deviations of the two coordinate dimensions (x-coordinate and y-coordinate) are both set to δ. Since the x-coordinate and the y-coordinate represent the row direction and the column direction of the image, they are completely orthogonal. Therefore, the correlation coefficient of the x-coordinate and the y-coordinate is set to 0, and the obtained two-dimensional Gaussian distribution The formula is:

[0131] ;

[0132] Among them, x and y represent the coordinate information of each weight inside the interpolation operator. Since the target pixel is the coordinate center and the coordinate changes of the x-axis and y-axis are related to the size of the interpolation operator, the coordinate changes of x and y conform to the following formula:

[0133] ;

[0134] Among them, k is the size of the interpolation operator, :1: indicates an interval of 1. For example, when the interpolation operator size is 5×5, the formula for coordinate change becomes [-2:1:2], which means that the x-coordinate or y-coordinate changes from -2 to 2 with an interval of 1, that is, the value range of the x-coordinate or y-coordinate is [-2, -1, 0, 1, 2]. At this time, an interpolation operator that conforms to the two-dimensional Gaussian distribution can be designed according to the two-dimensional Gaussian distribution formula, and the standard deviation δ is used in the range of (0, k).

[0135] When using the interpolation operator to reconstruct the image of each filter channel, in order to avoid affecting the reconstruction effect of edge pixels, such as Fig.21 As shown in the figure, when reconstructing the pixels at positions 1, 2, 3, and 4, the number of pixels in their neighborhood is not saturated, resulting in a lower calculated pixel value. To solve the above situation, the edge of the current spectrum image is expanded before reconstruction. The number of rows (columns) h for each edge expansion is h=sqrt(n)-1, where n is the number of channels in the current filter period, and the value of n is 9, 16, or 25. Sqrt represents the square root operation, that is, when the filter period of the spectral camera 7 is 3×3, 4×4, or 5×5, the number of rows (columns) that need to be expanded for each edge is 2, 3, or 4, respectively. Fig. 22As shown in the figure, taking the leftmost and topmost sides of the mosaic image with a filtering period of 3×3 as an example, the pixel values ​​in the red frame are expanded from the pixel values ​​in the green frame. Assuming that the size of the mosaic multispectral image before expansion is w×w, the size after edge expansion is (w+2)×(w+2). After demosaicing and reconstructing the image using the interpolation operator, the size of the image needs to be restored to its original size. The restoration method is as follows: Fig.23 As shown, each edge of the expanded image is reduced by two sizes to obtain an interpolated image. Each interpolated image is demosaiced using an interpolation operator to obtain a multispectral image of the current target. Because when the interpolation operator is used to demosaic the image, the condition that the minimum sampling rate is greater than or equal to 25% of the minimum sampling rate of the color image must be met. Therefore, when the filter period is 3×3, 4×4 or 5×5, the corresponding minimum sampling times of the three spectral cameras 7 are 2, 4 or 6 times. At the same time, when each spectral camera 7 performs the minimum sampling operation, the sampling dispersion principle should be implemented, that is, within the same filtering period, two adjacent samplings should ensure that the filtering channels of the two samples are not adjacent and are not in the same row or column. Taking the filtering period of 3×3 as an example, Fig.24 As shown, if the first sampling position is at the red frame position in the figure, the second sampling must be at any position of other color frames.

[0136] In the third demosaicing method, in order to quickly obtain a high-resolution multispectral image of the target to be measured, the number of exposures of the camera should be minimized, and the number of times the galvanometer module 6 is used to obtain a mosaic multispectral image should be reduced. That is, if the reconstruction rate of the multispectral image demosaicing is required to be fast, in the extreme case, the number of sampling times for any snapshot multispectral camera 7 is once, and only one value is true in each filtering period. When the filtering period of the snapshot multispectral camera 7 is 3×3, 4×4 or 5×5, the actual sampling rate is 11%, 6.25% or 4%. The method of demosaicing the obtained sparse multispectral image based on the interpolation operator will no longer be applicable. Therefore, the present invention designs a demosaicing algorithm based on deep learning to demosaic and reconstruct the obtained sparse multispectral image.

[0137] The de-mosaicing algorithm based on deep learning designed by the present invention has a structure diagram as shown in FIG. Fig.25As shown, taking the spectral camera 7 with a filtering period of 3×3 as an example, first, an interpolation operator is used to perform an initial demosaicing operation on the sparse multispectral image to obtain an initial demosaiced multispectral image; then an image reconstruction module composed of a depthwise separable convolution is designed to reconstruct the initial demosaiced multispectral image. N in the figure represents the number of repetitions of the image reconstruction module. When the filtering period of the snapshot multispectral camera 7 is 3×3, 4×4 or 5×5, N is 5, 7 or 9 respectively; finally, the image output by the image reconstruction module is added to the sparse multispectral image to obtain a multispectral image of the target to be measured.

[0138] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0139] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multispectral image demosaicing method based on a galvanometer, characterized in that: The specific steps include: S1: Building a de-mosaicing device, wherein the de-mosaicing device includes a lens, a galvanometer module, and a spectral camera arranged in sequence according to the direction of the light path; S2: Aim the lens at the target to be measured, and adjust the galvanometer module so that the target to be measured is imaged on the spectral camera at least once, and obtain a multispectral mosaic image corresponding to the number of imaging times; S3: according to the quantitative relationship between the number of imaging times in step S2 and the number of spectral channels in a single filtering cycle of the spectral camera, a demosaicing process is implemented on the multispectral mosaic image based on different demosaicing methods; In step S3, the number of imaging times in step S2 is compared with the number of spectral channels in a single filtering cycle of the spectral camera: If the number of imaging times in step S2 is the same as the number of spectral channels in a single filtering cycle of the spectral camera, the first demosaicing method is used to perform spectral splitting and merging on all multispectral mosaic images, so as to achieve demosaicing of the multispectral mosaic images and obtain a multispectral image of the target to be measured; The steps of the first demosaicing method specifically include: All multispectral mosaic images were split into single-spectral images with a single filtering channel; The single-spectral images with the same filtering channel information are spectrally merged to achieve demosaicing of the multi-spectral mosaic image and obtain the multi-spectral image of the target to be measured; If the number of imaging times in step S2 is different from the number of spectral channels in a single filtering cycle of the spectral camera, and the number of imaging times in step S2 is greater than or equal to one quarter of the number of spectral channels in a single filtering cycle of the spectral camera, the second demosaicing method is used to perform spectral splitting and merging on all multispectral mosaic images to achieve demosaicing of the multispectral mosaic images, and obtain a multispectral image of the target to be measured; The steps of the second demosaicing method specifically include: All multispectral mosaic images were split into single-spectral images with a single filtering channel; The single spectrum images with the same filtering channel information are spectrally merged to obtain a sparse single spectrum image; Constructing a pixel position matrix based on a set of pixel positions of missing spectral information of each sparse single-spectrum image; Perform edge expansion on each sparse single-spectrum image, and the number of expanded rows and columns is (m-1); The size of the interpolation operator is set to 2m-1, and the interpolation operator is used to perform weighted summation on the grayscale value of each pixel located at the center of the interpolation operator in each sparse single spectrum image, so as to realize image reconstruction of each sparse single spectrum image, and obtain a corresponding reconstructed single spectrum image; After restoring the size of each reconstructed single spectrum image, the reconstructed spectrum information is extracted from each reconstructed single spectrum image using the pixel position matrix, the pixel position where the reconstructed spectrum information of each reconstructed single spectrum image is located corresponds one to one with the pixel position where the missing spectrum information of each sparse single spectrum image is located, and each sparse single spectrum image is filled with the extracted spectrum information to realize the demosaicing of the multispectral mosaic image, and obtain the multispectral image of the target to be measured; If the number of imaging times in step S2 is greater than one and less than one quarter of the number of spectral channels in a single filtering cycle of the spectral camera, all multispectral mosaic images are spectrally split and merged to obtain a sparse multispectral image, and the third demosaicing method is used to perform demosaicing on the sparse multispectral image to obtain a multispectral image of the target to be measured; The steps of the third demosaicing method specifically include: All multispectral mosaic images were split into single-spectral images with a single filtering channel; The single spectrum images with the same filtering channel information are spectrally merged to obtain a sparse single spectrum image; Constructing a pixel position matrix based on a set of pixel positions of missing spectral information of each sparse single-spectrum image; Perform edge expansion on each sparse single-spectrum image, and the number of expanded rows and columns is (m-1); The size of the interpolation operator is set to 2m-1, and the interpolation operator is used to perform weighted summation on the grayscale value of each pixel located at the center of the interpolation operator in each sparse single spectrum image, so as to realize image reconstruction of each sparse single spectrum image, and obtain a corresponding reconstructed single spectrum image; After restoring the size of each reconstructed single spectrum image, the reconstructed spectrum information is extracted from each reconstructed single spectrum image using the pixel position matrix, the pixel position where the reconstructed spectrum information of each reconstructed single spectrum image is located corresponds to the pixel position of the missing spectrum information of each sparse single spectrum image one by one, and each sparse single spectrum image is filled with the extracted spectrum information to obtain an initial demosaiced image accordingly; A reconstruction module is constructed, and all the initial demosaiced images are sequentially input into the reconstruction module for demosaicing operation to obtain a multispectral image of the target to be measured.

2. The multispectral image demosaicing method based on a galvanometer according to claim 1, characterized in that: In step S1, the galvanometer module includes a reflector and a two-dimensional motor, and the reflector is connected to the two-dimensional motor; Taking the geometric center of the reflector as the origin, a two-dimensional rectangular coordinate system is established on the reflective surface of the reflector; The two-dimensional motor controls the reflector to rotate along the X-axis of the two-dimensional rectangular coordinate system or along the Y-axis of the two-dimensional rectangular coordinate system.

3. The multispectral image demosaicing method based on a galvanometer according to claim 1, characterized in that: The filtering period of the spectral camera is n×n; each filtering period includes m×m filtering channels; the number of filtering periods occupied by the mapping area of ​​the target to be measured on the spectral camera is (n-1)×(n-1).

4. The multispectral image demosaicing method based on a galvanometer according to claim 1, characterized in that: In step S2, at least one imaging corresponds to different pixels of the imaging target surface of the spectral camera.

5. The multispectral image demosaicing method based on a galvanometer according to claim 1, characterized in that: The reconstruction module includes N cascaded image reconstruction submodules, where N=2m-1; The image reconstruction submodule includes a first depthwise separable convolution module, a second depthwise separable convolution module and a third depthwise separable convolution module. The initial demosaiced image is input into the image reconstruction module. The initial demosaiced image is processed by the first depthwise separable convolution module and the second depthwise separable convolution module in sequence to obtain a feature map A1. The initial demosaiced image is input into the third depthwise separable convolution module for processing to obtain a feature map A2. The feature maps A1 and A2 are added to obtain an output feature map B1 of the image reconstruction submodule. The output feature map of the current image reconstruction submodule is used as the input feature of the next image reconstruction submodule, and the above operation is repeated until the output feature map B of the Nth image reconstruction submodule is obtained. N , the feature map B N The sparse multispectral image corresponding to the current initial demosaiced image is added to complete the demosaicing operation of the multispectral image and obtain the multispectral image of the target to be measured.

Images