A high-speed multispectral camera based on a light-splitting prism and a multi-channel TDI chip

Abstract

The application discloses a high-speed multispectral camera based on a split-beam prism and a multi-channel TDI chip, and relates to the technical field of optical imaging and high-speed sensing, which comprises an imaging objective, a light-splitting module, a sensor array and a processing chip; in use, external natural light reflected by an object is projected to the light-splitting module through the imaging objective, the light-splitting module decomposes the incident light into multiple wave bands to generate multi-channel color information, the sensor array receives the multi-channel color information generated by the light-splitting module, and finally the processing chip processes and fuses the multi-channel color information received by the sensor array into a color multispectral image. The application can realize fine decomposition of a spectrum, enables the camera to obtain rich and detailed spectral information, and can be used for analyzing chemical components and physical characteristics of an object; the application can complete image acquisition of multiple spectral wave bands in a short time, and is suitable for shooting fast-moving objects or transient phenomena.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical imaging and high-speed sensing technology, and particularly relates to a high-speed multispectral camera based on a light-splitting prism and a multi-channel TDI chip. BACKGROUND

[0002] Current color cameras are mainly realized through a Bayer filter scheme, in which a Bayer filter is arranged on a sensor to obtain a complete color image through color interpolation. Due to its low cost, color cameras using a Bayer filter have become the mainstream in the current market. However, the existing TDI chip cannot realize color through a Bayer filter, that is, the existing TDI chip is only black and white and cannot realize color imaging, so how to make the traditional TDI camera realize true color, multispectral and ultra-high-speed imaging has become a problem to be solved by those skilled in the art. SUMMARY

[0003] The present application provides a high-speed multispectral camera based on a light-splitting prism and a multi-channel TDI chip, comprising: an imaging objective lens (1), a light-splitting module (2), a sensor array (3) and a processing chip; in use, external natural light reflected by an object is projected to the light-splitting module (2) through the imaging objective lens (1), the light-splitting module (2) decomposes the incident light into multiple wavebands to generate multi-channel color information, the sensor array (3) receives the multi-channel color information generated by the light-splitting module (2), and finally the processing chip processes and fuses the multi-channel color information received by the sensor array (3) into a color multispectral image.

[0004] The high-speed multispectral camera based on a light-splitting prism and a multi-channel TDI chip as described above, wherein the light-splitting module (2) is provided with multiple light-splitting prisms, the arrangement order of the light-splitting prisms depends on the wavelengths of the color lights, and the end of each color channel generated by each light-splitting prism is integrated with an independent COMS-TDI sensor, and the line frequency is synchronously locked to a microsecond level of accuracy.

[0005] The high-speed multispectral camera based on a light-splitting prism and a multi-channel TDI chip as described above, wherein the light-splitting module (2) uses the refraction and total internal reflection of the light in each light-splitting prism and the dichroic film on the surface of the prism to decompose the incident polychromatic light into monochromatic light of different wavelengths, so that the light rays of different spectral wavebands propagate in different directions, thereby realizing spectral separation; the angle at which the light rays of different wavebands are incident on the inner surface of the prism should be greater than the total reflection angle, and the optical path of each waveband light ray exiting from different channels in the prism should remain consistent.

[0006] A high-speed multispectral camera based on a light-splitting prism and a multi-channel TDI chip fusion, wherein the light-splitting module (2) includes a first light-splitting prism (21), a second light-splitting prism (22), and a third light-splitting prism (23), which are used to separate blue light, red light, and green light in incident light, respectively; and the corresponding sensor array (3) includes a first COMS-TDI sensor (31), a second COMS-TDI sensor (32), and a third COMS-TDI sensor (33), which are respectively arranged at the ends of the blue light, red light, and green light channels to form multiple sensing channels and realize sensing and integration of light signals of different spectral bands.

[0007] A high-speed multispectral camera based on a light-splitting prism and a multi-channel TDI chip fusion, wherein the light-splitting module (2) is also used to expand the incident light into RGB+NIR or RGB+DUV four channels, if the incident light is expanded into RGB+NIR, a fourth light-splitting prism (24) is added in the light-splitting module (2) to separate infrared light in the incident light, if the incident light is expanded into RGB+DUV, the added fourth light-splitting prism (24) is used to separate ultraviolet light in the incident light, and a fourth COMS-TDI sensor (34) is added to receive fourth color channel information separated by the fourth light-splitting prism (24).

[0008] The application also provides a multi-channel color information processing and fusion method, which comprises:

[0009] Step 1, the sensor array converts the received light signals into digital signals and inputs them into the processing chip in real time;

[0010] Step 2, the processing chip restores the received digital signals into multiple two-dimensional image matrices and eliminates geometric distortion caused by optical path difference;

[0011] Step 3, the multiple two-dimensional image matrices are spliced and fused by using a pre-trained spectral mapping model to obtain a true-color multispectral image.

[0012] The multi-channel color information processing and fusion method, wherein the geometric distortion caused by the optical path difference is eliminated, and the method comprises the following sub-steps:

[0013] detecting corner points in each two-dimensional image matrix;

[0014] matching the detected corner points into a feature point group;

[0015] calculating optimal correction parameters according to the matched feature point group;

[0016] eliminating the geometric distortion caused by the optical path difference based on the optimal correction parameters.

[0017] The multi-channel color information processing fusion method as described above, wherein the spectral mapping model is an end-to-end learning model of U-Net structure, multi-scale feature matrices in a two-dimensional image matrix are extracted through an encoder-decoder structure, and a skip connection is used to retain spatial details, so as to realize adaptive mapping of multi-channel spectral information; an attention mechanism enhancement module is further added before the skip connection, to dynamically adjust the importance of different channels, and improve spectral fidelity, and the method specifically includes the following sub-steps:

[0018] Compress the multi-scale feature matrix extracted by the encoder into a channel description feature;

[0019] Calculate a channel connection vector according to the channel description feature and a description feature of an expected image;

[0020] Re-calibrate the extracted multi-scale feature matrix by using the channel connection vector;

[0021] Generate a final output image by performing skip connection on the re-calibrated multi-scale feature matrix through a decoder.

[0022] The beneficial effects realized by the application are as follows:

[0023] High spectral resolution: the light splitting prism can finely split light into multiple different spectral wavelengths, and each channel of the multi-channel sensor array can correspond to a specific narrow waveband, so that the camera can obtain rich and detailed spectral information, and the traditional black-and-white TDI camera can realize color multispectral imaging.

[0024] Ultra-high-speed imaging: the multi-channel sensor array has fast signal processing capability and high frame rate, combined with the high-efficiency light splitting of the light splitting prism, the camera can complete image acquisition of multiple spectral wavebands in a short time, and is suitable for shooting fast-moving objects or transient phenomena.

[0025] High sensitivity and low noise: the integration function of the multi-channel sensor array can accumulate light signals, improve the sensitivity of the camera to weak light, and reduce the noise level, so that high-quality multispectral images can be obtained under low light conditions.

[0026] Good spatial resolution: the fusion design of the light splitting prism and the multi-channel sensor array can ensure that high spatial resolution can be maintained under different spectral channels, and the details and morphology of the object can be clearly presented. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only show some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0028] Figure 1 is a high-speed multispectral camera schematic diagram based on a split prism and a multi-channel TDI chip provided by the first embodiment of the present application;

[0029] Figure 2 is a split module and sensor array schematic diagram for RGB spectrum provided by the first embodiment of the present application;

[0030] Figure 3 is a split module working principle schematic diagram for RGB spectrum provided by the first embodiment of the present application;

[0031] Figure 4 is a split module parameter design schematic diagram for RGB spectrum provided by the first embodiment of the present application;

[0032] Figure 5 is a split module and sensor array schematic diagram for RGB+NIR or RGB+DUV spectrum provided by the first embodiment of the present application;

[0033] Figure 6 is a split module working principle schematic diagram for RGB+NIR or RGB+DUV spectrum provided by the first embodiment of the present application;

[0034] Figure 7 is a split module parameter design schematic diagram for RGB+NIR or RGB+DUV spectrum provided by the first embodiment of the present application.

[0035] Reference signs:

[0036] 1, imaging objective; 2, split module; 3, sensor array; 21, first split prism; 22, second split prism; 23, third split prism; 24, fourth split prism; 31, first COMS-TDI sensor; 32, second COMS-TDI sensor; 33, third COMS-TDI sensor; 34, fourth COMS-TDI sensor. DETAILED DESCRIPTION

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1

[0039] like Figure 1 As shown, Embodiment 1 of this application provides a high-speed multispectral camera based on the fusion of a beam splitter prism and a multi-channel TDI chip, including: an imaging objective lens 1, a beam splitter module 2, a sensor array 3, and a processing chip; in use, the external natural light reflected by the object is projected onto the beam splitter module 2 through the imaging objective lens 1. The beam splitter module 2 decomposes the incident light into multiple bands to generate multi-channel color information. The sensor array 3 receives the multi-channel color information generated by the beam splitter module 2. Finally, the processing chip processes and fuses the multi-channel color information received by the sensor array 3 into a color multispectral image.

[0040] In other words, this application utilizes multiple CMOS-TDI sensors containing monochrome TDI chips to jointly receive multiple colors of light from natural light, and then achieves color imaging of a monochrome TDI camera through the fusion processing of multiple colors of light.

[0041] In this embodiment, the beam splitting module 2 is equipped with multiple beam splitting prisms. The arrangement order of the beam splitting prisms depends on the wavelength of each color light. It is necessary to ensure that the color light with the longest or shortest wavelength can be decomposed first. Moreover, each beam splitting prism generates an independent CMOS-TDI sensor at the end of the color channel. Its line frequency synchronization is locked to the microsecond level. Therefore, the corresponding sensor array 3 also includes multiple CMOS-TDI sensors. The beam splitting module 2 uses the refraction and total internal reflection of light in each beam splitting prism, as well as the dichroic thin film on the prism surface, to decompose the incident polychromatic light into monochromatic light of different wavelengths, so that the light of different spectral bands propagates in different directions, thereby achieving spectral separation.

[0042] Figure 1 Taking three beam-splitting prisms as an example, see [link / reference]. Figure 2The beam splitting module 2 includes a first beam splitting prism 21, a second beam splitting prism 22, and a third beam splitting prism 23, which are used to separate blue light, red light, and green light in the incident light, respectively. The corresponding sensor array 3 includes a first COMS-TDI sensor 31, a second COMS-TDI sensor 32, and a third COMS-TDI sensor 33, which are deployed at the ends of the blue light, red light, and green light channels, respectively, forming multiple sensing channels to realize the sensing and integration of light signals in different spectral bands. When the camera is working, each sensing channel corresponds to a specific spectral band separated by the beam splitting prism, receives and converts the light signal into an electrical signal, and after integration processing, improves the signal strength and quality, thereby improving the camera's ability to capture weak light signals and its spectral resolution.

[0043] The specific spectral splitting principle of the RGB spectrum splitting module 2 is as follows: See Figure 3 The a-surfaces (i.e., 101a and 102a) of prisms 101 and 102 are coated with specific filter films, and the b-surfaces (i.e., 101b and 102b) are used to achieve total internal reflection. When RGB incident light enters the beam splitter 101 in the horizontal direction, it first reaches the a-surface 101a of prism 101. Under the action of the specific filter film on 101a, the blue short-wavelength light in the incident light is separated first and undergoes total internal reflection through the prism 101b surface to enter the corresponding first channel. The remaining light enters prism 102. On the a-surface (102a) of prism 102, the red long-wavelength light in the incident light is separated by the filter film and then the light path is adjusted through total internal reflection through the 102b surface to enter the second channel. Finally, the remaining green band light enters the third channel, and the beam splitting is completed. It is important to note that the angle at which light rays of different wavelengths are incident on the surface of prism b should be greater than the angle of total internal reflection. The optical path length within the prism traversed by light rays of different wavelengths exiting from different channels should be consistent or approximately consistent. For specific parameter design, please refer to [reference needed]. Figure 4, the incident angle of the central field of view light on each surface and the geometric path length of each section in the prism are plotted, the degrees of the two angles determining the reflection angle in the prism 101 are A and 2A respectively, the degrees of the two angles determining the reflection angle in the prism 102 are 2B+A and B+A respectively, the degree of the angle adjacent to the lower side of the prism in the prism 103 is 90°-B, according to the geometric calculation of the above prism parameters, the incident angles of the light on the surfaces 101a, 102a where the filter films are located are A and B respectively, the incident angle of the blue waveband light on 101b after being reflected by the filter film 101a is 2A, which should satisfy 2A>arcsin(1 / n), n is the refractive index of the prism material, the incident angle of the red waveband light on 102b after being reflected by the filter film 102a is 2B+A, which should satisfy 2B+A>arcsin(1 / n); then, given the total optical path Ltotal in the prism, taking Ltotal=(L1+L11+L12)=(L1+L2+L21+L22)=(L1+L2+L3) as the reference, combined with the requirements of the shape and structure of each prism and the above total reflection angle conditions, a reasonable analytical solution that meets the design requirements is solved, and the following is an example of a set of key parameters that meet the conditions:

[0044]

[0045] The basic structure of the COMS-TDI sensor is usually composed of 32 to 256 rows of pixels, and the number of pixels in each row can reach 16K or higher. Its working process can be described by the following mathematical model:

[0046] S(x) = Σ(i=1 to N) Pi(x)

[0047] Where S(x) is the final output signal, N is the number of TDI stages (i.e. the number of pixel rows), and Pi(x) is the light signal received by the i-th row at position x. This formula clearly shows the core idea of TDI technology: enhancing signal strength through multiple accumulation.

[0048] The working process of TDI can be described in more detail as follows: a) When light shines on the first row of pixels, initial photocharges are generated. b) In the next clock cycle, these charges are accurately transferred to the second row of pixels. c) At the same time, the first row of pixels continues to receive new photons and generate new charges. d) This process continues in subsequent rows of pixels, with each row receiving new photons and accumulating charges from the previous row. e) When the charges reach the last row of pixels, the accumulated signal is read out, forming the final image signal.

[0049] The light splitting module 2 is also used to expand the incident light into RGB+NIR (near infrared) or RGB+DUV (deep ultraviolet) four channels, as shown in Figure 5If the extension is RGB+NIR, a fourth light splitting prism 24 is added in the light splitting module 2 to separate the infrared light in the incident light. If the extension is RGB+DUV, the fourth light splitting prism 24 is added to separate the ultraviolet light in the incident light. A fourth COMS-TDI sensor 34 is added to receive the fourth color channel information separated by the fourth light splitting prism 24.

[0050] The specific light splitting principle of the light splitting module 2 for RGB+NIR or RGB+DUV spectrum is as follows: referring to Figure 6 The a surface (i.e. 201a, 202a, 203a) of the prisms 201, 202, and 203 is coated with a specific filter film, and the b surface (i.e. 201b, 202b, 203b) is used to achieve total reflection. When the composite light enters the prism 201, it first reaches the a surface 201a. Under the action of the specific filter film on the 201a surface, the deep ultraviolet or near infrared band light is separated first. The separated deep ultraviolet or near infrared band light is totally reflected by 101b and enters the first channel. The remaining light enters the prism 202. Under the action of the specific filter film on the a surface (202a) of the prism 202, the blue band light is separated. The separated blue band light adjusts the optical path by the total reflection of the 202b surface and enters the second channel. The remaining red and green light enters the prism 203. Under the action of the specific filter film on the a surface (203a) of the prism 203, the red band light is separated. The separated red band light adjusts the optical path by the total reflection of the 203b surface and enters the third channel. Finally, the remaining green band light passes through the prism 204 and enters the fourth channel. Similarly, there are two points to note. First, the angle of incidence of different band lights on the b surface of the prism should be greater than the total reflection angle. Second, the prism internal optical path experienced by each band light exiting from different channels should be consistent or approximately consistent. For specific parameter design, refer to Figure 7, the degrees of the two angles in the prism 201 determining the reflection angle are A and 2A, the degrees of the two angles in the prism 202 determining the reflection angle are 2B+A and B+A, the degrees of the two angles in the prism 203 determining the reflection angle are 2C-B and C-B, and the degree of the angle in the prism 204 adjacent to the lower angle of the prism 202 is 90°-C. According to the geometric calculation of the prism parameters, the incident angles of the light on the surfaces 201a, 202a and 203a of the filter film are A, B and C respectively. The incident angle of the deep ultraviolet or near infrared waveband light on 101b after being reflected by the 201a filter film is 2A, which should satisfy 2A>arcsin(1 / n). The incident angle of the blue waveband light on 102b after being reflected by the 202a filter film is 2B+A, which should satisfy 2B+A>arcsin(1 / n). The incident angle of the red waveband light on 103b after being reflected by the 103a filter film is 2C-B, which should satisfy 2C-B>arcsin(1 / n). Then, the total optical path Ltotal in the prism is given, and the total optical path Ltotal is taken as the reference, that is, Ltotal=(L1+L11+L12)=(L1+L2+L21+L22)=(L1+L2+L3+L31+L32), and the reasonable analytical solution satisfying the design requirements is solved in combination with the shape structure requirements of the prisms and the above total reflection angle conditions. The following is an example of key parameters satisfying the conditions:

[0051]

[0052] It should be noted that Figure 7 The dashed part in the prism is cut off to reduce the volume of the light splitting module.

[0053] The parallel light splitting efficiency of the embodiment can be improved to more than 94%, which is 30% higher than the traditional scheme in terms of light utilization rate. The frame rate can reach 1000KHZ, supporting sub-pixel level true color capture in high-speed motion scenes. It is compatible with a wide spectrum coverage from deep ultraviolet 200nm to near infrared 1700nm, meeting the special needs of semiconductor detection, fluorescence imaging, etc. Modular design supports free combination of spectral channels, and one machine can be used for multiple purposes by replacing the light splitting module.

[0054] Embodiment two

[0055] The embodiment two of the application provides a multi-channel color information processing and fusion method, which comprises the following steps:

[0056] Step S10: The sensor array converts the received light signal into a digital signal and inputs it into the processing chip in real time.

[0057] The spectroscopic prism decomposes incident light into RGB, RGB+DUV, RGB+NIR multiple spectral channels according to wavelength, and each COMS-TDI sensor in the sensor array corresponds to receive light signals of these different spectral channels, so as to realize synchronous capture and processing of different spectral bands, and input the processed digital signals into the processing chip in real time.

[0058] Step S20: The processing chip restores the received digital signals into a plurality of two-dimensional image matrices, and eliminates geometric distortion caused by optical path difference;

[0059] The optical path difference will cause geometric distortion between multi-channel images, which is manifested as translation, rotation, scaling or nonlinear deformation. The method for eliminating the geometric distortion is specifically divided into the following sub-steps:

[0060] Step S21: detecting corner points in each two-dimensional image matrix;

[0061] The corner point detection in this embodiment uses Harris corner point detection algorithm, and FAST corner point detection can also be used, which is not limited here.

[0062] Step S22: matching the detected corner points into feature point groups;

[0063] First, the centroid direction of the neighborhood of the corner point is calculated as the matching factor of the corner point, and then any corner point detected in any two-dimensional image matrix is taken as the reference sequence; the Hamming distance of each corner point in the other two-dimensional image matrix from the first corner point in the reference sequence is calculated, and a group of corner points with the smallest distance is screened out from each of the other two-dimensional image matrices to form a first feature point group; then the Hamming distance of each corner point in the other two-dimensional image matrix from the second corner point in the reference sequence is calculated, and a group of corner points with the smallest distance is screened out from each of the other two-dimensional image matrices to form a second feature point group; and so on, until a feature point group is matched for each corner point in the reference sequence.

[0064] Step S23: calculating optimal correction parameters according to the matched feature point groups;

[0065] All the feature point groups are stored as a feature matrix of m*n, m is the total number of feature point groups, and n is the number of feature points contained in each feature point group, and then the feature matrix is brought into the optimal correction parameter calculation formula: wherein, the optimal correction parameters of each two-dimensional image matrix to the target two-dimensional image matrix are obtained and wherein is the mapping matrix of the i-th two-dimensional image matrix to the target two-dimensional image matrix, is the bias vector of the i-th two-dimensional image matrix, denotes the coordinates of the corner point with subscript j in the reference sequence, represents the coordinate of the i-th row and j-th feature point in the feature matrix, j takes value from 1 to m, and i takes value from 1 to n.

[0066] Step S24: eliminating the geometric distortion caused by the optical path difference based on the optimal correction parameter;

[0067] using the calculated correction parameter and mapping each two-dimensional image matrix to a target two-dimensional image matrix to eliminate the geometric distortion caused by the optical path difference.

[0068] Step S30: using the pre-trained spectral mapping model to splice and fuse the plurality of two-dimensional image matrices to obtain a true-color multi-spectral image;

[0069] Each two-dimensional image matrix contains color information of the object, that is, spectral information. After the plurality of two-dimensional image matrices are input into the pre-trained spectral mapping model, a true-color multi-spectral image of the object can be obtained, wherein the spectral mapping model is an end-to-end learning model of U-Net structure, which extracts a multi-scale feature matrix in the two-dimensional image matrix through an encoder-decoder structure, and uses a skip connection to retain spatial details and realize adaptive mapping of multi-channel spectral information. An attention mechanism enhancement module is added before the skip connection to dynamically adjust the importance of different channels and improve spectral fidelity. The specific steps include the following sub-steps:

[0070] Step S31: compressing the multi-scale feature matrix extracted by the encoder into a channel description feature;

[0071] The multi-scale feature matrix extracted from different two-dimensional image matrices is sequentially input into the formula: to obtain a description feature of different spectral channels , wherein represents a feature value of the q-th row and e-th column in the current multi-scale feature matrix, represents a mean value of the q-th row features in the current multi-scale feature matrix, represents a mean value of the e-th column features in the current multi-scale feature matrix, q takes value from 1 to Q, Q is the total number of rows of the multi-scale feature matrix, and e takes value from 1 to E, E is the total number of columns of the multi-scale feature matrix.

[0072] Step S32: calculating a channel connection vector according to the channel description feature and the description feature of the expected image;

[0073] The channel connection vector is a key to dynamically adjust the importance of different channels and improve spectral fidelity. The formula is used to calculate the connection vector of each channel , wherein The estimated value of the channel connection vector is iterated by a particle swarm optimization algorithm, represents the descriptive feature of channel d in the fth input sample, represents the descriptive feature of the expected image (obtained using the encoder) for the fth input sample, D is a set of spectral channels, f takes values from 1 to F, and F is the number of samples in a training batch.

[0074] Step S33: rescale the extracted multi-scale feature matrix using the channel connection vector;

[0075] Each channel connection vector is multiplied by its corresponding multi-scale feature matrix to obtain the rescaled multi-scale feature matrix.

[0076] Step S34: perform skip connection on the rescaled multi-scale feature matrix through the decoder to generate the final output image;

[0077] Skip connection is the core design of U-Net, which is used to solve the problem of gradient disappearance and spatial information loss in deep networks. The core idea is to fuse the high-resolution features of the encoder (downsampling path) with the low-resolution features of the decoder (upsampling path), thereby recovering the detailed information within the image and improving the quality of the model output.

[0078] In addition, a special loss function needs to be designed to specifically optimize the spectral mapping model to ensure the use ability and accuracy of the model. The loss function designed for the spectral mapping model is represented as: wherein is the loss value calculation result, , are the optimization weights of spectral accuracy and spatial detail, respectively, is the spectral value of the kth point in the expected image (image processed by artificial processing for comparison), is the spectral value of the kth point in the model output image, k takes values from 1 to w, and w is the number of spectral data points involved in the calculation, , represent the spectral mean of the expected image and the model output image, respectively, represents the spectral covariance of the expected image and the model output image, , represent the spectral variance of the expected image and the model output image, respectively, , is a small constant to prevent division by zero.

[0079] Corresponding to the above embodiment, the embodiment of the present application provides a computer storage medium, comprising: at least one memory and at least one processor;

[0080] a memory for storing one or more program instructions;

[0081] a processor for running the one or more program instructions to perform a processing fusion method of multi-channel color information.

[0082] Corresponding to the above-mentioned embodiments, the embodiments of the present application provide a computer readable storage medium, the computer readable storage medium containing one or more program instructions, the one or more program instructions being used for being executed by a processor to perform a processing fusion method of multi-channel color information.

[0083] The embodiments disclosed in the present application provide a computer readable storage medium, the computer readable storage medium storing computer program instructions, when the computer program instructions are run on a computer, the computer program instructions make the computer execute the above-mentioned processing fusion method of multi-channel color information.

[0084] In the embodiments of the present application, the processor can be an integrated circuit chip with signal processing capability. The processor can be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0085] The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The processor reads the information in the storage medium and combines the hardware to complete the steps of the above-mentioned method.

[0086] The storage medium can be a memory, for example, can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0087] The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory.

[0088] The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0089] The storage media described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0090] Those skilled in the art should be aware that, in one or more of the above examples, the functions described in the present application can be implemented in combination of hardware and software. When the software is applied, the corresponding functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes a computer storage medium and a communication medium, wherein the communication medium includes any medium that facilitates the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0091] The above detailed description of the specific implementation of the present application further explains the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above detailed description is only a specific implementation of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A high-speed multispectral camera based on the integration of a light-splitting prism and a multi-channel TDI chip, characterized in that, It comprises: An imaging objective (1), a light splitting module (2), a sensor array (3) and a processing chip; The light splitting module (2) is provided with a plurality of light splitting prisms, the arrangement order of the light splitting prisms depends on the wavelengths of the color lights, and each light splitting prism generates an independent COMS-TDI sensor at the end of a color channel, and the line frequency is synchronously locked to the microsecond level precision, in use, the external natural light reflected by an object is projected to the light splitting module (2) through the imaging objective (1), the light splitting module (2) decomposes the incident light into multiple wave bands to generate multiple channel color information, the sensor array (3) receives the multiple channel color information generated by the light splitting module (2), and finally the processing chip processes and fuses the multiple channel color information received by the sensor array (3) into a color multispectral image: The processing chip converts the received light signals into a plurality of two-dimensional image matrices, and then splices and fuses the plurality of two-dimensional image matrices by using a pre-trained spectral mapping model to obtain a true-color multispectral image, and the specific data processing procedure of the spectral mapping model is as follows: ①Compress the multi-scale feature matrix extracted by the encoder into a channel description feature; The multi-scale feature matrix extracted from different two-dimensional image matrices is sequentially brought into the formula: to obtain the description features of different spectral channels , wherein represents the feature value of the qth row and the e th column in the current multi-scale feature matrix, represents the mean value of the qth row features in the current multi-scale feature matrix, represents the mean value of the e th column features in the current multi-scale feature matrix, q takes a value from 1 to Q, Q is the total number of rows of the multi-scale feature matrix, e takes a value from 1 to E, E is the total number of columns of the multi-scale feature matrix; ②Calculate a channel connection vector according to the channel description feature and the description feature of an expected image; Using the formula Calculate the connection vector of each channel Wherein is the estimated value of the connection vector of channel d, which is iterated by the particle swarm optimization algorithm, represents the descriptive features of channel d in the fth input sample, represents the descriptive features of the expected image for the fth input sample, D is a set of spectral channels, f takes values from 1 to F, and F is the number of samples in a training batch; ③Re-calibrate the extracted multi-scale feature matrix by using the channel connection vector; Multiply each channel connection vector and the corresponding multi-scale feature matrix to obtain a calibrated multi-scale feature matrix; ④Generate a final output image by performing a skip connection on the re-calibrated multi-scale feature matrix through a decoder; The loss function designed for the spectral mapping model is expressed as: wherein is the loss value calculation result, are the optimization weights of spectral accuracy and spatial details respectively, is the spectral value of the kth point in the expected image, is the spectral value of the kth point in the model output image, k takes 1-w, and w is the number of spectral data points participating in the calculation, respectively represent the spectral mean of the expected image and the model output image, represents the spectral covariance of the expected image and the model output image, respectively represent the spectral variance of the expected image and the model output image, is a small constant to prevent division by zero.

2. The high-speed multispectral camera based on the combination of a light-splitting prism and a multi-channel TDI chip according to claim 1, characterized in that, The light splitting module (2) decomposes the incident polychromatic light into monochromatic light of different wavelengths by using the refraction and total internal reflection of the light in the light splitting prisms and the dichroic film on the prism surface, so that the light rays of different spectral bands propagate in different directions, thereby realizing spectral separation; the angle at which the light rays of different wave bands are incident on the inner surface of the prism should be greater than the total reflection angle, and the optical path of each wave band light ray exiting from different channels in the prism should remain consistent.

3. The high-speed multispectral camera based on the combination of a light-splitting prism and a multi-channel TDI chip according to claim 1, characterized in that, The light splitting module (2) comprises a first light splitting prism (21), a second light splitting prism (22) and a third light splitting prism (23) for separating blue light, red light and green light in the incident light respectively; and the corresponding sensor array (3) comprises a first COMS-TDI sensor (31), a second COMS-TDI sensor (32) and a third COMS-TDI sensor (33) respectively arranged at the end of the blue light, red light and green light channels to form a plurality of sensing channels and realize sensing and integration of light signals of different spectral bands.

4. The high-speed multispectral camera based on the combination of a light-splitting prism and a multi-channel TDI chip according to claim 3, characterized in that, In use, each sensing channel corresponds to a specific spectral band separated by the light splitting prism, receives and converts the light signal into an electrical signal, and after integration processing, the strength and quality of the signal are improved, thereby improving the capture ability of the camera for weak light signals and spectral resolution.

5. The high-speed multispectral camera based on the combination of a light-splitting prism and a multi-channel TDI chip according to claim 3, characterized in that, The light splitting module (2) is also used to expand the incident light into RGB+NIR or RGB+DUV four channels. If the incident light is expanded into RGB+NIR, a fourth light splitting prism (24) is added in the light splitting module (2) to separate the infrared light in the incident light. If the incident light is expanded into RGB+DUV, the fourth light splitting prism (24) is added to separate the ultraviolet light in the incident light. A fourth COMS-TDI sensor (34) is also added to receive the fourth color channel information separated by the fourth light splitting prism (24).

6. A method of processing and fusing multi-channel color information, characterized by, The method is applied to the high-speed multispectral camera based on the light splitting prism and the multi-channel TDI chip fusion according to any one of claims 1-5, and includes: Step 1, the sensor array converts the received light signal into a digital signal and inputs it into the processing chip in real time; Step 2, the processing chip restores the received digital signal into a plurality of two-dimensional image matrices and eliminates the geometric distortion caused by the optical path difference; Step 3, the plurality of two-dimensional image matrices are spliced and fused by using a pre-trained spectral mapping model to obtain a true-color multispectral image.

7. The method of claim 6, wherein the processing and fusing of multi-channel color information is performed by a computer system. The geometric distortion caused by the optical path difference is eliminated, which includes the following sub-steps: detecting the corner points in each two-dimensional image matrix; matching the detected corner points into a feature point group; calculating the optimal correction parameters according to the matched feature point group; eliminating the geometric distortion caused by the optical path difference based on the optimal correction parameters.

Images