High-speed multispectral camera based on fusion of beam splitter prism and multichannel TDI chip

Abstract

The invention discloses a high-speed multispectral camera based on fusion of a beam splitter prism and a multichannel TDI chip, which relates to the technical field of optical imaging and high-speed sensing and comprises an imaging objective lens, a beam splitting module, a sensor array and a processing chip. During use, external natural light reflected by an object is projected to the light splitting module through the imaging objective lens, the light splitting module decomposes incident light into a plurality of wave bands to generate multi-channel color information, and the sensor array receives the multi-channel color information generated by the light splitting module. And finally, the processing chip processes and fuses the multi-channel color information received by the sensor array into a color multispectral image. According to the invention, fine decomposition of the spectrum can be realized, so that a camera can obtain rich and detailed spectrum information, and the method can be used for analyzing chemical components and physical characteristics of an object; the system can complete the image collection of a plurality of spectral bands in a short time, and is suitable for shooting a fast moving object or a transient phenomenon.

Description

Technical Field

[0001] The present invention relates to the technical field of optical imaging and high-speed sensing, and particularly to a high-speed multispectral camera based on the integration of a beam splitter prism and a multi-channel TDI chip. Background Art

[0002] Currently, color cameras are mainly implemented through the Bayer filter scheme. Among them, the Bayer filter arranges filters of multiple colors on the sensor, and a complete color image is obtained by means of color interpolation. Due to its low cost, color cameras using the Bayer filter have become the mainstream in the current market. However, existing TDI chips cannot achieve color through the Bayer filter, that is, existing TDI chips are only black and white and cannot achieve color imaging. Therefore, how to enable traditional TDI cameras to achieve true color, multispectral, and ultra-high-speed imaging has become an urgent problem for those skilled in the art to solve. Summary of the Invention

[0003] The present invention provides a high-speed multispectral camera based on the integration of a beam splitter prism and a multi-channel TDI chip, including: an imaging objective lens (1), a beam splitting module (2), a sensor array (3), and a processing chip; when in use, external natural light reflected by an object is projected onto the beam splitting module (2) through the imaging objective lens (1). The beam splitting module (2) decomposes the incident light into multiple bands to generate multi-channel color information. The sensor array (3) receives the multi-channel color information generated by the beam splitting module (2), and finally the processing chip processes and fuses the multi-channel color information received by the sensor array (3) into a color multispectral image.

[0004] For a high-speed multispectral camera based on the integration of a beam splitter prism and a multi-channel TDI chip as described above, multiple beam splitter prisms are provided in the beam splitting module (2). The arrangement order of the beam splitter prisms depends on the wavelengths of the respective color lights, and an independent COMS-TDI sensor is integrated at the end of each color channel generated by each beam splitter prism, and its line frequency is synchronously locked to the microsecond level of accuracy.

[0005] For a high-speed multispectral camera based on the integration of a beam splitter prism and a multi-channel TDI chip as described above, the beam splitting module (2) uses the refraction and total internal reflection of light in each beam splitter prism, as well as the dichroic thin film on the prism surface, to decompose the incident polychromatic light into monochromatic lights of different wavelengths, so that the light rays of different spectral bands propagate in different directions, thereby achieving spectral separation; the angles of the light rays of different bands incident on the inner surface of the prism should be greater than the total reflection angle, and the optical paths of the light rays of each band exiting from different channels in the prism should be kept consistent.

[0006] A high-speed multispectral camera based on the fusion of a spectroscopic prism and a multi-channel TDI chip as described above, wherein the spectroscopic module (2) includes a first spectroscopic prism (21), a second spectroscopic prism (22), and a third spectroscopic prism (23), which are respectively used to separate blue light, red light, and green light in the incident light; correspondingly, the sensor array (3) includes a first COMS-TDI sensor (31), a second COMS-TDI sensor (32), and a third COMS-TDI sensor (33), which are respectively deployed at the ends of the blue light, red light, and green light channels to form multiple sensing channels, realizing the sensing and integration of optical signals in different spectral bands.

[0007] A high-speed multispectral camera based on the fusion of a spectroscopic prism and a multi-channel TDI chip as described above, wherein the spectroscopic module (2) is further used to expand the incident light into four channels of RGB+NIR or RGB+DUV. If it is expanded to RGB+NIR, an additional spectroscopic prism, i.e., a fourth spectroscopic prism (24), is added in the spectroscopic module (2) to separate the infrared light in the incident light. If it is expanded to RGB+DUV, the added fourth spectroscopic prism (24) is used to separate the ultraviolet light in the incident light. Correspondingly, an additional COMS-TDI sensor, i.e., a fourth COMS-TDI sensor (34), is added to receive the information of the fourth color channel separated by the fourth spectroscopic prism (24).

[0008] The present invention also provides a method for processing and fusing multi-channel color information, including: Step1: The sensor array converts the received optical signal into a digital signal and inputs it into the processing chip in real time; Step2: The processing chip restores the received digital signal into multiple two-dimensional image matrices and eliminates the geometric distortion caused by the optical path difference; Step3: Use the pre-trained spectral mapping model to splice and fuse the multiple two-dimensional image matrices to obtain a true-color multispectral image.

[0009] A method for processing and fusing multi-channel color information as described above, wherein eliminating the geometric distortion caused by the optical path difference is specifically divided into the following sub-steps: Detect the corner points in each two-dimensional image matrix; Match the detected corner points into a set of feature points; Calculate the optimal correction parameters according to the matched set of feature points; Eliminate the geometric distortion caused by the optical path difference based on the optimal correction parameters.

[0010] A method for processing and fusing multi-channel color information as described above, wherein the spectral mapping model is an end-to-end learning model with a U-Net structure. It extracts multi-scale feature matrices from a two-dimensional image matrix through an encoder-decoder structure, and uses skip connections to retain spatial details, realizing the adaptive mapping of multi-channel spectral information. An attention mechanism enhancement module is also added before the skip connections to dynamically adjust the importance of different channels and improve spectral fidelity. The specific steps are as follows: Compress the multi-scale feature matrix extracted by the encoder into a channel description feature; Calculate the channel connection vector based on the channel description feature and the description feature of the expected image; Use the channel connection vector to recalibrate the extracted multi-scale feature matrix; Generate the final output image through the decoder's skip connection of the recalibrated multi-scale feature matrix.

[0011] The beneficial effects achieved by the present invention are as follows: High spectral resolution: The spectroscope can finely decompose light into spectra of multiple different wavelengths. Each channel of the multi-channel sensor array can correspond to a specific narrow band, enabling the camera to obtain rich and detailed spectral information, and enabling traditional black-and-white TDI cameras to achieve color multi-spectral imaging.

[0012] Ultra-high-speed imaging: The multi-channel sensor array has fast signal processing capabilities and high frame rates. Combined with the efficient light splitting of the spectroscope, the camera can complete the image acquisition of multiple spectral bands in a short time, and is suitable for photographing fast-moving objects or transient phenomena.

[0013] High sensitivity and low noise: The integration function of the multi-channel sensor array can accumulate optical signals, improve the camera's sensitivity to weak light, and at the same time reduce the noise level, enabling high-quality multi-spectral images to be obtained under low-light conditions.

[0014] Good spatial resolution: The integrated design of the spectroscope and the multi-channel sensor array can ensure high spatial resolution in different spectral channels, clearly presenting the details and morphology of objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0016] Figure 1It is a schematic diagram of a high-speed multispectral camera based on the integration of a beam splitter prism and a multi-channel TDI chip provided in the first embodiment of the present application; Figure 2 It is a schematic diagram of a beam splitting module for RGB spectra and a sensor array provided in the first embodiment of the present application; Figure 3 It is a schematic diagram of the working principle of a beam splitting module for RGB spectra provided in the first embodiment of the present application; Figure 4 It is a schematic diagram of the parameter design of a beam splitting module for RGB spectra provided in the first embodiment of the present application; Figure 5 It is a schematic diagram of a beam splitting module for RGB+NIR or RGB+DUV spectra and a sensor array provided in the first embodiment of the present application; Figure 6 It is a schematic diagram of the working principle of a beam splitting module for RGB+NIR or RGB+DUV spectra provided in the first embodiment of the present application; Figure 7 It is a schematic diagram of the parameter design of a beam splitting module for RGB+NIR or RGB+DUV spectra provided in the first embodiment of the present application.

[0017] Reference numerals: 1, imaging objective lens; 2, beam splitting module; 3, sensor array; 21, first beam splitter prism; 22, second beam splitter prism; 23, third beam splitter prism; 24, fourth beam splitter prism; 31, first COMS-TDI sensor; 32, second COMS-TDI sensor; 33, third COMS-TDI sensor; 34, fourth COMS-TDI sensor. Detailed implementation manners

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0019] Embodiment 1 As Figure 1As shown in the figure, Embodiment 1 of the present application provides a high-speed multispectral camera based on the fusion of a beam splitting prism and a multi-channel TDI chip, including: an imaging objective lens 1, a beam splitting module 2, a sensor array 3, and a processing chip; when in use, the external natural light reflected by an object is projected onto the beam splitting module 2 through the imaging objective lens 1. The beam splitting module 2 decomposes the incident light into multiple bands to generate multi-channel color information. The sensor array 3 receives the multi-channel color information generated by the beam splitting module 2, and finally the processing chip processes and fuses the multi-channel color information received by the sensor array 3 into a color multispectral image.

[0020] That is to say, the present application uses multiple COMS-TDI sensors each containing a monochromatic TDI chip to jointly receive multiple color lights in natural light, and then realizes color imaging of a black-and-white TDI camera through the fusion processing of multiple color lights.

[0021] In the embodiment of the present application, multiple beam splitting prisms are provided in the beam splitting module 2. The arrangement order of the beam splitting prisms depends on the wavelengths of the respective color lights, and it is necessary to ensure that the color light with the longest or shortest wavelength can be decomposed first. Moreover, an independent COMS-TDI sensor is integrated at the end of each color channel generated by each beam splitting prism, and its line frequency is synchronized and locked to the microsecond level of accuracy. Therefore, the corresponding sensor array 3 also includes multiple COMS-TDI sensors; the beam splitting module 2 uses the refraction and total internal reflection of light in each beam splitting prism, as well as the dichroic thin film on the prism surface, to decompose the incident polychromatic light into monochromatic lights of different wavelengths, enabling the light rays in different spectral bands to propagate in different directions, thereby achieving spectral separation.

[0022] Figure 1 Taking three beam splitting prisms as an example, see Figure 2 , the beam splitting module 2 includes a first beam splitting prism 21, a second beam splitting prism 22, and a third beam splitting prism 23, which are respectively used to separate blue light, red light, and green light in the incident light; the corresponding sensor array 3 includes a first COMS-TDI sensor 31, a second COMS-TDI sensor 32, and a third COMS-TDI sensor 33, which are respectively deployed at the ends of the blue light, red light, and green light channels to form multiple induction channels, realizing the induction and integration of optical signals in different spectral bands. When the camera is working, each induction channel will correspond to a specific spectral band separated by the beam splitting prism, receive and convert the optical signal into an electrical signal, and after integral processing, improve the intensity and quality of the signal, thereby enhancing the camera's ability to capture weak light signals and spectral resolution.

[0023] For the RGB spectrum, the specific beam splitting principle of the beam splitting module 2 is as follows: See Figure 3, the a surfaces (i.e., 101a and 102a) of the prisms 101 and 102 are coated with a specific filter film, and the b surfaces (i.e., 101b and 102b) are used to achieve total internal reflection. When the RGB incident light enters the beam splitting prism 101 horizontally, it first reaches the a surface 101a of the prism 101. Under the action of the specific filter film on 101a, the short-wave blue light in the incident light is separated first and undergoes total internal reflection through the surface of the prism 101b and enters the corresponding first channel. The remaining light enters the prism 102. On the a surface (102a) of the prism 102, the long-wave red light in the incident light is separated by the filter film, and then the optical path is adjusted through total internal reflection on the surface 102b and enters the second channel; finally, the remaining green-band light enters the third channel, and the beam splitting is completed. It should be noted that the incident angles of light in different bands on the b surface of the prism should be greater than the total internal reflection angle, and the optical path lengths of each band of light exiting from different channels in the prism should be kept the same or approximately the same. For specific parameter design, refer to Figure 4 , the incident angles of the central field of view light on each surface and the geometric optical path lengths of each section in the prism are drawn in the figure. Let the degrees of the two angles that determine the reflection angle in the prism 101 be A and 2A respectively, the degrees of the two angles that determine the reflection angle in the prism 102 be 2B + A and B + A respectively, and the degree of the angle adjacent to the lower part of the prism in the prism 103 be 90° - B. According to the above prism parameters for geometric calculation, the incident angles of the light on the filter film surfaces 101a and 102a are A and B respectively. After the blue-band light is reflected by the 101a filter film, the incident angle on 101b is 2A, and it should satisfy 2A > arcsin(1 / n), where n is the refractive index of the prism material. After the red-band light is reflected by the 102a filter film, the incident angle on 102b is 2B + A, and it should satisfy 2B + A > arcsin(1 / n); then, given the total optical path L_total in the prism, with L_total = (L1 + L11 + L12) = (L1 + L2 + L21 + L22) = (L1 + L2 + L3) as the benchmark, combined with the shape and structure requirements of each prism and the above total internal reflection angle conditions, a reasonable analytical solution that meets the design requirements is solved. The following are a set of key parameters of an example that meets the conditions:

[0024] The basic structure of the COMS-TDI sensor usually consists of 32 to 256 rows of pixels, and the number of pixels per row can reach 16K or higher. Its working process can be described by the following mathematical model: S(x) = Σ(i = 1 to N) Pi(x) where S(x) is the final output signal, N is the number of TDI stages (i.e., the number of pixel rows), and Pi(x) is the optical signal received by the i-th row at position x. This formula clearly shows the core idea of TDI technology: enhancing the signal intensity through multiple accumulations.

[0025] The working process of the TDI can be described in more detail as follows: a) When light irradiates the first row of pixels, initial photo-charges are generated. b) In the next clock cycle, these charges are precisely transferred to the second row of pixels. c) Meanwhile, the first row of pixels continues to receive new photons and generate new charges. d) This process continues in the subsequent pixel rows, with each row receiving new photons and accumulating charges from the previous row. e) When the charges reach the last row of pixels, the accumulated signal is read out to form the final image signal.

[0026] The beam splitting module 2 is also used to expand the incident light into four channels of RGB+NIR (near-infrared) or RGB+DUV (deep ultraviolet). Refer to Figure 5 , if it is expanded to RGB+NIR, a beam splitting prism, namely the fourth beam splitting prism 24, is added in the beam splitting module 2 to separate the infrared light in the incident light. If it is expanded to RGB+DUV, the added fourth beam splitting prism 24 is used to separate the ultraviolet light in the incident light. Correspondingly, a COMS-TDI sensor, namely the fourth COMS-TDI sensor 34, is added to receive the fourth color channel information separated by the fourth beam splitting prism 24.

[0027] For the beam splitting principle of the beam splitting module 2 for the RGB+NIR or RGB+DUV spectrum, it is as follows: Refer to Figure 6 , the a surfaces (i.e., 201a, 202a, 203a) of the prisms 201, 202, 203 are coated with specific filter films, and the b surfaces (i.e., 201b, 202b, 203b) are used to achieve total reflection. When the composite light is incident on the prism 201, it will first reach its a surface 201a. Under the action of the specific filter film on the 201a surface, the deep ultraviolet or near-infrared band light is separated first. The separated deep ultraviolet or near-infrared band light undergoes total reflection through 101b and enters the first channel; the remaining light enters the prism 202. Under the action of the specific filter film on its a surface (202a), the blue band light is separated. The separated blue band light adjusts the optical path through the total reflection on the 202b surface and enters the second channel; the remaining red and green lights enter the prism 203. On its a surface (203a), the red band light is separated by the filter film. The separated red band light adjusts the optical path through the total reflection on the 203b surface and enters the third channel; finally, the remaining green band light passes through the prism 204 and enters the fourth channel. Similarly, there are two points to note. First, the incident angles of light in different bands on the b surface of the prism should be greater than the total reflection angle. Second, the optical paths of the light in different bands exiting from different channels in the prism should be kept the same or approximately the same. For the specific parameter design, refer to Figure 7, let the degrees of the two angles that determine the reflection angle in prism 201 be A and 2A respectively, the degrees of the two angles that determine the reflection angle in prism 202 be 2B + A and B + A respectively, the degrees of the two angles that determine the reflection angle in prism 203 be 2C - B and C - B respectively, and the degree of the angle adjacent to the lower part of prism 202 in prism 204 be 90° - C. According to the above prism parameters for geometric calculation, the incident angles of light on the surfaces 201a, 202a, and 203a where the filter film is located are A, B, and C respectively. After the deep ultraviolet or near-infrared band light is reflected by the 201a filter film, the incident angle on 101b is 2A, and it should satisfy 2A > arcsin(1 / n). After the blue band light is reflected by the 202a filter film, the incident angle on 102b is 2B + A, and it should satisfy 2B + A > arcsin(1 / n). After the red band light is reflected by the 103a filter film, the incident angle on 103b is 2C - B, and it should satisfy 2C - B > arcsin(1 / n); Subsequently, given the total optical path L_total in the prism, with L_total = (L1 + L2 + L3 + L4) = (L1 + L11 + L12) = (L1 + L2 + L21 + L22) = (L1 + L2 + L3 + L31 + L32) as the benchmark, combined with the shape and structure requirements of each prism and the above total reflection angle conditions, solve the reasonable analytical solution that meets the design requirements. The following are a set of key parameters of an example that meets the conditions:

[0028] It should be noted that Figure 7 The dashed part in it is the prism part without light passing through cut off to reduce the volume of the beam splitting module.

[0029] The parallel beam splitting efficiency of this embodiment can be increased to more than 94%, and the light utilization rate is increased by 30% compared with the traditional scheme; the frame rate can reach 1000K HZ, supporting sub-pixel true color capture in high-speed motion scenes; it is compatible with a wide spectral range covering deep ultraviolet 200nm to near infrared 1700nm, meeting special requirements such as semiconductor detection and fluorescence imaging; the modular design supports free combination of spectral channels, and by replacing the beam splitting module, the same machine can be used for multiple purposes.

[0030] Embodiment 2 Embodiment 2 of the present application provides a method for processing and fusing multi-channel color information, including: Step S10: The sensor array converts the received optical signal into a digital signal and inputs it into the processing chip in real time; The beam splitter prism decomposes the incident light into multiple spectral channels of RGB, RGB+DUV, and RGB+NIR according to wavelength. Each COMS-TDI sensor in the sensor array respectively receives the optical signals of these different spectral channels, realizes the synchronous capture and processing of different spectral bands, and inputs the processed digital signals into the processing chip in real time.

[0031] Step S20: The processing chip restores the received digital signals into multiple two-dimensional image matrices and eliminates the geometric distortion caused by the optical path difference. The optical path difference will cause geometric distortion between multi-channel images, manifested as translation, rotation, scaling, or non-linear deformation. The method for eliminating its geometric distortion is specifically divided into the following sub-steps: Step S21: Detect the corner points in each two-dimensional image matrix. In this embodiment, the Harris corner detection algorithm is used for corner detection, and the FAST corner detection can also be used, which is not limited here.

[0032] Step S22: Match the detected corner points into feature point groups. First, calculate the centroid direction of the corner point neighborhood as the matching factor of the corner point. Then, take the corner points detected in any two-dimensional image matrix as the reference sequence; calculate the Hamming distance between each corner point in other two-dimensional image matrices and the first corner point in the reference sequence, and screen out a corner point with the smallest distance from each of the other two-dimensional image matrices to form the first feature point group; then calculate the Hamming distance between each corner point in other two-dimensional image matrices and the second corner point in the reference sequence, and screen out a corner point with the smallest distance from each of the other two-dimensional image matrices to form the second feature point group; and so on, until a feature point group is matched for each corner point in the reference sequence.

[0033] Step S23: Calculate the optimal correction parameters according to the matched feature point groups. Store all the feature point groups as an m*n feature matrix, where m is the total number of feature point groups and n is the number of feature points included in each feature point group. Then, substitute the feature matrix into the optimal correction parameter calculation formula: to obtain the optimal correction parameters from each two-dimensional image matrix to the target two-dimensional image matrix and , where is the mapping matrix from the i-th two-dimensional image matrix to the target two-dimensional image matrix, is the bias vector of the i-th two-dimensional image matrix, represents the corner point coordinates with subscript j in the reference sequence, represents the coordinates of the j-th feature point in the i-th row of the feature matrix, where j takes values from 1 to m and i takes values from 1 to n.

[0034] Step S24: Eliminate the geometric distortion caused by the optical path difference based on the optimal correction parameters; Using the calculated correction parameters and map each two-dimensional image matrix to the target two-dimensional image matrix to eliminate the geometric distortion caused by the optical path difference.

[0035] Step S30: Use the pre-trained spectral mapping model to splice and fuse multiple two-dimensional image matrices to obtain a true-color multispectral image; Each two-dimensional image matrix contains a kind of color information of the object to be photographed, that is, spectral information. Input the corrected multiple two-dimensional image matrices into the pre-trained spectral mapping model, and a true-color multispectral image of the object can be obtained. The spectral mapping model is an end-to-end learning model with a U-Net structure. It extracts multi-scale feature matrices from the two-dimensional image matrices through an encoder-decoder structure and uses skip connections to retain spatial details to achieve adaptive mapping of multi-channel spectral information. An attention mechanism enhancement module is also added before the skip connection to dynamically adjust the importance of different channels and improve spectral fidelity. It specifically includes the following sub-steps: Step S31: Compress the multi-scale feature matrices extracted by the encoder into channel description features; The multi-scale feature matrices extracted from different two-dimensional image matrices are successively brought into the formula: to obtain the description features of different spectral channels , where represents the feature value of the q-th row and e-th column in the current multi-scale feature matrix, represents the mean value of the features in the q-th row within the current multi-scale feature matrix, represents the mean value of the features in the e-th column within the current multi-scale feature matrix. q takes values from 1 to Q, Q is the total number of rows of the multi-scale feature matrix, and e takes values from 1 to E, E is the total number of columns of the multi-scale feature matrix.

[0036] Step S32: Calculate the channel connection vectors according to the channel description features and the description features of the expected image; The channel connection vector is the key to dynamically adjusting the importance of different channels and improving spectral fidelity. Use the formula to calculate the connection vectors of each channel , where is the estimated value of the connection vector of channel d, which is iterated through the particle swarm optimization algorithm, represents the description feature of channel d in the f-th input sample, represents the description feature of its expected image (obtained using the encoder) for the f-th input sample. D is the set of spectral channels, and f takes values from 1 to F, F is the number of samples in a training batch.

[0037] Step S33: recalibrate the extracted multi-scale feature matrix using the channel connection vector; Multiply each channel connection vector by its corresponding multi-scale feature matrix to obtain the recalibrated multi-scale feature matrix.

[0038] Step S34: perform skip connections on the recalibrated multi-scale feature matrix through a decoder to generate the final output image; Skip Connection is the core design of U-Net, which is used to solve the problems of gradient disappearance and spatial information loss in deep networks. Its core idea is to fuse the high-resolution features of the encoder (downsampling path) with the low-resolution features of the decoder (upsampling path), so as to restore the detailed information in the image and improve the quality of the model output.

[0039] In addition, a special loss function needs to be designed to specifically optimize the spectral mapping model to ensure the usability and accuracy of the model. The loss function designed for the spectral mapping model is expressed as: where is the calculation result of the loss value, , are the optimization weights of spectral accuracy and spatial details respectively, is the spectral value at the kth position in the expected image (the image used for comparison after manual processing), is the spectral value at the kth position in the model output image, k takes values from 1 to w, and w is the number of spectral data points participating in the calculation, , represent the spectral means of the expected image and the model output image respectively, represents the spectral covariance between the expected image and the model output image, , represent the spectral variances of the expected image and the model output image respectively, , are small constants to prevent division by zero.

[0040] Corresponding to the above embodiments, an embodiment of the present invention provides a computer storage medium, including: at least one memory and at least one processor; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute a processing and fusion method for multi-channel color information.

[0041] Corresponding to the above embodiments, an embodiment of the present invention provides a computer-readable storage medium, which contains one or more program instructions for being executed by a processor to perform a processing and fusion method for multi-channel color information.

[0042] An embodiment of the present invention discloses a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions run on a computer, the computer is caused to execute the above-mentioned processing and fusion method for multi-channel color information.

[0043] In an embodiment of the present invention, the processor may be an integrated circuit chip with signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0044] It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and combines its hardware to complete the steps of the above method.

[0045] The storage medium may be a memory, for example, it may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0046] Among them, the non-volatile memory may be a read-only memory (ROM for short), a programmable read-only memory (PROM for short), an erasable programmable read-only memory (EPROM for short), an electrically erasable programmable read-only memory (EEPROM for short), or a flash memory.

[0047] The volatile memory may be a Random Access Memory (RAM) which serves as an external cache. By way of example but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0048] The storage media described in the embodiments of the present invention are intended to include but not limited to these and any other suitable types of memories.

[0049] Those skilled in the art should be able to realize that in one or more of the above examples, the functions described in the present invention can be implemented by a combination of hardware and software. When applying software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. The computer-readable medium includes computer storage media and communication media, where the communication media includes any medium that facilitates the transfer of a computer program from one place to another. The storage media can be any available medium accessible by a general or special purpose computer.

[0050] The above specific embodiments have further elaborated on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the protection scope of the present invention.

Claims

1. A high-speed multispectral camera based on the fusion of a spectroscopic prism and a multi-channel TDI chip, characterized in that, Comprising: An imaging objective lens (1), a beam splitting module (2), a sensor array (3), and a processing chip; The beam splitting module (2) is provided with multiple beam splitting prisms. The arrangement order of the beam splitting prisms depends on the wavelengths of each color light, and an independent COMS-TDI sensor is integrated at the end of each color channel generated by each beam splitting prism. Its line frequency is synchronously locked to the microsecond level of precision. During use, the external natural light reflected by an object is projected onto the beam splitting module (2) through the imaging objective lens (1). The beam splitting module (2) decomposes the incident light into multiple bands to generate multi-channel color information. The sensor array (3) receives the multi-channel color information generated by the beam splitting module (2), and finally the processing chip processes and fuses the multi-channel color information received by the sensor array (3) into a color multi-spectral image.

2. The high-speed multispectral camera based on the fusion of a beam splitter prism and a multi-channel TDI chip according to claim 1, characterized in that The beam splitting module (2) utilizes the refraction and total internal reflection of light in each beam splitting prism, as well as the dichroic film on the prism surface, to decompose the incident polychromatic light into monochromatic lights of different wavelengths, enabling the light rays of different spectral bands to propagate in different directions, thereby achieving spectral separation; the angles of the light rays of different bands incident on the inner surface of the prism should be greater than the total reflection angle, and the optical path lengths of the light rays of each band exiting from different channels in the prism should be kept consistent.

3. A high-speed multispectral camera based on the fusion of a beam splitter prism and a multi-channel TDI chip according to claim 1, characterized in that, The beam splitting module (2) includes a first beam splitting prism (21), a second beam splitting prism (22), and a third beam splitting prism (23), which are respectively used to separate blue light, red light, and green light in the incident light; the corresponding sensor array (3) includes a first COMS-TDI sensor (31), a second COMS-TDI sensor (32), and a third COMS-TDI sensor (33), which are respectively deployed at the ends of the blue light, red light, and green light channels to form multiple induction channels, realizing the induction and integration of optical signals of different spectral bands.

4. A high-speed multispectral camera based on the fusion of a beam splitter prism and a multi-channel TDI chip according to claim 3, characterized in that, When the camera is working, each induction channel corresponds to a specific spectral band separated by the beam splitting prism, receives the optical signal and converts it into an electrical signal. After integral processing, the intensity and quality of the signal are improved, thereby enhancing the camera's ability to capture weak light signals and spectral resolution.

5. The high-speed multispectral camera based on the fusion of a beam splitter prism and a multi-channel TDI chip according to claim 3, wherein The beam splitting module (2) is also used to expand the incident light into an RGB+NIR or RGB+DUV four-channel. If it is expanded to RGB+NIR, a beam splitting prism, namely a fourth beam splitting prism (24), is added to the beam splitting module (2) to separate the infrared light in the incident light. If it is expanded to RGB+DUV, the added fourth beam splitting prism (24) is used to separate the ultraviolet light in the incident light. Correspondingly, a COMS-TDI sensor, namely a fourth COMS-TDI sensor (34), is added to receive the fourth color channel information separated by the fourth beam splitting prism (24).

6. A processing and fusion method for multi-channel color information, characterized in that The method is applied to a high-speed multi-spectral camera based on the fusion of beam splitting prisms and multi-channel TDI chips as described in any one of claims 1-5, and includes: Step1: The sensor array converts the received optical signal into a digital signal and inputs it into the processing chip in real time; Step2: The processing chip restores the received digital signal into multiple two-dimensional image matrices and eliminates the geometric distortion caused by the optical path difference; Step 3. Use the pre-trained spectral mapping model to splice and fuse multiple two-dimensional image matrices to obtain a true-color multispectral image.

7. A method for processing and fusing multi-channel color information according to claim 6, characterized in that Eliminate the geometric distortion caused by the optical path difference, which is specifically divided into the following sub-steps: Detect the corner points in each two-dimensional image matrix; Match the detected corner points into feature point groups; Calculate the optimal correction parameters based on the matched feature point groups; Eliminate the geometric distortion caused by the optical path difference based on the optimal correction parameters.

Images