Dual-band imaging method and system integrating image sensor and liquid crystal shutter, and medium

Abstract

The embodiment of the invention provides a dual-band imaging method and system integrating an image sensor and a liquid crystal shutter, and a medium, and the method comprises the steps: obtaining the type of a CMOS image sensor, and configuring the working parameters of the CMOS image sensor; initializing the state of the liquid crystal shutter device, setting the operation parameters of the liquid crystal shutter device based on the working parameters of the CMOS image sensor, obtaining the operation frequency of the liquid crystal shutter device, and judging whether the operation frequency reaches the switching frequency; if not, entering a first imaging mode to obtain a first waveband image; if yes, optical filter switching is carried out, a second imaging mode is entered, and a second wave band image is obtained; performing fusion processing based on the first waveband image and the second waveband image to generate a dual-waveband image; different optical filters are dynamically switched through the liquid crystal shutter device, two images with different wave bands are obtained, then the images are enhanced and fused, double-wave-band imaging is achieved, and the imaging effect is improved.

Description

Technical Field

[0001] This application relates to the field of imaging technology. Specifically, it relates to a dual-band imaging method, system, and medium integrating an image sensor and a liquid crystal shutter. Background Art

[0002] Metal Oxide Semiconductor (MOS) Image Sensors (CIS) are widely used in various electronic imaging systems, such as cameras in mobile devices, as well as cameras for security, remote sensing, machine vision, and automotive applications. They can be used for dual-band imaging in the Visible Light (VIS) band (wavelength range of 400 - 700 nanometers) and the Near Infrared (NIR) band (wavelength range of 700 - 1100 nanometers) because silicon is sensitive to photons in these bands. Application examples of visible light / near infrared dual-band imaging systems include enhancing image quality in low light conditions, non-contact vital sign monitoring, driver monitoring, and image dehazing.

[0003] Although dual-frequency imaging (visible light and near infrared) can be achieved through two or more imaging elements, in applications where system size and weight are restricted, a single image sensor method is more preferred. One method of single-sensor dual-frequency imaging is to replace the filter in front of the image sensor through a mechanism. In existing imaging methods, an Infrared Cutoff Filter (IRCF) is placed in the optical path during the day and removed at night. This method does not use the entire array resolution simultaneously in the two spectral bands. Another method is based on an image sensor with a Color Filter Array (CFA) that includes four types of filters: Red (R), Green (G), Blue (B), and Infrared (IR), also known as RGB-IR CFA, and a visible light / near infrared dual-band pass filter (DBPF). This method allows imaging in both the visible light and near infrared light bands simultaneously, but the image processing flow requires non-standard processing modules, and the resolution of the color and near infrared images will be reduced. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a dual-band imaging method, system, and medium integrating an image sensor and a liquid crystal shutter. By dynamically switching different filters through a liquid crystal shutter device, two images in different bands are obtained, and then the images are enhanced and fused to achieve dual-band imaging and improve the imaging effect.

[0005] The embodiments of this application also provide a dual-band imaging method integrating an image sensor and a liquid crystal shutter, including: Obtain the type of the CMOS image sensor and configure the working parameters of the CMOS image sensor; Initialize the state of the liquid crystal shutter device, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, obtain the operating frequency of the liquid crystal shutter device, and determine whether the operating frequency reaches the switching frequency; If the switching frequency is not reached, control the liquid crystal shutter device to enter the first imaging mode, allow the light in the first band to pass through and reach the CMOS image sensor based on the first imaging mode, and obtain the first band image; If the switching frequency is reached, control the liquid crystal shutter device to perform filter switching, enter the second imaging mode, allow the light in the second band to pass through and reach the CMOS image sensor based on the first imaging mode, and obtain the second band image; Perform preprocessing on the first band image and the second band image to obtain a preprocessed image, and fuse the preprocessed image to generate a dual-band image.

[0006] Optionally, in the dual-band imaging method of the integrated image sensor and liquid crystal shutter described in the embodiments of the present application, obtain the type of the CMOS image sensor and configure the operating parameters of the CMOS image sensor, specifically including: Obtain the user manual and technical specification of the CMOS image sensor, and analyze the characteristics and performance parameter information of the CMOS image sensor based on the user manual and technical specification; Obtain the identification information of the CMOS image sensor, and analyze the marked model and manufacturer based on the identification information; Based on the characteristics and performance parameter information of the CMOS image sensor, the marked model and the manufacturer, obtain the type of the CMOS image sensor; Configure the operating parameters based on the type of the CMOS image sensor, and the operating parameters include exposure time, gain parameter and clock parameter.

[0007] Optionally, in the dual-band imaging method of the integrated image sensor and liquid crystal shutter described in the embodiments of the present application, initialize the state of the liquid crystal shutter device, and set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, specifically including: Obtain the power supply state information of the liquid crystal shutter device, and analyze the operating state information of the liquid crystal shutter device based on the power supply state information; Compare the operating state information with the set initial state information to obtain a state deviation rate; Determine whether the state deviation rate is greater than or equal to the set state deviation rate threshold; If so, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, and the operating parameters include switching frequency and operating voltage; If not, perform a reset operation on the liquid crystal shutter device to restore the current state of the liquid crystal shutter device to the initial state and set the operating parameters of the liquid crystal shutter device.

[0008] Optionally, in the dual-band imaging method of the integrated image sensor and liquid crystal shutter described in the embodiments of the present application, controlling the liquid crystal shutter device to enter the first imaging mode, and based on the first imaging mode, allowing the light of the first band to pass through and reach the CMOS image sensor to obtain a first-band image, specifically including: Obtain the current state information of the liquid crystal shutter device, and compare the current state information of the liquid crystal shutter device with the condition information of the first imaging mode; Determine whether the current state information of the liquid crystal shutter device meets the condition information of the first imaging mode; If it is satisfied, switch to the first band based on the first imaging mode. The first band is the visible light band. Project the visible light onto the CMOS image sensor based on the first band to obtain a first-band image, and the first-band image is a visible light band image; If it is not satisfied, adjust the current state parameters of the liquid crystal shutter device and control the liquid crystal shutter device to enter the first imaging mode.

[0009] Optionally, in the dual-band imaging method of the integrated image sensor and liquid crystal shutter described in the embodiments of the present application, controlling the liquid crystal shutter device to perform filter switching to enter the second imaging mode, and based on the first imaging mode, allowing the light of the second band to pass through and reach the CMOS image sensor to obtain a second-band image, specifically including: Obtain an imaging mode switching instruction, switch the first band to the second band based on the imaging mode switching instruction. The second band is the near-infrared light band, and control the liquid crystal shutter device to enter the second imaging mode; Set the light passing band based on the second imaging mode, obtain an optical signal, and analyze the band where the optical signal is located based on the optical signal; Process the optical signal based on the set light passing band, allow the optical signal in the set light passing band to pass through, and convert the optical signal into an electrical signal; Transmit the electrical signal to the CMOS image sensor to obtain a second-band image.

[0010] Optionally, in the dual-band imaging method of the integrated image sensor and liquid crystal shutter described in the embodiments of the present application, perform preprocessing on the first-band image and the second-band image to obtain a preprocessed image, and perform fusion processing on the preprocessed image to generate a dual-band image, specifically including: Obtain the first-band image, extract image features, analyze the image noise features based on the image features, and remove the noise features in the image based on the mean filtering or median filtering algorithm to obtain a first optimized image; Analyze the pixel information of the first optimized image, analyze the image dead pixels based on the pixel information, correct the image dead pixels, and obtain the first preprocessed image; Obtain the second-band image, extract the image features, analyze the image noise features based on the image features, and remove the noise features in the image based on the mean filtering or median filtering algorithm to obtain the second optimized image; Analyze the pixel information of the second optimized image, analyze the image dead pixels based on the pixel information, correct the image dead pixels, and obtain the second preprocessed image; Perform weighted fusion processing based on the pixel information of the first preprocessed image and the pixel information of the second preprocessed image to obtain the dual-band image.

[0011] In a second aspect, an embodiment of the present application provides a dual-band imaging system integrating an image sensor and a liquid crystal shutter. The system includes: a memory and a processor. The memory includes a program for the dual-band imaging method of integrating an image sensor and a liquid crystal shutter. When the program for the dual-band imaging method of integrating an image sensor and a liquid crystal shutter is executed by the processor, the following steps are implemented: Obtain the type of the CMOS image sensor and configure the operating parameters of the CMOS image sensor; Initialize the state of the liquid crystal shutter device, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, obtain the operating frequency of the liquid crystal shutter device, and determine whether the operating frequency reaches the switching frequency; If the switching frequency is not reached, control the liquid crystal shutter device to enter the first imaging mode, and allow the light of the first band to pass through and reach the CMOS image sensor to obtain the first-band image; If the switching frequency is reached, control the liquid crystal shutter device to perform filter switching, enter the second imaging mode, and allow the light of the second band to pass through and reach the CMOS image sensor to obtain the second-band image; Perform preprocessing on the first-band image and the second-band image to obtain a preprocessed image, and fuse the preprocessed image to generate a dual-band image.

[0012] Optionally, in the dual-band imaging system integrating an image sensor and a liquid crystal shutter according to the embodiment of the present application, obtaining the type of the CMOS image sensor and configuring the operating parameters of the CMOS image sensor specifically includes: Obtain the user manual and technical specification of the CMOS image sensor, and analyze the characteristics and performance parameter information of the CMOS image sensor based on the user manual and technical specification; Obtain the identification information of the CMOS image sensor, and analyze the marked model and manufacturer based on the identification information; Based on the characteristics and performance parameter information of the CMOS image sensor, the labeled model, and the manufacturer, the type of the CMOS image sensor is obtained; Configure the working parameters based on the type of the CMOS image sensor, where the working parameters include the exposure time, the gain parameter, and the clock parameter.

[0013] Optionally, in the dual-band imaging system of the integrated image sensor and the liquid crystal shutter described in the embodiments of the present application, initialize the state of the liquid crystal shutter device, and set the operating parameters of the liquid crystal shutter device based on the working parameters of the CMOS image sensor, specifically including: Obtain the power supply status information of the liquid crystal shutter device, and analyze the operating status information of the liquid crystal shutter device based on the power supply status information; Compare the operating status information with the set initial status information to obtain the status deviation rate; Determine whether the status deviation rate is greater than or equal to the set status deviation rate threshold; If so, set the operating parameters of the liquid crystal shutter device based on the working parameters of the CMOS image sensor, where the operating parameters include the switching frequency and the working voltage; If not, perform a reset operation on the liquid crystal shutter device, restore the current state of the liquid crystal shutter device to the initial state, and set the operating parameters of the liquid crystal shutter device.

[0014] In a third aspect, the embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium includes a program for the dual-band imaging method of the integrated image sensor and the liquid crystal shutter. When the program for the dual-band imaging method of the integrated image sensor and the liquid crystal shutter is executed by a processor, the steps of the dual-band imaging method of the integrated image sensor and the liquid crystal shutter as described in any one of the above are implemented.

[0015] As described above, a dual-band imaging method, system, and medium integrating an image sensor and a liquid crystal shutter provided by an embodiment of the present application obtain the type of a CMOS image sensor and configure the operating parameters of the CMOS image sensor; initialize the state of the liquid crystal shutter device, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, obtain the operating frequency of the liquid crystal shutter device, and determine whether the operating frequency reaches the switching frequency; if the switching frequency is not reached, control the liquid crystal shutter device to enter the first imaging mode, allow the light of the first band to pass through and reach the CMOS image sensor based on the first imaging mode, and obtain the first-band image; if the switching frequency is reached, control the liquid crystal shutter device to perform filter switching, enter the second imaging mode, allow the light of the second band to pass through and reach the CMOS image sensor based on the first imaging mode, and obtain the second-band image; perform preprocessing on the first-band image and the second-band image to obtain a preprocessed image, and fuse and process the preprocessed image to generate a dual-band image; dynamically switch different filters through the liquid crystal shutter device to obtain two images of different bands, and then perform image enhancement and fusion processing to achieve dual-band imaging and improve the imaging effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 It is a flowchart of the dual-band imaging method of the integrated image sensor and liquid crystal shutter provided by the embodiment of the present application; Figure 2 It is a flowchart of the working parameter configuration of the CMOS image sensor of the dual-band imaging method of the integrated image sensor and liquid crystal shutter provided by the embodiment of the present application; Figure 3 It is a flowchart of the method for setting the operating parameters of the liquid crystal shutter device of the dual-band imaging method of the integrated image sensor and liquid crystal shutter provided by the embodiment of the present application; Figure 4 It is a schematic diagram of the working principle of the first band of the dual-band imaging system of the integrated image sensor and liquid crystal shutter provided by the embodiment of the present application; Figure 5 It is a schematic diagram of the working principle of the second band of the dual-band imaging system of the integrated image sensor and liquid crystal shutter provided by the embodiment of the present application; Figure 6This is a timing diagram for the synchronous operation of the rolling shutter CIS and the LC shutter in the dual-band imaging system of the integrated image sensor and the liquid crystal shutter provided by the embodiments of the present application. Detailed implementation manners

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but merely represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts fall within the protection scope of the present application.

[0019] It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first" and "second" are only used for differential description and cannot be understood as indicating or implying relative importance.

[0020] Please refer to Figure 1 , Figure 1 which is a flowchart of a dual-band imaging method of an integrated image sensor and a liquid crystal shutter in some embodiments of the present application. The dual-band imaging method of the integrated image sensor and the liquid crystal shutter is used in a terminal device. The dual-band imaging method of the integrated image sensor and the liquid crystal shutter includes the following steps: S101, obtaining the type of the CMOS image sensor and configuring the working parameters of the CMOS image sensor; S102, initializing the state of the liquid crystal shutter device, setting the operating parameters of the liquid crystal shutter device based on the working parameters of the CMOS image sensor, obtaining the operating frequency of the liquid crystal shutter device, and determining whether the operating frequency reaches the switching frequency; S103, if the switching frequency is not reached, controlling the liquid crystal shutter device to enter the first imaging mode, allowing the light of the first band to pass through and reach the CMOS image sensor based on the first imaging mode, and obtaining the first-band image; S104, if the switching frequency is reached, controlling the liquid crystal shutter device to perform filter switching, entering the second imaging mode, allowing the light of the second band to pass through and reach the CMOS image sensor based on the first imaging mode, and obtaining the second-band image; S105. Preprocess based on the first-band image and the second-band image to obtain a preprocessed image, and fuse the preprocessed image to generate a dual-band image.

[0021] It should be noted that by configuring the working parameters of the CMOS image sensor, the CMOS image sensor is always in an efficient working state. At the same time, by analyzing the operating frequency of the liquid crystal shutter device, the imaging mode is dynamically switched to achieve dual-band imaging, thereby obtaining a dual-band image.

[0022] Please refer to Figure 2 , Figure 2 is a flowchart for configuring the working parameters of a CMOS image sensor in a dual-band imaging method of an integrated image sensor and a liquid crystal shutter in some embodiments of the present application. According to the embodiments of the present invention, the type of the CMOS image sensor is obtained, and the working parameters of the CMOS image sensor are configured, specifically including: S201. Obtain the user manual and technical specification of the CMOS image sensor, and analyze the characteristics and performance parameter information of the CMOS image sensor based on the user manual and technical specification; S202. Obtain the identification information of the CMOS image sensor, and analyze the marked model and manufacturer based on the identification information; S203. Based on the characteristics and performance parameter information of the CMOS image sensor, the marked model and manufacturer, obtain the type of the CMOS image sensor; S204. Configure the working parameters based on the type of the CMOS image sensor, and the working parameters include exposure time, gain parameter, and clock parameter.

[0023] It should be noted that the working parameters of the CMOS image sensor are set through a control circuit. For example, a suitable exposure time is set to ensure clear images under different lighting conditions; the gain parameter is adjusted according to the imaging requirements to improve the signal intensity of the image; parameters such as pixel clock are configured to ensure the acquisition speed and quality of the image. At the same time, the liquid crystal shutter device is initialized, and its switching frequency, working voltage, and other parameters are set to enable it to stably switch different-band filters.

[0024] Please refer to Figure 3 , Figure 3 is a flowchart for setting the operating parameters of a liquid crystal shutter device in a dual-band imaging method of an integrated image sensor and a liquid crystal shutter in some embodiments of the present application. According to the embodiments of the present invention, the state of the liquid crystal shutter device is initialized, and the operating parameters of the liquid crystal shutter device are set based on the working parameters of the CMOS image sensor, specifically including: S301. Obtain the power supply state information of the liquid crystal shutter device, and analyze the operating state information of the liquid crystal shutter device based on the power supply state information; S302. Compare the operating status information with the set initial status information to obtain the status deviation rate. S303. Determine whether the status deviation rate is greater than or equal to the set status deviation rate threshold. S304. If so, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor. The operating parameters include the switching frequency and the operating voltage. S305. If not, perform a reset operation on the liquid crystal shutter device, restore the current state of the liquid crystal shutter device to the initial state, and set the operating parameters of the liquid crystal shutter device.

[0025] It should be noted that the two polarization filters of the liquid crystal (LC) shutter device are designed to have a high extinction ratio in the visible light band and a very low extinction ratio in the near-infrared (NIR) band. The polarizer filters the incident light. Then, if the polarization rotator is activated, the polarization direction of the light passing through the polarization rotator will rotate; otherwise, the polarization direction will not rotate, and the polarization directions of the polarizer and the analyzer are perpendicular to each other. Under this setting, when the polarization direction of the light emerging from the polarization rotator matches the polarization direction of the analyzer, the shutter state is regarded as "open"; when it is perpendicular to the polarization direction of the analyzer, the shutter state is regarded as "closed". Since the light is blocked by the shutter when the polarization rotator is not activated, the shutter is in the "normally closed" state.

[0026] According to an embodiment of the present invention, controlling the liquid crystal shutter device to enter the first imaging mode, and based on the first imaging mode, allowing the light of the first band to pass through and reach the CMOS image sensor to obtain the first band image, specifically includes: Obtain the current state information of the liquid crystal shutter device, and compare the current state information of the liquid crystal shutter device with the condition information of the first imaging mode. Determine whether the current state information of the liquid crystal shutter device meets the condition information of the first imaging mode. If it is satisfied, switch to the first band based on the first imaging mode. The first band is the visible light band. Project the visible light onto the CMOS image sensor based on the first band to obtain the first band image. The first band image is a visible light band image. If it is not satisfied, adjust the current state parameters of the liquid crystal shutter device to control the liquid crystal shutter device to enter the first imaging mode.

[0027] It should be noted that during the light transmission process, the liquid crystal shutter device quickly switches the filter according to the control signal, so that the light of a specific band can pass through and reach the CMOS image sensor. For example, when it is necessary to collect an image in the visible light band, the liquid crystal shutter device switches to the visible light filter to allow the visible light to pass through; when it is necessary to collect an image in another band (such as near-infrared), it switches to the near-infrared filter again.

[0028] According to an embodiment of the present invention, controlling a liquid crystal shutter device to perform filter switching, entering a second imaging mode, allowing light in a second band to pass through and reach a CMOS image sensor based on a first imaging mode, and obtaining a second-band image, specifically including: Obtaining an imaging mode switching instruction, switching a first band to a second band based on the imaging mode switching instruction, where the second band is a near-infrared light band, and controlling the liquid crystal shutter device to enter the second imaging mode; Setting a light passing band based on the second imaging mode, obtaining an optical signal, and analyzing the band where the optical signal is located based on the optical signal; Processing the optical signal based on the set light passing band, allowing the optical signal in the set light passing band to pass through, and converting the optical signal into an electrical signal; Transmitting the electrical signal to the CMOS image sensor to obtain a second-band image.

[0029] It should be noted that in the rolling shutter operation, the data reading of each row is performed in sequence. When a reset signal is sent to the pixels of the first row (row 1), a new frame starts. Immediately afterwards, they start integrating for a period of time, called the "integration time" Tint. Then, the reset signal is sent to the next row in a continuous order, where the interval between rows cannot be shorter than the time required to read data from a single row, and the pixels of each row start integrating and resetting after they are sent the reset signal. After Tint ends, the pixel signals of each row are immediately read. The end of Tint of the last row (row N) marks the end of the frame. The liquid crystal shutter can be synchronized with the global shutter image sensor and is simpler because in the global shutter operation, the Tint of all rows starts and ends simultaneously.

[0030] According to an embodiment of the present invention, performing preprocessing based on the first-band image and the second-band image to obtain a preprocessed image, and performing fusion processing on the preprocessed image to generate a dual-band image, specifically including: Obtaining the first-band image, extracting image features, analyzing the image noise features based on the image features, removing the noise features in the image based on the mean filtering or median filtering algorithm, and obtaining a first optimized image; Analyzing the pixel information of the first optimized image, analyzing the image bad pixels based on the pixel information, and correcting the image bad pixels to obtain a first preprocessed image; Obtaining the second-band image, extracting image features, analyzing the image noise features based on the image features, removing the noise features in the image based on the mean filtering or median filtering algorithm, and obtaining a second optimized image; Analyzing the pixel information of the second optimized image, analyzing the image bad pixels based on the pixel information, and correcting the image bad pixels to obtain a second preprocessed image; Based on the pixel information of the first preprocessed image and the pixel information of the second preprocessed image, weighted fusion processing is performed to obtain a dual-band image.

[0031] It should be noted that preprocessing the digitized dual-band image data includes denoising processing, using algorithms such as mean filtering and median filtering to remove noise in the image; performing white balance adjustment to make the color restoration of the image accurate; and bad pixel correction can also be performed to repair possible bad pixels in the CMOS image sensor. If it is necessary to fuse the image information of two bands to obtain richer image content, an image fusion algorithm can be used. For example, a pixel-level fusion algorithm that performs operations such as weighted averaging on the corresponding pixel values of the two-band images; or a feature-level fusion algorithm that first extracts features such as edges and corners of the two-band images, then fuses these features, and then reconstructs the image based on the fused features.

[0032] In a second aspect, an embodiment of the present application provides a dual-band imaging system integrating an image sensor and a liquid crystal shutter. The system includes: a memory and a processor. The memory includes a program for the dual-band imaging method of the integrated image sensor and the liquid crystal shutter. When the program for the dual-band imaging method of the integrated image sensor and the liquid crystal shutter is executed by the processor, the following steps are implemented: Obtain the type of the CMOS image sensor and configure the operating parameters of the CMOS image sensor; Initialize the state of the liquid crystal shutter device, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, obtain the operating frequency of the liquid crystal shutter device, and determine whether the operating frequency reaches the switching frequency; If the switching frequency is not reached, control the liquid crystal shutter device to enter the first imaging mode, and based on the first imaging mode, make the light of the first band pass through and reach the CMOS image sensor to obtain a first-band image; If the switching frequency is reached, control the liquid crystal shutter device to perform filter switching, enter the second imaging mode, and based on the first imaging mode, make the light of the second band pass through and reach the CMOS image sensor to obtain a second-band image; Preprocess the first-band image and the second-band image to obtain preprocessed images, and perform fusion processing on the preprocessed images to generate a dual-band image.

[0033] It should be noted that by configuring the operating parameters of the CMOS image sensor, the CMOS image sensor is always in an efficient working state. At the same time, by analyzing the operating frequency of the liquid crystal shutter device, the imaging mode is dynamically switched to achieve dual-band imaging, thereby obtaining a dual-band image.

[0034] In a preferred embodiment of the present invention, as Figure 4As shown, the liquid crystal shutter consists of a liquid crystal film placed between two linear polarizing filters (i.e., a polarizer and an analyzer). When an electric field is applied, the liquid crystal film changes the polarization direction of the light passing through it. Therefore, in the invention, it is called a "polarization rotator". The extinction ratio of the polarizing filter is wavelength-dependent. If the band with a high extinction ratio covers a part of the wavelength band to which the detector is sensitive to light, this characteristic can be utilized to implement a dual-band imaging system. The transmission band of the camera filter should match the characteristics of the polarizing filter, and the normal operation of the system requires synchronization between the image sensor and the shutter activation.

[0035] The polarizer filters the incident light. When the polarization rotator is activated, the light passes through the analyzer perpendicular to the polarizer, and the shutter is in the open state. When the polarization rotator is not activated, the light is blocked by the analyzer, and the shutter is in the closed state.

[0036] As Figure 5 shown, in the second band, the extinction ratio of the polarizing filter is very low, i.e., about 1:1. Therefore, unpolarized NIR light passes through them without any polarization filtering. Thus, regardless of whether the polarization rotator is activated or not, the light flux after the analyzer does not change. Generally speaking, if optical losses and other minor defects are ignored for simplicity, then under broadband illumination with energy in the visible (VIS) and near-infrared (NIR) bands, the shutter operation can be described as follows: In the open state, the light emerging from the shutter includes polarized VIS light and unpolarized NIR light; in the closed state, the light emerging from the shutter only includes unpolarized NIR light. Therefore, if the illumination conditions remain unchanged, the VIS component of the shutter output light in the open state can be obtained by subtracting the closed state signal from the open state signal. The present invention utilizes this characteristic to implement a dual-band imaging system. Figure 5 Shown in

[0037] As Figure 6 shown, before the integration of the first row starts, the shutter state changes and remains unchanged until the integration time (Tint) of the last row in the same frame ends. The transition times Tf and Tr of the LC shutter should be considered, as Figure 6 shown in Frame 2 in

[0038] The input signal of the microcontroller comes from an image sensor and is used to indicate the start of a new frame, and its output signal controls a liquid crystal (LC) shutter. The microcontroller is pre-programmed to alternately switch the shutter state for each frame. To prevent image artifacts caused by the rise and fall times (Tr and Tf respectively) of the liquid crystal shutter, we calculated the timing to ensure that the shutter state is changed within a short period of time before the start of a new frame and within a short period of time after the end of the frame. In this application, Tr and Tf of the liquid crystal shutter are approximately 150 - 200 microseconds. Two consecutive frames of images need to be captured to generate a set of color and near-infrared images, where one frame is captured with the shutter open and the other frame is captured with the shutter closed. Therefore, the effective frame rate of the integrated device is half of the frame rate of the image sensor.

[0039] According to an embodiment of the present invention, obtain the type of the CMOS image sensor and configure the operating parameters of the CMOS image sensor, specifically including: Obtain the user manual and technical specification of the CMOS image sensor, and analyze the characteristics and performance parameter information of the CMOS image sensor based on the user manual and technical specification; Obtain the identification information of the CMOS image sensor, and analyze the marked model and manufacturer based on the identification information; Based on the characteristics and performance parameter information of the CMOS image sensor, the marked model and the manufacturer, obtain the type of the CMOS image sensor; Configure the operating parameters based on the type of the CMOS image sensor, and the operating parameters include exposure time, gain parameter, and clock parameter.

[0040] It should be noted that the operating parameters of the CMOS image sensor are set through a control circuit. For example, a suitable exposure time is set to ensure clear images under different lighting conditions; the gain parameter is adjusted according to the imaging requirements to enhance the signal intensity of the image; parameters such as pixel clock are configured to ensure the acquisition speed and quality of the image. At the same time, the liquid crystal shutter device is initialized, and parameters such as its switching frequency and operating voltage are set so that it can stably switch different band filters.

[0041] According to an embodiment of the present invention, initialize the state of the liquid crystal shutter device, and set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, specifically including: Obtain the power supply state information of the liquid crystal shutter device, and analyze the operating state information of the liquid crystal shutter device based on the power supply state information; Compare the operating state information with the set initial state information to obtain the state deviation rate; Judge whether the state deviation rate is greater than or equal to the set state deviation rate threshold; If so, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor. The operating parameters include the switching frequency and the operating voltage. If not, perform a reset operation on the liquid crystal shutter device, restore the current state of the liquid crystal shutter device to the initial state, and set the operating parameters of the liquid crystal shutter device.

[0042] It should be noted that the two polarization filters of the liquid crystal (LC) shutter device are designed to have a high extinction ratio in the visible light band and a very low extinction ratio in the near-infrared (NIR) band. The polarizer filters the incident light. Then, if the polarization rotator is activated, the polarization direction of the light passing through the polarization rotator will rotate; otherwise, the polarization direction will not rotate, and the polarization directions of the polarizer and the analyzer are perpendicular to each other. In this setting, when the polarization direction of the light emerging from the polarization rotator matches the polarization direction of the analyzer, the shutter state is regarded as "open"; when it is perpendicular to the polarization direction of the analyzer, the shutter state is regarded as "closed". Since the light is blocked by the shutter when the polarization rotator is not activated, the shutter is in the "normally closed" state.

[0043] According to an embodiment of the present invention, control the liquid crystal shutter device to enter the first imaging mode, and based on the first imaging mode, make the light in the first band pass through and reach the CMOS image sensor to obtain an image of the first band, specifically including: Obtain the current state information of the liquid crystal shutter device, and compare the current state information of the liquid crystal shutter device with the condition information of the first imaging mode; Judge whether the current state information of the liquid crystal shutter device meets the condition information of the first imaging mode; If it meets, switch to the first band based on the first imaging mode. The first band is the visible light band. Project the visible light onto the CMOS image sensor based on the first band to obtain an image of the first band. The image of the first band is an image in the visible light band; If it does not meet, adjust the current state parameters of the liquid crystal shutter device and control the liquid crystal shutter device to enter the first imaging mode.

[0044] It should be noted that during the light transmission process, the liquid crystal shutter device quickly switches the filter according to the control signal, so that the light in a specific band can pass through and reach the CMOS image sensor. For example, when an image in the visible light band needs to be collected, the liquid crystal shutter device switches to the visible light filter to allow the visible light to pass through; when an image in another band (such as near-infrared) needs to be collected, it switches to the near-infrared filter again.

[0045] According to an embodiment of the present invention, controlling a liquid crystal shutter device to perform filter switching and enter a second imaging mode, allowing light in a second band to pass through and reach a CMOS image sensor based on a first imaging mode, and obtaining a second band image, specifically including: Obtaining an imaging mode switching instruction, switching from a first band to a second band based on the imaging mode switching instruction, where the second band is a near-infrared light band, and controlling the liquid crystal shutter device to enter the second imaging mode; Setting a light passing band based on the second imaging mode, obtaining an optical signal, and analyzing the band where the optical signal is located based on the optical signal; Processing the optical signal based on the set light passing band, allowing the optical signal in the set light passing band to pass through, and converting the optical signal into an electrical signal; Transmitting the electrical signal to the CMOS image sensor to obtain a second band image.

[0046] It should be noted that in the rolling shutter operation, the data reading of each row is performed sequentially. When a reset signal is sent to the pixels of the first row (row 1), a new frame starts. Immediately afterwards, they start integrating for a period of time, called the "integration time" Tint. Then, the reset signal is sent to the next row in a continuous order, where the interval between rows cannot be shorter than the time required to read data from a single row, and the pixels of each row start integrating and resetting after they are sent the reset signal. After Tint ends, the pixel signals of each row are immediately read. The end of Tint of the last row (row N) marks the end of the frame. The liquid crystal shutter can be synchronized with the global shutter image sensor and is simpler because in the global shutter operation, the Tint of all rows starts and ends simultaneously.

[0047] According to an embodiment of the present invention, preprocessing is performed based on the first band image and the second band image to obtain a preprocessed image, and the preprocessed image is fused to generate a dual-band image, specifically including: Obtaining the first band image, extracting image features, analyzing the image noise features based on the image features, removing the noise features in the image based on the mean filtering or median filtering algorithm, and obtaining a first optimized image; Analyzing the pixel information of the first optimized image, analyzing the image bad pixels based on the pixel information, and correcting the image bad pixels to obtain a first preprocessed image; Obtaining the second band image, extracting image features, analyzing the image noise features based on the image features, removing the noise features in the image based on the mean filtering or median filtering algorithm, and obtaining a second optimized image; Analyzing the pixel information of the second optimized image, analyzing the image bad pixels based on the pixel information, and correcting the image bad pixels to obtain a second preprocessed image; Perform weighted fusion processing based on the pixel information of the first preprocessed image and the pixel information of the second preprocessed image to obtain a dual-band image.

[0048] It should be noted that preprocessing the digitized dual-band image data includes denoising processing, using algorithms such as mean filtering and median filtering to remove noise in the image; performing white balance adjustment to accurately restore the color of the image; and it is also possible to perform dead pixel correction to repair possible dead pixels in the CMOS image sensor. If it is necessary to fuse the image information of two bands to obtain richer image content, an image fusion algorithm can be used. For example, a pixel-level fusion algorithm that performs operations such as weighted averaging on the corresponding pixel values of the two-band images; or a feature-level fusion algorithm that first extracts features such as edges and corners of the two-band images, then fuses these features, and then reconstructs the image based on the fused features.

[0049] The third aspect of the present invention provides a computer-readable storage medium, which includes a program for a dual-band imaging method integrating an image sensor and a liquid crystal shutter. When the program for the dual-band imaging method integrating an image sensor and a liquid crystal shutter is executed by a processor, the steps of the dual-band imaging method of integrating an image sensor and a liquid crystal shutter as described in any one of the above are implemented.

[0050] A dual-band imaging method, system and medium integrating an image sensor and a liquid crystal shutter disclosed by the present invention, by obtaining the type of the CMOS image sensor, configuring the working parameters of the CMOS image sensor; initializing the state of the liquid crystal shutter device, setting the operating parameters of the liquid crystal shutter device based on the working parameters of the CMOS image sensor, obtaining the operating frequency of the liquid crystal shutter device, and determining whether the operating frequency reaches the switching frequency; if the switching frequency is not reached, controlling the liquid crystal shutter device to enter the first imaging mode, allowing the light of the first band to pass through and reach the CMOS image sensor based on the first imaging mode to obtain a first-band image; if the switching frequency is reached, controlling the liquid crystal shutter device to perform filter switching and enter the second imaging mode, allowing the light of the second band to pass through and reach the CMOS image sensor based on the first imaging mode to obtain a second-band image; performing preprocessing on the first-band image and the second-band image to obtain preprocessed images, and fusing the preprocessed images to generate a dual-band image; dynamically switching different filters through the liquid crystal shutter device to obtain two images of different bands, and then performing image enhancement and fusion processing to achieve dual-band imaging and improve the imaging effect.

[0051] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed with each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0052] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units; they can be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0053] In addition, each functional unit in the embodiments of the present invention can be all integrated in a processing unit, or each unit can be separately used as a unit, or two or more units can be integrated in one unit; the above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0054] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: removable storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other various media that can store program codes.

[0055] Or, if the above-mentioned integrated units of the present invention are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. And the foregoing storage medium includes: removable storage devices, ROM, RAM, magnetic disks or optical disks and other various media that can store program codes.

Claims

1. A dual-band imaging method integrating an image sensor and a liquid crystal shutter, characterized in that, Including: Obtain the type of the CMOS image sensor and configure the operating parameters of the CMOS image sensor; Initialize the state of the liquid crystal shutter device, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, obtain the operating frequency of the liquid crystal shutter device, and determine whether the operating frequency reaches the switching frequency; If the switching frequency is not reached, control the liquid crystal shutter device to enter the first imaging mode, and based on the first imaging mode, make the light of the first band pass through and reach the CMOS image sensor to obtain the first band image; If the switching frequency is reached, control the liquid crystal shutter device to perform filter switching, enter the second imaging mode, and based on the first imaging mode, make the light of the second band pass through and reach the CMOS image sensor to obtain the second band image; Perform preprocessing on the first band image and the second band image to obtain a preprocessed image, and fuse the preprocessed image to generate a dual-band image.

2. The dual-band imaging method integrating an image sensor and a liquid crystal shutter according to claim 1, characterized in that, Obtain the type of the CMOS image sensor and configure the operating parameters of the CMOS image sensor, specifically including: Obtain the user manual and technical specification of the CMOS image sensor, and analyze the characteristics and performance parameter information of the CMOS image sensor based on the user manual and technical specification; Obtain the identification information of the CMOS image sensor, and analyze the marked model and manufacturer based on the identification information; Based on the characteristics and performance parameter information of the CMOS image sensor, the marked model and the manufacturer, obtain the type of the CMOS image sensor; Configure the operating parameters based on the type of the CMOS image sensor, and the operating parameters include exposure time, gain parameter and clock parameter.

3. The dual-band imaging method integrating an image sensor and a liquid crystal shutter according to claim 2, characterized in that, Initialize the state of the liquid crystal shutter device, and set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, specifically including: Obtain the power supply state information of the liquid crystal shutter device, and analyze the operating state information of the liquid crystal shutter device based on the power supply state information; Compare the operating state information with the set initial state information to obtain a state deviation rate; Determine whether the state deviation rate is greater than or equal to the set state deviation rate threshold; If so, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, and the operating parameters include switching frequency and operating voltage; If not, perform a reset operation on the liquid crystal shutter device, restore the current state of the liquid crystal shutter device to the initial state, and set the operating parameters of the liquid crystal shutter device.

4. The dual-band imaging method integrating an image sensor and a liquid crystal shutter according to claim 3, characterized in that, Control the liquid crystal shutter device to enter the first imaging mode, and based on the first imaging mode, make the light of the first band pass through and reach the CMOS image sensor to obtain the first band image, specifically including: Obtain the current state information of the liquid crystal shutter device, and compare the current state information of the liquid crystal shutter device with the condition information of the first imaging mode; Determine whether the current state information of the liquid crystal shutter device meets the condition information of the first imaging mode; If it meets, switch to the first band based on the first imaging mode. The first band is the visible light band. Project the visible light onto the CMOS image sensor based on the first band to obtain the first band image, and the first band image is a visible light band image; If not satisfied, adjust the current state parameters of the liquid crystal shutter device and control the liquid crystal shutter device to enter the first imaging mode.

5. The dual-band imaging method integrating an image sensor and a liquid crystal shutter according to claim 4, characterized in that, Control the liquid crystal shutter device to perform filter switching and enter the second imaging mode. Based on the first imaging mode, make the light in the second band pass through and reach the CMOS image sensor to obtain a second-band image, specifically including: Obtain an imaging mode switching instruction, switch the first band to the second band based on the imaging mode switching instruction. The second band is the near-infrared light band, and control the liquid crystal shutter device to enter the second imaging mode; Set the light passing band based on the second imaging mode, obtain the optical signal, and analyze the band where the optical signal is located based on the optical signal; Process the optical signal based on the set light passing band, allow the optical signal in the set light passing band to pass through, and convert the optical signal into an electrical signal; Transmit the electrical signal to the CMOS image sensor to obtain a second-band image.

6. The dual-band imaging method integrating an image sensor and a liquid crystal shutter according to claim 5, characterized in that, Perform preprocessing on the first-band image and the second-band image to obtain a preprocessed image, and perform fusion processing on the preprocessed image to generate a dual-band image, specifically including: Obtain the first-band image, extract the image features, analyze the image noise features based on the image features, remove the noise features in the image based on the mean filtering or median filtering algorithm, and obtain the first optimized image; Analyze the pixel information of the first optimized image, analyze the image bad pixels based on the pixel information, and correct the image bad pixels to obtain the first preprocessed image; Obtain the second-band image, extract the image features, analyze the image noise features based on the image features, remove the noise features in the image based on the mean filtering or median filtering algorithm, and obtain the second optimized image; Analyze the pixel information of the second optimized image, analyze the image bad pixels based on the pixel information, and correct the image bad pixels to obtain the second preprocessed image; Perform weighted fusion processing on the pixel information of the first preprocessed image and the pixel information of the second preprocessed image to obtain a dual-band image.

7. A dual-band imaging system integrating an image sensor and a liquid crystal shutter, characterized in that, The system includes: a memory and a processor. The memory includes a program for the dual-band imaging method of the integrated image sensor and the liquid crystal shutter. When the program for the dual-band imaging method of the integrated image sensor and the liquid crystal shutter is executed by the processor, the following steps are implemented: Obtain the type of the CMOS image sensor and configure the working parameters of the CMOS image sensor; Initialize the state of the liquid crystal shutter device, set the operating parameters of the liquid crystal shutter device based on the working parameters of the CMOS image sensor, obtain the operating frequency of the liquid crystal shutter device, and determine whether the operating frequency reaches the switching frequency; If the switching frequency is not reached, control the liquid crystal shutter device to enter the first imaging mode. Based on the first imaging mode, make the light in the first band pass through and reach the CMOS image sensor to obtain a first-band image; If the switching frequency is reached, control the liquid crystal shutter device to perform filter switching and enter the second imaging mode. Based on the first imaging mode, make the light in the second band pass through and reach the CMOS image sensor to obtain a second-band image; Perform preprocessing on the first-band image and the second-band image to obtain a preprocessed image, and perform fusion processing on the preprocessed image to generate a dual-band image.

8. The dual-band imaging system integrating an image sensor and a liquid crystal shutter according to claim 7, characterized in that, Obtain the type of the CMOS image sensor and configure the operating parameters of the CMOS image sensor, specifically including: Obtain the user manual and technical specification of the CMOS image sensor, and analyze the characteristics and performance parameter information of the CMOS image sensor based on the user manual and technical specification; Obtain the identification information of the CMOS image sensor, and analyze the labeled model and manufacturer based on the identification information; Based on the characteristics and performance parameter information of the CMOS image sensor, the labeled model and the manufacturer, obtain the type of the CMOS image sensor; Configure the operating parameters based on the type of the CMOS image sensor, and the operating parameters include exposure time, gain parameter and clock parameter.

9. The dual-band imaging system integrating an image sensor and a liquid crystal shutter according to claim 8, characterized in that, Initialize the state of the liquid crystal shutter device, and set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, specifically including: Obtain the power supply status information of the liquid crystal shutter device, and analyze the operating status information of the liquid crystal shutter device based on the power supply status information; Compare the operating status information with the set initial status information to obtain the status deviation rate; Judge whether the status deviation rate is greater than or equal to the set status deviation rate threshold; If so, set the operating parameters of the liquid crystal shutter device based on the operating parameters of the CMOS image sensor, and the operating parameters include switching frequency and operating voltage; If not, perform a reset operation on the liquid crystal shutter device, restore the current state of the liquid crystal shutter device to the initial state, and set the operating parameters of the liquid crystal shutter device.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a dual-band imaging method program for an integrated image sensor and a liquid crystal shutter. When the dual-band imaging method program for the integrated image sensor and the liquid crystal shutter is executed by a processor, the steps of the dual-band imaging method for the integrated image sensor and the liquid crystal shutter as described in any one of claims 1 to 6 are implemented.

Images