Real-time high dynamic range image processing method and system based on sCMOS

Abstract

The invention provides a real-time high-dynamic-range image processing method and system based on sCMOS, and relates to the technical field of image processing.The method comprises the steps that a high-gain image and a low-gain image are collected at the same time through an sCMOS sensor and used for capturing dark part details and bright part details respectively; transmitting the high-gain and low-gain image data from the sCMOS sensor to an image processing module; storing the collected high-gain and low-gain image data in a memory, and carrying out preliminary preprocessing; carrying out linear segmentation on gray values of the high-gain image and the low-gain image, and carrying out gray stretching to enhance the image contrast; analyzing unsaturated pixel points in the high-gain image based on gray characteristics of the high-gain image and the low-gain image, and synthesizing the high-gain image and the low-gain image into an HDR image in combination with a fixed gain ratio k and an offset b; and the processed HDR image data is transmitted to an upper computer, so that an HDR image can be synthesized efficiently.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a real-time high dynamic range (HDR) image processing method and system based on sCMOS. Background Art

[0002] Traditional image sensors have limited dynamic range and are unable to capture both light and dark details in an image. To overcome this problem, HDR imaging technology has been proposed and widely studied. Existing HDR imaging methods usually require multiple exposures or multi-frame image synthesis, which is computationally complex and prone to ghosting. sCMOS sensors have high and low gain channels that can output two images of different brightness at the same time, which can reduce the amount of calculation while avoiding ghosting problems. However, existing hardware-based HDR image synthesis methods still have problems with large computational complexity and long computational time, and fast synthesis methods also have interval segmentation problems. Summary of the invention

[0003] In view of this, in order to solve the above problems, the present invention proposes a real-time high dynamic range image processing method and system based on sCMOS, which can efficiently synthesize HDR images, maintain image details and contrast, and keep consistent with the output frame rate of the high-speed imaging system.

[0004] The present invention is implemented by the following technical solutions:

[0005] In a first aspect, the present invention provides a real-time high dynamic range image processing method based on sCMOS, the method comprising the following steps:

[0006] The sCMOS sensor is used to simultaneously collect high-gain and low-gain images to capture dark and bright details respectively;

[0007] Transmitting high-gain and low-gain image data from the sCMOS sensor to the image processing module;

[0008] The collected high-gain and low-gain image data are stored in a memory and pre-processed;

[0009] Linearly segment the grayscale values ​​of high-gain and low-gain images, and perform grayscale stretching to enhance image contrast;

[0010] Based on the grayscale characteristics of high-gain and low-gain images, the unsaturated pixels in the high-gain image are analyzed, and the high-gain and low-gain images are synthesized into an HDR image by combining a fixed gain ratio k and an offset b.

[0011] The signal-to-noise ratio of the image is enhanced by the 2×2 average binning technology, and the pipeline is used to optimize floating-point operations while data storage and processing are carried out in parallel, and the processed HDR image data is transmitted to the host computer.

[0012] As a further solution of the present invention, the high-gain and low-gain image acquisition of the sCMOS sensor adopts a synchronous exposure mode.

[0013] As a further solution of the present invention, the image data acquisition module is connected to the image processing module via a high-speed data transmission interface.

[0014] As a further solution of the present invention, the gray value segmentation and stretching process includes histogram equalization and contrast enhancement processing of the image, performing image histogram statistics, and determining the gain and exposure time settings. If the gray range of a bit is evenly divided into b gray levels, where a and b are both even numbers, then the number of dark pixels (gray value 0) is defined (S low ), saturated pixel (grayscale value is 2 a -1) (S high ), calculate the number of pixels located in the upper and lower parts of the histogram of the two images as:

[0015]

[0016] Where P is the percentage of pixels in the histogram, s represents low gain, l represents high gain, H represents the image histogram, h represents the grayscale level (1-b), and N represents the number of all pixels. t,s ≈80%, P t,l ≈80%, then the gain and exposure time are adjusted; P t，s Indicates the percentage of pixels in the low gain histogram; H t，s (h) represents the number of pixels of gray level (1-b) in the low gain histogram; P t，l Indicates the percentage of pixels in the high gain histogram; H t，l (h) represents the number of pixels at gray level (1-b) in the high gain histogram.

[0017] As a further solution of the present invention, the signal-to-noise ratio enhancement is achieved by an average binning technique, including averaging the grayscale values ​​of a 2×2 pixel area. When the pixels are not saturated, the grayscale value relationship between the corresponding points of the high-gain image and the low-gain image is analyzed as follows:

[0018] H x，y =k*L x，y +bH x，y <2 a -1

[0019] In the formula, H x,y Represents the gray value of high-gain image, L x,y represents the gray value of the low-gain image, k and b represent coefficients. Each set of gain settings will result in a fixed set of k and b. x and y represent the pixel coordinates.

[0020] As a further solution of the present invention, grayscale stretching is performed on high-gain and low-gain images. When synthesizing an HDR image according to the grayscale value relationship, the idea of ​​directly synthesizing the HDR image with useful information in the high- and low-gain images is combined with the high- and low-gain grayscale characteristics, when the following formula is satisfied:

[0021]

[0022] In the formula, a L Indicates the segmentation point of the high-gain image histogram; a H Indicates the cut-off points of the high-gain image histogram.

[0023] Use linear stretching to x,y ≤a L and L x,y ≥a L The area is stretched so that the grayscale range of the fused image is distributed in the entire [0, S], then:

[0024]

[0025] Where H′ and L′ represent the images of H and L after a certain grayscale linear transformation.

[0026] The synthesized image G HDR It is expressed as:

[0027]

[0028] As a further solution of the present invention, the image display and storage module includes a communication interface with a host computer via an FPGA to transmit the HDR image data to the host computer for display and storage.

[0029] As a further solution of the present invention, the real-time high dynamic range image processing method also coordinates and controls the parameter configuration of the sensor, the timing management of image data acquisition, the control of the processing flow and the management of data transmission through a system control module. The system control module provides a user interface for the user to set and adjust the system parameters.

[0030] In a second aspect, the present invention further provides a real-time high dynamic range image processing system based on sCMOS, which is used to perform a real-time high dynamic range image processing method based on sCMOS. The dual-gain HDR synthesis system includes:

[0031] Dual-gain image acquisition module, used to use the sCMOS sensor to simultaneously acquire high-gain and low-gain images, respectively used to capture dark and bright details;

[0032] An image data acquisition module, used for transmitting high-gain and low-gain image data from the sCMOS sensor to an image processing module;

[0033] A data cache processing module is used to store the collected high-gain and low-gain image data into a memory and perform preliminary preprocessing;

[0034] Grayscale stretching module, used to linearly segment the grayscale values ​​of high-gain and low-gain images, and perform grayscale stretching to enhance image contrast;

[0035] An HDR image synthesis module is used to analyze unsaturated pixels in the high-gain image based on the grayscale characteristics of the high-gain and low-gain images, and to synthesize the high-gain and low-gain images into an HDR image by combining a fixed gain ratio k and an offset b;

[0036] The signal-to-noise ratio enhancement module is used to enhance the signal-to-noise ratio of the image through 2×2 average binning;

[0037] A real-time optimization processing module for optimizing floating-point operations while data storage and processing are performed in parallel using a pipeline;

[0038] The image display and storage module is used to transmit the processed HDR image data to the host computer to realize real-time display and storage of the image.

[0039] As a further solution of the present invention, the sCMOS-based real-time high dynamic range image processing system also includes a system control module for coordinating and controlling sensor parameter configuration, timing management of image data acquisition, control of processing flow, and management of data transmission.

[0040] The present invention also includes a computer device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the sCMOS-based real-time high dynamic range image processing method.

[0041] The present invention also includes a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the real-time high dynamic range image processing method based on sCMOS.

[0042] Compared with the prior art, the real-time high dynamic range image processing method and system based on sCMOS provided by the present invention has the following beneficial effects:

[0043] 1. Extended dynamic range: The present invention uses the sCMOS sensor to simultaneously capture high-gain and low-gain images, effectively capturing the dark and bright details in the scene, significantly extending the dynamic range of the image and avoiding the problem of detail loss in the traditional single-gain mode.

[0044] 2. Improved image quality: The present invention uses gray value segmentation and stretching modules to perform image preprocessing, and through histogram equalization and contrast enhancement technology, greatly improves the visual effect of the image, making the dark details clearer and the bright parts less likely to be saturated.

[0045] 3. Enhanced signal-to-noise ratio: The present invention effectively reduces the noise in the image and improves the signal-to-noise ratio of the image through the 2×2 average binning technology in the signal-to-noise ratio enhancement module, making the image purer and richer in details.

[0046] 4. Real-time processing performance: The present invention adopts a real-time optimization processing module with pipeline design and parallel processing technology, which effectively reduces processing delays and ensures that the system can process high dynamic range images under real-time conditions, which is suitable for application scenarios that require rapid response. The system control module provides a user interface, allowing users to set and adjust system parameters according to specific application requirements, enhancing the flexibility and adaptability of the system and being able to meet the needs of a variety of complex scenarios.

[0047] 5. Efficient data transmission: The image data acquisition module of the present invention is connected to the image processing module through a high-speed data transmission interface, ensuring the real-time transmission of high-gain and low-gain image data and reducing delays and packet loss during data transmission.

[0048] 6. Accurate HDR synthesis: The HDR synthesis module of the present invention synthesizes high-gain and low-gain images into high-quality HDR images by analyzing unsaturated pixels in high-gain images and combining fixed gain ratios and offsets, thereby ensuring the accuracy and consistency of the synthesis results.

[0049] In summary, the present invention provides a real-time high dynamic range image processing method and system based on sCMOS, which can realize real-time processing of high-quality HDR images in various complex scenes, significantly improving the effect and efficiency of image processing. The image display and storage module transmits the processed HDR image data to the host computer for display and storage through the FPGA and host computer communication interface, ensuring the efficient transmission and storage of image data. The system is suitable for various application scenarios such as medical imaging, industrial detection, astronomical observation and high-end photography, and has broad application prospects and market demand.

[0050] These and other aspects of the present invention will be more clearly understood in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the exemplary embodiments or related technical descriptions. The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0052] Figure 1 The figure is a flow chart of a real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0053] Figure 2 The figure is a diagram showing the binning principle and effect in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0054] Figure 3 This is a graph showing the relationship between grayscale values ​​of high and low gain channels in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0055] Figure 4 This is a timing diagram of RGMII output in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0056] Figure 5 This is a timing diagram of RGMII input in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0057] Figure 6 This is a diagram of the FPGA algorithm structure in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0058] Figure 7 This is a system structure diagram of the host computer software in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0060] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0061] The following will be combined with the drawings in the exemplary embodiments of the present invention to clearly and completely describe the technical solutions in the exemplary embodiments of the present invention. Obviously, the exemplary embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0062] Since the existing hardware-based HDR image synthesis methods still have the problems of large amount of calculation and long operation time, and the fast synthesis method also has the problem of interval segmentation. The present invention provides a real-time high dynamic range image processing method and system based on sCMOS, which can efficiently synthesize HDR images, maintain image details and contrast, and keep consistent with the output frame rate of the high-speed image system.

[0063] The technical solution of the present invention is further described below in conjunction with specific embodiments:

[0064] See also Figure 1 As shown, Figure 1 A flowchart of a real-time high dynamic range image processing method based on sCMOS provided by the present invention. A real-time high dynamic range image processing method based on sCMOS provided in one embodiment of the present invention comprises the following steps:

[0065] Step S10, using the sCMOS sensor to simultaneously collect two images, a high-gain image and a low-gain image, to capture dark and bright details respectively;

[0066] Step S20, transmitting the high-gain and low-gain image data from the sCMOS sensor to an image processing module;

[0067] Step S30, storing the collected high-gain and low-gain image data in a memory, and performing preliminary preprocessing;

[0068] Step S40, linearly segmenting the grayscale values ​​of the high-gain image and the low-gain image, and performing grayscale stretching to enhance the image contrast;

[0069] Step S50: based on the grayscale characteristics of the high-gain and low-gain images, analyze the unsaturated pixels in the high-gain image, combine the fixed gain ratio k and the offset b, and synthesize the high-gain and low-gain images into an HDR image;

[0070] Step S60: enhance the signal-to-noise ratio of the image by using a 2×2 average binning technique, optimize floating-point operations while performing data storage and processing in parallel using a pipeline, and transmit the processed HDR image data to a host computer.

[0071] In the real-time high dynamic range image processing method and system based on sCMOS of this embodiment, high gain and low gain image acquisition of the sCMOS sensor adopts a synchronous exposure mode, and the image data acquisition module is connected to the image processing module through a high-speed data transmission interface.

[0072] In this embodiment, the gray value segmentation and stretching process includes histogram equalization and contrast enhancement processing of the image, performing image histogram statistics, and determining the gain and exposure time settings. If the gray range of a bit is evenly divided into b gray levels, where a and b are both even numbers, then the number of dark pixels (gray value 0) is defined as S low ), saturated pixel (grayscale value is 2 a -1), the number of pixels located in the upper and lower parts of the histograms of the two images is calculated as:

[0073]

[0074]

[0075] Where P is the percentage of pixels in the histogram, s represents low gain, l represents high gain, H represents the image histogram, h represents the grayscale level (1-b), and N represents the number of all pixels. t,s ≈80%, P t,l ≈80%, then the gain and exposure time are adjusted.

[0076] Among them, the signal-to-noise ratio enhancement is achieved through the average binning technology, including averaging the grayscale values ​​of the 2×2 pixel area. In this embodiment, the Binning signal-to-noise ratio is enhanced. Since the selected sCMOS sensor is 2048×1152, the project requires the image output to be 512×512. Since the sensor chip itself does not have the analog binning function, it is necessary to perform 2×2 average binning on the read pixel values ​​on the FPGA to obtain a pixel size of 1024×576. According to the analysis, the signal-to-noise ratio after digital 2×2 average binning will be increased by 2 times, such as Figure 2 shown.

[0077] When the pixels are not saturated, the grayscale value relationship between the corresponding points of the high-gain image and the low-gain image is:

[0078] H x，y =k*L x，y +b H x，y <2 a -1

[0079] In the formula, H x,y Represents the gray value of high-gain image, L x,y represents the gray value of the low-gain image, k and b represent coefficients. Using multiple sets of images for verification, the gray value relationship curve of the high and low gain channels is plotted as follows: Figure 3 As shown, the horizontal axis is the low-gain brightness value, and the vertical axis represents the high-gain brightness value. From the corresponding relationships of multiple groups of different gain combinations, it can be concluded that in the unsaturated area, the two show an obvious linear relationship, and this relationship still holds true for shooting in different scenes, that is, each group of gain settings obtains a fixed set of k and b.

[0080] In this embodiment, grayscale stretching is performed on high-gain and low-gain images. When synthesizing an HDR image according to the grayscale value relationship, the idea of ​​directly synthesizing the HDR image with useful information in the high- and low-gain images is combined with the high- and low-gain grayscale characteristics, when the following formula is satisfied:

[0081]

[0082] Then {SH∪HL} can obtain an HDR image that retains the details in both images and the synthesized image is smooth. The algorithm proposed in this invention uses Lapray's method to obtain the original image, so a can be directly set H =S / 2, then the segmentation point a of the low-gain image can be directly obtained according to the formula L At the same time, in order to better display the image effect, grayscale stretching is added in the fusion process to achieve the effect of improving the image contrast.

[0083] Among them, linear stretching is used to x,y ≤a L and L x,y ≥a L The area is stretched so that the grayscale range of the fused image is distributed in the entire [0, S], then:

[0084]

[0085] In the formula, H′ and L′ represent the images of H and L after a certain grayscale linear transformation respectively;

[0086] The synthesized image G HDR It is expressed as:

[0087]

[0088] In this embodiment, the image display and storage module includes transmitting HDR image data to the host computer for display and storage through an FPGA and a host computer communication interface. The real-time high dynamic range image processing method also coordinates and controls the parameter configuration of the sensor, the timing management of image data acquisition, the control of the processing flow, and the management of data transmission through a system control module. The system control module provides a user interface for users to set and adjust system parameters.

[0089] The data transmission between the high dynamic range sCMOS camera FPGA software product and the host computer is an Ethernet interface with a transmission rate of 600Mbps. The Ethernet protocol uses TCP / IP. The RGMII output timing and RGMII input timing in the Ethernet interface timing are respectively Figure 4 and Figure 5 shown.

[0090] In this embodiment, communication with the host computer is achieved through the USB interface, and the camera sensor configuration instructions sent by the host computer are received through the interface. The instructions mainly include parameters such as exposure time and adjustable gain, and the instructions are forwarded to the sensor through the camera interface to correctly configure the camera; and some status data, temperature data and other status information inside the FPGA are packaged according to the specified data format and returned to the host computer software.

[0091] Communication with the host computer is achieved through the Ethernet interface. The camera sensor image data collected and processed inside the FPGA is packaged according to the agreed data format and returned to the host computer software for displaying the image data. Among them, in the camera sensor configuration function, the camera configuration parameters received are used to configure the camera sensor through the SPI bus, mainly including the camera exposure time, adjustable gain and other parameters. The timing generator function generates the corresponding read and write control timing according to the interface timing requirements of the camera sensor to achieve the communication function with the camera sensor. The image cache function receives the image data collected by the camera sensor and caches it in the memory on the FPGA chip. The image processing function performs image processing on the image data cached in the memory, mainly performing non-uniformity correction, large dynamic range synthesis, binning synthesis and other processing. According to the hardware implementation analysis of the algorithm, as shown in the figure: Figure 5 As shown in the figure, the most time-consuming process of implementing the above FPGA algorithm structure is to find the maximum grayscale value of the low-gain image of the whole frame image and the minimum value of the high-gain image. In this embodiment, a pipeline method is designed to calculate the maximum value of the whole frame image while the image data is stored in the input memory. After the maximum value is obtained, a request signal is sent to the memory module to read the data in the memory and enter the G HDR module, and implement the algorithm point by point. HDR The module involves processes such as floating-point operations, which will cause delays of several clocks. However, the pipeline design ensures that the entire HDR synthesis algorithm operation process will not affect the image output frame rate, that is, the HDR image synthesis frame rate will be consistent with the output image frame rate of the high-speed imaging system.

[0092] Among them, the host computer software has two main external interfaces in terms of hardware interface. One is a USB interface for USB communication function, which realizes communication with the host computer, mainly sends camera sensor configuration instructions, forwards the instructions to the camera through FPGA, and performs correct configuration; and receives some status data, temperature data and other status information returned from FPGA to display the status. The other interface is the Ethernet interface, which mainly receives the camera sensor image data collected by FPGA, and performs storage, processing and display, etc., see Figure 7As shown in the figure, the host computer software of the high dynamic range sCMOS camera can be divided into four functional modules: camera parameter configuration function, image data acquisition function, image data storage function, and image data display function; among them, the camera parameter configuration function is used to configure the camera's exposure time, gain adjustment and other parameters, and the image data acquisition function receives the image data collected by the high dynamic range sCMOS camera through the Ethernet interface; the image data storage function is used to store the received image data, and can save the source code data as well as the parsed data, and can store the image data. At the same time, the stored data supports query mode. When storing, a new file is created at 0 o'clock every day, which is convenient for checking image data by day; the image data display function is used to parse the received data according to the data format and display it in different areas on the host computer software interface. The image data collected by the camera is correctly displayed in the form of an image.

[0093] Among them, the main operation process of the host computer software includes:

[0094] 1. Communication establishment process: Detect the communication request instruction sent by the high dynamic range sCMOS camera control module according to the protocol, respond to the control module, complete the interface registration, and establish communication.

[0095] 2. Send data: The host computer software sends the camera configuration parameters such as exposure time and gain setting data to the control module. For the specific format, please refer to the camera data manual to configure the camera correctly. It also sends corresponding control instructions for turning on and off the cooling function and for turning on and off the heating function.

[0096] 3. Receive data: After the camera is configured and working normally, it starts to collect image data. The host computer software realizes the functions of image data collection, storage, and data display through the Ethernet interface.

[0097] During the electron microscope integration, system installation and docking, the development of a multi-channel scanning controller that matches electron beam scanning control, sCMOS camera triggering and phosphor screen fluorescence duration was completed; the compact structure design of the sCMOS camera was completed, and long-term stable working tests were carried out; the performance and application index requirements of high-precision electron backscatter diffraction detectors were achieved, and the detector assessment was passed.

[0098] In the real-time high dynamic range image processing method based on sCMOS of the present invention, high-gain and low-gain images are collected simultaneously by the sCMOS sensor, which effectively captures the dark and bright details in the scene, significantly expands the dynamic range of the image, and avoids the problem of detail loss in the traditional single gain mode. The present invention uses gray value segmentation and stretching modules for image preprocessing, and greatly improves the visual effect of the image through histogram equalization and contrast enhancement technology, making the dark details clearer and the bright parts less prone to saturation.

[0099] The present invention effectively reduces the noise in the image and improves the signal-to-noise ratio of the image through the 2×2 average binning technology in the signal-to-noise ratio enhancement module, making the image purer and richer in details. The present invention adopts a real-time optimization processing module with pipeline design and parallel processing technology, which effectively reduces the processing delay and ensures that the system can process high dynamic range images under real-time conditions, which is suitable for application scenarios that require rapid response. The system control module provides a user interface, allowing users to set and adjust system parameters according to specific application requirements, enhancing the flexibility and adaptability of the system and being able to meet the needs of a variety of complex scenarios.

[0100] The image data acquisition module of the present invention is connected to the image processing module through a high-speed data transmission interface, ensuring the real-time transmission of high-gain and low-gain image data, reducing the delay and packet loss during data transmission. The HDR synthesis module of the present invention synthesizes the high-gain and low-gain images into high-quality HDR images by analyzing the unsaturated pixels in the high-gain image, combining a fixed gain ratio and an offset, and ensuring the accuracy and consistency of the synthesis results.

[0101] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0102] In one embodiment, the present invention provides a real-time high dynamic range image processing system based on sCMOS, which is used to perform the above-mentioned real-time high dynamic range image processing method based on sCMOS. The dual-gain HDR synthesis method includes:

[0103] Dual-gain image acquisition module, used to use the sCMOS sensor to simultaneously acquire high-gain and low-gain images, respectively used to capture dark and bright details;

[0104] An image data acquisition module, used for transmitting high-gain and low-gain image data from the sCMOS sensor to an image processing module;

[0105] A data cache processing module is used to store the collected high-gain and low-gain image data into a memory and perform preliminary preprocessing;

[0106] Grayscale stretching module, used to linearly segment the grayscale values ​​of high-gain and low-gain images, and perform grayscale stretching to enhance image contrast;

[0107] An HDR image synthesis module is used to analyze unsaturated pixels in the high-gain image based on the grayscale characteristics of the high-gain and low-gain images, and to synthesize the high-gain and low-gain images into an HDR image by combining a fixed gain ratio k and an offset b;

[0108] The signal-to-noise ratio enhancement module is used to enhance the signal-to-noise ratio of the image through 2×2 average binning;

[0109] A real-time optimization processing module for optimizing floating-point operations while data storage and processing are performed in parallel using a pipeline;

[0110] Image display and storage module, used to transmit the processed HDR image data to the host computer to realize real-time display and storage of the image;

[0111] The system control module is used to coordinate and control the parameter configuration of the sensor, the timing management of image data acquisition, the control of the processing flow, and the management of data transmission.

[0112] In this embodiment, the camera adopts the steps of the aforementioned real-time high dynamic range image processing method based on sCMOS during execution. Therefore, the operation process of the camera will not be described in detail in this embodiment.

[0113] In summary, the real-time high dynamic range image processing method and system based on sCMOS of the present invention can realize real-time processing of high-quality HDR images in various complex scenes, significantly improving the effect and efficiency of image processing. The image display and storage module transmits the processed HDR image data to the host computer for display and storage through the FPGA and host computer communication interface, ensuring the efficient transmission and storage of image data. The system is suitable for various application scenarios such as medical imaging, industrial detection, astronomical observation and high-end photography, and has broad application prospects and market demand.

[0114] In one embodiment, a computer device is also provided in an embodiment of the present invention, comprising at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor performs the steps of the sCMOS-based real-time high dynamic range image processing method.

[0115] In one embodiment, the present invention further provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the steps of the sCMOS-based real-time high dynamic range image processing method.

[0116] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be realized by instructing the relevant hardware through a computer program represented by computer instructions, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0117] Non-volatile memory may include read-only memory, magnetic tape, floppy disk, flash memory or optical storage, etc. Volatile memory may include random access memory or external cache memory. As an illustration and not limitation, RAM may be in various forms, such as static random access memory or dynamic random access memory, etc.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time high dynamic range image processing method based on sCMOS, characterized in that: The method comprises the following steps: The sCMOS sensor is used to simultaneously collect high-gain and low-gain images to capture dark and bright details respectively; Transmitting high-gain and low-gain image data from the sCMOS sensor to the image processing module; The collected high-gain and low-gain image data are stored in a memory and pre-processed; Linearly segment the grayscale values ​​of high-gain and low-gain images, and perform grayscale stretching to enhance image contrast; Based on the grayscale characteristics of high-gain and low-gain images, the unsaturated pixels in the high-gain image are analyzed, and the high-gain and low-gain images are synthesized into an HDR image by combining a fixed gain ratio k and an offset b. The signal-to-noise ratio of the image is enhanced by the 2×2 average binning technology, and the pipeline is used to optimize floating-point operations while data storage and processing are carried out in parallel, and the processed HDR image data is transmitted to the host computer.

2. The real-time high dynamic range image processing method based on sCMOS according to claim 1, characterized in that: The high-gain and low-gain image acquisition of the sCMOS sensor adopts a synchronous exposure mode.

3. The real-time high dynamic range image processing method based on sCMOS as claimed in claim 2, characterized in that The image data acquisition module is connected to the image processing module via a high-speed data transmission interface.

4. The real-time high dynamic range image processing method based on sCMOS according to claim 1, characterized in that: The gray value segmentation and stretching process includes histogram equalization and contrast enhancement processing of the image, image histogram statistics, and when determining the gain and exposure time settings, if the gray range of a bit is evenly divided into b gray levels, where a and b are both even numbers, then the number of dark pixels S is defined as low , the gray value is 0, the number of saturated pixels is S high , gray value is 2 a -1, calculate the number of pixels in the upper and lower parts of the histogram of the two images as: Where P is the percentage of pixels in the histogram, s represents low gain, l represents high gain, H represents the image histogram, h represents the grayscale level (1-b), and N represents the number of all pixels. t,s ≈80%, P t,l ≈80%, then the gain and exposure time are adjusted; P t，s Indicates the percentage of pixels in the low gain histogram; H t，s (h) represents the number of pixels of gray level (1-b) in the low gain histogram; P t，l Indicates the percentage of pixels in the high gain histogram; H t，l (h) represents the number of pixels at gray level (1-b) in the high gain histogram.

5. The real-time high dynamic range image processing method based on sCMOS according to claim 4, characterized in that: The signal-to-noise ratio enhancement is achieved by the average binning technology, which includes averaging the grayscale values ​​of a 2×2 pixel area. When the pixels are not saturated, the grayscale value relationship between the corresponding points of the high-gain image and the low-gain image is analyzed as follows: H x，y =k*L x，y +bH x，y <2 a -1 In the formula, H x,y Represents the gray value of high-gain image, L x,y represents the gray value of the low-gain image, k and b represent coefficients. Each set of gain settings will result in a fixed set of k and b. x and y represent the pixel coordinates.

6. The real-time high dynamic range image processing method based on sCMOS according to claim 5, characterized in that: The high-gain and low-gain images are stretched grayscale. When synthesizing the HDR image according to the grayscale value relationship, the idea of ​​directly synthesizing the HDR image with useful information in the high- and low-gain images is combined with the high- and low-gain grayscale characteristics. When the following formula is satisfied: In the formula, a L Indicates the segmentation point of the high-gain image histogram; a H Indicates the segmentation point of the high-gain image histogram; Use linear stretching to x,y ≤a L and L x,y ≥a L The area is stretched so that the grayscale range of the fused image is distributed in the entire [0, S], then: In the formula, H′ and L′ represent the images of H and L after a certain grayscale linear transformation respectively; The synthesized image G HDR It is expressed as:

7. The real-time high dynamic range image processing method based on sCMOS according to claim 6, characterized in that: The image display and storage module includes a communication interface with a host computer through an FPGA to transmit HDR image data to the host computer for display and storage.

8. The real-time high dynamic range image processing method based on sCMOS according to claim 7, characterized in that: The real-time high dynamic range image processing method also coordinates and controls the parameter configuration of the sensor, the timing management of image data acquisition, the control of the processing flow and the management of data transmission through a system control module. The system control module provides a user interface for the user to set and adjust the system parameters.

9. A real-time high dynamic range image processing system based on sCMOS, characterized in that: The system is used to perform the sCMOS-based real-time high dynamic range image processing method according to any one of claims 1 to 8, comprising: Dual-gain image acquisition module, used to use the sCMOS sensor to simultaneously acquire high-gain and low-gain images, respectively used to capture dark and bright details; An image data acquisition module, used for transmitting high-gain and low-gain image data from the sCMOS sensor to an image processing module; A data cache processing module is used to store the collected high-gain and low-gain image data into a memory and perform preliminary preprocessing; Grayscale stretching module, used to linearly segment the grayscale values ​​of high-gain and low-gain images, and perform grayscale stretching to enhance image contrast; An HDR image synthesis module is used to analyze unsaturated pixels in the high-gain image based on the grayscale characteristics of the high-gain and low-gain images, and to synthesize the high-gain and low-gain images into an HDR image by combining a fixed gain ratio k and an offset b; The signal-to-noise ratio enhancement module is used to enhance the signal-to-noise ratio of the image through 2×2 average binning; A real-time optimization processing module for optimizing floating-point operations while data storage and processing are performed in parallel using a pipeline; The image display and storage module is used to transmit the processed HDR image data to the host computer to realize real-time display and storage of the image.

10. The sCMOS-based real-time high dynamic range image processing system according to claim 9, characterized in that: The sCMOS-based real-time high dynamic range image processing system also includes a system control module for coordinating and controlling parameter configuration of sensors, timing management of image data acquisition, control of processing flow, and management of data transmission.

Images