Real-time high dynamic range image processing method and system based on sCMOS

Abstract

The present invention provides a real-time high dynamic range image processing method and system based on sCMOS, which relates to the field of image processing technology. The method uses an sCMOS sensor to simultaneously collect two high-gain and low-gain images, respectively used to capture dark and bright details; transmits the high-gain and low-gain image data from the sCMOS sensor to an image processing module; stores the collected high-gain and low-gain image data in a memory and performs preliminary preprocessing; linearly segments the grayscale values ​​of the high-gain and low-gain images, and performs grayscale stretching to enhance image contrast; analyzes unsaturated pixels in the high-gain image based on the grayscale characteristics of the high-gain and low-gain images, and combines the high-gain and low-gain images into an HDR image by combining a fixed gain ratio k and an offset b; and transmits the processed HDR image data to a host computer, thereby efficiently synthesizing the HDR image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a real-time high dynamic range (HDR) image processing method and system based on sCMOS. Background Art

[0002] Due to their limited dynamic range, traditional image sensors have difficulty capturing both bright and dark details in an image. To overcome this problem, HDR imaging technology has been proposed and widely studied. Existing HDR imaging methods usually require multiple exposures or multi-frame image synthesis, which is computationally complex and prone to ghosting. sCMOS sensors have high and low gain channels that can output two images of different brightness simultaneously, reducing the amount of computation while avoiding ghosting. However, existing hardware-based HDR image synthesis methods still have problems with large computational complexity and long computational time, and fast synthesis methods also have interval segmentation problems. Summary of the Invention

[0003] In view of this, in order to solve the above problems, the present invention proposes a real-time high dynamic range image processing method and system based on sCMOS, which can efficiently synthesize HDR images, maintain image details and contrast, and keep consistent with the output frame rate of the high-speed imaging system.

[0004] The present invention is achieved by adopting the following technical solutions:

[0005] In a first aspect, the present invention provides a real-time high dynamic range image processing method based on sCMOS, the method comprising the following steps:

[0006] The sCMOS sensor is used to simultaneously capture high-gain and low-gain images to capture dark and bright details respectively.

[0007] Transmitting high-gain and low-gain image data from the sCMOS sensor to the image processing module;

[0008] The collected high-gain and low-gain image data are stored in a memory and pre-processed;

[0009] Linearly segment the grayscale values ​​of high-gain and low-gain images and perform grayscale stretching to enhance image contrast;

[0010] Based on the grayscale characteristics of high-gain and low-gain images, the unsaturated pixels in the high-gain image are analyzed, and the high-gain and low-gain images are synthesized into an HDR image by combining a fixed gain ratio k and an offset b.

[0011] The signal-to-noise ratio of the image is enhanced by the 2×2 average binning technology, and the floating-point operation optimization is performed while the data storage and processing are carried out in parallel using a pipeline, and the processed HDR image data is transmitted to the host computer.

[0012] As a further solution of the present invention, the high-gain and low-gain image acquisition of the sCMOS sensor adopts a synchronous exposure mode.

[0013] As a further solution of the present invention, the image data acquisition module is connected to the image processing module via a high-speed data transmission interface.

[0014] As a further solution of the present invention, the gray value segmentation and stretching process includes histogram equalization and contrast enhancement processing of the image, performing image histogram statistics, and determining the gain and exposure time settings. If the gray range of a bit is evenly divided into b gray levels, where a and b are both even numbers, then the number of dark pixels (gray value 0) is defined (S low ), saturated pixel (gray value is 2 a -1) (S high ), calculate the number of pixels in the upper and lower parts of the histogram of the two images as:

[0015]

[0016] Where P is the percentage of pixels in the histogram, s represents low gain, l represents high gain, H represents the image histogram, h represents the grayscale level (1-b), and N represents the number of all pixels. t,s ≈80%, P t,l ≈80%, then the gain and exposure time are adjusted; P t，s Indicates the percentage of pixels in the low-gain histogram; H t，s (h) represents the number of pixels of gray level (1-b) in the low gain histogram; P t，l Indicates the percentage of pixels in the high gain histogram; H t，l (h) represents the number of pixels at gray level (1-b) in the high gain histogram.

[0017] As a further solution of the present invention, the signal-to-noise ratio enhancement is achieved by an average binning technique, which includes averaging the grayscale values ​​of a 2×2 pixel area. When the pixels are not saturated, the grayscale value relationship between the corresponding points of the high-gain image and the low-gain image is analyzed as follows:

[0018] H x，y =k*L x，y +bH x，y <2 a -1

[0019] Where H x,y Represents the grayscale value of high-gain image, L x,y Represents the grayscale value of the low-gain image, k represents the fixed gain ratio, b represents the offset b. Each set of gain settings results in a set of fixed k and b; x and y represent the pixel coordinates.

[0020] As a further solution of the present invention, grayscale stretching is performed on high-gain and low-gain images. When synthesizing an HDR image based on the grayscale value relationship, the idea of ​​directly synthesizing the HDR image with useful information in the high- and low-gain images is adopted. In combination with the high- and low-gain grayscale characteristics, when the following formula is satisfied:

[0021]

[0022] Where a L Indicates the segmentation point of the low-gain image histogram; a H Indicates the split point of the high-gain image histogram.

[0023] Use linear stretching to x,y ≤a L and L x,y ≥a L The area is stretched so that the grayscale range of the fused image is distributed in the entire [0, S], where S represents the maximum value of the fused image. Then:

[0024]

[0025] Where H′ and L′ represent the images of H and L after a certain grayscale linear transformation.

[0026] The synthesized image G HDR Expressed as:

[0027]

[0028] Among them, SH represents the detail information of the bright part of the scene, and HL represents the detail information of the bright part of the scene.

[0029] As a further solution of the present invention, the image display and storage module includes a communication interface with a host computer via FPGA to transmit HDR image data to the host computer for display and storage.

[0030] As a further solution of the present invention, the real-time high dynamic range image processing method also coordinates and controls the parameter configuration of the sensor, the timing management of image data acquisition, the control of the processing flow, and the management of data transmission through a system control module. The system control module provides a user interface for the user to set and adjust the system parameters.

[0031] In a second aspect, the present invention further provides a real-time high dynamic range image processing system based on sCMOS, for performing a real-time high dynamic range image processing method based on sCMOS, the dual-gain HDR synthesis system comprising:

[0032] Dual-gain image acquisition module, used to simultaneously acquire high-gain and low-gain images using the sCMOS sensor, respectively for capturing dark and bright details;

[0033] an image data acquisition module for transmitting high-gain and low-gain image data from the sCMOS sensor to an image processing module;

[0034] A data cache processing module is used to store the collected high-gain and low-gain image data into a memory and perform preliminary preprocessing;

[0035] Grayscale stretching module, used to linearly segment the grayscale values ​​of high-gain and low-gain images and perform grayscale stretching to enhance image contrast;

[0036] An HDR image synthesis module is used to analyze the unsaturated pixels in the high-gain image based on the grayscale characteristics of the high-gain and low-gain images, and synthesize the high-gain and low-gain images into an HDR image by combining a fixed gain ratio k and an offset b;

[0037] The signal-to-noise ratio enhancement module is used to enhance the signal-to-noise ratio of the image through 2×2 average binning;

[0038] A real-time optimization processing module for optimizing floating-point operations while data storage and processing are performed in parallel using a pipeline;

[0039] The image display and storage module is used to transmit the processed HDR image data to the host computer to realize real-time display and storage of the image.

[0040] As a further solution of the present invention, the sCMOS-based real-time high dynamic range image processing system also includes a system control module for coordinating and controlling sensor parameter configuration, timing management of image data acquisition, control of processing flow, and management of data transmission.

[0041] The present invention also includes a computer device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the sCMOS-based real-time high dynamic range image processing method.

[0042] The present invention also includes a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the sCMOS-based real-time high dynamic range image processing method.

[0043] Compared with the prior art, the real-time high dynamic range image processing method and system based on sCMOS provided by the present invention has the following beneficial effects:

[0044] 1. Extended dynamic range: This invention uses the sCMOS sensor to simultaneously capture high-gain and low-gain images, effectively capturing details in both dark and bright areas of the scene, significantly expanding the dynamic range of the image and avoiding the detail loss problem in traditional single-gain mode.

[0045] 2. Improved image quality: This invention uses grayscale value segmentation and stretching modules for image preprocessing, and through histogram equalization and contrast enhancement technology, it greatly improves the visual effect of the image, making dark details clearer and bright areas less likely to be saturated.

[0046] 3. Enhanced signal-to-noise ratio: The present invention effectively reduces the noise in the image and improves the signal-to-noise ratio of the image through the 2×2 average binning technology in the signal-to-noise ratio enhancement module, making the image purer and richer in details.

[0047] 4. Real-time Processing Performance: This invention utilizes a real-time optimization processing module with pipeline design and parallel processing technology, effectively reducing processing latency and ensuring the system can process high dynamic range images in real time, making it suitable for applications requiring rapid response. The system control module provides a user interface, allowing users to set and adjust system parameters according to specific application requirements, enhancing the system's flexibility and adaptability to various complex scenarios.

[0048] 5. Efficient data transmission: The image data acquisition module of the present invention is connected to the image processing module through a high-speed data transmission interface, ensuring real-time transmission of high-gain and low-gain image data, and reducing delays and packet loss during data transmission.

[0049] 6. Accurate HDR synthesis: The HDR synthesis module of the present invention synthesizes high-gain and low-gain images into a high-quality HDR image by analyzing the unsaturated pixels in the high-gain image and combining them with a fixed gain ratio and offset, ensuring the accuracy and consistency of the synthesis results.

[0050] In summary, the present invention provides a real-time high dynamic range image processing method and system based on sCMOS, which can realize real-time processing of high-quality HDR images in various complex scenarios, significantly improving the effect and efficiency of image processing. The image display and storage module transmits the processed HDR image data to the host computer for display and storage through the FPGA and host computer communication interface, ensuring efficient transmission and storage of image data. The system is suitable for various application scenarios such as medical imaging, industrial inspection, astronomical observation and high-end photography, and has broad application prospects and market demand.

[0051] These and other aspects of the present invention will become more readily apparent in the following description of the embodiments. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for the exemplary embodiments or related technical descriptions. The drawings are used to provide a further understanding of the present invention and constitute part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the drawings:

[0053] Figure 1 Flowchart of a real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0054] Figure 2 This is a diagram showing the binning principle and effect in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0055] Figure 3 This is a graph showing the relationship between grayscale values ​​of high and low gain channels in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0056] Figure 4 This is a timing diagram of RGMII output in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0057] Figure 5 This is a timing diagram of RGMII input in the real-time high dynamic range image processing method based on sCMOS according to an embodiment of the present invention.

[0058] Figure 6 This is a diagram of the FPGA algorithm structure in the sCMOS-based real-time high dynamic range image processing method according to an embodiment of the present invention.

[0059] Figure 7This is a system structure diagram of the host computer software in the sCMOS-based real-time high dynamic range image processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0061] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to be different types.

[0062] The following will clearly and completely describe the technical solutions in the exemplary embodiments of the present invention in conjunction with the accompanying drawings of the exemplary embodiments of the present invention. Obviously, the exemplary embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] Existing hardware-based HDR image synthesis methods still suffer from high computational complexity and long operation times, and fast synthesis methods also face the problem of interval segmentation. The present invention provides a real-time high dynamic range image processing method and system based on sCMOS, which can efficiently synthesize HDR images, maintain image detail and contrast, and maintain consistency with the output frame rate of high-speed imaging systems.

[0064] The technical solution of the present invention is further described below with reference to specific embodiments:

[0065] See Figure 1 As shown, Figure 1 A flowchart of a real-time high dynamic range image processing method based on sCMOS provided by the present invention. A real-time high dynamic range image processing method based on sCMOS provided in one embodiment of the present invention includes the following steps:

[0066] Step S10: using the sCMOS sensor to simultaneously capture two images, one with high gain and one with low gain, to capture details of dark and bright areas, respectively;

[0067] Step S20: transmitting the high-gain and low-gain image data from the sCMOS sensor to an image processing module;

[0068] Step S30: storing the collected high-gain and low-gain image data in a memory and performing preliminary preprocessing;

[0069] Step S40: linearly segmenting the grayscale values ​​of the high-gain and low-gain images, and performing grayscale stretching to enhance image contrast;

[0070] Step S50: Based on the grayscale characteristics of the high-gain and low-gain images, analyze the unsaturated pixels in the high-gain image, and combine the high-gain and low-gain images into an HDR image by combining a fixed gain ratio k and an offset b.

[0071] Step S60: Enhance the signal-to-noise ratio of the image by using a 2×2 average binning technique, optimize floating-point operations while performing data storage and processing in parallel using a pipeline, and transmit the processed HDR image data to a host computer.

[0072] In the sCMOS-based real-time high dynamic range image processing method and system of this embodiment, the high-gain and low-gain image acquisition of the sCMOS sensor adopts a synchronous exposure mode, and the image data acquisition module is connected to the image processing module via a high-speed data transmission interface.

[0073] In this embodiment, the grayscale value segmentation and stretching process includes histogram equalization and contrast enhancement processing of the image, performing image histogram statistics, and determining the gain and exposure time settings. If the grayscale range of a bit is evenly divided into b grayscale levels, where a and b are both even numbers, then the number of dark pixels (grayscale value 0) is defined (S low ), saturated pixel (gray value is 2 a -1), the number of pixels in the upper and lower parts of the histograms of the two images is calculated as:

[0074]

[0075] Where P is the percentage of pixels in the histogram, s represents low gain, l represents high gain, H represents the image histogram, h represents the grayscale level (1-b), and N represents the number of all pixels. t,s ≈80%, P t,l ≈80%, then the gain and exposure time are adjusted.

[0076] Among them, the signal-to-noise ratio enhancement is achieved through the average binning technology, which includes averaging the grayscale values ​​of a 2×2 pixel area. In this embodiment, the binning signal-to-noise ratio is enhanced. Since the selected sCMOS sensor is 2048×1152 and the project requires an image output of 512×512, and since the sensor chip itself does not have the analog binning function, it is necessary to perform 2×2 average binning on the read pixel values ​​on the FPGA to obtain a pixel size of 1024×576. According to analysis, the signal-to-noise ratio after digital 2×2 average binning will be increased by 2 times, as shown in the following example. Figure 2 shown.

[0077] When the pixels are not saturated, the grayscale value relationship between the corresponding points of the high-gain image and the low-gain image is:

[0078] H x，y =k*L x，y +b H x，y <2 a -1

[0079] Where H x,y Represents the grayscale value of high-gain image, L x,y Represents the grayscale value of the low-gain image, k represents the fixed gain ratio, and b represents the offset b. Using multiple sets of images for verification, the relationship curve of the grayscale values ​​of the high and low gain channels is drawn as follows: Figure 3 As shown, the horizontal axis is the low-gain brightness value, and the vertical axis represents the high-gain brightness value. From the corresponding relationship between multiple groups of different gain combinations, it can be concluded that in the unsaturated area, the two show an obvious linear relationship, and this relationship is still valid for shooting different scenes, that is, each group of gain settings results in a fixed set of k and b.

[0080] In this embodiment, grayscale stretching is performed on high-gain and low-gain images. When synthesizing an HDR image based on the grayscale value relationship, the idea of ​​directly synthesizing the HDR image with useful information in the high- and low-gain images is adopted. In combination with the high- and low-gain grayscale characteristics, when the following formula is satisfied:

[0081]

[0082] Then {SH∪HL} can obtain an HDR image that retains the details of both images and the synthesized image is smooth. The algorithm proposed in this invention uses Lapray's method to obtain the original image, so it can directly set a H =S / 2, then the segmentation point a of the low-gain image can be directly obtained according to the formula L At the same time, in order to better display the image effect, grayscale stretching is added in the fusion process to achieve the effect of improving the image contrast.

[0083] Among them, linear stretching is used to x,y ≤a L and L x,y ≥a L The area is stretched so that the grayscale range of the fused image is distributed in the entire [0, S], where S represents the maximum value of the fused image. Then:

[0084]

[0085] Where H′ and L′ represent the images of H and L after a certain grayscale linear transformation respectively;

[0086] The synthesized image G HDR Expressed as:

[0087]

[0088] Among them, SH represents the detail information of the bright part of the scene, and HL represents the detail information of the bright part of the scene.

[0089] In this embodiment, the image display and storage module includes a communication interface with the host computer through the FPGA to transmit HDR image data to the host computer for display and storage. The real-time high dynamic range image processing method also coordinates and controls the parameter configuration of the sensor, the timing management of image data acquisition, the control of the processing flow, and the management of data transmission through the system control module. The system control module provides a user interface for the user to set and adjust the system parameters.

[0090] The data transmission between the high dynamic range sCMOS camera FPGA software product and the host computer is through the Ethernet interface with a transmission rate of 600Mbps. The Ethernet protocol uses TCP / IP. The RGMII output timing and RGMII input timing in the Ethernet interface timing are respectively shown in Figure 4 and Figure 5 shown.

[0091] In this embodiment, communication with the host computer is achieved through the USB interface. The camera sensor configuration instructions sent by the host computer are received through this interface. The instructions mainly include parameters such as exposure time and adjustable gain. The instructions are forwarded to the sensor through the camera interface to correctly configure the camera; and some status data, temperature data and other status information inside the FPGA are packaged according to the specified data format and returned to the host computer software.

[0092] Communication with the host computer is achieved through the Ethernet interface. The camera sensor image data collected and processed inside the FPGA is packaged according to the agreed data format and returned to the host computer software for displaying the image data. Among them, in the camera sensor configuration function, the camera configuration parameters received are used to configure the camera sensor through the SPI bus, mainly including the camera exposure time, adjustable gain and other parameters. The timing generator function generates the corresponding read and write control timing according to the interface timing requirements of the camera sensor to achieve the communication function with the camera sensor. The image cache function receives the image data collected by the camera sensor and caches it in the memory on the FPGA chip. The image processing function performs image processing on the image data cached in the memory, mainly performing non-uniformity correction, large dynamic range synthesis, binning synthesis and other processing. According to the hardware implementation analysis of the algorithm, Figure 5 As shown in the figure, the most time-consuming process of implementing the above FPGA algorithm structure is to find the maximum grayscale value of the low-gain image and the minimum grayscale value of the high-gain image of the entire frame. In this embodiment, a pipeline method is designed to calculate the maximum grayscale value of the entire frame while the image data is stored in the input memory. After the maximum grayscale value is obtained, a request signal is sent to the memory module to read the data in the memory and enter the G HDR Module, point by point calculation to implement the algorithm. HDR The module involves processes such as floating-point operations, which will cause delays of several clocks. However, the pipeline design ensures that the entire HDR synthesis algorithm operation process does not affect the image output frame rate. That is, the HDR image synthesis frame rate will be consistent with the output image frame rate of the high-speed imaging system.

[0093] The host computer software has two main external interfaces in terms of hardware interface. One is the USB interface for USB communication function, which realizes communication with the host computer. It mainly sends camera sensor configuration instructions, which are forwarded to the camera through the FPGA for correct configuration; and receives some status data, temperature data and other status information returned from the FPGA for status display. The other interface is the Ethernet interface, which mainly receives the camera sensor image data collected by the FPGA, and performs storage, processing and display. Figure 7As shown, the high-dynamic-range sCMOS camera host computer software can be divided into four functional modules: camera parameter configuration, image data acquisition, image data storage, and image data display. The camera parameter configuration function is used to configure parameters such as camera exposure time and gain adjustment. The image data acquisition function receives image data captured by the high-dynamic-range sCMOS camera via an Ethernet interface. The image data storage function is used to store both the source and parsed image data. The stored data is queryable, with a new file created each day at 0:00 AM, making it easy to access image data by day. The image data display function parses the received data according to its format and displays it in separate areas on the host computer software interface. The image data captured by the camera is accurately displayed in the form of images.

[0094] Among them, the main operation process of the host computer software includes:

[0095] 1. Communication establishment process: Detect the communication request instruction sent by the high dynamic range sCMOS camera control module according to the protocol, respond to the control module, complete the interface registration, and establish communication.

[0096] 2. Sending Data: The host computer software sends camera configuration parameters, such as exposure time and gain settings, to the control module. Refer to the camera data sheet for the specific format to configure the camera correctly. The host computer software also issues control commands for turning the cooling function on and off, and for turning the heating function on and off.

[0097] 3. Data Receiving: Once the camera is configured and operating normally, it begins collecting image data. The host computer software uses the Ethernet interface to collect, store, and display image data.

[0098] During the electron microscope integration, system installation and docking, the development of a multi-channel scanning controller that matches electron beam scanning control, sCMOS camera triggering and phosphor screen fluorescence duration was completed; the compact structure design of the sCMOS camera was completed, and long-term stable operation tests were carried out; the performance and application index requirements of the high-precision electron backscatter diffraction detector were achieved, and the detector assessment was passed.

[0099] In this sCMOS-based real-time high-dynamic-range image processing method, the sCMOS sensor simultaneously captures high-gain and low-gain images, effectively capturing details in both dark and bright areas of a scene. This significantly expands the image's dynamic range and avoids the detail loss typically associated with traditional single-gain modes. This method utilizes a grayscale value segmentation and stretching module for image preprocessing, and employs histogram equalization and contrast enhancement techniques to significantly enhance the visual quality of the image, making dark details clearer and highlights less prone to saturation.

[0100] This invention effectively reduces image noise and improves the image's signal-to-noise ratio (SNR) through the 2×2 average binning technique in the signal-to-noise ratio enhancement module, resulting in a purer and more detailed image. The invention also employs a real-time optimization processing module that utilizes pipeline design and parallel processing technology, effectively reducing processing latency and ensuring the system can process high-dynamic-range images in real time, making it suitable for applications requiring rapid response. The system control module provides a user interface that allows users to set and adjust system parameters based on specific application requirements, enhancing the system's flexibility and adaptability to meet the needs of a variety of complex scenarios.

[0101] The image data acquisition module of the present invention is connected to the image processing module via a high-speed data transmission interface, ensuring real-time transmission of high-gain and low-gain image data, reducing delays and packet loss during data transmission. The HDR synthesis module of the present invention synthesizes the high-gain and low-gain images into a high-quality HDR image by analyzing unsaturated pixels in the high-gain image and combining them with a fixed gain ratio and offset, ensuring the accuracy and consistency of the synthesis results.

[0102] It should be understood that, although the above is described in a certain order, these steps are not necessarily performed in sequence according to the above order. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0103] In one embodiment, the present invention provides a real-time high dynamic range image processing system based on sCMOS, which is used to perform the above-mentioned real-time high dynamic range image processing method based on sCMOS. The dual-gain HDR synthesis method includes:

[0104] Dual-gain image acquisition module, used to simultaneously acquire high-gain and low-gain images using the sCMOS sensor, respectively for capturing dark and bright details;

[0105] an image data acquisition module for transmitting high-gain and low-gain image data from the sCMOS sensor to an image processing module;

[0106] A data cache processing module is used to store the collected high-gain and low-gain image data into a memory and perform preliminary preprocessing;

[0107] Grayscale stretching module, used to linearly segment the grayscale values ​​of high-gain and low-gain images and perform grayscale stretching to enhance image contrast;

[0108] An HDR image synthesis module is used to analyze the unsaturated pixels in the high-gain image based on the grayscale characteristics of the high-gain and low-gain images, and synthesize the high-gain and low-gain images into an HDR image by combining a fixed gain ratio k and an offset b;

[0109] The signal-to-noise ratio enhancement module is used to enhance the signal-to-noise ratio of the image through 2×2 average binning;

[0110] A real-time optimization processing module for optimizing floating-point operations while data storage and processing are performed in parallel using a pipeline;

[0111] Image display and storage module, used to transmit the processed HDR image data to the host computer to realize real-time display and storage of the image;

[0112] The system control module is used to coordinate and control the parameter configuration of the sensor, the timing management of image data acquisition, the control of the processing flow, and the management of data transmission.

[0113] In this embodiment, the camera adopts the steps of the aforementioned sCMOS-based real-time high dynamic range image processing method during execution. Therefore, the operation process of the camera will not be described in detail in this embodiment.

[0114] In summary, the sCMOS-based real-time high dynamic range image processing method and system of the present invention can realize real-time processing of high-quality HDR images in various complex scenarios, significantly improving the effect and efficiency of image processing. The image display and storage module transmits the processed HDR image data to the host computer for display and storage through the FPGA and host computer communication interface, ensuring efficient transmission and storage of image data. The system is suitable for various application scenarios such as medical imaging, industrial inspection, astronomical observation, and high-end photography, and has broad application prospects and market demand.

[0115] In one embodiment, a computer device is further provided in an embodiment of the present invention, comprising at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the sCMOS-based real-time high dynamic range image processing method.

[0116] In one embodiment, the present invention further provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the steps of the sCMOS-based real-time high dynamic range image processing method.

[0117] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program represented by computer instructions. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above-mentioned methods. In addition, any reference to memory, storage, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0118] Non-volatile memory may include read-only memory, magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory or external cache memory. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory or dynamic random access memory.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A real-time high dynamic range image processing method based on sCMOS, characterized in that: The method comprises the following steps: The sCMOS sensor is used to simultaneously capture high-gain and low-gain images to capture dark and bright details respectively. Transmitting high-gain and low-gain image data from the sCMOS sensor to the image processing module; The collected high-gain and low-gain image data are stored in a memory and pre-processed; Linearly segment the grayscale values ​​of high-gain and low-gain images and perform grayscale stretching to enhance image contrast; Based on the grayscale characteristics of high-gain and low-gain images, the unsaturated pixels in the high-gain image are analyzed, and the high-gain and low-gain images are synthesized into an HDR image by combining a fixed gain ratio k and an offset b. The signal-to-noise ratio of the image is enhanced by the 2×2 average binning technology, and the floating-point operation optimization is performed while the data storage and processing are carried out in parallel using a pipeline, and the processed HDR image data is transmitted to the host computer.

2. The real-time high dynamic range image processing method based on sCMOS according to claim 1, characterized in that: The high-gain and low-gain image acquisition of the sCMOS sensor adopts a synchronous exposure mode.

3. The real-time high dynamic range image processing method based on sCMOS according to claim 2, characterized in that The image data acquisition module is connected to the image processing module via a high-speed data transmission interface.

4. The real-time high dynamic range image processing method based on sCMOS according to claim 1, characterized in that: The grayscale value segmentation and stretching process includes histogram equalization and contrast enhancement processing of the image, image histogram statistics, and determining the gain and exposure time settings. If the grayscale range of a bit is evenly divided into b grayscale levels, where a and b are both even numbers, then the number of dark pixels S is defined. low , the gray value is 0, the number of saturated pixels is S high , grayscale value is 2 a -1, calculate the number of pixels in the upper and lower parts of the histogram of the two images as: Where P is the percentage of pixels in the histogram, s represents low gain, l represents high gain, H represents the image histogram, h represents the grayscale level (1-b), and N represents the number of all pixels. t,s ≈80%, P t,l ≈80%, then the gain and exposure time are adjusted; P t，s Indicates the percentage of pixels in the low-gain histogram; H t，s (h) represents the number of pixels of gray level (1-b) in the low gain histogram; P t，l Indicates the percentage of pixels in the high gain histogram; H t，l (h) represents the number of pixels at gray level (1-b) in the high gain histogram.

5. The real-time high dynamic range image processing method based on sCMOS according to claim 4, characterized in that: The signal-to-noise ratio enhancement is achieved through the binning technique, which involves averaging the grayscale values ​​of a 2×2 pixel area. When the pixels are not saturated, the grayscale value relationship between the corresponding points of the high-gain image and the low-gain image is analyzed as follows: H x，y =k*L x，y +b H x，y <2 a -1 Where H x,y Represents the grayscale value of high-gain image, L x,y Represents the grayscale value of the low-gain image, k represents the fixed gain ratio, b represents the offset b. Each set of gain settings results in a set of fixed k and b; x and y represent the pixel coordinates.

6. The real-time high dynamic range image processing method based on sCMOS according to claim 5, characterized in that: When performing grayscale stretching on high-gain and low-gain images and synthesizing HDR images based on the grayscale value relationship, the idea of ​​directly synthesizing the useful information in high- and low-gain images into HDR images is adopted, and the high- and low-gain grayscale characteristics are combined, when the following formula is satisfied: Where a L Indicates the segmentation point of the low-gain image histogram; a H Indicates the segmentation point of the high-gain image histogram; Use linear stretching to x,y ≤a L and L x,y ≥a L The area is stretched so that the grayscale range of the fused image is distributed in the entire [0, S], where S represents the maximum value of the fused image. Then: Where H′ and L′ represent the images of H and L after grayscale linear transformation respectively; The synthesized image G HDR Expressed as: Among them, SH represents the detail information of the bright part of the scene, and HL represents the detail information of the bright part of the scene.

7. The real-time high dynamic range image processing method based on sCMOS according to claim 6, characterized in that: The image display and storage module includes a communication interface with a host computer through an FPGA to transmit HDR image data to the host computer for display and storage.

8. The sCMOS-based real-time high dynamic range image processing method according to claim 7, wherein: This real-time high dynamic range image processing method also coordinates and controls the sensor parameter configuration, image data acquisition timing management, processing flow control, and data transmission management through a system control module. The system control module provides a user interface for users to set and adjust system parameters.

9. A real-time high dynamic range image processing system based on sCMOS, characterized in that: A system for executing the sCMOS-based real-time high dynamic range image processing method according to any one of claims 1 to 8, comprising: Dual-gain image acquisition module, used to simultaneously acquire high-gain and low-gain images using the sCMOS sensor, respectively for capturing dark and bright details; an image data acquisition module for transmitting high-gain and low-gain image data from the sCMOS sensor to an image processing module; A data cache processing module is used to store the collected high-gain and low-gain image data into a memory and perform preliminary preprocessing; Grayscale stretching module, used to linearly segment the grayscale values ​​of high-gain and low-gain images and perform grayscale stretching to enhance image contrast; An HDR image synthesis module is used to analyze the unsaturated pixels in the high-gain image based on the grayscale characteristics of the high-gain and low-gain images, and synthesize the high-gain and low-gain images into an HDR image by combining a fixed gain ratio k and an offset b; The signal-to-noise ratio enhancement module is used to enhance the signal-to-noise ratio of the image through 2×2 average binning; A real-time optimization processing module for optimizing floating-point operations while data storage and processing are performed in parallel using a pipeline; The image display and storage module is used to transmit the processed HDR image data to the host computer to realize real-time display and storage of the image.

10. The sCMOS-based real-time high dynamic range image processing system according to claim 9, wherein: The sCMOS-based real-time high dynamic range image processing system also includes a system control module for coordinating and controlling sensor parameter configuration, timing management of image data acquisition, control of processing flow, and management of data transmission.

Images