Ultra-wide spectrum image processing method and system based on FPGA dynamic reconstruction

Abstract

The invention relates to the technical field of ultra-wide spectrum image processing, in particular to an ultra-wide spectrum image processing method and system based on FPGA dynamic reconstruction, and the method comprises the steps: carrying out the FPGA internal special logic operation of a to-be-processed heterogeneous image, obtaining a standardized visible light data stream, obtaining a standardized infrared data stream, and carrying out the FPGA internal special logic operation; processing the standardized visible light data stream and the standardized infrared data stream to obtain a YUV visible light image, an equalized thermal infrared image and a sensor control parameter, performing ISP algorithm structure recombination on an ultra-wide spectrum image processing module to obtain a recombined ISP processing assembly line, and obtaining an optimal ISP processing assembly line and a semantic enhancement fusion image; and completing ultra-wide spectrum image processing based on FPGA dynamic reconstruction based on the optimal ISP processing assembly line, the semantic enhancement fusion image and the sensor control parameters. According to the method, the FPGA resource utilization rate can be improved, and the ISP algorithm reconstruction time can be greatly shortened.

Description

Technical Field

[0001] This invention relates to the field of ultrawide spectral image processing technology, and in particular to an ultrawide spectral image processing method and system based on FPGA dynamic reconstruction. Background Technology

[0002] FPGA dynamic reconfiguration refers to the process of reloading some or all of the logic into the FPGA without completely shutting down the system. Ultrawidespectral imaging is image data encompassing a wide spectral range from ultraviolet to infrared.

[0003] Traditional FPGA-based dynamic reconstruction is limited to data processing in a single visible light band, lacking mechanisms for synchronous access and parallel processing of multi-modal optical signals (such as visible and infrared). Secondly, the system requires interrupting the current data processing pipeline during dynamic reconstruction, introducing millisecond-level delays that make it difficult to meet the strict timing constraints required for simultaneous processing of multi-source data in high-real-time applications. Furthermore, existing image processing chains lack complete ISP pipeline algorithm integration, supporting only partial preprocessing functions and lacking key modules such as color correction, noise suppression, and detail enhancement, resulting in limited image quality output and failing to meet the requirements of imaging accuracy and visual quality in practical applications. Therefore, improving FPGA resource utilization and significantly reducing ISP algorithm reconstruction time are urgent technical challenges. Summary of the Invention

[0004] This invention provides an ultrawide spectrum image processing method based on FPGA dynamic reconstruction and a computer-readable storage medium. Its main purpose is to improve the utilization of FPGA resources and greatly shorten the reconstruction time of the ISP algorithm.

[0005] To achieve the above objectives, this invention provides an ultrawideband image processing method based on FPGA dynamic reconstruction, comprising: The visible light sensor, infrared detector, and ultrawide spectrum image processing module were identified. The ultrawide spectrum image processing module includes: Zynq FPGA PS side, Zynq FPGA PL side, AXI bus, ISP processing module, and data interface conversion module. The visible light sensor and the infrared detector are driven separately to obtain heterogeneous visible light images and heterogeneous infrared images; A heterogeneous visible light image is input into the data interface conversion module to obtain a heterogeneous image to be processed. The heterogeneous image to be processed is subjected to dedicated logic operations within the FPGA to obtain a standardized visible light data stream. The standardized visible light data stream includes: a set of visible light values ​​and a valid visible light data signal. The set of visible light values ​​includes multiple visible light values. The dedicated logic operations within the FPGA include: physical layer conversion of differential signals to single-ended signals and cross-clock domain synchronization. Standardized infrared data streams are acquired based on heterogeneous infrared images. The standardized infrared data streams include: thermal infrared value sets and valid thermal infrared data signals. The standardized visible light data stream and the standardized infrared data stream were processed by the ultrawide spectrum image processing module to obtain YUV visible light images, equalized thermal infrared images and sensor control parameters. The ISP algorithm structure of the ultrawide spectrum image processing module is restructured using a pre-built dynamic algorithm routing unit to obtain a restructured ISP processing pipeline; The optimal ISP processing pipeline and semantically enhanced fused image are obtained based on YUV visible light images, equalized thermal infrared images, and reconstructed ISP processing pipelines. Ultrawide spectral image processing based on FPGA dynamic reconstruction is achieved by using the optimal ISP processing pipeline, semantically enhanced fused images, and sensor control parameters.

[0006] Optionally, the ultrawideband image processing module processes the standardized visible light data stream and the standardized infrared data stream respectively to obtain a YUV visible light image, an equalized thermal infrared image, and sensor control parameters, including: Visible light values ​​are extracted sequentially from the set of visible light values ​​in the standardized visible light data stream, and the following operations are performed on each extracted visible light value: If the visible light value is greater than the preset calibration black level value, the corrected visible light value is calculated based on the calibration black level value and the visible light value. The corrected visible light value is the value obtained by subtracting the calibration black level value from the visible light value. If the visible light value is less than or equal to the calibrated black level value, then the preset black level correction value is used as the corrected visible light value, wherein the black level correction value is 0; A digital gain operation is performed on the corrected visible light value to obtain the gain visible light value. The gain visible light values ​​are then summarized to obtain a gain visible light value set. A sensor bad pixel correction operation is then performed on the gain visible light value set to obtain a bad pixel corrected visible light pixel set. White balance correction is performed on the visible light pixel set for defective pixel correction to obtain a white balance visible light pixel set. Based on the white balance visible light pixel set, an RGB visible light image and sensor control parameters are obtained. An RGB to YUV conversion operation is performed on the RGB visible light image to obtain a YUV visible light image. The standardized infrared data stream is processed using an ultrawide spectrum image processing module to obtain an equalized thermal infrared image.

[0007] Optionally, the step of performing sensor dead pixel correction on the gain visible light value set to obtain a dead pixel corrected visible light pixel set includes: Extract the gain visible light values ​​sequentially from the gain visible light value set, and perform the following operations on each extracted gain visible light value: Create a sliding window, take the gain visible light value as the center pixel, obtain the pixel color and center pixel value of the center pixel, and obtain the set of pixel values ​​of the same color based on the sliding window, the center pixel and the pixel color; Calculate the pixel difference between the central pixel value and each pixel value of the same color in the set of pixel values ​​of the same color to obtain the pixel difference set. Obtain the difference sign set based on the pixel difference set, where the difference sign corresponds one-to-one with the pixel difference. Determine whether all the difference signs in the difference sign set are the same; If all the difference signs in the difference sign set are the same, then the center pixel is regarded as a normal pixel. If the difference signs in the difference sign set are not all the same, then determine whether there is a pixel difference in the pixel difference set that is not greater than a preset difference threshold. If there is a pixel difference in the pixel difference set that is not greater than the preset difference threshold, then the center pixel is regarded as a bad pixel. Perform median filtering on the set of pixels of the same color to obtain the median pixel value. Use the median pixel value to correct bad pixels and obtain the updated pixel. If there is no pixel difference in the pixel difference set that is not greater than a preset difference threshold, then the center pixel is regarded as a normal pixel. By summarizing the updated pixels and normal pixels, a set of visible light pixels for bad pixel correction is obtained.

[0008] Optionally, the step of performing white balance correction on the visible light pixel set for defective pixel correction to obtain a white balance visible light pixel set includes: The bad pixel correction image is identified based on the visible light pixel set for bad pixel correction. The bad pixel correction image is then divided to obtain an image division region set, which includes multiple image division regions. Image regions are extracted sequentially from the image region set. Red pixel value set, green pixel value set and blue pixel value set are obtained according to the image region. The mean value of red pixel, green pixel and blue pixel are calculated according to the red pixel value set, green pixel value set and blue pixel value set. The region parameters are calculated based on the average values ​​of red, green, and blue pixels. The region parameters include the region brightness value and the region saturation value. If the region parameters meet the preset standard region parameter range conditions, then the image region is considered a valid image region. If the region parameters do not meet the preset standard region parameter range conditions, the image region is removed from the image region set to obtain an updated region set. The updated region set is used as the image region set, and the process of extracting image regions sequentially from the image region set is repeated until all image regions in the image region set have been extracted. The effective image regions are summarized to obtain the effective image region set. The effective image region set is then calculated using the Zynq FPGA PS side to obtain the channel gain factor set. White balance correction is performed on the visible light pixel set for bad pixel correction based on the channel gain factor set to obtain the white balance visible light pixel set.

[0009] Optionally, the step of acquiring RGB visible light images and sensor control parameters based on a white balance visible light pixel set includes: Extract white balance visible light pixels sequentially from the white balance visible light pixel set, use the white balance visible light pixels as the center pixels to be denoised, create a denoising window, and obtain the set of visible light pixels to be denoised based on the center pixels to be denoised and the denoising window. Extract the visible light pixels to be denoised sequentially from the set of visible light pixels to be denoised, and perform the following operations on each extracted visible light pixel: The pre-constructed bilinear interpolation method is used to estimate the full color of the visible light pixels to be denoised, and the R value, G value and B value are obtained. The pixel gray value is calculated based on the R value, G value and B value. The pixel gray values ​​are summarized to obtain the window gray map corresponding to the set of visible light pixels to be denoised. Perform convolution operation on the window grayscale image to obtain the horizontal and vertical gradients, and calculate the gradient magnitude based on the horizontal and vertical gradients. If the gradient magnitude is less than the preset gradient threshold, then the center pixel to be denoised is subjected to a preset mean filter to obtain the filtered center pixel. If the gradient magnitude is greater than or equal to the preset gradient threshold, then the center pixel to be denoised is taken as the center pixel for denoising. By summing the center pixels of the filter and the center pixels of the noise reduction, a noise-reduced grayscale image is obtained. This grayscale image is then divided into grayscale partitioning sets. Using a pre-constructed center-weighted metering principle, weights are assigned to the grayscale partitioning sets to obtain region weight sets. The region weights are related to the grayscale partitioning... Figure 1 One-to-one correspondence; RGB visible light images and sensor control parameters are obtained based on grayscale partitioning atlases and region weight sets.

[0010] Optionally, the step of obtaining RGB visible light images and sensor control parameters based on grayscale partitioning atlases and region weight sets includes: Extract grayscale partition images sequentially from the grayscale partition image set, determine the target region weight from the region weight set based on the grayscale partition images, calculate the pixel mean based on the grayscale partition images, calculate the weighted average brightness value based on the pixel mean and the target region weight, and summarize the weighted average brightness values ​​to obtain the weighted average brightness value set. The ACE statistics are calculated based on the weighted average brightness value set. The ACE statistics are then fed back to the pre-built ARM processor via the AXI bus to obtain the sensor control parameters, which include: exposure time and exposure value. The denoised grayscale image is stretched to obtain a stretched grayscale image. Then, a demosaic operation is performed on the stretched grayscale image to obtain an RGB visible light image.

[0011] Optionally, the step of processing the standardized infrared data stream using the ultrawide spectral image processing module to obtain an equalized thermal infrared image includes: In the ultrawide spectrum image processing module, a non-uniformity correction operation is performed on the thermal infrared value set in the standardized infrared data stream to obtain the correction pixel response value set. The correction pixel set is then identified based on the correction pixel response value set, where there is a one-to-one correspondence between the correction pixel and the correction pixel response value. The correction pixel response values ​​are extracted sequentially from the correction pixel response value set, a correction sliding window is created, the correction pixel response values ​​are used as the center pixel response values, and the window pixel response value sequence is obtained based on the correction sliding window and the center pixel response values. The pixel mean is calculated based on the window pixel response value sequence, and the pixel absolute difference is calculated based on the pixel mean and the center pixel response value. If the absolute difference of the pixels is greater than the preset standard deviation threshold, then the corrected pixel corresponding to the central pixel response value in the corrected pixel set is taken as a blind pixel, and the blind pixel is corrected to obtain the corrected blind pixel. If the absolute difference of the pixels is not greater than the preset standard deviation threshold, then the corrected pixel corresponding to the central pixel response value in the corrected pixel set is regarded as a normal pixel. By summing the corrected blind pixels and normal pixels, a set of corrected blind pixels is obtained. Then, 2D noise reduction is performed on the set of corrected blind pixels to obtain a set of denoised pixels. A dual-platform histogram equalization image enhancement operation is performed on the denoised pixel set to obtain an equalized thermal infrared image.

[0012] Optionally, the step of obtaining the optimal ISP processing pipeline and semantically enhanced fused image based on YUV visible light image, equalized thermal infrared image, and reconstructed ISP processing pipeline includes: A pre-built scene recognition network was used to perform scene recognition on YUV visible light images to obtain scene recognition results, including nighttime scenes and foggy scenes. The optimal parameter set is generated based on the pre-constructed parameter regression neural network and scene recognition results. The optimal parameter set is then sent to the reconstructed ISP processing pipeline using the AXI bus to obtain the optimal ISP processing pipeline. A pre-built multimodal data module is used to fuse YUV visible light images and equalized thermal infrared images to obtain a semantically enhanced fused image. The multimodal data module includes a pixel-level fusion unit, a semantic segmentation network, and a content-aware enhancement unit.

[0013] Optionally, the step of fusing YUV visible light images and equalized thermal infrared images using a pre-built multimodal data module to obtain a semantically enhanced fused image includes: The pixel-level fusion unit is used to perform preliminary fusion of the YUV visible light image and the equalized thermal infrared image to obtain an initial fused image. The initial fused image is input into the semantic segmentation network to obtain the image to be segmented. Key target recognition is performed on the image to be segmented to obtain a target segmentation mask. The initial fused image is segmented according to the target segmentation mask to obtain a set of key target regions and a set of non-key target regions. The set of key target regions includes one or more key target regions, and the set of non-key target regions includes one or more non-key target regions. The key target regions include pedestrians or vehicles. The content-aware enhancement unit performs local image enhancement operations on each key target region in the key target region set to obtain a set of locally enhanced regions. The content-aware enhancement unit performs lightweight processing operations on each non-critical target region in the non-critical target region set to obtain a lightweight region set. The locally enhanced region set and the lightweight region set are stitched together to obtain a semantically enhanced fused image.

[0014] To achieve the above objectives, the present invention also provides an ultrawideband image processing system based on FPGA dynamic reconstruction, comprising: The sensor driver module is used to identify the visible light sensor, the infrared detector, and the ultrawide spectrum image processing module. The ultrawide spectrum image processing module includes: Zynq FPGA PS side, Zynq FPGA PL side, AXI bus, ISP processing module, and data interface conversion module, which drive the visible light sensor and the infrared detector respectively to obtain visible light heterogeneous images and infrared heterogeneous images. The data standardization processing module is used to input a visible light heterogeneous image into the data interface conversion module to obtain a heterogeneous image to be processed, and to perform FPGA internal dedicated logic operations on the heterogeneous image to be processed to obtain a standardized visible light data stream. The standardized visible light data stream includes: a set of visible light values ​​and a valid visible light data signal. The set of visible light values ​​includes multiple visible light values. The FPGA internal dedicated logic operations include: physical layer conversion of differential signals to single-ended signals and cross-clock domain synchronization. A standardized infrared data stream is obtained based on the infrared heterogeneous image. The standardized infrared data stream includes: a set of thermal infrared values ​​and a valid thermal infrared data signal. The algorithm reconstruction module is used to process the standardized visible light data stream and the standardized infrared data stream using the ultrawide spectrum image processing module to obtain YUV visible light images, equalized thermal infrared images and sensor control parameters. The pre-built dynamic algorithm routing unit is used to reconstruct the ISP algorithm structure of the ultrawide spectrum image processing module to obtain the reconstructed ISP processing pipeline. The FPGA dynamic reconstruction module is used to obtain the optimal ISP processing pipeline and semantically enhanced fused image based on YUV visible light image, equalized thermal infrared image and reconstructed ISP processing pipeline. Based on the optimal ISP processing pipeline, semantically enhanced fused image and sensor control parameters, it completes ultrawide spectrum image processing based on FPGA dynamic reconstruction.

[0015] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: Memory, storing at least one instruction; The processor executes the instructions stored in the memory to implement the ultrawide spectrum image processing method based on FPGA dynamic reconstruction described above.

[0016] To address the aforementioned issues, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned FPGA-based dynamic reconstruction ultrawide spectrum image processing method.

[0017] To address the problems described in the background art, this invention identifies a visible light sensor, an infrared detector, and an ultra-wideband image processing module. The ultra-wideband image processing module includes a Zynq FPGA PS side, a Zynq FPGA PL side, an AXI bus, an ISP processing module, and a data interface conversion module. The visible light sensor captures image information in the visible light band, while the infrared detector acquires image information in the infrared band. The combination of these two provides richer spectral information. Driving the visible light sensor and infrared detector separately yields heterogeneous visible light and infrared images. By driving the visible light sensor and infrared detector, this invention can acquire image data in different spectral ranges. The heterogeneous visible light image contains information such as the color and texture of an object under visible light, while the heterogeneous infrared image reflects the thermal radiation characteristics of an object. The combination of these two heterogeneous images provides more comprehensive... The scene information provided helps in target recognition and monitoring applications in complex environments. The heterogeneous visible light image is input into the data interface conversion module to obtain the heterogeneous image to be processed. The FPGA's internal dedicated logic operations are then executed on the heterogeneous image to obtain a standardized visible light data stream. This standardized visible light data stream includes a set of visible light values ​​and valid visible light data signals. The visible light value set includes multiple visible light values. The FPGA's internal dedicated logic operations include physical layer conversion from differential signals to single-ended signals and cross-clock domain synchronization. The physical layer conversion from differential signals to single-ended signals in the FPGA's internal dedicated logic operations of this invention can improve the signal's anti-interference capabilities. This invention ensures stable data transmission, and cross-clock domain synchronization solves data transmission problems between different clock domains, preventing data loss or errors, thus obtaining a standardized visible light data stream. Based on heterogeneous infrared images, a standardized infrared data stream is acquired, comprising: a thermal infrared value set and valid thermal infrared data signals. This standardized infrared data stream facilitates subsequent fusion processing with the standardized visible light data stream, improving the collaborative processing capability between different spectral data. An ultrawideband image processing module is used to process the standardized visible light data stream and the standardized infrared data stream respectively, obtaining a YUV visible light image, an equalized thermal infrared image, and a sensor image. By controlling the parameters, the visible light images in the YUV format of this invention are more suitable for color processing and compression, reducing data volume while maintaining image quality. Equalizing the thermal infrared image enhances image contrast, highlights the thermal features of objects, and facilitates subsequent target detection and recognition. A pre-built dynamic algorithm routing unit restructures the ISP algorithm of the ultra-wideband image processing module, resulting in a restructured ISP processing pipeline. This invention's dynamic algorithm routing unit can flexibly restructure the ISP algorithm according to different application requirements and image characteristics. This dynamic reconstruction capability allows the system to quickly adapt to different scenarios and tasks, improving the system's versatility and flexibility.Based on YUV visible light images, equalized thermal infrared images, and a reconstructed ISP processing pipeline, this invention obtains the optimal ISP processing pipeline and semantically enhanced fused images. The optimal processing pipeline fully leverages the advantages of different spectral data, achieving more efficient image processing. The semantically enhanced fused image integrates information from visible light and infrared images, preserving not only the color and texture information of objects but also enhancing the semantic information of the image, improving readability and understandability, and facilitating more accurate target recognition and scene analysis. Based on the optimal ISP processing pipeline, semantically enhanced fused images, and sensor control parameters, ultra-wideband image processing based on FPGA dynamic reconstruction is completed. Therefore, this invention can improve FPGA resource utilization and significantly shorten the ISP algorithm reconstruction time. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an embodiment of the ultrawide spectrum image processing method based on FPGA dynamic reconstruction provided by the present invention. Figure 2 A functional block diagram of an ultrawide spectrum image processing system based on FPGA dynamic reconstruction provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device that implements the FPGA-based dynamic reconstruction ultrawide spectrum image processing method according to an embodiment of the present invention.

[0019] Explanation of reference numerals in the attached figures: 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0020] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0022] This application provides an ultrawide spectral image processing method based on FPGA dynamic reconstruction. The execution entity of the FPGA-based ultrawide spectral image processing method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the FPGA-based ultrawide spectral image processing method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0023] Reference Figure 1The diagram shown is a flowchart illustrating an ultrawide spectral image processing method based on FPGA dynamic reconstruction according to an embodiment of the present invention. In this embodiment, the ultrawide spectral image processing method based on FPGA dynamic reconstruction includes: S1. The visible light sensor, infrared detector, and ultrawide spectrum image processing module are identified. The ultrawide spectrum image processing module includes: Zynq FPGA PS side, Zynq FPGA PL side, AXI bus, ISP processing module, and data interface conversion module.

[0024] It needs to be explained that a visible light sensor is a sensor capable of detecting visible light (typically light with wavelengths in the 400-700 nanometer range), converting light signals into electrical signals to generate two-dimensional image data. An infrared detector is a sensor capable of detecting infrared light (typically light with wavelengths above 700 nanometers), generating thermal images to display the temperature distribution of objects. The ultra-wideband image processing module is the hardware core of the entire image processing system; it is responsible for driving the sensors and processing all the data they transmit. The Zynq FPGA PS side refers to the processor system side inside the FPGA. A Zynq FPGA is a chip that integrates an ARM processor and programmable logic (FPGA). The PS side contains the ARM processor and related peripheral interfaces for running the operating system and performing high-level control tasks. The Zynq FPGA PL side refers to the programmable logic side inside the FPGA. The PL side is the FPGA portion used to implement custom digital logic and algorithms, providing high flexibility and parallel processing capabilities, suitable for performing complex image processing tasks. The AXI bus is the bus used to transfer data between the Zynq FPGA PS side and the Zynq FPGA PL side. The ISP processing module refers to the image signal processing module. The ISP processing module is responsible for preprocessing the raw image data, including noise reduction, white balance adjustment, and color correction, to generate high-quality images. The data interface conversion module is used to convert data formats and interfaces.

[0025] S2. Drive the visible light sensor and the infrared detector respectively to obtain the visible light heterogeneous image and the infrared heterogeneous image.

[0026] It should be explained that visible light heterogeneous images refer to two-dimensional image data generated by visible light sensors. Infrared heterogeneous images refer to two-dimensional image data generated by infrared detectors.

[0027] S3. Input the visible light heterogeneous image into the data interface conversion module to obtain the heterogeneous image to be processed. Perform FPGA internal dedicated logic operations on the heterogeneous image to be processed to obtain a standardized visible light data stream.

[0028] Specifically, the standardized visible light data stream includes: a set of visible light values ​​and a valid visible light data signal. The set of visible light values ​​includes multiple visible light values. The dedicated logic operations inside the FPGA include: physical layer conversion of differential signals to single-ended signals and cross-clock domain synchronization.

[0029] It should be explained that the heterogeneous image to be processed refers to the visible light heterogeneous image input to the data interface conversion module. The FPGA's internal dedicated logic operation refers to the physical layer conversion of the heterogeneous image data from differential signals to single-ended signals, obtaining the converted heterogeneous image data, and then performing cross-clock domain synchronization on this converted heterogeneous image data. Single-ended signal is a signal transmission method that uses one signal line and one ground line to represent a data signal. Physical layer conversion is the process of converting one signal format to another, usually performed at the physical layer. Physical layer conversion ensures signal compatibility between different transmission media or interfaces. Cross-clock domain synchronization is the process of synchronizing data between different clock domains, ensuring correct data transmission between different clock domains and avoiding data loss or errors. This invention's cross-clock domain synchronization can be processed using a two-stage pipeline, handshake mechanism, or asynchronous FIFO. Visible light value refers to the value of a single pixel in the image data captured by the visible light sensor. A valid visible light data signal is a signal used to indicate the validity of the visible light value set, helping to correctly process data and avoid processing invalid data during processing.

[0030] S4. Obtain a standardized infrared data stream based on the infrared heterogeneous image, wherein the standardized infrared data stream includes: thermal infrared value set and valid thermal infrared data signal.

[0031] It should be explained that the method for obtaining a standardized infrared data stream based on heterogeneous infrared images is the same as the method for obtaining a standardized visible light data stream based on heterogeneous visible light images, and will not be repeated here. A thermal infrared value set refers to the collection of all thermal infrared values ​​in the image data captured by the infrared detector. A thermal infrared value refers to the brightness or temperature information of a pixel in the infrared band. A valid thermal infrared data signal is a signal used to indicate the validity of the thermal infrared value set.

[0032] S5. The ultrawide spectrum image processing module is used to process the standardized visible light data stream and the standardized infrared data stream respectively to obtain the YUV visible light image, the equalized thermal infrared image and the sensor control parameters.

[0033] In detail, the ultrawide spectral image processing module processes the standardized visible light data stream and the standardized infrared data stream respectively to obtain a YUV visible light image, an equalized thermal infrared image, and sensor control parameters, including: Visible light values ​​are extracted sequentially from the set of visible light values ​​in the standardized visible light data stream, and the following operations are performed on each extracted visible light value: If the visible light value is greater than the preset calibration black level value, the corrected visible light value is calculated based on the calibration black level value and the visible light value. The corrected visible light value is the value obtained by subtracting the calibration black level value from the visible light value. If the visible light value is less than or equal to the calibrated black level value, then the preset black level correction value is used as the corrected visible light value, wherein the black level correction value is 0; A digital gain operation is performed on the corrected visible light value to obtain the gain visible light value. The gain visible light values ​​are then summarized to obtain a gain visible light value set. A sensor bad pixel correction operation is then performed on the gain visible light value set to obtain a bad pixel corrected visible light pixel set. White balance correction is performed on the visible light pixel set for defective pixel correction to obtain a white balance visible light pixel set. Based on the white balance visible light pixel set, an RGB visible light image and sensor control parameters are obtained. An RGB to YUV conversion operation is performed on the RGB visible light image to obtain a YUV visible light image. The standardized infrared data stream is processed using an ultrawide spectrum image processing module to obtain an equalized thermal infrared image.

[0034] It needs to be explained that the visible light value refers to the raw data value extracted from visible light data, representing the brightness or color information of pixels in a visible light image. The calibrated black level value is a pre-set reference value used to represent the output level of the image sensor under no-light (or extremely low-light) conditions. In image processing, the calibrated black level value is used to correct for the effects of dark current noise and other factors in the image sensor. By comparing and calculating with the visible light value, dark current noise in the image can be removed, making the dark areas of the image more accurate. The gain visible light value is the value obtained after performing a digital gain operation on the calibrated visible light value. Digital gain is a technique for adjusting image brightness or contrast by multiplying by a gain coefficient to amplify or reduce the visible light value. The gain visible light value set is the set obtained by summing all the gain visible light values ​​after digital gain operation. The bad pixel correction visible light pixel set is the set of bad pixel corrected visible light pixels obtained by performing sensor bad pixel correction operation on the pixels corresponding to each gain visible light value in the gain visible light value set. Sensor bad pixels are abnormal pixels in the image sensor caused by physical defects or damage. Defective pixel correction can detect and repair these abnormal pixels, making the visible light image more uniform and accurate, and providing more reliable data for subsequent white balance and other operations. The white balance visible light pixel set refers to the set of white balance visible light pixels obtained by performing white balance correction on each defective pixel in the defective pixel correction visible light pixel set. White balance correction can eliminate color casts in the image, making the image colors more realistic and natural. The RGB-to-YUV conversion operation on the RGB visible light image refers to performing brightness and saturation correction on the RGB visible light image, and then sharpening the image after brightness and saturation correction. Both brightness and saturation correction and image sharpening are based on mature and well-known digital image processing technologies and can be fully implemented using existing techniques. Brightness and saturation correction are achieved by applying independent linear gain and offset operations to the YUV components, while image sharpening uses a standard unsharpening mask technique. This involves extracting the low-frequency components of the image through Gaussian filtering, multiplying the difference between the original image and the low-frequency image by a specific factor, and then superimposing it back into the original image to enhance high-frequency details. YUV visible light images are images obtained by performing an RGB-to-YUV conversion on an RGB visible light image.

[0035] Specifically, the step of performing sensor dead pixel correction on the gain visible light value set to obtain a dead pixel corrected visible light pixel set includes: Extract the gain visible light values ​​sequentially from the gain visible light value set, and perform the following operations on each extracted gain visible light value: Create a sliding window, take the gain visible light value as the center pixel, obtain the pixel color and center pixel value of the center pixel, and obtain the set of pixel values ​​of the same color based on the sliding window, the center pixel and the pixel color; Calculate the pixel difference between the central pixel value and each pixel value of the same color in the set of pixel values ​​of the same color to obtain the pixel difference set. Obtain the difference sign set based on the pixel difference set, where the difference sign corresponds one-to-one with the pixel difference. Determine whether all the difference signs in the difference sign set are the same; If all the difference signs in the difference sign set are the same, then the center pixel is regarded as a normal pixel. If the difference signs in the difference sign set are not all the same, then determine whether there is a pixel difference in the pixel difference set that is not greater than a preset difference threshold. If there is a pixel difference in the pixel difference set that is not greater than the preset difference threshold, then the center pixel is regarded as a bad pixel. Perform median filtering on the set of pixels of the same color to obtain the median pixel value. Use the median pixel value to correct bad pixels and obtain the updated pixel. If there is no pixel difference in the pixel difference set that is not greater than a preset difference threshold, then the center pixel is regarded as a normal pixel. By summarizing the updated pixels and normal pixels, a set of visible light pixels for bad pixel correction is obtained.

[0036] It should be explained that creating a sliding window refers to creating a 5x5 sliding window. The center pixel is the pixel located at the geometric center of the sliding window. Pixel color refers to the color information represented by the center pixel. In bad pixel correction, pixel color is used to determine which pixels share the same color as the center pixel for more accurate comparison and analysis. The set of pixel values ​​of the same color refers to the set of pixel values ​​within the sliding window that have the same color as the center pixel. The center pixel value refers to the specific numerical value of the center pixel in the image data. The pixel difference is the difference between the center pixel value and the pixel values ​​of the same color. The pixel difference measures the difference between the center pixel value and the neighboring pixel values ​​of the same color. By calculating the pixel difference, it can be determined whether the center pixel has a significant difference from its neighboring pixels of the same color, thus determining whether it is a bad pixel. The set of pixel difference values ​​refers to the set of pixel differences between all pixel values ​​of the same color and the center pixel value. The sign of the difference refers to the sign (positive or negative) of the pixel difference. The set of difference signs refers to the set of all difference signs. If all difference signs in the difference sign set are the same, it indicates that the difference direction between the center pixel and the same-color pixel is consistent, and it may not be a bad pixel. If the difference signs in the difference sign set are not all the same, it indicates that the center pixel may be abnormal. A normal pixel is the center pixel corresponding to a difference sign set where all difference signs are the same. The difference threshold is a pre-set threshold used to determine whether the pixel difference is not greater than the difference threshold. A bad pixel is the center pixel in the pixel difference set whose pixel difference is not greater than the preset difference threshold. Median filtering is an image processing technique that removes noise by sorting the pixel values ​​in the image and taking the median. The median pixel value is the median obtained after performing median filtering on the set of pixel values ​​of the same color. Correcting bad pixels using the median pixel value means replacing the bad pixel value corresponding to the bad pixel with the median pixel value. The updated pixel refers to the new center pixel after correction.

[0037] Specifically, the step of performing white balance correction on the visible light pixel set for defective pixel correction to obtain a white balance visible light pixel set includes: The bad pixel correction image is identified based on the visible light pixel set for bad pixel correction. The bad pixel correction image is then divided to obtain an image division region set, which includes multiple image division regions. Image regions are extracted sequentially from the image region set. Red pixel value set, green pixel value set and blue pixel value set are obtained according to the image region. The mean value of red pixel, green pixel and blue pixel are calculated according to the red pixel value set, green pixel value set and blue pixel value set. The region parameters are calculated based on the average values ​​of red, green, and blue pixels. The region parameters include the region brightness value and the region saturation value. If the region parameters meet the preset standard region parameter range conditions, then the image region is considered a valid image region. If the region parameters do not meet the preset standard region parameter range conditions, the image region is removed from the image region set to obtain an updated region set. The updated region set is used as the image region set, and the process of extracting image regions sequentially from the image region set is repeated until all image regions in the image region set have been extracted. The effective image regions are summarized to obtain the effective image region set. The effective image region set is then calculated using the Zynq FPGA PS side to obtain the channel gain factor set. White balance correction is performed on the visible light pixel set for bad pixel correction based on the channel gain factor set to obtain the white balance visible light pixel set.

[0038] It should be explained that a bad pixel corrected image refers to an image that has undergone bad pixel correction processing, in which all bad pixel pixel values ​​have been corrected to the updated pixel values. Dividing the bad pixel corrected image refers to dividing the image using a preset division value. For example, the division value is a value set manually in advance. For example, the division value is 64. An image region set refers to the set of all image regions. An image region refers to a specific region within the image region set. Red pixel values ​​refer to the set of all red pixel values ​​in the image region. Green pixel values ​​refer to the set of all green pixel values ​​in the image region. Blue pixel values ​​refer to the set of all blue pixel values ​​in the image region. The average red pixel value refers to the average value of the red pixel value set. The average green pixel value refers to the average value of the green pixel value set. The average blue pixel value refers to the average value of the blue pixel value set. The step of calculating the region parameters based on the average red pixel value, average green pixel value, and average blue pixel value is as follows: The region brightness value is calculated using the following formula, as shown below: in, Indicates the brightness value of the area. This represents the average value of the red pixels. This represents the average value of the green pixels. This represents the average value of blue pixels. The red-to-green ratio is calculated based on the average values ​​of red and green pixels, and the blue-to-green ratio is calculated based on the average values ​​of green and blue pixels. The absolute differences between these red-to-green and blue-to-green ratios and the preset ideal neutral value are then calculated. These absolute differences are used as the region's saturation value. The red-to-green ratio is the value obtained by dividing the average value of red pixels by the average value of green pixels. The blue-to-green ratio is the value obtained by dividing the average value of blue pixels by the average value of green pixels. The absolute difference between red and green pixels is the absolute difference between the red-to-green ratio and the ideal neutral value. The ideal neutral value is a pre-set value. For example, the ideal neutral value is 1.

[0039] Understandably, if the region parameters meet the preset standard region parameter range conditions, it means that the region brightness value is within the preset brightness range, the absolute red-green difference is within the preset absolute red-green difference range, and the absolute green-blue difference is within the preset absolute green-blue difference range. If the region parameters do not meet the preset standard region parameter range conditions, it means that the region brightness value is not within the preset brightness range, the absolute red-green difference is not within the preset absolute red-green difference range, or the absolute green-blue difference is not within the preset absolute green-blue difference range. A valid image region refers to an image region whose region parameters meet the preset standard region parameter range conditions. Updating the region set refers to the new image region set obtained by removing image regions that do not meet the standard region parameter conditions from the image region set. The valid image region set refers to the set composed of all valid image regions. The steps for calculating the channel gain factor set using the Zynq FPGA PS side to calculate the effective image region set are as follows: All red-green and blue-green ratios in the effective image region set are statistically analyzed. These ratios are then imported into the Zynq FPGA PS side. The red, green, and blue gain factors are calculated using the Zynq FPGA PS side. The red, green, and blue gain factors are then summarized to obtain the channel gain factor set. The green gain factor is a pre-defined factor. For example, the green gain factor is 1.0. Red gain factor = (green gain factor / all red-green ratios). Blue gain factor = (green gain factor / all blue-green ratios). The steps for performing white balance correction on the visible light pixel set based on the channel gain factor set are as follows: Visible light pixels for bad pixel correction are extracted sequentially from the visible light pixel set. The pixel color and value are determined based on these pixels. The target channel gain factor is determined from the channel gain factor set based on the pixel color. The white balance visible light pixel value is calculated based on the target channel gain factor. The calculation formula is as follows: in, This represents the visible light pixel value for white balance. This indicates the visible light pixel value for dead pixel correction. Indicates the calibrated black level value. This represents the target channel gain factor. Pixel color and bad pixel correction visible light pixel value refer to the color and pixel value of the bad pixel correction visible light pixel, respectively. The target channel gain factor refers to the channel gain factor corresponding to the extracted bad pixel correction visible light pixel.

[0040] Specifically, the acquisition of RGB visible light images and sensor control parameters based on a white balance visible light pixel set includes: Extract white balance visible light pixels sequentially from the white balance visible light pixel set, use the white balance visible light pixels as the center pixels to be denoised, create a denoising window, and obtain the set of visible light pixels to be denoised based on the center pixels to be denoised and the denoising window. Extract the visible light pixels to be denoised sequentially from the set of visible light pixels to be denoised, and perform the following operations on each extracted visible light pixel: The pre-constructed bilinear interpolation method is used to estimate the full color of the visible light pixels to be denoised, and the R value, G value and B value are obtained. The pixel gray value is calculated based on the R value, G value and B value. The pixel gray values ​​are summarized to obtain the window gray map corresponding to the set of visible light pixels to be denoised. Perform convolution operation on the window grayscale image to obtain the horizontal and vertical gradients, and calculate the gradient magnitude based on the horizontal and vertical gradients. If the gradient magnitude is less than the preset gradient threshold, then the center pixel to be denoised is subjected to a preset mean filter to obtain the filtered center pixel. If the gradient magnitude is greater than or equal to the preset gradient threshold, then the center pixel to be denoised is taken as the center pixel for denoising. By summing the center pixels of the filter and the center pixels of the noise reduction, a noise-reduced grayscale image is obtained. This grayscale image is then divided into grayscale partitioning sets. Using a pre-constructed center-weighted metering principle, weights are assigned to the grayscale partitioning sets to obtain region weight sets. The region weights are related to the grayscale partitioning... Figure 1 One-to-one correspondence; RGB visible light images and sensor control parameters are obtained based on grayscale partitioning atlases and region weight sets.

[0041] It should be explained that a denoising window is a fixed-size rectangular area that slides across the image to denoise each pixel and its neighborhood. For example, a denoising window is a 5x5 sliding window. The center pixel to be denoised is the pixel located at the geometric center of the denoising window. The set of visible light pixels to be denoised is the set of all visible light pixels to be denoised within the denoising window, excluding the center pixel. A specific pixel within the set of visible light pixels to be denoised is a single pixel. Bilinear interpolation is an image processing technique used to interpolate visible light pixels to obtain a smoother image. R, G, and B values ​​refer to the pixel values ​​of the red, green, and blue channels, respectively. The formula for calculating the pixel grayscale value in the step of calculating the pixel grayscale value based on the R, G, and B values ​​is as follows: in, Represents the pixel grayscale value. Indicates the R value. Indicates the G value. This represents the B value. A window grayscale image is an image formed by summing the grayscale values ​​of all pixels in the visible light pixel set to be denoised. The window grayscale image is used for convolution operations to calculate the gradients in the horizontal and vertical directions to determine edge and texture information in the image. The convolution operation on the window grayscale image to obtain the horizontal and vertical gradients refers to using the Sobel operator to perform convolution operations on the window grayscale image to obtain the horizontal and vertical gradients. The horizontal gradient is the gradient value of the window grayscale image in the horizontal direction calculated through convolution operations. The horizontal gradient is used to represent the brightness change of the image in the horizontal direction. The vertical gradient is the gradient value of the window grayscale image in the vertical direction calculated through convolution operations. The vertical gradient is used to represent the brightness change of the image in the vertical direction. The calculation formula for calculating the gradient magnitude based on the horizontal and vertical gradients is as follows: in, Indicates the gradient magnitude. Represents the gradient in the horizontal direction. This represents the vertical gradient. The gradient magnitude indicates the edge strength of the center pixel in the grayscale image to be denoised. A larger gradient magnitude indicates a more pronounced edge at the center pixel. The gradient threshold is a pre-set threshold used to determine whether the gradient magnitude is significant. If the gradient magnitude is less than the pre-set threshold, the edge at the center pixel is not pronounced and needs smoothing. If the gradient magnitude is greater than or equal to the pre-set threshold, the edge at the center pixel is pronounced and needs to be preserved.

[0042] Understandably, the filtered center pixel refers to the center pixel to be denoised after mean filtering. The denoised center pixel refers to the original center pixel without filtering. The denoised grayscale image refers to the image formed by combining all filtered center pixels and denoised center pixels. The method for dividing the denoised grayscale image is the same as the method for dividing the bad pixel correction image, and will not be repeated here. Center-weighted metering is a metering method that focuses on measuring the brightness of the central region of the image while considering the brightness of the surrounding regions. Center-weighted metering is used to assign weights to the grayscale image set, so that the brightness of the central region of the image contributes more to the overall brightness. The region weight set refers to the set of region weights assigned to each grayscale image according to the center-weighted metering principle.

[0043] Specifically, the acquisition of RGB visible light images and sensor control parameters based on grayscale partitioning atlases and region weight sets includes: Extract grayscale partition images sequentially from the grayscale partition image set, determine the target region weight from the region weight set based on the grayscale partition images, calculate the pixel mean based on the grayscale partition images, calculate the weighted average brightness value based on the pixel mean and the target region weight, and summarize the weighted average brightness values ​​to obtain the weighted average brightness value set. The ACE statistics are calculated based on the weighted average brightness value set. The ACE statistics are then fed back to the pre-built ARM processor via the AXI bus to obtain the sensor control parameters, which include: exposure time and exposure value. The denoised grayscale image is stretched to obtain a stretched grayscale image. Then, a demosaic operation is performed on the stretched grayscale image to obtain an RGB visible light image.

[0044] It should be explained that: Target region weight refers to the region weight corresponding to the current grayscale partition image extracted from the region weight set. Pixel mean refers to the average value of all pixels in the grayscale partition image. Weighted average brightness value is the product of pixel mean and target region weight. Weighted average brightness value set refers to the set of weighted average brightness values ​​of all grayscale partition images, used to calculate the brightness characteristics of the overall denoised grayscale image, providing a basis for subsequent exposure control. ACE statistic refers to the sum of all weighted average brightness values ​​in the weighted average brightness value set. ARM processor refers to a high-performance processor used to process data fed back from the image processing module and perform corresponding control. Exposure time refers to the length of time the sensor acquires the image. Exposure value refers to the sensitivity of the sensor when acquiring the image.

[0045] Importantly, the step of stretching the denoised grayscale image is as follows: The denoised center pixel is extracted sequentially from the denoised grayscale image; the denoised center pixel value is determined based on the denoised center pixel; the stretched grayscale value is calculated based on the denoised center pixel value, the calibrated black level value, and a preset stretching gain factor; and the stretched grayscale values ​​are summarized to obtain the stretched grayscale image. The denoised center pixel value refers to the pixel value corresponding to the denoised center pixel. In the step of calculating the stretched grayscale value based on the denoised center pixel value, the calibrated black level value, and the preset stretching gain factor, the stretched grayscale value = {(denoised center pixel value - calibrated black level value)}. The stretching gain factor is 1024. A stretched grayscale image refers to the image obtained after stretching a denoised grayscale image. Image stretching is used to adjust the contrast of the denoised grayscale image, making its brightness range more reasonable and facilitating subsequent demosaic operations. The demosaic operation on the stretched grayscale image refers to converting the RAW data in the stretched grayscale image into a color image where each pixel contains complete RGB information.

[0046] In detail, the process of using the ultrawide spectral image processing module to process the standardized infrared data stream to obtain an equalized thermal infrared image includes: In the ultrawide spectrum image processing module, a non-uniformity correction operation is performed on the thermal infrared value set in the standardized infrared data stream to obtain the correction pixel response value set. The correction pixel set is then identified based on the correction pixel response value set, where there is a one-to-one correspondence between the correction pixel and the correction pixel response value. The correction pixel response values ​​are extracted sequentially from the correction pixel response value set, a correction sliding window is created, the correction pixel response values ​​are used as the center pixel response values, and the window pixel response value sequence is obtained based on the correction sliding window and the center pixel response values. The pixel mean is calculated based on the window pixel response value sequence, and the pixel absolute difference is calculated based on the pixel mean and the center pixel response value. If the absolute difference of the pixels is greater than the preset standard deviation threshold, then the corrected pixel corresponding to the central pixel response value in the corrected pixel set is taken as a blind pixel, and the blind pixel is corrected to obtain the corrected blind pixel. If the absolute difference of the pixels is not greater than the preset standard deviation threshold, then the corrected pixel corresponding to the central pixel response value in the corrected pixel set is regarded as a normal pixel. By summing the corrected blind pixels and normal pixels, a set of corrected blind pixels is obtained. Then, 2D noise reduction is performed on the set of corrected blind pixels to obtain a set of denoised pixels. A dual-platform histogram equalization image enhancement operation is performed on the denoised pixel set to obtain an equalized thermal infrared image.

[0047] It should be explained that the step of performing non-uniformity correction on the thermal infrared value set in the standardized infrared data stream to obtain the corrected pixel response value set is as follows: obtain the low temperature pixel response value set and the high temperature pixel response value set, extract the low temperature pixel response value sequentially from the low temperature pixel response value set, and confirm the target high temperature pixel response value from the high temperature pixel response value set based on the low temperature pixel response value. The gain parameter is calculated based on the low temperature pixel response value, the target high temperature pixel response value, the preset standard low temperature value, and the preset standard high temperature value. The offset parameter is then calculated based on the gain parameter. The gain parameters and offset parameters are summarized separately to obtain the gain parameter set and offset parameter set. The original pixel set is obtained according to the thermal infrared value set. The original pixels are extracted from the original pixel set in turn. The target gain parameters and target offset parameters are confirmed from the gain parameter set and offset parameter set respectively based on the original pixels. The corrected pixel response value of the original pixel is calculated based on the target gain parameter and the target offset parameter. The corrected pixel response values ​​are then summarized to obtain a set of corrected pixel response values. Low-temperature pixel response value refers to the response value of a pixel to infrared radiation under low-temperature conditions. The low-temperature pixel response value set refers to the set of all low-temperature pixel response values ​​in a frame of an image acquired under low-temperature conditions. High-temperature pixel response value refers to the response value of a pixel to infrared radiation under high-temperature conditions. The high-temperature pixel response value set refers to the set of all high-temperature pixel response values ​​in a frame of an image acquired under high-temperature conditions. In this invention, the low-temperature pixel response value and the high-temperature pixel response value are in one-to-one correspondence. Determining the target high-temperature pixel response value from the high-temperature pixel response value set based on the low-temperature pixel response value means determining the position in the image under low-temperature conditions based on the low-temperature pixel response value, and then determining the corresponding high-temperature pixel response value in the image under high-temperature conditions based on that position; this is the target high-temperature pixel response value. The standard low-temperature value refers to a pre-set standard response value under low-temperature conditions. The standard high-temperature value refers to a pre-set standard response value under high-temperature conditions. In the step of calculating the gain parameter based on the low-temperature pixel response value, the target high-temperature pixel response value, the preset standard low-temperature value, and the preset standard high-temperature value, the gain parameter = (standard high-temperature value - standard low-temperature value) / (target high-temperature pixel response value - low-temperature pixel response value). In the step of calculating the offset parameter based on the gain parameter, the offset parameter = standard low-temperature value - gain parameter. Low-temperature pixel response value. Gain parameter set refers to the set of all gain parameters. Offset parameter set refers to the set of all offset parameters. Original pixel set refers to the set of all pixels in the uncorrected thermal infrared data. Target gain parameter and target offset parameter refer to the gain parameter and offset parameter corresponding to the original pixel, respectively. In the step of calculating the corrected pixel response value of the original pixel based on the target gain parameter and target offset parameter, the corrected pixel response value = target gain parameter. Original pixel value + target offset parameter. The original pixel value refers to the numerical value corresponding to the original pixel. The corrected pixel response value set refers to the set of all corrected pixel response values. Specifically, the corrected pixel set refers to the set of pixels corresponding to each corrected pixel response value in the corrected pixel response value set.

[0048] Furthermore, a corrected sliding window refers to a moving window used to extract data from a local area within a set of cell response values. For example, a corrected sliding window might be a 3x3 sliding window. The center cell response value refers to the response value of the center cell within the corrected sliding window. The window cell response value sequence refers to the sequence of response values ​​for all cells within the corrected sliding window, excluding the center cell response value. For example, a corrected sliding window might be a 3x3 sliding window, i.e. ,in, Indicates the response value of the center pixel. , , , , , , , This represents the sequence of window cell response values. The cell mean is the average of all window cell response values ​​in the sequence. The absolute cell difference is the absolute difference between the cell mean and the center cell response value. The standard deviation threshold is a pre-set threshold used to determine whether a corrected cell is a blind cell. A blind cell is a corrected cell corresponding to the center cell response value whose absolute cell difference is greater than the preset standard deviation threshold. Correcting blind cells refers to replacing the center cell response value corresponding to the blind cell with the cell mean. A corrected blind cell is a blind cell after correction. A normal cell is a center cell corresponding to the center cell response value whose absolute cell difference is not greater than the preset standard deviation threshold. The corrected blind cell set is a set containing both corrected blind cells and normal cells. 2D denoising of the corrected blind cell set refers to performing 2D denoising on the corrected blind cell set using a median filtering algorithm. The denoised cell set is the set of denoised cells after 2D denoising. The specific steps for performing dual-platform histogram equalization image enhancement on the denoised pixel set are existing technologies and will not be described in detail here.

[0049] S6. The ISP algorithm structure of the ultrawide spectrum image processing module is reorganized using the pre-built dynamic algorithm routing unit to obtain the reorganized ISP processing pipeline.

[0050] It should be explained that the dynamic algorithm routing unit is a control module used to dynamically change the signal routing path, thereby configuring and switching algorithm modules. In the ultrawideband image processing module, the dynamic algorithm routing unit is responsible for dynamically selecting and switching different ISP (Image Signal Processing) algorithm sub-modules based on external control signals (such as enable signals). This dynamic switching allows certain algorithm modules to be enabled or disabled as needed at runtime, thereby optimizing the processing flow and resource utilization. ISP algorithm structure reconfiguration refers to the reconfiguration of the connection and sequence of ISP algorithm modules through the dynamic algorithm routing unit. ISP algorithm structure reconfiguration allows the order and activation status of algorithm modules in the ISP processing flow to be adjusted at runtime. In this way, the path and sequence of image processing can be flexibly changed to adapt to different processing needs. For example, in some cases, it may be necessary to enable certain advanced image processing algorithms, while in other cases only basic processing steps may be required. Reconfiguring the ISP processing pipeline refers to the ISP processing flow after reconfiguration by the dynamic algorithm routing unit, including multiple ISP algorithm sub-modules.

[0051] S7. Based on YUV visible light images, equalized thermal infrared images, and reconstructed ISP processing pipelines, obtain the optimal ISP processing pipeline and semantically enhanced fused images.

[0052] In detail, the step of obtaining the optimal ISP processing pipeline and semantically enhanced fused image based on YUV visible light images, equalized thermal infrared images, and reconstructed ISP processing pipelines includes: A pre-built scene recognition network was used to perform scene recognition on YUV visible light images to obtain scene recognition results, including nighttime scenes and foggy scenes. The optimal parameter set is generated based on the pre-constructed parameter regression neural network and scene recognition results. The optimal parameter set is then sent to the reconstructed ISP processing pipeline using the AXI bus to obtain the optimal ISP processing pipeline. A pre-built multimodal data module is used to fuse YUV visible light images and equalized thermal infrared images to obtain a semantically enhanced fused image. The multimodal data module includes a pixel-level fusion unit, a semantic segmentation network, and a content-aware enhancement unit.

[0053] It should be explained that a scene recognition network refers to a neural network model used to identify and classify the scene type of an input image. For example, a scene recognition network is a convolutional neural network. A night scene refers to a scene at night or under low light conditions. A foggy scene refers to a scene in foggy weather or under low visibility conditions. A parametric regression neural network refers to a neural network model used to regress optimal processing parameters based on input conditions (such as scene type). The optimal parameter set refers to the set of optimal parameters generated by the scene recognition results and the parametric regression neural network. In this invention, the optimal parameters in the optimal parameter set correspond one-to-one with the sub-modules of the ISP algorithm in the reconstructed ISP processing pipeline. The optimal ISP processing pipeline refers to the ISP processing pipeline obtained after distributing the optimal parameter set to each sub-module of the ISP algorithm in the reconstructed ISP processing pipeline. The optimal ISP processing pipeline improves image quality by applying the optimal parameter set and adjusting various parameters of ISP processing to adapt to different scene types. The multimodal data module refers to an integrated module used to process multi-source data (such as visible light images and infrared images) to achieve data fusion. Semantically enhanced fused images refer to images generated after processing by a multimodal data module, which combine the characteristics of visible light images and infrared images, and enhance semantic information.

[0054] In detail, the process of fusing YUV visible light images and equalized thermal infrared images using a pre-constructed multimodal data module to obtain a semantically enhanced fused image includes: The pixel-level fusion unit is used to perform preliminary fusion of the YUV visible light image and the equalized thermal infrared image to obtain an initial fused image. The initial fused image is input into the semantic segmentation network to obtain the image to be segmented. Key target recognition is performed on the image to be segmented to obtain a target segmentation mask. The initial fused image is segmented according to the target segmentation mask to obtain a set of key target regions and a set of non-key target regions. The set of key target regions includes one or more key target regions, and the set of non-key target regions includes one or more non-key target regions. The key target regions include pedestrians or vehicles. The content-aware enhancement unit performs local image enhancement operations on each key target region in the key target region set to obtain a set of locally enhanced regions. The content-aware enhancement unit performs lightweight processing operations on each non-critical target region in the non-critical target region set to obtain a lightweight region set. The locally enhanced region set and the lightweight region set are stitched together to obtain a semantically enhanced fused image.

[0055] It should be explained that a pixel-level fusion unit refers to a processing unit used to fuse image data from different modalities at the pixel level. Preliminary fusion refers to the weighted summation of pixel values ​​at each corresponding position in the YUV visible light image and the equalized thermal infrared image. This fusion method effectively preserves the advantages of both images, such as the rich color and high resolution of the visible light image, and the thermal information and penetration capability of the infrared image, providing a more comprehensive and accurate visual data foundation for subsequent image processing and analysis. The initial fused image refers to the image obtained after the preliminary fusion of the YUV visible light image and the equalized thermal infrared image by the pixel-level fusion unit. A semantic segmentation network refers to a neural network model used to segment an image into different semantic regions (such as sky, roads, buildings, etc.). For example, semantic segmentation networks use deep learning techniques, such as segmentation networks (such as U-Net, DeepLab, etc.). The image to be segmented refers to the initial fused image input into the semantic segmentation network. Key target recognition of the image to be segmented refers to identifying important target objects, such as pedestrians or vehicles, in the image to be segmented. The target segmentation mask refers to the mask generated by the semantic segmentation network, used to mark the location and shape of key targets. The target segmentation mask of this invention is a binary image of the same size as the input image, where the target region is marked as 1 and other regions as 0. A key target region set refers to a collection of one or more key target regions. Non-key target regions refer to all regions in the image except for the key target regions. A content-aware enhancement unit is a processing unit used to enhance specific regions or features of the image based on image content. Local image enhancement operations enhance specific regions of the image corresponding to key target regions to highlight the features and details of the target. A local enhancement region set refers to the collection of key target regions after local image enhancement operations. Lightweight processing operations refer to simple processing operations (such as filtering, noise reduction, etc.) performed on non-key target regions, aiming to reduce processing time and resource consumption without affecting the overall image quality. Lightweight regions refer to non-key target regions after lightweight processing operations. The process of stitching together the local enhancement region set and the lightweight region set involves preparing a blank canvas of the same size as the original image and then determining the position of each region based on the target segmentation mask. The target segmentation mask is like a map, marking the locations of key and non-key target regions. Each local enhancement region in the local enhancement region set is precisely placed on the corresponding position on the canvas. These regions will cover the corresponding key target parts in the initial fused image. Similarly, each non-key target region in the lightweight region set is placed on the corresponding position on the blank canvas, ultimately resulting in a semantically enhanced fused image.

[0056] S8. Based on the optimal ISP processing pipeline, semantic enhancement fusion image and sensor control parameters, complete the ultrawide spectrum image processing based on FPGA dynamic reconstruction.

[0057] Importantly, the FPGA-based dynamic reconfiguration method of this invention can fully leverage the parallel processing capabilities and flexibility of FPGAs to achieve efficient and real-time ultrawideband image processing. Simultaneously, the feedback mechanism of sensor control parameters can continuously optimize the image acquisition process, improve image quality, and enable the system to operate stably and reliably under different environments and tasks.

[0058] To address the problems described in the background art, this invention identifies a visible light sensor, an infrared detector, and an ultra-wideband image processing module. The ultra-wideband image processing module includes a Zynq FPGA PS side, a Zynq FPGA PL side, an AXI bus, an ISP processing module, and a data interface conversion module. The visible light sensor captures image information in the visible light band, while the infrared detector acquires image information in the infrared band. The combination of these two provides richer spectral information. Driving the visible light sensor and infrared detector separately yields heterogeneous visible light and infrared images. By driving the visible light sensor and infrared detector, this invention can acquire image data in different spectral ranges. The heterogeneous visible light image contains information such as the color and texture of an object under visible light, while the heterogeneous infrared image reflects the thermal radiation characteristics of an object. The combination of these two heterogeneous images provides more comprehensive... The scene information provided helps in target recognition and monitoring applications in complex environments. The heterogeneous visible light image is input into the data interface conversion module to obtain the heterogeneous image to be processed. The FPGA's internal dedicated logic operations are then executed on the heterogeneous image to obtain a standardized visible light data stream. This standardized visible light data stream includes a set of visible light values ​​and valid visible light data signals. The visible light value set includes multiple visible light values. The FPGA's internal dedicated logic operations include physical layer conversion from differential signals to single-ended signals and cross-clock domain synchronization. The physical layer conversion from differential signals to single-ended signals in the FPGA's internal dedicated logic operations of this invention can improve the signal's anti-interference capabilities. This invention ensures stable data transmission, and cross-clock domain synchronization solves data transmission problems between different clock domains, preventing data loss or errors, thus obtaining a standardized visible light data stream. Based on heterogeneous infrared images, a standardized infrared data stream is acquired, comprising: a thermal infrared value set and valid thermal infrared data signals. This standardized infrared data stream facilitates subsequent fusion processing with the standardized visible light data stream, improving the collaborative processing capability between different spectral data. An ultrawideband image processing module is used to process the standardized visible light data stream and the standardized infrared data stream respectively, obtaining a YUV visible light image, an equalized thermal infrared image, and a sensor image. By controlling the parameters, the visible light images in the YUV format of this invention are more suitable for color processing and compression, reducing data volume while maintaining image quality. Equalizing the thermal infrared image enhances image contrast, highlights the thermal features of objects, and facilitates subsequent target detection and recognition. A pre-built dynamic algorithm routing unit restructures the ISP algorithm of the ultra-wideband image processing module, resulting in a restructured ISP processing pipeline. This invention's dynamic algorithm routing unit can flexibly restructure the ISP algorithm according to different application requirements and image characteristics. This dynamic reconstruction capability allows the system to quickly adapt to different scenarios and tasks, improving the system's versatility and flexibility.Based on YUV visible light images, equalized thermal infrared images, and a reconstructed ISP processing pipeline, this invention obtains the optimal ISP processing pipeline and semantically enhanced fused images. The optimal processing pipeline fully leverages the advantages of different spectral data, achieving more efficient image processing. The semantically enhanced fused image integrates information from visible light and infrared images, preserving not only the color and texture information of objects but also enhancing the semantic information of the image, improving readability and understandability, and facilitating more accurate target recognition and scene analysis. Based on the optimal ISP processing pipeline, semantically enhanced fused images, and sensor control parameters, ultra-wideband image processing based on FPGA dynamic reconstruction is completed. Therefore, this invention can improve FPGA resource utilization and significantly shorten the ISP algorithm reconstruction time.

[0059] like Figure 2 The diagram shown is a functional block diagram of an ultrawide spectrum image processing system based on FPGA dynamic reconstruction provided in an embodiment of the present invention.

[0060] The FPGA-based dynamic reconstruction ultrawide spectral image processing system 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the FPGA-based dynamic reconstruction ultrawide spectral image processing system 100 may include a sensor driving module 101, a data normalization processing module 102, an algorithm reconstruction module 103, and an FPGA dynamic reconstruction completion module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device. The sensor driving module 101 is used to identify the visible light sensor, the infrared detector, and the ultrawide spectrum image processing module. The ultrawide spectrum image processing module includes: a Zynq FPGA PS side, a Zynq FPGA PL side, an AXI bus, an ISP processing module, and a data interface conversion module, which drive the visible light sensor and the infrared detector respectively to obtain visible light heterogeneous images and infrared heterogeneous images. The data standardization processing module 102 is used to input a visible light heterogeneous image into the data interface conversion module to obtain a heterogeneous image to be processed, and to perform FPGA internal dedicated logic operations on the heterogeneous image to be processed to obtain a standardized visible light data stream. The standardized visible light data stream includes: a set of visible light values ​​and a valid visible light data signal. The set of visible light values ​​includes multiple visible light values. The FPGA internal dedicated logic operations include: physical layer conversion of differential signals to single-ended signals and cross-clock domain synchronization. A standardized infrared data stream is obtained based on the infrared heterogeneous image. The standardized infrared data stream includes: a set of thermal infrared values ​​and a valid thermal infrared data signal. The algorithm reconstruction module 103 is used to process the standardized visible light data stream and the standardized infrared data stream using the ultrawide spectrum image processing module to obtain YUV visible light image, equalized thermal infrared image and sensor control parameters. The pre-built dynamic algorithm routing unit is used to reconstruct the ISP algorithm structure of the ultrawide spectrum image processing module to obtain the reconstructed ISP processing pipeline. The FPGA dynamic reconstruction completion module 104 is used to obtain the optimal ISP processing pipeline and semantic enhancement fusion image based on YUV visible light image, equalized thermal infrared image and reconstructed ISP processing pipeline, and complete ultrawide spectrum image processing based on FPGA dynamic reconstruction based on the optimal ISP processing pipeline, semantic enhancement fusion image and sensor control parameters.

[0061] In detail, the modules in the FPGA-based dynamic reconstruction ultrawide spectrum image processing system 100 described in this embodiment of the invention employ the same methods as described above. Figure 1 The method uses the same techniques as the FPGA-based dynamic reconstruction ultrawide spectrum image processing method described in the article and can produce the same technical effects, so it will not be repeated here.

[0062] like Figure 3 The diagram shown is a schematic representation of an electronic device that implements an ultrawideband image processing method based on FPGA dynamic reconstruction, according to an embodiment of the present invention.

[0063] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a program for an ultrawide spectrum image processing method based on FPGA dynamic reconstruction.

[0064] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as the portable hard drive of the electronic device 1. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of an ultra-wideband image processing method program based on FPGA dynamic reconstruction, but also to temporarily store data that has been output or will be output.

[0065] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., an ultrawideband image processing method program based on FPGA dynamic reconstruction) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0066] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0067] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0068] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0069] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0070] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0071] The ultrawide spectrum image processing method program based on FPGA dynamic reconstruction stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following: The visible light sensor, infrared detector, and ultrawide spectrum image processing module were identified. The ultrawide spectrum image processing module includes: Zynq FPGA PS side, Zynq FPGA PL side, AXI bus, ISP processing module, and data interface conversion module. The visible light sensor and the infrared detector are driven separately to obtain heterogeneous visible light images and heterogeneous infrared images; A heterogeneous visible light image is input into the data interface conversion module to obtain a heterogeneous image to be processed. The heterogeneous image to be processed is subjected to dedicated logic operations within the FPGA to obtain a standardized visible light data stream. The standardized visible light data stream includes: a set of visible light values ​​and a valid visible light data signal. The set of visible light values ​​includes multiple visible light values. The dedicated logic operations within the FPGA include: physical layer conversion of differential signals to single-ended signals and cross-clock domain synchronization. Standardized infrared data streams are acquired based on heterogeneous infrared images. The standardized infrared data streams include: thermal infrared value sets and valid thermal infrared data signals. The standardized visible light data stream and the standardized infrared data stream were processed by the ultrawide spectrum image processing module to obtain YUV visible light images, equalized thermal infrared images and sensor control parameters. The ISP algorithm structure of the ultrawide spectrum image processing module is restructured using a pre-built dynamic algorithm routing unit to obtain a restructured ISP processing pipeline; The optimal ISP processing pipeline and semantically enhanced fused image are obtained based on YUV visible light images, equalized thermal infrared images, and reconstructed ISP processing pipelines. Ultrawide spectral image processing based on FPGA dynamic reconstruction is achieved by using the optimal ISP processing pipeline, semantically enhanced fused images, and sensor control parameters.

[0072] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0073] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0074] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: The visible light sensor, infrared detector, and ultrawide spectrum image processing module were identified. The ultrawide spectrum image processing module includes: Zynq FPGA PS side, Zynq FPGA PL side, AXI bus, ISP processing module, and data interface conversion module. The visible light sensor and the infrared detector are driven separately to obtain heterogeneous visible light images and heterogeneous infrared images; A heterogeneous visible light image is input into the data interface conversion module to obtain a heterogeneous image to be processed. The heterogeneous image to be processed is subjected to dedicated logic operations within the FPGA to obtain a standardized visible light data stream. The standardized visible light data stream includes: a set of visible light values ​​and a valid visible light data signal. The set of visible light values ​​includes multiple visible light values. The dedicated logic operations within the FPGA include: physical layer conversion of differential signals to single-ended signals and cross-clock domain synchronization. Standardized infrared data streams are acquired based on heterogeneous infrared images. The standardized infrared data streams include: thermal infrared value sets and valid thermal infrared data signals. The standardized visible light data stream and the standardized infrared data stream were processed by the ultrawide spectrum image processing module to obtain YUV visible light images, equalized thermal infrared images and sensor control parameters. The ISP algorithm structure of the ultrawide spectrum image processing module is restructured using a pre-built dynamic algorithm routing unit to obtain a restructured ISP processing pipeline; The optimal ISP processing pipeline and semantically enhanced fused image are obtained based on YUV visible light images, equalized thermal infrared images, and reconstructed ISP processing pipelines. Ultrawide spectral image processing based on FPGA dynamic reconstruction is achieved by using the optimal ISP processing pipeline, semantically enhanced fused images, and sensor control parameters.

[0075] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0076] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0078] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for processing ultrawide spectral images based on FPGA dynamic reconstruction, characterized in that, The method includes: The visible light sensor, infrared detector, and ultrawide spectrum image processing module were identified. The ultrawide spectrum image processing module includes: Zynq FPGA PS side, Zynq FPGA PL side, AXI bus, ISP processing module, and data interface conversion module. The visible light sensor and the infrared detector are driven separately to obtain heterogeneous visible light images and heterogeneous infrared images; A heterogeneous visible light image is input into the data interface conversion module to obtain a heterogeneous image to be processed. The heterogeneous image to be processed is subjected to dedicated logic operations within the FPGA to obtain a standardized visible light data stream. The standardized visible light data stream includes: a set of visible light values ​​and a valid visible light data signal. The set of visible light values ​​includes multiple visible light values. The dedicated logic operations within the FPGA include: physical layer conversion of differential signals to single-ended signals and cross-clock domain synchronization. Standardized infrared data streams are acquired based on heterogeneous infrared images. The standardized infrared data streams include: thermal infrared value sets and valid thermal infrared data signals. The standardized visible light data stream and the standardized infrared data stream were processed by the ultrawide spectrum image processing module to obtain YUV visible light images, equalized thermal infrared images and sensor control parameters. The ISP algorithm structure of the ultrawide spectrum image processing module is restructured using a pre-built dynamic algorithm routing unit to obtain a restructured ISP processing pipeline; The optimal ISP processing pipeline and semantically enhanced fused image are obtained based on YUV visible light images, equalized thermal infrared images, and reconstructed ISP processing pipelines. Ultrawide spectral image processing based on FPGA dynamic reconstruction is achieved by using the optimal ISP processing pipeline, semantically enhanced fused images, and sensor control parameters.

2. The ultrawide spectral image processing method based on FPGA dynamic reconstruction as described in claim 1, characterized in that, The ultrawideband image processing module processes the standardized visible light data stream and the standardized infrared data stream respectively to obtain a YUV visible light image, an equalized thermal infrared image, and sensor control parameters, including: Visible light values ​​are extracted sequentially from the set of visible light values ​​in the standardized visible light data stream, and the following operations are performed on each extracted visible light value: If the visible light value is greater than the preset calibration black level value, the corrected visible light value is calculated based on the calibration black level value and the visible light value. The corrected visible light value is the value obtained by subtracting the calibration black level value from the visible light value. If the visible light value is less than or equal to the calibrated black level value, then the preset black level correction value is used as the corrected visible light value, wherein the black level correction value is 0; A digital gain operation is performed on the corrected visible light value to obtain the gain visible light value. The gain visible light values ​​are then summarized to obtain a gain visible light value set. A sensor bad pixel correction operation is then performed on the gain visible light value set to obtain a bad pixel corrected visible light pixel set. White balance correction is performed on the visible light pixel set for defective pixel correction to obtain a white balance visible light pixel set. Based on the white balance visible light pixel set, an RGB visible light image and sensor control parameters are obtained. An RGB to YUV conversion operation is performed on the RGB visible light image to obtain a YUV visible light image. The standardized infrared data stream is processed using an ultrawide spectrum image processing module to obtain an equalized thermal infrared image.

3. The ultrawide spectral image processing method based on FPGA dynamic reconstruction as described in claim 2, characterized in that, The step of performing sensor bad pixel correction on the gain visible light value set to obtain a bad pixel corrected visible light pixel set includes: Extract the gain visible light values ​​sequentially from the gain visible light value set, and perform the following operations on each extracted gain visible light value: Create a sliding window, take the gain visible light value as the center pixel, obtain the pixel color and center pixel value of the center pixel, and obtain the set of pixel values ​​of the same color based on the sliding window, the center pixel and the pixel color; Calculate the pixel difference between the central pixel value and each pixel value of the same color in the set of pixel values ​​of the same color to obtain the pixel difference set. Obtain the difference sign set based on the pixel difference set, where the difference sign corresponds one-to-one with the pixel difference. Determine whether all the difference signs in the difference sign set are the same; If all the difference signs in the difference sign set are the same, then the center pixel is regarded as a normal pixel. If the difference signs in the difference sign set are not all the same, then determine whether there is a pixel difference in the pixel difference set that is not greater than a preset difference threshold. If there is a pixel difference in the pixel difference set that is not greater than the preset difference threshold, then the center pixel is regarded as a bad pixel. Perform median filtering on the set of pixels of the same color to obtain the median pixel value. Use the median pixel value to correct bad pixels and obtain the updated pixel. If there is no pixel difference in the pixel difference set that is not greater than a preset difference threshold, then the center pixel is regarded as a normal pixel. By summarizing the updated pixels and normal pixels, a set of visible light pixels for bad pixel correction is obtained.

4. The ultrawide spectral image processing method based on FPGA dynamic reconstruction as described in claim 3, characterized in that, The step of performing white balance correction on the visible light pixel set for defective pixel correction to obtain a white balance visible light pixel set includes: The bad pixel correction image is identified based on the visible light pixel set for bad pixel correction. The bad pixel correction image is then divided to obtain an image division region set, which includes multiple image division regions. Image regions are extracted sequentially from the image region set. Red pixel value set, green pixel value set and blue pixel value set are obtained according to the image region. The mean value of red pixel, green pixel and blue pixel are calculated according to the red pixel value set, green pixel value set and blue pixel value set. The region parameters are calculated based on the average values ​​of red, green, and blue pixels. The region parameters include the region brightness value and the region saturation value. If the region parameters meet the preset standard region parameter range conditions, then the image region is considered a valid image region. If the region parameters do not meet the preset standard region parameter range conditions, the image region is removed from the image region set to obtain an updated region set. The updated region set is used as the image region set, and the process of extracting image regions sequentially from the image region set is repeated until all image regions in the image region set have been extracted. The effective image regions are summarized to obtain the effective image region set. The effective image region set is then calculated using the Zynq FPGA PS side to obtain the channel gain factor set. White balance correction is performed on the visible light pixel set for bad pixel correction based on the channel gain factor set to obtain the white balance visible light pixel set.

5. The ultrawide spectral image processing method based on FPGA dynamic reconstruction as described in claim 4, characterized in that, The acquisition of RGB visible light images and sensor control parameters based on white balance visible light pixel sets includes: Extract white balance visible light pixels sequentially from the white balance visible light pixel set, use the white balance visible light pixels as the center pixels to be denoised, create a denoising window, and obtain the set of visible light pixels to be denoised based on the center pixels to be denoised and the denoising window. Extract the visible light pixels to be denoised sequentially from the set of visible light pixels to be denoised, and perform the following operations on each extracted visible light pixel: The pre-constructed bilinear interpolation method is used to estimate the full color of the visible light pixels to be denoised, and the R value, G value and B value are obtained. The pixel gray value is calculated based on the R value, G value and B value. The pixel gray values ​​are summarized to obtain the window gray map corresponding to the set of visible light pixels to be denoised. Perform convolution operation on the window grayscale image to obtain the horizontal and vertical gradients, and calculate the gradient magnitude based on the horizontal and vertical gradients. If the gradient magnitude is less than the preset gradient threshold, then the center pixel to be denoised is subjected to a preset mean filter to obtain the filtered center pixel. If the gradient magnitude is greater than or equal to the preset gradient threshold, then the center pixel to be denoised is taken as the center pixel for denoising. The filtering center pixels and the noise reduction center pixels are summarized to obtain a noise reduction grayscale image. The noise reduction grayscale image is divided to obtain a grayscale partitioning image set. The grayscale partitioning image set is weighted using the pre-constructed center-weighted metering principle to obtain a region weight set, where the region weights correspond one-to-one with the grayscale partitioning image. RGB visible light images and sensor control parameters are obtained based on grayscale partitioning atlases and region weight sets.

6. The ultrawide spectral image processing method based on FPGA dynamic reconstruction as described in claim 5, characterized in that, The process of obtaining RGB visible light images and sensor control parameters based on grayscale partitioning atlases and region weight sets includes: Extract grayscale partition images sequentially from the grayscale partition image set, determine the target region weight from the region weight set based on the grayscale partition images, calculate the pixel mean based on the grayscale partition images, calculate the weighted average brightness value based on the pixel mean and the target region weight, and summarize the weighted average brightness values ​​to obtain the weighted average brightness value set. The ACE statistics are calculated based on the weighted average brightness value set. The ACE statistics are then fed back to the pre-built ARM processor via the AXI bus to obtain the sensor control parameters, which include: exposure time and exposure value. The denoised grayscale image is stretched to obtain a stretched grayscale image. Then, a demosaic operation is performed on the stretched grayscale image to obtain an RGB visible light image.

7. The ultrawide spectral image processing method based on FPGA dynamic reconstruction as described in claim 6, characterized in that, The process of using an ultrawide spectral image processing module to process the standardized infrared data stream to obtain an equalized thermal infrared image includes: In the ultrawide spectrum image processing module, a non-uniformity correction operation is performed on the thermal infrared value set in the standardized infrared data stream to obtain the correction pixel response value set. The correction pixel set is then identified based on the correction pixel response value set, where there is a one-to-one correspondence between the correction pixel and the correction pixel response value. The correction pixel response values ​​are extracted sequentially from the correction pixel response value set, a correction sliding window is created, the correction pixel response values ​​are used as the center pixel response values, and the window pixel response value sequence is obtained based on the correction sliding window and the center pixel response values. The pixel mean is calculated based on the window pixel response value sequence, and the pixel absolute difference is calculated based on the pixel mean and the center pixel response value. If the absolute difference of the pixels is greater than the preset standard deviation threshold, then the corrected pixel corresponding to the central pixel response value in the corrected pixel set is taken as a blind pixel, and the blind pixel is corrected to obtain the corrected blind pixel. If the absolute difference of the pixels is not greater than the preset standard deviation threshold, then the corrected pixel corresponding to the central pixel response value in the corrected pixel set is regarded as a normal pixel. By summing the corrected blind pixels and normal pixels, a set of corrected blind pixels is obtained. Then, 2D noise reduction is performed on the set of corrected blind pixels to obtain a set of denoised pixels. A dual-platform histogram equalization image enhancement operation is performed on the denoised pixel set to obtain an equalized thermal infrared image.

8. The ultrawide spectral image processing method based on FPGA dynamic reconstruction as described in claim 7, characterized in that, The process of obtaining the optimal ISP processing pipeline and semantically enhanced fused image based on YUV visible light images, equalized thermal infrared images, and reconstructed ISP processing pipelines includes: A pre-built scene recognition network was used to perform scene recognition on YUV visible light images to obtain scene recognition results, including nighttime scenes and foggy scenes. The optimal parameter set is generated based on the pre-constructed parameter regression neural network and scene recognition results. The optimal parameter set is then sent to the reconstructed ISP processing pipeline using the AXI bus to obtain the optimal ISP processing pipeline. A pre-built multimodal data module is used to fuse YUV visible light images and equalized thermal infrared images to obtain a semantically enhanced fused image. The multimodal data module includes a pixel-level fusion unit, a semantic segmentation network, and a content-aware enhancement unit.

9. The ultrawide spectral image processing method based on FPGA dynamic reconstruction as described in claim 8, characterized in that, The method of fusing YUV visible light images and equalized thermal infrared images using a pre-constructed multimodal data module to obtain a semantically enhanced fused image includes: The pixel-level fusion unit is used to perform preliminary fusion of the YUV visible light image and the equalized thermal infrared image to obtain an initial fused image. The initial fused image is input into the semantic segmentation network to obtain the image to be segmented. Key target recognition is performed on the image to be segmented to obtain a target segmentation mask. The initial fused image is segmented according to the target segmentation mask to obtain a set of key target regions and a set of non-key target regions. The set of key target regions includes one or more key target regions, and the set of non-key target regions includes one or more non-key target regions. The key target regions include pedestrians or vehicles. The content-aware enhancement unit performs local image enhancement operations on each key target region in the key target region set to obtain a set of locally enhanced regions. The content-aware enhancement unit performs lightweight processing operations on each non-critical target region in the non-critical target region set to obtain a lightweight region set. The locally enhanced region set and the lightweight region set are stitched together to obtain a semantically enhanced fused image.

10. An ultrawideband image processing system based on FPGA dynamic reconstruction, characterized in that, The system includes: The sensor driver module is used to identify the visible light sensor, the infrared detector, and the ultrawide spectrum image processing module. The ultrawide spectrum image processing module includes: Zynq FPGA PS side, Zynq FPGA PL side, AXI bus, ISP processing module, and data interface conversion module, which drive the visible light sensor and the infrared detector respectively to obtain visible light heterogeneous images and infrared heterogeneous images. The data standardization processing module is used to input a visible light heterogeneous image into the data interface conversion module to obtain a heterogeneous image to be processed, and to perform FPGA internal dedicated logic operations on the heterogeneous image to be processed to obtain a standardized visible light data stream. The standardized visible light data stream includes: a set of visible light values ​​and a valid visible light data signal. The set of visible light values ​​includes multiple visible light values. The FPGA internal dedicated logic operations include: physical layer conversion of differential signals to single-ended signals and cross-clock domain synchronization. A standardized infrared data stream is obtained based on the infrared heterogeneous image. The standardized infrared data stream includes: a set of thermal infrared values ​​and a valid thermal infrared data signal. The algorithm reconstruction module is used to process the standardized visible light data stream and the standardized infrared data stream using the ultrawide spectrum image processing module to obtain YUV visible light images, equalized thermal infrared images and sensor control parameters. The pre-built dynamic algorithm routing unit is used to reconstruct the ISP algorithm structure of the ultrawide spectrum image processing module to obtain the reconstructed ISP processing pipeline. The FPGA dynamic reconstruction module is used to obtain the optimal ISP processing pipeline and semantically enhanced fused image based on YUV visible light image, equalized thermal infrared image and reconstructed ISP processing pipeline. Based on the optimal ISP processing pipeline, semantically enhanced fused image and sensor control parameters, it completes ultrawide spectrum image processing based on FPGA dynamic reconstruction.

Images