Real-time automatic hyper-parameter optimization method and system for ISP
Through the combination of parameter network and ISP proxy model, ISP hyperparameters are directly predicted from the original image and tuned online, solving the real-time and adaptability problems of the ISP parameter tuning method, and achieving efficient image quality and processing speed improvement , provides strong support for smart edge devices.
Patent Information
- Application Number
- CN202410968307.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
The existing ISP parameter tuning methods have problems such as manual tuning is time-consuming and labor-intensive, automatic tuning is difficult to adapt to real-time changing scenarios, and it is difficult to meet the high real-time performance requirements of smart edge devices.
Using the combination of parametric network ParaNet and ISP proxy model, the hyperparameters are predicted directly from the original image by simulating the indifferentiated ISP hardware, and real-time parameter optimization is achieved through online tuning, and a dual-path hardware architecture is used to achieve real-time processing.
It significantly improves image quality and processing speed, is suitable for various practical application scenarios of smart edge devices, realizes real-time online search and fast and accurate optimization of ISP parameters, and provides strong support for real-time automated parameter optimization of ISP hardware systems.
Smart Images

Figure CN118941436A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image signal processing, and in particular relates to a real-time automatic hyperparameter optimization method and system for an ISP hardware system. Background Art
[0002] The main task of the image signal processing (ISP) hardware system is to convert the photoelectric signal captured by the camera sensor into a standard RGB image for subsequent image analysis and processing. In this process, the ISP hardware needs to execute complex image processing algorithms, including denoising, demosaicing, automatic white balance, color correction, sharpening, etc. The effect of the algorithm depends largely on the settings of multiple hyperparameters, such as denoising strength, white balance coefficient, etc.
[0003] At present, the parameter tuning of ISP hardware systems relies on manual experience. Engineers manually adjust various hyperparameters in the hope of obtaining the best image quality. This method is not only time-consuming and labor-intensive, but also requires a high level of expertise and experience from engineers. In addition, manual tuning is often targeted at specific shooting scenes and is difficult to adapt to the complex and changing environments in actual applications. In order to address the shortcomings of manual tuning, researchers have proposed some automatic tuning methods in recent years. For example, some methods use image quality assessment indicators as loss functions to train neural network models to automatically generate ISP parameters. However, most of these methods rely on offline training and optimization, and are unable to dynamically adjust ISP parameters in real-time applications, making it difficult to adapt to real-time changing scenarios in actual applications.
[0004] In addition, with the widespread application of intelligent edge devices, the real-time performance requirements of ISP hardware systems are becoming increasingly higher. ISP hardware needs to complete image processing within milliseconds. Traditional offline tuning methods and complex proxy models are often difficult to meet this high real-time requirement in practical applications, becoming a bottleneck limiting the performance improvement of ISP hardware systems.
[0005] In summary, existing ISP parameter tuning methods have problems such as manual tuning being time-consuming and labor-intensive, and automatic tuning being difficult to adapt to real-time changing scenarios. In order to solve these problems, the present invention proposes a real-time automatic hyperparameter optimization framework, which realizes real-time online optimization of ISP parameters by combining a parameter network and an online tuning module. The framework performs well in improving image quality and processing speed, and is suitable for various practical application scenarios of intelligent edge devices. Summary of the invention
[0006] The purpose of the present invention is to propose a real-time automatic hyperparameter optimization method for ISP to address the shortcomings of the ISP parameter tuning method in the prior art, realize real-time online search of ISP parameters, and significantly improve image quality and processing speed.
[0007] The real-time automatic hyperparameter optimization method for ISP proposed in this invention is shown in Figure 1 As shown. It includes building a parameter network ParaNet and an ISP proxy model, where:
[0008] The parameter network ParaNet is used to predict the hyperparameters of the image signal processing (ISP) hardware system directly from the original image. The parameter network ParaNet specifically adopts a DNN model, and the specific structure includes 6 layers of 8-channel 3*3 convolutional layers and 2 layers of fully connected layers; among them, the convolutional layer of the second layer uses dilated convolution with a step size of 2 to reduce the amount of data, and the remaining convolutional layers use regular convolution with a step size of 1 to extract image features. The 2 layers of fully connected layers are used to convert the dimensions of the image to the dimensions of the hyperparameters. By learning the mapping process from the original image to the hyperparameters, fast and accurate parameter prediction is achieved. The input of the parameter network is the original image, and the output is a set of ISP hyperparameters.
[0009] The ISP proxy model adopts an 8-layer U-shaped multi-scale residual connection 3*3 convolutional layer network structure; it is used to extract image features from multiple scales and to learn the mapping relationship between hyperparameters and image processing.
[0010] The real-time automatic hyperparameter optimization method includes three steps: simulating non-differentiable ISP hardware, mapping the original image to ISP hyperparameters, and online tuning;
[0011] (1) Simulating non-differentiable ISP hardware: Since the hyperparameters of ISP hardware are discrete, it becomes a non-differentiable black box function, making it difficult to optimize using DNN. To overcome this problem, the present invention makes the ISP hardware differentiable by training an ISP proxy model based on a U-shaped multi-scale residual connection 3*3 convolutional layer network structure. Specifically, first, a paired training dataset is constructed, which is generated by the ISP hardware and includes raw data and configuration files; the output of the ISP hardware is used as the true value of the training agent. The training process can be expressed as:
[0012] W p =argminW p (L2(f proxy (I RAW ,P,W p ),f I SP (I RAW ,P))), (1)
[0013] Among them, W p is the weight of the proxy model, P is the parameter of ISP, I RAW is the original data, f proxy represents the ISP proxy model, f I SPrepresents ISP hardware and L2 represents L2 loss function.
[0014] (2) Mapping the original image to ISP hyperparameters; since the original image contains a large amount of low-level information, this information can be used to generate a high-level interpretation of the image. For example, the average value of the original image can be used to determine ISO information or lighting information, and the standard deviation shows the complexity and noise intensity of the image. Therefore, the present invention uses a parameter network ParaNet to directly learn the process of mapping from the original image to hyperparameters, so that the hyperparameter optimization process can run online. The original data is input into the parameter network ParaNet to generate hyperparameters. These hyperparameters are calculated together with the original data through the parameter-frozen ISP proxy model to calculate the loss of the specific task. This loss is then used to update the neural network parameters of the parameter network ParaNet, that is, the connection weights inside the neural network. This process is expressed as:
[0015] W r =argminW r (L task (f proxy (I RAW ,f para (I RAW ,W r ),W p ))), (2)
[0016] Among them, f para Represents the parameter network ParaNet, W r Represents the connection weights inside the parameter network ParaNet.
[0017] (3) Online tuning: Integrate the parameter network ParaNet into the ISP hardware system, and input the original data into the parameter network ParaNet and the ISP hardware at the same time. ParaNet is connected to the configuration interface of the ISP to export the optimized parameters online. The specific process is as follows:
[0018] O rgb =f I SP (I RAW ,f para (I RAW ,W r )), (3)
[0019] Among them, O rgb is the output RGB image of ISP.
[0020] Formula (1), Formula (2), and Formula (3) represent the mathematical relationship of the real-time automatic hyperparameter optimization method. Based on the mathematical relationship, the method can effectively realize the real-time online search of ISP parameters and significantly improve the image quality and processing speed.
[0021] The present invention realizes real-time optimization of ISP hyperparameters by simulating non-differentiable ISP hardware, mapping original images to ISP hyperparameters, and online tuning. The present invention excels in improving the speed and accuracy of ISP parameter search, and provides strong support for real-time automatic parameter optimization of ISP hardware systems.
[0022] Based on the above-mentioned real-time automatic hyperparameter optimization method, the present invention also provides a real-time automatic hyperparameter optimization system. The real-time automatic hyperparameter optimization system adopts a dual-channel hardware architecture. Figure 2 As shown. Specifically, it includes two pathways, ParaNet and ISP, and real-time processing is achieved through a dual-path architecture. The ParaNet pathway and the ISP pathway are executed in parallel to achieve real-time processing. The uniqueness of this architecture design is that the sensor's raster scanning results in the sensor outputting one pixel per cycle, rather than one image block. Therefore, the sampling speed of the sensor becomes the computational bottleneck of the entire system, rather than ParaNet or ISP. In order to reduce hardware resources and ensure that there is no additional delay, we propose a hardware-friendly dual-path architecture.
[0023] In ParaNet hardware, the convolution module contains convolution kernels for the input channel (cin) and the output channel (cout). Since the input is downsampled RAW data, two adjacent pixels are read every 8 cycles, and the convolution calculation of the output channel is serialized using the downsampling interval. A semi-parallel architecture is involved, which parallelizes the input channel and serializes the calculation of the output channel to reduce the number of 3x3 convolution kernels. In the ParaNet architecture, both the convolution layer and the fully connected layer are implemented in a semi-parallel manner. This serialized output design saves a lot of hardware resources without affecting the system latency. On the other hand, due to the unique design of serialized output, the second layer of convolution in ParaNet hardware implements the serialization of calculation from 3*3 convolution kernel to 1*3 convolution kernel, further reducing hardware resources.
[0024] The output of ParaNet is first de-normalized by the Para_Gen module and temporarily stored in the configuration register. The ISP hardware then reads this register to configure each image processing module in the ISP path. The ISP contains a total of 7 image processing modules, namely digital gain (DGain), bad pixel removal (DPC), noise reduction (DNS), green balance (GB), color difference (DMC), edge enhancement (EE), and color matrix correction (CMC). ParaNet and ISP paths are processed in parallel, using the optimized hyperparameters generated by the previous frame for processing in the current frame. The uniqueness of this design is that, unlike the image block-based neural network architecture, we use the sampling bottleneck of the sensor to reduce the number of multiplier units of the convolution kernel and the fully connected layer, while ensuring that the computational latency and register configuration latency of ParaNet do not exceed the processing latency of the ISP, thereby achieving frame-by-frame search.
[0025] Verification of the design of the present invention on FPGA and ASIC shows that the present invention is available and has high performance. It realizes real-time online ISP parameter search and image processing while maintaining high precision and high speed. It is the fastest ISP automatic parameter search method in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a flowchart of the real-time automatic hyperparameter optimization method of the present invention.
[0027] Figure 2 It is a structural diagram of the real-time automatic hyperparameter optimization system of the present invention. DETAILED DESCRIPTION
[0028] The present invention is further described below by way of embodiments in conjunction with the accompanying drawings.
[0029] The design of the present invention is verified on FPGA and ASIC, specifically including:
[0030] (1) Execute real-time automatic hyperparameter optimization methods. Figure 1 As shown in step 1, first build a training data set and use the ISP hardware to generate 5,000 sets of training data, including the original data and configuration files as input, and the output of the ISP hardware as the true value.
[0031] (2) Train the ISP proxy model. The ISP proxy model is trained by minimizing the L2 loss function. The structure of the ISP proxy model is a multi-scale residual connection 3*3 convolutional layer network structure with an 8-layer U-shaped structure. This enables the proxy model to simulate the behavior of the ISP hardware. The training formula is detailed in Formula (1).
[0032] (3) Design ParaNet, which has a structure of 6 layers of 8-channel 3*3 convolutional layers and 2 layers of fully connected layers, where the second convolutional layer uses dilated convolution with a stride of 2. The parameter network ParaNet is used to predict ISP hyperparameters from the original image.
[0033] (4) Train ParaNet. Input the original data into ParaNet to generate hyperparameters. Then process the original data and hyperparameters through proxy ISP. Calculate the task loss to update the weight of ParaNet. The training loss function is mean square error and the learning rate is 1e-5. The training formula is shown in formula (2).
[0034] (5) Integrate ParaNet into the ISP hardware system, input the original data into ParaNet and ISP at the same time, and connect ParaNet to the configuration interface of ISP to export the optimized parameters online.
[0035] (6) Implement real-time optimization, configure the ISP hardware through the hyperparameters generated by ParaNet, and perform image processing in the current frame. The specific process is shown in formula (3).
[0036] (7) Design the hardware architecture of ParaNet in the dual-channel architecture. All convolution structures use semi-parallel computing. According to the number of output channels, the ParaNet hardware architecture uses serialized output computing. In this design, the number of output channels is 8. The designed code is functionally simulated and logically synthesized using FPGA-based simulation tools and ASIC process-based synthesis tools. The FPGA uses the Xilinx KC705 development board, and the ASIC process uses the Global Foundry 28nm process.
[0037] (8) Design the hardware architecture of ISP in dual-channel architecture. ISP includes 7 modules, namely digital gain, bad pixel removal, noise reduction, green balance, color difference, edge enhancement, and color matrix correction. All modules are processed in serial order. The designed code is functionally simulated and logically synthesized by FPGA-based simulation tools and ASIC process-based synthesis tools. The FPGA uses Xilinx KC705 development board and the ASIC process uses Global Foundry 28nm process.
[0038] (9) Integrate the ParaNet and ISP pathways, configure the ISP pathway, and the ISP hardware reads the parameters in the ParaNet configuration registers to configure each image processing module. The real-time automatic hyperparameter optimization system achieves frame rates of 75FPS and 240FPS at 1080P resolution.
[0039] The usability and high performance of the design of the present invention are demonstrated. Compared with the existing method based on ParaNet, the method of the present invention has fewer parameters (about 2.84M) and enhanced floating point computing power (about 98.84GFLOPs), while the ParaNet design only uses 0.0093M parameters, effectively reducing memory requirements. Through system evaluation, the design of the present invention is significantly better than the existing automatic ISP parameter search method, especially when dealing with unforeseen changes.
Claims
1. A real-time automatic hyperparameter optimization method for ISP, It is characterized in that It includes building a parameter network ParaNet and an ISP proxy model; among which: The parameter network ParaNet is used to predict the hyperparameters of the image signal processing (ISP) hardware system directly from the original image. The parameter network ParaNet adopts a DNN model, and its specific structure includes 6 layers of 8-channel 3*3 convolutional layers and 2 layers of fully connected layers. Among them, the convolutional layer of the second layer adopts dilated convolution with a step size of 2 to reduce the amount of data, and the remaining convolutional layers use convolution with a step size of 1 to extract image features. The 2 layers of fully connected layers are used to convert the dimensions of the image into the dimensions of the hyperparameters. By learning the mapping process from the original image to the hyperparameters, fast and accurate parameter prediction is achieved. The input of the parameter network is the original image, and the output is a set of ISP hyperparameters. The ISP proxy model uses an 8-layer U-shaped multi-scale residual connection 3*3 convolutional layer network structure to extract image features from multiple scales and learn the mapping relationship between hyperparameters and image processing; The real-time automatic hyperparameter optimization method comprises the following specific steps: (1) Simulate non-differentiable ISP hardware; By training the ISP proxy model, the ISP hardware is made differentiable. Specifically, a paired training dataset is first constructed. The dataset is generated by the ISP hardware and includes raw data and configuration files. The output of the ISP hardware is used as the true value of the training proxy. The training process is expressed as: W p =argminW p (L2(f proxy (I RAW ,P,W p ),f ISP (I RAW ,P))), (1) Among them, W p is the weight of the proxy model, P is the parameter of ISP, I RAW is the original data, f proxy represents the ISP proxy model, f ISP represents ISP hardware, L2 represents L2 loss function; (2) Mapping the original image to ISP hyperparameters; Specifically, the parameter network ParaNet is used to directly learn the process of mapping from the original image to the hyperparameters, so that the hyperparameter optimization process can be run online; the original data is input into the parameter network ParaNet to generate hyperparameters; these hyperparameters and the original data are passed through the parameter-frozen ISP proxy model to calculate the loss of the specific task; then this loss is used to update the network parameters of the parameter network ParaNet, that is, the connection weights inside the network; this process is expressed as: W r =argminW r (L task (f proxy (I RAW ,f para (I RAW ,W r ),W p ))), (2) Among them, f para Represents the parameter network ParaNet, W r Represents the connection weights inside the parameter network ParaNet; (3) Online tuning; The parameter network ParaNet is integrated into the ISP hardware system, and the original data is input into the parameter network ParaNet and the ISP hardware at the same time; ParaNet is connected to the configuration interface of the ISP to export the optimized parameters online; the specific process is expressed as follows: O rgb =f ISP (I RAW ,f para (I RAW ,W r )), (3) Among them, O rgb is the output of ISP, which is an RGB image.
2. A real-time automatic hyperparameter optimization system based on the real-time automatic hyperparameter optimization method according to claim 1, It is characterized in that Adopt dual-channel hardware architecture, including ParaNet and ISP channels, which are executed in parallel to achieve real-time processing; In ParaNet hardware, the convolution module contains convolution kernels for input channels (cin) and output channels (cout); the input is downsampled RAW data, and two adjacent pixels are read every 8 cycles. The convolution calculation of the output channel is serialized using the downsampling interval; the input channel is parallelized and the calculation of the output channel is serialized to reduce the number of 3x3 convolution kernels; in ParaNet, both the convolution layer and the fully connected layer are implemented in semi-parallel mode; the second layer of convolution in ParaNet hardware implements the serialization of calculation from 3*3 convolution kernel to 1*3 convolution kernel, which further reduces hardware resources; The output of the ParaNet hardware is first de-normalized by the Para_Gen module and temporarily stored in the configuration register; then, the ISP hardware reads the register to configure each image processing module in the ISP path; the ISP contains a total of 7 image processing modules, namely digital gain (DGain), bad pixel removal (DPC), noise reduction (DNS), green balance (GB), color difference (DMC), edge enhancement (EE), and color matrix correction (CMC); ParaNet and ISP paths are processed in parallel, using the optimized hyperparameters generated in the previous frame for processing in the current frame; in this way, the sampling bottleneck of the sensor is used to reduce the number of multiplier units of the convolution kernel and the fully connected layer, while ensuring that the calculation delay and register configuration delay of ParaNet do not exceed the processing delay of the ISP, so as to achieve frame-by-frame search.
Citation Information
Cited By
ISP hyper-parameter iterative optimization method and device and electronic equipment
CN121190778A
Method, apparatus and electronic device for iterative optimization of isp hyperparameters
CN121190778B
