Method, apparatus and electronic device for iterative optimization of isp hyperparameters

Abstract

The application provides an ISP hyperparameter iterative optimization method and device and electronic equipment, and the method comprises the following steps: obtaining a pre-constructed hyperparameter prediction model, and the hyperparameter prediction model is constructed according to a given original image and a corresponding downstream visual task. The ISP hyperparameter sequence in the training data is gradually masked through a forward process to obtain a partially masked ISP hyperparameter sequence. The hyperparameter prediction model is trained based on the partially masked ISP hyperparameter sequence and the corresponding original image. A fully masked ISP hyperparameter sequence is initialized for a target image. The target image and the fully masked ISP hyperparameter sequence are used as model inputs, and the masked hyperparameters are iteratively predicted through a reverse process. Through the coarse-to-fine hyperparameter prediction paradigm, the iterative prediction process is no longer strictly dependent on the fixed sequence of parameters, so that the model can naturally learn the context, effectively identify and utilize the mutual correlation between all parameters.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an iterative optimization method, apparatus, and electronic device for ISP hyperparameters. Background Technology

[0002] An image signal processor (ISP) is a complex system consisting of multiple image processing modules. Traditional ISP design employs a divide-and-conquer strategy, developing specialized algorithms for each module to break down the complex ISP design problem into multiple subtasks. These subtasks typically involve a large number of parameters (ranging from tens to hundreds) that require manual tuning and rely on the experience of imaging experts for optimization.

[0003] Currently, image processing experts primarily rely on manual optimization of numerous hyperparameters. However, the complex parameter space presents significant challenges to traditional manual tuning methods. This is especially true for different downstream vision tasks (such as object detection and image segmentation), where subjective experience alone is insufficient to determine the optimal hyperparameter configuration. Recent research has proposed various ISP hyperparameter optimization methods based on downstream task evaluation metrics. These methods mainly fall into two categories: those based on derivative-free optimization and those utilizing differentiable approximations for gradient optimization. However, these existing methods generally treat the high-dimensional parameter space as a holistic black box for optimization and prediction, neglecting the important interrelationships between hyperparameters. The simplistic approach of existing methods limits further improvements in optimization performance. Summary of the Invention

[0004] This invention provides an iterative optimization method, apparatus, and electronic device for ISP hyperparameters, addressing the shortcomings of existing technologies that treat the high-dimensional parameter space as a holistic black box for optimization and prediction, neglecting the important interrelationships between hyperparameters, leading to low optimization performance. The technical solution proposed by this invention is as follows:

[0005] In a first aspect, the present invention provides an iterative optimization method for ISP hyperparameters, comprising:

[0006] Obtain a pre-built hyperparameter prediction model, which is constructed based on a given original image and its corresponding downstream visual task;

[0007] By progressively masking the ISP hyperparameter sequence in the training data through a forward process, a partially masked ISP hyperparameter sequence is obtained.

[0008] Based on the ISP hyperparameter sequence of the partial mask and the corresponding original image, the hyperparameter prediction model is trained to obtain the trained hyperparameter prediction model.

[0009] Acquire the target image and initialize a full mask ISP hyperparameter sequence for the target image;

[0010] The target image and the fully masked ISP hyperparameter sequence are used as inputs to the trained hyperparameter prediction model. The masked hyperparameters are predicted iteratively through a reverse process, and the complete optimized ISP hyperparameter sequence is output.

[0011] Optionally, the reverse process includes:

[0012] Input the current ISP hyperparameter sequence and the target image into the trained hyperparameter prediction model to predict the masked hyperparameters and obtain the prediction results;

[0013] The current ISP hyperparameter sequence is updated based on the prediction results to obtain the updated ISP hyperparameter sequence;

[0014] Adjust the mask ratio, and remask the updated ISP hyperparameter sequence based on the adjusted mask ratio to generate a new ISP hyperparameter sequence, and update the current ISP hyperparameter sequence with the new ISP hyperparameter sequence;

[0015] Repeat the process of predicting masked hyperparameters, updating the sequence, adjusting the mask ratio, and re-masking until the mask ratio t drops to 0, at which point the complete optimized ISP hyperparameter sequence is output.

[0016] Optionally, the masking operation in the forward process includes:

[0017] Sample the ISP hyperparameter sequence from the training data;

[0018] Uniformly sample mask proportion t from uniform distribution U (0,1);

[0019] Each hyperparameter in the ISP hyperparameter sequence is independently masked with probability t, or... The probability of keeping the image unmasked is used to obtain a partially masked ISP hyperparameter sequence.

[0020] Optionally, the hyperparameter prediction model is trained by calculating regression loss only on the hyperparameters marked by the mask; the hyperparameter prediction model includes a visual feature extraction layer and a hyperparameter prediction layer.

[0021] The hyperparameter prediction model is trained based on the ISP hyperparameter sequence derived from the partial mask and the corresponding original image to obtain the trained hyperparameter prediction model, including:

[0022] Visual features are extracted from the original image based on the visual feature extraction layer;

[0023] The ISP hyperparameter sequence of the partial mask and the visual features are input into the hyperparameter prediction layer to obtain the prediction output of the masked hyperparameters;

[0024] The regression loss is calculated based on the predicted output of the masked hyperparameters and the true labels. The model parameters of the hyperparameter prediction layer are then updated through backpropagation based on the regression loss. The true labels are the true hyperparameter values ​​in the ISP hyperparameter sequence.

[0025] Optionally, the regression loss is determined as follows:

[0026] ;

[0027] In the formula, For regression loss; Indicates model parameters; Indicates the total length of the ISP hyperparameter sequence; Indicates a mask marker; Indicates an indicator function; Indicates the preset loss function; Represents the sequence of ISP hyperparameters for partial masking; express The i-th hyperparameter in; Represents the original image; This represents the hyperparameter prediction model using a partially masked ISP hyperparameter sequence. and the original image When given input, the predicted output for the i-th hyperparameter; This represents the actual hyperparameter value of the i-th hyperparameter in the ISP hyperparameter sequence.

[0028] Optionally, the visual feature extraction layer is a pre-trained convolutional neural network, and the hyperparameter prediction layer is a multi-layer bidirectional Transformer model.

[0029] Secondly, the present invention also provides an ISP hyperparameter iterative optimization device, comprising the following modules:

[0030] The model acquisition module is used to acquire a pre-built hyperparameter prediction model, which is constructed based on a given original image and its corresponding downstream visual task.

[0031] The model training module is used to progressively mask the ISP hyperparameter sequence in the training data through a forward process to obtain a partially masked ISP hyperparameter sequence; and to train the hyperparameter prediction model based on the partially masked ISP hyperparameter sequence and the corresponding original image to obtain the trained hyperparameter prediction model.

[0032] An iterative optimization module is used to acquire a target image, initialize a full-mask ISP hyperparameter sequence for the target image, and use the target image and the full-mask ISP hyperparameter sequence as input to the trained hyperparameter prediction model, iteratively predict the masked hyperparameters through a reverse process, and output a complete optimized ISP hyperparameter sequence.

[0033] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the ISP hyperparameter iterative optimization method as described in the first aspect above.

[0034] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the ISP hyperparameter iterative optimization method as described in the first aspect above.

[0035] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the ISP hyperparameter iterative optimization method as described in the first aspect above.

[0036] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0037] The iterative optimization method, apparatus, and electronic device for ISP hyperparameters provided by this invention progressively masks the ISP hyperparameter sequence in the training data through a forward process to obtain a partially masked ISP hyperparameter sequence. This approach does not process the entire high-dimensional parameter space all at once, but rather processes the parameters gradually and in stages, breaking the paradigm of treating the high-dimensional parameter space as a complete black box. This allows the model to progressively learn and understand the structure and characteristics of the parameter space. After acquiring the target image and initializing the fully masked ISP hyperparameter sequence, the masked hyperparameters are iteratively predicted through a reverse process based on the target image and the fully masked ISP hyperparameter sequence. This iterative prediction process further refines the processing of the parameter space, focusing only on a portion of the masked parameters in each iteration, gradually constructing a complete optimized ISP hyperparameter sequence, and avoiding the blindness caused by processing the entire black box.

[0038] A hyperparameter prediction model is trained based on a partially masked ISP hyperparameter sequence and the corresponding original image. During training, the partially masked sequence allows the model to consider the potential relationships between unmasked and masked parameters. Since the model needs to predict masked parameters based on known partial parameter information, it must learn the interrelationships between parameters, thus capturing important interrelationships between hyperparameters during training. Iterative prediction of masked hyperparameters in the reverse process ensures that each prediction is based on previously predicted or known parameter information. As iterations proceed, the model continuously uses the determined parameter information to further predict other parameters; this progressive approach strengthens the model's learning and utilization of the interrelationships between hyperparameters. In each iteration, the model can more accurately predict the next masked parameter based on the current parameter combinations, thus fully considering the mutual influence between hyperparameters and solving the problem of existing technologies neglecting this interrelationship. Through the aforementioned staged processing of the high-dimensional parameter space and the learning of the interrelationships between hyperparameters, the trained hyperparameter prediction model can more accurately understand the relationship between ISP hyperparameters, the original image, and downstream visual tasks. This more accurate model training provides a solid foundation for subsequent optimization processes, enabling the generation of optimized ISP hyperparameter sequences that better meet practical needs. The iterative prediction mechanism of the reverse process makes the optimization process a gradual approximation of the optimal solution. Each iteration improves upon the previous one, continuously utilizing newly predicted parameter information to optimize the entire parameter sequence. This iterative optimization approach allows for more precise adjustment of hyperparameters, avoiding the local optima problem that may occur in overall optimization, thereby effectively improving optimization performance and overcoming the shortcomings of existing technologies that suffer from low optimization efficiency due to simplified processing methods.

[0039] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0040] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the classic ISP optimization model framework provided by the present invention.

[0043] Figure 2 This is a flowchart illustrating the ISP hyperparameter iterative optimization method provided by the present invention.

[0044] Figure 3 This is a schematic diagram of the forward process provided by the present invention.

[0045] Figure 4 This is a schematic diagram of the reverse process provided by the present invention.

[0046] Figure 5 This is a schematic diagram of the ISP hyperparameter iterative optimization device provided by the present invention.

[0047] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] Currently, the industry primarily relies on image processing experts to manually adjust a large number of hyperparameters in image signal processing (ISP) hyperparameter optimization. However, ISP hyperparameters constitute a complex high-dimensional parameter space, posing a significant challenge to traditional manual tuning methods. Especially when dealing with different downstream vision tasks (such as object detection and image segmentation), it is difficult to determine the optimal hyperparameter configuration based solely on subjective experience.

[0050] Recent studies have proposed several methods for optimizing ISP hyperparameters based on downstream task evaluation metrics. Figure 1This paper presents a classic ISP optimization model framework. In the image processing workflow based on this framework, the original image is processed sequentially through modules such as Block1, Block2, Block3, Block4, and Block5 to ultimately reconstruct the image. The optimization of high-dimensional ISP hyperparameters is a crucial step in this process. Existing optimization methods mainly fall into two categories: those based on derivative-free optimization and those utilizing differentiable approximations for gradient optimization. These existing methods generally treat the high-dimensional parameter space as a complete black box for optimization and prediction, neglecting the important interrelationships between high-dimensional ISP hyperparameters. The simplified processing methods of existing approaches limit further improvements in optimization performance.

[0051] Based on this, this invention provides an iterative optimization method, apparatus, and electronic device for ISP hyperparameters, proposing a novel ISP hyperparameter optimization model framework that overcomes the inability of traditional ISP optimization methods to model the interrelationships between ISP hyperparameters. This architecture fully utilizes the low-level information of the input original image and the predicted hyperparameters to progressively deduce unpredicted parameters, iteratively predicting and optimizing the ISP high-dimensional parameter set. A coarse-to-fine hyperparameter prediction paradigm is designed, where the iterative prediction process no longer strictly follows a fixed sequence of dependent parameters, enabling the model to naturally learn context and effectively identify and utilize the interrelationships between all parameters. This invention is based on a mask diffusion mechanism and adopts a standard training process, but uses a diffusion sampling method in the inference prediction stage to predict ISP hyperparameters by progressively demasking. The model can predict multiple mask position parameters simultaneously at each step. The diffusion strategy guides the prediction steps, allowing the model to progressively predict all ISP hyperparameters in a coarse-to-fine manner. This prediction paradigm enables the model to progressively optimize and refine ISP hyperparameters, further improving the prediction and optimization performance of high-dimensional parameters.

[0052] Reference Figure 2 As shown, the iterative optimization method for ISP hyperparameters includes the following:

[0053] S110. Obtain a pre-built hyperparameter prediction model, which is constructed based on a given original image and its corresponding downstream visual task.

[0054] Obtain a pre-built hyperparameter prediction model. This model defines and constructs a distribution model for predicting ISP hyperparameters for a given original image and its corresponding downstream visual task (such as image classification, object detection, image enhancement, denoising, etc.). The original image I is typically raw RAW data or unprocessed image data from the ISP input. The downstream visual task defines the objective and direction for the model to optimize the ISP hyperparameters. The goal of this model is to learn how to predict a suitable sequence of ISP hyperparameters based on image content and task requirements, thereby optimizing the performance of the visual task.

[0055] S120. By progressively masking the ISP hyperparameter sequence in the training data through a forward process, a partially masked ISP hyperparameter sequence is obtained.

[0056] Collect a large number of sample triples (I,T,P) ∗ ): I represents the original image. T represents the identifier or features of the downstream visual task (such as task type, loss function, requirements, etc.). P ∗ The optimal ISP hyperparameter configuration (obtainable through empirical adjustment, manual annotation, or automatic search) corresponds to the input image and task requirements. This data is used to train the hyperparameter prediction model and establish the mapping relationship between the input and the hyperparameters.

[0057] During the forward pass, the independent parameters in the ISP hyperparameter sequence P0 of the training data are progressively masked. Specifically, using the real ISP hyperparameter sequence from the training data, the parameters are masked according to different masking ratios t∈{t0,t1,...,t...}. n The hyperparameters in the sequence are partially masked. The masking ratio t indicates that in this step, approximately t proportions of the hyperparameters in the sequence are randomly masked, while the remaining parameters remain unchanged. As the masking ratio t gradually increases from 0 to 1, the hyperparameters in the sequence are gradually masked until t=1, at which point the sequence is completely masked. Specifically, each hyperparameter in the sequence is masked with a probability of t, or with a probability of ( The probability of the hyperparameter sequence remains unmasked. This process gradually transforms the complete optimized ISP hyperparameter sequence into a partially masked or even fully masked form, creating a distribution process of the hyperparameter mask sequence. Through this process, the model can perceive the transition of the hyperparameter sequence from a complete state to a state disturbed by noise (masking), improving its ability to recover missing information.

[0058] S130. Based on the ISP hyperparameter sequence of the partial mask and the corresponding original image, train the hyperparameter prediction model to obtain the trained hyperparameter prediction model.

[0059] By using the ISP hyperparameter sequences of partial masks in the training data, combined with the corresponding original images, a hyperparameter prediction model is trained. The training objective is to enable the model to learn how to predict and recover the masked hyperparameter values, approximating the true distribution of the original hyperparameters, given the partial absence of ISP parameters. After training, a hyperparameter prediction model capable of reasonably inferring and completing masked conditions is obtained.

[0060] Specifically, a hyperparameter prediction model is trained using partial masked ISP hyperparameter sequences with different masking ratios and their corresponding input images. The training objective is to enable the model to recover or predict the masked parameters given data containing missing hyperparameters, approximating the true distribution of the original data. The model thus possesses the ability to infer and complete missing parameters from the mask.

[0061] The hyperparameter prediction model aims to maximize the following objective function:

[0062] ;

[0063] In the formula, This represents the optimal model parameters. The goal of hyperparameter prediction models is to find a set of parameters that optimizes the model's performance on a given task. This means finding a set of model parameters such that The value reaches its minimum. These are the parameters of the hyperparameter prediction model, which are continuously adjusted. The value of minimizes the overall loss of the model on the training samples, thereby enabling the model to better predict the appropriate ISP hyperparameters based on the input image to meet the requirements of downstream vision tasks. Indicates model parameters; This represents the number of training samples, i.e., the number of original images used to train the hyperparameter prediction model and the number of data pairs for their corresponding downstream visual tasks. It is a task-related loss function used to measure the model's prediction of the ISP-processed image. The difference between the model and the real target is used to guide the optimization of the model on specific downstream vision tasks. This represents the i-th original image, which is the input data for the model. This refers to ISP (Image Signal Processor) operation, which is based on the predicted ISP hyperparameter sequence. right Process it. It is a hyperparameter prediction model with and model parameters Given that the ISP system consists of a series of packaged hardware processing units and is highly opaque, this invention reformulates the above optimization objective as follows:

[0064] ;

[0065] In the formula, Indicates the original image Optimal ISP parameter configuration for specific downstream vision tasks. This represents the preset loss function used to measure the difference between the predicted ISP hyperparameter sequence P and... The difference between them measures the accuracy of hyperparameter predictions. The aim is to improve the accuracy by adjusting model parameters. This enables the model to accurately predict the optimal ISP hyperparameters based on the input image, thus meeting the needs of downstream vision tasks.

[0066] The core of this invention is a hyperparameter prediction model, which is a parametric model. It will predict the ISP hyperparameters of the partially masked part. and the original image As input, simultaneously predict all masked hyperparameters (denoted as ). The model is trained only on the regression loss computed on the masked prediction parameters.

[0067] S140. Obtain the target image and initialize the full mask ISP hyperparameter sequence for the target image.

[0068] In practical applications, for the target image that needs to be optimized, a fully masked ISP hyperparameter sequence is initialized for the target image, that is, all hyperparameters in the sequence are unknown and are completely masked.

[0069] S150. The target image and the fully masked ISP hyperparameter sequence are used as inputs to the trained hyperparameter prediction model. The masked hyperparameters are predicted iteratively through a reverse process, and the complete optimized ISP hyperparameter sequence is output.

[0070] Taking the target image and the ISP hyperparameter sequence of the full mask as input, prediction is performed through an iterative inverse process. In the inverse process, starting from the ISP hyperparameter sequence of the full mask, the masked hyperparameters are predicted iteratively. A masking ratio t is introduced, starting from t=1 and gradually decreasing. In each step, the masked hyperparameters are predicted based on the current masking ratio t and the hyperparameter prediction model. This process continues, and as t gradually decreases to 0, all hyperparameters are gradually predicted, ultimately outputting a complete and optimized ISP hyperparameter sequence. Essentially, the inverse process is based on a trained hyperparameter prediction model, starting from the full mask sequence, to gradually recover and predict the masked parameter values. Each iteration uses the current state to estimate the distribution of the masked hyperparameters and updates the sequence. As iterations proceed, the masking ratio gradually decreases, and the hyperparameter sequence gradually approaches the complete and optimized ISP parameter configuration. The inverse process is essentially a probabilistic inverse extrapolation space based on a priori defined prediction model, achieving fine-grained optimization of hyperparameters through multiple iterations.

[0071] Specifically, taking the target image and the full mask parameter sequence as input, a trained hyperparameter prediction model is used to predict the target image according to a preset mask ratio sequence t. n →t0 Iteratively recovers the hyperparameters. Each iteration predicts the distribution of parameters under the current masked state using the model, thus predicting the masked hyperparameters. Starting from t=1, the mask ratio is gradually reduced, causing the hyperparameter sequence to tend towards completeness and optimization. Iteration continues until the mask ratio t=0, obtaining the optimized complete ISP hyperparameter configuration. This method effectively optimizes the ISP hyperparameter configuration to adapt to different input images and downstream vision tasks, thereby improving image processing performance and effectiveness. Simultaneously, by adjusting the mask ratio t, the prediction difficulty of the model and the stability of the training process can be controlled.

[0072] The iterative optimization method for ISP hyperparameters provided in this invention progressively masks the ISP hyperparameter sequence in the training data through a forward process, obtaining a partially masked ISP hyperparameter sequence. This approach does not process the entire high-dimensional parameter space all at once, but rather processes the parameters gradually and in stages, breaking the paradigm of treating the high-dimensional parameter space as a complete black box. This allows the model to progressively learn and understand the structure and characteristics of the parameter space. After acquiring the target image and initializing the fully masked ISP hyperparameter sequence, the masked hyperparameters are iteratively predicted through a reverse process based on the target image and the fully masked ISP hyperparameter sequence. This iterative prediction process further refines the processing of the parameter space, focusing only on a portion of the masked parameters in each iteration, gradually constructing a complete optimized ISP hyperparameter sequence, and avoiding the blindness caused by processing the entire black box.

[0073] A hyperparameter prediction model is trained based on a partially masked ISP hyperparameter sequence and the corresponding original image. During training, the partially masked sequence allows the model to consider the potential relationships between unmasked and masked parameters. Since the model needs to predict masked parameters based on known partial parameter information, it must learn the interrelationships between parameters, thus capturing important interrelationships between hyperparameters during training. Iterative prediction of masked hyperparameters in the reverse process ensures that each prediction is based on previously predicted or known parameter information. As iterations proceed, the model continuously uses the determined parameter information to further predict other parameters; this progressive approach strengthens the model's learning and utilization of the interrelationships between hyperparameters. In each iteration, the model can more accurately predict the next masked parameter based on the current parameter combinations, thus fully considering the mutual influence between hyperparameters and solving the problem of existing technologies neglecting this interrelationship. Through the aforementioned staged processing of the high-dimensional parameter space and the learning of the interrelationships between hyperparameters, the trained hyperparameter prediction model can more accurately understand the relationship between ISP hyperparameters, the original image, and downstream visual tasks. This more accurate model training provides a solid foundation for subsequent optimization processes, enabling the generation of optimized ISP hyperparameter sequences that better meet practical needs. The iterative prediction mechanism of the reverse process makes the optimization process a gradual approximation of the optimal solution. Each iteration improves upon the previous one, continuously utilizing newly predicted parameter information to optimize the entire parameter sequence. This iterative optimization approach allows for more precise adjustment of hyperparameters, avoiding the local optima problem that may occur in overall optimization, thereby effectively improving optimization performance and overcoming the shortcomings of existing technologies that suffer from low optimization efficiency due to simplified processing methods.

[0074] As an optional embodiment, the masking operation in the forward process includes:

[0075] S210. Sample the ISP hyperparameter sequence from the training data.

[0076] During training, the first step is to select an ISP hyperparameter sequence from the existing training dataset. The training data contains numerous combinations of ISP hyperparameters under various settings, collected from different image scenes, shooting conditions, or for different downstream vision tasks. The sampling operation involves randomly selecting a specific hyperparameter sequence from this dataset as the input for subsequent masking operations.

[0077] S220, uniform sampling mask ratio t from uniform distribution U (0,1).

[0078] This invention employs a diffusion sampling strategy, mask ratio The values ​​vary randomly between 0 and 1. U(0,1) represents a uniform distribution between 0 and 1. A value is sampled from this distribution as the mask ratio t, which serves as the probability basis for the masking operation. The uniform distribution ensures the randomness of the sampling, making the t value obtained from each sampling have an equal probability between 0 and 1, thus enabling a comprehensive exploration of the situation under different mask ratios.

[0079] S230. Mask each hyperparameter in the ISP hyperparameter sequence independently with probability t, or... The probability of keeping the image unmasked is used to obtain a partially masked ISP hyperparameter sequence.

[0080] For each hyperparameter in the sampled ISP hyperparameter sequence, an independent evaluation is performed. Specifically, a random number between 0 and 1 is generated. If this random number is less than the sampled value t, the corresponding hyperparameter is masked, for example, by setting its value to a specific mask identifier (such as 0 or a special symbol). If the random number is greater than or equal to the value t, the original value of the hyperparameter remains unchanged. After this independent evaluation and processing of all hyperparameters in the sequence, a partially masked ISP hyperparameter sequence is obtained.

[0081] This invention generates a large number of partially masked sequences by sampling different ISP hyperparameter sequences from training data and performing masking operations based on different sampling masking ratios t. This significantly increases the diversity of training data, allowing the model to encounter various combinations of hyperparameters with missing information, thus helping the model learn more robust features and patterns. In practical applications, some ISP hyperparameter information may be missing due to various reasons (such as sensor failure, data transmission errors, etc.). This masking operation method can simulate such scenarios, allowing the model to learn and predict under incomplete information during training, improving the model's adaptability and stability in practical applications. Since each hyperparameter is masked independently, this method can randomly hide different combinations of hyperparameters. When training a hyperparameter prediction model based on the partially masked ISP hyperparameter sequences and the corresponding original images, the model needs to infer the masked hyperparameters based on the remaining unmasked hyperparameter information. This forces the model to learn the interrelationships between hyperparameters, because only by considering these relationships can the masked parameters be predicted more accurately, thereby improving the model's understanding and utilization of the relationships between hyperparameters. Random masking introduces a degree of randomness and uncertainty, preventing the model from relying solely on fixed hyperparameter combinations during training. This helps prevent overfitting to specific hyperparameter sequences, improving the model's generalization ability and enabling it to perform well even with new and unseen hyperparameter conditions.

[0082] Specifically, during the training phase, the first step is to sample ISP hyperparameter sequences P0 from the existing training dataset. These sequences represent the hyperparameter configurations required by the ISP under different image processing scenarios. (Refer to...) Figure 3 As shown, the features of the original image I are represented in blue, while each hyperparameter in the ISP hyperparameter sequence P0 is represented in yellow. Next, a masking proportion t is randomly sampled from the uniform distribution U(0,1). This proportion t determines what percentage of the hyperparameters will be masked in subsequent steps. For each hyperparameter in the sampled ISP hyperparameter sequence P0, each marker is independently masked with the same probability t, either by replacing the hyperparameter with a special [MASK] marker or by keeping the hyperparameter unchanged with a probability of 1-t. After the masking operation, a partially masked ISP hyperparameter sequence P is obtained. t Some hyperparameters are masked, while others remain unchanged. If a hyperparameter is marked, it will be replaced with [MASK]; otherwise, it will remain unchanged.

[0083] Combine the original image I and the hyperparameter sequence P of the partial mask. tInput into the hyperparameter prediction model (i.e. Figure 3 In the Transformer hyperparameter predictor, the hyperparameter prediction model makes predictions based on the input image features and partially known hyperparameter information. The hyperparameter prediction model outputs the ISP (Image Signal Processor) predicted hyperparameters. ,in This represents the complete sequence of predicted hyperparameters. This prediction process is based on partially known hyperparameter information and original image features, aiming to recover the masked hyperparameter values.

[0084] During inference, the model iteratively decodes from scratch, predicting and generating a complete sequence of ISP hyperparameters. The algorithm starts with the input target image and all masked hyperparameters, predicting hyperparameters at all locations. In the next step, the predicted hyperparameters are masked, and both the masked and masked hyperparameters are used as input to the model, generating predicted hyperparameters at the masked locations. This entire process runs iteratively, ultimately achieving iterative optimization prediction of the high-dimensional ISP parameter set.

[0085] This invention uses randomized masking of hyperparameters at different proportions, requiring the model to adapt to varying degrees of partial information loss during training. This training method helps the model learn more robust feature representations, thereby improving its generalization ability on unseen data. The masking operation in the forward process simulates the gradual revelation of information in black-box optimization problems. In practical applications, it is often impossible to obtain the optimal values ​​of all hyperparameters at once; instead, multiple trials and adjustments are needed to gradually approach the optimal solution. The masking operation in the forward process allows the model to adapt to this gradual solution process during training. By introducing a masking proportion t and randomly masking hyperparameters, the model needs to predict the masked hyperparameter values ​​more accurately during training. This training method helps the model learn the complex relationships between hyperparameters and more accurately predict the complete optimized ISP hyperparameter sequence during inference. To improve the model's ability to handle variable-length data, 1% of the training data is set to a random length. This setting, combined with the masking operation, allows the model to adapt to ISP hyperparameter sequence inputs of different lengths, further enhancing the model's flexibility and practicality. The masking operation in the forward process simulates the stepwise solution process of the black-box optimization problem by randomly masking the hyperparameters in the ISP hyperparameter sequence, which enhances the generalization ability and prediction accuracy of the model, and supports training on variable-length data.

[0086] As an optional embodiment, the reverse process includes:

[0087] S310. Input the current ISP hyperparameter sequence and the target image into the trained hyperparameter prediction model to predict the masked hyperparameters and obtain the prediction results.

[0088] The current ISP hyperparameter sequence and the corresponding target image are input into the already trained hyperparameter prediction model. This model, having been trained previously with a partially masked ISP hyperparameter sequence and the corresponding original image, possesses the ability to predict the masked hyperparameters based on known hyperparameter information and target image features. The model analyzes and calculates the input information to output the prediction results for the masked hyperparameters. In the first iteration, the aforementioned current ISP hyperparameter sequence is the initialized fully masked ISP hyperparameter sequence.

[0089] S320. Update the current ISP hyperparameter sequence based on the prediction result to obtain the updated ISP hyperparameter sequence.

[0090] Based on the masked hyperparameters predicted by the model, the current ISP hyperparameter sequence is updated. The previously masked positions are filled with the predicted hyperparameter values, resulting in an updated ISP hyperparameter sequence. This updated sequence contains more accurate hyperparameter information than the previous sequence.

[0091] S330. Adjust the mask ratio, and remask the updated ISP hyperparameter sequence based on the adjusted mask ratio to generate a new ISP hyperparameter sequence, and update the current ISP hyperparameter sequence to the new ISP hyperparameter sequence.

[0092] The previously used masking ratio t is adjusted, which can be done by gradually decreasing the masking ratio in steps. Based on the adjusted masking ratio, the updated ISP hyperparameter sequence is masked again. This means that some predicted and filled hyperparameters may be masked again, while some predicted hyperparameters are retained. Specifically, based on the new masking ratio t, each hyperparameter in the sequence is independently judged, and masked with a probability of t or with a probability of t. The probability is kept unmasked, thus generating a new ISP hyperparameter sequence. The currently processed ISP hyperparameter sequence is then updated with this newly generated sequence.

[0093] S340, Repeat the process of predicting the masked hyperparameters, updating the sequence, adjusting the mask ratio, and re-masking until the mask ratio t drops to 0, and output the complete optimized ISP hyperparameter sequence.

[0094] Repeat the process of predicting masked hyperparameters, updating the sequence, adjusting the mask ratio, and remasking, i.e., repeat steps S310-S330. In each loop, the mask ratio t gradually decreases, meaning the number of masked hyperparameters in the sequence gradually decreases. When the mask ratio t finally drops to 0, there are no more masked hyperparameters in the sequence, resulting in a complete optimized ISP hyperparameter sequence, which is then output as the optimized result.

[0095] This invention utilizes the model's predictive ability for masked hyperparameters to fill in the unknown parts of the sequence through a continuous process of prediction, updating, and remasking. Each iteration further optimizes the hyperparameter sequence based on the previous one, making the hyperparameter values ​​in the sequence increasingly closer to the optimal solution, thereby gradually constructing a complete optimized ISP hyperparameter sequence. During the reverse process, the trained hyperparameter prediction model continues to play a role. As the mask ratio decreases, the model needs to predict the remaining masked parameters based on less and less known hyperparameter information. This places higher demands on the model's predictive ability and also prompts the model to make fuller use of the learned relationships between hyperparameters and the feature information of the target image, improving the accuracy and reliability of the model's predictions. Adjusting the mask ratio and remasking allows the entire optimization process to adapt to the needs of different stages. In the initial stage, a higher mask ratio allows the model to learn and predict from relatively ambiguous information; as the mask ratio gradually decreases, the model can make more accurate predictions based on more known information. This dynamic adjustment helps improve the efficiency and effectiveness of optimization. When the mask ratio drops to 0, it means that all hyperparameters have undergone the model's prediction and optimization process. By taking into account the interrelationships between hyperparameters and the features of the target image through this iterative approach, the resulting complete optimized ISP hyperparameter sequence can better adapt to the needs of the target image and downstream vision tasks, thereby improving the accuracy and applicability of the optimization results.

[0096] Reference Figure 4 As shown, in the initial state (masking ratio t=1), all ISP hyperparameters are masked, meaning all positions in the ISP hyperparameter sequence are masked. The current fully masked ISP hyperparameter sequence is then compared with the target image (i.e.,...). Figure 4 The original image I) is input into the hyperparameter prediction model (i.e. Figure 4 The hyperparameter prediction model, based on the input information, predicts the masked hyperparameter values ​​and fills them into the corresponding positions in the sequence. Figure 4The predicted hyperparameter values ​​are indicated by yellow fill in the image. Based on the prediction results of the hyperparameter prediction model, the current ISP hyperparameter sequence is updated. At this point, some positions in the sequence have been filled with predicted values, resulting in the updated ISP hyperparameter sequence. The masking ratio t is adjusted and gradually decreased. As t decreases, some hyperparameter positions that have already been predicted and filled may be masked again (…). Figure 4 (The yellow fill indicates areas partially covered by cross lines). A new ISP hyperparameter sequence is generated, containing some known hyperparameter values ​​and some masked positions. The current ISP hyperparameter sequence is updated to this new ISP hyperparameter sequence. The new ISP hyperparameter sequence is then input back into the hyperparameter prediction model for the next round of prediction. The process of prediction, updating, adjusting the mask ratio, and remasking is repeated until the mask ratio t drops to 0. When the mask ratio t drops to 0, all hyperparameter positions have been filled with predicted values, at which point the complete optimized ISP hyperparameter sequence is output. Figure 4 (As shown at the bottom of the middle section). By progressively adjusting the masking ratio and remasking, the reverse process allows the model to update the hyperparameter sequence at each step based on the current best prediction, thus gradually approaching the optimal solution. As the masking ratio decreases, the model can utilize more known information to predict the masked hyperparameters, thereby improving prediction accuracy. Through repeated prediction and update processes, the model can learn the complex relationships between hyperparameters and adapt to prediction tasks under different masking ratios, thereby enhancing the model's robustness. The entire reverse process achieves automated optimization of ISP hyperparameters without manual intervention, improving optimization efficiency.

[0097] As an optional embodiment, the hyperparameter prediction model is trained by calculating regression loss only on the masked hyperparameters. During training, the model only focuses on the masked hyperparameter locations. For unmasked hyperparameters, since their true values ​​are already given, no loss needs to be calculated. The regression loss is calculated based on the difference between the predicted and true values ​​of the masked hyperparameters. The hyperparameter prediction model includes a visual feature extraction layer and a hyperparameter prediction layer. The visual feature extraction layer is responsible for extracting useful visual features from the original image. The hyperparameter prediction layer then predicts the masked hyperparameters based on these visual features and the partially masked ISP hyperparameter sequence.

[0098] The above-described S130 method, which trains the hyperparameter prediction model based on the ISP hyperparameter sequence of the partial mask and the corresponding original image to obtain the trained hyperparameter prediction model, includes:

[0099] S1301. Visual features are extracted from the original image based on the visual feature extraction layer.

[0100] The visual feature extraction layer in the hyperparameter prediction model is responsible for extracting features from the original image. This process utilizes deep learning techniques, such as operations within Convolutional Neural Networks (CNNs). Through a series of convolutions, pooling, and other operations, the visual feature extraction layer captures visual features of different levels and abstractions from the original image, such as image edges, textures, and color distribution. These visual features provide image contextual information for subsequent hyperparameter prediction.

[0101] S1302. Input the ISP hyperparameter sequence of the partial mask and the visual features into the hyperparameter prediction layer to obtain the prediction output of the masked hyperparameters.

[0102] The partially masked ISP hyperparameter sequence obtained through the forward pass and the visual features obtained from the visual feature extraction layer are input together into the hyperparameter prediction layer. The hyperparameter prediction layer is a neural network module with a specific structure that receives these two inputs and fuses and processes them. During this process, the model utilizes the image information provided by the visual features and some known hyperparameter information, through internal neuron computation and parameter adjustment, to predict the masked hyperparameters and output the prediction result.

[0103] S1303. Calculate the regression loss based on the predicted output of the masked hyperparameters and the true label, and update the model parameters of the hyperparameter prediction layer through backpropagation based on the regression loss; wherein, the true label is the true hyperparameter value in the ISP hyperparameter sequence.

[0104] Since the hyperparameter prediction model only calculates the regression loss on the masked hyperparameters, the regression loss is calculated based on the predicted output of the masked hyperparameters and the true hyperparameter values ​​(i.e., the true labels) in the ISP hyperparameter sequence. Common regression loss metrics such as mean squared error (MSE) can be used, measuring the model's prediction accuracy by calculating the difference between the predicted and true values. Based on the calculated regression loss, the model parameters of the hyperparameter prediction layer are updated using the backpropagation algorithm. The backpropagation algorithm adjusts the model parameters according to certain optimization strategies (such as stochastic gradient descent and its variants) based on the gradient information of the loss function with respect to the model parameters, making the model's prediction results gradually approach the true values, thereby continuously optimizing the model's performance.

[0105] This invention calculates regression loss only on the masked hyperparameters, allowing the model to focus on optimizing unknown hyperparameters that need to be predicted during training. This approach avoids ineffective computation and parameter tuning on known hyperparameters, improving training efficiency and focus, and enabling the model to learn more effectively how to accurately predict masked hyperparameters. The combination of the visual feature extraction layer and the hyperparameter prediction layer achieves effective fusion of image visual features and ISP hyperparameter information. Visual features provide background information on image scene and content for hyperparameter prediction, while some known hyperparameter information provides a direct reference for the model to predict masked parameters. This fusion method helps the model better understand the intrinsic relationship between the image and hyperparameters, thereby improving the accuracy of hyperparameter prediction. By updating model parameters through backpropagation based on regression loss, the hyperparameter prediction model can continuously adjust its parameter settings to reduce the gap between predicted and true values. As training progresses, the model's predictive ability gradually improves, enabling it to more accurately predict masked hyperparameters based on some known hyperparameter information and image visual features, laying the foundation for subsequently generating a complete optimized ISP hyperparameter sequence. This training method enables the model to not only learn the relationships between hyperparameters but also consider the influence of image visual features on hyperparameters. Therefore, the trained model can better utilize its learned knowledge to make predictions when faced with new, unseen images and partially masked hyperparameter sequences, exhibiting strong generalization ability and adapting to the ISP hyperparameter optimization needs in different scenarios.

[0106] As an optional implementation, the regression loss is determined in the following manner:

[0107] ;

[0108] In the formula, For regression loss; Indicates model parameters; This represents the total length of the ISP hyperparameter sequence, i.e., the total number of hyperparameters in the sequence; Indicates a mask marker; Indicates the indicator function. When the i-th hyperparameter in the ISP hyperparameter sequence of the partial mask... , equal to mask mark When the condition is met, the indicator function is 1; otherwise, it is 0. This ensures that the regression loss is calculated only for the masked hyperparameters. Indicates the preset loss function; The sequence of ISP hyperparameters representing a partial mask is sampled from the forward process; express The i-th hyperparameter in; Represents the original image; This represents the hyperparameter prediction model using a partially masked ISP hyperparameter sequence. and the original image When given input, the predicted output for the i-th hyperparameter; This represents the true hyperparameter value of the i-th hyperparameter in the ISP hyperparameter sequence, sampled from the training data.

[0109] During training, for the ISP hyperparameter sequence of partial masks For each hyperparameter position i in the algorithm, the indicator function is first used to determine whether the position is masked. If it is masked, the hyperparameter prediction model's prediction output for that position is then passed to the algorithm. Compared with the actual hyperparameter values Substitute them together into the preset loss function In the first step, calculate the loss value at that location. Then, sum the loss values ​​for all masked locations to obtain the regression loss of the entire model. .

[0110] Based on the calculated regression loss The loss function is calculated with respect to the model parameters using the backpropagation algorithm. The gradient is then calculated. Based on this gradient information, optimization algorithms (such as stochastic gradient descent, Adam, etc.) are used to optimize the model parameters. The model is updated so that its predictions gradually approach the true values, thereby continuously optimizing the model's performance.

[0111] This invention utilizes an indicator function, where the regression loss is calculated only for the masked hyperparameters. This allows the model to precisely focus on the hyperparameters to be predicted during training, avoiding ineffective calculations and parameter adjustments on known hyperparameters, improving training efficiency and focus, and enabling the model to learn more effectively how to accurately predict masked hyperparameters. A pre-defined loss function (such as mean squared error) effectively measures the difference between the hyperparameter prediction model's predicted output and the true value. By calculating this difference, the accuracy of the model in predicting each masked hyperparameter can be clearly understood, providing clear guidance for model optimization. Based on the backpropagation and parameter update process of the regression loss, the model can automatically adjust its parameters according to the prediction error. As training progresses, the model continuously optimizes parameter settings to reduce the gap between predicted and true values, thereby improving the model's prediction accuracy for masked hyperparameters. This loss calculation and parameter update method allows the model to not only focus on the correlation between hyperparameters during the learning process but also consider the influence of image information on hyperparameter prediction. Therefore, when faced with new, unseen images and partially masked hyperparameter sequences, the trained model can better utilize the learned knowledge to make predictions, has strong generalization ability, and can adapt to the ISP hyperparameter optimization needs in different scenarios.

[0112] As an optional embodiment, the visual feature extraction layer of the hyperparameter prediction model is a pre-trained convolutional neural network, and the hyperparameter prediction layer is a multi-layer bidirectional Transformer model. The training process of the hyperparameter prediction model is as follows:

[0113] ① A pre-trained convolutional neural network (CNN, such as ResNet) is used to extract visual features v from the original image I. Pre-trained CNNs, trained on large-scale datasets, possess powerful image feature extraction capabilities, capturing both low-level features (such as edges and textures) and high-level semantic features (such as object categories and scene information). After the original image I is input into the pre-trained CNN, it undergoes multiple convolutional and pooling operations to output fixed-dimensional visual features. This feature vector serves as one of the inputs to the subsequent Transformer model.

[0114] ② Sample the masking proportion t from the uniform distribution U(0,1). Based on t, independently mask the ISP hyperparameter sequence P0 used for training. Each hyperparameter is masked with probability t (replaced with the [MASK] label), and remains unchanged with probability 1-t, generating a partially masked ISP hyperparameter sequence P. t .

[0115] ③ Combine the visual feature v with the ISP hyperparameter sequence P of the partial mask. tThe input to the Transformer model is formed by concatenating or merging the hyperparameters. Visual features provide image contextual information, and the partially masked ISP hyperparameter sequence provides known hyperparameter constraints. Positional encoding is added to each label in the sequence to preserve sequence order information. The Transformer model utilizes information from all positions in the sequence simultaneously for prediction through a bidirectional attention mechanism. Unlike unidirectional models (such as TransformerDecoder), bidirectional models can capture contextual dependencies, improving prediction accuracy. The model outputs the predicted values ​​of the masked hyperparameters, filling the [MASK] positions in the sequence. Regression loss is calculated only at the masked positions, based on the predicted output of the masked hyperparameters and the true labels. The model parameters of the Transformer model are updated through backpropagation based on the regression loss. Optimize prediction performance.

[0116] Pre-trained CNNs, trained on large-scale datasets, can extract general and robust visual features, reducing the model's dependence on specific tasks and improving generalization ability. A bidirectional attention mechanism enables the model to utilize information from all positions in the sequence simultaneously, capturing complex dependencies between hyperparameters and improving prediction accuracy. By progressively reducing the mask ratio *t*, the model achieves a coarse-to-fine prediction paradigm, reducing the optimization difficulty in high-dimensional parameter spaces. Each step only requires predicting a subset of parameters, improving training stability. Unlike black-box optimization methods, this invention explicitly models the interrelationships between hyperparameters through iterative prediction and contextual learning, improving optimization performance. A random masking strategy simulates information loss in real-world scenarios, allowing the model to adapt to noisy environments during training, enhancing robustness in practical applications. From the original image input to the complete hyperparameter sequence output, the entire process is automated end-to-end, requiring no manual intervention and improving optimization efficiency.

[0117] The hyperparameter prediction model proposed in this invention can be effectively optimized for various downstream tasks. Specifically, this invention utilizes existing large-scale sRGB datasets and combines them with synthetic ISP for experiments. The proposed inverse mapping method is used in the experiments to convert sRGB images into RAW format as input data for the ISP. This invention is evaluated on the widely used COCO dataset, using a pre-trained YOLOv3 as the downstream object detection model, and employing the standard mean average precision (mAP) metric for performance evaluation. The method of this invention effectively improves the performance of downstream tasks by adaptively adjusting the texture features and color representation of the image.

[0118] The hyperparameter prediction model proposed in this invention can effectively optimize the high-dimensional parameter set of ISP. Existing ISP hyperparameter tuning methods, whether employing iterative optimization or direct prediction strategies, generally treat the ISP hyperparameter space as a holistic black box system, attempting to estimate all parameters in a single operation. In contrast, the method proposed in this invention introduces a stepwise iterative prediction paradigm, effectively reducing the overall complexity of the high-dimensional optimization space. Through coarse-to-fine overall prediction, more accurate parameter prediction is achieved. In the proposed iterative prediction framework, the model predicts the masked hyperparameters at each time step, gradually reducing the number of masked parameters to predict all parameters.

[0119] Taking image classification as an example of a downstream visual task, the construction and training process of the above hyperparameter prediction model is explained:

[0120] In image classification tasks, ISP hyperparameters need to be optimized to improve classification accuracy (e.g., Top-1 Accuracy). The hyperparameter prediction model needs to predict a set of ISP hyperparameters so that when the image processed by ISP is input into a classification model (e.g., ResNet), the classification performance is optimal. In the image classification task scenario, the objective function of the above hyperparameter prediction model... , used to represent the original image The optimal ISP parameter configuration for image classification tasks.

[0121] The visual feature extraction layer of the hyperparameter prediction model uses a pre-trained CNN, specifically the first few layers of ResNet (such as Stages 1-3 of ResNet-18) as feature extractors, from the original image. Extract low- to mid-level visual features v i (e.g., edges, textures, local semantics). Output feature map v i ∈R H×W×C Where R represents a three-dimensional tensor in real space; H and W are spatial dimensions, representing height and width respectively; and C is the number of channels, i.e., the feature dimension at each spatial location.

[0122] The hyperparameter prediction model employs a multi-layer bidirectional Transformer model for its hyperparameter prediction layer, which incorporates visual features v. i Flattened into sequence v i seq ∈R L×C (L=H×W), and the ISP hyperparameter sequence P of the partial mask. t The sequences are concatenated to form the input sequence X∈R of a multi-layer bidirectional Transformer model. (L+N’)×C(L is the length of the visual feature sequence, N' is the number of hyperparameters). Hyperparameters of a random scale t (t~U(0,1)) are masked, with the mask positions replaced by [MASK]. A multi-layer bidirectional Transformer model captures the global context through a self-attention mechanism to predict the masked hyperparameter values. The complete ISP hyperparameter sequence is output. ∈R N’ This includes noise reduction intensity, color matrix, sharpening parameters, etc.

[0123] The training process of the hyperparameter prediction model is as follows:

[0124] ① Data preparation: Collect RAW image datasets (such as SIDD, RENOIR), and label the true class y. i The true values ​​of the ISP hyperparameters are predefined (obtained through traditional tuning methods or expert experience). These true values ​​are for the original image. Optimal ISP parameter configuration for image classification tasks. RAW images are randomly cropped, flipped, and have their brightness adjusted to simulate different shooting conditions.

[0125] ② Input By transferring the data to ResNet, visual features v are obtained. i The hyperparameters of the random mask t ratio, and the ISP hyperparameter sequence P for generating partial masks. t Visual features v i Flattened into sequence v i seq , will v i seq and P t Input a multilayer bidirectional Transformer model and predict the hyperparameters of the masked component. Based on the predicted hyperparameters of the masked component and the true values ​​of the ISP hyperparameters, calculate the regression loss only for the masked component's hyperparameter positions using the regression loss formula above. Update the model parameters of the multilayer bidirectional Transformer model through backpropagation based on this regression loss.

[0126] The reasoning process of the hyperparameter prediction model is as follows:

[0127] During the inference phase, all hyperparameters are initialized to [MASK], resulting in a fully masked ISP hyperparameter sequence. Visual features v of the target image (RAW format) are extracted using ResNet. v and the current mask sequence (i.e., the full ISP hyperparameter sequence) are input into a multi-layer bidirectional Transformer model to predict the masked hyperparameters. The current ISP hyperparameter sequence is updated based on the prediction results, resulting in an updated ISP hyperparameter sequence. The mask ratio t is progressively reduced, and some hyperparameters of the updated ISP hyperparameter sequence are remasked based on the adjusted mask ratio to generate a new ISP hyperparameter sequence. The current ISP hyperparameter sequence is then updated to this new sequence. This iteration is repeated until t decreases to 0. After iteration, a complete optimized ISP hyperparameter sequence is obtained, which is used for ISP image processing. The processed image is then input into a classification model to obtain the predicted category.

[0128] The ISP hyperparameter iterative optimization device provided by the present invention is described below. The ISP hyperparameter iterative optimization device described below and the ISP hyperparameter iterative optimization method described above can be referred to in correspondence.

[0129] The ISP hyperparameter iterative optimization device provided by this invention refers to... Figure 5 As shown, it includes:

[0130] The model acquisition module 410 is used to acquire a pre-built hyperparameter prediction model, which is constructed based on a given original image and its corresponding downstream visual task.

[0131] The model training module 420 is used to progressively mask the ISP hyperparameter sequence in the training data through a forward process to obtain a partially masked ISP hyperparameter sequence; and to train the hyperparameter prediction model based on the partially masked ISP hyperparameter sequence and the corresponding original image to obtain the trained hyperparameter prediction model.

[0132] The iterative optimization module 430 is used to acquire a target image, initialize a full-mask ISP hyperparameter sequence for the target image, and use the target image and the full-mask ISP hyperparameter sequence as input to the trained hyperparameter prediction model, iteratively predict the masked hyperparameters through a reverse process, and output a complete optimized ISP hyperparameter sequence.

[0133] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communications bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communications bus 540. The processor 510 can call logical instructions from the memory 530 to execute the ISP hyperparameter iterative optimization method.

[0134] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the ISP hyperparameter iterative optimization method provided by the above methods.

[0136] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the ISP hyperparameter iterative optimization method provided by the methods described above.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An iterative optimization method for ISP hyperparameters, characterized in that, include: Obtain a pre-built hyperparameter prediction model, which is constructed based on a given original image and its corresponding downstream visual task; By progressively masking the ISP hyperparameter sequence in the training data through a forward process, a partially masked ISP hyperparameter sequence is obtained. Based on the ISP hyperparameter sequence of the partial mask and the corresponding original image, the hyperparameter prediction model is trained to obtain the trained hyperparameter prediction model. Acquire the target image and initialize a full mask ISP hyperparameter sequence for the target image; The target image and the fully masked ISP hyperparameter sequence are used as inputs to the trained hyperparameter prediction model. The masked hyperparameters are predicted iteratively through a reverse process, and the complete optimized ISP hyperparameter sequence is output. The reverse process includes: Input the current ISP hyperparameter sequence and the target image into the trained hyperparameter prediction model to predict the masked hyperparameters and obtain the prediction results; The current ISP hyperparameter sequence is updated based on the prediction results to obtain the updated ISP hyperparameter sequence; Adjust the mask ratio, and remask the updated ISP hyperparameter sequence based on the adjusted mask ratio to generate a new ISP hyperparameter sequence, and update the current ISP hyperparameter sequence with the new ISP hyperparameter sequence; Repeat the process of predicting masked hyperparameters, updating the sequence, adjusting the mask ratio, and re-masking until the mask ratio t drops to 0, then output the complete optimized ISP hyperparameter sequence. The hyperparameter prediction model is trained by calculating regression loss only on the masked hyperparameters; the hyperparameter prediction model includes a visual feature extraction layer and a hyperparameter prediction layer; The hyperparameter prediction model is trained based on the ISP hyperparameter sequence derived from the partial mask and the corresponding original image to obtain the trained hyperparameter prediction model, including: Visual features are extracted from the original image based on the visual feature extraction layer; The ISP hyperparameter sequence of the partial mask and the visual features are input into the hyperparameter prediction layer to obtain the prediction output of the masked hyperparameters; The regression loss is calculated based on the predicted output of the masked hyperparameters and the true labels. The model parameters of the hyperparameter prediction layer are then updated through backpropagation based on the regression loss. The true labels are the true hyperparameter values ​​in the ISP hyperparameter sequence.

2. The iterative optimization method for ISP hyperparameters according to claim 1, characterized in that, The masking operation in the forward process includes: Sample the ISP hyperparameter sequence from the training data; Uniformly sample mask proportion t from uniform distribution U (0,1); Each hyperparameter in the ISP hyperparameter sequence is independently masked with probability t, or... The probability of keeping the image unmasked is used to obtain a partially masked ISP hyperparameter sequence.

3. The iterative optimization method for ISP hyperparameters according to claim 1, characterized in that, The regression loss is determined as follows: ； In the formula, For regression loss; Indicates model parameters; Indicates the total length of the ISP hyperparameter sequence; Indicates a mask marker; Indicates an indicator function; Indicates the preset loss function; Represents the sequence of ISP hyperparameters for partial masking; express The i-th hyperparameter in; Represents the original image; This represents the hyperparameter prediction model using a partially masked ISP hyperparameter sequence. and the original image When given input, the predicted output for the i-th hyperparameter; This represents the actual hyperparameter value of the i-th hyperparameter in the ISP hyperparameter sequence.

4. The iterative optimization method for ISP hyperparameters according to claim 1, characterized in that, The visual feature extraction layer is a pre-trained convolutional neural network, and the hyperparameter prediction layer is a multi-layer bidirectional Transformer model.

5. An iterative optimization device for ISP hyperparameters, characterized in that, include: The model acquisition module is used to acquire a pre-built hyperparameter prediction model, which is constructed based on a given original image and its corresponding downstream visual task. The model training module is used to progressively mask the ISP hyperparameter sequence in the training data through a forward process to obtain a partially masked ISP hyperparameter sequence; and to train the hyperparameter prediction model based on the partially masked ISP hyperparameter sequence and the corresponding original image to obtain the trained hyperparameter prediction model. An iterative optimization module is used to acquire a target image, initialize a full-mask ISP hyperparameter sequence for the target image, and use the target image and the full-mask ISP hyperparameter sequence as input to the trained hyperparameter prediction model, iteratively predict the masked hyperparameters through a reverse process, and output a complete optimized ISP hyperparameter sequence. The reverse process includes: Input the current ISP hyperparameter sequence and the target image into the trained hyperparameter prediction model to predict the masked hyperparameters and obtain the prediction results; The current ISP hyperparameter sequence is updated based on the prediction results to obtain the updated ISP hyperparameter sequence; Adjust the mask ratio, and remask the updated ISP hyperparameter sequence based on the adjusted mask ratio to generate a new ISP hyperparameter sequence, and update the current ISP hyperparameter sequence with the new ISP hyperparameter sequence; Repeat the process of predicting masked hyperparameters, updating the sequence, adjusting the mask ratio, and re-masking until the mask ratio t drops to 0, then output the complete optimized ISP hyperparameter sequence. The hyperparameter prediction model is trained by calculating regression loss only on the masked hyperparameters; the hyperparameter prediction model includes a visual feature extraction layer and a hyperparameter prediction layer; The hyperparameter prediction model is trained based on the ISP hyperparameter sequence derived from the partial mask and the corresponding original image to obtain the trained hyperparameter prediction model, including: Visual features are extracted from the original image based on the visual feature extraction layer; The ISP hyperparameter sequence of the partial mask and the visual features are input into the hyperparameter prediction layer to obtain the prediction output of the masked hyperparameters; The regression loss is calculated based on the predicted output of the masked hyperparameters and the true labels. The model parameters of the hyperparameter prediction layer are then updated through backpropagation based on the regression loss. The true labels are the true hyperparameter values ​​in the ISP hyperparameter sequence.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the ISP hyperparameter iterative optimization method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the ISP hyperparameter iterative optimization method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the ISP hyperparameter iterative optimization method as described in any one of claims 1 to 4.

Images