Image signal processor optimization method and device

Abstract

The invention relates to an image signal processor optimization method and device. An optimization apparatus for an image signal processor (ISP) is provided, including processing circuitry. The processing circuit is configured to use a simulator of the image signal processor to process an original sample picture used for image signal processor optimization based on an ISP configuration parameter set to obtain a processed sample picture, and use an unsupervised pre-training task model for realizing a specific task to obtain an unsupervised evaluation score based on the processed sample picture, and updating the ISP configuration parameter set at least based on the unsupervised evaluation score, wherein the unsupervised evaluation score evaluates an effect of performing the specific task on the processed sample picture using the unsupervised task model.

Description

Technical Field

[0001] The present disclosure relates to image signal processing, and more particularly to optimization of image signal processors. Background Art

[0002] As electronic photographic equipment, such as various digital cameras and portable devices equipped with photographic equipment, becomes increasingly popular, more and more people use electronic photographic equipment to obtain photos, videos, etc. of various scenes.

[0003] Image signal processor (ISP) is the underlying image processing device in electronic photographic equipment. It is used to convert the original light signal captured by the optical sensor in the electronic photographic equipment to obtain images that can be viewed by the human eye on various display devices. It is widely used in current digital cameras, mobile phone cameras and other devices. The performance of ISP has a great impact on the quality of the final image captured. ISP generally provides a large number of configuration parameters for adjustment, and ISP manufacturers often have experts to tune the configuration parameters. Generally, the tuning target of ISP is the visual perception of the human eye, such as texture clarity, visual noise, etc.

[0004] Unless otherwise stated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Likewise, unless otherwise stated, problems identified with respect to one or more approaches should not be assumed to have been identified in any prior art based on this section. Summary of the invention

[0005] This Summary is provided to introduce concepts in a simplified form that are described in detail later in the Detailed Description.

[0006] In one aspect of the present disclosure, an optimization device for an image signal processor (ISP) is provided, the optimization device comprising a processing circuit configured to: use an image signal processor simulator to process an original sample image for image signal processor optimization based on an ISP configuration parameter set to obtain a processed sample image; based on the processed sample image, use an unsupervised pre-trained task model that implements a specific task to obtain an unsupervised evaluation score, wherein the unsupervised evaluation score evaluates the effect of performing the specific task on the processed sample image using the unsupervised task model; and update the ISP configuration parameter set based at least on the unsupervised evaluation score.

[0007] In another aspect of the present disclosure, an optimization method for an image signal processor (ISP) is provided, comprising: using an image signal processor simulator to process an original sample image for image signal processor optimization based on an ISP configuration parameter set to obtain a processed sample image; based on the processed sample image, using an unsupervised pre-trained task model that implements a specific task to obtain an unsupervised evaluation score, wherein the unsupervised evaluation score evaluates the effect of performing the specific task on the processed sample image using the unsupervised task model; and updating the ISP configuration parameter set based at least on the unsupervised evaluation score.

[0008] In yet another aspect, a device is provided, comprising at least one processor and at least one storage device having instructions stored thereon, which when executed by the at least one processor may cause the at least one processor to perform the method as described herein.

[0009] In yet another aspect, a storage medium storing instructions is provided, which when executed by a processor may cause the optimization method as described herein to be performed.

[0010] In still another aspect, an electronic device is provided, comprising an image signal processor; and the optimization device as described above, for optimizing a configuration parameter set of the image signal processor.

[0011] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The drawings described herein are used to provide a further understanding of the present disclosure, and each drawing together with the following detailed description is included in this specification and forms a part of the specification for explaining the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation on the present disclosure.

[0013] Figure 1 The conceptual arrangement of ISP in the capture architecture is shown.

[0014] Figure 2 An exemplary application scenario to which ISP tuning according to an embodiment of the present disclosure can be applied is shown.

[0015] Figure 3 is a flow chart illustrating an optimization method for ISP according to an embodiment of the present disclosure.

[0016] Figure 4 is a flow chart illustrating an optimization method for ISP according to an embodiment of the present disclosure.

[0017] Figure 5 is a flow chart illustrating a correction method for ISP optimization according to an embodiment of the present disclosure.

[0018] Figure 6 is a flowchart illustrating a method for ISP optimization based on multi-granularity processing according to an embodiment of the present disclosure.

[0019] Figure 7 is a flow chart of a method for ISP optimization according to an embodiment of the present disclosure.

[0020] Figure 8 An exemplary ISP automatic tuning framework according to an embodiment of the present disclosure is shown.

[0021] Fig. 9 An exemplary ISP automatic tuning process according to an embodiment of the present disclosure is shown.

[0022] Fig.10 Shows the effect of the global consistency loss model over training iterations.

[0023] Fig.11A and Fig. 11B Shows the effect of using different tuning methods.

[0024] Fig.12 The effect of unsupervised tuning according to an embodiment of the present disclosure is shown.

[0025] Fig.13 The average performance of unsupervised tuning according to an embodiment of the present disclosure is shown.

[0026] Fig.14 is a block diagram showing a tuning device for ISP according to an embodiment of the present disclosure.

[0027] Fig.15 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0028] Fig.16 A block diagram illustrating an exemplary hardware configuration of a computer system capable of implementing an embodiment of the present invention is shown.

[0029] It should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. The same or similar reference numerals are used in the various drawings to represent the same or similar parts. Therefore, once an item is defined in one drawing, it may not be further discussed in subsequent drawings. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure, but it is obvious that the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The drawings and the description of the embodiments below are actually only illustrative and are by no means any limitation to the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments described here.

[0031] In addition, when describing exemplary embodiments of the present disclosure in conjunction with the accompanying drawings below, for the sake of clarity and conciseness, not all features of the embodiments are described in the specification. It should be noted that in order to avoid obscuring the present disclosure due to unnecessary details, only processing steps and / or device structures closely related to at least the scheme according to the present disclosure are shown in the accompanying drawings, while other details that are not closely related to the present disclosure are omitted.

[0032] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be performed in different orders, and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown in the execution. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0033] In the present disclosure, the terms "first", "second", etc. are used merely to distinguish elements or steps, but are not intended to indicate time sequence, priority or importance.

[0034] ISP (Image Signal Processor) is an important component of electronic photographic equipment, such as various digital cameras and portable devices equipped with photographic equipment. Figure 1The conceptual arrangement of ISP in the shooting architecture is shown. In particular, when taking a photo, light enters the lens module through the lens module, and then the image sensor converts the light received by the lens into an electrical signal, which will be sent to the image signal processor. The image signal processor generates an image based on the received electrical signal to be presented to the user, or further processed, such as by an image processor (e.g., GPU), and the processing result can be presented to the user. In operation, the image signal processor (ISP) unit can perform a series of signal processing processes to make the image more beautiful in visual presentation and suitable for user observation. The processing performed by the ISP may include but is not limited to AEC (automatic exposure control), AGC (automatic gain control), AWB (automatic white balance), denoising, demosaicing, sharpening, color correction, gamma mapping, tone mapping, compression, etc. The signal generated by the ISP unit can be in any appropriate format, as long as the signal can be further processed or suitable for user viewing, for example, it can be an image in a format such as JPEG, JPG, etc.

[0035] ISP can be widely used in various image applications. For example, with the development of machine learning, a large number of images are used for computer vision tasks. However, ISP optimization / tuning, especially for advanced computer vision tasks such as autonomous driving, is still difficult.

[0036] Some attempts at ISP automatic tuning have been proposed.

[0037] One approach is to automatically tune the ISP based on expert understanding. For example, based on expert understanding, it is believed that the sharpness and contrast of an image will be helpful for computer vision tasks such as registration and pedestrian detection. Therefore, some parameters of the ISP are adjusted to enhance the image features that experts believe are effective. Although this approach does not require manual adjustment by experts, it still requires expert domain knowledge for judgment. At the same time, the expert's conclusion is often only superficially related to the task evaluation, which affects the final effect.

[0038] Another approach is to use the evaluation index of the task as the optimization target, but to simplify the optimization, the ISP is approximated or partially optimized. For example, the ISP function is abstracted into several CNNs and trained as a whole for downstream tasks. The problem is that there is no accurate correspondence between the CNN and the ISP parameters, and the results cannot be applied to the tuning of the ISP parameters, and the existing hardware cannot be utilized. For another example, the independent modules in the ISP are optimized separately. Although it reduces the difficulty of optimization, it also ignores the mutual influence between the modules, so only suboptimal solutions can be obtained. For another example, a proxy neural network is trained to simulate the ISP as a whole. However, since the noise caused by the simulation cannot be ignored, the results obtained based on the proxy neural network are also suboptimal.

[0039] Another idea is to treat ISP as a black box, take the task evaluation index as the optimization target, and directly tune the ISP parameters through a black box optimizer. The main difficulty is that there are many ISP parameters, and the evaluation index only contains the final prediction effect of the model, and the amount of information is single, making optimization more difficult. For example, an improved CMA-ES optimizer can be used to perform black box tuning on ISP parameters, which can explicitly tune each parameter while treating ISP as a whole. However, the main problem is that this tuning method only relies on the evaluation score of the computer vision task, so the tuning efficiency is low and the optimization effect may not be good.

[0040] In addition, some existing ISP optimization methods are often optimized for specific downstream tasks, such as image classification or object detection. The optimized ISP can only achieve good results for specific tasks, but cannot achieve results across multiple tasks. For example, an ISP optimized for image classification is more effective for image classification, but may not be good for object detection.

[0041] The present disclosure provides an improved ISP tuning scheme. The present disclosure uses an unsupervised pre-trained task model to perform a specific task on a sample image processed by an ISP simulator based on an ISP configuration parameter set, calculates an unsupervised evaluation score to evaluate the effect (e.g., loss) of the execution of the specific task on the processed sample image, and updates the ISP configuration parameter set based at least on the unsupervised evaluation score, so that the updated ISP configuration parameter set is more likely to result in a better evaluation score. In the scheme of the present disclosure, a pre-trained unsupervised task model (i.e., a deep learning backbone network) can be used without the participation of experts. Moreover, the ISP is not simplified, but the ISP of actual application is simulated, so that the scheme is more suitable for actual application. In addition, the scheme of the present disclosure does not optimize for a specific downstream computer vision task, but uses the effect of the unsupervised pre-trained task model (i.e., a deep learning backbone network) performing a specific task on the processed sample image as an evaluation index, which is highly correlated with the deep learning backbone network, thereby showing robustness to different downstream tasks. Therefore, using this evaluation index is more conducive to the optimizer to tune the ISP parameters, achieving better results than directly tuning based on specific task evaluation indicators.

[0042] On the one hand, the present disclosure can use the effect (loss) of the unsupervised pre-trained model on the ISP-processed picture to reflect the quality of the ISP-processed picture input to the model, and on the other hand, it can also combine the consistency loss to evaluate whether the content of the ISP-processed picture is consistent with the content of the original picture. The present disclosure can also use a multi-objective strategy to comprehensively optimize the loss of the unsupervised model and the loss of the consistency model, so that the ISP-processed picture can ultimately achieve better results for various downstream tasks.

[0043] Figure 2 An exemplary application scenario to which ISP tuning according to an embodiment of the present disclosure can be applied is shown.

[0044] As shown in the figure, the optical sensor processes the incident light to generate an electrical signal and then converts it into a digital image, such as a 16-bit or 24-bit Bayer image. It can be called a raw picture.

[0045] The optical sensor sends the original image to the image signal processor (ISP). The ISP performs various processing on the original image based on its ISP configuration parameters, such as white balance, demosaicing, denoising, sharpening, tone and color correction, compression, etc., and outputs an 8-bit RGB image, that is, the image processed by the ISP.

[0046] On the one hand, a pre-trained unsupervised computer vision model can be used to perform a specific task on the ISP-processed graphics, and an unsupervised evaluation score can be obtained to evaluate the performance (e.g., loss) of the specific task on the ISP-processed image.

[0047] In this exemplary scenario, the unsupervised computer vision model is, for example, based on a generated unsupervised visual task model (e.g., MAE (Masked Autoencoders)), which performs random masking on an 8-bit RGB image to generate a random mask image. The random mask image is input into the pre-trained unsupervised computer vision model (e.g., MAE). The unsupervised computer vision model outputs a picture reconstruction prediction, i.e., a predicted reconstructed picture. The loss of the reconstructed picture relative to the picture processed by ISP is calculated as the unsupervised evaluation score.

[0048] On the other hand, a consistency loss model can be used to calculate the global consistency loss for 8-bit RGB images.

[0049] The ISP parameters can be adjusted based on the supervised evaluation score and the global consistency loss. Specifically, based on the unsupervised evaluation score and the global consistency loss, a multi-objective learning strategy can be used to calculate the total evaluation score, and then the ISP configuration parameters can be adjusted / updated based on the total evaluation score.

[0050] Figure 3 is a flow chart illustrating an optimization method 300 for ISP according to an embodiment of the present disclosure.

[0051] As shown in diagram 300 , method 300 may include operation 301 , in which an original sample picture for image signal processor optimization is processed based on an ISP configuration parameter set using a simulator of an image signal processor to obtain a processed sample picture.

[0052] The simulator of the image signal processor can simulate a specific hardware ISP, and its configuration parameters can correspond to the configuration parameters of the hardware ISP. The effect obtained by a set of configuration parameters on the ISP simulator corresponds to the effect achieved on the corresponding hardware ISP. The ISP simulator can provide an interface for adjusting the configuration parameters. The ISP simulator itself can be a black box. The ISP simulator used in the present disclosure can be, for example, the hardware ISP of the Sony FUJI sensor, and the basic functions can include demosaicing, white balance, noise reduction, sharpening, tone mapping, bit length compression, etc.

[0053] In some embodiments, the ISP simulator may include a noise reducer based on bilateral filtering and Gaussian filtering, an edge enhancement based on high-pass filtering, and a tone mapper based on the Durand tone mapping algorithm. The ISP simulator may simulate several important functions of the SonyFuji Family ISP. In order to simulate the discrete characteristics of the parameters in the hardware ISP, the parameters used in the ISP simulator may also be discrete.

[0054] The ISP configuration parameter set may be generated by an optimizer (eg, a black box optimizer). An ISP configuration parameter set may include multiple configuration parameters corresponding to multiple functions to be implemented by the optimizer.

[0055] The optimizer can generate multiple continuous values ​​that are the same as the number of configuration parameters of the hardware ISP based on the current internal state of the optimizer. These values ​​have a one-to-one correspondence with the configuration parameters of the hardware ISP. In some embodiments, the generated multiple values ​​can be processed according to the range and value type of the actual parameters of the ISP hardware to make them meet the parameter requirements of the ISP hardware. For example, the values ​​that exceed the corresponding parameter range can be truncated, scaled, etc. to make them meet the parameter range requirements. For another example, if the parameter type is discrete, the continuous values ​​can be converted into discrete values ​​by rounding. The processed multiple values ​​can be used as ISP configuration parameters provided to the ISP simulator.

[0056] The original image sample is an original image that has not been processed by the image signal processor. The original sample image used for image signal processor optimization is, for example, an original image input by a sensor and not processed by the ISP, such as a Bayer image.

[0057] A plurality of original sample images may be input into the ISP simulator in the form of a data stream, and the ISP simulator may output a plurality of ISP-processed sample images in the form of a data stream.

[0058] Method 300 may also include operation 303, in which an unsupervised evaluation score is obtained based on the processed sample image using an unsupervised pre-trained task model that implements a specific task, wherein the unsupervised evaluation score evaluates the effect of performing the specific task on the processed sample image using the unsupervised task model.

[0059] An unsupervised pre-trained task model refers to an unsupervised task model that has been pre-trained using training data.

[0060] In some embodiments, the unsupervised pre-training task model can be an unsupervised pre-training task model based on generation, including but not limited to MAE, BeiT and BeiTv2. In some embodiments, the unsupervised pre-training task model can be an unsupervised pre-training task model based on comparison, such as SimCLR. Those skilled in the art will appreciate that other unsupervised pre-training task models may also be used.

[0061] Existing methods rely on effective labeling of data sets for ISP tuning, but there are problems such as errors in data set labeling errors and complex data labeling work. According to some embodiments of the present disclosure, for example, a MAE model (or a Vistion Transformer trained using MAE-like techniques) can be used to unsupervisedly calculate the loss score of the model reconstruction result under the current image quality. Because the MAE model is trained using a large-scale image data set, the pre-trained MAE model contains prior information about the image ISP parameters. The model uses this prior information and uses, for example, 25% of the image area as known information to predict the reconstruction of the image color and texture of, for example, 75% of the unknown image area, and obtains the loss score of the current input image based on the prediction results. Compared with the prior art method that uses a large number of manually labeled data sets for optimization, the model can obtain a quantitative evaluation of the image quality of the current ISP parameters without labels.

[0062] According to some embodiments of the present disclosure, the unsupervised method does not require dataset annotation, so multiple existing public datasets are used for simultaneous training and testing to reflect the improvement in the effect of ISP automatic tuning. KITTI and Cityscape are commonly used datasets in the field of autonomous driving, and COCO is a commonly used dataset for daily life scenes. Generally, object recognition is performed based on the KITTI and COCO datasets, and image segmentation is performed based on the Cityscape dataset. When training an unsupervised task model, pre-training without annotation can be performed across multiple datasets. This enables the images output by the ISP that is tuned for task execution effects based on pre-trained unsupervised models to provide generally good effects and robustness across a variety of downstream visual tasks.

[0063] Different unsupervised pre-training task models implement different specific tasks and calculate unsupervised evaluation scores differently. For example, MAE randomly masks the ISP-processed image, predicts and reconstructs the randomly masked image to obtain the reconstructed image, and then calculates the loss based on the mean square error of the pixels in the reconstructed area and the corresponding area in the ISP-processed image as the unsupervised evaluation score. SimCLR can perform different amplifications on each image in a batch of ISP-processed images, extract features from the amplified images, and calculate the contrast loss as the unsupervised evaluation score.

[0064] The unsupervised evaluation score can evaluate the effect of the unsupervised pre-trained task model in performing its specific task on the ISP-processed images, that is, the loss. If the unsupervised evaluation score shows a large loss, it means that the quality of the ISP-processed images is not ideal, which in turn reflects that the settings of the ISP configuration parameters may need further optimization; if the unsupervised evaluation score shows a small loss, it means that the quality of the ISP-processed images is relatively ideal, which in turn reflects that the settings of the ISP configuration parameters are relatively ideal.

[0065] Method 300 may also include operation 305, in which the ISP configuration parameter set is updated based at least on the unsupervised evaluation score.

[0066] In some embodiments, the optimizer may update the internal state of the optimizer based at least on the unsupervised evaluation score, thereby generating an updated ISP configuration parameter set based on the updated internal state. The updated ISP configuration parameter set will make the evaluation score corresponding to the updated ISP configuration parameter set closer to a better evaluation score and away from a worse evaluation score.

[0067] In some embodiments, the updating of the ISP configuration parameter set may be performed iteratively. This may include, in subsequent iterations, using a simulator of an image signal processor to process a next original sample image based on the updated ISP configuration parameter set to obtain a next processed sample image, obtaining an unsupervised evaluation score based on the next processed sample image using an unsupervised pre-trained task model that implements a specific task, and further updating the updated ISP configuration parameter set based at least on the unsupervised evaluation score.

[0068] The termination condition of the iteration may be that the evaluation score corresponding to the ISP configuration parameter set meets the preset threshold condition, or the evaluation score meets the preset threshold condition continuously for a preset number of times, or the number of iterations reaches a preset threshold number, etc.

[0069] Figure 4 is a flow chart illustrating an optimization method 400 for ISP according to an embodiment of the present disclosure.

[0070] As shown in diagram 400 , method 400 includes operation 401 , in which an original sample picture for image signal processor optimization is processed based on an ISP configuration parameter set using a simulator of an image signal processor to obtain a processed sample picture.

[0071] Method 400 also includes operation 403, in which an unsupervised evaluation score is obtained based on the processed sample image using an unsupervised pre-trained task model that implements a specific task, wherein the unsupervised evaluation score evaluates the effect of performing the specific task on the processed sample image using the unsupervised task model.

[0072] Operation 401 and 403 and Figure 3 Operations 301 and 303 are similar and will not be described in detail here.

[0073] The method 400 may also include an operation 405 , in which an overall evaluation score is calculated based at least on the unsupervised evaluation score.

[0074] In some embodiments, the total evaluation score may be calculated based only on the unsupervised evaluation score. For example, the calculated unsupervised evaluation score is used as the total evaluation score.

[0075] In other embodiments, a global consistency loss may be calculated based on the original sample images and the processed sample images, and a multi-objective learning strategy may be used to calculate a total evaluation score based on both the unsupervised evaluation score and the global consistency loss.

[0076] In some embodiments, the global consistency loss can be calculated based on the similarity of the pixel value distribution between the original sample image and the processed sample image or the similarity of the size relationship between the pixel and the surrounding pixels. The pixel values ​​of the image before and after ISP processing cannot be directly compared by the value itself, so the content consistency can be judged by the pixel value distribution or the size relationship between the pixel and the surrounding pixel values.

[0077] In some embodiments, the unsupervised task model can be a MAE model, which depends on the surrounding image pixels. In the early stage of ISP tuning, because the initial parameters are random, the quality of the processed image is generally poor. Therefore, the MAE model is prone to excessive information loss during the ISP optimization process, resulting in the optimizer may not be able to obtain enough information to determine the direction of parameter optimization, resulting in special cases of all black or all white. According to some embodiments, a consistency loss model for the original image samples and the processed image samples can be established, and the global consistency loss can be calculated for the overall pixel distribution of the image before and after ISP processing. A linear function passing through the origin can be used to calculate the global consistency loss.

[0078] An exemplary calculation flow of the global consistency loss can be as follows:

[0079] (1) Using the least squares method, calculate the linear model, slope

[0080] (2) Calculate the global consistency loss according to the following formula:

[0081] Among them, x i ,y i are the corresponding pixel values ​​of the original image and the processed image respectively.

[0082] Since the MAE score changes directly according to the image quality, there is still a good tuning signal in the early stage, so it can guide ISP tuning more quickly. At the same time, the consistency loss can reflect the structural information of the image, which can prevent the MAE model from over-optimizing the ISP. Fig.10 shown. Fig.10 The effect of the global consistency loss model in training iterations is shown, and the pixel distribution of the corresponding image is approximately linear. Fig.12 In the pixel distribution diagram, the horizontal axis represents the pixel value before ISP processing after taking the log value, and the vertical axis represents the pixel value after ISP processing. Fig.10 It can be seen that after unsupervised combined with global consistency loss tuning, the image information obtained through the unsupervised training model is retained, and the pixel distribution of the original image and the image after ISP processing is basically the same, that is, the content of the image before and after processing is basically the same.

[0083] A multi-objective learning strategy can be used to comprehensively consider both the unsupervised evaluation score and the consistency loss score. The multi-objective learning strategy may include, but is not limited to, dynamic weighted averaging, gradient normalization, dynamic task priority, and uncertainty weighting.

[0084] In some embodiments, a dynamic weighted average can be used. According to the multi-objective learning strategy, the respective weight coefficients can be calculated in real time for the unsupervised evaluation score and the global consistency loss, and the weighted average of the two can be calculated as the total evaluation score.

[0085] In some embodiments, the optimization goal of the optimizer is to optimize both the unsupervised evaluation score and the consistency loss. In different cases, different weights can be used for the unsupervised evaluation score and the consistency loss. The unsupervised evaluation score (e.g., the MAE prediction loss evaluation score) and the consistency loss evaluation score are calculated in real time according to the dynamic weighted average strategy during multiple rounds of iterative optimization. The loss evaluation score and the corresponding weight are multiplied and summed to obtain the total evaluation score, and the effect is as follows: Fig.11A shown. Fig.11A Shows the effect of using different tuning methods. Fig.11A Shows real pictures of the dataset, the effect of tuning using existing methods, and the effect of tuning using only MAE score, tuning using only consistency loss, tuning using MAE+consistency loss, and tuning using MAE+consistency loss+dynamic weighted average.

[0086] Method 400 may also include operation 405, in which the ISP configuration parameter set is updated based on the total evaluation score.

[0087] In some embodiments, the total evaluation score calculated based on the current ISP configuration parameter set may be input to the optimizer, so that the optimizer can optimize its internal state based on the total evaluation score, thereby generating an updated ISP configuration parameter set based on the updated internal state.

[0088] Figure 5 is a flow chart showing a correction method 500 for ISP optimization according to an embodiment of the present disclosure.

[0089] like Figure 5 As shown, method 500 may include operation 501 , in which it is determined whether the unsupervised evaluation score is less than a threshold.

[0090] The unsupervised evaluation score is, for example, the unsupervised evaluation score calculated by operation 303 or 403 .

[0091] Generally speaking, it is desirable to continuously optimize the unsupervised evaluation score to be as small as possible. However, when the unsupervised evaluation score is less than an extremely low value or close to zero, it may indicate that the ISP configuration parameter set generated by the previous iteration is inappropriate. Those skilled in the art may set the threshold based on experience.

[0092] Method 500 may further include operation 503 , in which, in response to determining that the unsupervised evaluation score is less than the threshold, the original sample image used in the previous ISP optimization iteration is reloaded and the unsupervised evaluation score is recalculated.

[0093] The method 500 may further include an operation 505 of downscaling the recalculated unsupervised evaluation score and using the downscaled unsupervised evaluation score as a new unsupervised evaluation score.

[0094] When the unsupervised evaluation score calculated during the current ISP optimization iteration is less than the threshold, this means that there may be a problem with the ISP configuration parameter set generated by the previous ISP optimization iteration. Therefore, the original sample image used in the previous iteration is reloaded, the unsupervised evaluation score is recalculated, and the recalculated unsupervised evaluation score is processed before completing the current ISP optimization iteration. The recalculated unsupervised evaluation score can be reduced by a predetermined ratio, for example, to one tenth of the original score, to reduce the impact of the unsupervised evaluation score on the total evaluation score. This correction strategy can promptly detect errors in the ISP optimization iteration process and make effective corrections.

[0095] Figure 6 is a flow chart showing a method 600 for ISP optimization based on multi-granularity processing according to an embodiment of the present disclosure.

[0096] like Figure 6 As shown, method 600 may include operation 601, in which a plurality of pictures with granularity (ie, resolution) ranging from high to low are generated based on the processed sample picture.

[0097] After processing, the sample image is Figure 3 Operation 301 or Figure 4 The sample image processed by the ISP simulator is obtained in operation 401. Methods for generating multiple images with high to low granularity include but are not limited to image pyramid and downsampling.

[0098] The method 600 may include an operation 603 , in which, for each of the plurality of images, an unsupervised evaluation score and a global consistency loss of the image are respectively calculated.

[0099] The method 600 may include an operation 605 , in which a total evaluation score corresponding to the processed sample image is calculated.

[0100] In some embodiments, this may include obtaining a multi-granularity unsupervised evaluation score corresponding to the processed sample image by calculating the weighted sum of the corresponding multiple unsupervised evaluation scores of the multiple images, obtaining a multi-granularity global consistency loss corresponding to the processed sample image by calculating the weighted sum of the corresponding multiple global consistency losses of the multiple images, and calculating a total evaluation score corresponding to the processed sample image based on the multi-granularity unsupervised evaluation scores and the multi-granularity global consistency losses using a multi-objective learning strategy.

[0101] For pictures with higher granularity, the weight of the unsupervised evaluation score can be set higher, and the weight of the global consistency loss score can be set lower. For pictures with lower granularity, the weight of the unsupervised evaluation score can be set lower, and the weight of the global consistency loss score can be set higher. Those skilled in the art can set specific weight values ​​as needed.

[0102] Method 600 may include operation 607 , in which the ISP configuration parameter set is updated based on the total evaluation score corresponding to the processed sample picture.

[0103] In the disclosed embodiments, multiple data sets in different computer vision fields can be used for ISP tuning, and the image sizes of different data sets may be different. Since the unsupervised evaluation scores (such as MAE prediction loss evaluation scores) and consistency loss evaluation scores focus on information understanding capabilities. The unsupervised task model reconstructs the image by understanding the image content, so the unsupervised task model understands large-scale spatial semantic information. The consistency loss model calculates the pixel distribution before and after ISP processing, so the consistency loss model understands the texture information of smaller-sized images. Therefore, for image data from different data sets, image data of different scales are randomly cropped according to a certain ratio, and the weights of the unsupervised evaluation score and the consistency loss score of the current image are linearly obtained according to the image scale. The evaluation score of the current ISP-processed image is obtained based on the weight calculation. From the back Fig.12 It can be seen that better results can be achieved when the image pyramid is used to generate and optimize images of 8 granularities.

[0104] Figure 7 is a flow chart of a method 700 for ISP optimization according to an embodiment of the present disclosure.

[0105] As shown in the figure, method 700 may include operation 701, in which multiple total evaluation scores are obtained, each total evaluation score corresponding to an ISP configuration parameter set generated by an optimizer for an image signal processor simulator based on a current internal state, and associated with a processed sample image obtained by processing the same original sample image based on the ISP parameter set.

[0106] In order to achieve optimization stability, based on the current internal state of the optimizer, multiple ISP configuration parameter sets can be continuously generated. For the same input original sample image, the ISP simulator can use these multiple ISP configuration parameter sets to process the same original sample image respectively, and obtain multiple corresponding processed sample images. For these multiple processed sample images, the total evaluation scores are calculated respectively, and multiple corresponding total evaluation scores can be obtained.

[0107] The method 700 may further include an operation 703 , in which an internal state of the optimizer is updated based on the plurality of total evaluation scores and the corresponding plurality of ISP configuration parameter sets.

[0108] The obtained corresponding multiple total evaluation scores and the corresponding multiple ISP configuration parameter sets can be input into the optimizer so that the optimizer can update the internal state accordingly.

[0109] Method 700 may further include operation 705, in which an updated ISP configuration parameter set is obtained using an optimizer based on the updated internal state, wherein the updated ISP configuration parameter set is capable of causing a total evaluation score corresponding to the updated ISP configuration parameter set to be close to a better total evaluation score among the multiple total evaluation scores and away from a worse total evaluation score among the multiple total evaluation scores.

[0110] Based on the updated internal state, the optimizer can generate a new set of ISP configuration parameters for use in the next ISP optimization iteration.

[0111] Figure 8 An exemplary ISP automatic tuning framework according to an embodiment of the present disclosure is shown.

[0112] As shown in Figure 800, the automatic tuning framework includes an ISP simulator, a black box optimizer, a consistency loss model, a multi-objective learning strategy, an evaluation index calculation module, and involves a tuning sample module and an unsupervised computer vision model.

[0113] Those skilled in the art will understand that Figure 8 The automatic tuning framework shown may include more or fewer modules. For example, the tuning sample module and the unsupervised computer vision model may be part of the tuning framework or may be independent of the tuning framework. In some cases, one or more of the consistency loss model, the multi-objective learning strategy, or the evaluation index calculation module may be omitted.

[0114] Fig. 9 An exemplary ISP automatic tuning process according to an embodiment of the present disclosure is shown.

[0115] The tuning sample module may, for example, obtain / provide a sample image for ISP optimization. The sample image may be a raw image output from a sensor without ISP processing.

[0116] The ISP automatic tuning framework may read the tuning sample (operation 901). The tuning sample is input into the ISP simulator for processing.

[0117] The ISP automatic tuning framework may use an optimizer to generate one or more sets of configuration parameters for an ISP simulator, and call the ISP simulator (operation 902) to process an input image, such as a read tuning sample image, according to the configuration parameters.

[0118] The image output by the ISP simulator, that is, the ISP-processed image, can be used as input to an unsupervised computer vision model, which can be called (operation 903) to perform a specific task on the ISP-processed image.

[0119] The ISP automatic tuning framework may receive the result output by the unsupervised computer vision model, such as an unsupervised evaluation score. In some embodiments, a MAE loss score may be calculated (operation 905) and the MAE loss score may be fed back (operation 906).

[0120] In some embodiments, for the results obtained due to model prediction errors, a model score correction strategy can be called to obtain a scaled output result. The model score correction strategy is implemented, for example, as described in method 500.

[0121] The ISP automatic tuning framework can receive the input sample image and the ISP processed image output by the ISP simulator, call the consistency loss model (operation 903) for calculation and processing, and obtain the output result of the consistency loss model, such as the data distribution consistency evaluation index. That is, the consistency loss score can be calculated (operation 907) and the consistency loss score can be fed back (operation 908).

[0122] The ISP automatic tuning framework may receive the unsupervised visual model output and the consistency loss model output, and calculate the overall evaluation index according to the multi-objective learning strategy. A dynamic weighted average weight may be determined for the unsupervised evaluation score and the consistency loss, and the overall evaluation index may be calculated (operation 909).

[0123] The ISP automatic tuning framework may feed back the total evaluation index to the optimizer to update the optimizer state (operation 910).

[0124] The optimizer generates updated ISP parameters according to the updated optimizer state (operation 911).

[0125] In some embodiments, the tuning sample images can be input in the form of data streams, the ISP simulator can be called for processing, and the processed images can be output to the unsupervised computer vision model in the form of data streams. Since the tuning data set contains multiple samples, this plays an important role in improving the efficiency of the framework and saving system resources.

[0126] In some embodiments, the unsupervised computer vision model is based on a generated unsupervised computer vision model (e.g., Mean Squared Error, MAE), and the evaluation index calculation module can calculate the mean square error (MSE) loss based on the output result of the model reconstruction and the image output by the ISP, as an unsupervised evaluation score to evaluate the task effect. At the same time, the evaluation index calculation module can evaluate the difference in data distribution based on the overall pixel value of the input sample image and the image output by the ISP simulator, and calculate the global consistency loss. Afterwards, the weights can be dynamically learned during the optimization process based on a multi-objective learning strategy. Based on the loss evaluation score and weight, the evaluation index calculation module can give an evaluation index that reflects the effect of fitting the data distribution after ISP processing, which is used to update the internal state of the optimizer.

[0127] The ISP simulator can receive input raw image samples (such as 24-bit Bayer images) and provide an interface for adjusting parameters. According to the configuration parameters and the input raw image samples, the ISP simulator outputs the ISP-processed image. The ISP simulator itself can be a black box, and the implementation of the internal structure has no effect on the implementation of the present disclosure. The ISP simulator can simulate a specific hardware ISP, and its parameter effects can correspond one-to-one with the hardware ISP. The effect obtained by a set of parameters on the ISP simulator is consistent with the effect on the corresponding hardware ISP. The ISP simulator used in the present disclosure can correspond to the hardware ISP of the Sony FUJI sensor, and the basic functions can include but are not limited to de-mosaicing, white balance, noise reduction, sharpening, tone mapping and bit length compression.

[0128] The input of the computer vision model used in the present disclosure can be an image processed by ISP (e.g., a 3-channel 8-bit RGB image), and the output is the execution result of the corresponding task. The computer vision model in the present disclosure is a convolutional neural network (CNN). In the present disclosure, CNN is a neural network that has been trained to achieve a specific task.

[0129] Different models can be used for different tasks. When training computer vision models for different specific tasks, you can choose the corresponding training set. For example, for object detection, you can use the KITTI and COCO datasets to train YOLOv8. For image segmentation, you can use the Cityscape dataset to train BiseNetv2.

[0130] Unsupervised computer vision models, such as MAE, BeiT, BeiTv2, or SimCLR, can use large-scale unlabeled training datasets. Unsupervised computer vision models can use multiple existing public datasets for training and testing at the same time to reflect the improvement of ISP automatic tuning. KITTI and Cityscape are commonly used datasets in the field of autonomous driving, and COCO is a commonly used dataset for daily life scenes. Object recognition is generally based on KITTI and COCO datasets, and image segmentation is based on Cityscape datasets. After ISP tuning based on the task execution effect of the unsupervised computer vision model, the images processed by ISP can show generally good results and robustness across various downstream visual tasks.

[0131] The optimizer may be a CMA-ES optimizer. The optimizer may update the internal state of the optimizer according to the total evaluation score fed back from the evaluation index calculation module. The optimizer may first sort the evaluation scores from best to worst, and use the values ​​corresponding to the best evaluation scores as positive values, and the values ​​corresponding to the worst evaluation scores as negative values. The optimizer may update the internal state according to the positive values ​​and the negative values, so that the mean of the newly generated values ​​will be closer to the positive values, and more likely to be away from the negative values.

[0132] According to the embodiments of the present disclosure, optimization can be performed based on the CMA-ES optimizer. The optimization goal is to simultaneously reduce the prediction loss of the computer vision model for the mask area of ​​the image and the consistency alignment loss before and after the image sample processing. The ultimate goal is to enable the image after ISP processing to obtain higher evaluation indicators in multiple downstream computer vision tasks.

[0133] An exemplary ISP optimization method according to an embodiment of the present disclosure is described below, which may include the following steps:

[0134] a) Randomly select a set of numbers equal to the number of internal parameters of the optimizer as the initial value of the optimizer.

[0135] b) Automatic tuning frameworks (e.g. Figure 8 The automatic tuning framework shown in FIG. 1 calls the optimizer to generate a set of multiple continuous values ​​equal to the number of ISP parameters. These values ​​have a one-to-one correspondence with the ISP parameters. The framework processes the multiple values ​​generated according to the range and value type of the actual ISP parameters to make them meet the ISP parameter requirements. The processing may include truncating, scaling, or reflecting the values ​​that exceed the corresponding parameter range to make them meet the parameter range requirements. If the parameter type is discrete, the continuous value is converted into discrete by rounding.

[0136] c) The framework uses the processed numbers in b) to set ISP parameters, and calls the ISP simulator to process the original image samples based on the set parameters to obtain sample images after ISP processing.

[0137] d) The framework calls the unsupervised computer vision model to operate on the random mask image generated after the sample image processed by ISP is randomly masked, and obtains the prediction result of the mask area of ​​the image output by the model.

[0138] e) The image predicted by the model is the same size as the sample image after ISP processing. The reconstructed area of ​​the image is compared with the sample image after ISP processing. The mean square error (MSE) loss is calculated based on each pixel in the mask area of ​​the reconstructed image and the corresponding pixel in the sample image after ISP processing. The reconstruction result of the mask area of ​​the image can reflect the difference between the image processed by ISP parameters and the massive image data used in unsupervised model training. This difference in data domain is used as the unsupervised evaluation score of the sample image after ISP processing.

[0139] f) When the unsupervised evaluation score of the image is less than the specified threshold, it is determined that the ISP optimization iteration has failed. A correction strategy is used to reload the image data used in the previous ISP optimization iteration, and the weight of the unsupervised evaluation score is reduced proportionally to obtain a new evaluation score.

[0140] g) The framework calculates the overall pixel distribution of the image before and after ISP processing to obtain the global consistency loss. The specific calculation formula is as follows:

[0141]

[0142] Among them, x i ,y i are the corresponding pixel values ​​of the original image and the processed image respectively. f(·,θ) is a monotonically increasing function. The simplest one can be a linear function, while the more complex one can be a neural network.

[0143] h) The framework calculates the weight coefficients of the unsupervised evaluation score and the global consistency loss in real time according to the multi-objective learning strategy. For example, dynamic weighted averaging can be used, and the weight coefficient is calculated as follows: where ω i (t) represents the weight of each task i, L n (t-1),r n (t-1) represents the loss score and training speed of task n at step t-1, N represents the total number of tasks, and T is a constant. The loss evaluation score is multiplied by the corresponding weight and then summed to get the total evaluation score.

[0144] i) When using the multi-granularity image learning strategy, generate multi-granularity images based on the ISP-processed images in c), and calculate the corresponding unsupervised evaluation scores and global consistency losses and corresponding weights for different granularity images according to steps from d) to h). According to the different granularity of the images, the unsupervised evaluation scores of images with high granularity are given higher weights, while the global consistency losses of images with low granularity are given higher weights, and the overall evaluation score is obtained by summing the weights.

[0145] j) In order to optimize more stably, steps b) to h) or i) may be repeated for a predetermined number of times to obtain multiple sets of values ​​and corresponding loss scores generated by the current optimizer. For example, it may be repeated 16 times.

[0146] When steps b) to h) or i) are repeatedly executed, the same original sample image is used. Different original sample images may be used for different rounds of ISP optimization iterations.

[0147] k) When the number of executions from b) to h) or i) in step j) is 1, the internal state of the optimizer is updated according to the total evaluation score calculated by the ISP optimization iteration, and then the optimizer generates updated ISP parameters based on the updated internal state; when the number of executions from b) to h) or i) in step j) is, for example, 16 times, 16 sets of ISP parameters and corresponding 16 total evaluation scores are input into the optimizer to update the internal state of the optimizer, and then the optimizer generates updated ISP parameters based on the updated internal state. The optimizer compares the evaluation scores and updates the internal state, so that the newly generated value is more likely to be close to the value with a good evaluation score and more likely to be away from the value with a poor evaluation score.

[0148] l) In order to obtain better ISP tuning results, b) to k) may be repeated until the evaluation score for the first threshold number of consecutive times does not improve compared with the previous repetition, or reaches a preset second threshold number. For example, the first threshold number may be 50 times, and the second threshold number may be 500 times. Those skilled in the art may set different iteration termination conditions as needed.

[0149] Those skilled in the art will appreciate that various modifications may be made to the specific process of the exemplary ISP optimization method within the scope of the present disclosure.

[0150] In some embodiments, the unsupervised task model may adopt a contrast-based unsupervised task model, such as SimCLR.

[0151] SimCLR can perform two different data augmentations (random rotation, random cutting, random grayscale, random color perturbation, etc.) on each image in a batch of multiple images. Then, SimCLR is used to extract features from the augmented images, and the feature vectors corresponding to the two augmentations of the same image are required to have a high similarity, while the feature vectors of different images have a low similarity.

[0152] According to the embodiments of the present disclosure, each of a batch of multiple (e.g., 256) ISP-processed images can be subjected to random rotation, random cutting, and random grayscale data amplification (considering that random color perturbation will affect the color regeneration of ISP, random color perturbation amplification is not used). After amplifying the 256 images, SimCLR is used for feature extraction, and the contrast loss is calculated as the unsupervised loss score.

[0153] SimCLR can make the features extracted from a picture after different amplifications have a high degree of similarity, which may make the edges and textures of the picture after ISP processing particularly prominent (so they are not easily affected by random amplification), while other areas may suffer a large loss of content. In some embodiments, better results can be achieved by combining consistency loss and dynamic loss weighted average. Fig. 11B The comparison of the effects of using SimCLR and using the weighted average of SimCLR, consistency loss and dynamic loss is shown.

[0154] In some embodiments, one or more steps may be omitted, for example, one or more of consistency loss evaluation, multi-objective learning strategy, and multi-granularity learning strategy may be omitted. The number of repetitions from step b) to h) or i) may be set as needed.

[0155] Fig.12 The effect of unsupervised tuning according to an embodiment of the present disclosure is shown.

[0156] When the tuning dataset is small, the evaluation index changes more sparsely, so the existing methods directly optimize the evaluation index based on the downstream visual task, which requires the dataset to have good manual labels. However, since the MAE score changes directly according to the image quality, it does not need to be based on the evaluation index obtained by the label, and can also provide image quality information to guide ISP tuning. Therefore, under the training of different types of images, it can achieve more improvement than the existing methods.

[0157] Fig.12In the example, the computer vision task is object detection. MAE is used for unsupervised tuning, and the model used for supervised tuning is Yolo-V8. The evaluation score is mAP@0.5 value (hereinafter referred to as mAP). The mAP value is calculated as follows: 1. For a certain category, first set the detection confidence threshold, and eliminate model predictions below the threshold; 2. Calculate the intersection area and union area of ​​the remaining model prediction detection box and the manually labeled detection box respectively. If the intersection area is greater than 0.5 times the union area, it is considered a correct detection, otherwise it is an error; 3. Based on the number of correct and incorrect detections in 2, calculate the corresponding precision and recall values; 4. By adjusting the confidence threshold in 1, you can get a curve of the change in precision value with respect to the recall value; calculate the area under the curve as the AP value of the category; average the AP values ​​of all categories to get the mAP value. As shown Fig.12 As shown in Figure 3, when the number of unlabeled data exceeds 300, the unsupervised tuning method can outperform the supervised tuning.

[0158] Fig.13 The average performance of unsupervised tuning according to an embodiment of the present disclosure is shown.

[0159] The existing supervised ISP tuning method uses different models for different tasks, so it is difficult to fully share information in different data sets. The unsupervised tuning method of the embodiment of the present disclosure uniformly uses an unsupervised visual basic model for tuning, so better results can be achieved in multi-tasking.

[0160] Fig.13 In computer vision, the tasks are object detection and semantic segmentation. Fig.14 The object detection involved is the same. MAE is used for unsupervised tuning, and the model used for supervised ISP tuning is BiseNetV2. The semantic segmentation evaluation score is mIoU (hereinafter referred to as mIoU). The calculation method of mIoU is: 1. At each pixel point, first classify it into different categories according to the model prediction results; 2. Compare the model prediction results with the manual annotation results, and calculate the intersection and union; 3. According to the values ​​in 2, calculate the intersection and union ratio of each category; 4. Finally, take the average of all category IoU as the final mIoU value.

[0161] Comparison of the effects of supervised and unsupervised tuning when using 256 images for object detection and semantic segmentation at the same time. Dark heights represent the effect of object detection tuning, and light heights represent the effect of semantic segmentation tuning. Fig.13 As shown in the figure, compared with existing methods, the unsupervised method has achieved improved accuracy in both object detection and semantic segmentation tasks.

[0162] The ISP tuning solution according to the embodiment of the present disclosure is particularly suitable for various multi-tasking scenarios, such as autonomous driving, unmanned retail, etc.

[0163] The application of autonomous driving requires the support of a series of computer vision tasks, including lane detection, signal light recognition, sign recognition, vehicle and pedestrian detection, etc. Lane detection requires segmenting each lane based on the image. This task requires accurate identification of the lane lines on the road surface. It does not require the ISP to have good color accuracy, but requires clear edge features to help curve recognition. Signal light recognition requires the ability to identify the position and color of the signal light, which mainly requires the ISP to accurately correct the color and prevent overexposure. Sign recognition requires identifying lane signs on the road or road signs on the roadside, and requires the ISP to stably obtain the color of the sign for positioning and classification. Vehicle and pedestrian detection is the most complex, requiring strong overall performance of the ISP.

[0164] Unmanned retail applications mainly require face detection, face verification, and product detection functions. Among them, face detection requires the ISP to accurately reflect the skin color and shape of the face; face verification requires the ISP to clearly display the facial feature points. Product detection requires the ISP to accurately reflect the texture features of the packaging bag, such as color, pattern, and text.

[0165] According to the embodiment of the present disclosure, the ISP tuning solution can achieve better results across a variety of different downstream tasks.

[0166] The tuning framework and tuning method according to the present disclosure may be implemented in an optimization device. Fig.14 1 is a block diagram showing a tuning device 1000 for ISP according to an embodiment of the present disclosure. The tuning device 1000 may include a processing circuit 1003, which may be configured to implement various modules and interactions between modules involved in the tuning method as described above.

[0167] In some embodiments, the optimization device of the present disclosure can be integrated into any device including an image signal processor (ISP), such as a photographic device or other image acquisition / processing device, for example, in the form of an integrated circuit, a processor, or even in the existing processing circuit of the device; or it can also be detachably connected to the device as a separate device, for example, as a separate module, or together with other components that can be detachably connected to the device. In some embodiments, it can even be set on a remote device with which the device can communicate.

[0168] In some embodiments, the solution of the present disclosure can be implemented by a software algorithm, so that it can be easily integrated into various types of devices including an image signal processor (ISP), such as a camera, a still camera such as a SLR camera, a micro single-lens camera, etc., as well as a portable photographic device, and other image acquisition / processing devices. In particular, the method of the present disclosure can be executed as a computer program, an instruction, etc. by a processor of a photographic device to perform tuning for the ISP.

[0169] The optimization device according to the embodiment of the present disclosure can be incorporated into an electronic device having an image signal processor to optimize the configuration parameter set of the image signal processor. The electronic device can be, for example, a camera, a mobile phone, and the like. Fig.15 is a block diagram showing an electronic device 1100 according to an embodiment of the present disclosure, wherein the electronic device 1100 includes an image signal processor 1102 and an optimization device 1104 for optimizing a configuration parameter set of the image signal processor 1102 .

[0170] In addition, it should be understood that the above series of processes and devices can also be implemented by software and / or firmware. In the case of being implemented by software and / or firmware, from a storage medium or a network to a computer with a dedicated hardware structure, such as Fig.16 The general-purpose personal computer 1100 shown installs the programs constituting the software, and when the various programs are installed, the computer can execute various functions and the like. Fig.16 1 is a block diagram showing an example structure of a personal computer of an optimization device that can be adopted in an embodiment of the present disclosure. In one example, the personal computer can correspond to the above-mentioned exemplary optimization device according to the present disclosure.

[0171] exist Fig.16 In the embodiment, a central processing unit (CPU) 1601 executes various processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage section 1608 to a random access memory (RAM) 1603. In the RAM 1603, data required when the CPU 1601 executes various processes and the like is also stored as needed.

[0172] The CPU 1601, the ROM 1602, and the RAM 1603 are connected to one another via a bus 1604. To the bus 1604, an input / output interface 1605 is also connected.

[0173] The following components are connected to the input / output interface 1605: an input section 1606 including a keyboard, a mouse, etc.; an output section 1607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN card, a modem, etc. The communication section 1609 performs communication processing via a network such as the Internet.

[0174] A drive 1610 is also connected to the input / output interface 1605 as needed. A removable medium 1611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1610 as needed so that a computer program read therefrom is installed into the storage section 1608 as needed.

[0175] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 1611 .

[0176] Those skilled in the art should understand that such storage media is not limited to Fig.16 The removable medium 1611 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1611 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be the ROM 1602, a hard disk included in the storage portion 1608, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0177] It should be noted that the methods and devices described herein may be implemented as software, firmware, hardware, or any combination thereof. Some components may, for example, be implemented as software running on a digital signal processor or microprocessor. Other components may, for example, be implemented as hardware and / or an application specific integrated circuit.

[0178] In addition, the method and system of the present invention can be implemented in a variety of ways. For example, the method and system of the present invention can be implemented by software, hardware, firmware or any combination thereof. The order of the steps of the method described above is only illustrative, and unless otherwise specifically stated, the steps of the method of the present invention are not limited to the order specifically described above. In addition, in some embodiments, the present invention can also be embodied as a program recorded in a recording medium, including machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also covers the recording medium storing the program for implementing the method according to the present invention. Such storage media may include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, and the like.

[0179] Those skilled in the art will appreciate that the boundaries between the above operations are merely illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed in additional operations, and operations can be performed at least partially overlapping in time. Moreover, alternative embodiments may include multiple instances of specific operations, and the order of operations may be changed in other various embodiments. However, other modifications, variations, and replacements are equally possible. Therefore, this specification and accompanying drawings should be considered illustrative, not restrictive.

[0180] In addition, embodiments of the present disclosure may further include the following illustrative examples.

[0181] 1. An optimization device for an image signal processor (ISP), the optimization device comprising a processing circuit configured to:

[0182] Using an image signal processor simulator to process an original sample image for image signal processor optimization based on an ISP configuration parameter set to obtain a processed sample image;

[0183] Obtaining an unsupervised evaluation score based on the processed sample image using an unsupervised pre-trained task model that implements a specific task, wherein the unsupervised evaluation score evaluates the effect of using the unsupervised task model to perform the specific task on the processed sample image;

[0184] The set of ISP configuration parameters is updated based at least on the unsupervised evaluation score.

[0185] 2. The optimization device of claim 1, wherein the processing circuit is further configured to:

[0186] calculating an overall evaluation score based at least on the unsupervised evaluation score; and

[0187] The ISP configuration parameter set is updated based on the total evaluation score.

[0188] 3. The optimization device according to claim 2, wherein the processing circuit is further configured to:

[0189] Calculating a global consistency loss based on the original sample image and the processed sample image; and

[0190] Based on the unsupervised evaluation score and the global consistency loss, a multi-objective learning strategy is used to calculate an overall evaluation score.

[0191] 4. An optimization device as described in item 3, wherein the global consistency loss is calculated based on the similarity of pixel value distribution between the original sample image and the processed sample image or the similarity of size relationship between a pixel and surrounding pixels.

[0192] 5. The optimization device according to any one of items 2 to 4, wherein the processing circuit is further configured to:

[0193] The update of the ISP configuration parameter set is performed iteratively.

[0194] 6. The optimization device according to any one of items 2 to 4, wherein the processing circuit is further configured to:

[0195] Using an optimizer to generate the ISP configuration parameter set based on the current internal state;

[0196] Updating an internal state of the optimizer based on the total evaluation score corresponding to the ISP configuration parameter set; and

[0197] Based on the updated internal state, an optimizer is used to generate an updated set of ISP configuration parameters.

[0198] 7. The optimization device of item 6, wherein the processing circuit is further configured to:

[0199] Obtaining a plurality of total evaluation scores, each total evaluation score corresponding to an ISP configuration parameter set generated by the optimizer for a simulator of an image signal processor based on a current internal state, and associated with a processed sample image obtained by processing the same original sample image based on the ISP parameter set;

[0200] updating an internal state of the optimizer based on the plurality of total evaluation scores and the corresponding plurality of ISP configuration parameter sets; and

[0201] Based on the updated internal state, an updated ISP configuration parameter set is obtained using an optimizer, wherein the updated ISP configuration parameter set can make the total evaluation score corresponding to the updated ISP configuration parameter set close to the better total evaluation score among the multiple total evaluation scores and away from the worse total evaluation score among the multiple total evaluation scores.

[0202] 8. The optimization device according to any one of items 1 to 4, wherein the processing circuit is further configured to:

[0203] Determining whether the unsupervised evaluation score is less than a threshold;

[0204] In response to determining that the unsupervised evaluation score is less than the threshold, reloading the original sample image used in the previous ISP optimization iteration, and recalculating the unsupervised evaluation score,

[0205] The recomputed unsupervised evaluation score is scaled down and the scaled unsupervised evaluation score is used as the new unsupervised evaluation score.

[0206] 9. The optimization device according to any one of clauses 3 to 4, wherein the processing circuit is further configured to:

[0207] Based on the processed sample image, generate multiple images with varying granularity from high to low;

[0208] For each of the multiple images, respectively calculate an unsupervised evaluation score and a global consistency loss of the image;

[0209] Obtaining a multi-granularity unsupervised evaluation score corresponding to the processed sample image by calculating a weighted sum of the corresponding multiple unsupervised evaluation scores of the multiple images;

[0210] Obtaining a multi-granularity global consistency loss corresponding to the processed sample image by calculating a weighted sum of the corresponding multiple global consistency losses of the multiple images; and

[0211] Based on the multi-granularity unsupervised evaluation scores and the multi-granularity global consistency loss, a multi-objective learning strategy is used to calculate the total evaluation score corresponding to the processed sample image;

[0212] The ISP configuration parameter set is updated based on the total evaluation score corresponding to the processed sample image.

[0213] 10. An optimization device as described in item 1, wherein the unsupervised pre-training task model includes at least one of a generation-based unsupervised pre-training task model and a comparison-based unsupervised pre-training task model.

[0214] 11. An optimization device as described in item 1, wherein the unsupervised pre-training task model includes at least one of the following: MAE, BeiT, BeiTv2 or SimCLR.

[0215] 12. An optimization method for an image signal processor (ISP), comprising:

[0216] Using an image signal processor simulator to process an original sample image for image signal processor optimization based on an ISP configuration parameter set to obtain a processed sample image;

[0217] Obtaining an unsupervised evaluation score based on the processed sample image using an unsupervised pre-trained task model that implements a specific task, wherein the unsupervised evaluation score evaluates the effect of using the unsupervised task model to perform the specific task on the processed sample image;

[0218] The set of ISP configuration parameters is updated based at least on the unsupervised evaluation score.

[0219] 13. The optimization method according to item 12, further comprising:

[0220] calculating an overall evaluation score based at least on the unsupervised evaluation score; and

[0221] The ISP configuration parameter set is updated based on the total evaluation score.

[0222] 14. The optimization method according to item 13, further comprising:

[0223] Calculating a global consistency loss based on the original sample image and the processed sample image; and

[0224] Based on the unsupervised evaluation score and the global consistency loss, a multi-objective learning strategy is used to calculate an overall evaluation score.

[0225] 15. An optimization method as described in item 14, wherein the global consistency loss is calculated based on the similarity of pixel value distribution between the original sample image and the processed sample image or the similarity of size relationship between a pixel and surrounding pixels.

[0226] 16. The optimization method according to any one of items 13 to 15, further comprising:

[0227] The update of the ISP configuration parameter set is performed iteratively.

[0228] 17. The optimization method according to any one of items 13 to 15, further comprising:

[0229] Using an optimizer to generate the ISP configuration parameter set based on the current internal state;

[0230] Updating an internal state of the optimizer based on the total evaluation score corresponding to the ISP configuration parameter set; and

[0231] Based on the updated internal state, an optimizer is used to generate an updated set of ISP configuration parameters.

[0232] 18. The optimization method according to item 17, further comprising:

[0233] Obtaining a plurality of total evaluation scores, each total evaluation score corresponding to an ISP configuration parameter set generated by the optimizer for a simulator of an image signal processor based on a current internal state, and associated with a processed sample image obtained by processing the same original sample image based on the ISP parameter set;

[0234] updating an internal state of the optimizer based on the plurality of total evaluation scores and the corresponding plurality of ISP configuration parameter sets; and

[0235] Based on the updated internal state, an updated ISP configuration parameter set is obtained using an optimizer, wherein the updated ISP configuration parameter set can make the total evaluation score corresponding to the updated ISP configuration parameter set close to the better total evaluation score among the multiple total evaluation scores and away from the worse total evaluation score among the multiple total evaluation scores.

[0236] 19. The optimization method according to any one of items 12 to 15, further comprising:

[0237] Determining whether the unsupervised evaluation score is less than a threshold;

[0238] In response to determining that the unsupervised evaluation score is less than the threshold, reloading the original sample image used in the previous ISP optimization iteration, and recalculating the unsupervised evaluation score,

[0239] The recomputed unsupervised evaluation score is scaled down and the scaled unsupervised evaluation score is used as the new unsupervised evaluation score.

[0240] 20. The optimization method according to any one of items 13 to 15, further comprising:

[0241] Based on the processed sample image, generate multiple images with varying granularity from high to low;

[0242] For each of the multiple images, respectively calculate an unsupervised evaluation score and a global consistency loss of the image;

[0243] Obtaining a multi-granularity unsupervised evaluation score corresponding to the processed sample image by calculating a weighted sum of the corresponding multiple unsupervised evaluation scores of the multiple images;

[0244] Obtaining a multi-granularity global consistency loss corresponding to the processed sample image by calculating a weighted sum of the corresponding multiple global consistency losses of the multiple images; and

[0245] Based on the multi-granularity unsupervised evaluation scores and the multi-granularity global consistency loss, a multi-objective learning strategy is used to calculate the total evaluation score corresponding to the processed sample image;

[0246] The ISP configuration parameter set is updated based on the total evaluation score corresponding to the processed sample image.

[0247] 21. The optimization method of item 12, wherein the unsupervised pre-training task model comprises at least one of a generation-based unsupervised pre-training task model and a comparison-based unsupervised pre-training task model.

[0248] 22. The optimization method of item 12, wherein the unsupervised pre-training task model comprises at least one of the following: MAE, BeiT, BeiTv2 or SimCLR.

[0249] 23. A device comprising

[0250] at least one processor; and

[0251] At least one storage device storing instructions thereon, which, when executed by the at least one processor, causes the at least one processor to perform the optimization method according to any one of items 12-22.

[0252] 24. A storage medium storing instructions, which, when executed by a processor, enable execution of the optimization method according to any one of items 12-22.

[0253] 25. An electronic device, comprising:

[0254] an image signal processor; and

[0255] An optimization device as described in any of items 1-11, used to optimize the configuration parameter set of the image signal processor.

[0256] Although the present disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions and transformations can be made without departing from the spirit and scope of the present disclosure as defined by the appended claims. Moreover, the terms "including", "comprising" or any other variants of the embodiments of the present disclosure are intended to cover non-exclusive inclusions, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "including one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0257] Although some specific embodiments of the present disclosure have been described in detail, it should be understood by those skilled in the art that the above embodiments are merely illustrative and do not limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be combined, modified or replaced without departing from the scope and essence of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An optimization device for an image signal processor (ISP), the optimization device comprising a processing circuit configured to: Using an image signal processor simulator to process an original sample image for image signal processor optimization based on an ISP configuration parameter set to obtain a processed sample image; Obtaining an unsupervised evaluation score based on the processed sample image using an unsupervised pre-trained task model that implements a specific task, wherein the unsupervised evaluation score evaluates the effect of using the unsupervised task model to perform the specific task on the processed sample image; The set of ISP configuration parameters is updated based at least on the unsupervised evaluation score.

2. The optimization device of claim 1, wherein the processing circuit is further configured to: calculating an overall evaluation score based at least on the unsupervised evaluation score; and The ISP configuration parameter set is updated based on the total evaluation score.

3. The optimization device of claim 2, wherein the processing circuit is further configured to: Calculating a global consistency loss based on the original sample image and the processed sample image; and Based on the unsupervised evaluation score and the global consistency loss, a multi-objective learning strategy is used to calculate an overall evaluation score.

4. The optimization device as described in claim 3, wherein the global consistency loss is calculated based on the similarity of pixel value distribution between the original sample image and the processed sample image or the similarity of size relationship between a pixel and surrounding pixels.

5. The optimization device according to any one of claims 2 to 4, wherein the processing circuit is further configured to: The update of the ISP configuration parameter set is performed iteratively.

6. The optimization device according to any one of claims 2 to 4, wherein the processing circuit is further configured to: Using an optimizer to generate the ISP configuration parameter set based on the current internal state; Updating an internal state of the optimizer based on the total evaluation score corresponding to the ISP configuration parameter set; and Based on the updated internal state, an optimizer is used to generate an updated set of ISP configuration parameters.

7. The optimization device of claim 6, wherein the processing circuit is further configured to: Obtaining a plurality of total evaluation scores, each total evaluation score corresponding to an ISP configuration parameter set generated by the optimizer for a simulator of an image signal processor based on a current internal state, and associated with a processed sample image obtained by processing the same original sample image based on the ISP parameter set; updating an internal state of the optimizer based on the plurality of total evaluation scores and the corresponding plurality of ISP configuration parameter sets; as well as Based on the updated internal state, an updated ISP configuration parameter set is obtained using an optimizer, wherein the updated ISP configuration parameter set can make the total evaluation score corresponding to the updated ISP configuration parameter set close to the better total evaluation score among the multiple total evaluation scores and away from the worse total evaluation score among the multiple total evaluation scores.

8. The optimization device according to any one of claims 1 to 4, wherein the processing circuit is further configured to: Determining whether the unsupervised evaluation score is less than a threshold; In response to determining that the unsupervised evaluation score is less than the threshold, reloading the original sample image used in the previous ISP optimization iteration, and recalculating the unsupervised evaluation score, The recomputed unsupervised evaluation score is scaled down and the scaled unsupervised evaluation score is used as the new unsupervised evaluation score.

9. The optimization device according to any one of claims 3 to 4, wherein the processing circuit is further configured to: Based on the processed sample image, generate multiple images with varying granularity from high to low; For each of the multiple images, respectively calculate an unsupervised evaluation score and a global consistency loss of the image; Obtaining a multi-granularity unsupervised evaluation score corresponding to the processed sample image by calculating a weighted sum of the corresponding multiple unsupervised evaluation scores of the multiple images; Obtaining a multi-granularity global consistency loss corresponding to the processed sample image by calculating a weighted sum of multiple global consistency losses corresponding to the multiple images; as well as Based on the multi-granularity unsupervised evaluation scores and the multi-granularity global consistency loss, a multi-objective learning strategy is used to calculate the total evaluation score corresponding to the processed sample image; The ISP configuration parameter set is updated based on the total evaluation score corresponding to the processed sample image.

10. The optimization device according to claim 1, wherein: The unsupervised pre-training task model includes at least one of a generation-based unsupervised pre-training task model and a comparison-based unsupervised pre-training task model.

Images