ISP parameter setting method and device, medium and program product

Abstract

The invention relates to an ISP parameter setting method and device, a medium and a program product. The method comprises the following steps: acquiring original image data and scene metadata, wherein the scene metadata is used for representing a target shooting scene when the original image data is shot; inputting the original image data and the scene metadata into a trained ISP parameter setting model, so as to generate a target value of each ISP parameter matched with the target shooting scene by the trained ISP parameter setting model; and respectively updating the current numerical value of each ISP parameter in the ISP chip into a corresponding target numerical value, so that the ISP chip processes the original image data based on the updated numerical value to obtain a target image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method, device, medium, and program product for setting ISP parameters. Background Technology

[0002] An Image Signal Processor (ISP) is a key component of modern digital imaging systems, primarily used to convert raw image data output from image sensors (such as CMOS or CCD sensors) into RGB images. ISP chip design is complex, typically involving a series of functional modules. Taking a mobile phone as an example, an ISP chip includes modules for image noise reduction, automatic white balance, autofocus, color correction, and high dynamic range, each involving different algorithms and hardware. Therefore, debugging ISP parameters is a complex and tedious task, requiring significant manpower. Summary of the Invention

[0003] This disclosure provides a method, device, medium, and program product for setting ISP parameters to address the shortcomings of related technologies.

[0004] According to a first aspect of the present disclosure, a method for setting ISP parameters is provided, comprising:

[0005] Acquire raw image data and scene metadata, wherein the scene metadata is used to characterize the target shooting scene when the raw image data was captured;

[0006] The original image data and the scene metadata are input into a trained ISP parameter setting model, so that the trained ISP parameter setting model generates target values ​​for each ISP parameter that match the target shooting scene;

[0007] The current values ​​of each ISP parameter in the ISP chip are updated to the corresponding target values, so that the ISP chip processes the original image data based on the updated values ​​to obtain the target image.

[0008] According to a second aspect of the present disclosure, a training method for an ISP parameter setting model is proposed, comprising:

[0009] Acquire sample original image data, sample scene metadata, and sample target image corresponding to the sample original image data. The sample scene metadata is used to characterize the sample shooting scene when the sample original image data was captured.

[0010] The original image data of the sample and the metadata of the sample scene are input into the ISP parameter setting model so that the ISP parameter setting model can generate sample prediction values ​​of each ISP parameter that match the shooting scene of the sample.

[0011] The original image data of the sample is processed using the sample prediction values ​​of each ISP parameter to obtain the sample prediction image corresponding to the shooting scene of the sample.

[0012] The loss value is calculated based on the predicted image of the sample and the target image of the sample, and the ISP parameter setting model is iteratively trained based on the loss value.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0014] processor;

[0015] Memory used to store processor-executable instructions;

[0016] The processor is configured to implement the method described in the embodiments of the first or second aspect above.

[0017] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the embodiments of the first or second aspect above.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the methods described in the embodiments of the first or second aspect above.

[0019] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0020] As described in the above embodiments, this disclosure inputs the original image data and scene metadata representing the target shooting scene in which the original image data is located into a trained ISP parameter setting model, and updates the values ​​of each ISP parameter in the ISP chip using the target values ​​generated by the trained ISP parameter setting model. Then, the updated values ​​are used to process the original image data to obtain the target image. Since the trained ISP parameter setting model can generate target values ​​for each ISP parameter that match the target shooting scene based on the input original image data and scene metadata, processing the original image data captured in the target shooting scene using the target values ​​of each ISP parameter can significantly improve the imaging quality of the target image. The above method realizes automatic optimization of ISP parameters in different shooting scenes, which not only effectively improves the optimization efficiency of ISP, but also obtains the optimal ISP parameter values ​​suitable for each shooting scene.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating an ISP parameter setting system according to an embodiment of the present disclosure.

[0024] Figure 2 This is a schematic flowchart illustrating a method for setting ISP parameters according to an embodiment of the present disclosure.

[0025] Figure 3 This is a schematic diagram illustrating an application scenario of an ISP parameter setting method according to an embodiment of the present disclosure.

[0026] Figure 4 This is a schematic diagram illustrating an application scenario of another method for setting ISP parameters according to an embodiment of the present disclosure.

[0027] Figure 5 This is a flowchart illustrating a training method for an ISP parameter setting model according to an embodiment of the present disclosure.

[0028] Figure 6 This is a flowchart illustrating the training process of an ISP parameter setting model according to an embodiment of the present disclosure.

[0029] Figure 7 This is a schematic block diagram illustrating an ISP parameter setting apparatus according to an embodiment of the present disclosure.

[0030] Figure 8 This is a schematic block diagram of a training apparatus for an ISP parameter setting model according to an embodiment of the present disclosure.

[0031] Figure 9 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0032] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0033] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. The singular forms “a” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of embodiments of this disclosure, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0035] For the sake of brevity and ease of understanding, this document uses the terms "greater than" or "less than", "higher than" or "lower than" to describe size relationships. However, it will be understood by those skilled in the art that the term "greater than" also includes the meaning of "greater than or equal to", and "less than" also includes the meaning of "less than or equal to"; the term "higher than" also includes the meaning of "higher than or equal to", and "lower than" also includes the meaning of "lower than or equal to".

[0036] In related technologies, machine learning models are often used to replace some functional modules in an ISP chip, or even to replace the entire ISP chip, in order to improve image quality. However, this approach, integrating the machine learning model with the ISP chip, not only consumes significant computing power but also increases the difficulty of integrating the various functional modules of the ISP chip. Secondly, the machine learning model is essentially a black box; its operation is unpredictable, making it difficult to pinpoint the cause of any anomalies during image capture. Finally, at the hardware level, the machine learning model relies on a dedicated NPU (Neural Processing Unit) computing module, and integrating and scheduling the NPU and ISP hardware modules is quite challenging.

[0037] In view of this, this disclosure proposes a scheme to adjust the values ​​of various ISP parameters in the ISP chip using a machine learning model. This scheme can improve the shooting effect in various shooting scenarios without integrating the machine learning model with the ISP chip.

[0038] The following describes one or more embodiments of this disclosure in detail.

[0039] The embodiments of this disclosure can be applied to electronic devices that deploy an ISP chip and a trained ISP parameter setting model. These electronic devices may include, but are not limited to, smartphones, desktop computers, tablets, laptops, e-book readers, smartwatches, and smart bracelets. One or more embodiments of this disclosure are not limited to these. During operation, the electronic device can run the ISP parameter setting system to achieve the purpose of setting ISP parameters. The application program for the ISP parameter setting system can be pre-installed on the electronic device, allowing the ISP parameter setting system to be started and run on the device. Of course, when using technologies such as HTML5, it is not necessary to install the corresponding application on the electronic device to obtain and run the ISP parameter setting system.

[0040] In another embodiment, the ISP parameter setting system may include, for example: Figure 1 The diagram shows a server 11, a network 12, and an electronic device 13. During operation, the server 11 can run the server-side program of the ISP parameter setting system to perform related data processing functions. Meanwhile, the electronic device 13 can run the client-side program of the ISP parameter setting system to perform related display and human-computer interaction functions. Thus, the server 11 and the electronic device 13 work together to implement the ISP parameter setting scheme. It should be noted that the server 11 can also communicate with multiple electronic devices to set ISP parameters on multiple devices.

[0041] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted by a host cluster. Electronic device 13 can be any type of device, such as a PC, tablet, laptop, PDA (Personal Digital Assistants), mobile phone, wearable device (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments disclosed herein do not limit this. The network 12 for interaction between electronic device 13 and server 11 can include various types of wired or wireless networks. In one embodiment, network 12 can include the Public Switched Telephone Network (PSTN) and the Internet. The client application of the ISP parameter setting system can be pre-installed on the electronic device, allowing the client to be launched and run on the electronic device; of course, when using an online "client" such as HTML5 technology, it is not necessary to install the corresponding application on the electronic device to obtain and run the client.

[0042] exist Figure 1 In the ISP parameter setting system shown, an ISP chip is deployed on electronic device 13, and a trained ISP parameter setting model is deployed on server 11. The ISP parameter setting process is as follows: Electronic device 13 acquires raw image data and scene metadata, and submits the raw image data and scene metadata to server 11. Server 11 inputs the raw image data and scene metadata into the trained ISP parameter setting model, and returns the target values ​​of each ISP parameter output by the trained ISP parameter setting model to electronic device 13, so that electronic device 13 can use the received target values ​​to process the raw image data and obtain the target image.

[0043] In yet another embodiment, embodiments of this disclosure can be applied to a server, on which an ISP chip and a trained ISP parameter setting model are deployed. Figure 1 Taking server 11 as an example, server 11 can receive raw image data and scene metadata submitted by any electronic device, and use a trained ISP parameter setting model to generate target values ​​for each ISP parameter corresponding to the raw image data and scene metadata. Then, server 11 processes the raw image data using the target values ​​of each ISP parameter, and returns the processed target image to the electronic device that submitted the raw image data and scene metadata.

[0044] Figure 2 This is a schematic flowchart illustrating a method for setting ISP parameters according to an embodiment of this disclosure. Figure 2As shown, the process may include the following steps:

[0045] S201: Obtain raw image data and scene metadata, wherein the scene metadata is used to characterize the target shooting scene when the raw image data was captured.

[0046] Typically, raw image data and scene metadata can be acquired using various sensors pre-deployed in a photographic device. In this embodiment, the photographic device refers to an electronic device with photo / video capture capabilities, including but not limited to smartphones, smartwatches, and smart glasses. A variety of sensors can be deployed in the photographic device, such as image sensors, illumination sensors, and color temperature sensors. Among these, the image sensor can acquire raw image data (such as RAW images). In this embodiment, all sensors other than the image sensor are referred to as "other sensors," and the data acquired by these other sensors is referred to as "scene metadata." Therefore, other sensors may include illumination sensors, color temperature sensors, distance sensors, and IMUs (Inertial Measurement Units). Illumination sensors measure the ambient light intensity of the shooting scene, used to adjust exposure and other shooting parameters. Color temperature sensors measure the color temperature of the ambient light in the shooting scene, used for automatic white balance adjustment. Distance sensors measure the distance between the target object and the camera, used for depth sensing, autofocus, etc. IMUs measure the attitude, angular velocity, and acceleration of the photographic device, which can be used to stabilize images and prevent camera shake. Therefore, scene metadata can include: lighting data, color temperature data, distance data, camera pose data, and so on. Typically, scene metadata differs for different shooting scenarios, so scene metadata can be used to characterize the target shooting scene in which the raw image data was captured.

[0047] In one embodiment, it is understood that when a user activates the camera device but does not initiate a shooting request, the camera device acquires raw image data continuously output by the image sensor. That is, S201 acquires a large amount of raw image data. Therefore, for the acquired raw image data and scene metadata, it is necessary to perform timestamp matching between the two to filter out scene metadata that matches the timestamp of the raw image data. Taking a frame of raw image data as an example, scene metadata matching the timestamp of that frame of raw image data (hereinafter referred to as the "target timestamp") refers to: scene metadata with the same timestamp as the target timestamp, or scene metadata with a timestamp that is a preset time interval from the target timestamp. Then, the filtered scene metadata is associated and cached with the raw image data so that when the user initiates a shooting request, the ISP parameter values ​​applicable to the target shooting scene can be determined based on the raw image data and the filtered scene metadata.

[0048] Because the timestamps of the filtered scene metadata match the timestamps of the original image data, the filtered scene metadata can more accurately reflect the target shooting scene when the original image data was acquired. This helps to determine more accurate ISP parameter values ​​that are more suitable for the target shooting scene based on the filtered scene metadata and the original image data, so as to obtain a target image with higher imaging quality using the accurate ISP parameter values.

[0049] S202: Input the original image data and the scene metadata into the trained ISP parameter setting model, so that the trained ISP parameter setting model generates target values ​​for each ISP parameter that match the target shooting scene.

[0050] A deep learning model or a large language model can be selected as the ISP parameter setting model. The selected ISP parameter setting model is then trained to ensure excellent performance in the vertical task of generating ISP parameters. The training method for the ISP parameter setting model will be detailed later in this disclosure and will not be described here. The trained ISP parameter setting model can generate target values ​​for each ISP parameter that match the target shooting scene based on the input raw image data and scene metadata.

[0051] In one embodiment, in conjunction with the aforementioned timestamp matching embodiment, the filtered scene metadata can be associated with its matching original image data and input into a trained ISP parameter setting model. The trained ISP parameter setting model can capture additional information based on the association between the original image data and the scene metadata, thereby generating more accurate target values ​​for each ISP parameter based on the captured information.

[0052] In one embodiment, the ISP parameter values ​​adapted to different shooting scenes differ, while the ISP parameter values ​​adapted to the same shooting scene are usually consistent. Therefore, after obtaining the target values ​​of each ISP parameter that match a certain shooting scene, the target values ​​can be associated with and stored with the shooting scene. During the next shooting, it can be first checked whether the current shooting scene is consistent with the stored shooting scene. If the detection result shows that the current shooting scene is consistent with the stored shooting scene, the ISP parameter values ​​corresponding to the stored shooting scene can be directly used for image processing. If the detection result shows that the current shooting scene is inconsistent with the stored shooting scene, i.e., a change in the shooting scene is detected, the original image data and scene metadata obtained under the current shooting scene need to be input into the trained ISP parameter setting model so that the trained ISP parameter setting model can generate the target values ​​of each ISP parameter that match the current shooting scene.

[0053] The above method only uses the trained ISP parameter setting model to generate target values ​​for each ISP parameter when a change in the shooting scene is detected. This helps to improve the efficiency of obtaining ISP parameter values, thereby improving the efficiency of the ISP chip in processing images, while reducing the overhead of the system in generating ISP parameter values.

[0054] S203: Update the current values ​​of each ISP parameter in the ISP chip to the corresponding target values, so that the ISP chip processes the original image data based on the updated values ​​to obtain the target image.

[0055] Combination Figure 1 In various embodiments, depending on the application of these embodiments, the ISP chip can be deployed on electronic devices or servers. Taking deployment on a mobile phone as an example, there are multiple deployment methods for the ISP chip. For instance, the ISP chip can be integrated with the phone's computing chip, or it can be a standalone external component on the phone. This disclosure does not limit the specific deployment method of the ISP chip.

[0056] After obtaining the target values ​​for each ISP parameter, the current values ​​of each ISP parameter in the ISP chip can be updated to the corresponding target values, allowing the ISP chip to process the original image data based on the updated values. This processing can include image noise reduction, automatic white balance, autofocus, color correction, high dynamic range adjustment, and other operations. After processing, the target image corresponding to the original image data can be obtained.

[0057] In the above embodiments, since scene metadata can characterize the target shooting scene in which the original image data was captured, and the trained ISP parameter setting model can generate target values ​​for each ISP parameter that match the target shooting scene based on the input original image data and scene metadata, processing the original image data captured in the target shooting scene using the target values ​​of each ISP parameter can significantly improve the imaging quality of the target image. This method achieves automatic ISP parameter tuning under different shooting scenes, effectively improving ISP tuning efficiency and obtaining optimal ISP parameter values ​​suitable for various shooting scenes.

[0058] In one embodiment, evaluation feedback on the target image can be obtained. For example, after showing the target image to the user, evaluation prompts are output to the user, allowing them to evaluate the target image based on the prompts. If the evaluation feedback indicates that the target image quality is high, it means that the target values ​​of each ISP parameter are relatively accurate. If the evaluation feedback indicates that the target image quality is low, it means that the target values ​​of each ISP parameter are not accurate enough. In this case, the model parameters of the trained ISP parameter setting model can be adjusted based on the target image and the evaluation feedback to further improve the ISP parameter setting model's ability to generate ISP parameter values.

[0059] In this embodiment, the ISP parameter setting model is continuously optimized and adjusted based on the quality of the target image obtained during the actual implementation of the present disclosure scheme. This allows the optimized ISP parameter setting model to generate more accurate target values ​​for each ISP parameter, thereby improving the imaging quality.

[0060] Figure 3 This is a schematic diagram illustrating an application scenario of an ISP parameter setting method according to an embodiment of this disclosure. Figure 3 In this system, the ISP chip and a trained ISP parameter setting model are deployed on a server. Any client 31 submits the acquired raw image data and scene metadata to the server 32, which then inputs these data into the trained ISP parameter setting model. The trained ISP parameter setting model generates target values ​​for each ISP parameter that match the target shooting scene based on the input and sends these target values ​​to the ISP chip. The ISP chip updates the current values ​​of each ISP parameter to the corresponding target values ​​and processes the raw image data using the updated ISP parameter values ​​to output the target image. The server 32 returns the target image output by the ISP chip to the client 31, allowing the client 31 to preview the target image on the interface for user viewing.

[0061] Figure 4 This is a schematic diagram illustrating an application scenario of another method for setting ISP parameters according to an embodiment of this disclosure. Figure 4Taking a mobile phone as an example for photography, a mobile phone deploys a camera module, a System on a Chip (SOC) chip, an ISP chip, and a trained ISP parameter setting model. When the user activates the camera, the camera module acquires MIPI signals from the image sensor and other sensors, and sends the acquired MIPI signals to the SOC chip. The SOC chip processes the MIPI signals, parsing them into raw image data (RAW data format) and scene metadata. Then, the SOC chip processes the raw image data and scene metadata, such as performing timestamp matching between the raw image data and scene metadata, filtering out scene metadata that matches the timestamp of the raw image data. When the user issues a shooting command, the SOC chip associates the raw image data and scene metadata and sends them to the trained ISP parameter setting model. The trained ISP parameter setting model generates target values ​​for each ISP parameter that match the target shooting scene based on the input raw image data and scene metadata, and sends the generated target values ​​to the ISP chip. The ISP chip updates the current values ​​of each ISP parameter to the corresponding target values, and uses the updated ISP parameter values ​​to process the original image data, outputting the target image (RGB data format). The target image can be stored on a hard drive or other storage module.

[0062] Figure 5 This is a flowchart illustrating a training method for an ISP parameter setting model according to embodiments of the present disclosure. Figure 5 As shown, the training process may include the following steps:

[0063] S501: Obtain sample original image data, sample scene metadata, and sample target image corresponding to the sample original image data. The sample scene metadata is used to characterize the sample shooting scene when the sample original image data was taken.

[0064] In this embodiment, the original sample image data, sample scene metadata, and sample target image are all known and defined. The methods for obtaining the original sample image data and sample scene metadata can refer to the embodiment shown in S201 above, and will not be repeated here. The sample target image can be obtained by taking a picture of the same sample object in the same sample shooting scene using a professional camera. Because the images obtained by professional cameras are of high quality, the sample target images obtained by professional cameras can be used as ground truth labels in the training process. It should be noted that for professional cameras, their optical parameters need to be consistent with the optical parameters of the photographic equipment used to obtain the original sample image data and sample scene metadata, thereby ensuring that the output images of both are feature-aligned at the pixel level. If the optical parameters cannot be kept consistent, calibration tools and algorithms can be used to perform feature transformation on the images obtained by professional cameras, and the transformed image can be used as the sample target image, thereby achieving complete alignment at the pixel feature level.

[0065] S502: Input the original image data of the sample and the metadata of the sample scene into the ISP parameter setting model, so that the ISP parameter setting model generates sample prediction values ​​of each ISP parameter that match the shooting scene of the sample.

[0066] In one embodiment, the original sample image data and sample scene metadata need to be timestamped to filter out sample scene metadata that matches the timestamp of the original sample image data. Then, the original sample image data and sample scene metadata are associated and input into the ISP parameter setting model. The specific matching process can be referred to the embodiment shown in S201 above, and will not be repeated here.

[0067] Because the timestamps of the filtered sample scene metadata match the timestamps of the original sample image data, the filtered sample scene metadata can more accurately reflect the sample shooting scene when the original sample image data was collected. Therefore, determining the correlation between the original sample image data and the sample scene metadata helps to improve the training effect of the ISP parameter setting model.

[0068] In one embodiment, the original sample image data and sample scene metadata can be vector-converted to obtain a sample original image data vector corresponding to the original sample image data and a sample scene metadata vector corresponding to the sample scene metadata. The vector conversion operation may include normalization, which typically refers to converting the data to a value within a specific range, usually scaling the data to the [0,1] or [-1,1] interval. Normalization eliminates the influence of data dimensions, ensuring that the sample original image data vector and the sample scene metadata vector have the same scale. The sample original image data vector and the sample scene metadata vector are then concatenated, and the concatenated vector is input into the ISP parameter setting model. This method significantly improves the ISP parameter setting model's ability to understand and analyze sample original image data vectors and sample scene metadata vectors of the same scale, avoiding the impact of dimensional differences between the sample original image data and the sample scene metadata on model training.

[0069] S503: The original image data of the sample is processed using the sample prediction values ​​of each ISP parameter to obtain a sample prediction image corresponding to the shooting scene of the sample.

[0070] During training, the values ​​of each ISP parameter in the ISP chip can be updated to the corresponding sample prediction values, and the updated values ​​can be used to process the original sample image data to obtain the sample prediction image corresponding to the sample shooting scene.

[0071] S504: Calculate the loss value based on the sample predicted image and the sample target image, and iteratively train the ISP parameter setting model based on the loss value.

[0072] The loss function can be used to calculate the loss value between the predicted image and the target image. For example, the loss function can be L1 Loss, SSIM Loss, color loss, or a weighted combination of the above three loss functions. This disclosure does not limit this. Then, the ISP parameter setting model is backpropagated based on the result of the loss function to continuously update the model parameters. Training of the ISP parameter setting model is completed when it meets the predefined training objective or reaches the predefined number of iterations, and it can then be used to achieve, for example... Figure 2 The ISP parameter setting scheme is shown below.

[0073] The training process described above requires significant computing power. When mobile devices have sufficient computing power, this process can be completed independently using a real ISP chip deployed on the mobile device. However, considering that PCs typically have higher computing power than mobile devices, the training process can also be achieved through a combination of PC and mobile devices. For example, an ISP parameter setting model is deployed on the PC, and a real ISP chip is deployed on the mobile device. First, a communication connection is established between the PC and the mobile device. During training, the original sample image data, sample scene metadata, and sample target images are sent to the PC. The ISP parameter setting model deployed on the PC takes the original sample image data and sample scene metadata as input, generates and outputs the sample prediction values ​​for each ISP parameter. Next, the PC sends the sample prediction values ​​to the mobile device. The mobile device updates the current values ​​of each ISP parameter in the real ISP chip to the corresponding sample prediction values ​​and processes the original sample image data using the real ISP chip to obtain the sample prediction image. The mobile device then sends the sample prediction image back to the PC. The PC will calculate the loss value based on the received sample prediction image and sample target image, and update the model parameters of the ISP parameter setting model based on the loss value.

[0074] In one embodiment, in order to reduce communication interaction between the mobile device and the PC during training and improve training efficiency, a simulated ISP chip can be deployed on the PC to perform the first stage of iterative training based on the simulated ISP chip, and then the real ISP chip deployed on the mobile device can be used for the second stage of iterative training. Figure 6 This is a flowchart illustrating the training process of an ISP parameter setting model according to an embodiment of this disclosure. Figure 6As shown, the ISP parameter setting model and optimizer are deployed on the GPU of the PC, and a simulated ISP chip is deployed on the CPU of the PC. The first stage of iterative training may include: vectorizing the original sample image data and sample scene metadata, concatenating the resulting original sample image data vector with the sample scene metadata vector, and sending the concatenated vector to the ISP parameter setting model. The ISP parameter setting model generates sample prediction values ​​for each ISP parameter based on the input vector and sends the sample prediction values ​​to the simulated ISP chip. The simulated ISP chip processes the original sample image data based on the sample prediction values ​​and outputs the sample prediction image. It calculates the loss value based on the similarity between the sample prediction image and the sample target image and sends the loss value to the optimizer. The optimizer outputs optimized model parameters based on the loss value and sends the optimized model parameters to the ISP parameter setting model. The ISP parameter setting model updates its own model parameters to the optimized model parameters and makes predictions again based on the optimized model parameters. This continues until the ISP parameter setting model converges on the simulated ISP chip. At this point, the ISP parameter setting model is migrated to the mobile device, and fine-tuned based on the actual ISP chip deployed on the mobile device (i.e., the second stage of iterative training). The sample data and fine-tuning process used in the fine-tuning process are the same as in the first stage of iterative training, until the ISP parameter setting model converges again on the actual ISP chip. At this point, the training of the ISP parameter setting model is complete, and it can be used to implement, for example... Figure 2 The ISP parameter setting scheme is shown below.

[0075] The above training method can significantly reduce data interaction between mobile devices and PCs, greatly improve model training efficiency, and, based on training with real ISP chips deployed on mobile devices, can obtain ISP parameter setting models adapted to real ISP chips. This helps to improve the matching degree between real ISP chips and ISP parameter setting models, thereby improving the performance of ISP parameter setting models in actual use.

[0076] Corresponding to the aforementioned embodiments of the ISP parameter setting method, this disclosure also provides embodiments of the ISP parameter setting apparatus.

[0077] Please see Figure 7 , Figure 7 This is a schematic block diagram of an ISP parameter setting apparatus provided in an exemplary embodiment. The apparatus may include: a data acquisition unit 702, an input unit 704, and an update unit 706.

[0078] The data acquisition unit 702 is configured to acquire raw image data and scene metadata, wherein the scene metadata is used to characterize the target shooting scene when the raw image data is captured.

[0079] The input unit 704 is configured to input the original image data and the scene metadata into a trained ISP parameter setting model so that the trained ISP parameter setting model generates target values ​​for each ISP parameter that match the target shooting scene.

[0080] The update unit 706 is configured to update the current values ​​of each ISP parameter in the ISP chip to the corresponding target values, so that the ISP chip processes the original image data based on the updated values ​​to obtain the target image.

[0081] Optionally, the device further includes:

[0082] The matching unit 708 is configured to perform timestamp matching between the original image data and the scene metadata in order to filter out the scene metadata that matches the timestamp of the original image data.

[0083] The input unit 704 is specifically used to: associate the original image data with the selected scene metadata and input it into the trained ISP parameter setting model.

[0084] Optionally, the input unit 704 is specifically used to: input the original image data and scene metadata corresponding to the changed shooting scene into the trained ISP parameter setting model when a change in the shooting scene is detected.

[0085] Optionally, the device further includes:

[0086] Feedback unit 710 is configured to acquire evaluation feedback for the target image, and to adjust the model parameters of the trained ISP parameter setting model based on the evaluation feedback and the target image.

[0087] Corresponding to the aforementioned embodiments of the training method for the ISP parameter setting model, this disclosure also provides embodiments of a training apparatus for the ISP parameter setting model.

[0088] Please see Figure 8 , Figure 8 This is a schematic block diagram of a training apparatus for an ISP parameter setting model provided in an exemplary embodiment. The apparatus includes: a sample acquisition unit 802, a sample input unit 804, a sample processing unit 806, and an iterative training unit 808. Wherein:

[0089] The sample acquisition unit 802 is configured to acquire sample original image data, sample scene metadata, and sample target image corresponding to the sample original image data. The sample scene metadata is used to characterize the sample shooting scene when the sample original image data is captured.

[0090] The sample input unit 804 is configured to input the original image data of the sample and the sample scene metadata into the ISP parameter setting model so that the ISP parameter setting model can generate sample prediction values ​​of each ISP parameter that match the sample shooting scene.

[0091] The sample processing unit 806 is configured to process the original image data of the sample using the sample prediction values ​​of each ISP parameter to obtain a sample prediction image corresponding to the shooting scene of the sample.

[0092] The iterative training unit 808 is configured to calculate a loss value based on the sample predicted image and the sample target image, and to iteratively train the ISP parameter setting model based on the loss value.

[0093] Optionally, the device further includes:

[0094] The sample matching unit 810 is configured to perform timestamp matching between the original sample image data and the sample scene metadata in order to filter out sample scene metadata that matches the timestamp of the original sample image data.

[0095] The sample input unit 804 is specifically used to: associate the original image data of the sample with the selected sample scene metadata and input it into the ISP parameter setting model.

[0096] Optionally, the device further includes:

[0097] The conversion unit 812 is configured to perform vector conversion on the original sample image data and the sample scene metadata to obtain a sample original image data vector corresponding to the original sample image data and a sample scene metadata vector corresponding to the sample scene metadata; and to concatenate the original sample image data vector and the sample scene metadata vector to input the concatenated vector into the ISP parameter setting model.

[0098] Optionally, the iterative training of the ISP parameter setting model includes: a first-stage iterative training using a simulated ISP chip deployed on a PC, and / or a second-stage iterative training using a real ISP chip deployed on a mobile device.

[0099] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant methods, and will not be elaborated upon here.

[0100] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0101] Figure 9 This is a schematic block diagram illustrating an electronic device according to embodiments of the present disclosure. For example, the electronic device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0102] Reference Figure 9 The electronic device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0103] Processing component 902 typically controls the overall operation of electronic device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0104] Memory 904 is configured to store various types of data to support the operation of electronic device 900. Examples of this data include instructions for any application or method operating on electronic device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0105] Power supply component 906 provides power to various components of electronic device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 900.

[0106] Multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0107] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when electronic device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0108] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0109] Sensor assembly 914 includes one or more sensors for providing state assessments of various aspects of electronic device 900. For example, sensor assembly 914 can detect the on / off state of electronic device 900, the relative positioning of components such as the display and keypad of electronic device 900, changes in position of electronic device 900 or a component of electronic device 900, the presence or absence of user contact with electronic device 900, orientation or acceleration / deceleration of electronic device 900, and temperature changes of electronic device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0110] Communication component 916 is configured to facilitate wired or wireless communication between electronic device 900 and other devices. Electronic device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 9G NR, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0111] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described in any of the above embodiments.

[0112] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of an electronic device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0113] In an exemplary embodiment, this disclosure also provides a computer program product including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0114] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0115] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for setting ISP parameters, characterized in that, include: Acquire raw image data and scene metadata, wherein the scene metadata is used to characterize the target shooting scene when the raw image data was captured; The original image data and the scene metadata are input into a trained ISP parameter setting model, so that the trained ISP parameter setting model generates target values ​​for each ISP parameter that match the target shooting scene; The current values ​​of each ISP parameter in the ISP chip are updated to the corresponding target values, so that the ISP chip processes the original image data based on the updated values ​​to obtain the target image.

2. The method according to claim 1, characterized in that, It also includes: matching the original image data with the scene metadata using timestamps to filter out scene metadata that matches the timestamps of the original image data; The step of inputting the original image data and the scene metadata into the trained ISP parameter setting model includes: associating the original image data with the selected scene metadata and inputting it into the trained ISP parameter setting model.

3. The method according to claim 1, characterized in that, The step of inputting the original image data and the scene metadata into the trained ISP parameter setting model includes: When a change in the shooting scene is detected, the original image data and scene metadata corresponding to the changed shooting scene are input into the trained ISP parameter setting model.

4. The method according to claim 1, characterized in that, Also includes: Obtain evaluation feedback for the target image, and adjust the model parameters of the trained ISP parameter setting model based on the evaluation feedback and the target image.

5. A training method for an ISP parameter setting model, characterized in that, include: Acquire sample original image data, sample scene metadata, and sample target image corresponding to the sample original image data. The sample scene metadata is used to characterize the sample shooting scene when the sample original image data was captured. The original image data of the sample and the metadata of the sample scene are input into the ISP parameter setting model so that the ISP parameter setting model can generate sample prediction values ​​of each ISP parameter that match the shooting scene of the sample. The original image data of the sample is processed using the sample prediction values ​​of each ISP parameter to obtain the sample prediction image corresponding to the shooting scene of the sample. The loss value is calculated based on the predicted image of the sample and the target image of the sample, and the ISP parameter setting model is iteratively trained based on the loss value.

6. The method according to claim 5, characterized in that, It also includes: matching the original sample image data with the sample scene metadata using timestamps to filter out sample scene metadata that matches the timestamps of the original sample image data; The step of inputting the original sample image data and the sample scene metadata into the ISP parameter setting model includes: associating the original sample image data with the selected sample scene metadata and inputting it into the ISP parameter setting model.

7. The method according to claim 5, characterized in that, Also includes: The original sample image data and the sample scene metadata are vectorized to obtain the original sample image data vector corresponding to the original sample image data and the sample scene metadata vector corresponding to the sample scene metadata. The original image data vector of the sample and the metadata vector of the sample scene are concatenated, and the concatenated vector is input into the ISP parameter setting model.

8. The method according to claim 5, characterized in that, The iterative training of the ISP parameter setting model includes: a first-stage iterative training using a simulated ISP chip deployed on a PC, and / or a second-stage iterative training using a real ISP chip deployed on a mobile device.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 8 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 8.

11. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 8.

Images