A method for optimizing image signal processing

Abstract

A method and apparatus for optimizing image signal processing, wherein the image signal processing comprises a plurality of individual subtasks performed in a predetermined order, wherein the method comprises the steps of: defining a first group of subtasks, the first group having a starting point and an end point in the predetermined order and comprising at least two or more subtasks; performing the subtasks in the predetermined order until the starting point of the first group; performing the subtasks of the first group with a predetermined first set of parameters until the end point of the first group; modifying the predetermined first set of parameters; and performing the subtasks of the first group with the modified first set of parameters until the end point of the first group.

Description

Technical Field

[0001] This invention relates to a method, corresponding software and apparatus for processing image signals, and an apparatus for achieving optimized image signal processing. Background Technology

[0002] With the increasing prevalence of mobile device photography, the demand for image quality and performance is also rising. Therefore, the camera sensors of such devices capture light from the scene and transform this information into an image that meets the user's expectations. This transformation is performed by an image signal processing (ISP) pipeline that includes several algorithms or subtasks to reduce unwanted effects or interference on the image. Such subtasks include, but are not limited to, noise reduction, white balance, color correction, lens shading correction, defect pixel correction, gamma correction, tone mapping (including adjustments to the image's brightness and contrast), etc. Typically, ISP is implemented in specific software or hardware to accelerate image processing. Therefore, for dedicated devices or camera sensors, the subtasks or algorithms must be adequately adapted, which is often accomplished by simulating ISP on general-purpose hardware.

[0003] In systems that process images, or image sequences such as videos, or any combination thereof, processing time often becomes a bottleneck because the vast majority of image processing tasks are not performed by dedicated hardware. An example of this scenario is processing raw frames through an ISP simulator, where the user's task is to iteratively adjust subtasks or algorithm parameters to maximize image quality. Therefore, simulating an ISP for optimization is extremely time-consuming.

[0004] Traditionally, ISP simulators are capable of simulating the following images: (i) from start to finish; (ii) from start to any stop point; and (iii) simulating a single algorithm or subtask in a pipeline.

[0005] One of the well-known drawbacks of ISP simulations is that they are typically slow, often failing to assess the success of subtask adjustments by focusing on individual blocks, failing to evaluate the image before sufficient processing is completed (when the subtask is early in the ISP process), and the subtask can mask problems. Slowness is the most significant issue because, despite these drawbacks, simulations often only perform adjustments on individual subtasks or stop simulating after adjustments, neglecting the impact on subsequent subtasks. Summary of the Invention

[0006] Therefore, the object of the present invention is to provide a method for optimizing image signal processing, and in particular an improved image signal processing simulation.

[0007] The problem is solved by the method of claim 1, the software of claim 10, the device of claim 11, and the apparatus for implementing ISP optimization according to claim 12.

[0008] In a first aspect, a computer-implemented method for optimizing image signal processing is provided, wherein the image signal processing includes a plurality of individual subtasks executed in a predetermined order, wherein the method includes the following steps:

[0009] a) Define the subtasks of the first group, which has a start point and an end point in a predetermined order and includes at least two or more subtasks;

[0010] b) Execute subtasks in a predetermined order until the starting point of the first group;

[0011] c) Execute the subtasks of the first group using the predetermined first parameter set until the end of the first group;

[0012] d) Modify the predefined first parameter set; and

[0013] e) Execute the subtasks of the first group using the modified first parameter set until the end of the first group.

[0014] The invention implements subtasks using corresponding algorithms to convert light information captured by a camera sensor into a final image. Image signal processing (ISP) is then performed in a predetermined sequence or pipeline, simulated according to the steps described above. The corresponding subtasks are adjusted by modifying a predetermined first set of parameters to optimize the ISP for user needs. This predetermined first set of parameters may include a set of parameters for a specific subtask, or it may include a predetermined set of parameters for more than one subtask in a first group. Modifications can be performed manually or based on evaluations performed by an evaluation algorithm (e.g., artificial intelligence or machine learning). However, the invention is not limited to, and in particular does not relate to, a specific manner of modifying and optimizing parameters. However, since the individual subtasks to be adjusted are defined within a group that preferably includes subtask-dependent subtasks, the results of parameter modifications can be directly evaluated after simulating the group, and undesirable effects and influences can be excluded. Specifically, it is neither necessary to simulate all subtasks from the beginning to the end of the ISP pipeline (which is very time-consuming) nor to limit the simulation to a single or individual subtask. Therefore, simulation processing is accelerated because not every adjustment step requires simulation of the entire ISP, and simulation is not limited to optimizing a subtask, thus avoiding unnecessary and unpredictable results.

[0015] Preferably, steps (d) and (e) are repeated at least once, and more preferably multiple times, wherein the first parameter set modified in step (e) is further modified in the repeated step (d) until the optimization of one or more dedicated subtasks of the first group is completed. Therefore, optimization can be performed as an iterative process, wherein only the individual subtasks of the first group are simulated at least once, and preferably multiple times, until the desired adjustment result is achieved. These steps are repeated until a modified first parameter set that provides the desired result is found. The first parameter set serves as the final first parameter set for the individual subtasks of the first group or for the subtasks themselves, and can be used by simulations of the remaining subtasks in the ISP pipeline, or in the final product of implementing the optimized ISP.

[0016] Preferably, before step (c), the result of the last subtask before the first group is cached and used as the starting point for the simulation of executing the subtasks of the first group.

[0017] Preferably, after step e), the method may include the following steps:

[0018] f) Define a second group of subtasks, wherein the second group includes at least two subtasks, wherein at least one subtask is different from the subtasks of the first group, and wherein the second group has a start point and an end point in a predetermined order.

[0019] g) Execute the subtasks of the second group using the predetermined second parameter set until the end of the second group;

[0020] h) Modify the predefined second parameter set; and

[0021] i) Execute the subtasks of the second group using the modified second parameter set until the end of the second group.

[0022] Therefore, using the second group, another dedicated subtask to be adjusted can be defined and included within the second group, and adjusted again in a process similar to that described above. Here, the predetermined second parameter set is the set of parameters for a single dedicated subtask within the subtasks of the second group. Alternatively, the second parameter set includes parameters for more than one subtask within the subtasks of the second group. Thus, using the second group, a second dedicated subtask to be adjusted can be defined by simulating the ISP from start to finish, and adjustment can be performed without repeatedly simulating the entire ISP.

[0023] Preferably, steps (h) and (i) are repeated at least once, and preferably multiple times, wherein, for each repetition, the modified second parameter set is defined and used to execute the subtasks of the second group. Thus, optimization can be performed as an iterative process, wherein only the individual subtasks of the second group are simulated at least once, and preferably multiple times, until the desired adjustment result is achieved. These steps are repeated until a modified second parameter set that provides the desired result is found. This second parameter set serves as the final second parameter set for one or more subtasks of the second group, and can be used for simulations of the remaining subtasks in the ISP pipeline, or in the final product of implementing the optimized ISP.

[0024] Preferably, the second group is arranged at least partially after the first group. This means that at least one subtask of the second group follows the first group in a predetermined subtask order and therefore does not belong to the first group. Alternatively or additionally, the second group is arranged at least partially parallel to the first group. Since the ISP pipeline may also include branches, the subtasks can be parallel, wherein the first group may at least partially form a first branch, and the second group may form at least one of the subtasks of the parallel branch. In all alternatives, the first and second groups may overlap and share common subtasks belonging to both groups.

[0025] Preferably, the last subtask of the first group is the first subtask of the second group. Alternatively, the first subtask of the first group that is no longer part of the first group is the first subtask of the second group. Therefore, the second group immediately follows the first group.

[0026] Preferably, each subtask of the ISP is assigned to one or more groups so that each group can adjust its respective subtask.

[0027] Preferably, each group includes at least one subtask to be adjusted.

[0028] Preferably, at least one group, more preferably each group, includes at least one, particularly all, dependent subtasks that depend on the processing results of the dedicated subtask to be adjusted. Therefore, modifying the first set of parameters used to adjust the dedicated subtask within the first group may also affect the processing results of the dependent subtasks. Combining these dependent subtasks in the first group results in iterative simulation of their common behavior in the ISP. This avoids undesirable side effects caused by neglecting the dependent subtasks while adjusting the dedicated subtask.

[0029] Preferably, the remaining subtasks are executed after the end of the last group (i.e., the last subtask of the last group) until the ISP ends.

[0030] In another aspect of the invention, a computer program is provided, the program including instructions which, when executed by a computer, cause the computer to perform the steps of the method described above.

[0031] In another aspect of the invention, an apparatus is provided, the apparatus including a processor and a non-volatile memory storage medium storing machine-executable instructions, wherein the machine-executable instructions are executable by the processor and cause the processor to perform the steps of the method described above.

[0032] In another aspect of the invention, an apparatus is provided comprising an image sensor for capturing image data and a processor connected to the image sensor, wherein the processor is configured to perform image signal processing optimized by the method described above. Thus, the ISP can be implemented as dedicated hardware or software, wherein an optimized set of parameters for each adjustment subtask is implemented and used to convert the light information captured by the image sensor into a final image.

[0033] This invention provides an improvement to the simulation of an ISP, thereby avoiding the drawbacks of currently known simulation methods. For example, the slowness resulting from simulating the entire ISP by individually tweaking each subtask for each subtask, or the disadvantages and undesirable results caused by pipeline dependencies between individual subtasks when selecting individual subtasks and limiting the simulation to those individual subtasks. Therefore, this invention achieves more accurate and predictable simulation results by tweaking and considering all dependent subtasks that depend on the tweaked subtask before evaluating the tweaking results for each individual subtask. Furthermore, unnecessary simulation of subtasks is avoided by repeating the simulation only for the subtask to be tweaked and at least the individual groups of its dependent subtasks. Attached Figure Description

[0034] The invention will be described in further detail with reference to the accompanying drawings.

[0035] The attached diagram shows:

[0036] Figure 1 Example of an ISP

[0037] Figure 2 According to the steps of the method of the present invention,

[0038] Figure 3 : An ISP that implements the method according to the present invention,

[0039] Figure 4 According to the steps of the method in another embodiment,

[0040] Figures 5A to 5D : ISP implementation methods for the different subtasks that have been adjusted, and

[0041] Figure 6 : A device for implementing optimized IPS. Detailed Implementation

[0042] Figure 1 An example of an ISP pipeline is shown, which transmits raw image data 10 to a final image 20, thereby converting pure light information captured by a camera sensor or image sensor into an image that meets the user's requirements for quality and performance. The ISP includes multiple subtasks 30, which perform different tasks within the ISP to convert the raw image data 10 into the final image data 20. These subtasks may be associated with noise reduction, white balance, color correction, color interpolation, lens shading correction, defect pixel correction, gamma correction, tone mapping, contrast adjustment, and brightness adjustment, etc., which are necessary to meet the user's requirements for the final image 20. The invention is not limited to a specific order of subtasks, nor to any particular subtask itself. The subtasks can be implemented as algorithms. Figure 1 As shown, subtask 30 is executed in a predetermined order and may also include branches 34, wherein different subtasks are executed in parallel. Furthermore, the invention is not limited to a specific number of subtasks or parallel branches.

[0043] To adapt the ISP to new equipment, new camera sensors, new hardware, etc., the ISP must be adjusted to provide the desired results. Therefore, the parameter set of the corresponding subtask 36 is adjusted until the desired result is achieved. The parameters describe the specific behavior or result of each subtask. Adjustments can also be made repeatedly for one or more other subtasks.

[0044] However, typically, the results of the adjustments cannot be viewed immediately after each of the adjusted subtasks 36; instead, other subtasks that can be considered dependent on subtask 32 need to be run. The adjusted subtask 36 naturally depends on previous subtask adjustments. Therefore, in the prior art, the process is such that subtasks are adjusted one by one, following specific adjustment guidelines specific to the ISP and nearby tasks. This is a very time-consuming process, which typically slows down time-to-market and further relies heavily on the user 38's experience in adjusting the ISP.

[0045] Figure 2 The steps of the present invention are shown:

[0046] In step S01, the subtasks of the first group are defined as having a start point and an end point in a predetermined order and including at least two or more subtasks.

[0047] In step S02, subtasks are executed or simulated in a predetermined order until the starting point of the first group.

[0048] In step S03, the subtasks of the first group are executed or simulated using a predetermined first parameter set until the end of the first group.

[0049] In step S04, the predetermined first parameter set is modified.

[0050] In step S05, the subtasks of the first group are executed or simulated using the modified first parameter set until the end of the first group.

[0051] Among them, such as Figure 1 As shown, steps S04 and S05 can be repeated to determine the set of optimization parameters for the subtasks to be adjusted, which is then used to adjust further steps of the remaining subtasks and also to implement ISP in the device.

[0052] Figure 3 An example illustrates the process of simulating and optimizing a dedicated subtask. Among other things, Figure 3 The ISP includes a certain number of subtasks 30.

[0053] - Upon completion of the first run starting from pipeline (A), the simulation stops at point (B) and the results at this point are cached. Point (B) indicates the starting point of the first group, which in this example comprises several subtasks 42, including branches of the ISP.

[0054] - After this, the user can make any modifications to the subtasks 42 of the first group between points (B) and (C) or their parameters, modifying the corresponding parameter set. This method allows for the simultaneous adjustment of one or more subtasks between points (B) and (C), making the pipeline and tuning tool architecture more flexible compared to existing technologies.

[0055] - After these desired modifications are made, the simulation can continue and will run to point (C) where the results can be analyzed.

[0056] - If the user sees that the modifications made between points (B) and (C) in the first group do not produce satisfactory results, he / she can further modify the algorithm / parameters and repeat the simulation of the region between points (B) and (C).

[0057] - Once the adjustments between points (B) and (C) in the first group are satisfactory, the user can move these trimming points forward in the pipeline if needed. This avoids the need to repeat simulations between points (A) and (C), thus saving time. After this, the above steps can be repeated by defining another group comprising two or more subtasks.

[0058] Once the results are satisfactory, the simulation can continue until the end of pipeline (D).

[0059] refer to Figure 4 It illustrates the steps for defining a second set of tasks with at least two subtasks, wherein these steps are referenced above. Figure 2 The described steps S01 to S05 are executed after or in parallel. The second group includes at least one subtask to be adjusted, and may also include one or more dependent subtasks that depend on the subtask to be adjusted.

[0060] In step S06, a second group of subtasks is defined, comprising at least two subtasks, wherein at least one subtask is different from the subtasks in the first group, and wherein the second group has a start point and an end point in a predetermined order.

[0061] In step S07, the subtasks of the second group are executed or simulated using a predetermined second parameter set until the end of the second group.

[0062] In step S08, the predetermined second parameter set is modified.

[0063] In step S09, the subtasks of the second group are executed or simulated using the modified second parameter set until the end of the second group.

[0064] Among them, such as Figure 3 As shown, steps S08 and S09 can be repeated to determine the set of optimization parameters for the subtasks to be adjusted, which is then used to adjust further steps of the remaining subtasks in the ISP and also to implement the ISP in the device.

[0065] Reference Figures 5A to 5D It illustrates an exemplary process for adjusting an ISP, wherein the ISP has several subtasks 30 executed in a predetermined order, similar to... Figure 1 Convert the raw data into the final image.

[0066] Among them, according to Figure 5A It is possible to define the first group 50, which includes the first subtask to be adjusted. Figure 5A In the example, the first group 50 includes three subtasks. At least one subtask in the first group 50 can be adjusted. Alternatively, more than one or all subtasks of the first group 50 can be adjusted, i.e., the parameters used to perform the subtasks are modified by a human user or due to machine evaluation, for example, by artificial intelligence or machine learning. Furthermore, the first group 50 may include one or more dependent subtasks that depend on the one or more subtasks to be adjusted. To optimize the parameters and find the final parameter set for each subtask of the first group 50, the subtasks of the first group 50 are iteratively repeated / simulated, wherein, for each step, one or more sets of parameters are modified accordingly until the desired result defined by the final parameter set for the first group 50 is obtained. Subsequently, as... Figure 5BThe depicted design defines a second group 52, which in this example includes six subtasks of the first branch of the ISP, and includes one or more subtasks to be tuned. Furthermore, the second group 52 may include one or more dependent subtasks that depend on the subtasks to be tuned. It is noted that the first subtask of the second group 52 is a subsequent subtask of the last subtask of the first group 50, such that there is no overlap between the first and second groups. The subtasks of the second group 52 are repeatedly simulated to find the final parameter set of the subtasks to be tuned, such that the results of the second group of subtasks provide the desired results. Subsequently, as... Figure 5C As illustrated, the third group 54 is defined, comprising four subtasks of the parallel branch and a fifth subtask after the branch reorganization. Note that the fifth subtask of the third group 54 is also included as the last subtask in the second group, such that the second group 52 and the third group 54 at least partially overlap. The subtasks of the third group 54 are simulated repeatedly to find the final parameter set of the subtasks to be adjusted so that the results of the third group of subtasks provide the desired results. Subsequently, as... Figure 5D As illustrated, a fourth group 56 is defined, which comprises four subtasks. The subtasks of the fourth group 54 are repeatedly simulated to find the final parameter set of the subtasks to be adjusted so that the results of the fourth group of subtasks provide the desired results. Wherein... Figures 5A to 5D In the example, the fourth group of 56 subtasks provides the final image of the optimized ISP.

[0067] Figures 5A to 5D The examples are certainly not limited to the number of subtasks and branches shown. Furthermore, the selected groups are not limited to any particular number, and each group can include two or more subtasks.

[0068] Furthermore, these groups can overlap, follow one another sequentially, or be separated by one or more independent subtasks, which, as known in the prior art, can be simulated in isolation. These isolated subtasks may not include any dependent subtasks. Furthermore, as... Figures 5A to 5D The illustrated ISP ends with a fourth group, 56. However, the invention is not limited to the case where the ISP ends with a specific group. Instead, in another embodiment, the fourth group may be followed by one or more additional subtasks, or isolated subtasks or subtasks with dependent subtasks, or subtasks with more groups. Each group may include at least one subtask to be adjusted, and may or may not include dependent subtasks that depend on the subtask to be adjusted.

[0069] Therefore, the defined group can be customized based on the specific dependencies of the subtask to be tuned, avoiding redundant simulations that include simulations of all subtasks in the ISP. Furthermore, it avoids limiting the simulation to a single subtask to be tuned, ignoring dependent subtasks that might lead to undesirable results. Thus, by improving the evaluation capability of the modified parameter set for each tuned subtask, the desired results can be obtained more quickly, thereby accelerating the overall simulation process.

[0070] Figure 6 An apparatus, such as terminal device 200, is shown, which includes a camera sensor or image sensor 220 connected to a processor 240. Furthermore, the processor 240 is connected to a memory 260. The memory 260 stores the final set of parameters found through the simulation described above in order to provide an optimized ISP that transforms the raw image data captured by the image sensor 220 and provides the final image to the user of the terminal device 200. The processor 240 with memory 260 can be implemented as a dedicated image processor that implements the ISP in hardware, or the ISP can be implemented in software stored in memory 260 and processed by the processor 240.

Claims

1. A computer-implemented method for optimizing image signal processing, wherein, The image signal processing includes multiple individual subtasks executed in a predetermined order, wherein the method includes: a) Define a first group of subtasks, wherein the first group has a start point and an end point in the predetermined order and includes at least two or more subtasks executed sequentially, wherein the at least two or more subtasks included in the first group are some of the multiple individual subtasks; the first group includes one or more dependent subtasks that depend on one or more subtasks to be adjusted. b) Execute the subtasks in the predetermined order until the beginning of the first group; c) Execute the subtasks of the first group using a predetermined first parameter set until the end of the first group; d) Modify the predetermined first parameter set; and e) Execute the subtasks of the first group using the modified first parameter set until the end of the first group; Repeat steps d) and e) once or more until the desired adjustment result is achieved; f) Define a second group of subtasks, wherein the subtasks of the second group include at least two subtasks executed sequentially, wherein at least one subtask is different from the subtasks of the first group, wherein the second group has a start point and an end point in the predetermined order, the second group is arranged at least partially after the first group in the predetermined order, and the last subtask of the first group is the first subtask of the second group, or the direct successor subtask of the first group is the first subtask of the second group. g) Execute the subtasks of the second group using the predetermined second parameter set until the end of the second group.

2. The method according to claim 1, wherein, Before step c), cache the results of the last subtask preceding the first group.

3. The method according to any one of claims 1 to 2, wherein, The method includes: h) Modify the predetermined second parameter set; and i) Execute the subtasks of the second group using the modified second parameter set until the end of the second group.

4. The method according to claim 3, wherein, Steps h) and i) are repeated at least once, and preferably multiple times.

5. The method according to any one of claims 1 to 4, wherein, Before step g), cache the results of the last subtask preceding the second group.

6. The method according to any one of claims 1 to 4, wherein, After the end of the last group, the remaining subtasks are executed until the predetermined sequence is completed.

7. A data processing device, comprising a processor, wherein, The processor is configured to perform the steps of the method according to any one of claims 1 to 6.

8. An apparatus wherein, The device has an image sensor for capturing image data and a processor connected to the image sensor, wherein the processor is configured to perform image signal processing optimized by the method according to any one of claims 1 to 6 or the device according to claim 7.

Images