Image processing method and device, computer equipment, chip and readable storage medium

Abstract

The invention relates to an image processing method and device, computer equipment, a chip and a readable storage medium. The method comprises the following steps: acquiring a first output result and a second output result; the first output result comprises an output result of image processing performed by the last ISP algorithm module in the series branch, and the second output result comprises an output result of image processing performed by the AI algorithm module of each branch in the parallel branch; performing weight parameter adjustment on the first output result and the second output result to obtain fusion weights corresponding to the first output result and the second output result respectively; and carrying out weighted fusion on the first output result and the second output result by adopting a fusion weight to obtain an output result of the ISP algorithm module group. According to the method, the final required image style is adjusted by adjusting the weight value corresponding to each algorithm module.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer equipment, chip and readable storage medium. Background Art

[0002] Image Signal Processor (ISP) is a dedicated chip module for processing camera images or video signals. It usually processes image signals serially using pipelines. The core of ISP is this series of serial processing algorithms solidified on the chip, which is usually called ISP Pipeline.

[0003] With the development of artificial intelligence (AI) technology, ISP-related AI algorithms have continued to emerge, and the concept and implementation of AI ISP have gradually emerged. However, it is difficult to adjust the traditional parameters for AI ISP to achieve the desired effect. Summary of the invention

[0004] Based on this, it is necessary to provide an image processing method, device, computer equipment, chip and readable storage medium that can adjust the effect in response to the above technical problems.

[0005] In a first aspect, the present application provides an image processing method, which is applied to an image processing system, wherein the image processing system includes an image signal processing ISP algorithm module group, the ISP algorithm module group includes multiple algorithm modules with the same or similar image processing effects, the multiple algorithm modules include an AI algorithm module and at least one ISP algorithm module constituting a series branch, wherein the series branch is connected in parallel with one or more AI algorithm modules to form a parallel branch; the method includes:

[0006] Obtain a first output result and a second output result; the first output result includes an output result of the image processing performed by the last ISP algorithm module in the series branch, and the second output result includes an output result of the image processing performed by the AI ​​algorithm modules of each branch in the parallel branch;

[0007] Adjusting the weight parameters of the first output result and the second output result to obtain the fusion weights corresponding to the first output result and the second output result respectively;

[0008] The first output result and the second output result are weightedly fused using the fusion weight to obtain the output result of the ISP algorithm module group.

[0009] In one embodiment, the method further comprises:

[0010] Receive the original image and input the original image into the ISP algorithm module group.

[0011] In one embodiment, weight parameter adjustment is performed on the first output result and the second output result, including:

[0012] Perform weight parameter adjustment on the first output result and the second output result based on a weight adjustment strategy; the weight adjustment strategy includes a weight value determination rule and a weight value adjustment rule;

[0013] Among them, the weight value determination rule includes determining a preset fixed value according to the acquisition scenario and image usage of the original image, or obtaining a dynamically adjusted value by calculating based on the original image; the weight value adjustment rule includes a shared weight or per-pixel adjustment for the region of interest.

[0014] In one embodiment, the method further includes:

[0015] When the weight value adjustment rule is per-pixel adjustment for the region of interest, filter the fusion weight using a filtering algorithm.

[0016] In one embodiment, when each algorithm module corresponds to different image processing functions in the same image processing task, the fusion weight includes negative weights and positive weights.

[0017] In one embodiment, the image processing system further includes a direct-through branch connected in parallel at both ends of the series branch, and each branch of the parallel branch includes an AI algorithm module; the fusion weight corresponding to the first output result is the first weight value, and the fusion weight corresponding to the second output result is the second weight value;

[0018] Perform weighted fusion on the first output result and the second output result using the fusion weight to obtain the output result of the ISP algorithm module group, including:

[0019] Obtain the third output result of the direct-through branch, and the fusion weight corresponding to the third output result is the third weight value; among them, the sum of the first weight value, the second weight value, and the third weight value is 1;

[0020] Output a weighted average according to the first weight value, the second weight value, and the third weight value to obtain the output result of the ISP algorithm module group.

[0021] In one embodiment, the first weight value, the second weight value, and the third weight value are all between 0 and 1.

[0022] Second aspect, the present application also provides an image processing apparatus, which is applied to an image processing system. The image processing system includes an Image Signal Processing (ISP) algorithm module group. The ISP algorithm module group includes multiple algorithm modules with the same or similar image processing effects. The multiple algorithm modules include an AI algorithm module and at least one ISP algorithm module forming a series branch. Wherein, the series branch is connected in parallel with one or more AI algorithm modules to form a parallel branch; the apparatus includes:

[0023] An output result acquisition module, configured to acquire a first output result and a second output result; the first output result includes the output result of image processing by the last ISP algorithm module in the series branch, and the second output result includes the output results of image processing by the AI algorithm modules of each branch in the parallel branch;

[0024] A weight parameter adjustment module, configured to adjust the weight parameters of the first output result and the second output result, and acquire the fusion weights corresponding to the first output result and the second output result respectively;

[0025] A weighted fusion module, configured to perform weighted fusion on the first output result and the second output result by using the fusion weights to obtain the output result of the ISP algorithm module group.

[0026] Third aspect, the present application also provides an image processing system, including an Image Signal Processing (ISP) algorithm module group. The ISP algorithm module group includes multiple algorithm modules with the same or similar image processing effects. The multiple algorithm modules include an AI algorithm module and at least one ISP algorithm module forming a series branch. Wherein, the series branch is connected in parallel with one or more AI algorithm modules to form a parallel branch;

[0027] Wherein, the output result of the ISP algorithm module group is obtained by using the steps of the above image processing method.

[0028] In one embodiment, each branch of the parallel branch includes one AI algorithm module.

[0029] In one embodiment, the image processing system further includes an input module and an output module;

[0030] The input module is configured to receive the original image and input the original image into the ISP algorithm module group;

[0031] The output module is configured to output the output result of the ISP algorithm module group.

[0032] In one embodiment, the image processing system further includes a direct-through branch connected in parallel at both ends of the series branch.

[0033] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0034] In a fourth aspect, the present application also provides a chip, which includes a programmable logic circuit and / or program instructions, and when the chip runs, the steps of the above method are implemented.

[0035] In a fifth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0036] In a sixth aspect, the present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0037] The above image processing method, device, computer device, chip and readable storage medium, the image signal processing ISP algorithm module group in the image processing system, includes a plurality of algorithm modules with the same or similar image processing effects. The plurality of algorithm modules include an AI algorithm module and at least one ISP algorithm module constituting a series branch. Among them, the series branch is connected in parallel with one or more AI algorithm modules to form a parallel branch. Furthermore, by adjusting the weights of the output results of each algorithm module, the result output by the ISP algorithm module group is optimized. In the embodiments of the present application, for algorithm modules with similar effects or close effects, they are placed in the Pipeline in a parallel manner. In terms of the realization of the final effect, by adjusting the weight values corresponding to each algorithm module, the required image style is adjusted. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0039] Figure 1 It is a block diagram of the module structure of a traditional ISP Pipeline;

[0040] Figure 2 It is an application environment diagram of the image processing method in an embodiment;

[0041] Figure 3 It is a schematic flowchart of the image processing method in an embodiment;

[0042] Figure 4 It is a schematic flowchart of the weighted average step in an embodiment;

[0043] Figure 5 is the block diagram of the module structure of the ISP Pipeline in one embodiment;

[0044] Figure 6 is the block diagram of the module structure of the ISP Pipeline in another embodiment;

[0045] Figure 7 is the block diagram of the module structure of the ISP Pipeline in yet another embodiment;

[0046] Figure 8 is the block diagram of the structure of the image processing device in one embodiment;

[0047] Figure 9 is the internal structure diagram of a computer device in one embodiment. Detailed implementation manners

[0048] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] It can be understood that terms such as "first" and "second" in this application are only used to distinguish similar objects and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. It can be understood that "at least one" means one or more, and "a plurality" means two or more.

[0051] When used herein, the singular forms of "a", "an" and "the" may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" etc. specify the presence of the stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the related listed items.

[0052] Traditional ISP Pipeline (such as Figure 1As shown in the figure, the processing algorithms are divided into several different image formats (RAW, RGB, YUV, etc.). The algorithms in each format are further divided into different modules according to the processing required for the image. Different modules complete a single task (noise reduction, white balance, color adjustment, etc.). In the traditional ISP Pipeline, the algorithm of each single module has parameters with a certain meaning that can be adjusted. After the basic framework of the algorithm is determined, the entire ISP Pipeline can be adjusted according to the specific meaning of the parameters of each module and the effects required by the project to meet the final needs. It can be understood that Figure 1 The n and N in the above formula can be set according to the requirements. For example, n and N are positive integers respectively.

[0053] At present, AI ISP can be an end-to-end replacement for traditional ISP, or it can replace one or several modules in traditional ISP. In most cases, AI ISP algorithms have better performance than traditional ISP algorithms, but AI ISP algorithms have certain difficulties in adjusting parameters. Usually, AI ISP parameters are obtained by training neural networks, and it is difficult to make interpretable adjustments. In addition, AI ISP algorithms are still under development. A single neural network completes the entire ISPPipeline, or a single neural network completes the tasks of one or more modules in the ISP Pipeline. New AI algorithms and traditional algorithms or existing AI algorithms are constantly compared, replaced, or complemented.

[0054] In view of the difficulty in adjusting the parameters of AI ISP in traditional solutions for the final effect, the embodiment of the present application proposes a pipeline including an AI ISP algorithm that can adjust the parameters of AI ISP for effect orientation. It should be noted that the beneficial effects or technical problems solved by the embodiment of the present application are not limited to this one, and may also be other implicit or related problems, which can be specifically referred to the description of the following embodiment.

[0055] The image processing method provided in the embodiment of the present application can be applied to Figure 2 In the application environment shown, Figure 2 1 is a schematic diagram of the structure of a camera provided by an embodiment of the present application, wherein the camera may include: a body 102, a lens 104 disposed on the body 102, an image sensor 106 and an ISP chip 108 disposed in the body 102; the image sensor 106 collects a raw image (such as a camera image or a video signal) through the lens 104, and the ISP chip 108 may receive the raw image from the image sensor 106 and process the raw image. It can be understood that the raw image may be the original RAW data.

[0056] The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. The following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0057] In an exemplary embodiment, as Figure 3 shown, an image processing method is provided. Taking the application of this method to an image processing system as an example, the method may include the following steps 202 to 206.

[0058] Step 202, obtain a first output result and a second output result; the first output result includes the output result of image processing by the last ISP algorithm module in the series branch, and the second output result includes the output results of image processing by the AI algorithm modules in each branch of the parallel branch.

[0059] Among them, an image processing system is proposed in an embodiment of the present application. This image processing system can be used as the overall framework of the ISP Pipeline. Further, an embodiment of the present application also provides an arrangement method of the AI algorithm modules. Exemplarily, the image processing system may include an image signal processing ISP algorithm module group. The ISP algorithm module group includes multiple algorithm modules with the same or similar image processing effects. The multiple algorithm modules include AI algorithm modules and at least one ISP algorithm module constituting a series branch. Among them, the series branch is connected in parallel with one or more AI algorithm modules to form a parallel branch.

[0060] It should be noted that the AI algorithm module in the image processing system can be understood as an AI ISP algorithm module, and the ISP algorithm module can be understood as a traditional ISP Pipeline; for multiple image processing algorithms used to achieve the same or similar image processing effects, at the software level, each image processing algorithm corresponds to an algorithm module. Algorithm modules with the same or similar image processing effects can be simply referred to as algorithm modules with the same or similar effects. Exemplarily, algorithm modules with the same or similar image processing effects are directed to the same image processing task (or the same type of image processing tasks).

[0061] Optionally, the AI ISP algorithm in the embodiment of the present application may include, but is not limited to, noise reduction, white balance, color adjustment, brightness adjustment, and wide dynamic range algorithm, etc.

[0062] The main difference between the ISP Pipeline in this application and the traditional ISP Pipeline lies in that for algorithm modules with similar or close effects, there is no need to arrange them in the Pipeline in a way of selecting one from multiple or serial processing. In this application, for algorithm modules with similar or close effects, whether they are not processed, simply format-converted, various traditional algorithms, or various AI algorithms, they are all placed in the Pipeline in parallel. In addition, based on the above image processing system, this application provides a parameter adjustment method related to the AI ISP algorithm module, so that in terms of the final effect realization, instead of selecting one from multiple algorithms, weights are assigned to the results obtained by different image processing algorithms.

[0063] Specifically, multiple algorithm modules with the same or similar image processing effects can refer to simultaneously selecting multiple algorithms for the same type of image processing effect. Moreover, through the parallel branches formed between the AI algorithm module and at least one ISP algorithm module constituting a series branch, the embodiments of this application can parallelize multiple algorithms (the specific operations of the algorithms are different, but similar results can be obtained). Exemplarily, these similar results can be fused by means of weighting.

[0064] Among them, based on the image processing system, a first output result and a second output result can be obtained. The first output result can include the output result of the last ISP algorithm module in the series branch for image processing, and the second output result can include the output results of the AI algorithm modules in each branch of the parallel branch for image processing, that is, the output results of multiple algorithms for the same type of image processing effect are obtained.

[0065] It should be noted that the series branch in the embodiments of this application is a series branch formed by sequentially connecting at least one ISP algorithm module. Exemplarily, taking the input of the original image as an example, according to the execution order, the first ISP algorithm module among multiple ISP algorithm modules can perform image processing on the original image to obtain the output image of the first ISP algorithm module, and each of the other ISP algorithm modules except the first ISP algorithm module can perform image processing on the output image of the previous ISP algorithm module of this ISP algorithm module to obtain the output image of this ISP algorithm module. Furthermore, the output image of the last ISP algorithm module in the series branch can be used as the above-mentioned first output result.

[0066] Step 204: Adjust the weight parameters of the first output result and the second output result to obtain the fusion weights corresponding to the first output result and the second output result respectively.

[0067] Specifically, by adjusting the weight parameters of the first output result and the second output result, the fusion weights corresponding to the first output result and the second output result can be obtained. Among them, by adjusting the weight parameters, the weight values of each fusion weight (i.e., the weight values of the fusion weights) can be changed. For example, for the selection of weight parameters, a certain algorithm weight can be set to 1 and the others to 0, which is equivalent to selecting the algorithm with a weight of 1. Other selection methods of weight parameters can also be set according to needs.

[0068] In the embodiment of the present application, by assigning corresponding weights to various algorithms, the advantages of each algorithm can be aggregated, and the output result can be optimized. It can be understood that different fusion weights can refer to assigning different weight values to the results output by different algorithms.

[0069] Step 206: Perform weighted fusion on the first output result and the second output result using the fusion weights to obtain the output result of the ISP algorithm module group.

[0070] Specifically, the first output result and the second output result can be weighted and fused using the fusion weights to obtain the output result of the ISP algorithm module group. In the embodiment of the present application, by adjusting the weight parameters of the results obtained by various different algorithms and fusing the results of each algorithm to obtain the final result, the problem that the parameters of the AI ISP algorithm are difficult to adjust directionally is solved.

[0071] Based on the image processing method of the embodiment of the present application, multiple AI neural network algorithms, or multiple groups of training parameters with different styles of a certain neural network framework, can all be applied in the same ISP Pipeline, and the final required image style can be adjusted by simply adjusting the weight values corresponding to each algorithm.

[0072] It can be understood that the above weight parameter adjustment and weighted fusion can also adopt other forms, not limited to the forms already mentioned in the above embodiments, as long as they can achieve the functions of assigning corresponding weights to various algorithms and adjusting the weight values corresponding to each algorithm.

[0073] In one of the embodiments, the method may further include:

[0074] Receive the original image and input the original image into the ISP algorithm module group.

[0075] Specifically, the image processing system may include an input module. As the input of the Pipeline, the input module can receive the original image and input the original image into the ISP algorithm module group. Exemplarily, the original image may be the original RAW data.

[0076] In addition, the image processing system may further include an output module for outputting the output results of the ISP algorithm module group. Exemplarily, in the embodiments of the present application, the output results of the AI algorithm module may be in the same output format as the output results of the ISP algorithm module, such as the YUV format.

[0077] In one embodiment, adjusting the weight parameters of the first output result and the second output result may include:

[0078] Adjusting the weight parameters of the first output result and the second output result based on a weight adjustment strategy; the weight adjustment strategy includes a weight value determination rule and a weight value adjustment rule;

[0079] Among them, the weight value determination rule includes determining a preset fixed value according to the acquisition scenario and image usage of the original image, or calculating a dynamically adjusted value based on the original image; the weight value adjustment rule includes sharing weights or per-pixel adjustment for the region of interest.

[0080] Specifically, in the embodiments of the present application, the weight parameters of the first output result and the second output result may be adjusted based on a weight adjustment strategy. Exemplarily, the weight adjustment strategy may include a weight value determination rule and a weight value adjustment rule. Among them, the weight value determination rule may be to determine a preset fixed value according to the acquisition scenario and image usage of the original image, or it may also be to calculate a dynamically adjusted value based on the original image; in addition, the weight value adjustment rule may be to share weights, or it may be per-pixel adjustment for the region of interest.

[0081] Exemplarily, the weight value determination rule may refer to determining whether to use a preset fixed value or a dynamically adjusted value for the weight value. Among them, a preset fixed value (abbreviated as "fixed") may be preset according to the image acquisition scenario and the purpose of image acquisition, or a dynamically adjusted value (abbreviated as "dynamic adjustment") may be obtained by calculating the acquired image. In addition, the weight value adjustment rule may refer to the weight value adjustment method. Sharing weights may mean that a single image uses the same weight value, and per-pixel adjustment for the region of interest may be to perform per-pixel adjustment on the image according to the region of interest (ROI).

[0082] Regarding per-pixel adjustment, it can be understood that different weight values are assigned to different objects, motion states, pixel brightness, pixel saturation, etc. For example, when processing an algorithm that focuses on the face region, the pixels covered by the face region can be found first through a face detection algorithm, and the weight values of these pixels can be adjusted, while a general weight strategy can be used for other non-ROI regions. Similarly, different weight strategies can be used for the moving object region and the background region in the picture, according to the brightness and saturation adjustment strategies, etc. Per-pixel adjustment can also combine the above multiple pixel features to adjust the weight values.

[0083] Optionally, the above weight value determination rule and weight value adjustment rule can be independent, that is, fixed value and dynamic adjustment, image - shared weight and per - pixel adjustment. Combining them can result in four options: ① fixed value + image - shared weight; ② fixed value + per - pixel adjustment; ③ dynamic adjustment + image - shared weight; ④ dynamic adjustment + per - pixel adjustment. Appropriate selection can be made according to actual needs.

[0084] As described above, in the embodiments of the present application, in addition to setting weights in a globally unified manner, weights can also be set in a per - pixel manner, so that different algorithmic logics can be applied to different scenes in the image.

[0085] In some embodiments, the method may further include:

[0086] When the weight value adjustment rule is per - pixel adjustment for the region of interest, a filtering algorithm is used to filter the fusion weights.

[0087] Specifically, when adjusting the weights per - pixel for the ROI, a filtering algorithm can be added to make the transition of weights smooth in the image, so that there will be no sudden change in the final image processing effect. Such filtering can be in the spatial domain or in the temporal domain.

[0088] Exemplarily, the filtering algorithm may include spatial filtering and temporal filtering. In the embodiments of the present application, when using per - pixel adjustment of weight values, there may be a large difference in weight values between adjacent pixels, which is manifested as unnatural and non - smooth transition in the image. For example, when adjusting the weight values for the pixels in the face region, if no filtering is performed, the processing traces on the face may be obvious and the transition may be unnatural. At this time, adding spatial filtering to the weights can make the overall picture effect more coordinated and the transition natural. In addition, in the processing of consecutive image frames (videos), due to the jump of weight values between the front and back frames, sudden changes in the brightness, color, and image style of the front and back frames of the video may also occur. At this time, applying temporal filtering can suppress such sudden changes and make the transition between the front and back frames of the video natural. Optionally, the spatial filtering in the filtering algorithm can be used for the per - pixel (ROI) weight adjustment method, and the temporal filtering can be used for the dynamic adjustment method of weight values.

[0089] In an exemplary embodiment, the image processing system further includes a direct - through branch connected in parallel across the two ends of the series branch. Each branch of the parallel branch includes an AI algorithm module; the fusion weight corresponding to the first output result is the first weight value, and the fusion weight corresponding to the second output result is the second weight value.

[0090] As Figure 4 shown, weighted fusion of the first output result and the second output result is performed using the fusion weights to obtain the output result of the ISP algorithm module group, which may include steps 302 to 304.

[0091] Step 302: Obtain the third output result of the direct path branch. The fusion weight corresponding to the third output result is the third weight value. The sum of the first weight value, the second weight value, and the third weight value is 1.

[0092] Specifically, the direct path branch may refer to an algorithm of simple direct pass (i.e., no processing or simple format conversion). The third output result may refer to a simple direct pass output. The third weight value may refer to assigning a weight to the simple direct pass output. The constraint condition is that the sum of the first weight value, the second weight value, and the third weight value is 1.

[0093] Step 304: Output a weighted average according to the first weight value, the second weight value, and the third weight value to obtain the output result of the ISP algorithm module group.

[0094] Specifically, a weighted average of multiple algorithms can be output according to different weight values, so that the effect is balanced between the AI ISP algorithm and the traditional ISP Pipeline. In the embodiments of the present application, for a single module of the ISP, instead of selecting one from various algorithms, since the node formats between the ISP Pipeline modules are consistent and the image effects are similar or complementary, corresponding weights are assigned to various algorithms, so that the advantages of each algorithm can be aggregated and the output result can be optimized.

[0095] In one embodiment, the first weight value, the second weight value, and the third weight value are all between 0 and 1.

[0096] Specifically, when the weight value of the direct path branch is 1 and the other weight values are all 0, that is, this series of algorithm modules are skipped without processing the image data (the input and output formats are the same) or only performing format conversion (the output format is different from the input format); when the weight value of a certain algorithm module is 1 and the other weight values are all 0, that is, only the output of this algorithm is selected as the output of this series of modules. Exemplarily, the weight values corresponding to the outputs of each algorithm module can be between 0 and 1.

[0097] In the embodiments of the present application, for each parallel algorithm module, a weight is assigned to the result of its output. The results output by each parallel algorithm module match the corresponding weights, and after performing a weighted average, the final output of the module group is obtained. Usually, the weights corresponding to each individual algorithm module take numbers between 0 and 1. For some cases, negative weights can also be taken. The sum of the corresponding weights assigned to the results of each parallel algorithm module is 1.

[0098] In one embodiment, when each algorithm module corresponds to different image processing functions in the same image processing task, the fusion weight includes negative weights and positive weights.

[0099] Specifically, different image processing functions in the same image processing task may refer to certain enhancement or weakening processes performed based on the result of a basic algorithm.

[0100] Taking the negative weight as an example, usually the final result is a certain enhancement or weakening process based on the result of a basic algorithm. In this case, the weight value of the output result of this basic algorithm is 1. The remaining algorithm results are adjusted based on the basic algorithm, and the corresponding weight values of the remaining algorithm results can be positive or negative. Exemplarily, if the remaining algorithm is a noise reduction algorithm (the algorithm result is the noise residual), the noise reduction algorithm usually reduces noise but also reduces clarity. If noise reduction is required, the weight of the noise residual value should be negative, that is, subtract the noise residual; if clarity improvement is required, the weight of the noise residual value should be positive. Similarly, for image contrast enhancement or reduction, saturation enhancement or reduction, warm and cold color tone adjustment, color temperature increase or decrease, etc., the above similar weight strategies can be used. It should be noted that when negative weights and / or positive weights are adopted, the weight values may not be restricted by the total weight value being 1.

[0101] To further illustrate the present application, specific examples are given below for explanation:

[0102] As Figure 5 shown, an end-to-end AI ISP algorithm (AI algorithm module) is connected in parallel with a series branch, and the series branch includes a plurality of ISP algorithm modules connected in series in sequence (i.e., the traditional ISP Pipeline), namely Module 1 to Module N.

[0103] Among them, the input of the ISP Pipeline is the original RAW data, which is respectively used as the input of the AI ISP algorithm and the traditional ISP Pipeline. The output formats of the AI ISP algorithm and the traditional ISP Pipeline can be the same (such as the YUV format). Exemplarily, the output of the AI ISP algorithm is assigned a weight of Weight(Pipeline, AI), and the output of the traditional ISP Pipeline is assigned a weight of Weight(Pipeline, Conventional). The constraint is that Weight(Pipeline, AI) + Weight(Pipeline, Conventional) = 1. If Weight(Pipeline, AI) = 1, then Weight(Pipeline, Conventional) = 0. At this time, the Pipeline is equivalent to only using the end-to-end AI ISP algorithm. If Weight(Pipeline, AI) = 0, then Weight(Pipeline, Conventional) = 1. At this time, the Pipeline is equivalent to only using the traditional ISP Pipeline. Optionally, Weight(Pipeline, AI) and Weight(Pipeline, Conventional) can be between 0 and 1. In this case, the Pipeline outputs the weighted average of the two algorithms according to the weight value, and the effect is balanced between the AI ISP algorithm and the traditional ISP Pipeline.

[0104] As Figure 6 shown, in a complete ISP Pipeline, the AI ISP algorithm can only cover several modules. Taking an example where there are two AI algorithm modules, one traditional ISP algorithm, and one simple pass-through (i.e., no processing or simple format conversion), and a total of four algorithm outputs on one ISP Pipeline node, the two AI algorithms can select inputs from the same node or from different nodes ( Figure 6Shown is the selection of inputs from different nodes. Exemplarily, the output of AI algorithm module AIm1 is assigned a weight of Weight(MultiModule, AIm1), the output of AI algorithm module AIm2 is assigned a weight of Weight(MultiModule, AIm2), the output of the traditional ISP algorithm is assigned a weight of Weight(MultiModule, Conventional), and the simple pass-through output is assigned a weight of Weight(MultiModule, Simple). The constraint is that Weight(MultiModule, AIm1) + Weight(MultiModule, AIm2) + Weight(MultiModule, Conventional) + Weight(MultiModule, Simple) = 1. When Weight(MultiModule, Simple) = 1 and the other weight values are all 0, the image data is not processed by this series of modules (the input and output formats are the same) or only the format is converted (the output format is different from the input format); when one of the weight values is 1 and the other weight values are all 0, only the output of that algorithm is selected as the output of this series of modules; optionally, the weight values corresponding to the outputs of each module can all be between 0 and 1.

[0105] As Figure 7 shown, taking the case where the AI ISP algorithm only covers one module (the processing flow diagram is similar to the above example), the AI algorithm only covers one module and completes the processing task of a single module. Further, taking the case where there are two AI algorithms, one traditional ISP algorithm, and one simple pass-through (i.e., no processing or simple format conversion) with a total of four algorithm outputs on one ISPPipeline node as an example, the output of AI algorithm module AIs1 is assigned a weight of Weight(SingleModule, AIs1), the output of AI algorithm module AIs2 is assigned a weight of Weight(SingleModule, AIs2), the output of the traditional ISP algorithm is assigned a weight of Weight(SingleModule, Conventional), and the simple pass-through output is assigned a weight of Weight(SingleModule, Simple). The constraint is that Weight(SingleModule, AIs1) + Weight(SingleModule, AIs2) + Weight(SingleModule, Conventional) + Weight(SingleModule, Simple) = 1.

[0106] In addition, it is also possible to use only one AI algorithm module in parallel with a simple direct-through algorithm, and the weights of the two branch algorithms can only be selected as 0 or 1, and the weights of the two cannot be 0 at the same time or 1 at the same time. That is, a switch option is provided for the AI ISP algorithm module. When Weight(AI algorithm) = 1, it means that the AI algorithm is selected; when Weight(AI algorithm) = 0, it means skipping the AI algorithm and using the simple direct-through.

[0107] As described above, for a single module of the ISP in the image processing method of the present application, instead of selecting one from various algorithms, since the node formats between ISP Pipeline modules are consistent, and the image effects are similar or complementary, corresponding weights are assigned to various algorithms, so that the advantages of each algorithm can be integrated, and the output result can reach the optimal. In addition, the embodiment of the present application solves the problem that the parameters of the AIISP algorithm are difficult to adjust directionally. Multiple AI neural network algorithms, or multiple sets of training parameters with different styles of a certain neural network framework, can all be applied in the same ISP Pipeline, and the final required image style can be adjusted by simply adjusting the weight values corresponding to each algorithm.

[0108] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown sequentially according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times, and the execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0109] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the following image processing devices can refer to the limitations on the image processing method in the above text, and will not be repeated here.

[0110] In an exemplary embodiment, as Figure 8As shown, an image processing apparatus 800 is provided, which is applied to an image processing system. The image processing system includes an image signal processing (ISP) algorithm module group. The ISP algorithm module group includes multiple algorithm modules with the same or similar image processing effects. The multiple algorithm modules include an AI algorithm module and at least one ISP algorithm module forming a series branch. Among them, the series branch is connected in parallel with one or more AI algorithm modules to form a parallel branch. The apparatus 800 includes:

[0111] An output result acquisition module 801, configured to acquire a first output result and a second output result. The first output result includes the output result of the last ISP algorithm module in the series branch for image processing, and the second output result includes the output results of the AI algorithm modules in each branch of the parallel branch for image processing.

[0112] A weight parameter adjustment module 802, configured to adjust the weight parameters of the first output result and the second output result to obtain the fusion weights corresponding to the first output result and the second output result respectively.

[0113] A weighted fusion module 803, configured to perform weighted fusion on the first output result and the second output result using the fusion weights to obtain the output result of the ISP algorithm module group.

[0114] In one embodiment, the apparatus 800 further includes:

[0115] A raw image input module, configured to receive a raw image and input the raw image into the ISP algorithm module group.

[0116] In one embodiment, the weight parameter adjustment module 802 is configured to adjust the weight parameters of the first output result and the second output result based on a weight adjustment strategy. The weight adjustment strategy includes a weight value determination rule and a weight value adjustment rule. Among them, the weight value determination rule includes determining a preset fixed value according to the acquisition scenario and image usage of the raw image, or obtaining a dynamically adjusted value by calculating based on the raw image. The weight value adjustment rule includes a shared weight or a per-pixel adjustment for the region of interest.

[0117] In one embodiment, the apparatus 800 further includes:

[0118] A filtering module, configured to, when the weight value adjustment rule is a per-pixel adjustment for the region of interest, filter the fusion weights using a filtering algorithm.

[0119] In one embodiment, when each algorithm module corresponds to different image processing functions in the same image processing task, the fusion weights include negative weights and positive weights.

[0120] In one embodiment, the image processing system further includes a direct path branch connected in parallel across the series branch, and each branch of the parallel branch includes an AI algorithm module; the fusion weight corresponding to the first output result is a first weight value, and the fusion weight corresponding to the second output result is a second weight value;

[0121] A weighted fusion module 803, configured to obtain a third output result of the direct path branch, where the fusion weight corresponding to the third output result is a third weight value; wherein, the sum of the first weight value, the second weight value, and the third weight value is 1; and output a weighted average according to the first weight value, the second weight value, and the third weight value to obtain the output result of the ISP algorithm module group.

[0122] In one embodiment, the first weight value, the second weight value, and the third weight value are all between 0 and 1.

[0123] Each module in the above image processing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0124] In an exemplary embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 9As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0125] Those skilled in the art can understand that Figure 9 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0126] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0127] In an exemplary embodiment, a chip is provided. The chip includes a programmable logic circuit and / or program instructions, and when the chip runs, the steps of the above method are implemented.

[0128] In some embodiments, the chip may be an ISP chip.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0130] In one embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps in the above method embodiments.

[0131] Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, Resistive Random Access Memory (ReRAM), Magnetoresistive Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene memory, etc. Volatile memory can include Random Access Memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, Artificial Intelligence (AI) processors, etc., without limitation.

[0132] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0133] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. An image processing method, characterized in that: Applied to an image processing system, the image processing system includes an image signal processing ISP algorithm module group, the ISP algorithm module group includes multiple algorithm modules with the same or similar image processing effects, the multiple algorithm modules include an AI algorithm module and at least one ISP algorithm module constituting a series branch, wherein the series branch is connected in parallel with one or more of the AI ​​algorithm modules to form a parallel branch; the method includes: Obtain a first output result and a second output result; the first output result includes an output result of the image processing performed by the last ISP algorithm module in the series branch, and the second output result includes an output result of the image processing performed by the AI ​​algorithm modules in each branch in the parallel branch; Adjusting weight parameters of the first output result and the second output result to obtain fusion weights corresponding to the first output result and the second output result respectively; The first output result and the second output result are weightedly fused using the fusion weight to obtain the output result of the ISP algorithm module group.

2. The method according to claim 1, characterized in that The method further comprises: An original image is received, and the original image is sent to the ISP algorithm module group.

3. The method according to claim 2, characterized in that The adjusting weight parameters of the first output result and the second output result includes: Adjusting weight parameters of the first output result and the second output result based on a weight adjustment strategy; the weight adjustment strategy includes a weight value determination rule and a weight value adjustment rule; Among them, the weight value determination rule includes determining a preset fixed value based on the acquisition scene and image purpose of the original image, or calculating a dynamic adjustment value based on the original image; the weight value adjustment rule includes sharing weights or pixel-by-pixel adjustment for the area of ​​interest.

4. The method according to claim 3, characterized in that The method further comprises: When the weight value adjustment rule is the pixel-by-pixel adjustment for the region of interest, a filtering algorithm is used to filter the fusion weight.

5. The method according to any one of claims 1 to 4, characterized in that: When each of the algorithm modules corresponds to a different image processing function in the same image processing task, the fusion weight includes a negative weight and a positive weight.

6. The method according to any one of claims 1 to 4, characterized in that: The image processing system further includes a through branch connected in parallel at both ends of the series branch, and each branch of the parallel branch includes one of the AI ​​algorithm modules; the fusion weight corresponding to the first output result is a first weight value, and the fusion weight corresponding to the second output result is a second weight value; The adopting the fusion weight to weightedly fuse the first output result and the second output result to obtain the output result of the ISP algorithm module group includes: Obtaining a third output result of the through branch, wherein the fusion weight corresponding to the third output result is a third weight value; wherein the sum of the first weight value, the second weight value and the third weight value is 1; The output result of the ISP algorithm module group is obtained by outputting a weighted average according to the first weight value, the second weight value and the third weight value.

7. The method according to claim 6, characterized in that The first weight value, the second weight value and the third weight value are all between 0 and 1.

8. An image processing device, characterized in that: Applied to an image processing system, the image processing system includes an image signal processing ISP algorithm module group, the ISP algorithm module group includes multiple algorithm modules with the same or similar image processing effects, the multiple algorithm modules include an AI algorithm module and at least one ISP algorithm module constituting a series branch, wherein the series branch is connected in parallel with one or more of the AI ​​algorithm modules to form a parallel branch; the device includes: An output result acquisition module, used to acquire a first output result and a second output result; the first output result includes an output result of the image processing performed by the last ISP algorithm module in the series branch, and the second output result includes an output result of the image processing performed by the AI ​​algorithm modules in each branch in the parallel branch; A weight parameter adjustment module, used to adjust the weight parameters of the first output result and the second output result, and obtain the fusion weights corresponding to the first output result and the second output result respectively; A weighted fusion module is used to use the fusion weight to perform weighted fusion on the first output result and the second output result to obtain the output result of the ISP algorithm module group.

9. An image processing system, characterized in that: It includes an image signal processing ISP algorithm module group, the ISP algorithm module group includes multiple algorithm modules with the same or similar image processing effects, the multiple algorithm modules include an AI algorithm module and at least one ISP algorithm module constituting a series branch, wherein the series branch is connected in parallel with one or more of the AI ​​algorithm modules to form a parallel branch; Wherein, the output result of the ISP algorithm module group is obtained by using the image processing method described in any one of claims 1 to 7.

10. The image processing system according to claim 9, characterized in that: Each branch of the parallel branches includes one AI algorithm module.

11. The image processing system according to claim 9, characterized in that: The image processing system also includes an input module and an output module; The input module is used to receive the original image and send the original image to the ISP algorithm module group; The output module is used to output the output results of the ISP algorithm module group.

12. The image processing system according to claim 9, characterized in that: The image processing system further includes a through branch connected in parallel at both ends of the series branch.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A chip, characterized in that: The chip comprises a programmable logic circuit and / or program instructions, and the steps of the method according to any one of claims 1 to 7 are implemented when the chip is executed.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images