Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and the method is used for determining a truncation optimization value of a first pixel after a to-be-processed image is subjected to truncation processing, and the first pixel is any pixel in the to-be-processed image; the method comprises the following steps: determining a non-cut value of a first pixel in a to-be-processed image before the to-be-processed image is subjected to cut processing; according to the parameters of the first truncation processing, calculating a truncation theoretical value of the non-truncation value after the truncation processing; fusing the non-cut value and the cut theoretical value to obtain an expected non-cut value of the first pixel; for the expected non-truncated value, calculating a non-linear compressed pixel value of the first pixel after the non-linear compression processing; and performing truncation processing on the non-linear compressed pixel value according to the parameter of the second truncation processing to obtain a truncation optimized value of the first pixel. According to the method, the color rendition intensity of the highlight interval can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Image Signal Processor (ISP) is widely used in image processing fields such as mobile phones, SLRs, security, and autonomous driving. A good ISP can greatly determine the image quality. When the ISP performs digital signal processing, due to the gain control adjustment in the digital signal processing process, when data saturation occurs in the calculation of gain-related modules, the common method to deal with the saturation problem is data clipping, but data clipping can easily lead to the loss of image information, which is reflected in the image quality, causing the highlight saturated area to lose color. Summary of the invention

[0003] The present application provides an image processing method, device, electronic device and storage medium for improving the color restoration intensity in a highlight range.

[0004] An embodiment of the present application provides an image processing method for determining a truncation optimization value of a first pixel of an image to be processed after truncation processing, wherein the first pixel is any pixel in the image to be processed; the method comprises: determining, for a first pixel in the image to be processed, an untruncated value of the first pixel before the image to be processed is truncation processed; calculating a truncation theoretical value of the untruncated value after truncation processing according to parameters of a first truncation processing; fusing the untruncated value and the truncation theoretical value to obtain an expected untruncated value of the first pixel; calculating a nonlinear compressed pixel value of the first pixel after nonlinear compression processing for the expected untruncated value; and truncation processing the nonlinear compressed pixel value according to parameters of a second truncation processing to obtain a truncation optimization value of the first pixel.

[0005] The embodiment of the present application provides an image processing device, which is used to determine the truncation optimization value of the first pixel of an image to be processed after truncation processing, wherein the first pixel is any pixel in the image to be processed; the device includes: a determination module, which is used to determine the untruncated value of the first pixel in the image to be processed before the image to be processed is truncation processed; a first truncation module, which is used to calculate the truncation theoretical value of the untruncated value after truncation processing according to the parameters of the first truncation processing; a fusion module, which is used to fuse the untruncated value and the truncation theoretical value to obtain the expected untruncated value of the first pixel; a nonlinear compression module, which is used to calculate the nonlinear compressed pixel value of the first pixel after nonlinear compression processing according to the expected untruncated value; and a second truncation module, which is used to truncate the nonlinear compressed pixel value according to the parameters of the second truncation processing to obtain the truncation optimization value of the first pixel.

[0006] An embodiment of the present application provides an electronic device, including: one or more processors; a memory, on which one or more programs are stored. When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the image processing methods in the embodiments of the present application.

[0007] An embodiment of the present application provides a storage medium, which stores a computer program. When the computer program is executed by a processor, any image processing method in the embodiment of the present application is implemented.

[0008] According to the image processing method of the embodiment of the present application, the untruncated value before truncation and the truncated theoretical value after truncation of any pixel in the processed image can be fused, and the expected untruncated value of the pixel obtained by fusion can be subjected to nonlinear compression processing to obtain a nonlinear compressed pixel value, and then the nonlinear compressed pixel value can be subjected to truncation processing to obtain a truncated optimized value of the first pixel. According to this method, the untruncated value and the truncation theoretical value of any pixel in the processed image are fused to realize the correction of the untruncated value using the truncation theoretical value, which is beneficial to the protection of the highlight interval, and reduces the impact of the nonlinear problem of the near-saturation interval data on the restoration of the highlight area, and nonlinear compression processing is performed based on the expected untruncated value obtained by fusion, and the color restoration intensity of the highlight interval is improved through the nonlinear compression intensity control method to solve the problem of insufficient color restoration intensity.

[0009] With regard to the above embodiments and other aspects of the present application and their implementation, further description is provided in the accompanying drawings, detailed description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of an image processing method according to an embodiment of the present application is shown.

[0011] Figure 2a This is a schematic diagram of the positional relationship between a highlight recovery module and multiple gain modules according to an example of an embodiment of the present application.

[0012] Figure 2b This is a schematic diagram of the positional relationship between a highlight recovery module and a plurality of gain modules according to another example of an embodiment of the present application.

[0013] Figure 3 A schematic diagram of the RAW format of the RGGB sensor according to an embodiment of the present application is shown.

[0014] Figure 4 A schematic diagram showing the structure of an HLR module combined with a gain module in an embodiment of the present application.

[0015] Figure 5 A detailed flowchart showing an image processing method according to an exemplary embodiment of the present application.

[0016] Figure 6 A schematic diagram showing the structure of an image processing device according to an embodiment of the present application is shown.

[0017] Figure 7 A structural diagram showing an exemplary hardware architecture of a computing device capable of implementing the method and apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0019] The present application will be described more fully below with reference to the accompanying drawings, but the embodiments shown may be embodied in different forms, and the present application should not be construed as being limited to the embodiments set forth below. On the contrary, the purpose of providing these embodiments is to make the present application thorough and complete, and to enable those skilled in the art to fully understand the scope of the present application.

[0020] The accompanying drawings of the embodiments of the present application are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification. Together with the detailed embodiments, they are used to explain the present application, and do not constitute a limitation of the present application. By describing the detailed embodiments with reference to the accompanying drawings, the above and other features and advantages will become more obvious to those skilled in the art.

[0021] The present application may be described with reference to plan views and / or cross-sectional views by means of ideal schematic views of the present application. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.

[0022] The terms used in this application are only used to describe specific embodiments and are not intended to limit the application. The term "and / or" as used in this application includes any and all combinations of one or more related enumerated items. The singular forms "one" and "the" as used in this application are also intended to include plural forms, unless the context clearly indicates otherwise. The terms "including", "made of..." as used in this application specify the presence of features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0023] Unless otherwise defined, all terms (including technical and scientific terms) used in this application have the same meaning as those commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this application.

[0024] ISP can receive digital signals input by sensors and perform a series of pipeline processing on digital images, including lens and sensor-related corrections, noise removal, image tone restoration and enhancement, detail preservation and enhancement, etc. Good ISP performance greatly determines the image quality. ISP is widely used in image processing fields such as mobile phones, SLRs, security, and autonomous driving.

[0025] In some scenarios, digital images are processed sequentially module by module in the ISP pipeline. Due to hardware implementation constraints, the processing modules have input and output bit width restrictions. Due to the gain control adjustment in the digital signal processing process, the calculation of gain-related modules usually includes data saturation problems. The common method to deal with the saturation problem is data truncation, which causes the loss of image information, which is reflected in the image quality and causes the highlight saturated area to lose color. In addition, due to the nonlinear characteristics of the module (such as the display module), the digital signal has a large nonlinearity in the near-saturation area, and the digital signals of different color channels have inconsistent nonlinearity in the near-saturation range, further affecting the accuracy of color restoration related functions.

[0026] In the related art, when performing highlight recovery (HLR) on digital signals to solve the impact of saturation truncation operation on image quality, the brightness adjustment of the module has non-linear characteristics, which increases the difficulty of implementing HLR. Therefore, the HLR scheme in the related art usually has the problem of insufficient color restoration intensity in the highlight area and the color cast problem of the restored color caused by the non-linear characteristics of the module.

[0027] The embodiments of the present application provide an image processing method, device, electronic device and storage medium for improving the color restoration intensity in a highlight range and solving the problem of insufficient color restoration intensity.

[0028] The image processing method according to the embodiment of the present application can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a computing device, or a vehicle-mounted device; it can also be a product device including an image processor such as a SLR camera, a surveillance camera, or a vehicle-mounted camera. The image processing method can be implemented by a processor calling a computer-readable program instruction stored in a memory. The server can include an independent physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.

[0029] In the description of the following embodiments, the image processing method of the embodiment of the present application may correspond to an image processing device in an image processor, and the image processing device may have the same or equivalent functions as the HLR module in the image processor (ISP). The image processing device or the HLR module in the image processor can be used to implement the image processing method of the embodiment of the present application.

[0030] First, refer to Figure 1 An embodiment of the present application provides an image processing method, which is used to determine the truncation optimization value of the first pixel of the image to be processed after truncation processing. The first pixel is any pixel in the image to be processed. The image processing method includes but is not limited to the following steps.

[0031] S110, determining, for a first pixel in the image to be processed, an un-truncated value of the first pixel before the image to be processed is truncated.

[0032] In this step, the truncation process is used to control the bit width of the first pixel in the image to be processed within the allowed bit width range. In the description of the following embodiment, the un-truncation value of the first pixel can be expressed as P i,j,unclip .

[0033] For example, the un-truncated value may be a gain process or a color correction process performed on the first pixel in the image to be processed before the truncation process.

[0034] S120, calculating a truncation theoretical value of the untruncated value after the truncation processing according to the parameters of the first truncation processing.

[0035] In this step, the parameter of the first truncation process is used to indicate the lower limit value and the upper limit value of the first bit width range corresponding to the truncation process of the untruncated value. The lower limit value and the upper limit value of the first bit width range can be called: the upper truncation threshold value and the lower truncation threshold value of the first bit width range. In the description of the following embodiment, the truncation theoretical value can be expressed as P i,j,clip .

[0036] S130: Fusing the untruncated value and the truncated theoretical value to obtain an expected untruncated value of the first pixel.

[0037] In the above steps, due to the nonlinear characteristics of the display screen module (hereinafter referred to as the module), the untruncated value of each pixel in the image to be processed, especially in the near-saturation interval, has more significant nonlinear characteristics. The corresponding pixel value calculated by directly performing image processing (gain processing or color correction processing) may be inaccurate. Therefore, the untruncated value of the first pixel can be corrected by using the truncated theoretical value of the first pixel through pixel value fusion, which is beneficial to protecting the highlight interval and reducing the impact of the nonlinear problem of the near-saturation interval data on the recovery of the highlight area, thereby helping to improve the color restoration of the highlight area.

[0038] In this step, the fusion may be weighted fusion, and the fusion weight value may be a dynamic weight value. Exemplarily, the fusion weight value is a value greater than or equal to 0 and less than or equal to 1. It should be understood that the embodiment of the present application does not specifically limit the weight value of the untruncated value and the weight value of the truncated theoretical value. Specifically, a fixed weight may be configured or a weight value may be set according to the untruncated value P of the first pixel. i,j,unclip and the theoretical value P of the first pixel i,j,clip Perform adaptive weight design.

[0039] S140: Calculate the nonlinear compressed pixel value of the first pixel after the nonlinear compression process for the expected untruncated value.

[0040] In this step, the color restoration intensity of the highlight interval can be effectively controlled by processing using a nonlinear compression method.

[0041] S150, performing truncation processing on the nonlinear compressed pixel value according to the parameters of the second truncation processing to obtain a truncation optimization value of the first pixel.

[0042] Exemplarily, the nonlinear compressed pixel value of the first pixel is truncated according to the bit width range allowed to be output by the HLR module in the ISP to obtain the truncated optimized value of the first pixel and the truncated optimized value of each pixel in the image to be processed, thereby forming the processing result of the image to be processed.

[0043] According to the image processing method of the embodiment of the present application, the untruncated value before truncation and the truncated theoretical value after truncation of any pixel in the processed image can be fused, and the expected untruncated value of the pixel obtained by fusion can be subjected to nonlinear compression processing to obtain a nonlinear compressed pixel value, and then the nonlinear compressed pixel value can be subjected to truncation processing to obtain a truncated optimized value of the first pixel. According to this method, the untruncated value and the truncation theoretical value of any pixel in the processed image are fused to realize the correction of the untruncated value using the truncation theoretical value, which is beneficial to the protection of the highlight interval, and reduces the impact of the nonlinear problem of the near-saturation interval data on the restoration of the highlight area, and nonlinear compression processing is performed based on the expected untruncated value obtained by fusion, and the color restoration intensity of the highlight interval is improved through the nonlinear compression intensity control method to solve the problem of insufficient color restoration intensity.

[0044] In some embodiments, the image processing method can be applied to an image processor, and before calculating the truncated theoretical value of the untruncated value after truncation processing according to the parameters of the first truncation processing, the image to be processed is also subjected to the first image processing; step S110 may specifically include: S01, determining the current pixel value of the first pixel that has not been truncated and has not been subjected to the first image processing; S02, calculating the first processed value of the first pixel of the image to be processed after the first image processing according to the parameters of the first image processing, as the untruncated value of the first pixel.

[0045] In this embodiment, the current pixel value may be a pixel value that has not been subjected to truncation processing and first image processing. If the first image processing is performed on the image to be processed according to the parameters of the first image processing, the current pixel value of the first pixel in the image to be processed may be converted into the first processing value. Exemplarily, if the first image processing is gain processing, the parameters of the first image processing are gain parameters used in the gain processing.

[0046] Step S150 may specifically include: S11, calculating a decoupling value of the nonlinear compressed pixel value that eliminates the influence of the first image processing according to the parameters of the first image processing; S12, performing truncation processing on the decoupling value of the nonlinear compressed pixel value that eliminates the influence of the first image processing according to the parameters of the second truncation processing, to obtain a truncation optimized value of the first pixel.

[0047] In step S11, eliminating the influence of the first image processing is an inverse operation of the first image processing, that is, there is a corresponding relationship between the two, and eliminating the influence of the first image processing can offset the effect of the first image processing. For example, if the first image processing needs to be multiplied by the parameter of the first image processing, then the first image processing parameter needs to be divided when eliminating the influence of the first image processing; in step S12, the parameter of the second truncation processing is used to indicate: the decoupling value of the nonlinear compressed pixel value that has eliminated the influence of the first image processing, the lower limit value and the upper limit value of the second bit width range corresponding to the truncation processing, and the lower limit value and the upper limit value of the second bit width range can be called: the upper truncation threshold value and the lower truncation threshold value of the second bit width range.

[0048] In this embodiment, if in the HLR module, the first image processing is performed on the first pixel that has not been truncated and has not been subjected to the first image processing, then before the HLR module outputs the processing result, the influence of the first image processing can be eliminated first. This is beneficial for decoupling the first image processing in the HLR module in the ISP from the first image processing of subsequent modules, and does not affect the first image processing of subsequent modules.

[0049] In some embodiments, the first image processing is one or more gain processing, and the parameters of the first image processing are one or more gain parameters.

[0050] In this embodiment, the product of the current pixel value of the first pixel and the parameter corresponding to each gain process is calculated as the un-truncated value of the first pixel.

[0051] Exemplarily, the gain processing includes but is not limited to at least one of Lens Shading Correction (LSC), Digital Gain (DG) and Auto White Balance (AWB), and the gain parameters include but are not limited to: a gain parameter corresponding to the LSC processing, a gain parameter corresponding to the DG processing and a gain parameter corresponding to the AWB processing. Each gain processing can be performed by a corresponding gain module in the image processor, for example, the image processor includes but is not limited to: at least one of an LSC module that performs LSC processing, a DG module that performs DG processing and an AWB module that performs AWB processing.

[0052] Through the specific steps of the above-mentioned step S110, the untruncated value of the first pixel after gain processing can be calculated for the current pixel value of the first pixel that has not been processed by gain and has not been processed by truncation, and in step S120, the untruncated value of the first pixel that has been processed by gain (without truncation) can be specifically calculated for the truncation theoretical value after truncation.

[0053] Specifically, in the above step S11, a decoupling value for eliminating the influence of the gain processing may be calculated for the nonlinear compressed pixel value. For example, in the above embodiment, when the gain processing is performed, if the pixel value of the first pixel is multiplied by the total gain parameter, then in the subsequent calculation of the decoupling value for eliminating the influence of the gain processing, the nonlinear compressed pixel value (the pixel value in the RAW domain) may be divided by the total gain parameter value to eliminate the influence of the gain processing performed on the first pixel, wherein the total gain parameter value is the product of the gain parameters of each gain processing performed on the current pixel value of the first pixel that has not been subjected to the gain processing and the truncated processing.

[0054] In some embodiments, one or more gain parameters include non-pixel level gain parameters; before the above step S12, the method also includes: interpolating the non-pixel level gain parameters to obtain interpolated pixel level gain parameters; and updating one or more gain parameters according to the pixel level gain parameters.

[0055] Exemplarily, the gain parameter of the LSC module, the gain parameter of the DG module, and the gain parameter of the AWB module may be at a pixel level or a block level, where the level is used to indicate: the smallest processing unit when performing a gain operation. Taking the pixel level as an example, it is used to indicate that the smallest processing unit when performing a gain operation is a pixel.

[0056] As a specific example, in some scenarios, the gain parameter of the LSC module can selectively use the meshgrid gain as input (for example, with the grid as the minimum processing unit). When calculating the first processing value of the first pixel of the image to be processed after gain processing, that is, before performing the gain operation on the first pixel in the image to be processed, the gain parameter of the LSC module can be interpolated to obtain the pixel-level gain parameter, and the interpolated pixel-level gain parameter is used as the updated gain parameter of the LSC module; for another example: when the AWB gain parameter is a frame-level or block-level gain parameter, the gain parameter of the AWB module can also be interpolated to obtain the pixel-level gain parameter before the first pixel in the image to be processed, and then the interpolated pixel-level gain parameter is used as the updated gain parameter of the AWB module.

[0057] In this embodiment, before performing a gain operation on the first pixel in the image to be processed, it is ensured that the gain parameters of each gain module are gain parameters at the pixel level. The most basic unit of a digital image is a pixel. Performing a gain operation on any pixel in the image to be processed using a gain parameter at the pixel level is beneficial to improving the accuracy of the gain effect.

[0058] In some embodiments, when the first image processing includes lens shading correction, digital gain and automatic white balance, the above step S12 may specifically include:

[0059] The un-truncated value of the first pixel is calculated using the following expression (1):

[0060] P i,j,unclip =P i,j ×gain lsc ×gain digital ×gain awb (1)

[0061] Among them, P i,j is the current pixel value of the first pixel that has not been subjected to the truncation process and the first image processing, P i,j,unclip is the uncut value of the first pixel, gain lsc 、gain digital and gain awb They are the gain parameters corresponding to lens shading correction, digital gain, and automatic white balance, respectively. i,j,unclip Untruncated value.

[0062] Specifically, gain lsc is the gain parameter corresponding to LSC processing. digital is the gain parameter corresponding to DG processing. awb is the gain parameter of the corresponding module when the gain processing is AWB processing; where P i,j and P i,j,unclip In scalar value form.

[0063] In the embodiment of the present application, the current pixel value of any pixel in the image to be processed that has not been subjected to the truncation process and the first image processing can be defined as P i,j , where the index (i, j) represents the two-dimensional spatial position of this pixel in the current frame, and the value calculated by the above expression (1) is used as the untruncated value of the first pixel. i,j,unclip It can be expressed as: i,j Compensation gain lsc 、gain digital and gain awb The untruncated value of the first pixel that has not been truncated, in actual applications, the digital signal corresponding to the untruncated value of the first pixel can be represented by a high bit width.

[0064] In some embodiments, when the bit width is limited, the untruncated value of the first pixel may be appropriately truncated according to a set bit width range. Specifically, in the above step S120, the truncated theoretical value of the untruncated value after the truncation process is calculated according to the parameters of the first truncation process, which may specifically include:

[0065] The untruncated value is truncated using the following expression (2):

[0066] P i,j,clip =clip(P i,j,unclip ,0,wl1) (2)

[0067] Among them, clip(.) represents the first truncation processing, P i,j,clip represents the calculated theoretical value of truncation, 0 and wl1 represent the lower limit value (also called the lower truncation threshold value) and upper limit value (also called the upper truncation threshold value) of the first bit width range used in the truncation process, and the upper limit value of the first bit width range is the maximum bit width supported by the gain processing module corresponding to the specified gain processing, and the specified gain processing is any gain processing among multiple gain processing. In this expression, P i,j,clip In scalar value form.

[0068] As an example, wl1 in the above expression (2) can be the maximum bit width of the last gain-related module that the HLR module is expected to process. For example, assuming that the above three gain modules are sorted according to the LSC module, the DG module and the AWB module, the last gain-related module that the HLR module is expected to process can be preset, for example, it can be set to any one of the LSC module, the DG module and the AWB module. It should be understood that the sorting of the above three gain modules is only a schematic illustration, and in actual application scenarios, the sorting of the above three gain modules is not specifically limited.

[0069] In this embodiment, each pixel in the processed image can be truncated according to the first bit width range to obtain a truncated theoretical value of each pixel; the upper limit value of the first bit width range is the upper limit value of the bit width range of the pre-specified gain module in all gain modules.

[0070] In some embodiments, before calculating the truncated theoretical value of the un-truncated value after truncation processing according to the parameters of the first truncation processing in step S120, the image to be processed is also subjected to the first image processing; step S110 may specifically include: for the first pixel in the image to be processed, determining the second processed value of the first pixel of the image to be processed that has undergone the first image processing but has not undergone the truncation processing, as the un-truncated value of the first pixel.

[0071] As an example, in the case where the first image processing is one or more gain processings, when the HLR module is located after any gain-related module, it means that the image to be processed is input into the HLR module after gain processing by at least one gain module. At this time, the untruncated value of the first pixel in the image to be processed is: the second processed value of the first pixel of the image to be processed that has been processed by the first image but not truncated.

[0072] In this example, when the HLR module is located before all gain modules, it means that the image to be processed that has not been processed by the first image and has not been truncated is input into the HLR module, that is, the untruncated value of the first pixel in the image to be processed is: the current pixel value of the first pixel that has not been truncated and has not been processed by the first image.

[0073] In order to better understand the position relationship between the HLR module and each gain module in the ISP pipeline, the following Figure 2a and Figure 2b , exemplarily illustrating the implementation scheme of the HLR module corresponding to the image processing method of an embodiment of the present application in the ISP pipeline. Figure 2a A schematic diagram of the positional relationship between an HLR module and a plurality of gain modules provided as an example in an embodiment of the present application; Figure 2b A schematic diagram of the positional relationship between an HLR module and a plurality of gain modules provided in another example of an embodiment of the present application.

[0074] like Figure 2a and Figure 2b As shown, it is assumed that the ISP pipeline includes an LSC module 210, a DG module 220 and an AWB module 230 in sequence, as shown in FIG. Figure 2a As shown, the LSC module 210 is the first gain module, and the HLR module 200 can be located before the LSC module 210, that is, the HLR module 200 can be located before the first gain processing module in the ISP pipeline; Figure 2b As shown, the HLR module 200 is located between the LSC module 210 and the DG module 220, that is, the HLR module 200 can be located between any two gain processing modules. For example, in other examples, the HLR module 200 can also be located between the DG module 220 and the AWB module 230, or can be located after the AWB module 230.

[0075] In the embodiment of the present application, the HLR module can be located before all gain modules (LSC module, DG module and AWB module), or the HLR module can be located between multiple gain-related modules. The positional relationship between the HLR module and each gain module can be customized according to actual needs, and the embodiment of the present application does not make specific limitations.

[0076] In some embodiments, before calculating the truncated theoretical value of the untruncated value after truncation according to the parameters of the first truncation processing in step S120, the image to be processed is further subjected to the first image processing, and the first image processing includes multiple gain processing; the above step S110 may specifically include:

[0077] S31, for the first pixel in the image to be processed, determine the current pixel value of the first pixel of the image to be processed that has undergone N1 gain processing but has not undergone truncation processing; N1 is an integer greater than or equal to 1; S32, according to the gain parameters corresponding to the N2 gain processing, calculate the third processed value of the first pixel of the image to be processed after the N2 gain processing as the untruncated value of the first pixel; wherein N2 is an integer greater than or equal to 1, and the N1 gain processing and the N2 gain processing are different gain processing.

[0078] The above-mentioned step S150 may specifically include: calculating a decoupling value for the nonlinear compressed pixel value that eliminates the influence of the N2 gain processings according to the gain parameters corresponding to the N2 gain processings; and performing truncation processing on the decoupling value of the nonlinear compressed pixel value that has eliminated the influence of the N2 gain processings according to the parameters of the second truncation processing to obtain the truncation optimization value of the first pixel.

[0079] In this embodiment, when the first image processing includes multiple gain processings, each gain processing corresponds to a gain module, and the HLR module can be combined with one or more gain modules.

[0080] For example, when the HLR module is combined with the LSC module, after the LSC module performs the corresponding gain processing on the image to be processed, the truncation operation may not be performed, but the image to be processed after the LSC processing is used as the input of the HLR module. In this case, there is no need to perform the LSC processing again in the above step S110, that is, the gain in the above expression (1) is lsc The value is 1, and when the influence of the gain processing is eliminated in the above step S150, there is no need to eliminate the influence of the LSC processing, that is, when the influence of the gain processing is eliminated, the corresponding gain lsc The value is also 1.

[0081] In order to better understand the implementation of the merging operation of the HLR module and the gain module in the ISP pipeline, the following Figure 4 The processing method of combining the HLR module and the gain module is schematically described.

[0082] Figure 4 A schematic diagram of a structure in which an HLR module and a gain module are combined according to an embodiment of the present application. Figure 4 and Figure 2a and Figure 2bThe same or equivalent modules have the same functions. Figure 4 In the embodiment, the HLR module 200 can be combined with the LSC module 210 as an integral module. In this case, the image to be processed received by the HLR module 200 is the image output by the LSC module 210. When the two are combined as an integral module, the gain parameter gain of the LSC module 210 is lsc The value of is 1. When the influence of the gain processing of the LSC module 210 is removed later, the gain parameter gain of the LSC module 210 is lsc The value of is also 1, that is, no pre-compensation of the corresponding gain module is required.

[0083] In an embodiment of the present application, the HLR module can be independent of the gain module in the ISP pipeline, or it can be combined with at least one gain module in the ISP pipeline as an integral module. For example, if it is combined with a certain gain module, the gain parameter of the gain module needs to be set to 1.

[0084] In the embodiment of the present application, one or more HLR modules may be included in the ISP pipeline. For example, in the ISP pipeline, multiple HLR modules may be implemented in the RAW domain and the RGB domain. For any HLR module, the HLR module may be located before all gain-related modules in the ISP pipeline, or between multiple gain-related modules in the ISP pipeline. Any HLR module may also be combined with one or more gain modules into an integral module. The specific configuration may be based on actual needs, and the embodiment of the present application does not make specific limitations.

[0085] In some embodiments, the image to be processed is an image in a non-red, green, and blue color mode, and step S110 may specifically include: converting the image to be processed in a non-red, green, and blue color mode into an image to be processed in a red, green, and blue color mode; for the first pixel in the image to be processed in the red, green, and blue color mode, determining the un-truncated value of the first pixel of the image to be processed in the red, green, and blue color mode before truncation processing.

[0086] In this embodiment, the image to be processed is collected by an image sensor and input into an image processor for image processing; when the image sensor is an image sensor of a non-red, green, and blue color mode, the image to be processed is an image of a non-red, green, and blue color mode, so it is necessary to first select a corresponding conversion method according to the type of the image sensor of the non-red, green, and blue color mode, convert the image to be processed of the non-red, green, and blue color mode into an image to be processed based on the red, green, and blue color mode, and then determine the un-truncated value of each pixel in the image to be processed based on the red, green, and blue color mode obtained by conversion.

[0087] In some embodiments, the image sensor may be a red-green-green-blue (RED Green Green Blue, RGGB) based sensor or other types of sensors. Other types of sensors may, for example, include at least one of: a blue-green-green-red (Blue Green Green RED, BGGR) based sensor, a green-red-blue-green (Green RED BlueGreen, GRBG) based sensor, and a green-blue-red-green (Green Blue RED Green, GBRG) based sensor.

[0088] In some embodiments, the image sensor may also be a sensor of other special styles, rather than an RGB-based sensor. Such other special-style sensors may include, but are not limited to: a sensor based on red, white, white, and blue (Red Clear Clear Blue, RCCB), a sensor based on red, white, white, and blue (Red Clear Clear Clear, RCCC), a sensor based on red, yellow, yellow, and cyan (Red Yellow Yellow Cyan, RYYCy), and a sensor based on red, green, blue, and infrared (RED Green Blue Infrared, RGB-IR). Among them, C stands for Clear, which is a white filter.

[0089] In some embodiments, the image sensor may further include: other sensor types including three channels or more channels that can correctly restore the color of an object after proper processing.

[0090] As an example, for sensors whose partial channels are transmitted through, such as RCCB and RCCC, and for sensors whose partial channels are replaced with other color channels, such as RYYCy, the C channel and the Y channel can be treated as equivalent to the G channel in RGGB; for RGB-IR, for example, a sensor in RGB-IR 4x4 mode or a sensor in 2x2 sensor mode can be included, and the corresponding pixel-level RGB three-channel values ​​can be calculated by selecting the corresponding color interpolation method or color correction method, and then subsequent steps can be executed.

[0091] The image processing method in the embodiment of the present application can be applicable to RGB-based sensors, and can also support other special styles such as RCCB, RCCC, RYYCy and RGB-IR sensors through format conversion, thereby improving the flexibility of the image processing method in the embodiment of the present application and the scope of applicability to sensor styles.

[0092] In some embodiments, the image to be processed is an image in an unprocessed format, and the above step S130 may specifically include: performing demosaicing on the untruncated value to obtain a first demosaiced value; performing demosaicing on the truncated theoretical value to obtain a second demosaiced value; and fusing the first demosaiced value and the second demosaiced value to obtain an expected untruncated value of the first pixel.

[0093] In this step, the unprocessed (RAW) image to be processed is the image (or the image signal corresponding to the image to be processed) directly output after the image sensor samples the captured light source signal through photoelectric conversion and analog-to-digital conversion (ADC). It is unprocessed data and can be used to represent the intensity of various lights received by the sensor. The original meaning of RAW is unprocessed, which means that the image signal corresponding to the image to be processed is lossless, and the pixel value of the image to be processed contains the original color information of the image.

[0094] In the embodiment of the present application, compared with RGB images, images in RAW format can ensure the highest image quality and details, and images in RAW format have a wider dynamic range, richer image formats and color gamut.

[0095] Figure 3 This is a schematic diagram of the RAW format of the RGGB sensor provided in the embodiment of the present application. Figure 3 As shown, assuming that the pixel size of the RGGB sensor is 4*4 (resolution is 4*4), then the sensor has 4*4 photosites, and each photosite can be a transistor. Figure 3 It can be seen that the G component is twice the B and R components because the human eye is more sensitive to green, so the weight of the G component at the photosite is increased and the sampling of green information is increased.

[0096] In the embodiments of the present application, the images to be processed output by various sensors need to be input to the ISP, and the ISP performs pipeline processing such as lens and sensor-related correction, noise removal, image tone restoration and enhancement, detail preservation and enhancement on the input images to be processed to ensure the corresponding image quality.

[0097] For example, the untruncated value and the truncated theoretical value may be demosaiced (DMC) by the following expressions (3) and (4).

[0098] P i,j,unclip,DMC =[r i,j,unclip ,g i,j,unclip ,b i,j,unclip ] (3)

[0099] P i,j,clip,DMC =[ri,j,clip ,g i,j,clip ,b i,j,clip ] (4)

[0100] In the above expression (3), P i,j,unclip,DMC Represents the RGB three-channel vector value in matrix form obtained after DMC processing of the untruncated value, where r i,j,unclip is the vector value of the R channel without truncated value; g i,j,unclip is the vector value of the G channel without truncated value, b i,j,unclip A vector value of the B channel without truncated values.

[0101] In the above expression (4), P i,j,clip,DMC Represents the RGB three-channel vector value in matrix form obtained after DMC processing of the truncated theoretical value, where r i,j,clip is the vector value of the R channel of the truncated theoretical value; g i,j,clip is the vector value of the G channel of the truncated theoretical value, b i,j,clip The vector value of the B channel that truncates the theoretical value.

[0102] In the disclosed embodiments, DMC processing can be implemented by one or more methods such as directional interpolation, color difference interpolation, color ratio interpolation, and bilinear interpolation. The specific method can be selected according to actual conditions. The embodiments of the present application do not impose any specific restrictions on the specific implementation method of DMC.

[0103] In this embodiment, the image to be processed in RAW format can be reconstructed into an image to be processed in RGB format through DMC processing, providing a data basis for subsequent weighted fusion of untruncated values ​​and truncated theoretical values ​​in the RGB domain.

[0104] In some embodiments, when the image to be processed is an image to be processed based on a red, green and blue color model, the untruncated value and the truncated theoretical value are weightedly fused to obtain an expected untruncated value of the first pixel.

[0105] In this embodiment, if the image to be processed is an RGB-based image to be processed, the above-mentioned demosaicing process is not required, and the untruncated value and the truncated theoretical value are directly weighted fused to correct the untruncated value using the truncated theoretical value.

[0106] Exemplarily, weighted fusion processing may be performed by the following expression (5).

[0107] P i,j,expect =w0×P i,j,clip +(1-w0)×P i,j,unclip (5)

[0108] In the above expression (5), P i,j,expectrepresents the expected un-truncated value of the first pixel, P i,j,clip is the theoretical value of truncation, P i,j,unclip is the untruncated value, w0 is the weight corresponding to the truncated theoretical value, (1-w0) is the untruncated value, w0∈[0,1], indicating that w0 is a weight value greater than or equal to 0 and less than or equal to 1; where the expected untruncated value P of the first pixel is i,j,expect , interception theoretical value P i,j,clip and the untruncated value P i,j,unclip All are in matrix form.

[0109] It should be understood that the value of w0 can be configured as a fixed weight or according to P i,j,expect and P i,j,unclip Design adaptive weights. For example, if P i,j,unclip If you do not need to use P i,j,clip To make corrections, we can set the value of w0 to 0. i,j,expect Directly take the value as the untruncated value P i,j,unclip .

[0110] In the embodiment of the present application, taking into account the nonlinear characteristics of the module, especially in the near-saturation range, the nonlinear characteristics are more significant, and the calculated untruncated value may be inaccurate. The untruncated value can be corrected by using the truncation theoretical value through the set dynamic weight to obtain the expected untruncated value of the first pixel.

[0111] In an embodiment of the present application, the gain module is usually implemented in the ISP RAW domain. Through the DMC processing of the above embodiment, the untruncated value of the RAW domain and the truncated theoretical value of the RAW domain can be converted into the untruncated value of the RGB domain and the truncated theoretical value of the RGB domain, and then the corresponding untruncated value and the corresponding truncated theoretical value are weightedly fused in the RGB domain; and for the gain-related module of the RGB domain, for example, when the digital gain is implemented in the RGB domain, the DMC processing can be skipped, and the corresponding untruncated value and the corresponding truncated theoretical value can be weightedly fused in the RGB domain.

[0112] In some embodiments, when the image to be processed is an image to be processed based on a red, green and blue color mode, the above step S140 may specifically include the following steps.

[0113] S41, adjusting the proportion of each channel of the expected untruncated value, taking values ​​of each adjusted channel value according to a set method, and linearizing the obtained corresponding values ​​to obtain linearized processed values ​​of the expected untruncated value.

[0114] Exemplarily, the proportion of each channel of the expected untruncated value can be adjusted by the following expression (6).

[0115] Pi,j,expect,gamma =gamma(P i,j,expect ) (6)

[0116] In the above expression (6), P i,j,expect is the expected untruncated value, gamma(.) represents the gamma function, which can operate on each element of the vector separately; using the gamma function to i,j,expect Adjust the proportion of different channels to adjust the color restoration strength of the expected untruncated value, and obtain the linearized processing value P of the expected untruncated value. i,j,expect,gamma Among them, the linearized value P of the expected untruncated value is i,j,expect,gamma and the expected untruncated value P i,j,expect All are in matrix form.

[0117] Exemplarily, the linearization process can be implemented by using a degamma function, and the value taking and linearization process according to the setting method can be implemented by the following expression (7).

[0118] Val expect =degamma(func(P i,j,expect,gamma )) (7)

[0119] In the above expression (7), func(.) represents a feasibility function that takes values ​​according to a set method. For example, the set method may be to take the mean or maximum value of each channel value, or may be to take the variance or standard deviation of each channel value. The specific method of taking values ​​may be selected according to actual conditions, and the present embodiment does not make any specific limitation. degamm(.) represents a degamma function, Val expect is the linearized value of the expected untruncated value, P i,j,expect,gamma The meaning of is the same as P in the above expression (6) i,j,expect,gamma The same is not repeated here.

[0120] Exemplarily, the gamma function can be implemented by a variety of methods. For example, a look-up table (LUT) can be used, or the gamma function can be implemented by the following expression (8).

[0121] gamma(x)=(x / wl) 1 / c (8)

[0122] Where x represents the input value, such as the expected untruncated value P i,j,expect Or the theoretical value of interception P i,j,clip , the coefficient c represents the intensity, and the meaning of wl can refer to the meaning of wl1 in the above expression (2), which will not be repeated here.

[0123] Exemplarily, referring to the gamma function implemented by expression (8) above, the de-gamma function can be implemented by the following expression (9) accordingly.

[0124] degamma(x)=(x / wl) c (9)

[0125] The values ​​of x, wl and c are the same as those in the above expression (8) and will not be repeated here.

[0126] It can be seen from the above expressions (8) and (9) that the gamma function and the degamma function are inverse functions of each other.

[0127] S42, adjusting the proportion of each channel of the truncation theoretical value, taking values ​​of each adjusted channel value according to a set method, and linearizing the obtained corresponding values ​​to obtain a linearized processed value of the truncation theoretical value.

[0128] Exemplarily, the proportion of each channel of the truncation theoretical value can be adjusted by the following expression (10).

[0129] P i,j,clip,gamma =gamma(P i,j,clip ) (10)

[0130] In the above expression (10), P i,j,clip is the theoretical value of truncation, and the gamma function is used to calculate P i,j,clip Adjust the proportion of different channels, and then adjust the color restoration strength of the interception theoretical value to obtain the linear processing value P of the interception theoretical value. i,j,clip,gamma Among them, the linear processing value P of the truncated theoretical value i,j,clip,gamma and the theoretical value of interception P i,j,clip All are in matrix form.

[0131] Exemplarily, the value taking and linearization processing according to the setting method can be implemented by the following expression (11).

[0132] Val clip =degamma(func(P i,j,clip,gamma )) (11)

[0133] In the above expression (11), Val clip is the linearized value of the truncated theoretical value; P i,j,clip,gamma For the meaning of , please refer to the above expression (10), for the meaning of func(.) and the meaning of degamm(.), please refer to the above expression (7), which will not be repeated here.

[0134] S43, calculating the nonlinear compressed pixel value of the first pixel after the nonlinear compression processing according to the expected untruncated value, the linearized processed value of the expected untruncated value and the linearized processed value of the truncated theoretical value.

[0135] Through the above steps S41-S43, the nonlinear compressed pixel value of the first pixel after the nonlinear compression process can be calculated for the expected untruncated value.

[0136] In some embodiments, step S43 may specifically include:

[0137] The nonlinear compressed pixel value of the first pixel after nonlinear compression processing can be calculated by the following expression (12):

[0138] P i,j,sup =(P i,j,expect / Val expect )×Val clip (12)

[0139] Among them, P i,j,sup is the nonlinear compressed pixel value, P i,j,expect is the expected un-truncated value of the first pixel; Val expect is the linearized value of the expected untruncated value, Val clip is the linearized value of the truncated theoretical value.

[0140] In this expression, P i,j,expect and P i,j,sup All are in matrix form.

[0141] In this embodiment, the nonlinear compressed pixel value of the first pixel after nonlinear compression processing can be calculated based on the correspondence between the expected untruncated value of the first pixel, the linearized processed value of the expected untruncated value and the linearized processed value of the truncated theoretical value.

[0142] In some embodiments, after step S43, the method further includes: S44, performing channel dimension downsampling processing on the nonlinear compressed pixel value to obtain a downsampled value of the nonlinear compressed pixel value as the nonlinear compressed pixel value of the first pixel after the nonlinear compression processing.

[0143] For example, the channel dimension downsampling process can be performed by the following expression (13):

[0144] P i,j,sup =ds(P i,j,sup ) (13)

[0145] In the above expression (13), ds(.) is the channel dimension downsampling function, P i,j,sup Represents the nonlinear compressed pixel value in matrix form, ds(Pi,j,sup ) represents the pixel value P of the nonlinear compression in matrix form i,j,sup Perform channel dimension downsampling, P i,j,sup =ds(P i,j,sup ) represents the down-sampling value of the nonlinear compressed pixel value as the nonlinear compressed pixel value of the first pixel after the nonlinear compression process, so that the nonlinear compressed pixel value can be restored from the RGB domain to the RAW domain, P i,j,sup In scalar form.

[0146] In this embodiment, if the image to be processed is an image in the RGB domain, step S44 can be ignored; if the image to be processed is an image in the RAW format, the image to be processed can be restored from the RGB domain to the RAW domain by using the above expression (13).

[0147] In some embodiments, the Gr and Gb channel information can be retained during the DMC processing of the untruncated values ​​and the truncated theoretical values ​​by the above expressions (3) and (4), so that the corresponding untruncated values ​​and the truncated theoretical values ​​can still effectively retain the Gr and Gb information after the de-mosaicing processing. Here, Gr and Gb can be used to represent the G channel in the same row as the R and B channels in the 2x2 block, respectively. The same is true for the RCCB, RYYCy, RGB-IR 4x4 and RGB-IR 2x2 sensor styles; for example, Gr and Gb can be the C and Y channels in RCCB and RYYCy; after linear interpolation of the untruncated values ​​and the truncated theoretical values ​​of RGB-IR 4x4 and RGB-IR 2x2 is performed to obtain the RGB channels, Gr and Gb can be the G channel.

[0148] In some embodiments, the above step S150 may specifically include the following steps:

[0149] S51, fusing the nonlinear compression pixel value and the truncation theoretical value to obtain a nonlinear compression optimization value of the first pixel.

[0150] For example, the nonlinear compressed pixel value and the truncated theoretical value can be fused by the following expression (14):

[0151] P i,j,sup,prot =w1×P i,j,clip +(1-w1)×P i,j,sup (14)

[0152] Among them, w1∈[0,1], that is, w1 is a weight value greater than or equal to 0 and less than or equal to 1.

[0153] In the above expression (14), P i,j,clip represents the theoretical value of interception, P i,j,suprepresents the nonlinear compressed pixel value (after channel dimension downsampling), w1 is the weight value corresponding to the truncation theoretical value, and (1-w1) is P i,j,sup The corresponding weight value, P i,j,sup,prot is the nonlinear compression optimization value of the first pixel; where P i,j,sup , P i,j,clip and P i,j,sup,prot All are in scalar value form.

[0154] It should be understood that the value of w1 can be configured as a fixed weight, or according to P i,j,clip and P i,j,sup For example, if you do not need to set the P i,j,sup To make adjustments, you can set the value of w1 to 0. i,j,sup,prot Directly take the value as P i,j,sup .

[0155] In this step, assuming that the pixel value of the first pixel in the image to be processed received by the HLR module is close to saturation, the effect of HLR may not be very obvious after nonlinear compression. In order to improve the effect of HLR, it is necessary to adjust the nonlinear compressed pixel value of the first pixel through weight matching, that is, considering the nonlinear characteristics of the module, the result after nonlinear compression can be selectively further protected to improve the highlight recovery effect.

[0156] S52, calculating a decoupling value for eliminating the influence of the first image processing on the nonlinear compression optimization value according to the parameters of the first image processing.

[0157] Exemplarily, the decoupling value that eliminates the influence of gain processing can be calculated for the nonlinear compression optimization value by the following expression (15):

[0158] P i,j,hlr =P i,j,sup,prot / (gain lsc ×gain digital ×gain awb ) (15)

[0159] In the above expression (15), gain lsc 、gain digital and gain awb They are the gain parameters corresponding to lens shading correction, digital gain, and automatic white balance. lsc ×gain digital ×gain awb Denoted as the total gain parameter, P i,j,sup,prot is the nonlinear compression optimization value, P i,j,hlrThe decoupled value, P, which is the value optimized for nonlinear compression and eliminates the effects of gain processing i,j,sup,prot and P i,j,hlr All are in scalar value form.

[0160] In an embodiment of the present application, if the first image processing is gain processing, after the nonlinear compression optimization value is calculated, the nonlinear compression optimization value can be divided by the total gain parameter, which is beneficial to decoupling the calculated nonlinear compression optimization value from the subsequent gain-related modules and does not affect the gain processing of the subsequent gain-related modules.

[0161] S53, performing truncation processing on the decoupling value of the nonlinear compression optimization value from which the influence of the first image processing has been eliminated according to the parameters of the second truncation processing, to obtain the truncation optimization value of the first pixel.

[0162] Exemplarily, the first image processing is one or more gain processings, and the nonlinear compression optimization value may be truncated by the following expression (16):

[0163] P i,j,hlr,clip =clip(P i,j,jlr ,0,wl hlr ) (16)

[0164] Among them, clip(.) represents the second truncation process, P i,j,hlr The decoupled value representing the nonlinear compression optimization value that has eliminated the effect of gain processing, P i,j,hlr,clip Indicates the truncated optimization value of the first pixel, 0 and wl hlr represents the lower limit and upper limit of the second bit width range used in the bit truncation process, and the upper limit of the second bit width range is the maximum bit width supported by the device executing the image processing method. i,j,hlr and P i,j,hlr,clip All are in the form of scalar values. Exemplarily, the maximum bit width supported by the device executing the image processing method can be: the maximum bit width supported by the HLR module in the image processor.

[0165] In this embodiment, the decoupling value that eliminates the influence of the first image processing can be truncated according to the second bit width range to obtain the truncated optimized value of the first pixel, and the calculated truncated optimized value of each pixel in the image to be processed can be used as the image processing result of the image to be processed.

[0166] In the embodiment of the present application, due to the nonlinear characteristics of the module, the processing result after nonlinear compression can be selectively further protected by truncation processing, so as to calculate the graph limit after near-saturation channel protection.

[0167] In some embodiments, before calculating the truncated theoretical value of the untruncated value after truncation processing according to the parameters of the first truncation processing in step S120, the image to be processed is also subjected to a second image processing, and the second image processing is a color adjustment processing; step S110 may specifically include: determining the current pixel value of the first pixel that has not been truncated, has not been processed by the first image, and has not been processed by the second image; calculating the fourth processed value of the first pixel of the image to be processed after the second image processing as the untruncated value of the first pixel.

[0168] As an example, the color adjustment processing may be color correction management (CCM). That is, the image processing method of the embodiment of the present application may be used as a general solution to the saturation truncation in the ISP pipeline. For the saturation truncation problem caused by the color adjustment type of processing other than the first image processing, the first image processing may not be performed on the processed image but the color adjustment processing may be performed. Accordingly, since the first image processing is not performed, the step of removing the influence of the first image processing described in the above embodiment needs to be skipped.

[0169] It should be understood that color adjustment processing can also be, for example, brightness, saturation, hue and contrast control processing, or it can be sharpening enhancement processing, enhanced color interpolation processing, and other operations that can change the saturation of the image to be processed; the embodiments of the present application do not make specific limitations.

[0170] In some embodiments, the first image processing is one or more gain processing and the second image processing is a color adjustment process.

[0171] In this embodiment, for the saturation truncation problem caused by color adjustment type processing other than gain processing, the processed image may be subjected to color adjustment processing instead of gain processing. Accordingly, since gain processing is not performed, the steps of removing the influence of gain processing described in the above embodiment need to be skipped.

[0172] The image processing method of the embodiment of the present application can fuse the untruncated value and the truncated theoretical value of any pixel in the processed image, and realize the correction of the untruncated value by using the truncated theoretical value, which is beneficial to the protection of the highlight interval and reduces the influence of the nonlinear problem of the near-saturation interval data on the recovery of the highlight area. It also performs nonlinear compression processing based on the expected untruncated value obtained by fusion, and improves the color restoration intensity of the highlight interval through the nonlinear compression intensity control method to solve the problem of insufficient color restoration intensity. In addition, the image processing method of the embodiment of the present application can be adapted to a variety of sensor formats, and the highlight recovery module corresponding to the image processing method can be flexibly adapted to different positions in the ISP pipeline, and can be decoupled from other gain modules for independent operation, or combined with one or more gain modules for operation, thereby having the characteristics of high versatility, high flexibility and low complexity.

[0173] Figure 5 Detailed flowchart of the image processing method according to an exemplary embodiment of the present application is shown. Figure 5 As shown, the image processing method includes the following steps.

[0174] S501, for a first pixel in an image to be processed, determining an un-truncated value of the first pixel before truncation processing, and a truncation theoretical value of the un-truncated value after truncation processing.

[0175] In this step, the untruncated value of the first pixel can be calculated by the above expression (1), and the untruncated value can be truncated by the above expression (2) to obtain the truncated theoretical value of the first pixel.

[0176] S502, calculating a demosaicing value.

[0177] In this step, the untruncated value and the truncated theoretical value are demosaiced by using the above expressions (3) and (4). The demosaicing process can convert the image to be processed in the RAW format into the image to be processed in the RGB format.

[0178] S503, calculating the expected value after highlight interval protection.

[0179] In this step, the demosaic value of the untruncated value and the demosaic value of the truncated theoretical value can be weightedly fused by the above expression (5) to obtain the expected untruncated value of the first pixel.

[0180] S504, calculating the nonlinear compressed pixel value after nonlinear compression.

[0181] In this step, the proportion of each channel of the expected untruncated value can be adjusted by the above expression (6), and then the corresponding pixel value after adjustment can be firstly taken according to the set method (for example, taking the average value) by the above expression (7), and then the de-gamma calculation is performed to obtain the linearized processing value of the expected untruncated value; and the proportion of each channel of the truncated theoretical value can be adjusted by the above expression (10), and then the corresponding pixel value after adjustment can be firstly taken according to the set method (for example, taking the average value) by the above expression (11), and then the de-gamma calculation is performed to obtain the linearized processing value of the truncated theoretical value.

[0182] In this step, the expected untruncated value of the first pixel, the linearized value of the expected untruncated value, and the linearized value of the truncated theoretical value can be used to calculate the nonlinear compressed pixel value of the first pixel after the nonlinear compression processing through the above expression (12), and then the nonlinear compressed pixel value after the nonlinear compression processing can be downsampled through the above expression (13) to restore the nonlinear compressed pixel value from the RGB domain to the RAW domain.

[0183] S505, calculating the nonlinear compression optimization value after near-saturation channel protection.

[0184] In this step, due to the nonlinear characteristics of the module, the nonlinear compressed pixel value and the truncation theoretical value can be weighted fused by the above expression (14) to obtain the nonlinear compression optimization value of the first pixel, thereby further protecting the result after nonlinear compression.

[0185] S506, calculating the result after the pre-compensation gain.

[0186] In this step, the influence of the gain processing on the nonlinear compression optimization value can be eliminated by the above expression (15), thereby obtaining a decoupled value of the nonlinear compression optimization value from which the influence of the gain processing has been eliminated.

[0187] In some embodiments, only the gain suppression of one or more modules among the LSC module, DG module and AWB module may be processed. For example, if the LSC gain suppression is not performed, the gain parameter of the LSC module in expression (15) may be set to 1.

[0188] S507, performing output bit width truncation operation.

[0189] In this step, the decoupling value of the nonlinear compression optimization value that has eliminated the influence of gain processing can be truncated by the above expression (16) to obtain the truncated optimization value of the first pixel. The truncated optimization value of the first pixel can be used as the output result of the image processing method in the embodiment of the present application, that is, the output result of the HLR module that executes the image processing method in the image processing device.

[0190] According to the image processing method of the embodiment of the present application, the untruncated value and the truncated theoretical value of any pixel in the processed image can be fused to realize the correction of the untruncated value by using the truncated theoretical value, which is beneficial to the protection of the highlight interval and the reduction of the influence of the nonlinear problem of the near-saturation interval data on the recovery of the highlight area. Nonlinear compression processing is performed based on the expected untruncated value obtained by fusion, and the color restoration intensity of the highlight interval is improved through the nonlinear compression intensity control method to solve the problem of insufficient color restoration intensity. In addition, the image processing method of the embodiment of the present application can be adapted to a variety of sensor formats, and the highlight recovery module corresponding to the image processing method can be flexibly adapted to different positions in the ISP pipeline. It can be decoupled from other gain modules and operated independently, or combined with one or more gain modules for operation, so as to have the characteristics of high versatility, high flexibility and low complexity.

[0191] It can be understood that the above-mentioned various method embodiments mentioned in this application can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, this application will not go into details. Those skilled in the art can understand that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined by its function and possible internal logic.

[0192] In addition, the present application also provides an image processing device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any image processing method provided by the present application. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0193] In a second aspect, the present application provides an image processing device. Figure 6 , the image processing device 600 provided in the embodiment of the present application includes:

[0194] A determination module 610 is used to determine, for a first pixel in the image to be processed, an un-truncated value of the first pixel before the image to be processed is truncated;

[0195] A first truncation module 620, configured to calculate a truncation theoretical value of an untruncated value after truncation according to the parameters of the first truncation process;

[0196] A fusion module 630 is used to fuse the untruncated value and the truncated theoretical value to obtain an expected untruncated value of the first pixel;

[0197] A non-linear compression module 640, configured to calculate a non-linear compressed pixel value of the first pixel after non-linear compression processing according to the expected untruncated value;

[0198] The second truncation module 650 is used to perform truncation processing on the nonlinear compressed pixel value according to the parameters of the second truncation processing to obtain the truncation optimization value of the first pixel.

[0199] In some embodiments, the image processing device 600 also includes: a first image processing module, which is used to perform first image processing on the image to be processed before calculating the truncated theoretical value of the untruncated value after truncating according to the parameters of the first truncation processing; a determination module 610, which is specifically used to: determine the current pixel value of the first pixel that has not been truncated and has not been subjected to the first image processing; calculate the first processed value of the first pixel of the image to be processed after the first image processing according to the parameters of the first image processing, as the untruncated value of the first pixel; a second truncation module 650, which is specifically used to: calculate a decoupling value that eliminates the influence of the first image processing for the nonlinear compression pixel value according to the parameters of the first image processing; and perform truncation processing on the decoupling value of the nonlinear compression pixel value that has eliminated the influence of the first image processing according to the parameters of the second truncation processing to obtain the truncation optimization value of the first pixel.

[0200] In some embodiments, the first image processing is one or more gain processing, and the parameters of the first image processing are one or more gain parameters.

[0201] In some embodiments, one or more gain parameters include non-pixel level gain parameters; the image processing device 600 also includes: an interpolation module, which is used to interpolate the non-pixel level gain parameters before calculating the first processing value of the first pixel of the image to be processed after the first image processing, so as to obtain the interpolated pixel level gain parameters; and update one or more gain parameters according to the pixel level gain parameters.

[0202] In some embodiments, when the first image processing includes lens shading correction, digital gain, and automatic white balance, the determination module 610 is used to calculate, according to the parameters of the first image processing, a first processed value of a first pixel of the image to be processed after the first image processing as the un-truncated value of the first pixel, specifically to:

[0203] Using P i,j,unclip =P i,j ×gain lsc ×gain digital ×gain awb Calculate the uncut value of the first pixel; where P i,j is the first pixel without truncation processing and first image processing, gain lsc 、gain digital and gain awb are the gain parameters corresponding to lens shading correction, digital gain, and automatic white balance, respectively. i,j,unclipis the untruncated value;

[0204] The first truncation module 620 is specifically used to: i,j,clip =clip(P i,j,unclip ,0,wl1) to perform truncation processing on the untruncated value; where clip(.) represents the first truncation processing, P i,j,clip It represents the theoretical value of truncation, 0 and wl1 represent the lower and upper limits of the first bit width range used in the truncation process. The upper limit of the first bit width range is the maximum bit width supported by the gain processing module corresponding to the specified gain processing. The specified gain processing is any one of the multiple gain processings.

[0205] In some embodiments, the first image processing module is used to perform first image processing on the image to be processed before calculating the truncated theoretical value of the untruncated value after truncation processing according to the parameters of the first truncation processing; the determination module 610 is specifically used to: for the first pixel in the image to be processed, determine the second processed value of the first pixel of the image to be processed that has undergone the first image processing but not undergone the truncation processing, as the untruncated value of the first pixel.

[0206] In some embodiments, a first image processing module is used to perform a first image processing on the image to be processed before calculating a truncated theoretical value of an untruncated value after truncating according to parameters of a first truncation processing, wherein the first image processing includes multiple gain processings; a determination module 610 is specifically used to: determine the current pixel value of the first pixel of the image to be processed that has undergone N1 gain processings but has not undergone truncation processing for a first pixel in the image to be processed, wherein N1 is an integer greater than or equal to 1; calculate a third processed value of the first pixel of the image to be processed after N2 gain processings according to gain parameters corresponding to N2 gain processings as the untruncated value of the first pixel; wherein N2 is an integer greater than or equal to 1, and the N1 gain processings and the N2 gain processings are different gain processings; a second truncation module 650 is specifically used to: calculate a decoupling value that eliminates the influence of the N2 gain processings for the nonlinear compression pixel value according to the gain parameters corresponding to the N2 gain processings; and perform truncation processing on the decoupling value of the nonlinear compression pixel value that has eliminated the influence of the N2 gain processings according to the parameters of the second truncation processing to obtain a truncation optimization value of the first pixel.

[0207] In some embodiments, the image to be processed is an image in a non-red, green and blue color mode; the determination module 610 is specifically used to: convert the image to be processed in a non-red, green and blue color mode into an image to be processed in a red, green and blue color mode; for the first pixel in the image to be processed in the red, green and blue color mode, determine the un-truncated value of the first pixel of the image to be processed in the red, green and blue color mode before the truncation processing.

[0208] In some embodiments, the image to be processed is an image in an unprocessed format, and the fusion module 630 is specifically used to: perform demosaicing on the untruncated value to obtain a first demosaiced value; perform demosaicing on the truncated theoretical value to obtain a second demosaiced value; and fuse the first demosaiced value and the second demosaiced value to obtain an expected untruncated value of the first pixel.

[0209] In some embodiments, the nonlinear compression module 640 is specifically used to: adjust the proportion of each channel of the expected untruncated value, take values ​​for each adjusted channel value according to a set method, and linearize the corresponding values ​​obtained to obtain the linearized processed value of the expected untruncated value; adjust the proportion of each channel of the truncated theoretical value, take values ​​for each adjusted channel value according to a set method, and linearize the corresponding values ​​obtained to obtain the linearized processed value of the truncated theoretical value; calculate the nonlinear compressed pixel value of the first pixel after nonlinear compression processing according to the expected untruncated value, the linearized processed value of the expected untruncated value and the linearized processed value of the truncated theoretical value.

[0210] In some embodiments, the nonlinear compression module 640, when used to calculate the nonlinear compressed pixel value of the first pixel after nonlinear compression processing according to the expected untruncated value, the linearized processed value of the expected untruncated value, and the linearized processed value of the truncated theoretical value, is specifically used to:

[0211] Using P i,j,sup =(P i,j,expect / Val expect )×Val clip Calculate the nonlinear compressed pixel value; where P i,j,sup is the nonlinear compressed pixel value, P i,j,expect is the expected un-truncated value of the first pixel; Val expect is the linearized value of the expected untruncated value, Val clip is the linearized value of the truncated theoretical value.

[0212] In some embodiments, the image processing device 600 also includes: a downsampling module, which is used to perform channel dimension downsampling processing on the nonlinear compressed pixel value after calculating the nonlinear compressed pixel value of the first pixel after nonlinear compression processing, so as to obtain a downsampled value of the nonlinear compressed pixel value as the nonlinear compressed pixel value of the first pixel after nonlinear compression processing.

[0213] In some embodiments, the second truncation module 650 is specifically used to: fuse the nonlinear compression pixel value and the truncation theoretical value to obtain the nonlinear compression optimization value of the first pixel; calculate a decoupling value of the nonlinear compression optimization value that eliminates the influence of the first image processing according to the parameters of the first image processing; and perform truncation processing on the decoupling value of the nonlinear compression optimization value that has eliminated the influence of the first image processing according to the parameters of the second truncation processing to obtain the truncation optimization value of the first pixel.

[0214] In some embodiments, the image processing device 600 also includes: a second image processing module, which is used to perform a second image processing on the image to be processed before calculating the truncated theoretical value of the untruncated value after truncation according to the parameters of the first truncation processing, and the second image processing is a color adjustment process; a determination module 610, which is specifically used to: determine the current pixel value of the first pixel that has not been truncated, has not been processed by the first image, and has not been processed by the second image; calculate the fourth processed value of the first pixel of the image to be processed after the second image processing as the untruncated value of the first pixel.

[0215] In some embodiments, the first image processing is one or more gain processing and the second image processing is a color adjustment process.

[0216] According to the image processing device of the embodiment of the present application, the untruncated value and the truncated theoretical value of any pixel in the processed image can be fused to realize the correction of the untruncated value by using the truncated theoretical value, which is beneficial to the protection of the highlight interval and the reduction of the influence of the nonlinear problem of the near-saturation interval data on the recovery of the highlight area. Nonlinear compression processing is performed based on the expected untruncated value obtained by fusion, and the color restoration intensity of the highlight interval is improved through the nonlinear compression intensity control method to solve the problem of insufficient color restoration intensity. In addition, the image processing method of the embodiment of the present application can be adapted to a variety of sensor formats, and the highlight recovery module corresponding to the image processing method can be flexibly adapted to different positions in the ISP pipeline, and can be decoupled from other gain modules for independent operation, or combined with one or more gain modules for operation, thereby having the characteristics of high versatility, high flexibility and low complexity.

[0217] It should be clear that the present application is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the convenience and brevity of description, a detailed description of the known methods is omitted here, and the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here.

[0218] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0219] The image processing device of the embodiment of the present application can be implemented as an HLR module in an ISP. The HLR module can be used as an enhancement module of the ISP digital image processor, effectively improving the highlight saturation problem caused by the saturation truncation operation in the ISP pipeline processing, and can be widely used in mobile phones, SLRs, surveillance cameras, car cameras and other products that contain ISP digital image processors, thereby being widely used in consumer electronics, smart security monitoring, smart homes, AR / XR / MR, autonomous driving and other fields.

[0220] In a third aspect, the present application also provides an electronic device. Figure 7 The electronic device includes: at least one processor 701; at least one memory 702, and one or more I / O interfaces 703, connected between the processor 701 and the memory 702; wherein the memory 702 stores one or more computer programs that can be executed by the at least one processor 701, and the one or more computer programs are executed by the at least one processor 701 so that the at least one processor 701 can perform the above-mentioned image processing method.

[0221] Among them, the processor is a device with data processing capabilities, including but not limited to the central processing unit (CPU); the memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, and can realize information exchange between the memory and the processor, including but not limited to the data bus (Bus), etc.

[0222] The present application also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned image processing method when executed by the processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0223] Those skilled in the art will appreciate that all or some of the steps, systems, and functional modules / units in the apparatus disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0224] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.

[0225] Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-temporary medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, magnetic tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0226] The present application has disclosed example embodiments, and although specific terms are used, they are only used and should only be interpreted as general illustrative meanings, and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. An image processing method, characterized in that: The method is used to determine the truncation optimization value of the first pixel of the image to be processed after the truncation processing, wherein the first pixel is any pixel in the image to be processed; the method comprises: For a first pixel in the image to be processed, determining an un-truncated value of the first pixel before the image to be processed is truncated; Calculate the truncated theoretical value of the untruncated value after the truncation process according to the parameters of the first truncation process; fusing the untruncated value and the truncated theoretical value to obtain an expected untruncated value of the first pixel; For the expected untruncated value, calculating a nonlinear compressed pixel value of the first pixel after nonlinear compression processing; The nonlinear compressed pixel value is truncated according to the parameters of the second truncation process to obtain the truncation optimization value of the first pixel.

2. The method according to claim 1, characterized in that Before calculating the truncation theoretical value of the un-truncation value after the truncation processing according to the parameters of the first truncation processing, the image to be processed is further subjected to the first image processing; The step of determining, for a first pixel in the image to be processed, an un-truncated value of the first pixel before the image to be processed is truncated, comprises: Determine a current pixel value of the first pixel that has not been subjected to truncation processing and first image processing; Calculating, according to the first image processing parameter, a first processed value of the first pixel of the image to be processed after the first image processing as an untruncated value of the first pixel; The step of performing truncation processing on the nonlinear compressed pixel value according to the parameter of the second truncation processing to obtain the truncation optimization value of the first pixel includes: Calculating, based on the parameters of the first image processing, a decoupling value for eliminating the influence of the first image processing for the nonlinear compressed pixel value; According to the parameters of the second truncation processing, a truncation processing is performed on the decoupling value of the nonlinear compressed pixel value from which the influence of the first image processing has been eliminated, so as to obtain a truncation optimized value of the first pixel.

3. The method according to any one of claims 2, characterized in that The first image processing is one or more gain processings, and the parameters of the first image processing are one or more gain parameters.

4. The method according to claim 3, characterized in that The one or more gain parameters include a non-pixel-level gain parameter; before calculating the first processing value of the first pixel after the image to be processed is processed by the first image, the method further includes: Performing interpolation processing on the non-pixel-level gain parameter to obtain an interpolated pixel-level gain parameter; The one or more gain parameters are updated according to the pixel-level gain parameters.

5. The method according to claim 3, characterized in that: In a case where the first image processing includes lens shading correction, digital gain, and automatic white balance, calculating, according to the parameters of the first image processing, a first processed value of the first pixel of the image to be processed after the first image processing as an un-truncated value of the first pixel, includes: Using P i,j,unclip =P i,j ×gain lsc ×gain digital ×gain awb calculating an untruncated value of the first pixel; Among them, P i,j is the first pixel that has not been subjected to the truncation process and the first image process, gain lsc 、gain digital and gain awb are the gain parameters corresponding to the lens shading correction, the gain parameters corresponding to the digital gain, and the gain parameters corresponding to the automatic white balance, respectively. i,j,unclip is the untruncated value; The calculating the truncated theoretical value of the untruncated value after the truncation process according to the parameters of the first truncation process comprises: Using P i,j,clip =clip(P i,j,unclip ,0,wl1) truncate the untruncated value; Among them, clip(.) represents the first truncation processing, P i,j,clip represents the theoretical value of truncation, 0 and wl1 represent the lower limit and upper limit of the first bit width range used for truncation processing, the upper limit of the first bit width range is the maximum bit width supported by the gain processing module corresponding to the specified gain processing, and the specified gain processing is any gain processing among multiple gain processings.

6. The method according to claim 1, characterized in that Before calculating the truncation theoretical value of the un-truncation value after the truncation processing according to the parameters of the first truncation processing, the image to be processed is further subjected to the first image processing; The step of determining, for a first pixel in the image to be processed, an un-truncated value of the first pixel before the image to be processed is truncated, comprises: For a first pixel in the image to be processed, a second processed value of the first pixel of the image to be processed that has been processed by the first image but not been processed by truncation is determined as an un-truncation value of the first pixel.

7. The method according to claim 1, characterized in that Before calculating the truncation theoretical value of the un-truncated value after the truncation processing according to the parameters of the first truncation processing, the image to be processed is further subjected to a first image processing, wherein the first image processing includes a plurality of gain processings; The step of determining, for a first pixel in the image to be processed, an un-truncated value of the first pixel before the image to be processed is truncated, comprises: For a first pixel in the image to be processed, determining a current pixel value of the first pixel of the image to be processed that has undergone N1 gain processing but has not undergone truncation processing; N1 is an integer greater than or equal to 1; Calculating, according to the gain parameters corresponding to the N2 gain processings, a third processed value of the first pixel of the image to be processed after the N2 gain processings as the untruncated value of the first pixel; wherein N2 is an integer greater than or equal to 1, and the N1 gain processing and the N2 gain processing are different gain processings; The step of performing truncation processing on the nonlinear compressed pixel value according to the parameter of the second truncation processing to obtain the truncation optimization value of the first pixel includes: Calculating, for the nonlinear compressed pixel value, a decoupling value that eliminates the influence of the N2 gain processing according to the gain parameters corresponding to the N2 gain processing; According to the parameters of the second truncation processing, a truncation processing is performed on the decoupling value of the nonlinear compressed pixel value from which the influence of the N2 gain processings has been eliminated, so as to obtain a truncation optimized value of the first pixel.

8. The method according to claim 1, characterized in that: The image to be processed is an image in a non-red, green and blue color mode; and determining, for a first pixel in the image to be processed, an un-truncated value of the first pixel before the image to be processed is subjected to truncation processing, comprises: Converting the image to be processed in a non-red, green and blue color mode into an image to be processed in a red, green and blue color mode; For a first pixel in the image to be processed in the red, green and blue color mode, an un-truncated value of the first pixel before the image to be processed in the red, green and blue color mode is subjected to truncation processing is determined.

9. The method according to claim 1, characterized in that: The image to be processed is an image in an unprocessed format, and fusing the untruncated value and the truncated theoretical value to obtain the expected untruncated value of the first pixel includes: Performing a demosaicing process on the untruncated value to obtain a first demosaicing value; Performing a demosaicing process on the truncated theoretical value to obtain a second demosaicing value; The first demosaic value and the second demosaic value are merged to obtain an expected un-truncated value of the first pixel.

10. The method according to claim 9, characterized in that The step of calculating the nonlinear compressed pixel value of the first pixel after nonlinear compression processing for the expected untruncated value includes: Adjusting the proportion of each channel of the expected untruncated value, taking values ​​of each adjusted channel value according to a set method, and performing linearization processing on the obtained corresponding values ​​to obtain the linearized processed value of the expected untruncated value; Adjusting the proportion of each channel of the truncation theoretical value, taking values ​​of each adjusted channel value according to the setting method, and performing linear processing on the corresponding values ​​obtained to obtain the linear processing value of the truncation theoretical value; The nonlinear compressed pixel value of the first pixel after nonlinear compression processing is calculated according to the expected untruncated value, the linearized processed value of the expected untruncated value and the linearized processed value of the truncated theoretical value.

11. The method according to claim 10, characterized in that The calculating, according to the expected untruncated value, the linearized value of the expected untruncated value and the linearized value of the truncated theoretical value, the nonlinear compressed pixel value of the first pixel after nonlinear compression processing comprises: Using P i,j,sup =(P i,j,expect / Val expect )×Val clip Calculating the nonlinear compressed pixel value; Among them, P i,j,sup is the nonlinear compressed pixel value, P i,j,expect is the expected untruncated value of the first pixel; Val expect is the linearized value of the expected untruncated value, Val clip is the linearized value of the truncated theoretical value.

12. The method according to claim 10, characterized in that After calculating the nonlinear compressed pixel value of the first pixel after the nonlinear compression process, the method further includes: The nonlinear compressed pixel value is subjected to channel dimension downsampling processing to obtain a downsampled value of the nonlinear compressed pixel value as the nonlinear compressed pixel value of the first pixel after the nonlinear compression processing.

13. The method according to claim 10, characterized in that The step of performing truncation processing on the nonlinear compressed pixel value according to the parameter of the second truncation processing to obtain the truncation optimization value of the first pixel includes: The nonlinear compression pixel value and the truncation theoretical value are merged to obtain a nonlinear compression optimization value of the first pixel; Calculating, based on the parameters of the first image processing, a decoupling value for the nonlinear compression optimization value to eliminate the influence of the first image processing; According to the parameters of the second truncation processing, the decoupling value of the nonlinear compression optimization value from which the influence of the first image processing has been eliminated is truncation processed to obtain the truncation optimization value of the first pixel.

14. The method according to claim 2, characterized in that Before calculating the truncation theoretical value of the un-truncation value after the truncation processing according to the parameters of the first truncation processing, the image to be processed is further subjected to a second image processing, wherein the second image processing is a color adjustment processing; The step of determining, for a first pixel in the image to be processed, an un-truncated value of the first pixel before the image to be processed is truncated, comprises: Determine a current pixel value of the first pixel that has not been subjected to truncation processing, first image processing, and second image processing; A fourth processed value of the first pixel after the image to be processed is processed by the second image is calculated as an un-truncated value of the first pixel.

15. The method according to claim 14, characterized in that The first image processing is one or more gain processings, and the second image processing is a color adjustment processing.

16. An image processing device, characterized in that: The device is used to determine the truncation optimization value of the first pixel of the image to be processed after the truncation processing, wherein the first pixel is any pixel in the image to be processed; the device comprises: A determination module, configured to determine, for a first pixel in the image to be processed, an un-truncated value of the first pixel before the image to be processed is subjected to truncation processing; A first truncation module, used for calculating a truncation theoretical value of the untruncated value after the truncation process according to the parameters of the first truncation process; a fusion module, configured to fuse the untruncated value and the truncated theoretical value to obtain an expected untruncated value of the first pixel; A non-linear compression module, used for calculating a non-linear compressed pixel value of the first pixel after non-linear compression processing according to the expected untruncated value; The second truncation module is used to perform truncation processing on the nonlinear compressed pixel value according to the parameters of the second truncation processing to obtain the truncation optimization value of the first pixel.

17. An electronic device comprising: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method according to any one of claims 1 to 15.

18. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 15 is implemented.

Images