Image processing device and method, storage medium and electronic equipment

Abstract

The invention discloses an image processing device and method, a storage medium and electronic equipment. The image processing device comprises a processor, and the processor is configured to obtain a first image collected by a first RGB-IR image sensor in a live-action light source environment; acquiring a first removal parameter calibrated for the first preset color temperature; wherein the first removal parameter is a parameter used for completely removing the IR component of the image; obtaining a white balance Planck curve of the first RGB-IR image sensor; determining an estimated color temperature of the live-action light source environment based on the first image, a white balance Planck curve and a first removal parameter; determining a second removal parameter adapted to the estimated color temperature; wherein the second removal parameter is a parameter used for partially removing the IR component of the image; and performing IR component removal on the first image according to the second removal parameter to obtain a second image. According to the embodiment of the invention, great brightness and signal-to-noise ratio loss can be avoided, and the subsequent image processing effect can be ensured.

Description

Technical Field

[0001] This disclosure relates to image technology, and in particular to an image processing apparatus, method, storage medium, and electronic device. Background Technology

[0002] Currently, RGB-IR image sensors are being used more and more widely; where R represents red, G represents green, B represents blue, and IR represents infrared.

[0003] Generally speaking, for images captured by RGB-IR image sensors during the day, the IR component needs to be removed from the image before subsequent image processing. However, the removal methods in related technologies will result in a significant loss of brightness and signal-to-noise ratio (SNR), affecting the effect of subsequent image processing. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides an image processing apparatus, method, storage medium, and electronic device.

[0005] According to one aspect of the present disclosure, an image processing apparatus is provided, comprising: a processor configured to: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain a first removal parameter for a first preset color temperature calibration; wherein the first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the white balance Planck curve, and the first removal parameter, the estimated color temperature of the real-world light source environment is determined; Determine a second removal parameter that is compatible with the estimated color temperature; wherein the second removal parameter is a parameter used to partially remove the IR components of the image; According to the second removal parameters, the first image is subjected to IR component removal to obtain the second image.

[0006] According to another aspect of the present disclosure, an image processing method is provided, comprising: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain a first removal parameter for a first preset color temperature calibration; wherein the first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the white balance Planck curve, and the first removal parameter, the estimated color temperature of the real-world light source environment is determined; Determine a second removal parameter that is compatible with the estimated color temperature; wherein the second removal parameter is a parameter used to partially remove the IR components of the image; According to the second removal parameters, the first image is subjected to IR component removal to obtain the second image.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that is executed by a processor to implement the above-described image processing method.

[0008] According to another aspect of the present disclosure, a computer program product is provided that, when instructions in the computer program product are executed by a processor, performs the above-described image processing method.

[0009] Based on the image processing apparatus, method, storage medium, electronic device, and program product provided in the above embodiments of this disclosure, the first image acquired by the first RGB-IR image sensor in a real-world lighting environment is affected by the color temperature of the light source. The white balance Planck curve of the first RGB-IR image sensor is related to the color temperature. A first removal parameter calibrated for a first preset color temperature is used to completely remove the IR components of the image, which helps to eliminate the adverse effects of the image IR components on color temperature estimation. Therefore, based on the image, the white balance Planck curve, and the first removal parameter, the estimated color temperature of the real-world lighting environment can be determined relatively accurately. Following a second removal parameter adapted to the estimated color temperature, IR component removal is performed on the first image to obtain a second image, which can then be used for subsequent image processing. Since the second removal parameter is used to partially remove the IR components of the image, removing IR components from the first image according to the second removal parameter will only remove a portion of the IR components, not all of them. This helps to avoid the significant loss of brightness and signal-to-noise ratio caused by the complete removal of IR components. Thus, there will be no obvious noise in the second image, which helps to ensure the effect of subsequent image processing. Attached Figure Description

[0010] Figure 1 This is a system architecture diagram to which some exemplary embodiments of this disclosure apply.

[0011] Figure 2 This is a schematic diagram of the structure of an image processing apparatus provided by some exemplary embodiments of the present disclosure.

[0012] Figure 3-1 This is a schematic diagram of the data arrangement in the first image in some exemplary embodiments of this disclosure.

[0013] Figure 3-2 This is a schematic diagram of the data arrangement in the second image in some exemplary embodiments of this disclosure.

[0014] Figure 3-3 This is a schematic diagram showing the random and discrete statistical landing points of the white area under different color temperatures in some exemplary embodiments of this disclosure.

[0015] Figure 3-4 This is a schematic diagram illustrating the distribution pattern of the white balance Planck curve in some exemplary embodiments of this disclosure.

[0016] Figure 4 This is a schematic diagram of the processor structure in some exemplary embodiments of this disclosure.

[0017] Figure 5-1 This is a schematic diagram of the calibration process for removing parameters in some exemplary embodiments of this disclosure.

[0018] Figure 5-2 This is a schematic diagram illustrating the calibration process and application of the white balance Planck curve in some exemplary embodiments of this disclosure.

[0019] Figure 5-3 This is a flowchart illustrating the operation of a processor in an image processing apparatus provided by some exemplary embodiments of this disclosure.

[0020] Figure 6 This is one of the schematic flowcharts of an image processing method provided by some exemplary embodiments of this disclosure.

[0021] Figure 7 This is a second schematic flowchart of an image processing method provided by some exemplary embodiments of this disclosure.

[0022] Figure 8 This is the third flowchart illustrating an image processing method provided by some exemplary embodiments of this disclosure.

[0023] Figure 9 This is the fourth flowchart illustrating an image processing method provided by some exemplary embodiments of this disclosure.

[0024] Figure 10 This is the fifth of several exemplary embodiments of the image processing method provided in this disclosure.

[0025] Figure 11 This is a schematic flowchart of an image processing method provided by some exemplary embodiments of this disclosure.

[0026] Figure 12 This is a schematic diagram of the structure of an electronic device provided by some exemplary embodiments of this disclosure. Detailed Implementation

[0027] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0028] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0029] Application Overview Understandably, an RGB-IR image sensor is an image sensor that can respond to both the visible light band and the infrared band.

[0030] Generally, for images captured by RGB-IR image sensors during the daytime, the IR component needs to be removed from the image before subsequent image processing. Because RGB-IR image sensors have a high response value to the infrared band, for example, 50% of the peak value of the response value to the visible light band, removing the IR component may result in a significant loss of brightness and signal-to-noise ratio, which can easily affect the effect of subsequent image processing.

[0031] Exemplary System Figure 1 This is a system architecture diagram applicable to some exemplary embodiments of the present disclosure, including an RGB-IR image sensor 12 and an image processing device 14; wherein the image processing device 14 may be electrically connected to the RGB-IR image sensor 12.

[0032] Optionally, the image processing device 14 may be a chip or electronic device for performing various processing on the images acquired by the RGB-IR image sensor 12. The chip here may be a system-on-a-chip (SOC), such as, but not limited to, intelligent driving chips, smart cockpit chips, etc. The electronic device here includes, but is not limited to, drones, action cameras, virtual reality devices, augmented reality devices, etc.

[0033] In the embodiments of this disclosure, for images acquired by the RGB-IR image sensor 12 during the day, the image processing device 14 can first remove some IR components from the image before performing subsequent image processing. Since the IR components are only partially removed, rather than completely removed, it is beneficial to avoid the significant loss of brightness and signal-to-noise ratio caused by the complete removal of the IR components, thereby ensuring the effect of subsequent image processing.

[0034] Exemplary device Figure 2This is a schematic diagram of the structure of an image processing apparatus 14 provided in some exemplary embodiments of this disclosure. For example... Figure 2 As shown, the image processing device 14 includes a processor 140, which may be, for example, an image signal processor (ISP). The processor 140 is configured to: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain the first removal parameter for the first preset color temperature calibration; wherein, the first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the white balance Planck curve, and the first removal parameter, the estimated color temperature of the real-world light source environment is determined; Determine a second removal parameter that is compatible with the estimated color temperature; wherein the second removal parameter is a parameter used to partially remove the IR components of the image; According to the second removal parameters, the IR component is removed from the first image to obtain the second image.

[0035] Optionally, Figure 1 The RGB-IR image sensor 12 in the image processing device 14 can serve as the first RGB-IR image sensor. Taking the case where the image processing device 14 is a smart cockpit chip as an example, the vehicle equipped with the smart cockpit chip can include a smart cockpit. The first RGB-IR image sensor can be an RGB-IR image sensor installed inside the smart cockpit, and the real-scene lighting environment can be a dynamically changing lighting environment composed of all natural and artificial light sources inside the smart cockpit. Taking the case where the image processing device 14 is a drone as an example, the first RGB-IR image sensor can be an RGB-IR image sensor installed on the drone's fuselage or gimbal, and the real-scene lighting environment can be a dynamically changing lighting environment composed of various light sources in the real three-dimensional space where the drone is performing its mission. The processor 140 can acquire the RGB-IR image collected by the first RGB-IR image sensor in the real-scene lighting environment and use this RGB-IR image as the first image. Clearly, the first image is the raw image collected by the first RGB-IR image sensor; therefore, the first image can also be called RGB-IR raw data.

[0036] In an optional example, the data arrangement in the first image can be seen in [reference needed]. Figure 3-1 Then the data arrangement in the first image satisfies: The area of ​​size is treated as a repeating unit. The four pixels in the first row of this repeating unit correspond to the B channel, G channel, R channel, and G channel, respectively. The four pixels in the second row of this repeating unit correspond to the G channel, IR channel, G channel, and IR channel, respectively. The four pixels in the third row of this repeating unit correspond to the R channel, G channel, B channel, and G channel, respectively. The four pixels in the fourth row of this repeating unit correspond to the G channel, IR channel, G channel, and IR channel, respectively.

[0037] Optionally, the first preset color temperature can be the color temperature corresponding to a standard light source, such as, but not limited to, light source A, light source D40, light source D50, light source D65, light source D75, etc., and the first preset color temperature can be, for example, but not limited to, 2856K corresponding to light source A, 4000K corresponding to light source D40, 5000K corresponding to light source D50, 6500K corresponding to light source D65, 7500K corresponding to light source D75, etc. The first removal parameter corresponding to the first preset color temperature for completely removing (i.e., 100% removal) the IR components of the image can be pre-calibrated and stored in memory, and the processor 140 can obtain the first removal parameter by reading the memory; wherein, the first removal parameter may include the removal coefficients corresponding to the R channel, G channel, and B channel, respectively.

[0038] In some optional embodiments of this disclosure, the calibration process for the first removal parameter may include: (a1) At the first preset color temperature, using the first RGB-IR image sensor (which can also be replaced by another RGB-IR image sensor with the same optical specifications as the first RGB-IR image sensor, such as the second RGB-IR image sensor mentioned below), RGB-IR images of a 24-color card without an IR cutoff filter (hereinafter referred to as the original image P1 for ease of description) and RGB-IR images of a 24-color card with an IR cutoff filter (hereinafter referred to as the original image P2 for ease of description) are acquired respectively; wherein, the original image P2 has the same size as the original image P1.

[0039] (a2) Using a preset interpolation algorithm, interpolate the original image P1 to obtain four full-resolution images with the same size as the original image P1: P1-1 (R channel), P1-2 (G channel), P1-3 (B channel), and P1-4 (IR channel). Using the same preset interpolation algorithm, interpolate the original image P2 to obtain four full-resolution images with the same size as the original image P2: P2-1 (R channel), P2-2 (G channel), P2-3 (B channel), and P2-4 (IR channel). The preset interpolation algorithm includes, but is not limited to, bilinear interpolation, trilinear interpolation, and image gradient-guided interpolation algorithms.

[0040] (a3) Take the 20th, 21st, 22nd, or 23rd color patch in the 24-color chart as the target color patch. Calculate the following pixel values: the first mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P1-1 corresponding to the R channel; the second mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P1-2 corresponding to the G channel; the third mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P1-3 corresponding to the B channel; the fourth mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P1-4 corresponding to the IR channel; the fifth mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P2-1 corresponding to the R channel; the sixth mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P2-2 corresponding to the G channel; and the seventh mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P2-3 corresponding to the B channel. Using (first mean - fifth mean) / fourth mean as the removal parameter corresponding to the R channel in the first removal parameter, using (second mean - sixth mean) / fourth mean as the removal parameter corresponding to the G channel in the first removal parameter, and using (third mean - seventh mean) / fourth mean as the removal parameter corresponding to the B channel in the first removal parameter, the complete first removal parameter can be obtained.

[0041] Optionally, the white balance Planck curve of the first RGB-IR image sensor can be a curve with the ratio of the pixel value of the R channel to the pixel value of the G channel (which can be expressed as R / G ratio) as the abscissa and the ratio of the pixel value of the B channel to the pixel value of the G channel (which can be expressed as B / G ratio) as the ordinate. This curve is used to characterize the trend of the ratio of the pixel value of the R channel to the pixel value of the G channel, and the ratio of the pixel value of the B channel to the pixel value of the G channel, as a function of color temperature (which is usually in Kelvin, which can also be expressed as K). The white balance Planck curve of the first RGB-IR image sensor can be pre-calibrated and stored in memory, and the processor 140 can obtain the white balance Planck curve of the first RGB-IR image sensor by reading the memory.

[0042] Optionally, the processor 140 can estimate the color temperature of the real-world lighting environment based on the first image, the white balance Planck curve, and the first removal parameter to obtain the estimated color temperature of the real-world lighting environment; wherein, the estimated color temperature of the real-world lighting environment can also be referred to as the scene color temperature (Correlated Color Temperature, CCT) of the real-world lighting environment.

[0043] Optionally, the processor 140 can determine a second removal parameter adapted to the estimated color temperature; wherein, the second removal parameter adapted to the estimated color temperature can be understood as a parameter that, at the estimated color temperature, can balance the image color and signal-to-noise ratio, and is used to partially remove (i.e., not 100% remove) the IR components of the image. Similar to the first removal parameter, the second removal parameter may also include removal coefficients corresponding to the R channel, G channel, and B channel, respectively. The processor 140 can remove the IR components of the first image according to the removal coefficients included in the second removal parameter to obtain the second image.

[0044] In some optional embodiments of this disclosure, the removal coefficients corresponding to the R channel, G channel, and B channel in the second removal parameter are respectively represented as k1, k2, and k3. Therefore, the process of removing IR components from the first image may include: (b1) Using a preset interpolation algorithm, the first image (hereinafter referred to as the original image P3 for ease of description) is interpolated to obtain four full-resolution images with the same size as the original image P3, namely the full-resolution image P3-1 corresponding to the R channel, the full-resolution image P3-2 corresponding to the G channel, the full-resolution image P3-3 corresponding to the B channel, and the full-resolution image P3-4 corresponding to the IR channel.

[0045] (b2) For any pixel in any row and column of the original image P3 (e.g., the pixel in the i-th row and j-th column), the pixel value x1 of the pixel in the i-th row and j-th column can be obtained from the full-resolution image P3-1 corresponding to the R channel, the pixel value x2 of the pixel in the i-th row and j-th column can be obtained from the full-resolution image P3-2 corresponding to the G channel, the pixel value x3 of the pixel in the i-th row and j-th column can be obtained from the full-resolution image P3-3 corresponding to the B channel, and the pixel value x4 of the pixel in the i-th row and j-th column can be obtained from the full-resolution image P3-4 corresponding to the IR channel. Based on x4 and k1, the pixel value x1' corresponding to x1 after IR component removal can be calculated according to the following formula: Based on x4 and k2, the pixel value x2' after IR component removal corresponding to x2 can be calculated using the following formula: Based on x4 and k3, the pixel value x3' corresponding to x3 after IR component removal can be calculated using the following formula: By updating the pixel value of each pixel in the full-resolution image P3-1 corresponding to the R channel to the pixel value after removing the IR component, we can obtain the full-resolution image P4-1 corresponding to the R channel; by updating the pixel value of each pixel in the full-resolution image P3-2 corresponding to the G channel to the pixel value after removing the IR component, we can obtain the full-resolution image P4-2 corresponding to the G channel; and by updating the pixel value of each pixel in the full-resolution image P3-3 corresponding to the B channel to the pixel value after removing the IR component, we can obtain the full-resolution image P4-3 corresponding to the B channel.

[0046] (b3) According to the data arrangement rules that conform to the Bayer array, the full-resolution image P4-1 corresponding to the R channel, the full-resolution image P4-2 corresponding to the G channel, and the full-resolution image P4-3 corresponding to the B channel are sampled to obtain an image of the Bayer type, which can be used as the second image.

[0047] In an optional example, the data arrangement in the second image can be seen in [reference needed]. Figure 3-2 Then the data arrangement in the second image satisfies: The area of ​​size is treated as a repeating unit. The two pixels in the first row of this repeating unit correspond to the B channel and the G channel, respectively, and the two pixels in the second row of this repeating unit correspond to the G channel and the R channel, respectively.

[0048] Optionally, taking the case where the image processing device 14 is a smart cockpit chip as an example, the processor 140 can perform subsequent image processing on the second image, such as but not limited to de-mosaic processing, white balance compensation processing, color restoration processing, etc. On this basis, it can further run perception and recognition algorithms such as driver status monitoring and passenger posture and activity monitoring.

[0049] In the embodiments of this disclosure, the first image acquired by the first RGB-IR image sensor under a real-world lighting environment is affected by the color temperature of the light source. The white balance Planck curve of the first RGB-IR image sensor is correlated with the color temperature. A first removal parameter calibrated for a first preset color temperature is used to completely remove the IR components of the image, which helps to eliminate the adverse effects of the image IR components on color temperature estimation. Therefore, based on the image, the white balance Planck curve, and the first removal parameter, the estimated color temperature of the real-world lighting environment can be determined relatively accurately. Following a second removal parameter adapted to the estimated color temperature, IR component removal is performed on the first image to obtain a second image for subsequent image processing. Since the second removal parameter is used to partially remove the IR components of the image, removing IR components from the first image according to the second removal parameter will only remove a portion of the IR components, not all of them. This helps to avoid the significant loss in brightness and signal-to-noise ratio caused by the complete removal of IR components. Thus, there will be no obvious noise in the second image, which helps to ensure the effect of subsequent image processing.

[0050] In some optional examples, the first preset color temperature is one of several preset color temperatures. Optionally, 2856K corresponding to light source A can be one preset color temperature, 4000K corresponding to light source D40 can be another preset color temperature, 5000K corresponding to light source D50 can be yet another preset color temperature, 6500K corresponding to light source D65 can be yet another preset color temperature, and 7500K corresponding to light source D75 can be yet another preset color temperature. The first preset color temperature can be any of the above preset color temperatures.

[0051] When processor 140 determines the second removal parameter that matches the estimated color temperature, it is specifically configured as follows: Based on the difference between estimated color temperature and preset color temperature, a second preset color temperature is selected from multiple preset color temperatures; Obtain the third removal parameter corresponding to the second preset color temperature; wherein, the third removal parameter is obtained by optimizing the fourth removal parameter calibrated for the second preset color temperature, and the fourth removal parameter is a parameter used to completely remove the IR components of the image; Based on the third removal parameter, a second removal parameter that is compatible with the estimated color temperature is determined.

[0052] Optionally, the preset color temperature difference condition may include: higher than a given color temperature and closest to a given color temperature. In this case, the processor 140 can select the preset color temperature that is higher than and closest to the estimated color temperature from a variety of preset color temperatures. The selected preset color temperature can be used as a second preset color temperature. In this case, there is only one type of second preset color temperature. Alternatively, the preset color temperature difference condition may include: higher than a given color temperature and closest to a given color temperature, and lower than a given color temperature and closest to a given color temperature. In this case, the processor 140 can select the preset color temperature that is higher than and closest to the estimated color temperature and the preset color temperature that is lower than the estimated color temperature and closest to the estimated color temperature from a variety of preset color temperatures. This can select two preset color temperatures, and the two selected preset color temperatures can each be used as a second preset color temperature. In this case, there are two types of second preset color temperatures.

[0053] In one optional example, the preset color temperatures are 2856K, 4000K, 5000K, 6500K, and 7500K, with an estimated color temperature of 3000K. If the preset color temperature difference criteria include being higher than a given color temperature and closest to it, then 4000K is selected as a second preset color temperature. If the preset color temperature difference criteria include being higher than a given color temperature and closest to it, and being lower than a given color temperature and closest to it, then 2856K and 4000K are each selected as a second preset color temperature.

[0054] Optionally, the fourth removal parameter corresponding to the second preset color temperature, used to completely remove the IR components of the image, can be pre-calibrated. The calibration process for the fourth removal parameter is the same as the calibration process for the first removal parameter described above, and will not be repeated here. Alternatively, the fourth removal parameter can be pre-optimized by considering both image color and signal-to-noise ratio as constraints to obtain the third removal parameter corresponding to the second preset color temperature. This third removal parameter is stored in memory, and memory 140 can obtain it by reading from memory. Memory 140 can also determine a second removal parameter adapted to the estimated color temperature based on the third removal parameter.

[0055] In some optional embodiments of this disclosure, when the processor 140 determines a second removal parameter that matches the estimated color temperature based on a third removal parameter, it is specifically configured as follows: In response to the fact that the number of second preset color temperatures is one, a third removal parameter corresponding to one second preset color temperature is determined as a second removal parameter that is adapted to the estimated color temperature; or, Since there are two types of second preset color temperatures, interpolation is performed between the third removal parameters corresponding to the two second preset color temperatures to obtain second removal parameters that are adapted to the estimated color temperature.

[0056] As described above, if the preset color temperature difference conditions include: higher than the given color temperature and closest to the given color temperature, then there is only one second preset color temperature. In this case, the processor 140 can directly determine the third removal parameter corresponding to one second preset color temperature as the second removal parameter that is compatible with the estimated color temperature, thus efficiently and quickly determining the second removal parameter.

[0057] As described above, if the preset color temperature difference conditions include: higher than a given color temperature and closest to a given color temperature, and lower than a given color temperature and closest to a given color temperature, then there are two types of second preset color temperatures. In this case, the processor 140 can perform linear or nonlinear interpolation between the third removal parameters corresponding to the two second preset color temperatures to obtain new removal parameters. These new removal parameters can be used as second removal parameters adapted to the estimated color temperature. Thus, by using the third removal parameters corresponding to both second preset color temperatures as the basis for determining the second removal parameters, it is beneficial to avoid introducing large errors from a single third removal parameter, thereby ensuring the accuracy and reliability of the determined second removal parameters.

[0058] In the embodiments of this disclosure, the estimated color temperature and the preset color temperature difference condition can be used as screening criteria to reasonably select a second preset color temperature from multiple preset color temperatures. The third removal parameter corresponding to the second preset color temperature can be obtained by optimizing the fourth removal parameter corresponding to the second preset color temperature by taking both image color and signal-to-noise ratio as constraints (the specific implementation method can be found in the application of evaluation index parameters including total color deviation, color deviation, signal-to-noise ratio, etc. below). Then, using the third removal parameter as the basis for determining the second removal parameter is beneficial for determining a second removal parameter that can take both image color and signal-to-noise ratio into account. In this way, according to the second removal parameter, the first image is subjected to IR component removal to obtain the second image, and subsequent image processing is performed on the obtained second image, which helps to ensure the effect of subsequent image processing.

[0059] In some optional examples, the processor 140 optimizes the fourth removal parameter for the second preset color temperature calibration to obtain the third removal parameter, specifically configured as follows: A third image is acquired by the second RGB-IR image sensor under a standard light source environment that meets the second preset color temperature; wherein the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor. The fourth removal parameter is adjusted to obtain the fifth removal parameter; wherein the fourth removal parameter includes the first removal coefficients corresponding to the R channel, G channel and B channel respectively, and the fifth removal parameter includes the second removal coefficients corresponding to the R channel, G channel and B channel respectively, each second removal coefficient is less than or equal to the corresponding first removal coefficient, and at least one second removal coefficient is different from the corresponding first removal coefficient. The third removal parameter is determined based on the third image and the fifth removal parameter.

[0060] Optionally, the fact that the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor can be understood as follows: the effective imaging surface physical size, pixel size, pixel array arrangement, total resolution, spectral response curve (including the spectral response curve for the visible light band and the spectral response curve for the infrared band), and cover glass parameters of the second RGB-IR image sensor are all the same as those of the first RGB-IR image sensor.

[0061] Optionally, the standard light source environment conforming to the second preset color temperature refers to a controlled light source environment constructed using professional equipment (such as a standard light box), which conforms to international standards in terms of spectral power distribution, color rendering, illuminance, and uniformity, and whose color temperature is the second preset color temperature. The processor 140 can acquire the RGB-IR image collected by the second RGB-IR image sensor in the standard light source environment conforming to the second preset color temperature, and use this RGB-IR image as the third image.

[0062] Optionally, the fourth removal parameter includes first removal coefficients corresponding to the R, G, and B channels, respectively. The processor 140 can adjust the fourth removal parameter according to a preset adjustment rule to obtain a fifth removal parameter including second removal coefficients corresponding to the R, G, and B channels, respectively. The preset adjustment rule may include, for example, proportionally reducing the first removal coefficients corresponding to the R, G, and B channels, or randomly reducing the first removal coefficients corresponding to the R, G, and B channels. The processor 140 can determine the third removal parameter based on the third image and the fifth removal coefficients.

[0063] In some optional embodiments of this disclosure, when the processor 140 determines the third removal parameter based on the third image and the fifth removal parameter, it is specifically configured as follows: The third image is subjected to IR component removal according to the second removal coefficient included in the fifth removal parameter to obtain the fourth image; Based on the fourth image, determine the fifth image, which belongs to the visualization type; The fifth image is evaluated to obtain evaluation index parameters; In response to the evaluation index parameter being within the preset parameter range, the fifth removal parameter is determined as the third removal parameter; If the evaluation index parameter is outside the preset parameter range, return to perform the operation of adjusting the fourth removal parameter to obtain the fifth removal parameter.

[0064] Optionally, the processor 140 can remove the IR components from the third image according to the second removal coefficients included in the fifth removal parameter to obtain the fourth image. The specific method for obtaining the fourth image is described above in the section on obtaining the second image, and will not be repeated here. Similar to the second image, the fourth image can be a Bayer type image. Therefore, the processor 140 can convert the fourth image from the Bayer type to a visualization type to obtain the fifth image. For example, the processor 140 can convert the fourth image from the Bayer type to the RGB type, and the resulting RGB image can be used as the fifth image. Alternatively, the processor 140 can convert the fourth image from the Bayer type to the RGB type, and then from the RGB type to the YUV type (where Y represents luminance, and U and V represent chrominance, respectively), and the resulting YUV image can be used as the fifth image.

[0065] Optionally, the processor 140 can evaluate the fifth image to obtain evaluation index parameters. For example, the processor 140 can evaluate the color accuracy of the fifth image to obtain the total color deviation (ΔE), which represents the magnitude of the total color difference perceived by the human eye, and the color deviation (ΔC), which represents the deviation of color intensity or saturation. ΔE and ΔC can be used as evaluation index parameters. As another example, the processor 140 can evaluate the noise level of the fifth image to obtain the signal-to-noise ratio (SNR), which represents the ratio of useful signal strength to noise strength. SNR can be used as an evaluation index parameter. For ΔE, ΔC, and SNR, corresponding preset parameter ranges can be preset; these preset parameter ranges can be understood as reasonable ranges for the parameters. If the evaluation index parameters include ΔE, ΔC, and SNR, and all of ΔE, ΔC, and SNR are within their respective preset parameter ranges, the processor 140 can determine that the color and SNR of the fifth image meet the requirements, and the evaluation of the fifth image passes. Furthermore, the fifth removal parameter applied to the fifth image can be determined as the third removal parameter. If the evaluation metrics include ΔE, ΔC, and SNR, and at least one of ΔE, ΔC, and SNR falls outside the corresponding preset parameter range, the processor 140 can determine that the color and / or signal-to-noise ratio of the fifth image do not meet the requirements, and the evaluation of the fifth image fails. It can then return to the process of adjusting the fourth removal parameter to obtain the fifth removal parameter. By performing this operation, a new fifth removal parameter can be obtained. Based on the new fifth removal parameter, a new fourth image, a new fifth image, and new evaluation metrics are obtained sequentially. Based on this, it is determined whether the evaluation of the new fifth image passes. If the evaluation of the new fifth image passes, the processor 140 can determine the new fifth removal parameter as the third removal parameter; otherwise, it can return to the process of adjusting the fourth removal parameter again to obtain the fifth removal parameter. Subsequent operations follow the same logic, which will not be elaborated further here.

[0066] In an optional example, processor 140 can reduce each of the first removal coefficients in the fourth removal parameter to 90% of its original value to obtain the fifth removal coefficient for the first time. Based on the fifth removal coefficient obtained in the first instance, processor 140 can sequentially obtain the fourth image, the fifth image, and the evaluation index parameter. If the evaluation index parameter is outside the preset parameter range, processor 140 can reduce each of the first removal coefficients in the fourth removal parameter to 85% of its original value to obtain the fifth removal coefficient for the second time. Based on the fifth removal coefficient obtained in the second instance, processor 140 can sequentially obtain the fourth image, the fifth image, and the evaluation index parameter. If the evaluation index parameter is outside the preset parameter range, processor 140 can reduce each of the first removal coefficients in the fourth removal parameter to 80% of its original value to obtain the fifth removal coefficient for the third time. This process continues in the same manner, and will not be elaborated further here.

[0067] In this way, by obtaining the fourth image, the fifth image, and the evaluation index parameters in sequence, and combining them with the preset parameter range, the processor 140 can search for a fifth removal parameter that can take into account both image color and signal-to-noise ratio as the third removal parameter. Using the third removal parameter as the basis for determining the second removal parameter is beneficial for determining the second removal parameter that can take into account both image color and signal-to-noise ratio.

[0068] Of course, the implementation of determining the third removal parameter based on the third image and the fifth removal parameter by the processor 140 is not limited to this. For example, after obtaining the fifth image based on the third image and the fifth removal parameter, the processor 140 can control the display screen to display the fifth image and obtain a first user input operation for the fifth image. The first user input operation includes, but is not limited to, text input, voice input, touch input, etc. If the first user input operation indicates that the fifth image displayed on the display screen meets the requirements, the processor 140 can determine the fifth removal parameter as the third removal parameter. If the first user input operation indicates that the fifth image displayed on the display screen does not meet the requirements, the processor 140 can return to perform the operation of adjusting the fourth removal parameter to obtain the fifth removal parameter.

[0069] In the embodiments of this disclosure, the fourth removal parameter is adjusted according to the preset adjustment rules, which can efficiently and quickly obtain the fifth removal parameter. Combined with the third image acquired by the second RGB-IR image sensor under a standard light source environment that meets the second preset color temperature, a certain search algorithm can be used to search for the third removal parameter that can take into account both image color and signal-to-noise ratio, which helps to ensure the rationality of the third removal parameter.

[0070] In some optional examples, when processor 140 determines the estimated color temperature of the real-world lighting environment based on the first image, the white balance Planck curve, and the first removal parameter, it is specifically configured as follows: According to the first removal parameters, the IR component is removed from the first image to obtain the sixth image; In the coordinate system where the white balance Planck curve is located, determine the first position corresponding to each of the multiple image blocks included in the sixth image; Based on the white balance Planck curve and the first position corresponding to multiple image blocks, the estimated color temperature of the real-world light source environment is determined.

[0071] Optionally, the processor 140 can remove IR components from the first image according to the removal coefficients included in the first removal parameters to obtain a sixth image. The specific method for obtaining the sixth image is the same as the description of the specific method for obtaining the second image above, and will not be repeated here. Similar to the second image, the sixth image can be a Bayer-type image.

[0072] Optionally, the processor 140 can divide the sixth image into multiple image blocks, for example, into equal parts. There are several image patches; where the values ​​of m and n can be preset, for example, m and n can both be set to 32 or 64. For... For each image patch, the first ratio of the mean pixel value of the R channel to the mean pixel value of the G channel, and the second ratio of the mean pixel value of the B channel to the mean pixel value of the G channel, can be calculated. In the coordinate system of the white balance Planck curve, the horizontal axis represents the first ratio corresponding to the image patch, and the vertical axis represents the position of the second ratio corresponding to the image patch. This position can be used as the first location of the image patch. Similarly, the processor 140 can determine the first locations corresponding to multiple image patches. Combined with the white balance Planck curve, the processor 140 can determine the estimated color temperature of the real-world lighting environment.

[0073] In some optional embodiments of this disclosure, when the processor 140 determines the estimated color temperature of the real-world light source environment based on the white balance Planck curve and the first positions corresponding to the multiple image patches, it is specifically configured as follows: Based on the white balance Planck curve and the first positions corresponding to multiple image patches, the weights of the first positions corresponding to multiple image patches are determined. By using the weights of the first positions corresponding to multiple image blocks respectively, the second position is obtained by weighting the first positions corresponding to multiple image blocks respectively; Determine multiple reference locations to define the white balance Planck curve; Based on the second position and preset distance conditions, the target reference position is selected from multiple reference positions; Based on the preset color temperature that maps to the target reference position among various preset color temperatures, the estimated color temperature of the actual light source environment is determined.

[0074] Optionally, the processor 140 can determine the distance information (e.g., distance values) between the first positions corresponding to multiple image patches and the white balance Planck curve, and determine the weights of the first positions corresponding to the multiple image patches based on the distance information. For example, a function can be pre-set where the independent variable is the distance information, the dependent variable is the weight, and the independent and dependent variables are negatively correlated. For each image patch among the multiple image patches, the distance information corresponding to that image patch can be used as the value of the independent variable and substituted into the above function for calculation to obtain the corresponding value of the dependent variable. The obtained value can be used as the weight of the first position corresponding to that image patch.

[0075] Of course, the method for determining the weights of the first positions corresponding to multiple image patches is not limited to this. For example, the white balance Planck curve can be used as the center line to determine a region of a set size (for ease of description, this region can be referred to as the target region). For each image patch, if the first position corresponding to the image patch is outside the target region, the weight of the first position corresponding to the image patch can be determined to be zero. If the first position corresponding to the image patch is within the target region, the weight of the first position corresponding to the image patch can be determined based on the distance information between the first position corresponding to the image patch and the white balance Planck curve, as well as the function mentioned above.

[0076] Optionally, the processor 140 can use the weights of the first positions corresponding to the multiple image blocks to perform a weighted average of the first positions corresponding to the multiple image blocks to obtain the second position. Assuming the above... =N, where the first positions corresponding to multiple image patches can be represented as R1, R2, R3, ..., RN, and the weights of the first positions corresponding to multiple image patches can be represented as Q1, Q2, Q3, ..., QN, and the second position can be represented as R. Then:

[0077] For ease of description, the coordinate system in which the white balance Planck curve is located will be referred to as the target coordinate system in the following text. Optionally, the calibration procedure for the white balance Planck curve may include: (c1) Under each of the various preset color temperatures, the second RGB-IR image sensor is used to acquire the RGB-IR image of the 24-color card (or gray card) (for ease of description, it will be referred to as the fourth original image).

[0078] (c2) For each of the multiple preset color temperatures, the corresponding parameters for completely removing the IR components of the image can be calibrated according to the relevant description above (for example, for the first preset color temperature, the first removal parameter can be calibrated). For each of the multiple preset color temperatures, the IR components of the fourth original image corresponding to that preset color temperature can be removed according to the parameters for completely removing the IR components of the image corresponding to that preset color temperature, to obtain an image belonging to the Bayer format.

[0079] (c3) For each of the various preset color temperatures, a target gray area is selected from the corresponding Bayer format image. For example, a second user input operation can be received, and the gray area selected by the second user input operation in the Bayer format image can be used as the target gray area. The second user input operation includes, but is not limited to, text input, voice input, and touch input. For the target gray area, a third ratio of the mean pixel value of the R channel to the mean pixel value of the G channel, and a fourth ratio of the mean pixel value of the B channel to the mean pixel value of the G channel can be calculated. These third and fourth ratios can form a ratio group corresponding to the preset color temperature. In a similar manner, ratio groups corresponding to various preset color temperatures can be obtained.

[0080] (c4) For each preset color temperature in the various preset color temperature ratio groups, the third ratio in the ratio group can be used as the abscissa and the fourth ratio in the ratio group as the ordinate to locate a position in the target coordinate system (this position can be considered to map to that preset color temperature). In this way, multiple positions in the target coordinate system can be located, and these positions can be smoothly connected in sequence to form a curve, which can be used as the calibrated white balance Planck curve.

[0081] Optionally, the processor 140 can determine multiple positions in the target coordinate system located based on the ratio groups corresponding to various preset color temperatures as multiple reference positions for defining the white balance Planck curve.

[0082] Optionally, the preset distance condition may include being located on the white balance Planck curve and closest to a given position. The processor 140 can determine a third position on the white balance Planck curve that is closest to the second position; this third position can serve as a position satisfying the preset distance condition with the second position. The processor 140 can also search for the nearest reference position to the left and right of the third position, thus finding two reference positions that can serve as two target reference positions. The processor 140 can perform linear or non-linear interpolation between the two preset color temperatures mapped to the two target reference positions to obtain a new color temperature, which can serve as an estimated color temperature for the actual light source environment.

[0083] In some embodiments, the third position is the same as one of the plurality of reference positions. The processor 140 can determine the reference position as the target reference position and determine the preset color temperature mapped to the target reference position as the estimated color temperature of the actual light source environment.

[0084] In this way, by combining the white balance Planck curve and the first positions corresponding to multiple image patches, the weights of the first positions corresponding to each image patch can be reasonably determined. For example, the weights of the first positions corresponding to multiple image patches can be determined according to the following rules: the closer the first position corresponding to an image patch is to the white balance Planck curve, the greater its weight; the farther the first position corresponding to an image patch is from the white balance Planck curve, the smaller its weight. By using the weights of the first positions corresponding to multiple image patches, a weighted average can be applied to the first positions, which can efficiently and quickly obtain the second position. The second position reflects the overall influence of the light source color temperature on the image. Combining the second position and a preset distance condition, a target reference position can be reasonably selected from multiple reference positions so that the estimated color temperature of the real-world light source environment can be determined efficiently and reliably based on the preset color temperature mapped to the target reference position.

[0085] Of course, the implementation method for determining the estimated color temperature of the real-world lighting environment based on the Planck curve of white balance and the first positions corresponding to multiple image patches is not limited to this. For example, the target area mentioned above can be determined first, and each first position located within the target area can be selected from the first positions corresponding to multiple image patches; then, the weight of each first position located within the target area can be determined, and the determined weights can be used to weight each first position located within the target area to obtain a second position; then, based on the second position and a preset distance condition, a target reference position can be selected, and the estimated color temperature of the real-world lighting environment can be determined based on the preset color temperature mapped to the target reference position.

[0086] In the embodiments of this disclosure, the white balance Planck curve and the first positions corresponding to multiple image blocks can be used as the basis for determining the estimated color temperature of the real-world light source environment. Since multiple image blocks belong to the sixth image, and the sixth image is obtained by removing the IR components of the first image according to the first removal parameter used to completely remove the image's IR components, this helps to eliminate the adverse effects caused by the residue of the image's IR components. These adverse effects may include, for example, the disordered and discrete statistical landing points of the white area under different color temperatures (e.g., ...). Figure 3-3 (Illustrated case), and the distribution pattern of the white balance Planck curve (e.g.) Figure 3-4 The illustrated situation does not match, resulting in a white balance color cast. Furthermore, in the embodiments of this disclosure, the second image is obtained by removing the IR components of the first image according to a second removal parameter used for partially removing the IR components of the image, which helps to avoid a significant loss of signal-to-noise ratio. Therefore, the embodiments of this disclosure can achieve a balance between white balance and noise.

[0087] In some optional examples, such as Figure 4 As shown, the processor 140 includes: The first processing path 1402 is configured to perform operations to determine the estimated color temperature of the real-world light source environment based on the first image, the white balance Planck curve, and the first removal parameter, and to determine the second removal parameter that is compatible with the estimated color temperature. The second processing path 1404 is configured to perform an operation of removing IR components from the first image according to the second removal parameters to obtain the second image.

[0088] Optionally, the processor 140 may include two processing paths: a first processing path 1402 and a second processing path 1404. The first processing path 1402 can be used to perform white balance statistical calculations, and the second processing path 1404 can be used to perform IR component removal operations. The second processing path 1404 can be electrically connected to the first processing path 1402. In an optional example, the first processing path 1402 may also be referred to as a white balance statistical calculation path; and the second processing path 1404 may also be referred to as an IR component removal path.

[0089] In the embodiments of this disclosure, the first processing path 1402 can remove the IR components of the first image according to the first removal parameters for completely removing the IR components of the image, resulting in a sixth image. Combined with the Planck curve for white balance, the estimated color temperature of the real-world lighting environment is determined, which helps eliminate the white balance color cast caused by the residual IR components of the image. Furthermore, the second processing path 1404 can remove the IR components of the first image according to the second removal parameters for partially removing the IR components, which helps avoid significant signal-to-noise ratio loss. Therefore, in the embodiments of this disclosure, the coordinated operation of the two processing paths achieves a balance between white balance and noise.

[0090] In some optional examples, calibration of the parameters and calibration of the white balance Planck curve can be performed during the calibration phase.

[0091] The calibration procedure for removing parameters can be found in [reference needed]. Figure 5-1For example, under the preset color temperature corresponding to each of the standard light sources (A, D40, D50, D65, and D75), a second RGB-IR image sensor can be used to acquire RGB-IR images of a 24-color chart without and with an IR cutoff filter. For each of the two RGB-IR images corresponding to each preset color temperature, interpolation processing can be performed to obtain four full-resolution images. Based on this, the first to seventh average values ​​mentioned above can be calculated for that preset color temperature, and the IR removeratio for 100% removal of the image IR components can be further obtained (e.g., the first removal parameter mentioned above). Next, the IR remove ratios for 100% removal of the image IR components corresponding to various preset color temperatures can be optimized to balance image color and signal-to-noise ratio, resulting in optimized non-100% removal IR remove ratios for various preset color temperatures (e.g., the third removal parameter mentioned above). Optionally, it can also output the optimized IR removeratio (not 100% removal) corresponding to various preset color temperatures.

[0092] The calibration procedure for the white balance Planck curve can be found in [reference needed]. Figure 5-2 For example, under the preset color temperature corresponding to each of the standard light sources (A, D40, D50, D65, and D75), the second RGB-IR image sensor can acquire the RGB-IR image of a 24-color chart and, based on... Figure 5-1 The IRremove ratio obtained by calibration is used to remove the IR components of the image 100%. The IR components are removed from each of the acquired RGB-IR images to obtain the corresponding images belonging to the Bayer format. The pixel value information in the target gray area of ​​the obtained images belonging to the Bayer format is statistically analyzed, and the white balance Planck curve can be constructed accordingly.

[0093] like Figure 5-2 As shown, in the actual image processing stage, the processor 140 can acquire the RGB-IR image captured by the first RGB-IR image sensor under real-world lighting conditions. According to the IR remove ratio corresponding to any preset color temperature for 100% IR component removal, the RGB-IR image can be subjected to IR component removal to obtain the sixth image mentioned above. The sixth image can be evenly divided into... Each image patch can be determined by the processor 140 in the target coordinate system. By taking the first position corresponding to each image patch and combining it with the white balance Planck curve, the estimated color temperature of the real-world light source environment can be determined.

[0094] Optionally, such as Figure 5-3 As shown, the processor 140 may include a white balance statistical calculation path and an IR component removal path. The white balance statistical calculation path can obtain an IR remove ratio for 100% removal of the image's IR components and perform IR component removal on the RGB-IR image accordingly. The white balance statistical calculation path can also determine an estimated color temperature by combining the white balance Planck curve and each first position. Based on the estimated color temperature, the white balance statistical calculation path can also determine an IR remove ratio for non-100% removal of the image's IR components. The IR component removal path can perform IR component removal on the RGB-IR image according to the non-100% removal IR remove ratio determined by the white balance statistical calculation path, thereby obtaining an image with non-100% IR component removal, such as the second image mentioned above. The IR component removal path can also output the image with non-100% IR component removal and perform subsequent image processing, such as de-mosaic processing, white balance compensation processing, and color restoration processing.

[0095] Thus, the embodiments of this disclosure can separate white balance statistical calculation and IR component removal into two processing paths; wherein, the white balance statistical calculation path can remove IR components 100%, so that the processed image meets the distribution law of the white balance Planck curve and the white balance can be correctly corrected; the IR component removal path can remove IR components non-100% to avoid significant loss of brightness and signal-to-noise ratio, which is beneficial to ensuring the effect of subsequent image processing.

[0096] Exemplary methods Figure 6 This is a schematic flowchart illustrating an image processing method provided by some exemplary embodiments of this disclosure. The image processing method can be applied to the image processing apparatus 14 in any of the above embodiments. Figure 6 As shown, the image processing methods include: Step 610: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Step 620: Obtain the first removal parameter for the first preset color temperature calibration; wherein, the first removal parameter is a parameter used to completely remove the IR components of the image; Step 630: Obtain the white balance Planck curve of the first RGB-IR image sensor; Step 640: Based on the first image, the white balance Planck curve, and the first removal parameter, determine the estimated color temperature of the real-world lighting environment; Step 650: Determine a second removal parameter that matches the estimated color temperature; wherein the second removal parameter is a parameter used to partially remove the IR components of the image; Step 660: Remove the IR components from the first image according to the second removal parameters to obtain the second image.

[0097] In some optional examples, the first preset color temperature is one of a variety of preset color temperatures; like Figure 7 As shown, step 650 includes: Step 710: Based on the estimated color temperature and the difference between preset color temperatures, select a second preset color temperature from multiple preset color temperatures; Step 720: Obtain the third removal parameter corresponding to the second preset color temperature; wherein, the third removal parameter is obtained by optimizing the fourth removal parameter calibrated for the second preset color temperature, and the fourth removal parameter is a parameter used to completely remove the IR components of the image; Step 730: Based on the third removal parameter, determine the second removal parameter that matches the estimated color temperature.

[0098] In some optional examples, step 730 includes: In response to the fact that the number of second preset color temperatures is one, a third removal parameter corresponding to one second preset color temperature is determined as a second removal parameter that is adapted to the estimated color temperature; or, Since there are two types of second preset color temperatures, interpolation is performed between the third removal parameters corresponding to the two second preset color temperatures to obtain second removal parameters that are adapted to the estimated color temperature.

[0099] In some optional examples, such as Figure 8 As shown, step 720 includes: Step 810: Acquire a third image from the second RGB-IR image sensor under a standard light source environment that meets the second preset color temperature; wherein the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor. Step 820: Adjust the fourth removal parameter to obtain the fifth removal parameter; wherein the fourth removal parameter includes the first removal coefficients corresponding to the R channel, G channel and B channel respectively, and the fifth removal parameter includes the second removal coefficients corresponding to the R channel, G channel and B channel respectively, each second removal coefficient is less than or equal to the corresponding first removal coefficient, and at least one second removal coefficient is different from the corresponding first removal coefficient. Step 830: Determine the third removal parameter based on the third image and the fifth removal parameter.

[0100] In some optional examples, such as Figure 9As shown, step 830 includes: Step 910: Remove the IR components from the third image according to the second removal coefficient included in the fifth removal parameter to obtain the fourth image; Step 920: Based on the fourth image, determine the fifth image that belongs to the visualization type; Step 930: Evaluate the fifth image to obtain evaluation index parameters; Step 940: In response to the evaluation index parameter being within the preset parameter range, the fifth removal parameter is determined as the third removal parameter; Step 950: In response to the evaluation index parameter being outside the preset parameter range, return to perform the operation of adjusting the fourth removal parameter to obtain the fifth removal parameter.

[0101] In some optional examples, such as Figure 10 As shown, step 640 includes: Step 1010: Remove the IR components from the first image according to the first removal parameters to obtain the sixth image; Step 1020: In the coordinate system where the white balance Planck curve is located, determine the first position corresponding to each of the multiple image blocks included in the sixth image; Step 1030: Based on the white balance Planck curve and the first positions corresponding to multiple image blocks, determine the estimated color temperature of the real-world light source environment.

[0102] In some optional examples, such as Figure 11 As shown, step 1030 includes: Step 1110: Based on the white balance Planck curve and the first positions corresponding to the multiple image blocks respectively, determine the weights of the first positions corresponding to the multiple image blocks respectively; Step 1120: Using the weights of the first positions corresponding to the multiple image blocks respectively, the first positions corresponding to the multiple image blocks are weighted to obtain the second position; Step 1130: Determine multiple reference locations for defining the white balance Planck curve; Step 1140: Based on the second position and preset distance conditions, filter the target reference position from multiple reference positions; Step 1150: Determine the estimated color temperature of the actual light source environment based on the preset color temperature mapped to the target reference position among multiple preset color temperatures.

[0103] In some alternative examples, steps 640 and 650 are executed by a first processing path in the processor included in the image processing apparatus, and step 660 is executed by a second processing path in the processor included in the image processing apparatus.

[0104] In the methods disclosed herein, the various optional embodiments, optional implementation methods and optional examples disclosed in the above exemplary device section can be flexibly selected and combined as needed to achieve the corresponding functions and effects, and this disclosure does not list them all.

[0105] The beneficial technical effects corresponding to the exemplary embodiments of this method can be found in the corresponding beneficial technical effects of the exemplary device section above, and will not be repeated here.

[0106] Exemplary electronic devices Figure 12 The illustration shows a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 1200 includes one or more processors 1210 and a memory 1220.

[0107] The processor 1210 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1200 to perform desired functions.

[0108] The memory 1220 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1210 may execute one or more computer program instructions to implement the image processing method or instruction execution method and / or other desired functions of the various embodiments of this disclosure described above.

[0109] In one example, the electronic device 1200 may also include an input device 1230 and an output device 1240, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0110] The input device 1230 may also include, for example, a keyboard, a mouse, etc.

[0111] The output device 1240 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0112] Of course, for the sake of simplicity, Figure 12Only some of the components of the electronic device 1200 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1200 may include any other suitable components depending on the specific application.

[0113] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0114] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0115] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0116] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0117] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. The specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0118] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An image processing apparatus, comprising: Processor, the processor being configured to: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain a first removal parameter for a first preset color temperature calibration; wherein the first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the white balance Planck curve, and the first removal parameter, the estimated color temperature of the real-world light source environment is determined; Determine a second removal parameter that is compatible with the estimated color temperature; wherein the second removal parameter is a parameter used to partially remove the IR components of the image; According to the second removal parameters, the first image is subjected to IR component removal to obtain the second image.

2. The image processing apparatus according to claim 1, wherein, The first preset color temperature is one of a variety of preset color temperatures; When the processor determines the second removal parameter that matches the estimated color temperature, it is specifically configured as follows: Based on the estimated color temperature and the preset color temperature difference condition, a second preset color temperature is selected from a variety of preset color temperatures; Obtain the third removal parameter corresponding to the second preset color temperature; wherein, the third removal parameter is obtained by optimizing the fourth removal parameter calibrated for the second preset color temperature, and the fourth removal parameter is a parameter used to completely remove the IR components of the image; Based on the third removal parameter, a second removal parameter that is compatible with the estimated color temperature is determined.

3. The image processing apparatus according to claim 2, wherein, When the processor determines the second removal parameter that matches the estimated color temperature based on the third removal parameter, it is specifically configured as follows: In response to the fact that the number of the second preset color temperatures is one, the third removal parameter corresponding to one of the second preset color temperatures is determined as the second removal parameter that is adapted to the estimated color temperature; or, In response to the fact that there are two types of the second preset color temperature, interpolation is performed between the third removal parameters corresponding to the two types of the second preset color temperature to obtain the second removal parameters that are adapted to the estimated color temperature.

4. The image processing apparatus according to claim 2, wherein, When the processor optimizes the fourth removal parameter for the second preset color temperature calibration to obtain the third removal parameter, it is specifically configured as follows: A third image is acquired by a second RGB-IR image sensor under a standard light source environment that meets the second preset color temperature; wherein the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor. The fourth removal parameter is adjusted to obtain the fifth removal parameter; wherein the fourth removal parameter includes first removal coefficients corresponding to the R channel, G channel and B channel respectively, and the fifth removal parameter includes second removal coefficients corresponding to the R channel, G channel and B channel respectively, each second removal coefficient is less than or equal to the corresponding first removal coefficient, and at least one second removal coefficient is different from the corresponding first removal coefficient. The third removal parameter is determined based on the third image and the fifth removal parameter.

5. The image processing apparatus according to claim 4, wherein, When the processor determines the third removal parameter based on the third image and the fifth removal parameter, it is specifically configured as follows: The third image is subjected to IR component removal according to the second removal coefficient included in the fifth removal parameter to obtain the fourth image; Based on the fourth image, determine the fifth image, which belongs to the visualization type; The fifth image is evaluated to obtain evaluation index parameters; In response to the evaluation index parameter being within the preset parameter range, the fifth removal parameter is determined as the third removal parameter; In response to the evaluation index parameter being outside the preset parameter range, the operation of adjusting the fourth removal parameter to obtain the fifth removal parameter is returned.

6. The image processing apparatus according to claim 1, wherein, When the processor determines the estimated color temperature of the real-world light source environment based on the first image, the white balance Planck curve, and the first removal parameter, it is specifically configured as follows: According to the first removal parameters, the first image is subjected to IR component removal to obtain the sixth image; In the coordinate system where the white balance Planck curve is located, determine the first positions corresponding to the multiple image blocks included in the sixth image; Based on the white balance Planck curve and the first positions corresponding to the multiple image blocks, the estimated color temperature of the real-world light source environment is determined.

7. The image processing apparatus according to claim 6, wherein, When the processor determines the estimated color temperature of the real-world light source environment based on the white balance Planck curve and the first positions corresponding to the multiple image patches, it is specifically configured as follows: Based on the white balance Planck curve and the first positions corresponding to the multiple image patches respectively, the weights of the first positions corresponding to the multiple image patches are determined. By using the weights of the first positions corresponding to the multiple image blocks respectively, the first positions corresponding to the multiple image blocks are weighted to obtain the second position; Determine multiple reference positions for defining the white balance Planck curve; Based on the second position and the preset distance condition, a target reference position is selected from multiple reference positions; The estimated color temperature of the actual light source environment is determined based on the preset color temperature that maps to the target reference position among various preset color temperatures.

8. The image processing apparatus according to any one of claims 1-7, wherein, The processor includes: The first processing path is configured to perform the operation of determining the estimated color temperature of the real-world light source environment based on the first image, the white balance Planck curve, and the first removal parameter, and the operation of determining the second removal parameter that is compatible with the estimated color temperature; The second processing path is configured to perform the operation of removing IR components from the first image according to the second removal parameters to obtain the second image.

9. An image processing method, comprising: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain a first removal parameter for a first preset color temperature calibration; wherein the first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the white balance Planck curve, and the first removal parameter, the estimated color temperature of the real-world light source environment is determined; Determine a second removal parameter that is compatible with the estimated color temperature; wherein the second removal parameter is a parameter used to partially remove the IR components of the image; According to the second removal parameters, the first image is subjected to IR component removal to obtain the second image.

10. A computer-readable storage medium storing a computer program that is executed by a processor to implement the image processing method of claim 9.

11. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method of claim 9.

Images