Image processing device and method, storage medium and electronic equipment

Abstract

The invention discloses an image processing device and method, a storage medium and electronic equipment. The image processing device comprises a processor, and the processor is configured to obtain a first image collected by a first RGB-IR image sensor in a live-action light source environment; acquiring a first removal parameter calibrated for the first preset color temperature; the first removal parameter is used for completely removing the IR component of the image; obtaining a white balance Planck curve of the first RGB-IR image sensor; based on the first image, the white balance Planck curve and a first removal parameter corresponding to the first preset color temperature, determining an estimated color temperature of the live-action light source environment; determining the change trend of the estimated color temperature; determining a second removal parameter based on the estimated color temperature and the variation trend; the second removal parameter is used for partially removing the IR component of the image; and performing IR component removal on the first image according to the second removal parameter to obtain a second image. According to the invention, great brightness and signal-to-noise ratio loss can be avoided, and the subsequent image processing effect can be guaranteed.

Description

Technical Field

[0001] This disclosure relates to image technology, and in particular to an image processing apparatus, method, storage medium, and electronic device. Background Technology

[0002] Currently, RGB-IR image sensors are being used more and more widely; where R represents red, G represents green, B represents blue, and IR represents infrared.

[0003] Generally speaking, for images captured by RGB-IR image sensors during the day, the IR component needs to be removed from the image before subsequent image processing. However, the removal methods in related technologies will result in a significant loss of brightness and signal-to-noise ratio (SNR), affecting the effect of subsequent image processing. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides an image processing apparatus, method, storage medium, and electronic device.

[0005] According to one aspect of the present disclosure, an image processing apparatus is provided, comprising: a processor configured to: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain a first removal parameter for a first preset color temperature calibration; wherein the first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the white balance Planck curve, and the first removal parameter corresponding to the first preset color temperature, the estimated color temperature of the real-world light source environment is determined; Determine the trend of the estimated color temperature; Based on the estimated color temperature and the changing trend, a second removal parameter is determined; wherein, the second removal parameter is a parameter used to partially remove the IR component of the image; According to the second removal parameters, the first image is subjected to IR component removal to obtain the second image.

[0006] According to another aspect of the present disclosure, an image processing method is provided, comprising: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain a first removal parameter for a first preset color temperature calibration; wherein the first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the white balance Planck curve, and the first removal parameter corresponding to the first preset color temperature, the estimated color temperature of the real-world light source environment is determined; Determine the trend of the estimated color temperature; Based on the estimated color temperature and the changing trend, a second removal parameter is determined; wherein, the second removal parameter is a parameter used to partially remove the IR component of the image; According to the second removal parameters, the first image is subjected to IR component removal to obtain the second image.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that is executed by a processor to implement the above-described image processing method.

[0008] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method described above.

[0009] According to another aspect of the present disclosure, a computer program product is provided that, when instructions in the computer program product are executed by a processor, performs the above-described image processing method.

[0010] Based on the image processing apparatus, method, storage medium, electronic device, and program product provided in the above embodiments of this disclosure, the first image acquired by the first RGB-IR image sensor in a real-world lighting environment is affected by the color temperature of the light source. The white balance Planck curve of the first RGB-IR image sensor is related to the color temperature. A first removal parameter calibrated for a first preset color temperature is used to completely remove the IR components of the image, which helps to eliminate the adverse effects of the image IR components on color temperature estimation. Therefore, based on the first image, the white balance Planck curve, and the first removal parameter corresponding to the first preset color temperature, the estimated color temperature of the real-world lighting environment can be determined relatively accurately. Based on the estimated color temperature and its changing trend, a second removal parameter can be adaptively determined to remove the IR components from the first image, thereby obtaining a second image for subsequent image processing. Since the second removal parameter is used to partially remove the IR components of the image, removing the IR components of the first image according to the second removal parameter will only remove a portion of the IR components in the first image, rather than removing all of the IR components. This helps to avoid the significant loss of brightness and signal-to-noise ratio caused by the complete removal of the IR components. In this way, there will be no obvious noise on the second image, which helps to ensure the effect of subsequent image processing. Attached Figure Description

[0011] Figure 1 This is a system architecture diagram to which some exemplary embodiments of this disclosure apply.

[0012] Figure 2 This is a schematic diagram of the structure of an image processing apparatus provided by some exemplary embodiments of the present disclosure.

[0013] Figure 3-1 This is a schematic diagram of the data arrangement in the first image in some exemplary embodiments of this disclosure.

[0014] Figure 3-2 This is a schematic diagram of the data arrangement in the second image in some exemplary embodiments of this disclosure.

[0015] Figure 4 This is one of the schematic diagrams of a first preset mapping relationship and a second preset mapping relationship in some exemplary embodiments of this disclosure.

[0016] Figure 5 This is a second schematic diagram of a first preset mapping relationship and a second preset mapping relationship in some exemplary embodiments of this disclosure.

[0017] Figure 6-1 This is a flowchart illustrating the implementation of image IR component removal in some exemplary embodiments of this disclosure.

[0018] Figure 6-2This is a diagram showing the effect of removing the IR component from two RGB-IR images acquired by an RGB-IR image sensor under two light source environments, D65 and A light.

[0019] Figure 6-3 This is an example of an exemplary embodiment of the present disclosure, showing the effect of removing the IR component from two RGB-IR images acquired by an RGB-IR image sensor under two light source environments, D65 and A light.

[0020] Figure 7 This is one of the schematic flowcharts of an image processing method provided by some exemplary embodiments of this disclosure.

[0021] Figure 8 This is a second schematic flowchart of an image processing method provided by some exemplary embodiments of this disclosure.

[0022] Figure 9 This is the third flowchart illustrating an image processing method provided by some exemplary embodiments of this disclosure.

[0023] Figure 10 This is the fourth flowchart illustrating an image processing method provided by some exemplary embodiments of this disclosure.

[0024] Figure 11 This is the fifth of several exemplary embodiments of the image processing method provided in this disclosure.

[0025] Figure 12 This is a schematic flowchart of an image processing method provided by some exemplary embodiments of this disclosure.

[0026] Figure 13 This is the seventh flowchart illustrating an image processing method provided by some exemplary embodiments of this disclosure.

[0027] Figure 14 This is the eighth schematic flowchart of an image processing method provided by some exemplary embodiments of this disclosure.

[0028] Figure 15 This is a schematic diagram of the structure of an electronic device provided by some exemplary embodiments of this disclosure. Detailed Implementation

[0029] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0030] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0031] Application Overview Understandably, an RGB-IR image sensor is an image sensor that can respond to both the visible light band and the infrared band.

[0032] Generally, for images captured by RGB-IR image sensors during the daytime, the IR component needs to be removed from the image before subsequent image processing. Because RGB-IR image sensors have a high response value to the infrared band, for example, 50% of the peak value of the response value to the visible light band, removing the IR component may result in a significant loss of brightness and signal-to-noise ratio, which can easily affect the effect of subsequent image processing.

[0033] Exemplary System Figure 1 This is a system architecture diagram applicable to some exemplary embodiments of the present disclosure, including an RGB-IR image sensor 12 and an image processing device 14; wherein the image processing device 14 may be electrically connected to the RGB-IR image sensor 12.

[0034] Optionally, the image processing device 14 may be a chip or electronic device for performing various processing on the images acquired by the RGB-IR image sensor 12. The chip here may be a system-on-a-chip (SOC), such as, but not limited to, intelligent driving chips, smart cockpit chips, etc. The electronic device here may be, but is not limited to, drones, action cameras, virtual reality devices, augmented reality devices, etc.

[0035] In the embodiments of this disclosure, for images acquired by the RGB-IR image sensor 12 during the day, the image processing device 14 can first remove some IR components from the image before performing subsequent image processing. Since the IR components are only partially removed, rather than completely removed, it is beneficial to avoid the significant loss of brightness and signal-to-noise ratio caused by the complete removal of the IR components, thereby ensuring the effect of subsequent image processing.

[0036] Exemplary device Figure 2 This is a schematic diagram of the structure of an image processing apparatus 14 provided in some exemplary embodiments of this disclosure. For example... Figure 2 As shown, the image processing device 14 includes a processor 140, which may be, for example, an image signal processor (ISP). The processor 140 is configured to: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain the first removal parameter for the first preset color temperature calibration; wherein, the first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the Planck curve of white balance, and the first removal parameter corresponding to the first preset color temperature, the estimated color temperature of the real-world light source environment is determined. Determine the estimated trend of color temperature change; Based on the estimated color temperature and its changing trend, a second removal parameter is determined; wherein, the second removal parameter is a parameter used to partially remove the IR component of the image; According to the second removal parameters, the IR component is removed from the first image to obtain the second image.

[0037] Optionally, Figure 1 The RGB-IR image sensor 12 in the image processing device 14 can serve as the first RGB-IR image sensor. Taking an image processing device 14 that is a smart cockpit chip as an example, a vehicle equipped with a smart cockpit chip may include a smart cockpit. The first RGB-IR image sensor can be an RGB-IR image sensor located inside the smart cockpit, and the real-scene lighting environment can be a dynamically changing lighting environment composed of all natural and artificial light sources inside the smart cockpit. Taking an image processing device 14 that is a drone as an example, the first RGB-IR image sensor can be an RGB-IR image sensor located on the drone's fuselage or gimbal, and the real-scene lighting environment can be a dynamically changing lighting environment composed of various light sources in the real three-dimensional space where the drone is performing its mission. The processor 140 can acquire the RGB-IR image collected by the first RGB-IR image sensor in a real-scene lighting environment and use this RGB-IR image as the first image. Clearly, the first image is the raw image collected by the first RGB-IR image sensor; therefore, the first image can also be called RGB-IR raw data.

[0038] In an optional example, the data arrangement in the first image can be seen in [reference needed]. Figure 3-1 Then the data arrangement in the first image satisfies: The area of ​​size is treated as a repeating unit. The four pixels in the first row of this repeating unit correspond to the B channel, G channel, R channel, and G channel, respectively. The four pixels in the second row of this repeating unit correspond to the G channel, IR channel, G channel, and IR channel, respectively. The four pixels in the third row of this repeating unit correspond to the R channel, G channel, B channel, and G channel, respectively. The four pixels in the fourth row of this repeating unit correspond to the G channel, IR channel, G channel, and IR channel, respectively.

[0039] Optionally, the first preset color temperature can be the color temperature corresponding to a standard light source, such as, but not limited to, light source A, light source D40, light source D50, light source D65, light source D75, etc., and the first preset color temperature can be, for example, but not limited to, 2856K corresponding to light source A, 4000K corresponding to light source D40, 5000K corresponding to light source D50, 6500K corresponding to light source D65, 7500K corresponding to light source D75, etc. The first removal parameter corresponding to the first preset color temperature for completely removing (i.e., 100% removal) the IR components of the image can be pre-calibrated and stored in memory, and the processor 140 can read the first removal parameter corresponding to the first preset color temperature from the memory. The first removal parameter corresponding to the first preset color temperature may include the removal coefficients corresponding to the R channel, G channel, and B channel, respectively.

[0040] In some optional embodiments of this disclosure, the calibration process for the first removal parameter corresponding to the first preset color temperature may include: (1) At the first preset color temperature, using the first RGB-IR image sensor (or other RGB-IR image sensor with the same optical specifications as the first RGB-IR image sensor, such as the second RGB-IR image sensor mentioned below), respectively acquire the RGB-IR image of the 24-color card without an IR cutoff filter (hereinafter referred to as the original image P1 for ease of description) and the RGB-IR image of the 24-color card with an IR cutoff filter (hereinafter referred to as the original image P2 for ease of description); wherein, the original image P2 has the same size as the original image P1.

[0041] (2) Using a preset interpolation algorithm, interpolate the original image P1 to obtain four full-resolution images with the same size as the original image P1, namely, full-resolution image P1-1 corresponding to the R channel, full-resolution image P1-2 corresponding to the G channel, full-resolution image P1-3 corresponding to the B channel, and full-resolution image P1-4 corresponding to the IR channel. Using a preset interpolation algorithm, interpolate the original image P2 to obtain four full-resolution images with the same size as the original image P2, namely, full-resolution image P2-1 corresponding to the R channel, full-resolution image P2-2 corresponding to the G channel, full-resolution image P2-3 corresponding to the B channel, and full-resolution image P2-4 corresponding to the IR channel. The preset interpolation algorithm is, for example, but not limited to, bilinear interpolation algorithm, trilinear interpolation algorithm, and image gradient-guided interpolation algorithm.

[0042] (3) Take the 20th, 21st, 22nd, or 23rd color patch in the 24-color chart as the target color patch. Calculate the following pixel values: the first mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P1-1 corresponding to the R channel; the second mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P1-2 corresponding to the G channel; the third mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P1-3 corresponding to the B channel; the fourth mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P1-4 corresponding to the IR channel; the fifth mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P2-1 corresponding to the R channel; the sixth mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P2-2 corresponding to the G channel; and the seventh mean of the pixel values ​​in the region where the target color patch is located in the full-resolution image P2-3 corresponding to the B channel. The (first mean - fifth mean) / fourth mean is used as the removal parameter for the R channel in the first removal parameter corresponding to the first preset color temperature. The (second mean - sixth mean) / fourth mean is used as the removal parameter for the G channel in the first removal parameter corresponding to the first preset color temperature. The (third mean - seventh mean) / fourth mean is used as the removal parameter for the B channel in the first removal parameter corresponding to the first preset color temperature.

[0043] Optionally, the white balance Planck curve of the first RGB-IR image sensor can be a curve with the ratio of the pixel value of the R channel to the pixel value of the G channel (which can be expressed as R / G ratio) as the abscissa and the ratio of the pixel value of the B channel to the pixel value of the G channel (which can be expressed as B / Gatio) as the ordinate. This curve is used to characterize the trend of the ratio of the pixel value of the R channel to the pixel value of the G channel, and the ratio of the pixel value of the B channel to the pixel value of the G channel, as a function of color temperature (which is usually in Kelvin, which can also be expressed as K). The white balance Planck curve of the first RGB-IR image sensor can be pre-calibrated and stored in memory, and the processor 140 can read the white balance Planck curve from memory.

[0044] Optionally, the processor 140 can estimate the color temperature of the real-world lighting environment based on the first image, the Planck curve of white balance, and the first removal parameter corresponding to the first preset color temperature, to obtain the estimated color temperature of the real-world lighting environment; wherein, the estimated color temperature of the real-world lighting environment can also be called the scene color temperature (Correlated Color Temperature, CCT) of the real-world lighting environment. The trend of the estimated color temperature can be understood as the change of color temperature over time. In some implementations, the first RGB-IR image sensor can acquire multiple first images in the real-world lighting environment, and then the corresponding estimated color temperature can be determined sequentially for multiple first images. If the most recently determined estimated color temperature is lower than the previously determined estimated color temperature, the trend of the estimated color temperature can be from high to low; if the most recently determined estimated color temperature is higher than the previously determined estimated color temperature, the trend of the estimated color temperature can be from low to high.

[0045] Optionally, the second removal parameter can be understood as a parameter that balances image color, signal-to-noise ratio, and stability under the estimated color temperature and its changing trend, and is used to partially remove (i.e., not 100% remove) the IR components of the image. Similar to the first removal parameter, the second removal parameter may also include removal coefficients corresponding to the R channel, G channel, and B channel, respectively. The processor 140 can remove the IR components of the first image according to the removal coefficients included in the second removal parameter to obtain the second image.

[0046] In some optional embodiments of this disclosure, the removal coefficients corresponding to the R channel, G channel, and B channel in the second removal parameter are respectively represented as k1, k2, and k3. Therefore, the process of removing IR components from the first image may include: (1) Using a preset interpolation algorithm, the first image (hereinafter referred to as the original image P3 for ease of description) is interpolated to obtain four full-resolution images with the same size as the original image P3, namely the full-resolution image P3-1 corresponding to the R channel, the full-resolution image P3-2 corresponding to the G channel, the full-resolution image P3-3 corresponding to the B channel, and the full-resolution image P3-4 corresponding to the IR channel.

[0047] (2) For any pixel in any row and column of the original image P3 (e.g., the pixel in the i-th row and j-th column), the pixel value x1 of the pixel in the i-th row and j-th column can be obtained from the full-resolution image P3-1 corresponding to the R channel, the pixel value x2 of the pixel in the i-th row and j-th column can be obtained from the full-resolution image P3-2 corresponding to the G channel, the pixel value x3 of the pixel in the i-th row and j-th column can be obtained from the full-resolution image P3-3 corresponding to the B channel, and the pixel value x4 of the pixel in the i-th row and j-th column can be obtained from the full-resolution image P3-4 corresponding to the IR channel. Based on x4 and k1, the pixel value x1' corresponding to x1 after IR component removal can be calculated according to the following formula: Based on x4 and k2, the pixel value x2' after IR component removal corresponding to x2 can be calculated using the following formula: Based on x4 and k3, the pixel value x3' corresponding to x3 after IR component removal can be calculated using the following formula: By updating the pixel value of each pixel in the full-resolution image P3-1 corresponding to the R channel to the pixel value after removing the IR component, we can obtain the full-resolution image P4-1 corresponding to the R channel; by updating the pixel value of each pixel in the full-resolution image P3-2 corresponding to the G channel to the pixel value after removing the IR component, we can obtain the full-resolution image P4-2 corresponding to the G channel; and by updating the pixel value of each pixel in the full-resolution image P3-3 corresponding to the B channel to the pixel value after removing the IR component, we can obtain the full-resolution image P4-3 corresponding to the B channel.

[0048] (3) According to the data arrangement rules that conform to the Bayer array, the full resolution image P4-1 corresponding to the R channel, the full resolution image P4-2 corresponding to the G channel, and the full resolution image P4-3 corresponding to the B channel are sampled and processed to obtain an image of the Bayer type, which can be used as the second image.

[0049] In an optional example, the data arrangement in the second image can be seen in [reference needed]. Figure 3-2 Then the data arrangement in the second image satisfies: The area of ​​size is treated as a repeating unit. The two pixels in the first row of this repeating unit correspond to the B channel and the G channel, respectively, and the two pixels in the second row of this repeating unit correspond to the G channel and the R channel, respectively.

[0050] Optionally, taking the image processing device 14 as an example of a smart cockpit chip, the processor 140 can perform subsequent image processing on the second image, such as but not limited to de-mosaic processing, white balance compensation processing, color restoration processing, etc. On this basis, it can further run perception and recognition algorithms such as driver status monitoring and passenger posture and activity monitoring.

[0051] In the embodiments of this disclosure, the first image acquired by the first RGB-IR image sensor under a real-world lighting environment is affected by the color temperature of the light source. The white balance Planck curve of the first RGB-IR image sensor is correlated with the color temperature. A first removal parameter calibrated for a first preset color temperature is used to completely remove the IR components of the image, which helps to eliminate the adverse effects of the image IR components on color temperature estimation. Therefore, based on the first image, the white balance Planck curve, and the first removal parameter corresponding to the first preset color temperature, the estimated color temperature of the real-world lighting environment can be determined relatively accurately. Based on the estimated color temperature and its changing trend, a second removal parameter can be adaptively determined to remove the IR components of the first image, thereby obtaining a second image for subsequent image processing. Since the second removal parameter is used to partially remove the IR components of the image, removing the IR components of the first image according to the second removal parameter will only remove a portion of the IR components, not all of them. This helps to avoid the significant loss of brightness and signal-to-noise ratio caused by the complete removal of the IR components. Thus, there will be no obvious noise in the second image, which helps to ensure the effect of subsequent image processing.

[0052] In some optional examples, when processor 140 determines the second removal parameter based on the estimated color temperature and its changing trend, it is specifically configured as follows: Based on the changing trend, a target preset mapping relationship is determined from a variety of preset mapping relationships; wherein, each preset mapping relationship is a mapping relationship between multiple color temperature segments in a predetermined color temperature range and the corresponding third removal parameter, the third removal parameter is a parameter used to partially remove the IR component of the image, and the first preset color temperature is located in the predetermined color temperature range; Based on the target preset mapping relationship, determine the third removal parameter of the color temperature range mapping to which the estimated color temperature belongs; The third removal parameter of the color temperature range mapping to which the estimated color temperature belongs is determined as the second removal parameter.

[0053] Optionally, the predetermined color temperature range is a range that can cover all color temperatures that may occur under real-world lighting conditions. The lower and upper limits of the predetermined color temperature range can be represented as a1 and a2, respectively, and the first preset color temperature can be located between a1 and a2. a1 can be, for example, 1600K, 1800K, etc., and a2 can be, for example, 9000K, 10000K, etc., and will not be listed here.

[0054] Optionally, multiple preset mapping relationships can be pre-built and stored in memory. Each preset mapping relationship can be a mapping relationship between multiple color temperature segments formed by dividing a predetermined color temperature range and their corresponding third removal parameters. Each preset mapping relationship can be adapted to a trend of color temperature changing over time. For example, one preset mapping relationship can be adapted to a trend of color temperature changing from high to low, and another preset mapping relationship can be adapted to a trend of color temperature changing from low to high. In addition, similar to the first removal parameter, the third removal parameter can also include removal coefficients corresponding to the R channel, G channel, and B channel, respectively.

[0055] As described above, the trend of change in the embodiments of this disclosure can be understood as the change in color temperature over time. This trend can be obtained by comparing the most recently determined estimated color temperature with the previously determined estimated color temperature. In the embodiments of this disclosure, a preset mapping relationship that matches the trend of change can be selected from a variety of preset mapping relationships. The selected preset mapping relationship can be used as a target preset mapping relationship. The target preset mapping relationship is the mapping relationship between multiple color temperature segments and their corresponding third removal parameters. By comparing the estimated color temperature with each of the multiple color temperature segments, the color temperature segment to which the estimated color temperature belongs can be determined. Furthermore, the third removal parameter mapped to the color temperature segment can be found in the target preset mapping relationship, and this third removal parameter is determined as the second removal parameter. In this way, the estimated color temperature and the trend of change can be effectively applied in the process of determining the second removal parameter for IR component removal of the first image, thereby ensuring the compatibility between the second removal parameter and the estimated color temperature and the trend of change. Therefore, the embodiments of this disclosure can adaptively control the IR component removal ratio based on color temperature information, which is beneficial to ensuring the accuracy of color reproduction at medium and high color temperatures, and also beneficial to ensuring the signal-to-noise ratio at low color temperatures.

[0056] In some optional examples, processor 140 is also configured as follows: For each of the multiple preset color temperatures within the predetermined color temperature range, a corresponding first removal parameter is assigned; wherein, the first preset color temperature is one of the multiple preset color temperatures; The first removal parameters corresponding to various preset color temperatures are optimized to obtain the fourth removal parameters corresponding to various preset color temperatures. From a variety of preset color temperatures, at least one color temperature group is determined; wherein each color temperature group includes two adjacent preset color temperatures from the variety of preset color temperatures; Determine the color temperature threshold and the corresponding color temperature jitter threshold for at least one color temperature group; Based on the predetermined color temperature range, the fourth removal parameters corresponding to various preset color temperatures, and the color temperature thresholds and color temperature jitter thresholds corresponding to at least one color temperature group, various preset mapping relationships are generated.

[0057] Optionally, 2856K corresponding to light source A can be used as one preset color temperature, 4000K corresponding to light source D40 can be used as another preset color temperature, 5000K corresponding to light source D50 can be used as yet another preset color temperature, 6500K corresponding to light source D65 can be used as yet another preset color temperature, and 7500K corresponding to light source D75 can be used as yet another preset color temperature. All of the above preset color temperatures can be within the predetermined color temperature range (that is, a1 is less than 2856K and a2 is greater than 7500K). The first preset color temperature can be any of the above preset color temperatures.

[0058] Optionally, the calibration method for the first removal parameter corresponding to the various preset color temperatures can refer to the relevant introduction to the calibration process of the first removal parameter corresponding to the first preset color temperature above, and will not be repeated here.

[0059] Optionally, the processor 140 can use both image color and signal-to-noise ratio as constraints (the specific implementation can be found in the application of evaluation index parameters including total color deviation, color deviation, and signal-to-noise ratio below) to optimize the first removal parameters corresponding to various preset color temperatures, thereby obtaining fourth removal parameters corresponding to various preset color temperatures. The fourth removal parameters can be parameters used to partially remove the IR components of the image. Similar to the first removal parameters, the fourth removal parameters can also include removal coefficients corresponding to the R channel, G channel, and B channel, respectively.

[0060] Optionally, the processor 140 can combine any two adjacent preset color temperatures from a variety of preset color temperatures to obtain multiple candidate color temperature groups, such that each candidate color temperature group includes two adjacent preset color temperatures from the variety of preset color temperatures. Then, each of the multiple candidate color temperature groups can be treated as a separate color temperature group, or at least one candidate color temperature group can be selected from the multiple candidate color temperature groups, and the selected at least one candidate color temperature group can be treated as a separate color temperature group. In this way, at least one color temperature group can be obtained.

[0061] Optionally, the color temperature threshold corresponding to any color temperature group can be understood as: a color temperature located between the two preset color temperatures included in the color temperature group. For the same image acquired by an RGB-IR image sensor at this color temperature, using the fourth removal parameters corresponding to these two preset color temperatures to remove the IR components of the image can achieve the same or essentially the same removal effect, and the removal effect can meet business requirements. The color temperature jitter threshold corresponding to any color temperature group can be understood as: a jitter value applied to the color temperature threshold corresponding to the color temperature group, which can be much smaller than the color temperature threshold corresponding to the color temperature group. In an optional example, the two preset color temperatures included in the color temperature group are 2856K and 4000K, the color temperature threshold corresponding to the color temperature group can be 3000K, and the color temperature jitter threshold corresponding to the color temperature group can be 100K.

[0062] Optionally, the multiple preset mapping relationships may include a first preset mapping relationship and a second preset mapping relationship. The first preset mapping relationship can be adapted to a trend of color temperature decreasing from high to low, and the second preset mapping relationship can be adapted to a trend of color temperature increasing from low to high. The color temperature jitter threshold corresponding to each color temperature group may include a first jitter threshold and a second jitter threshold. The first jitter threshold can be understood as a negative jitter value applied to the color temperature threshold corresponding to that color temperature group, and the second jitter threshold can be understood as a positive jitter value applied to the color temperature threshold corresponding to that color temperature group. As an example, the first jitter threshold and the second jitter threshold can be determined based on a first user input operation, such as, but not limited to, text input, voice input, touch input, etc. Of course, the first jitter threshold and the second jitter threshold can also be preset jitter thresholds. Furthermore, the first jitter threshold and the second jitter threshold can be the same or different.

[0063] Accordingly, when the processor 140 generates multiple preset mapping relationships based on a predetermined color temperature range, the fourth removal parameters corresponding to multiple preset color temperatures, and the color temperature thresholds and color temperature jitter thresholds corresponding to at least one color temperature group, it is specifically configured as follows: For each color temperature group, the color temperature threshold corresponding to the color temperature group is subtracted from the corresponding first jitter threshold to obtain the first adjusted color temperature; the color temperature threshold corresponding to the color temperature group is added to the corresponding second jitter threshold to obtain the second adjusted color temperature. Each first color temperature adjustment is used as a dividing point to divide the predetermined color temperature range into multiple first color temperature segments; Each second color temperature adjustment is used as a dividing point to divide the predetermined color temperature range into multiple second color temperature segments; Based on the fourth removal parameters corresponding to various preset color temperatures, the third removal parameters mapped to various first color temperature segments are determined, and a first preset mapping relationship between various first color temperature segments and the determined corresponding third removal parameters is generated. Based on the fourth removal parameters corresponding to various preset color temperatures, the third removal parameters mapped to various second color temperature segments are determined, and a second preset mapping relationship between various second color temperature segments and the determined corresponding third removal parameters is generated.

[0064] Assuming the color temperature threshold corresponding to any color temperature group in at least one color temperature group is denoted as c, the first jitter threshold corresponding to that color temperature group is denoted as d1, and the second jitter threshold corresponding to that color temperature group is denoted as d2, then the first adjusted color temperature corresponding to that color temperature group is c - d1, and the second adjusted color temperature corresponding to that color temperature group is c + d2. Thus, through subtraction, at least one first adjusted color temperature corresponding to at least one color temperature group can be obtained; through addition, at least one second adjusted color temperature corresponding to at least one color temperature group can be obtained. By dividing the predetermined color temperature range using each of the at least one first adjusted color temperature as a dividing point, multiple first color temperature segments can be formed. For example, if the predetermined color temperature range is a1~a2, and at least one first adjusted color temperature is two first adjusted color temperatures, with the lower one being b1 and the higher one being b2, then by dividing the predetermined color temperature range, three first color temperature segments can be formed: a1~b1, b1~b2, and b2~a2. Similarly, by using at least one second adjusted color temperature as a dividing point to divide the predetermined color temperature range, multiple second color temperature segments can be formed.

[0065] Optionally, the multiple first color temperature segments can be arranged in order of lower color temperature segments first, followed by higher color temperature segments. For any first color temperature segment that is not at the end of the multiple first color temperature segments, a color temperature threshold that is higher than all color temperatures in that first color temperature segment and closest to it can be selected. The lower of the two preset color temperatures included in the color temperature group corresponding to that threshold is determined, and the fourth removal parameter corresponding to the lower value is determined as the third removal parameter for the mapping of that first color temperature segment. For the first color temperature segment that is at the end of the multiple first color temperature segments, a color temperature threshold within that first color temperature segment can be determined. The higher of the two preset color temperatures included in the color temperature group corresponding to that threshold is determined, and the fourth removal parameter corresponding to the higher value is determined as the third removal parameter for the mapping of that first color temperature segment. Based on this, a first preset mapping relationship can be constructed between the multiple first color temperature segments and their respective third removal parameters.

[0066] Optionally, the multiple second color temperature segments can be arranged in order of lower color temperature segments first, followed by higher color temperature segments. For any second color temperature segment that is not at the end of the multiple second color temperature segments, a color temperature threshold within that segment can be determined. The lower of the two preset color temperatures included in the color temperature group corresponding to that threshold is then determined, and the fourth removal parameter corresponding to that lower value is determined as the third removal parameter for mapping that second color temperature segment. For the second color temperature segment at the end of the multiple second color temperature segments, a color temperature threshold that is lower than all color temperatures within that segment and closest to it can be selected. The higher of the two preset color temperatures included in the color temperature group corresponding to that threshold is then determined, and the fourth removal parameter corresponding to that higher value is determined as the third removal parameter for mapping that second color temperature segment. Based on this, a second preset mapping relationship can be constructed between the multiple second color temperature segments and their respective third removal parameters.

[0067] In an optional example, the fourth removal parameter for the preset color temperature of 2856K (corresponding to light source A) is denoted as IR_remove_ratio_A, the fourth removal parameter for the preset color temperature of 4000K (corresponding to light source D40) is denoted as IR_remove_ratio_D40, and the fourth removal parameter for the preset color temperature of 5000K (corresponding to light source D50) is denoted as IR_remove_ratio_D50. The color temperature threshold corresponding to the color temperature group including 2856K and 4000K is denoted as temper_th[A-D40], the first adjusted color temperature corresponding to the color temperature group including 2856K and 4000K is denoted as e1, and the second adjusted color temperature corresponding to the color temperature group including 2856K and 4000K is denoted as e2. The color temperature threshold corresponding to the color temperature groups including 4000K and 5000K is represented as temper_th[D40-D50], the first adjusted color temperature corresponding to the color temperature groups including 4000K and 5000K is represented as e3, and the second adjusted color temperature corresponding to the color temperature groups including 4000K and 5000K is represented as e4.

[0068] Assuming e1 and e3 are used as dividing points to segment the predetermined color temperature range, three first color temperature segments can be formed, ordered by lower color temperature segments first and higher color temperature segments last. These three first color temperature segments are a1~e1, e1~e3, and e3~a2, respectively. For the first color temperature segment a1~e1, it is not the last first color temperature segment. The color temperature threshold that is higher than all color temperatures in this first color temperature segment and closest to it is temper_th[A-D40]. The lower of the two preset color temperatures included in the color temperature group corresponding to temper_th[A-D40] is 2856K. Therefore, IR_remove_ratio_A corresponding to 2856K can be determined as the third removal parameter for the first color temperature segment a1~e1. For the first color temperature range e1~e3, which is not the last color temperature range, the color temperature threshold that is higher than all color temperatures in this first color temperature range and closest to this first color temperature range is temper_th[D40-D50]. The lower of the two color temperature thresholds included in the color temperature group corresponding to temper_th[D40-D50] is 4000K. Therefore, IR_remove_ratio_D40 corresponding to 4000K can be determined as the third removal parameter corresponding to the first color temperature range e1~e3. For the first color temperature range e3~a2, which is the last of the three color temperature ranges, the color temperature threshold within this range is temper_th[D40-D50]. The higher of the two color temperature thresholds included in the temper_th[D40-D50] color temperature group is 5000K. Therefore, IR_remove_ratio_D50 corresponding to 5000K can be determined as the third removal parameter for the first color temperature range e3~a2. In this way, a mapping relationship can be constructed between the three first color temperature ranges a1~e1, e1~e3, and e3~a2 and their corresponding third removal parameters. For example, see [link to documentation]. Figure 4 The content shown in the upper half can be used as the first preset mapping relationship.

[0069] Assuming e2 and e4 are used as dividing points to segment the predetermined color temperature range, three second color temperature segments can be formed, ordered from lowest to highest color temperature. These three second color temperature segments are a1~e2, e2~e4, and e4~a2. For the second color temperature segment a1~e2, it is not the last segment. The color temperature threshold within this segment is temper_th[A-D40]. The lower of the two preset color temperatures included in the color temperature group corresponding to temper_th[A-D40] is 2856K. Therefore, IR_remove_ratio_A corresponding to 2856K can be determined as the third removal parameter for the second color temperature segment a1~e2. For the second color temperature range e2~e4, which is not the last second color temperature range, the color temperature threshold in this second color temperature range is temper_th[D40-D50]. The lower of the two color temperature thresholds included in the color temperature group corresponding to temper_th[D40-D50] is 4000K. Therefore, IR_remove_ratio_D40 corresponding to 4000K can be determined as the third removal parameter corresponding to the second color temperature range e2~e4. For the second color temperature range e4~a2, which is the last of the two color temperature ranges, the color temperature threshold closest to all other color temperatures in this range is temper_th[D40-D50]. The higher of the two color temperature thresholds included in the color temperature group corresponding to temper_th[D40-D50] is 5000K. Therefore, IR_remove_ratio_D50 corresponding to 5000K can be determined as the third removal parameter for the second color temperature range e4~a2. In this way, a mapping relationship can be constructed between the three second color temperature ranges a1~e2, e2~e4, and e4~a2 and their corresponding third removal parameters. For example, see [link to documentation]. Figure 4 The content shown in the lower half can be used as a second preset mapping relationship.

[0070] In this embodiment, based on the color temperature threshold, the corresponding first jitter threshold, and the corresponding second jitter threshold for each of at least one color temperature group, subtraction and addition operations can be used to efficiently and reliably obtain the first adjusted color temperature and the corresponding second adjusted color temperature for each of the at least one color temperature group, which can then be used for segmenting a predetermined color temperature range. This allows for the efficient and reliable formation of multiple first color temperature segments and multiple second color temperature segments. Furthermore, based on the fourth removal parameters corresponding to various preset color temperatures, corresponding third removal parameters can be determined for each first color temperature segment and each second color temperature segment. On this basis, a first preset mapping relationship and a second preset mapping relationship can be efficiently and reliably constructed.

[0071] Given the establishment of the first and second preset mapping relationships, when the processor 140 determines the target preset mapping relationship from multiple preset mapping relationships based on the changing trend, it is specifically configured as follows: In response to the changing trend of color temperature decreasing from high to low, the first preset mapping relationship is determined as the target preset mapping relationship; In response to the changing trend of color temperature increasing from low to high, the second preset mapping relationship is determined as the target preset mapping relationship.

[0072] If the trend is that the color temperature changes from high to low, you can Figure 4 The first preset mapping relationship illustrated in the upper half is determined as the target preset mapping relationship. Therefore, if the estimated color temperature is between a1 and e1, IR_remove_ratio_A can be determined as the second removal parameter; if the estimated color temperature is between e1 and e3, IR_remove_ratio_40 can be determined as the second removal parameter; and if the estimated color temperature is between e3 and a2, IR_remove_ratio_50 can be determined as the second removal parameter.

[0073] If the trend is that the color temperature changes from low to high, you can Figure 4 The second preset mapping relationship illustrated in the lower half is determined as the target preset mapping relationship. Therefore, if the estimated color temperature is between a1 and e2, IR_remove_ratio_A can be determined as the second removal parameter; if the estimated color temperature is between e2 and e4, IR_remove_ratio_40 can be determined as the second removal parameter; and if the estimated color temperature is between e4 and a2, IR_remove_ratio_50 can be determined as the second removal parameter.

[0074] In an optional example, Figure 4 In the given information, `temper_th[A-D40]` is 3000K, `temper_th[D40-D50]` is 4600K, `e1` is 2900K, `e2` is 3100K, `e3` is 4500K, and `e4` is 4700K. Therefore, we can... Figure 4 Updated to Figure 5The first RGB-IR image sensor acquired one first image at time to, time t1 (which is the next time after time t0), time t2 (which is the next time after time t1), time t3 (which is the next time after time t2), and time t4 (which is the next time after time t3). The estimated color temperature of the first image acquired at time t0 is 2300K, the estimated color temperature of the first image acquired at time t1 is 2950K, the estimated color temperature of the first image acquired at time t2 is 3020K, the estimated color temperature of the first image acquired at time t3 is 4750K, and the estimated color temperature of the first image acquired at time t4 is 4300K. For the first image acquired at time t1 with an estimated color temperature of 2950K, compared to the first image acquired at time t0 with an estimated color temperature of 2300K, the color temperature trend is from low to high. Therefore, according to the second preset mapping relationship, the IR_remove_ratio_A corresponding to the color temperature range a1~3100K to which 2950K belongs can be determined as the second removal parameter for removing the IR components of the first image acquired at time t1. For the first image acquired at time t2 with an estimated color temperature of 3020K, compared to the first image acquired at time t1 with an estimated color temperature of 2950K, the color temperature trend is from low to high. Therefore, according to the second preset mapping relationship, the IR_remove_ratio_A corresponding to the color temperature range a1~3100K to which 3020K belongs can be determined as the second removal parameter for removing the IR components of the first image acquired at time t2. For the first image acquired at time t3, with an estimated color temperature of 4750K, compared to the first image acquired at time t2 with an estimated color temperature of 3020K, the color temperature trend is from low to high. Therefore, according to the second preset mapping relationship, the IR_remove_ratio_D50 corresponding to the color temperature range of 4700K to a2, to which 4750K belongs, can be determined as the second removal parameter for removing the IR components of the first image acquired at time t3. For the first image acquired at time t4, with an estimated color temperature of 4300K, compared to the first image acquired at time t2 with an estimated color temperature of 4700K, the color temperature trend is from high to low. Therefore, according to the first preset mapping relationship, the IR_remove_ratio_D40 corresponding to the color temperature range of 2900K to 4500K, to which 4300K ​​belongs, can be determined as the second removal parameter for removing the IR components of the first image acquired at time t4.

[0075] Depend on Figure 5It is known that the second removal parameter exhibits good stability when the estimated color temperature varies around the threshold of 3000K. For example, if the estimated color temperature increases from 2900K to 3100K, the second removal parameter remains constant at IR_remove_ratio_A. Similarly, if the estimated color temperature decreases from 3100K to 2900K, the second removal parameter remains constant at IR_remove_ratio_40. Likewise, the second removal parameter also exhibits good stability when the estimated color temperature varies around the threshold of 4700K. Therefore, in this embodiment, the second removal parameter used for IR component removal does not switch when the estimated color temperature changes slightly, which helps to avoid image abrupt changes after IR component removal. In addition, when the estimated color temperature changes significantly, such as from 3020K to 4750K or from 4750K to 4300K, the second removal parameter used for IR component removal will be switched accordingly. This allows the IR component removal ratio to be adaptively controlled based on the color temperature information, which is beneficial for ensuring the accuracy of color reproduction at medium and high color temperatures, as well as for ensuring the signal-to-noise ratio at low color temperatures.

[0076] The above describes the scenario where multiple preset mapping relationships consist of two preset mapping relationships (i.e., the first preset mapping relationship and the second preset mapping relationship). In specific implementations, multiple preset mapping relationships can also consist of more than two preset mapping relationships. For example, in some embodiments, the situation where the color temperature changes from high to low can be further divided into more situations based on the rate at which the color temperature changes from high to low, and the situation where the color temperature changes from low to high can also be further divided into more situations based on the rate at which the color temperature changes from low to high. For each of the further subdivided situations, a corresponding preset mapping relationship can be constructed.

[0077] In the embodiments of this disclosure, by calibrating for multiple preset color temperatures, first removal parameters corresponding to each preset color temperature can be obtained efficiently and reliably. Optimizing the first removal parameters corresponding to each preset color temperature facilitates the acquisition of fourth removal parameters corresponding to each preset color temperature that balance image color and signal-to-noise ratio. Furthermore, at least one color temperature group, and color temperature thresholds and corresponding color temperature jitter thresholds for each color temperature group can be determined. By combining a predetermined color temperature range, the fourth removal parameters corresponding to each preset color temperature, and the color temperature thresholds and corresponding color temperature jitter thresholds for each color temperature group, multiple preset mapping relationships are generated. Applying these generated preset mapping relationships avoids image abrupt changes, ensures accurate color reproduction at medium to high color temperatures, and guarantees a good signal-to-noise ratio at low color temperatures.

[0078] In some optional examples, when the processor 140 optimizes the first removal parameter corresponding to the first preset color temperature to obtain the fourth removal parameter corresponding to the first preset color temperature, it is specifically configured as follows: A third image is acquired by the second RGB-IR image sensor under a standard light source environment that meets the first preset color temperature; wherein the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor. The first removal parameter corresponding to the first preset color temperature is adjusted to obtain the fifth removal parameter; wherein, the first removal parameter includes the first removal coefficients corresponding to the R channel, G channel and B channel respectively, and the fifth removal parameter includes the second removal coefficients corresponding to the R channel, G channel and B channel respectively, each second removal coefficient is less than or equal to the corresponding first removal coefficient, and at least one second removal coefficient is different from the corresponding first removal coefficient. Based on the third image and the fifth removal parameter, the fourth removal parameter corresponding to the first preset color temperature is determined.

[0079] Optionally, the fact that the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor can be understood as follows: the effective imaging surface physical size, pixel size, pixel array arrangement, total resolution, spectral response curve (including the spectral response curve for the visible light band and the spectral response curve for the infrared band), and cover glass parameters of the second RGB-IR image sensor are all the same as those of the first RGB-IR image sensor.

[0080] Optionally, a standard light source environment conforming to the first preset color temperature refers to a controlled light source environment constructed using professional equipment (such as a standard light box), which conforms to international standards in terms of spectral power distribution, color rendering, illuminance, and uniformity, and whose color temperature is the first preset color temperature. The processor 140 can acquire the RGB-IR image collected by the second RGB-IR image sensor in a standard light source environment conforming to the first preset color temperature, and use this RGB-IR image as the third image.

[0081] Optionally, the first removal parameter corresponding to the first preset color temperature includes first removal coefficients corresponding to the R channel, G channel, and B channel, respectively. The processor 140 can adjust the first removal parameter corresponding to the first preset color temperature according to a preset adjustment rule to obtain a fifth removal parameter including second removal coefficients corresponding to the R channel, G channel, and B channel, respectively. The preset adjustment rule may include, for example, proportionally reducing the first removal coefficients corresponding to the R channel, G channel, and B channel, or randomly reducing the first removal coefficients corresponding to the R channel, G channel, and B channel. The processor 140 can determine a fourth removal parameter corresponding to the first preset color temperature based on the third image and the fifth removal coefficient.

[0082] In some optional embodiments of this disclosure, when the processor 140 determines the fourth removal parameter corresponding to the first preset color temperature based on the third image and the fifth removal parameter, it is specifically configured as follows: The third image is subjected to IR component removal according to the second removal coefficient included in the fifth removal parameter to obtain the fourth image; Based on the fourth image, determine the fifth image, which belongs to the visualization type; The fifth image is evaluated to obtain the first evaluation index parameters; In response to the first evaluation index parameter being within the preset parameter range, the fifth removal parameter is determined to be the fourth removal parameter corresponding to the first preset color temperature; In response to the first evaluation index parameter being outside the preset parameter range, the operation of adjusting the first removal parameter corresponding to the first preset color temperature to obtain the fifth removal parameter is performed.

[0083] Optionally, the processor 140 can remove the IR components from the third image according to the second removal coefficients included in the fifth removal parameter to obtain the fourth image. The specific method for obtaining the fourth image is described above in the section on obtaining the second image, and will not be repeated here. Similar to the second image, the fourth image can be a Bayer type image. Therefore, the processor 140 can convert the fourth image from the Bayer type to a visualization type to obtain the fifth image. For example, the processor 140 can convert the fourth image from the Bayer type to the RGB type, and the resulting RGB image can be used as the fifth image. Alternatively, the processor 140 can convert the fourth image from the Bayer type to the RGB type, and then from the RGB type to the YUV type (where Y represents luminance, and U and V represent chrominance, respectively), and the resulting YUV image can be used as the fifth image.

[0084] Optionally, the processor 140 can evaluate the fifth image to obtain first evaluation index parameters. For example, the processor 140 can evaluate the color accuracy of the fifth image to obtain the total color deviation (ΔE), which represents the magnitude of the total color difference perceived by the human eye, and the color deviation (ΔC), which represents the deviation of color intensity or saturation. ΔE and ΔC can be used as the first evaluation index parameters. As another example, the processor 140 can evaluate the noise level of the fifth image to obtain the signal-to-noise ratio (SNR), which represents the ratio of useful signal strength to noise strength. SNR can be used as the first evaluation index parameter. For ΔC, ΔC, and SNR, corresponding preset parameter ranges can be preset; these preset parameter ranges can be understood as reference ranges for the parameters. If the first evaluation index parameters include ΔE, ΔC, and SNR, and all of ΔE, ΔC, and SNR are within their respective preset parameter ranges, the processor 140 can determine that the color and signal-to-noise ratio of the fifth image meet the requirements, and the evaluation of the fifth image passes. Furthermore, the fifth removal parameter applied to the fifth image can be determined as the fourth removal parameter corresponding to the first preset color temperature. If the first evaluation metric parameters include ΔE, ΔC, and SNR, and at least one of ΔE, ΔC, and SNR falls outside the corresponding preset parameter range, the processor 140 can determine that the color and / or signal-to-noise ratio of the fifth image does not meet the requirements, and the evaluation of the fifth image fails. It can then return to the process of adjusting the first removal parameter corresponding to the first preset color temperature to obtain the fifth removal parameter. By performing this operation, a new fifth removal parameter can be obtained. Based on the new fifth removal parameter, a new fourth image, a new fifth image, and new first evaluation metric parameters are obtained sequentially. Based on this, it is determined whether the evaluation of the new fifth image passes. If the evaluation of the new fifth image passes, the processor 140 can determine the new fifth removal parameter as the fourth removal parameter corresponding to the first preset color temperature; otherwise, it can return to the process of adjusting the first removal parameter corresponding to the first preset color temperature to obtain the fifth removal parameter. Subsequent operations follow the same principle and will not be elaborated further here.

[0085] In an optional example, processor 140 can reduce each of the first removal coefficients in the first removal parameters corresponding to the first preset color temperature to 90% of their original values ​​to obtain the fifth removal coefficient for the first time. Based on the fifth removal coefficient obtained in the first instance, processor 140 can sequentially obtain the fourth image, the fifth image, and the first evaluation index parameter. If the first evaluation index parameter is outside the preset parameter range, processor 140 can reduce each of the first removal coefficients in the first removal parameters corresponding to the first preset color temperature to 85% of their original values ​​to obtain the fifth removal coefficient for the second time. Based on the fifth removal coefficient obtained in the second instance, processor 140 can sequentially obtain the fourth image, the fifth image, and the first evaluation index parameter. If the first evaluation index parameter is outside the preset parameter range, processor 140 can reduce each of the first removal coefficients in the fourth removal parameter to 80% of their original values ​​to obtain the fifth removal coefficient for the third time. This process continues in the same manner, and will not be elaborated further here.

[0086] In this way, through image evaluation, the processor 140 can search for a fifth removal parameter that can take into account both image color and signal-to-noise ratio as the fourth removal parameter corresponding to the first preset color temperature, which is beneficial to determining the fourth removal parameter that can take into account both image color and signal-to-noise ratio.

[0087] Of course, the implementation method by which the processor 140 determines the fourth removal parameter corresponding to the first preset color temperature based on the third image and the fifth removal parameter is not limited to this. For example, after obtaining the fifth image based on the third image and the fifth removal parameter, the processor 140 can control the display screen to display the fifth image and obtain a second user input operation. The second user input operation is, for example, but not limited to, text input, voice input, touch input, etc. If the second user input operation indicates that the fifth image displayed on the display screen meets the requirements, the processor 140 can determine the fifth removal parameter as the fourth removal parameter corresponding to the first preset color temperature. If the second user input operation indicates that the fifth image displayed on the display screen does not meet the requirements, the processor 140 can return to perform the operation of adjusting the first removal parameter corresponding to the first preset color temperature to obtain the fifth removal parameter.

[0088] In the embodiments of this disclosure, the first removal parameter corresponding to the first preset color temperature is adjusted according to the preset adjustment rules, which can efficiently and quickly obtain the fifth removal parameter. Combined with the third image acquired by the second RGB-IR image sensor under a standard light source environment that meets the first preset color temperature, the fourth removal parameter that can take into account both image color and signal-to-noise ratio can be searched through image evaluation, which is conducive to ensuring the rationality and reliability of the fourth removal parameter.

[0089] In some optional examples, the two preset color temperatures in each color temperature group are represented as a second preset color temperature and a third preset color temperature. When the processor 140 determines the color temperature threshold corresponding to the color temperature group, it is specifically configured as follows: Determine a reference color temperature from the color temperature range defined by the second preset color temperature and the third preset color temperature; A sixth image is acquired by the second RGB-IR image sensor under a test light source environment that conforms to the reference color temperature; wherein the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor. According to the fourth removal parameter corresponding to the second preset color temperature, the IR component is removed from the sixth image to obtain the seventh image; According to the fourth removal parameter corresponding to the third preset color temperature, the IR component of the sixth image is removed to obtain the eighth image; The color temperature threshold is determined based on the reference color temperature, the seventh image, and the eighth image.

[0090] Assuming the second preset color temperature is lower than the third preset color temperature, the color temperature range defined by the second and third preset color temperatures refers to a range where the lower limit is the second preset color temperature and the upper limit is the third preset color temperature. From this range, a color temperature can be selected as a reference color temperature. The selection method here can be either random selection or selection according to a set step size (for example, each time selection, the difference between the selected color temperature and the previous selected color temperature is set to a preset difference).

[0091] Optionally, a test light source environment conforming to the reference color temperature refers to a controlled light source environment constructed using professional equipment, which conforms to international standards in terms of spectral power distribution, color rendering, illuminance, and uniformity, and whose color temperature is the reference color temperature. The processor 140 can acquire the RGB-IR image collected by the second RGB-IR image sensor in the test light source environment conforming to the reference color temperature, and use this RGB-IR image as the sixth image.

[0092] Optionally, the processor 140 can remove IR components from the sixth image according to the removal coefficients included in the fourth removal parameters corresponding to the second preset color temperature to obtain the seventh image, and remove IR components from the sixth image according to the removal coefficients included in the fourth removal parameters corresponding to the third preset color temperature to obtain the eighth image. The specific methods for obtaining the seventh and eighth images are the same as those described above for obtaining the second image, and will not be repeated here. The processor 140 can determine the color temperature threshold based on the reference color temperature, the seventh image, and the eighth image.

[0093] In some optional embodiments of this disclosure, when the processor 140 determines the color temperature threshold based on the reference color temperature, the seventh image, and the eighth image, it is specifically configured as follows: Based on the seventh image, the ninth image is identified as belonging to the visualization type; Based on the eighth image, the tenth image is identified as belonging to the visualization type; The ninth image is evaluated to obtain the second evaluation index parameters; The tenth image is evaluated to obtain the third evaluation index parameters; In response to the fact that both the second and third evaluation index parameters are within the preset parameter range and the second and third evaluation index parameters meet the preset consistency condition, the reference color temperature is determined as the color temperature threshold. In response to at least one of the second and third evaluation index parameters being outside the preset parameter range, and / or the second and third evaluation index parameters not meeting the preset consistency conditions, the operation of determining a reference color temperature from the color temperature range defined by the second and third preset color temperatures is returned.

[0094] Optionally, the methods for determining the ninth and tenth images can be the same as those for determining the fifth image described above, and will not be repeated here. Similarly, the methods for obtaining the second and third evaluation index parameters can be the same as those for obtaining the first evaluation index parameters described above, and will not be repeated here.

[0095] Optionally, the preset consistency condition may include, for example, that the similarity between the evaluation index parameters is greater than a preset similarity. Here, some conventional similarity algorithms can be used to calculate the similarity between the second and third evaluation index parameters. If the calculated similarity is greater than the preset similarity, it can be determined that the second and third evaluation index parameters meet the preset consistency condition; if the calculated similarity is less than or equal to the preset similarity, it can be determined that the second and third evaluation index parameters do not meet the preset consistency condition.

[0096] In the embodiments of this disclosure, if both the second and third evaluation index parameters are within the preset parameter range, and the second and third evaluation index parameters meet the preset consistency condition, it can be determined that for the sixth image, removing the IR component using the fourth removal parameter corresponding to the second preset color temperature and removing the IR component using the fourth removal parameter corresponding to the third preset color temperature can achieve the same or substantially the same removal effect, and the removal effect can meet the business requirements. Therefore, the reference color temperature can be determined as the color temperature threshold. If at least one of the second and third evaluation index parameters is outside the preset parameter range, and / or the second and third evaluation index parameters do not meet the preset consistency condition, it can be determined that for the sixth image, removing the IR component using the fourth removal parameter corresponding to the second preset color temperature and removing the IR component using the fourth removal parameter corresponding to the third preset color temperature cannot achieve the same or substantially the same removal effect, and / or the removal effect cannot meet the business requirements. Therefore, the operation of determining the reference color temperature from the color temperature range defined by the second and third preset color temperatures can be returned, and this process can be iterated until a certain reference color temperature is determined as the color temperature threshold.

[0097] In some embodiments, after obtaining the ninth and tenth images, the processor 140 can control the display screen to display the ninth and tenth images and acquire a third user input operation. The third user input operation may include, but is not limited to, text input, voice input, and touch input. If the third user input operation indicates that the ninth and tenth images have achieved the same or substantially the same removal effect, and the removal effect meets business requirements, the processor 140 can determine a reference color temperature as a color temperature threshold. If the second user input operation indicates that the ninth and tenth images have not achieved the same or substantially the same removal effect, and the removal effect does not meet business requirements, the processor 140 can return to performing the operation of determining a reference color temperature from a color temperature range defined by a second preset color temperature and a third preset color temperature.

[0098] In the embodiments of this disclosure, for the reference color temperature determined from the color temperature range defined by the second preset color temperature and the third preset color temperature, the IR component can be removed from the sixth image according to the fourth removal parameter corresponding to the second preset color temperature and the fourth removal parameter corresponding to the third preset color temperature, respectively, to obtain the seventh image and the eighth image, which are used to evaluate the rationality of the reference color temperature. Based on this, it can be determined whether to use the reference color temperature as the color temperature threshold, or to re-search for the reference color temperature and use it as the color temperature threshold after the evaluation is passed, which helps to ensure the rationality and reliability of the determined color temperature threshold.

[0099] In some optional examples, when processor 140 determines the estimated color temperature of the real-world lighting environment based on the first image, the white balance Planck curve, and the first removal parameter corresponding to the first preset color temperature, it is specifically configured as follows: According to the first removal parameters corresponding to the first preset color temperature, the IR component of the first image is removed to obtain the eleventh image; In the coordinate system of the white balance Planck curve, determine the first position corresponding to each of the multiple image blocks included in the eleven images; Based on the white balance Planck curve and the first positions corresponding to multiple image patches, the weights of the first positions corresponding to multiple image patches are determined. By using the weights of the first positions corresponding to multiple image blocks respectively, the second position is obtained by weighting the first positions corresponding to multiple image blocks respectively; Based on the white balance Planck curve and the second position, the estimated color temperature of the real-world light source environment is determined.

[0100] Optionally, the processor 140 can remove the IR components from the first image according to the removal coefficients included in the first removal parameters corresponding to the first preset color temperature to obtain the eleventh image. The specific method for obtaining the eleventh image can be found in the description of the specific method for obtaining the second image above, and will not be repeated here. Similar to the second image, the eleventh image can be a Bayer-type image.

[0101] Optionally, the processor 140 can divide the eleventh image into multiple image blocks, for example, into an average of 100 blocks. There are several image patches; where the values ​​of m and n can be preset, for example, m and n can both be set to 32 or 64. For... For each image patch, the first ratio of the mean pixel value of the R channel to the mean pixel value of the G channel, and the second ratio of the mean pixel value of the B channel to the mean pixel value of the G channel, can be calculated. In the coordinate system of the white balance Planck curve, the horizontal axis represents the first ratio, and the vertical axis represents the position of the second ratio corresponding to that image patch. This position can be used as the first position of that image patch. Similarly, the processor 140 can determine the first positions corresponding to multiple image patches.

[0102] Optionally, the processor 140 can determine the distance information (e.g., distance values) between the first positions corresponding to multiple image patches and the white balance Planck curve, and determine the weights of the first positions corresponding to the multiple image patches based on the distance information. For example, a function can be pre-set where the independent variable is the distance information, the dependent variable is the weight, and the independent and dependent variables are negatively correlated. For each image patch among the multiple image patches, the distance information corresponding to that image patch can be used as the value of the independent variable and substituted into the above function for calculation to obtain the corresponding value of the dependent variable. The obtained value can be used as the weight of the first position corresponding to that image patch.

[0103] Of course, the method for determining the weights of the first positions corresponding to multiple image patches is not limited to this. For example, the white balance Planck curve can be used as the center line to determine a region of a set size (for ease of description, this region can be referred to as the target region). For each image patch, if the first position corresponding to the image patch is outside the target region, the weight of the first position corresponding to the image patch can be determined to be zero. If the first position corresponding to the image patch is within the target region, the weight of the first position corresponding to the image patch can be determined based on the distance information between the first position corresponding to the image patch and the white balance Planck curve, as well as the function mentioned above.

[0104] Optionally, the processor 140 can use the weights of the first positions corresponding to the multiple image blocks to perform a weighted average of the first positions corresponding to the multiple image blocks to obtain the second position. Assuming the above... =N, where the first positions corresponding to multiple image patches can be represented as R1, R2, R3, ..., RN, and the weights of the first positions corresponding to multiple image patches can be represented as Q1, Q2, Q3, ..., QN, and the second position can be represented as R. Then: For ease of description, the coordinate system in which the white balance Planck curve is located will be referred to as the target coordinate system in the following text. Optionally, the calibration procedure for the white balance Planck curve may include: (1) Under each of the various preset color temperatures, the second RGB-IR image sensor is used to acquire the RGB-IR image of the 24-color card (or the gray card) (for ease of description, it will be referred to as the fourth original image).

[0105] (2) For each preset color temperature among multiple preset color temperatures, the IR component of the fourth original image corresponding to the preset color temperature can be removed according to the first removal parameter corresponding to the preset color temperature to obtain an image belonging to the Bayer format.

[0106] (3) For each preset color temperature, a target gray area is selected from the corresponding Bayer format image. For example, a fourth user input operation can be received, and the gray area selected by the fourth user input operation in the Bayer format image can be used as the target gray area. The fourth user input operation includes, but is not limited to, text input, voice input, and touch input. For the target gray area, a third ratio of the mean pixel value of the R channel to the mean pixel value of the G channel, and a fourth ratio of the mean pixel value of the B channel to the mean pixel value of the G channel can be calculated. The third ratio and the fourth ratio can form a ratio group corresponding to the preset color temperature. In a similar manner, ratio groups corresponding to various preset color temperatures can be obtained.

[0107] (4) For each preset color temperature in the multiple preset color temperature ratio groups, the third ratio in the ratio group can be used as the abscissa and the fourth ratio in the ratio group as the ordinate to locate a position in the target coordinate system (this position can be considered to be mapped to the preset color temperature). In this way, multiple positions in the target coordinate system can be located, and these positions can be connected smoothly in sequence to form a curve, which can be used as the white balance Planck curve obtained by calibration.

[0108] Optionally, the processor 140 can determine multiple positions in the target coordinate system located based on ratio sets corresponding to various preset color temperatures as multiple reference positions for defining the white balance Planck curve. The processor 140 can determine a third position closest to the second position on the white balance Planck curve, and search for the nearest reference position to the left and right of the third position, thus finding two reference positions. The processor 140 can perform linear or non-linear interpolation between the two preset color temperatures mapped to the two reference positions to obtain a new color temperature, which can be used as an estimated color temperature for the real-world lighting environment.

[0109] In some embodiments, the third position is the same as one of the plurality of reference positions, and the processor 140 can determine the preset color temperature mapped by the reference position as the estimated color temperature of the real-world lighting environment.

[0110] Of course, the method for determining the estimated color temperature of a real-world lighting environment is not limited to this. For example, one can first determine the target area mentioned above, and then select each first position located within the target area from the first positions corresponding to multiple image blocks; then determine the weight of each first position located within the target area, and use the determined weights to weight each first position located within the target area to obtain the second position; then, based on the white balance Planck curve and the second position, determine the estimated color temperature of the real-world lighting environment.

[0111] In the embodiments of this disclosure, since multiple image blocks belong to the eleventh image, and the eleventh image is obtained by removing the IR components of the first image according to the first removal parameter used to completely remove the IR components of the image, it is beneficial to eliminate the adverse effects caused by the residue of the image IR components. These adverse effects may include, for example, the random and discrete statistical landing points of the white areas at different color temperatures, which do not conform to the distribution pattern of the white balance Planck curve, leading to white balance color cast. Furthermore, in the embodiments of this disclosure, the second image is obtained by removing the IR components of the first image according to the second removal parameter used to partially remove the IR components of the image, which is beneficial to avoid significant signal-to-noise ratio loss. Therefore, the embodiments of this disclosure can achieve a balance between white balance and noise.

[0112] In some optional examples, such as Figure 6-1 As shown, processor 140 can execute the following method flow: (1) Initial calibration of IR_remove_ratio: For various preset color temperatures, the corresponding first removal parameters are calibrated. For example, under the preset color temperature corresponding to each of the standard light sources A, D40, D50, D65, and D75, RGB-IR images of a 24-color chart without an IR cutoff filter and RGB-IR images of a 24-color chart with an IR cutoff filter are acquired respectively. Based on these RGB-IR images, the first removal parameters corresponding to various preset color temperatures can be calculated. The first removal parameters include the removal coefficients corresponding to the R channel, G channel, and B channel, which can be represented as R_rate, G_rate, and B_rate respectively.

[0113] (2) IR_remove_ratio tuning: The first removal parameters corresponding to various preset color temperatures are optimized to obtain the fourth removal parameters corresponding to various preset color temperatures. For example, for the first removal parameters, R_rate, G_rate, and B_rate can be adjusted according to the proportional adjustment rule, and the corresponding fourth removal parameters can be obtained on this basis.

[0114] (3) White balance Planck curve calibration: Based on the first removal parameters corresponding to various preset color temperatures, the white balance Planck curve is calibrated.

[0115] (4) Color temperature estimation: Based on the first image acquired by the first RGB-IR image sensor in the real-world light source environment, the white balance Planck curve, and the first removal parameter corresponding to any preset color temperature, the estimated color temperature of the real-world light source environment is determined.

[0116] (5) IR_remove_ratio segmented control and distribution: Based on the estimated color temperature change trend, the target preset mapping relationship is determined from the first preset mapping relationship and the second preset mapping relationship. Based on the estimated color temperature, the second removal parameter is determined by searching the target preset mapping relationship for the removal of the IR component of the first image.

[0117] In related technologies, the effect of removing the IR component from two RGB-IR images acquired by an RGB-IR image sensor under both D65 and A light source environments can be found in the following image. Figure 6-2 (Where the left side corresponds to D65 and the right side corresponds to A light). In the embodiments of this disclosure, the effect of removing the IR component from two RGB-IR images acquired by the RGB-IR image sensor under both D65 and A light light source environments can be found in the following diagram. Figure 6-3 (The left side corresponds to D65, and the right side corresponds to A beam). Through comparison... Figure 6-2 and Figure 6-3 As can be seen, the image noise in the related technologies is severe and the color reproduction effect is poor. The image noise in the embodiments of this disclosure can be significantly improved, and the embodiments of this disclosure can achieve good color reproduction effect.

[0118] In summary, the embodiments of this disclosure can adaptively control the IR component removal ratio through segmented color temperature adaptation, which is beneficial to ensuring the accuracy of color reproduction at medium and high color temperatures, and also beneficial to ensuring the signal-to-noise ratio at low color temperatures.

[0119] Exemplary methods Figure 7 This is a schematic flowchart illustrating an image processing method provided by some exemplary embodiments of this disclosure. The image processing method can be applied to the image processing apparatus 14 in any of the above embodiments. Figure 7 As shown, the image processing methods include: Step 710: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Step 720: Obtain the first removal parameter for the first preset color temperature calibration; wherein, the first removal parameter is a parameter used to completely remove the IR components of the image; Step 730: Obtain the white balance Planck curve of the first RGB-IR image sensor; Step 740: Based on the first image, the white balance Planck curve, and the first removal parameter corresponding to the first preset color temperature, determine the estimated color temperature of the real-world light source environment; Step 750: Determine the estimated trend of color temperature change; Step 760: Based on the estimated color temperature and its changing trend, determine the second removal parameter; wherein, the second removal parameter is a parameter used to partially remove the IR component of the image; Step 770: According to the second removal parameters, the IR component is removed from the first image to obtain the second image.

[0120] In some optional examples, such as Figure 8 As shown, step 760 includes: Step 810: Based on the changing trend, determine the target preset mapping relationship from multiple preset mapping relationships; wherein, each preset mapping relationship is a mapping relationship between multiple color temperature segments in a predetermined color temperature range and the corresponding third removal parameter, the third removal parameter is a parameter used to partially remove the IR component of the image, and the first preset color temperature is located in the predetermined color temperature range; Step 820: Determine the third removal parameter for the color temperature range mapping to which the estimated color temperature belongs, according to the target preset mapping relationship; Step 830: Determine the third removal parameter of the color temperature range mapping to which the estimated color temperature belongs as the second removal parameter.

[0121] In some optional examples, such as Figure 9 As shown, the method provided in the embodiments of this disclosure further includes: Step 910: For each of the multiple preset color temperatures within the predetermined color temperature range, a corresponding first removal parameter is assigned; wherein, the first preset color temperature is one of the multiple preset color temperatures. Step 920: Optimize the first removal parameters corresponding to the various preset color temperatures to obtain the fourth removal parameters corresponding to the various preset color temperatures; Step 930: Determine at least one color temperature group from a variety of preset color temperatures; wherein each color temperature group includes two adjacent preset color temperatures from the variety of preset color temperatures. Step 940: Determine the color temperature threshold and the corresponding color temperature jitter threshold for at least one color temperature group; Step 950: Based on the predetermined color temperature range, the fourth removal parameters corresponding to the various preset color temperatures, and the color temperature thresholds and color temperature jitter thresholds corresponding to at least one color temperature group, generate a variety of preset mapping relationships.

[0122] In some optional examples, multiple preset mapping relationships include a first preset mapping relationship and a second preset mapping relationship, and the color temperature jitter threshold corresponding to each color temperature group includes a first jitter threshold and a second jitter threshold; like Figure 10 As shown, step 950 includes: Step 1010: For each color temperature group, subtract the color temperature threshold corresponding to the color temperature group from the corresponding first jitter threshold to obtain the first adjusted color temperature; add the color temperature threshold corresponding to the color temperature group to the corresponding second jitter threshold to obtain the second adjusted color temperature. Step 1020: Use each first adjusted color temperature as a dividing point to divide the predetermined color temperature range into multiple first color temperature segments; Step 1030: Use each second adjusted color temperature as a dividing point to divide the predetermined color temperature range into multiple second color temperature segments; Step 1040: Based on the fourth removal parameters corresponding to various preset color temperatures, determine the third removal parameters mapped to various first color temperature segments respectively, and generate a first preset mapping relationship between various first color temperature segments and the determined corresponding third removal parameters. Step 1050: Based on the fourth removal parameters corresponding to various preset color temperatures, determine the third removal parameters mapped to various second color temperature segments respectively, and generate a second preset mapping relationship between various second color temperature segments and the determined corresponding third removal parameters.

[0123] In some optional examples, step 810 includes: In response to the changing trend of color temperature decreasing from high to low, the first preset mapping relationship is determined as the target preset mapping relationship; In response to the changing trend of color temperature increasing from low to high, the second preset mapping relationship is determined as the target preset mapping relationship.

[0124] In some optional examples, such as Figure 11 As shown, step 920 includes: Step 1110: Acquire a third image from the second RGB-IR image sensor under a standard light source environment that meets the first preset color temperature; wherein the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor. Step 1120: Adjust the first removal parameter corresponding to the first preset color temperature to obtain the fifth removal parameter; wherein, the first removal parameter includes the first removal coefficients corresponding to the R channel, G channel and B channel respectively, and the fifth removal parameter includes the second removal coefficients corresponding to the R channel, G channel and B channel respectively, each second removal coefficient is less than or equal to the corresponding first removal coefficient, and at least one second removal coefficient is different from the corresponding first removal coefficient. Step 1130: Based on the third image and the fifth removal parameter, determine the fourth removal parameter corresponding to the first preset color temperature.

[0125] In some optional examples, such as Figure 12 As shown, step 1130 includes: Step 1210: Remove the IR components from the third image according to the second removal coefficient included in the fifth removal parameter to obtain the fourth image; Step 1220: Based on the fourth image, determine the fifth image that belongs to the visualization type; Step 1230: Evaluate the fifth image to obtain the first evaluation index parameters; Step 1240: In response to the first evaluation index parameter being within the preset parameter range, the fifth removal parameter is determined to be the fourth removal parameter corresponding to the first preset color temperature; Step 1250: In response to the first evaluation index parameter being outside the preset parameter range, return to perform the operation of adjusting the first removal parameter corresponding to the first preset color temperature to obtain the fifth removal parameter.

[0126] In some optional examples, the two preset color temperatures in each color temperature group are represented as a second preset color temperature and a third preset color temperature, such as... Figure 13 As shown, for this color temperature group, at least one color temperature threshold corresponding to each color temperature group is determined, including: Step 1310: Determine a reference color temperature from the color temperature range defined by the second preset color temperature and the third preset color temperature; Step 1320: Acquire a sixth image from the second RGB-IR image sensor under a test light source environment that conforms to the reference color temperature; wherein the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor. Step 1330: According to the fourth removal parameter corresponding to the second preset color temperature, remove the IR component from the sixth image to obtain the seventh image; Step 1340: According to the fourth removal parameter corresponding to the third preset color temperature, remove the IR component from the sixth image to obtain the eighth image; Step 1350: Determine the color temperature threshold based on the reference color temperature, the seventh image, and the eighth image.

[0127] In some optional examples, step 1350 includes: Based on the seventh image, the ninth image is identified as belonging to the visualization type; Based on the eighth image, the tenth image is identified as belonging to the visualization type; The ninth image is evaluated to obtain the second evaluation index parameters; The tenth image is evaluated to obtain the third evaluation index parameters; In response to the fact that both the second and third evaluation index parameters are within the preset parameter range and the second and third evaluation index parameters meet the preset consistency condition, the reference color temperature is determined as the color temperature threshold. In response to at least one of the second and third evaluation index parameters being outside the preset parameter range, and / or the second and third evaluation index parameters not meeting the preset consistency conditions, the operation of determining a reference color temperature from the color temperature range defined by the second and third preset color temperatures is returned.

[0128] In some optional examples, such as Figure 14 As shown, step 740 includes: Step 1410: Remove the IR component from the first image according to the first removal parameter corresponding to the first preset color temperature to obtain the eleventh image; Step 1420: In the coordinate system where the white balance Planck curve is located, determine the first position corresponding to each of the multiple image blocks included in the eleven images; Step 1430: Based on the white balance Planck curve and the first positions corresponding to the multiple image blocks respectively, determine the weights of the first positions corresponding to the multiple image blocks respectively; Step 1440: Using the weights of the first positions corresponding to the multiple image blocks respectively, the first positions corresponding to the multiple image blocks are weighted to obtain the second position; Step 1450: Determine the estimated color temperature of the real-world lighting environment based on the white balance Planck curve and the second position.

[0129] In the methods disclosed herein, the various optional embodiments, optional implementation methods and optional examples disclosed in the above exemplary device section can be flexibly selected and combined as needed to achieve the corresponding functions and effects, and this disclosure does not list them all.

[0130] The beneficial technical effects corresponding to the exemplary embodiments of this method can be found in the corresponding beneficial technical effects of the exemplary device section above, and will not be repeated here.

[0131] Exemplary electronic devices Figure 15 The illustration shows a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 1500 includes one or more processors 1510 and memory 1520.

[0132] The processor 1510 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1500 to perform desired functions.

[0133] The memory 1520 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1510 may execute one or more computer program instructions to implement the image processing method or instruction execution method and / or other desired functions of the various embodiments of this disclosure described above.

[0134] In one example, the electronic device 1500 may also include an input device 1530 and an output device 1540, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0135] The input device 1530 may also include, for example, a keyboard, a mouse, etc.

[0136] The output device 1540 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0137] Of course, for the sake of simplicity, Figure 15 Only some of the components of the electronic device 1500 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1500 may include any other suitable components depending on the specific application.

[0138] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0139] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0140] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0141] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0142] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. The specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0143] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An image processing apparatus, comprising a processor, the processor being configured to: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain the first removal parameter for the first preset color temperature calibration; wherein... The first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the white balance Planck curve, and the first removal parameter corresponding to the first preset color temperature, the estimated color temperature of the real-world light source environment is determined; Determine the trend of the estimated color temperature; Based on the estimated color temperature and the changing trend, a second removal parameter is determined; wherein, the second removal parameter is a parameter used to partially remove the IR component of the image; According to the second removal parameters, the first image is subjected to IR component removal to obtain the second image.

2. The image processing apparatus according to claim 1, wherein, When the processor determines the second removal parameter based on the estimated color temperature and the changing trend, it is specifically configured as follows: Based on the changing trend, a target preset mapping relationship is determined from a variety of preset mapping relationships; wherein each preset mapping relationship is a mapping relationship between multiple color temperature segments in a predetermined color temperature range and a corresponding third removal parameter, the third removal parameter is a parameter used to partially remove the IR component of the image, and the first preset color temperature is located in the predetermined color temperature range; According to the target preset mapping relationship, determine the third removal parameter of the color temperature range to which the estimated color temperature belongs; The third removal parameter, which maps the estimated color temperature to the color temperature range, is determined as the second removal parameter.

3. The image processing apparatus according to claim 2, wherein, The processor is also configured to: For each of the various preset color temperatures within the predetermined color temperature range, a corresponding first removal parameter is assigned; wherein, the first preset color temperature is one of the various preset color temperatures; The first removal parameters corresponding to the various preset color temperatures are optimized to obtain the fourth removal parameters corresponding to the various preset color temperatures. From a plurality of preset color temperatures, at least one color temperature group is determined; wherein each color temperature group includes two adjacent preset color temperatures from the plurality of preset color temperatures; Determine the color temperature threshold and the corresponding color temperature jitter threshold for at least one of the color temperature groups; Based on the predetermined color temperature range, the fourth removal parameters corresponding to the various preset color temperatures, and the color temperature threshold and color temperature jitter threshold corresponding to at least one color temperature group, a variety of preset mapping relationships are generated.

4. The image processing apparatus according to claim 3, wherein, The various preset mapping relationships include a first preset mapping relationship and a second preset mapping relationship, and the color temperature jitter threshold corresponding to each color temperature group includes a first jitter threshold and a second jitter threshold; When the processor generates multiple preset mapping relationships based on the predetermined color temperature range, the fourth removal parameters corresponding to the various preset color temperatures, and the color temperature thresholds and color temperature jitter thresholds corresponding to at least one of the color temperature groups, it is specifically configured as follows: For each color temperature group, the color temperature threshold corresponding to the color temperature group is subtracted from the corresponding first jitter threshold to obtain the first adjusted color temperature; the color temperature threshold corresponding to the color temperature group is added to the corresponding second jitter threshold to obtain the second adjusted color temperature. Each of the first adjusted color temperatures is used as a dividing point to divide the predetermined color temperature range into multiple first color temperature segments; Each second adjusted color temperature is used as a dividing point to divide the predetermined color temperature range into multiple second color temperature segments; Based on the fourth removal parameters corresponding to the various preset color temperatures, the third removal parameters mapped to the various first color temperature segments are determined, and the first preset mapping relationship between the various first color temperature segments and the determined corresponding third removal parameters is generated. Based on the fourth removal parameters corresponding to the various preset color temperatures, third removal parameters corresponding to the various second color temperature segments are determined, and second preset mapping relationships between the various second color temperature segments and the determined corresponding third removal parameters are generated.

5. The image processing apparatus according to claim 4, wherein, When the processor determines the target preset mapping relationship from multiple preset mapping relationships based on the changing trend, it is specifically configured as follows: In response to the trend of color temperature decreasing from high to low, the first preset mapping relationship is determined as the target preset mapping relationship; In response to the trend of color temperature increasing from low to high, the second preset mapping relationship is determined as the target preset mapping relationship.

6. The image processing apparatus according to claim 3, wherein, When the processor optimizes the first removal parameter corresponding to the first preset color temperature to obtain the fourth removal parameter corresponding to the first preset color temperature, it is specifically configured as follows: A third image is acquired by a second RGB-IR image sensor under a standard light source environment that meets the first preset color temperature; wherein the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor. The first removal parameter corresponding to the first preset color temperature is adjusted to obtain the fifth removal parameter; wherein, the first removal parameter includes the first removal coefficients corresponding to the R channel, G channel and B channel respectively, and the fifth removal parameter includes the second removal coefficients corresponding to the R channel, G channel and B channel respectively, each of the second removal coefficients is less than or equal to the corresponding first removal coefficient, and at least one of the second removal coefficients is different from the corresponding first removal coefficient. Based on the third image and the fifth removal parameter, the fourth removal parameter corresponding to the first preset color temperature is determined.

7. The image processing apparatus according to claim 6, wherein, When the processor determines the fourth removal parameter corresponding to the first preset color temperature based on the third image and the fifth removal parameter, it is specifically configured as follows: The third image is subjected to IR component removal according to the second removal coefficient included in the fifth removal parameter to obtain the fourth image; Based on the fourth image, a fifth image belonging to the visualization type is determined; The fifth image is evaluated to obtain the first evaluation index parameter; In response to the first evaluation index parameter being within the preset parameter range, the fifth removal parameter is determined to be the fourth removal parameter corresponding to the first preset color temperature; In response to the first evaluation index parameter being outside the preset parameter range, the operation of adjusting the first removal parameter corresponding to the first preset color temperature to obtain the fifth removal parameter is returned.

8. The image processing apparatus according to claim 3, wherein, The two preset color temperatures in each color temperature group are represented as a second preset color temperature and a third preset color temperature. When the processor determines the color temperature threshold corresponding to the color temperature group, it is specifically configured as follows: A reference color temperature is determined from the color temperature range defined by the second preset color temperature and the third preset color temperature; A sixth image is acquired by the second RGB-IR image sensor under a test light source environment that conforms to the reference color temperature; wherein the optical specifications of the second RGB-IR image sensor are the same as those of the first RGB-IR image sensor. According to the fourth removal parameter corresponding to the second preset color temperature, the IR component is removed from the sixth image to obtain the seventh image; According to the fourth removal parameter corresponding to the third preset color temperature, the IR component is removed from the sixth image to obtain the eighth image; The color temperature threshold is determined based on the reference color temperature, the seventh image, and the eighth image.

9. The image processing apparatus according to claim 8, wherein, When the processor determines the color temperature threshold based on the reference color temperature, the seventh image, and the eighth image, it is specifically configured as follows: Based on the seventh image, a ninth image belonging to the visualization type is determined; Based on the eighth image, the tenth image, which belongs to the visualization type, is determined; The ninth image is evaluated to obtain the second evaluation index parameters; The tenth image is evaluated to obtain the third evaluation index parameter; In response to the fact that both the second evaluation index parameter and the third evaluation index parameter are within the preset parameter range, and the second evaluation index parameter and the third evaluation index parameter meet the preset consistency condition, the reference color temperature is determined as the color temperature threshold. In response to at least one of the second evaluation index parameter and the third evaluation index parameter being outside the preset parameter range, and / or the second evaluation index parameter and the third evaluation index parameter not meeting the preset consistency condition, the operation of determining the reference color temperature from the color temperature range defined by the second preset color temperature and the third preset color temperature is returned.

10. The image processing apparatus according to claim 1, wherein, When the processor determines the estimated color temperature of the real-world light source environment based on the first image, the white balance Planck curve, and the first removal parameter corresponding to the first preset color temperature, it is specifically configured as follows: According to the first removal parameter corresponding to the first preset color temperature, the first image is subjected to IR component removal to obtain the eleventh image; In the coordinate system where the white balance Planck curve is located, determine the first position corresponding to each of the multiple image blocks included in the eleven images; Based on the white balance Planck curve and the first positions corresponding to the multiple image patches respectively, the weights of the first positions corresponding to the multiple image patches are determined. By using the weights of the first positions corresponding to the multiple image blocks respectively, the first positions corresponding to the multiple image blocks are weighted to obtain the second position; Based on the white balance Planck curve and the second position, the estimated color temperature of the real-world light source environment is determined.

11. An image processing method, comprising: Acquire the first image captured by the first RGB-IR image sensor under real-world lighting conditions; Obtain a first removal parameter for a first preset color temperature calibration; wherein the first removal parameter is a parameter used to completely remove the IR components of the image; Obtain the white balance Planck curve of the first RGB-IR image sensor; Based on the first image, the white balance Planck curve, and the first removal parameter corresponding to the first preset color temperature, the estimated color temperature of the real-world light source environment is determined; Determine the trend of the estimated color temperature; Based on the estimated color temperature and the changing trend, a second removal parameter is determined; wherein, the second removal parameter is a parameter used to partially remove the IR component of the image; According to the second removal parameters, the first image is subjected to IR component removal to obtain the second image.

12. A computer-readable storage medium storing a computer program that is executed by a processor to implement the image processing method of claim 11.

13. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image processing method of claim 11.

Images