Image restoration methods, apparatus, computer-readable storage media, and electronic devices

Abstract

A method, apparatus, computer-readable storage medium, and electronic device for image restoration are disclosed. The method includes: acquiring an image to be processed, the image to be processed including an IR channel; determining the pixel brightness value of a pixel in the image to be processed, and a first pixel value of the pixel in the IR channel; determining the image color temperature of the image to be processed; removing the IR component from the pixel based on the pixel brightness value, the first pixel value, and the image color temperature; and determining a restored image based on the image to be processed after IR component removal. Embodiments of this disclosure can improve image restoration performance.

Description

Technical Field

[0001] This disclosure relates to image technology, and in particular to an image restoration method, apparatus, computer-readable storage medium, and electronic device. Background Technology

[0002] The application of RGBIR pattern sensors (i.e., RGBIR array sensors) is becoming increasingly widespread; where R represents red, G represents green, B represents blue, and IR represents infrared.

[0003] Generally speaking, after obtaining the image acquired by the RGBIR pattern sensor, the acquired image needs to be interpolated to include four channels, and then the IR component is removed to restore the true RGB color image. Summary of the Invention

[0004] To address the technical problem of unsatisfactory image restoration results in current methods, this disclosure is proposed. Embodiments of this disclosure provide an image restoration method, apparatus, computer-readable storage medium, and electronic device.

[0005] According to one aspect of the present disclosure, an image restoration method is provided, comprising:

[0006] Acquire an image to be processed, the image to be processed including an IR channel;

[0007] Determine the pixel brightness value of a pixel in the image to be processed, and the first pixel value of the pixel in the IR channel;

[0008] Determine the color temperature of the image to be processed;

[0009] Based on the pixel brightness value, the first pixel value, and the image color temperature, the IR component of the pixel is removed;

[0010] Based on the image to be processed after removing the IR components from the pixels, the restored image is determined.

[0011] According to another aspect of the present disclosure, an image restoration apparatus is provided, comprising:

[0012] The acquisition module is used to acquire the image to be processed, the image to be processed including an IR channel;

[0013] The first determining module is used to determine the pixel brightness value of the pixel in the image to be processed acquired by the acquisition module, and the first pixel value of the pixel in the IR channel;

[0014] The second determining module is used to determine the image color temperature of the image to be processed acquired by the acquiring module;

[0015] The removal module is used to remove the IR component from the pixel based on the pixel brightness value and the first pixel value determined by the first determining module and the image color temperature determined by the second determining module.

[0016] The third determining module is used to determine the restored image based on the image to be processed after the IR component is removed from the pixels by the removing module.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the above-described image restoration method.

[0018] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0019] processor;

[0020] Memory used to store the processor's executable instructions;

[0021] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image restoration method described above.

[0022] Based on the image restoration method, apparatus, computer-readable storage medium, and electronic device provided in the above embodiments of this disclosure, IR component removal can be performed on pixels by referring to the pixel brightness value, the first pixel value of the pixel in the IR channel, and the image color temperature of the image to be processed. This allows for the determination of the restored image based on the image to be processed after IR component removal. Since pixel brightness value and image color temperature are both important information carried by the image to be processed, guided by these values, IR component removal can be performed on pixels reasonably according to their brightness level. This achieves IR component removal without causing brightness reversal, ensuring effective IR component removal and more realistically restoring image color information. It also avoids brightness reversal in bright areas of the image, thereby improving the image restoration effect.

[0023] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0024] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 This is a schematic flowchart of an image restoration method provided in an exemplary embodiment of this disclosure.

[0026] Figure 2 This is a flowchart illustrating an image restoration method provided in another exemplary embodiment of this disclosure.

[0027] Figure 3 This is a schematic flowchart of an image restoration method provided in another exemplary embodiment of the present disclosure.

[0028] Figure 4-1 This is a schematic diagram of the first function in an exemplary embodiment of this disclosure.

[0029] Figure 4-2 This is a schematic diagram of the second function in an exemplary embodiment of this disclosure.

[0030] Figure 4-3 This is a schematic diagram of a third function in an exemplary embodiment of this disclosure.

[0031] Figure 5 This is a schematic diagram of the first rule in an exemplary embodiment of this disclosure.

[0032] Figure 6 This is a schematic flowchart of an image restoration method provided in yet another exemplary embodiment of this disclosure.

[0033] Figure 7 This is a schematic diagram of the second rule in an exemplary embodiment of this disclosure.

[0034] Figure 8 This is a schematic diagram of the structure of an image restoration apparatus provided in an exemplary embodiment of the present disclosure.

[0035] Figure 9 This is a schematic diagram of the structure of an image restoration apparatus provided in another exemplary embodiment of the present disclosure.

[0036] Figure 10 This is a schematic diagram of the structure of an image restoration apparatus provided in another exemplary embodiment of the present disclosure.

[0037] Figure 11 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0038] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0039] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0040] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0041] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0042] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0043] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0044] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0045] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0046] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0047] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0048] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0049] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0050] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0051] Application Overview

[0052] After obtaining the image acquired by the RGBIR pattern sensor, the following two steps can be performed: First, interpolate the acquired image into an image including four channels (which can be simply referred to as a 4-channel image); second, remove the IR components from the interpolated image to restore the true RGB color image. The combination of the above two steps can be considered as remosaic processing of the acquired image.

[0053] Assuming that the pixel value of a pixel in the interpolated image is represented as R in the R channel, G in the G channel, B in the B channel, and IR in the IR channel, the following formula can be used to remove the IR component of the pixel: R' = R - IR, G' = G - IR, B' = B - IR; where R' represents the pixel value of the pixel after removing the IR component in the R channel, G' represents the pixel value of the pixel after removing the IR component in the G channel, and B' represents the pixel value of the pixel after removing the IR component in the B channel.

[0054] In developing this disclosure, the inventors discovered that in low-brightness areas of an image, R>IR, G>IR, and B>IR. Using these formulas usually yields correct R', G', and B', ensuring accurate image restoration. However, as brightness increases, R, G, B, and IR all increase, with IR increasing more significantly than R, G, and B. This results in R', G', and B' being zero in high-brightness areas of the image. In other words, during image restoration, brightness reversal occurs in high-brightness areas. Therefore, current image restoration methods are ineffective and fail to meet practical needs.

[0055] Exemplary methods

[0056] Figure 1 This is a schematic flowchart of an image restoration method provided in an exemplary embodiment of this disclosure. Figure 1 The method shown can be applied to electronic devices. Figure 1 The method shown may include steps 110, 120, 130, 140 and 150, each of which will be explained below.

[0057] Step 110: Obtain the image to be processed, which includes the IR channel.

[0058] In step 110, an image can be acquired in real time using an RGBIR pattern sensor to obtain a real-time acquired image. The real-time acquired image is then interpolated into an image that includes four channels, and the interpolated image can be used as the image to be processed.

[0059] Of course, in step 110, historical images acquired by the RGBIR pattern sensor can also be obtained from the image library, and the image to be processed can be obtained by interpolation of the historical images.

[0060] It should be noted that the image to be processed may include multiple pixels. When performing image restoration, the processing methods for each of the multiple pixels are similar. Therefore, the embodiments of this disclosure mainly focus on the processing methods for a single pixel.

[0061] Step 120: Determine the pixel brightness value of the pixel in the image to be processed, and the first pixel value of the pixel in the IR channel.

[0062] Optionally, the pixel brightness value of a pixel can be used to characterize the brightness level of the pixel. The pixel brightness value of a pixel can be calculated based on the pixel values ​​of the pixel in the channels other than the IR channel. For clarity, the specific calculation method will be illustrated with examples later.

[0063] Optionally, the first pixel value of a pixel in the IR channel, as well as the pixel values ​​in the other channels besides the IR channel, can be directly extracted from the image to be processed.

[0064] Step 130: Determine the color temperature of the image to be processed.

[0065] In step 130, a color temperature detection algorithm can be used to detect the color temperature of the image to be processed in order to obtain the image color temperature of the image to be processed.

[0066] Step 140: Based on the pixel brightness value, the first pixel value, and the image color temperature, remove the IR component from the pixel.

[0067] In step 140, by removing the IR component from the pixel, the IR component removal result of the pixel can be obtained. The IR component removal result can include: the pixel value of the pixel after removing the IR component in each channel other than the IR channel.

[0068] Step 150: Determine the restored image based on the image to be processed after removing the IR components from the pixels.

[0069] In step 150, the IR component removal results of multiple pixels in the image to be processed can be used to generate a restored image. The restored image can be an RGB color image, and the multiple pixels in the restored image can correspond one-to-one with the multiple pixels in the image to be processed. Each pixel in the restored image can be used to represent the IR component removal result of the corresponding pixel in the image to be processed.

[0070] Based on the image restoration method provided in the above embodiments of this disclosure, the IR component of each pixel can be removed by referring to the pixel brightness value, the first pixel value of the pixel in the IR channel, and the color temperature of the image to be processed. This allows for the determination of the restored image based on the image after IR component removal. Since pixel brightness value and image color temperature are both important information carried by the image to be processed, guided by these values, IR component removal can be performed reasonably on each pixel according to its brightness level. This achieves IR component removal without causing brightness reversal, ensuring effective IR component removal and more realistically restoring the image color information. It also avoids brightness reversal in bright areas of the image, thereby improving the image restoration effect.

[0071] exist Figure 1 Based on the illustrated embodiment, the image to be processed further includes an R channel, a G channel, and a B channel. Step 120 involves determining the pixel brightness values ​​of the pixels in the image to be processed, including... Figure 2 Steps 1201, 1203, 1205, and 1207 in the process.

[0072] Step 1201: Determine the minimum and maximum pixel values ​​from the second pixel value in the R channel, the third pixel value in the G channel, and the fourth pixel value in the B channel of the pixel in the image to be processed.

[0073] In step 1201, the pixel values ​​of the pixels in the R channel can be extracted from the image to be processed as the second pixel value, the pixel values ​​of the pixels in the G channel can be extracted as the third pixel value, and the pixel values ​​of the pixels in the B channel can be extracted as the fourth pixel value.

[0074] By comparing the second, third, and fourth pixel values ​​in pairs, the pixel value with the smallest value can be selected as the minimum pixel value, and the pixel value with the largest value can be selected as the maximum pixel value.

[0075] Step 1203: Determine the first weight corresponding to the minimum pixel value and the second weight corresponding to the maximum pixel value.

[0076] Optionally, a weight can be preset. The preset weight can be 0.5, 0.6, 0.7, etc., which will not be listed here.

[0077] In step 1203, a pre-set weight can be used as the first weight corresponding to the minimum pixel value, and the difference between 1.0 and the first weight can be used as the second weight corresponding to the maximum pixel value; or, a pre-set weight can be used as the second weight corresponding to the maximum pixel value, and the difference between 1.0 and the second weight can be used as the first weight corresponding to the minimum pixel value.

[0078] Step 1205: Using the first weight and the second weight, the minimum pixel value and the maximum pixel value are weighted to obtain a weighted value.

[0079] In step 1205, the minimum pixel value and the maximum pixel value can be weighted and summed to obtain a weighted value.

[0080] Step 1207: Determine the pixel brightness value of the pixel based on the weighted value.

[0081] In step 1207, the weighted value can be directly used as the pixel brightness value of the pixel; or, the weighted value can be pre-processed, and the processing result corresponding to the pre-processing can be used as the pixel brightness value of the pixel.

[0082] Optionally, the pre-defined processing for any value includes, but is not limited to: multiplying the value by a preset coefficient, adding the value to a preset value, and subtracting the value from a preset value. It should be noted that the pre-defined processing mentioned below refers to the relevant description in this paragraph, and will not be elaborated upon further below.

[0083] Assuming the first weight is represented as 1.0 - blending_rgb_rate, the minimum pixel value is represented as min_rgb, the second weight is represented as blending_rgb_rate, the maximum pixel value is represented as max_rgb, and the pixel brightness value is represented as blending_rgb, then the pixel brightness value can be calculated using the following formula: blending_rgb = blending_rgb_rate * max_rgb + (1.0 - blending_rgb_rate) * min_rgb.

[0084] In the embodiments of this disclosure, by obtaining the pixel values ​​of the pixel in the R channel, G channel, and B channel respectively, and combining simple operation logic such as pixel value comparison operation and weighted operation, the pixel brightness value of the pixel can be determined efficiently and reliably. The pixel brightness value can effectively characterize the brightness level of the pixel.

[0085] Of course, in practice, the method of determining the pixel brightness value of a pixel is not limited to this. For example, after obtaining the pixel values ​​of a pixel in the R channel, G channel, and B channel, the three pixel values ​​can be weighted to obtain the pixel brightness value of the pixel.

[0086] Thus, assuming the second pixel value is represented as R, the third pixel value as G, and the fourth pixel value as B, the pixel brightness value is represented as blending_rgb. The pixel brightness value can also be calculated using the following formula: blending_rgb=R*77+G*128+B*51; where 77, 128, and 51 in this formula are values ​​determined based on experience, and users can adjust 77, 128, and 51 to other values ​​according to the actual situation.

[0087] exist Figure 1 Based on the illustrated embodiment, the image to be processed also includes an R channel, a G channel, and a B channel, such as... Figure 3 As shown, step 140 includes steps 1401, 1403, 1405 and 1407.

[0088] Step 1401: Based on the image color temperature, determine the first basic ratio corresponding to the R channel, the second basic ratio corresponding to the G channel, and the third basic ratio corresponding to the B channel.

[0089] Optionally, the first basic ratio corresponding to the R channel can be represented as r_rate, the second basic ratio corresponding to the G channel can be represented as g_rate, and the third basic ratio corresponding to the B channel can be represented as b_rate.

[0090] In an optional example, step 1401, determining the first base ratio corresponding to the R channel based on the image color temperature, includes:

[0091] Determine the preset brightness value corresponding to the R channel;

[0092] The pixel brightness value is compared with the preset brightness value to obtain the comparison result;

[0093] Based on the image color temperature and comparison results, the first basic ratio corresponding to the R channel is determined.

[0094] Optionally, the correspondence between channels and brightness values ​​can be preset. Based on the preset correspondence, the brightness value corresponding to the R channel can be determined efficiently and reliably. The determined brightness value can be used as the preset brightness value corresponding to the R channel. The preset brightness value can be used to compare the size with the pixel brightness value.

[0095] In one optional implementation, based on the image color temperature and comparison results, the first basic ratio corresponding to the R channel is determined, including:

[0096] In response to the comparison result indicating that the pixel brightness value is less than or equal to the preset brightness value and the image color temperature is the same as the first preset base color temperature, the first base ratio corresponding to the R channel is determined based on the R channel base ratio obtained by pre-calibration for the first preset base color temperature.

[0097] In response to the comparison result indicating that the pixel brightness value is less than or equal to the preset brightness value, the image color temperature is distributed between the second preset base color temperature and the third preset base color temperature, and the second preset base color temperature and the third preset base color temperature are two adjacent preset base color temperatures. Interpolation is performed between the corresponding R channel base ratios obtained by pre-calibrating the second preset base color temperature and the third preset base color temperature, and the first base ratio corresponding to the R channel is determined based on the interpolation result.

[0098] It should be noted that multiple preset base color temperatures can be set, and each preset base color temperature can be calibrated in advance through experiments to obtain the calibration results corresponding to each preset base color temperature. Among them, the calibration results corresponding to any preset base color temperature may include: the base ratio of the R channel, the base ratio of the G channel, and the base ratio of the B channel corresponding to this preset base color temperature.

[0099] If the comparison result indicates that the pixel brightness value is less than or equal to a preset brightness value, and the image color temperature is the same as one of the multiple preset base color temperatures, then this preset base color temperature can be used as the first preset base color temperature, and the R channel base ratio can be obtained from the calibration result corresponding to the first preset base color temperature. Next, the obtained R channel base ratio can be directly used as the first base ratio corresponding to the R channel; alternatively, the obtained R channel base ratio can be pre-processed, and the processing result corresponding to this pre-processing can be used as the first base ratio corresponding to the R channel.

[0100] If the comparison result indicates that the pixel brightness is less than or equal to a preset brightness value, and the image color temperature is different from each of the multiple preset base color temperatures, and the image color temperature is exactly distributed between two adjacent preset base color temperatures (e.g., between the second and third preset base color temperatures), then the R-channel base ratio can be obtained from the calibration results corresponding to the second and third preset base color temperatures, thus obtaining two R-channel base ratios. Referring to the differences between the image color temperature and the second and third preset base color temperatures, linear or non-linear interpolation can be performed between the obtained two R-channel base ratios to obtain a first interpolation base ratio. Next, the first interpolation base ratio can be directly used as the first base ratio corresponding to the R channel; alternatively, the first interpolation base ratio can be pre-processed, and the processing result corresponding to the pre-processing can be used as the first base ratio corresponding to the R channel.

[0101] In this implementation, by pre-calibrating multiple preset base color temperatures, for low-brightness areas of the image, the distribution of the image color temperature can be referenced to determine the first base ratio corresponding to the R channel in an appropriate manner, so as to ensure the rationality and reliability of the determined first base ratio.

[0102] In another alternative implementation, based on the image color temperature and comparison results, the first basic ratio corresponding to the R channel is determined, including:

[0103] In response to the comparison result indicating that the pixel brightness value is greater than the preset brightness value, the first basic ratio corresponding to the R channel is determined based on the pre-constructed mapping relationship between the brightness value and the basic ratio of the R channel, the pixel brightness value, and the image color temperature.

[0104] It should be noted that multiple preset base color temperatures can be set, and through prior experiments, a mapping relationship between luminance values ​​and the base ratio of the R channel can be established for each of these preset base color temperatures. This mapping relationship can be represented as a first function; where the independent variable of the first function can be the luminance value, and the dependent variable can be the base ratio of the R channel. Optionally, the first function can be found in [reference needed]. Figure 4-1 The slash part in the text represents the function, in Figure 4-1 In the diagram, the horizontal axis represents the brightness value, the vertical axis represents the base ratio of the R channel, max_r represents the preset brightness value corresponding to the R channel, and max_v represents the maximum value in the range of image pixel values ​​(e.g., 255). Thus, the first function can be expressed as the following formula: f(x) = ax + b, where f(x) represents the base ratio of the R channel, x represents the brightness value, and x lies between max_r and max_v.

[0105] If the comparison result indicates that the pixel brightness value is greater than a preset brightness value, and the image color temperature is the same as one of the multiple preset base color temperatures, then this preset base color temperature can be used as the first preset base color temperature. Based on the mapping relationship constructed for the first preset base color temperature, the base ratio of the R channel mapped to the pixel brightness value is determined. Next, the determined base ratio of the R channel can be used as the first base ratio corresponding to the R channel; alternatively, the determined base ratio of the R channel can undergo predetermined processing, and the processing result corresponding to this predetermined processing can be used as the first base ratio corresponding to the R channel.

[0106] If the comparison result indicates that the pixel brightness value is greater than the preset brightness value, and the image color temperature is different from each of the multiple preset base color temperatures, and the image color temperature is exactly distributed between two adjacent preset base color temperatures (e.g., between the second preset base color temperature and the third preset base color temperature), then the R-channel base ratio mapped to the pixel brightness value can be determined based on the mapping relationship constructed for the second and third preset base color temperatures, thus obtaining two R-channel base ratios. By performing linear or non-linear interpolation between the two obtained R-channel base ratios, a second interpolated base ratio can be obtained. Next, the second interpolated base ratio can be directly used as the first base ratio corresponding to the R channel; alternatively, the second interpolated base ratio can be pre-processed, and the processing result corresponding to the pre-processed value can be used as the first base ratio corresponding to the R channel.

[0107] Of course, in practical implementation, instead of constructing separate mapping relationships between luminance values ​​and R-channel base ratios for multiple preset base color temperatures, a single unified mapping relationship can be constructed. If the comparison result indicates that the pixel luminance value is greater than the preset luminance value, the R-channel base ratio corresponding to the pixel luminance value can be determined based on this unified mapping relationship. Then, referring to the image color temperature, adjustments are made based on the determined R-channel base ratio, and the adjusted result is used as the first base ratio corresponding to the R-channel. In some embodiments, the R-channel base ratio corresponding to the pixel luminance value can also be directly determined based on this unified mapping relationship, and the determined R-channel base ratio can be used as the first base ratio corresponding to the R-channel.

[0108] In this implementation, by pre-constructing a mapping relationship, for the bright areas of the image, the first basic ratio corresponding to the R channel can be reasonably determined by referring to the pixel brightness value and the image color temperature. Regardless of whether the image color temperature is high or low, the rationality and reliability of the determined first basic ratio can be well guaranteed.

[0109] Combining the two implementation methods above, it can be seen that, regardless of whether it is a bright or dark area, the reference image color temperature and the comparison result between the pixel brightness value and the preset brightness value corresponding to the R channel can reasonably determine the first basic ratio corresponding to the R channel for image restoration, thereby helping to improve the image restoration effect.

[0110] It should be noted that the method for determining the first basic ratio corresponding to the R channel is not limited to the two implementation methods mentioned above. For example, the preset basic color temperature that is closest to the image color temperature among multiple preset basic color temperatures can be directly determined, the basic ratio of the R channel can be obtained from the calibration results corresponding to the determined preset basic color temperature, and the obtained basic ratio of the R channel can be used as the first basic ratio corresponding to the R channel.

[0111] The above details the method for determining the first basic ratio corresponding to the R channel. The methods for determining the second basic ratio corresponding to the G channel and the third basic ratio corresponding to the B channel can be found in the description of the method for determining the first basic ratio.

[0112] It should be noted that when determining the second base ratio, a mapping relationship between the luminance value and the G channel base ratio can be established. This mapping relationship can be represented as a second function, which can be found in [reference needed]. Figure 4-2 The slashed portion in the text represents the function. When determining the third base ratio, a mapping relationship can be established between the luminance value and the B channel base ratio. This mapping relationship can be represented as a third function, which can be found in [reference needed]. Figure 4-3 The slashed part in the diagram represents the function. The descriptions of the second and third functions are the same as those for the first function described above, and will not be repeated here.

[0113] Step 1403: Determine the first type of correction coefficient based on the relationship between the pixel brightness value and the first preset brightness value range.

[0114] Optionally, the first preset brightness value range can be represented as [thres1, thres2], that is, the minimum brightness value in the first preset brightness value range is thres1, and the maximum brightness value in the first preset brightness value range is thres2. Here, the user can adjust the values ​​of thres1 and thres2 according to actual needs.

[0115] In one optional example, step 1403 includes:

[0116] In response to the pixel brightness value being less than or equal to the minimum brightness value in the first preset brightness value range, determine that the first type of correction coefficient is the first preset maximum correction coefficient; or,

[0117] In response to the pixel brightness value being greater than the minimum brightness value in the first preset brightness value range and less than the maximum brightness value in the first preset brightness value range, determine that the first type of correction coefficient is negatively correlated with the pixel brightness value; or,

[0118] In response to the pixel brightness value being greater than or equal to the maximum brightness value in the first preset brightness value range, determine that the first type of correction coefficient is the first preset minimum correction coefficient.

[0119] Optionally, the first preset maximum correction coefficient can be 1.0, and the first preset minimum correction coefficient can be 0. Assuming that the pixel brightness value is represented as blending_rgb and the first type of correction coefficient is represented as global_ratio, then global_ratio can be determined according to the following first rule:

[0120] (1) blending_rgb <= thres1, global_ratio = 1.0.

[0121] (2) thres1 < blending_rgb < thres2, global_ratio decreases monotonically until it becomes 0.

[0122] (3) blending_rgb >= thres2, global_ratio = 0.

[0123] The first rule can be represented by a function composed of the flat line part and the slant line part in Figure 5 or, the first rule can be represented by a function in the following formula form:

[0124]

[0125] where y represents the pixel brightness value and f(y) represents the first type of correction coefficient.

[0126] In this way, by adopting the first rule, the first type of correction coefficient can be reasonably determined by referring to the pixel brightness value, so as to ensure the rationality and reliability of the determined first type of correction coefficient.

[0127] Step 1405: Use the first type of correction coefficient to correct the first base ratio, the second base ratio, and the third base ratio respectively, to obtain the first correction ratio corresponding to the first base ratio, the second correction ratio corresponding to the second base ratio, and the third correction ratio corresponding to the third base ratio.

[0128] In step 1405, the product of the first type of correction coefficient and the first base ratio can be used as the first correction ratio, the product of the first type of correction coefficient and the second base ratio can be used as the second correction ratio, and the product of the first type of correction coefficient and the third base ratio can be used as the third correction ratio. Thus, the first correction ratio can be expressed as global_ratio*r_rate, the second correction ratio can be expressed as global_ratio*g_rate, and the third correction ratio can be expressed as global_ratio*b_rate.

[0129] Of course, in practice, the correction of the first, second, and third base ratios can also be achieved through other methods. For example, the product of the square of the first type of correction coefficient and the first base ratio can be used as the first correction ratio, the product of the square of the first type of correction coefficient and the second base ratio can be used as the second correction ratio, and the product of the square of the first type of correction coefficient and the third base ratio can be used as the third correction ratio.

[0130] Step 1407: Based on the first correction ratio, the second correction ratio, the third correction ratio, and the first pixel value, IR component removal is performed on the pixel.

[0131] In an optional example, such as Figure 6 As shown, step 1407 includes steps 14071, 14073, 14075, 14077 and 14079.

[0132] Step 14071: Based on the second pixel value, the first pixel value, and the first correction ratio of the pixel in the R channel, determine the first reference pixel value of the pixel after removing the IR component in the R channel.

[0133] In one alternative implementation, step 14071 includes:

[0134] Determine the product of the first pixel value and the first correction ratio;

[0135] Determine the difference between the second pixel value and the product;

[0136] Based on the difference, determine the first reference pixel value of the pixel after removing the IR component in the R channel.

[0137] Optionally, the product of the first pixel value and the first correction ratio can be expressed as global_ratio*r_rate*IR, and the difference between the second pixel value and the product can be expressed as R - global_ratio*r_rate*IR. Here, R - global_ratio*r_rate*IR can be directly used as the first reference pixel value after removing the IR component from the pixel in the R channel; or, R - global_ratio*r_rate*IR can be pre-processed, and the processing result corresponding to the pre-processing can be used as the first reference pixel value after removing the IR component from the pixel in the R channel.

[0138] In this implementation, the first reference pixel value of a pixel after removing the IR component in the R channel can be calculated efficiently and reliably through simple operational logic such as multiplication and subtraction.

[0139] Of course, the implementation of step 14071 is not limited to this. For example, in the process of determining the first reference pixel value, in addition to multiplication and subtraction, division, exponentiation or other operation logic can also be used.

[0140] Step 14073: Based on the third pixel value, the first pixel value, and the second correction ratio of the pixel in the G channel, determine the second reference pixel value of the pixel after removing the IR component in the G channel.

[0141] Step 14075: Based on the fourth pixel value, the first pixel value, and the third correction ratio of the pixel in the B channel, determine the third reference pixel value of the pixel after removing the IR component in the B channel.

[0142] It should be noted that the specific implementation methods of steps 14073 and 14075 can be referred to the description of the implementation method of step 14071, and will not be repeated here.

[0143] Step 14077: Determine the second type of correction coefficient based on the relationship between the pixel brightness value and the second preset brightness value range.

[0144] Optionally, the second preset brightness value range can be represented as [thres3, thres4], that is, the minimum brightness value in the second preset brightness value range is thres3, and the maximum brightness value in the second preset brightness value range is thres4. Here, the user can adjust the values ​​of thres3 and thres4 according to actual needs.

[0145] In one alternative implementation, step 14077 includes:

[0146] In response to the pixel brightness value being less than or equal to the minimum brightness value in the second preset brightness value range, determining that the second type of correction coefficient is the second preset maximum correction coefficient; or,

[0147] In response to the pixel brightness value being greater than the minimum brightness value in the second preset brightness value range and less than the maximum brightness value in the second preset brightness value range, determining that the second type of correction coefficient is negatively correlated with the pixel brightness value; or,

[0148] In response to the pixel brightness value being greater than the maximum brightness value in the second preset brightness value range, determining that the second type of correction coefficient is the second preset minimum correction coefficient.

[0149] Optionally, the second preset maximum correction coefficient can be expressed as scale_rate_max, the second preset minimum correction coefficient can be expressed as scale_rate_min, and both scale_rate_max and scale_rate_min can be obtained through manual debugging.

[0150] Assuming that the pixel brightness value is expressed as blending_rgb and the second type of correction coefficient is expressed as scale_ratio, then scale_ratio can be determined according to the following second rule:

[0151] (1) blending_rgb <= thres3, scale_ratio = scale_rate_max.

[0152] (2) thres3 < blending_rgb < thres4, scale_ratio decreases monotonically until it reaches scale_rate_min (which can generally be 1.0).

[0153] (3) blending_rgb >= thres4, scale_ratio = scale_rate_min.

[0154] The second rule can be represented by a function composed of the flat line part and the斜线 part in Figure 7 or the second rule can be represented by a function in the following formula form:

[0155]

[0156] where z represents the pixel brightness value and f(z) represents the second type of correction coefficient.

[0157] In this way, by adopting the second rule, the pixel brightness value can be referenced to reasonably determine the second type of correction coefficient, so as to ensure the rationality and reliability of the determined second type of correction coefficient. By making the second type of correction coefficient corresponding to the low brightness area of ​​the image greater than the second type of correction coefficient corresponding to the high brightness area of ​​the image, the signal of the dark area of ​​the image can be amplified, the brightness step can be shortened, and overexposure in the high brightness area can be avoided, thereby further improving the image restoration effect.

[0158] Step 14079: Using the second type of correction coefficient, correct the first reference pixel value, the second reference pixel value, and the third reference pixel value respectively to obtain the first target pixel value corresponding to the first reference pixel value, the second target pixel value corresponding to the second reference pixel value, and the third target pixel value corresponding to the third reference pixel value.

[0159] In step 14079, the product of the second type of correction coefficient and the first reference pixel value can be used as the first target pixel value, the product of the second type of correction coefficient and the second reference pixel value can be used as the second target pixel value, and the product of the second type of correction coefficient and the third reference pixel value can be used as the third target pixel value. Assuming the first target pixel value, the second target pixel value, and the third target pixel value are represented as R', G', and B' respectively, then:

[0160] R'=scale_ratio*(R-global_ratio*r_rate*IR)

[0161] G'=scale_ratio*(G-global_ratio*g_rate*IR)

[0162] B'=scale_ratio*(B-global_ratio*b_rate*IR)

[0163] Of course, in practice, the correction of the first reference pixel value, the second reference pixel value, and the third reference pixel value can also be achieved in other ways. For example, the product of the square of the second type of correction coefficient and the first reference pixel value can be used as the first target pixel value, the product of the square of the second type of correction coefficient and the second reference pixel value can be used as the second target pixel value, and the product of the square of the second type of correction coefficient and the third reference pixel value can be used as the third target pixel value.

[0164] It should be noted that the first target pixel value, the second target pixel value, and the third target pixel value can form the IR component removal result of the pixel. By using the IR component removal results of multiple pixels in the image to be processed, the restored image can be generated.

[0165] In the embodiments of this disclosure, by referring to the color temperature of the image, the first basic ratio corresponding to the R channel, the second basic ratio corresponding to the G channel, and the third basic ratio corresponding to the B channel can be reasonably determined. By referring to the pixel brightness value and applying the first preset brightness value range, the first type of correction coefficient can be reasonably determined based on the brightness level of the pixel. Using the first type of correction coefficient, the correction of the first to the third basic ratios can be reliably achieved to obtain the corresponding three correction ratios. Using the obtained three correction ratios, the IR component of the pixel can be reasonably removed to ensure the IR component removal effect and correctly restore the image color information, thereby improving the image restoration effect.

[0166] In summary, the image restoration method adopted in the embodiments of this disclosure can effectively improve the image restoration effect, avoid overexposure in bright areas, and ensure the dynamic range of the image.

[0167] Any of the image restoration methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image restoration methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image restoration methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0168] Exemplary device

[0169] Figure 8 This is a schematic diagram of the structure of an image restoration apparatus provided in an exemplary embodiment of the present disclosure. Figure 8 The apparatus shown includes an acquisition module 810, a first determination module 820, a second determination module 830, a removal module 840, and a third determination module 850.

[0170] The acquisition module 810 is used to acquire the image to be processed, which includes an IR channel;

[0171] The first determining module 820 is used to determine the pixel brightness value of the pixel in the image to be processed acquired by the acquiring module 810, and the first pixel value of the pixel in the IR channel.

[0172] The second determining module 830 is used to determine the image color temperature of the image to be processed acquired by the acquiring module 810;

[0173] The removal module 840 is used to remove the IR component of the pixel based on the pixel brightness value and the first pixel value determined by the first determining module 820 and the image color temperature determined by the second determining module 830.

[0174] The third determining module 850 is used to determine the restored image based on the image to be processed after IR component removal by the pixel removal module 840.

[0175] In an optional example, the image to be processed also includes R channels, G channels, and B channels, such as... Figure 9 As shown, the removal module 840 includes:

[0176] The first determining submodule 8401 is used to determine the first basic ratio corresponding to the R channel, the second basic ratio corresponding to the G channel, and the third basic ratio corresponding to the B channel based on the image color temperature determined by the second determining module 830.

[0177] The second determining submodule 8403 is used to determine a first type of correction coefficient based on the relationship between the pixel brightness value determined by the first determining module 820 and the first preset brightness value range.

[0178] The correction submodule 8405 is used to correct the first base ratio, the second base ratio and the third base ratio determined by the first determination submodule 8401 using the first type of correction coefficient determined by the second determination submodule 8403, respectively, to obtain the first correction ratio corresponding to the first base ratio, the second correction ratio corresponding to the second base ratio and the third correction ratio corresponding to the third base ratio.

[0179] The removal submodule 8407 is used to remove the IR component from the pixel based on the first correction ratio, the second correction ratio, the third correction ratio determined by the correction submodule 8405, and the first pixel value determined by the first determination module 820.

[0180] In one optional example, the second determining submodule 8403 includes:

[0181] The first determining unit is configured to, in response to the first determining module 820 determining that the pixel brightness value is less than or equal to the minimum brightness value within a first preset brightness value range, determine the first type of correction coefficient as the first preset maximum correction coefficient; or...

[0182] The second determining unit is configured to, in response to the first determining module 820 determining that the pixel brightness value is greater than the minimum brightness value in the first preset brightness value range and less than the maximum brightness value in the first preset brightness value range, determine that the first type of correction coefficient is negatively correlated with the pixel brightness value determined by the first determining module 820; or,

[0183] The third determining unit is used to determine the first type of correction coefficient as the first preset minimum correction coefficient in response to the first determining module 820 determining that the pixel brightness value is greater than or equal to the maximum brightness value in the first preset brightness value range.

[0184] In an optional example, such as Figure 9 As shown, the first determining submodule 8401 includes:

[0185] The fourth determining unit 84011 is used to determine the preset brightness value corresponding to the R channel;

[0186] The comparison unit 84013 is used to compare the pixel brightness value determined by the first determining module 820 with the preset brightness value determined by the fourth determining unit 84011 to obtain a comparison result;

[0187] The fifth determining unit 84015 is used to determine the first basic ratio corresponding to the R channel based on the image color temperature determined by the second determining module 830 and the comparison result obtained by the comparison unit 84013.

[0188] In one optional example, the fifth determining unit 84015 includes:

[0189] The first determining subunit is used to respond to the comparison result obtained by the comparison unit 84013, which indicates that the pixel brightness value determined by the first determining module 820 is less than or equal to the preset brightness value, and the image color temperature determined by the second determining module 830 is the same as the first preset base color temperature. Based on the R channel base ratio obtained by pre-calibrating for the first preset base color temperature, the first base ratio corresponding to the R channel is determined.

[0190] The second determining subunit is used to respond to the comparison result obtained by the comparison unit 84013, which indicates that the pixel brightness value determined by the first determining module 820 is less than or equal to a preset brightness value. The image color temperature determined by the second determining module 830 is distributed between the second preset base color temperature and the third preset base color temperature, and the second preset base color temperature and the third preset base color temperature are two adjacent preset base color temperatures. The subunit interpolates between the corresponding R channel base ratios obtained by pre-calibrating the second preset base color temperature and the third preset base color temperature, and determines the first base ratio corresponding to the R channel based on the interpolation result.

[0191] In an optional example, the fifth determining unit 84015 is specifically used for:

[0192] In response to the comparison result obtained by the comparison unit 84013 indicating that the pixel brightness value determined by the first determining module 820 is greater than the preset brightness value, the first basic ratio corresponding to the R channel is determined based on the pre-constructed mapping relationship between the brightness value and the basic ratio of the R channel, the pixel brightness value determined by the first determining module 820, and the image color temperature determined by the second determining module 830.

[0193] In an optional example, such as Figure 9 As shown, removing submodule 8407 includes:

[0194] The sixth determining unit 84071 is used to determine the first reference pixel value of the pixel after removing the IR component in the R channel based on the second pixel value of the pixel in the R channel, the first pixel value determined by the first determining module 820, and the first correction ratio determined by the correction submodule 8405.

[0195] The seventh determining unit 84073 is used to determine the second reference pixel value of the pixel after removing the IR component in the G channel based on the third pixel value of the pixel in the G channel, the first pixel value determined by the first determining module 820, and the second correction ratio determined by the correction submodule 8405.

[0196] The eighth determining unit 84075 is used to determine the third reference pixel value of the pixel after removing the IR component in the B channel based on the fourth pixel value of the pixel in the B channel, the first pixel value determined by the first determining module 820, and the third correction ratio determined by the correction submodule 8405.

[0197] The ninth determining unit 84077 is used to determine a second type of correction coefficient based on the relationship between the pixel brightness value determined by the first determining module 820 and the second preset brightness value range.

[0198] The correction unit 84079 is used to correct the first reference pixel value determined by the sixth determination unit 84071, the second reference pixel value determined by the seventh determination unit 84073, and the third reference pixel value determined by the eighth determination unit 84075 using the second type of correction coefficient determined by the ninth determination unit 84077, respectively, to obtain the first target pixel value corresponding to the first reference pixel value, the second target pixel value corresponding to the second reference pixel value, and the third target pixel value corresponding to the third reference pixel value.

[0199] In one optional example, the ninth determining unit 84077 includes:

[0200] The third determining subunit is configured to, in response to the first determining module 820 determining that the pixel brightness value is less than or equal to the minimum brightness value within a second preset brightness value range, determine the second type of correction coefficient as the second preset maximum correction coefficient; or...

[0201] The fourth determining subunit is configured to, in response to the first determining module 820 determining that the pixel brightness value is greater than the minimum brightness value in the second preset brightness value range and less than the maximum brightness value in the second preset brightness value range, determine that the second type of correction coefficient is negatively correlated with the pixel brightness value determined by the first determining module 820; or,

[0202] The fifth determining subunit is used to determine the second type of correction coefficient as the second preset minimum correction coefficient in response to the first determining module 820 determining that the pixel brightness value is greater than the maximum brightness value in the second preset brightness value range.

[0203] In an optional example, the sixth determining unit 84071 includes:

[0204] The sixth determining subunit is used to determine the product of the first pixel value determined by the first determining module 820 and the first correction ratio determined by the correction subunit 8405;

[0205] The seventh determining subunit is used to determine the difference between the second pixel value and the product determined by the sixth determining subunit;

[0206] The eighth determining subunit is used to determine the first reference pixel value of the pixel after removing the IR component in the R channel based on the difference determined by the seventh determining subunit.

[0207] In an optional example, the image to be processed also includes R channels, G channels, and B channels, such as... Figure 10 As shown, the first determining module 820 includes:

[0208] The third determining submodule 8201 is used to determine the minimum and maximum pixel values ​​of the pixels in the image to be processed obtained from the acquisition module 810, among the second pixel value in the R channel, the third pixel value in the G channel, and the fourth pixel value in the B channel.

[0209] The fourth determining submodule 8203 is used to determine the first weight corresponding to the minimum pixel value determined by the third determining submodule 8201 and the second weight corresponding to the maximum pixel value determined by the third determining submodule 8201.

[0210] The weighting submodule 8205 is used to weight the minimum pixel value and the maximum pixel value using the first weight and the second weight determined by the third determination submodule 8201 to obtain a weighted value.

[0211] The fifth determining submodule 8207 is used to determine the pixel brightness value of a pixel based on the weighted value obtained from the weighting submodule 8205.

[0212] In the apparatus disclosed herein, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve the corresponding functions and effects, and this disclosure does not list them all.

[0213] Exemplary electronic devices

[0214] Below, for reference Figure 11 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0215] Figure 11 A block diagram of an electronic device 1100 according to an embodiment of the present disclosure is shown.

[0216] like Figure 11 As shown, the electronic device 1100 includes one or more processors 1110 and memory 1120.

[0217] The processor 1110 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1100 to perform desired functions.

[0218] The memory 1120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1110 may execute the program instructions to implement the image restoration methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0219] In one example, the electronic device 1100 may also include an input device 1130 and an output device 1140, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0220] For example, when electronic device 1100 is a first device or a second device, the input device 1130 may be a microphone or a microphone array. When electronic device 1100 is a standalone device, the input device 1130 may be a communication network connector for receiving acquired input signals from the first device and the second device.

[0221] In addition, the input device 1130 may also include, for example, a keyboard, a mouse, etc.

[0222] The output device 1140 can output various information to the outside. The output device 1140 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0223] Of course, for the sake of simplicity, Figure 11Only some of the components of the electronic device 1100 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1100 may include any other suitable components depending on the specific application.

[0224] Exemplary computer program products and computer-readable storage media

[0225] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image restoration methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0226] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0227] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the image restoration methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0228] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0229] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. The specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0230] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0231] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0232] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0233] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0234] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0235] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An image restoration method, comprising: Acquire an image to be processed, the image to be processed including an IR channel, an R channel, a G channel and a B channel; Determine the pixel brightness value of a pixel in the image to be processed, and the first pixel value of the pixel in the IR channel; Determine the color temperature of the image to be processed; Based on the pixel brightness value, the first pixel value, and the image color temperature, the IR component of the pixel is removed; Based on the image to be processed after removing the IR components from the pixels, the restored image is determined; The step of removing the IR component from the pixel based on the pixel brightness value, the first pixel value, and the image color temperature includes: Based on the image color temperature, determine the first basic ratio corresponding to the R channel, the second basic ratio corresponding to the G channel, and the third basic ratio corresponding to the B channel; Based on the relationship between the pixel brightness value and the first preset brightness value range, a first type of correction coefficient is determined; Using the first type of correction coefficient, the first base ratio, the second base ratio, and the third base ratio are corrected respectively to obtain the first correction ratio corresponding to the first base ratio, the second correction ratio corresponding to the second base ratio, and the third correction ratio corresponding to the third base ratio. Based on the first correction ratio, the second correction ratio, the third correction ratio, and the first pixel value, the IR component is removed from the pixel.

2. The method according to claim 1, wherein, The step of determining the first type of correction coefficient based on the relationship between the pixel brightness value and the first preset brightness value range includes: In response to the pixel brightness value being less than or equal to the minimum brightness value within the first preset brightness value range, the first type of correction coefficient is determined to be the first preset maximum correction coefficient; or... In response to the pixel brightness value being greater than the minimum brightness value in the first preset brightness value range and less than the maximum brightness value in the first preset brightness value range, it is determined that the first type of correction coefficient is negatively correlated with the pixel brightness value; or... In response to the pixel brightness value being greater than or equal to the maximum brightness value in the first preset brightness value range, the first type of correction coefficient is determined to be the first preset minimum correction coefficient.

3. The method according to claim 1, wherein, The step of determining the first base ratio corresponding to the R channel based on the image color temperature includes: Determine the preset brightness value corresponding to the R channel; The pixel brightness value is compared with the preset brightness value to obtain a comparison result; Based on the image color temperature and the comparison results, the first base ratio corresponding to the R channel is determined.

4. The method according to claim 3, wherein, The step of determining the first base ratio corresponding to the R channel based on the image color temperature and the comparison result includes: In response to the comparison result indicating that the pixel brightness value is less than or equal to the preset brightness value and the image color temperature is the same as the first preset base color temperature, the first base ratio corresponding to the R channel is determined based on the R channel base ratio obtained in advance for the first preset base color temperature. In response to the comparison result indicating that the pixel brightness value is less than or equal to the preset brightness value, the image color temperature is distributed between the second preset base color temperature and the third preset base color temperature, and the second preset base color temperature and the third preset base color temperature are two adjacent preset base color temperatures. Interpolation is performed between the corresponding R channel base ratios obtained by pre-calibrating the second preset base color temperature and the third preset base color temperature, and the first base ratio corresponding to the R channel is determined based on the interpolation result.

5. The method according to claim 3, wherein, The step of determining the first base ratio corresponding to the R channel based on the image color temperature and the comparison result includes: In response to the comparison result indicating that the pixel brightness value is greater than the preset brightness value, the first basic ratio corresponding to the R channel is determined based on the pre-constructed mapping relationship between the brightness value and the basic ratio of the R channel, the pixel brightness value, and the image color temperature.

6. The method according to claim 1, wherein, The step of removing IR components from the pixel based on the first correction ratio, the second correction ratio, the third correction ratio, and the first pixel value includes: Based on the second pixel value of the pixel in the R channel, the first pixel value, and the first correction ratio, determine the first reference pixel value of the pixel after removing the IR component in the R channel; Based on the third pixel value of the pixel in the G channel, the first pixel value, and the second correction ratio, a second reference pixel value of the pixel after removing the IR component in the G channel is determined; Based on the fourth pixel value of the pixel in the B channel, the first pixel value, and the third correction ratio, the third reference pixel value of the pixel after removing the IR component in the B channel is determined; Based on the relationship between the pixel brightness value and the second preset brightness value range, a second type of correction coefficient is determined; Using the second type of correction coefficient, the first reference pixel value, the second reference pixel value, and the third reference pixel value are corrected respectively to obtain the first target pixel value corresponding to the first reference pixel value, the second target pixel value corresponding to the second reference pixel value, and the third target pixel value corresponding to the third reference pixel value.

7. The method according to claim 6, wherein, The step of determining the second type of correction coefficient based on the relationship between the pixel brightness value and the second preset brightness value range includes: In response to the pixel brightness value being less than or equal to the minimum brightness value within the second preset brightness value range, the second type of correction coefficient is determined as the second preset maximum correction coefficient; or... In response to the pixel brightness value being greater than the minimum brightness value in the second preset brightness value range and less than the maximum brightness value in the second preset brightness value range, it is determined that the second type of correction coefficient is negatively correlated with the pixel brightness value; or, In response to the pixel brightness value being greater than the maximum brightness value in the second preset brightness value range, the second type of correction coefficient is determined to be the second preset minimum correction coefficient.

8. The method according to claim 6, wherein, The step of determining the first reference pixel value of the pixel after removing the IR component in the R channel based on the second pixel value of the pixel in the R channel, the first pixel value, and the first correction ratio includes: Determine the product of the first pixel value and the first correction ratio; Determine the difference between the second pixel value and the product; Based on the difference, the first reference pixel value of the pixel after removing the IR component from the R channel is determined.

9. The method according to claim 1, wherein, Determining the pixel brightness value of a pixel in the image to be processed includes: The minimum and maximum pixel values ​​are determined from the second pixel value of the R channel, the third pixel value of the G channel, and the fourth pixel value of the B channel in the image to be processed. Determine the first weight corresponding to the minimum pixel value and the second weight corresponding to the maximum pixel value; Using the first weight and the second weight, the minimum pixel value and the maximum pixel value are weighted to obtain a weighted value; The pixel brightness value of the pixel is determined based on the weighted value.

10. An image restoration apparatus, comprising: An acquisition module is used to acquire an image to be processed, the image to be processed including an IR channel, an R channel, a G channel and a B channel; The first determining module is used to determine the pixel brightness value of the pixel in the image to be processed acquired by the acquiring module, and the first pixel value of the pixel in the IR channel; The second determining module is used to determine the image color temperature of the image to be processed acquired by the acquiring module; The removal module is used to remove the IR component from the pixel based on the pixel brightness value and the first pixel value determined by the first determining module and the image color temperature determined by the second determining module. The third determining module is used to determine the restored image based on the image to be processed after the IR component is removed from the pixels by the removing module. Specifically, the removal module is used for: Based on the image color temperature, determine the first basic ratio corresponding to the R channel, the second basic ratio corresponding to the G channel, and the third basic ratio corresponding to the B channel; Based on the relationship between the pixel brightness value and the first preset brightness value range, a first type of correction coefficient is determined; Using the first type of correction coefficient, the first base ratio, the second base ratio, and the third base ratio are corrected respectively to obtain the first correction ratio corresponding to the first base ratio, the second correction ratio corresponding to the second base ratio, and the third correction ratio corresponding to the third base ratio. Based on the first correction ratio, the second correction ratio, the third correction ratio, and the first pixel value, the IR component is removed from the pixel.

11. A computer-readable storage medium storing a computer program for performing the image restoration method according to any one of claims 1-9.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image restoration method according to any one of claims 1-9.

Images