Image data interpolation method, device, storage medium and electronic device

Abstract

The present application discloses an interpolation method, device, storage medium and electronic device for image data, which determine image data respectively collected by a three-color unit and an infrared unit in image data, determine the three-color unit and other infrared units adjacent to the infrared unit as adjacent units according to the positional relationship of each unit in the image sensor, determine the diagonal component change of green light at the infrared unit according to the image data collected by the infrared unit and the adjacent unit, and finally determine the image data of the adjacent units based on which to restore the green image data at the infrared unit according to the diagonal component change. Since the color change of the adjacent units of the infrared unit is taken into account when performing the difference, it is possible to specifically determine which image data to use to determine the interpolation result, so as to restore more accurate image data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image data interpolation method, device, storage medium and electronic device. Background Art

[0002] At present, image sensors are used in more and more scenarios. The traditional Bayer image sensor is composed of a red, blue and green (RGB) three-color unit array. Each color unit is used to sense visible light of different bands to obtain image data of different colors, and a color image is generated based on the three-color image data.

[0003] In the prior art, in order to expand the application scenarios of image sensors, an infrared (IR) unit that senses infrared band light is added to the three-color unit array to form a color-infrared (RGB-IR) sensor, which can sense visible light and infrared light at the same time, further expanding the application scenarios of image sensors.

[0004] However, since the current RGB-IR sensor usually replaces the green light unit (i.e., G unit) in the sensor array with an IR unit, the image signal processing unit (Image Signal Processor, ISP) for subsequent image processing needs to be redeveloped, which increases the cost of using the RGB-IR sensor. Based on this, the present application provides an image data interpolation method. Summary of the invention

[0005] The present application provides an image data interpolation method, device, storage medium and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0006] This application adopts the following technical solutions:

[0007] The present application provides an image data interpolation method, comprising:

[0008] Acquire image data collected by the color-infrared image sensor, and determine the image data collected by each three-color unit and each infrared unit in the image data;

[0009] According to the positional relationship of each unit in the image sensor, for at least part of the infrared units, determining other adjacent infrared units and adjacent three-color units on the diagonal line of the infrared unit as adjacent units;

[0010] Determine, according to the image data collected by the infrared unit and the adjacent units, the component interpolation of the green light at the infrared unit when collecting the image data, and the magnitude relationship of the gradient value of the image data;

[0011] Determining green image data interpolated at the infrared unit according to the size relationship and the component interpolation;

[0012] Image data for generating a color image is determined based on the image data obtained by interpolation and the image data respectively collected by the three-color units in the image data.

[0013] Optionally, in the color-infrared image sensor, the infrared unit is adjacent to the red unit and the blue unit, and is adjacent to the green unit in a diagonal direction.

[0014] Optionally, determining other infrared units adjacent to the infrared unit on a diagonal line as adjacent units includes:

[0015] Determine that other infrared units on the main diagonal line of the infrared unit, which are separated by one green unit, are adjacent units;

[0016] It is determined that other infrared units on the secondary diagonal line of the infrared unit, which are separated by one green unit, are adjacent units.

[0017] Optionally, determining, according to the image data collected by the infrared unit and the adjacent unit, component interpolation of green light at the infrared unit when collecting the image data comprises:

[0018] Determine the component interpolation of the green light on the main diagonal according to the difference of the image data of the green light unit on the main diagonal of the infrared unit, the difference of the image data of the infrared unit and other infrared units on the main diagonal, and the image data of the infrared unit;

[0019] The component interpolation of green light on the sub-diagonal line is determined according to the difference of image data of the green light unit on the sub-diagonal line of the infrared unit, the difference of image data between the infrared unit and other infrared units on the sub-diagonal line, and the image data of the infrared unit.

[0020] Optionally, determining, according to the image data collected by the infrared unit and the adjacent unit, a magnitude relationship of the gradient value of the image data when collecting the image data includes:

[0021] Determining a main first-order gradient according to the difference of the green light units on the main diagonal line of the infrared unit;

[0022] Determine a main second-order gradient according to differences in image data between the infrared unit and other adjacent infrared units on the main diagonal line;

[0023] Determining a secondary first-order gradient according to the difference of the green light units on the secondary diagonal line of the infrared unit;

[0024] Determine a secondary second-order gradient according to differences in image data between the infrared unit and other adjacent infrared units on the secondary diagonal line;

[0025] Determine a main diagonal gradient according to the main first-order gradient and the main second-order gradient, and determine a secondary diagonal gradient according to the secondary first-order gradient and the secondary second-order gradient;

[0026] The magnitude relationship between the main diagonal gradient and the secondary diagonal gradient is determined.

[0027] Optionally, determining green image data interpolated at the infrared unit according to the size relationship and the component interpolation includes:

[0028] When the main diagonal gradient is greater than the secondary diagonal gradient, interpolating the green light components of the secondary diagonal to determine the green image data interpolated at the infrared unit;

[0029] When the main diagonal gradient is less than the sub-diagonal gradient, interpolating the green light components of the main diagonal to determine the green image data interpolated at the infrared unit;

[0030] When the main diagonal gradient is equal to the sub-diagonal gradient, green image data interpolated at the infrared unit is determined according to an average of the interpolated components of the green light of the main diagonal and the interpolated components of the green light of the sub-diagonal.

[0031] Optionally, determining the magnitude relationship between the main diagonal gradient and the secondary diagonal gradient includes:

[0032] According to a preset redundant parameter, determining whether a difference between the main diagonal gradient and the secondary diagonal gradient is less than the redundant parameter;

[0033] If so, determining that the main diagonal gradient is equal to the secondary diagonal gradient;

[0034] If not, then according to the main diagonal gradient being equal to the secondary diagonal gradient, the magnitude relationship between the main diagonal gradient and the secondary diagonal gradient is determined.

[0035] Optionally, determining image data for generating a color image according to the image data obtained by interpolation and the image data respectively collected by the three-color units in the image data includes:

[0036] According to the position of the infrared unit in the image sensor, the image data of the infrared unit in the image data is replaced with the interpolated green image data to determine the image data for generating the color image;

[0037] And according to the image data of each infrared unit, image data for generating the infrared image is determined.

[0038] The present application provides an interpolation device for image data.

[0039] The device is applied to a color-infrared image sensor, comprising:

[0040] An acquisition module, which acquires image data collected by the color-infrared image sensor, and determines the image data collected by each three-color unit and each infrared unit in the image data;

[0041] A positioning module, for at least some infrared units, determining other infrared units and adjacent three-color units on a diagonal line of the infrared unit as adjacent units according to the positional relationship of the units in the image sensor;

[0042] a gradient determination module, which determines, based on the image data collected by the infrared unit and the adjacent units, the component interpolation of the green light at the infrared unit when collecting the image data, and the magnitude relationship of the gradient value of the image data;

[0043] an interpolation determination module, which determines green image data interpolated at the infrared unit according to the size relationship and the component interpolation;

[0044] The data combination module determines the image data used to generate the color image according to the image data obtained by interpolation and the image data respectively collected by the three-color units in the image data.

[0045] The present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method for interpolating the above-mentioned image data is implemented.

[0046] The present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method of interpolating image data when executing the program.

[0047] At least one of the above technical solutions adopted in this application can achieve the following beneficial effects:

[0048] First, the image data collected by the three-color unit and the infrared unit respectively in the image data are determined, and then according to the positional relationship of each unit in the image sensor, for at least part of the infrared units, other infrared units and adjacent three-color units adjacent to the diagonal line of the infrared unit are determined as adjacent units of the infrared unit, and according to the image data collected by the infrared unit and the adjacent units, the component interpolation of green light in two diagonal directions at the infrared unit and the size relationship of the gradient values ​​of the image data on the two diagonals are determined, and it is identified whether there is a significant change in the color gradient in the diagonal direction at the infrared unit, and the image data of which adjacent units are used to restore the green image data at the infrared unit based on the identification result, so that the green difference can be performed more accurately to obtain the image data for generating a color image.

[0049] Since the color changes of the adjacent units of the infrared unit are taken into consideration when performing the difference, it is possible to specifically determine which image data to use to determine the interpolation result, thereby restoring more accurate image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0051] Figure 1 A schematic diagram of a flow chart of a method for interpolating image data provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of image data provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of the arrangement of the units in the RGB-IR sensor provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of adjacent units of an infrared unit provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of the gradient change of pixel values ​​in image data provided by an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a device structure for interpolating image data provided in an embodiment of the present application;

[0057] Figure 7 A method corresponding to the embodiment of the present application is provided Figure 1 Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0059] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0060] Figure 1 A schematic flow chart of a method for interpolating image data provided in this application includes the following steps:

[0061] S100: Acquire image data collected by a color-infrared image sensor, and determine the image data collected by each three-color unit and each infrared unit in the image data.

[0062] In order to solve the problem that the ISP chip that processes the image data collected by the RGB sensor cannot be directly reused to process the image data collected by the color-infrared image sensor (hereinafter referred to as the RGB-IR sensor), in one or more embodiments of the present application, a method for interpolating image data is provided, so that RGB image data can be restored based on the RGB-IR image data collected by the RGB-IR sensor.

[0063] Since the RGB-IR sensor usually replaces some green units in the three-color unit array of the sensor with infrared units, the image data obtained by the image sensor cannot be directly reused for processing by the existing ISP chip.

[0064] Figure 2 The image data diagram provided for this application shows that the left side is the image data collected by the traditional RGB sensor, and the right side is the image data collected by the RGB-IR sensor. It can be seen that since the G unit is replaced by the IR unit, the collected data lacks some G image data, and after removing the IR image data, the image data will be missing, and the ISP chip cannot be used directly for processing. The missing image data needs to be interpolated.

[0065] Since the purpose is to solve the problem of how to interpolate the image data transmitted to the ISP chip, in one or more embodiments of the present application, the interpolation method can be executed by the front-end device that collects the image. For example, it is executed by an RGB-IR sensor, or by other processors, as long as it is performed before sending to the ISP chip, and this application does not make specific restrictions. For the convenience of description, the following explanation is taken as an example that the execution subject is the front-end device.

[0066] Specifically, after receiving the image data collected by the RGB-IR sensor, the front-end device can determine the corresponding relationship between the image data and each unit of the RGB-IR sensor. Figure 2 The corresponding relationship between different three-color image data and three-color units, and the corresponding relationship between infrared image data and infrared units are determined as shown. The three-color units include: red units, green units and blue units. The image data is specifically a pixel value.

[0067] In addition, it should be noted that the present application does not restrict the arrangement of the units in the RGB-IR sensor. Of course, since the photoelectric sensor is less sensitive to green light, more green units are usually set to ensure the color accuracy of the output image. Therefore, the RGB-IR sensor usually replaces half of the green units with infrared units. This application also uses this as an example for explanation. In addition, since there are many ways to arrange the three-color units, for the convenience of description, the following will be used as Figure 3 The arrangement shown is used as an example for explanation.

[0068] Figure 3 This is a schematic diagram of the arrangement of the units in the RGB-IR sensor provided in the present application, wherein G represents a green unit, R represents a red unit, B represents a blue unit, and IR represents an infrared unit. Figure 3 Only a partially enlarged area of ​​the sensor is shown.

[0069] S102: According to the positional relationship of each unit in the image sensor, for at least part of the infrared units, determine other infrared units and adjacent three-color units adjacent to the infrared unit on the diagonal line as adjacent units.

[0070] In one or more embodiments of the present application, since it is necessary to determine the interpolated green image data corresponding to the infrared unit, and the specific interpolation value should be determined based on the image data collected by the units around the infrared unit, the front-end device can determine other infrared units adjacent to the infrared unit, as well as adjacent three-color units, as adjacent units of the infrared unit for at least part of the infrared units.

[0071] Specifically, Figure 3 For example, in the RGB-IR sensor, the infrared units adjacent in the horizontal direction are red units, the infrared units adjacent in the vertical direction are blue units, and the infrared units adjacent in the diagonal direction are green units. Then, for any infrared unit, the front-end device can determine that the other infrared units on the main diagonal of the infrared unit that are separated by one green unit are adjacent units, and determine that the other infrared units on the secondary diagonal of the infrared unit that are separated by one green unit are adjacent units. Figure 4 shown.

[0072] Figure 4 A schematic diagram of adjacent units of the infrared unit provided in this application. Figure 4 Take the infrared unit filled with thin diagonal lines as an example, the adjacent green units on the diagonal of the infrared unit are the three-color units adjacent to the infrared unit. The infrared units filled with thick diagonal lines are other infrared units adjacent to the infrared unit. Among them, the straight line passing through the upper left corner IR1 and the lower right corner IR2 is the main diagonal of the infrared unit IRc, and the straight line passing through the upper right corner IR1 and the lower left corner IR2 is the secondary diagonal of the infrared unit IRc.

[0073] S104: Determine, based on the image data collected by the infrared unit and the adjacent units, component interpolation of green light at the infrared unit when collecting the image data, and a magnitude relationship of a gradient value of the image data.

[0074] In one or more embodiments of the present application, after determining the adjacent units of the infrared unit, the front-end device can determine the amount of change in the green light at the infrared unit in different directions and the amount of change in the infrared light in the same direction when the RGB-IR sensor collects image data. Compared with directly using the average value of the green image data of the green units near the infrared unit as the interpolated green image data of the infrared unit, the present application can identify whether there is a "gradient" change in the green light at the infrared unit through subsequent steps by determining the horizontal component change and the vertical component change, that is, it can be determined whether there is a high-frequency color change at the infrared unit, thereby determining a more accurate green interpolation. This avoids inaccurate green image data obtained by averaging and improves the accuracy of interpolation.

[0075] For example, there are usually large color changes at the edges of objects in the image, and pixel value gradient changes occur, such as Figure 5 shown. Figure 5 This is a schematic diagram of the color of the edge of an object provided in this application, in which the background color outside the visible object is obviously darker than the color of the object itself. If the infrared unit is exactly at the edge of the object, there will be a large color gradient between the other units around the infrared unit, but this color gradient has a direction. Figure 5 As shown in , the color gradients of the left and right sides of the infrared unit at point A are larger, while the color gradients in the up and down directions are smaller, while the color gradients of the upper right corner and other areas of the infrared unit at point B are larger. If the green image data collected by the surrounding green units for the infrared units at points A and B are averaged to determine the interpolation result, the color will be inaccurate.

[0076] Especially when there are many objects in the image, or the color of the objects changes at a high frequency in a small range, the simple mean interpolation method will not only cause color errors, but may also cause moiré patterns in the image, affecting the authenticity of the image.

[0077] The present application determines the gradient value of the image data of green light at the infrared unit, so that the subsequent steps can determine whether the infrared unit is at a position where the color changes, and determine the interpolation of the green image data based on the component interpolation of the green light at the determined infrared unit.

[0078] Specifically, Figure 3 Taking the layout of each unit in the RGB-IR sensor as an example, the change of the green image data at the infrared unit can be determined based on the change of the image data of the green unit adjacent to the infrared unit. Of course, since the green unit is on the diagonal of the infrared unit, the change of the determined green image data is also divided into two changes in the diagonal direction. That is, the change on the main diagonal and the change on the sub-diagonal. The front-end device can respectively determine the difference of the green units on the main diagonal as the gradient value of the image data of the infrared unit in the main diagonal direction, and determine the difference of the green units on the sub-diagonal as the gradient value of the image data of the infrared unit in the sub-diagonal direction. The following is referred to as the main diagonal gradient and the sub-diagonal gradient, respectively, and denoted as Gradp and Gradn. Of course, the main diagonal and sub-diagonal are borrowed from the concept of matrix diagonals in linear functions for easy understanding.

[0079] In addition, determining the gradient value can only determine whether there is a "direction" of color change, and it is also necessary to determine the specific green image data that needs to be interpolated. Since the gradient is determined according to the diagonal direction, the interpolation can also be determined according to the diagonal direction when determining the interpolation. Of course, this specification does not limit how to determine the interpolation. The front-end device can also determine the green interpolation corresponding to different directions based on the horizontal direction and the vertical direction, according to the red unit and the blue unit adjacent to the infrared unit.

[0080] Taking the method of determining interpolation according to the diagonal as an example, the front-end device can determine the component interpolation of the green light on the main diagonal according to the difference of the image data of the green light unit on the main diagonal of the infrared unit, the difference of the image data of the infrared unit and other infrared units on the main diagonal, and the image data of the infrared unit. And, determine the component interpolation of the green light on the sub-diagonal according to the difference of the image data of the green light unit on the sub-diagonal of the infrared unit, the difference of the image data of the infrared unit and other infrared units on the sub-diagonal, and the image data of the infrared unit.

[0081] by Figure 4Taking the layout shown in the figure as an example, if the component interpolation of the green light of the main diagonal line is expressed by a formula, it is:

[0082] IntpGp=IRc+(G1+G2) / 2-(IR1+2*IRc+IR2) / 4

[0083] Among them, IntpGp is ​​the component interpolation of the green light on the main diagonal, IRc is the image data of the infrared unit, G1 and G2 are respectively the image data of the adjacent green light units on the main diagonal of the infrared unit, and IR1 and IR2 are respectively the image data of other infrared units separated by one green unit on the secondary diagonal of the infrared unit. In other words, the component interpolation of the green light on the main diagonal is determined by the difference of the main diagonal of the infrared light at the infrared unit and the average of the green light on the main diagonal.

[0084] The interpolation of the green light component of the above sub-diagonal is:

[0085] IntpGn=IRc+(G3+G4) / 2-(IR3+2*IRc+IR4) / 4

[0086] Among them, IntpGn is the component interpolation of the green light of the secondary diagonal, G3 and G4 are the image data of the adjacent green light units on the secondary diagonal of the infrared unit, and IR3 and IR4 are the image data of other infrared units separated by one green unit on the main diagonal of the infrared unit. In other words, the component interpolation of the green light of the secondary diagonal is determined by the difference of the infrared light on the secondary diagonal at the infrared unit and the average value of the green light on the secondary diagonal.

[0087] In addition, Figure 4 Taking the layout shown in the figure as an example, the above determination of the main diagonal gradient can be based on the determination of the green cells on the main diagonal, and the following formula can be used: Gradp=abs(diff(G1, G2)).

[0088] The secondary diagonal gradient is determined using the following formula: Gradn=abs(diff(G3, G4)).

[0089] Alternatively, in one or more embodiments of the present application, the front-end device may also determine the main diagonal gradient based on the difference of the infrared units on the main diagonal, and determine the secondary diagonal gradient based on the difference of the infrared units on the secondary diagonal. The corresponding formulas are Gradp = abs(diff(2*IRc, IR1+IR2)), Gradn = abs(diff(2*IRc, IR3+IR4)).

[0090] S106: Determine green image data interpolated at the infrared unit according to the size relationship and the component interpolation.

[0091] In one or more embodiments of the present application, after determining the gradient value on the diagonal of the green light at the infrared unit, the front-end device can determine the magnitude relationship between the main diagonal gradient and the main diagonal gradient. To determine whether the infrared unit is at the edge of an object in the image or a position where the color changes greatly. If so, based on the magnitude relationship, determine whether to interpolate the green light components based on the main diagonal to determine the green interpolation, or to interpolate the green light components based on the sub-diagonal to determine the green interpolation. If not, determine the green interpolation based on the green image data collected by the green units in the adjacent units on average.

[0092] Specifically, when the main diagonal gradient is greater than the sub-diagonal gradient, the front-end device determines the green image data interpolated at the infrared unit according to the component interpolation of the green light of the sub-diagonal. When the main diagonal gradient is less than the sub-diagonal gradient, the green image data interpolated at the infrared unit is determined according to the component interpolation of the green light of the main diagonal. When the main diagonal gradient is equal to the sub-diagonal gradient, the green image data interpolated at the infrared unit is determined according to the average of the component interpolation of the green light of the main diagonal and the component interpolation of the green light of the sub-diagonal.

[0093] The formula is:

[0094] if Gradp>Gradn, then IntpG=IntpGn

[0095] if Gradp <Gradn,then IntpG=IntpGp

[0096] if Gradp=Gradn, then IntpG=(IntpGn+IntpGp) / 2

[0097] S108: Determine image data for generating a color image according to the image data obtained by interpolation and the image data respectively collected by the three-color units in the image data.

[0098] In one or more embodiments of the present application, the front-end device may use the above steps S104 to S106 to determine the interpolation of green image data of at least part of the infrared units. Then, when transmitting the image data to the ISP chip at the back end of the RGB-IR sensor, the infrared image data in the image data may be replaced with the interpolation of green image data according to the position of each infrared unit and the interpolation of green image data corresponding to each infrared unit, so as to determine the image data used to generate a color image. At this time, the structure of the image data is consistent with that of the image data collected before the RGB sensor replaces the green unit with the infrared unit. Even if the ISP chip is not adjusted, the ISP chip can process the image data normally. The cost of redeveloping the ISP chip is avoided, and the mature ISP chip can be reused.

[0099] At the same time, since the gradient direction of the green color change is taken into account when determining the interpolation, the interpolated green image is closer to the actual image data collected, making the obtained image color and texture details more accurate, and can greatly reduce defects such as moiré in the image.

[0100] based on Figure 1 The interpolation method of image data shown in the figure first determines the image data collected by the three-color unit and the infrared unit respectively in the image data, and then, according to the positional relationship of each unit in the image sensor, for at least part of the infrared units, determines other infrared units and adjacent three-color units adjacent to the infrared unit on the diagonal line as the adjacent units of the infrared unit, and according to the image data collected by the infrared unit and the adjacent units, determines the component interpolation of green light in two diagonal directions at the infrared unit, as well as the magnitude relationship of the gradient values ​​of the image data on the two diagonals, identifies whether there is a significant change in the color gradient in the diagonal direction at the infrared unit, and determines based on the identification result which adjacent units' image data are used to restore the green image data at the infrared unit, so that the green difference can be performed more accurately to obtain the image data for generating a color image.

[0101] Since the color changes of the adjacent units of the infrared unit are taken into consideration when performing the difference, it is possible to specifically determine which image data to use to determine the interpolation result, thereby restoring more accurate image data.

[0102] In one or more embodiments of the present application, as described in step S104, the front-end device may determine the interpolation of green light based only on the color change of adjacent units of one color. However, after all, the prediction is based on green light, not the actual measurement value. In addition, the color of the object itself may have different effects on other units, and the interpolation determined based on a single color unit (including infrared) may be accurate. Therefore, the front-end device can also determine the component changes on the diagonal based on each color, and comprehensively determine the horizontal component changes and the vertical component changes.

[0103] For example, the main diagonal interpolation of green light at the infrared unit is determined according to the mean or weighted mean of the main diagonal component change determined by the red unit in the adjacent units, the main diagonal component change determined by the blue unit in the adjacent units, and the main diagonal component change determined by other infrared units in the adjacent units. Among them, since the main diagonal component change determined by the infrared unit belongs to the result determined by the "in-situ" sampling data, its weight can be increased. The specific weight can be set as needed, for example, the weights of red light, blue light, and infrared light are set at 1:1:1.5 respectively, and the present application does not limit it.

[0104] Similarly, the same method can also be used to determine the change of the secondary diagonal component, which will not be described in detail in this application.

[0105] Furthermore, the aforementioned units at different positions have an influence on determining the interpolation accuracy. In one or more embodiments of the present application, the front-end device can also determine the main diagonal gradient and the secondary diagonal gradient based on the first-order gradient and the second-order gradient.

[0106] Specifically, the front-end device can determine the main first-order gradient according to the difference of the green light unit on the main diagonal of the infrared unit in step S104, and then determine the main second-order gradient according to the difference of the image data of the infrared unit and other adjacent infrared units on the main diagonal, and determine the main diagonal gradient according to the main first-order gradient and the main second-order gradient.

[0107] The main diagonal gradient is determined using the following formula:

[0108] Gradp=abs(diff(G1, G2))+abs(diff(2*IRc, IR1+IR2))

[0109] Among them, abs(diff(G1, G2)) is the main first-order gradient, that is, the gradient value of the main diagonal line determined in step S104, and abs(diff(2*IRc, IR1+IR2)) is the main second-order gradient.

[0110] In addition, based on the same principle, the front-end device can determine the secondary first-order gradient according to the difference of the green light unit on the secondary diagonal of the infrared unit, and then determine the secondary second-order gradient according to the difference of the image data of the infrared unit and other adjacent infrared units on the secondary diagonal. The secondary diagonal gradient is determined according to the secondary first-order gradient and the secondary second-order gradient.

[0111] The secondary diagonal gradient is determined using the following formula:

[0112] Gradn=abs(diff(G3, G4))+abs(diff(2*IRc, IR3+IR4))

[0113] Among them, abs(diff(G3, G4)) is the secondary first-order gradient, that is, the gradient value of the secondary diagonal determined in step S104, and abs(diff(2*IRc, IR3+IR4)) is the secondary second-order gradient.

[0114] Furthermore, in one or more embodiments of the present specification, the front-end device may introduce redundant parameters when determining the magnitude relationship between the main diagonal gradient and the secondary diagonal gradient, so as to misjudge the presence of high-frequency features at the infrared unit when the difference between the gradients in different directions is only a small difference.

[0115] Specifically, the front-end device determines whether the difference between the main diagonal gradient and the secondary diagonal gradient is less than the redundant parameter based on the preset redundant parameter. If so, it determines that the main diagonal gradient is equal to the secondary diagonal gradient. If not, based on the main diagonal gradient being equal to the secondary diagonal gradient, the magnitude relationship between the main diagonal gradient and the secondary diagonal gradient is determined.

[0116] The formula is:

[0117] if Gradp>Gradn+threshold, then IntpG=IntpGn

[0118] if Gradp <Gradn-threshold,then IntpG=IntpGp

[0119] if abs(Gradp-Gradn)≤threshold, then IntpG=(IntpGn+IntpGp) / 2

[0120] Among them, threshold is a redundant parameter and can be set as needed. This manual does not impose any restrictions on this. For example, it can be set to 1.

[0121] When the difference between the main diagonal gradient and the secondary diagonal gradient is not greater than the redundant parameter parameter, the main diagonal gradient is considered to be the same as the secondary diagonal gradient, and the interpolation is determined according to the mean value of the green units around the infrared unit.

[0122] It should be noted that in the above-mentioned embodiments of the present application, only the structure of each unit of the RGB-IR sensor is described as follows: the infrared unit in the color-infrared image sensor is adjacent to the red unit and the blue unit, and the arrangement mode is adjacent to the green unit in the diagonal direction. It is foreseeable that if the position of the three-color sensor in the RGB-IR sensor changes, the corresponding calculation method will also change. Assuming that the RGGB structure is no longer adopted, but the RRGB structure is used as the basis, and the red unit is replaced by the infrared unit, then the calculation determines the interpolation of red light. Similarly, even if the RGGB structure is still used as the basis, the infrared unit is adjacent to the blue unit on the left and right, and adjacent to the red unit on the top and bottom, then the calculation results of step S104 and step S106 also need to be adjusted accordingly. In the embodiments disclosed in the present application, only a specific unit arrangement structure is used as an example. If the unit in the same position is replaced by other units, the corresponding step calculation interpolation can also be determined based on the same reason.

[0123] The above are one or more implementation methods of the present application. Based on the same idea, the present application also provides a corresponding image data interpolation device, such as Figure 6 shown.

[0124] Figure 6 A schematic diagram of an image data interpolation device provided in the present application, the device applied to a color-infrared image sensor includes:

[0125] An acquisition module 200 acquires image data collected by the color-infrared image sensor, and determines the image data collected by each three-color unit and each infrared unit in the image data;

[0126] A positioning module 202, based on the positional relationship of each unit in the image sensor, determines, for at least some infrared units, other infrared units and adjacent three-color units on the diagonal line of the infrared unit as adjacent units;

[0127] An interpolation gradient determination module 204 determines, based on the image data collected by the infrared unit and the adjacent units, the component interpolation of the green light at the infrared unit when collecting the image data, and the magnitude relationship of the gradient value of the image data;

[0128] An interpolation determination module 206 determines green image data interpolated at the infrared unit according to the size relationship and the component interpolation;

[0129] The data combination module 208 determines the image data for generating a color image according to the image data obtained by interpolation and the image data respectively collected by the three-color units in the image data.

[0130] Optionally, in the color-infrared image sensor, the infrared unit is adjacent to the red unit and the blue unit, and is adjacent to the green unit in a diagonal direction.

[0131] Optionally, the positioning module 202 determines that other infrared units on the main diagonal of the infrared unit that are separated by one green unit are adjacent units, and determines that other infrared units on the secondary diagonal of the infrared unit that are separated by one green unit are adjacent units.

[0132] Optionally, the gradient determination module 204 determines the component interpolation of the green light on the main diagonal based on the difference in image data of the green light unit on the main diagonal of the infrared unit, the difference in image data between the infrared unit and other infrared units on the main diagonal, and the image data of the infrared unit, and determines the component interpolation of the green light on the sub-diagonal based on the difference in image data of the green light unit on the sub-diagonal of the infrared unit, the difference in image data between the infrared unit and other infrared units on the sub-diagonal, and the image data of the infrared unit.

[0133] Optionally, the gradient determination module 204 determines a main first-order gradient according to the difference of green light units on the main diagonal of the infrared unit, determines a main second-order gradient according to the difference of image data between the infrared unit and other adjacent infrared units on the main diagonal, determines a secondary first-order gradient according to the difference of green light units on the secondary diagonal of the infrared unit, determines a secondary second-order gradient according to the difference of image data between the infrared unit and other adjacent infrared units on the secondary diagonal, determines a main diagonal gradient according to the main first-order gradient and the main second-order gradient, determines a secondary diagonal gradient according to the secondary first-order gradient and the secondary second-order gradient, and determines the magnitude relationship between the main diagonal gradient and the secondary diagonal gradient.

[0134] Optionally, the interpolation determination module 208 determines the green image data interpolated at the infrared unit based on the component interpolation of the green light of the secondary diagonal when the main diagonal gradient is greater than the secondary diagonal gradient; determines the green image data interpolated at the infrared unit based on the component interpolation of the green light of the main diagonal when the main diagonal gradient is less than the secondary diagonal gradient; and determines the green image data interpolated at the infrared unit based on the average of the component interpolation of the green light of the main diagonal and the component interpolation of the green light of the secondary diagonal when the main diagonal gradient is equal to the secondary diagonal gradient.

[0135] Optionally, the gradient determination module 204 determines whether the difference between the main diagonal gradient and the secondary diagonal gradient is less than the redundant parameter based on a preset redundant parameter. If so, it determines that the main diagonal gradient is equal to the secondary diagonal gradient. If not, based on the main diagonal gradient being equal to the secondary diagonal gradient, it determines the magnitude relationship between the main diagonal gradient and the secondary diagonal gradient.

[0136] The present application also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides an interpolation method for image data.

[0137] This application also provides Figure 7 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Figure 7 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The image data interpolation method described above. Figure 7 The internal bus, network interface, memory, and non-volatile memory are omitted, and only the processor, RBG-IR sensor, and ISP chip are shown.

[0138] Of course, in addition to software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0139] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0140] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0141] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0142] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0143] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0144] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0147] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0148] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0149] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0150] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0152] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0153] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for interpolating image data, characterized in that: include: Acquire image data collected by a color-infrared image sensor, and determine image data collected by each three-color unit and each infrared unit in the image data, wherein the infrared unit in the color-infrared image sensor is adjacent to the red unit and the blue unit, and is adjacent to the green unit in a diagonal direction; According to the positional relationship of each unit in the image sensor, for at least part of the infrared units, determining other adjacent infrared units and adjacent three-color units on the diagonal line of the infrared unit as adjacent units; Determining, according to the image data collected by the infrared unit and the adjacent unit, a component interpolation of green light at the infrared unit when collecting the image data; According to the difference of the green light unit on the main diagonal of the infrared unit, the main first-order gradient is determined; according to the difference of the image data of the infrared unit and other adjacent infrared units on the main diagonal, the main second-order gradient is determined; according to the difference of the green light unit on the secondary diagonal of the infrared unit, the secondary first-order gradient is determined; according to the difference of the image data of the infrared unit and other adjacent infrared units on the secondary diagonal, the secondary second-order gradient is determined; Determine a main diagonal gradient according to the main first-order gradient and the main second-order gradient, and determine a secondary diagonal gradient according to the secondary first-order gradient and the secondary second-order gradient; According to a preset redundant parameter, determine whether the absolute value of the difference between the main diagonal gradient and the secondary diagonal gradient is less than or equal to the redundant parameter; if so, determine that the main diagonal gradient is equal to the secondary diagonal gradient; if not, determine the magnitude relationship between the main diagonal gradient and the secondary diagonal gradient; Determining green image data interpolated at the infrared unit according to the size relationship and the component interpolation; Image data for generating a color image is determined based on the image data obtained by interpolation and the image data respectively collected by the three-color units in the image data.

2. The method according to claim 1, characterized in that Determining other infrared units adjacent to the infrared unit on a diagonal line as adjacent units includes: Determine that other infrared units on the main diagonal line of the infrared unit, which are separated by one green unit, are adjacent units; It is determined that other infrared units on the secondary diagonal line of the infrared unit, which are separated by one green unit, are adjacent units.

3. The method according to claim 2, characterized in that Determining, according to the image data collected by the infrared unit and the adjacent unit, component interpolation of green light at the infrared unit when collecting the image data, comprises: According to the formula , determine the component interpolation of the green light of the main diagonal; According to the formula , determine the component interpolation of the green light of the secondary diagonal; Among them, IRc is the image data of the infrared unit, G1 and G2 are the image data of the green light units adjacent to the main diagonal of the infrared unit, IR1 and IR2 are the image data of other infrared units on the main diagonal of the infrared unit separated by a green unit, G3 and G4 are the image data of the green light units adjacent to the secondary diagonal of the infrared unit, and IR3 and IR4 are the image data of other infrared units on the secondary diagonal of the infrared unit separated by a green unit.

4. The method according to claim 3, characterized in that: Determining green image data interpolated at the infrared unit according to the size relationship and the component interpolation includes: When the main diagonal gradient is greater than the secondary diagonal gradient, interpolating the green light component of the secondary diagonal to determine the green image data interpolated at the infrared unit; When the main diagonal gradient is less than the secondary diagonal gradient, interpolating the green light components of the main diagonal to determine the green image data interpolated at the infrared unit; When the main diagonal gradient is equal to the sub-diagonal gradient, green image data interpolated at the infrared unit is determined according to the interpolation of the green light components of the main diagonal and the average of the interpolation of the green light components of the sub-diagonal.

5. The method according to claim 1, characterized in that Determining image data for generating a color image according to the image data obtained by interpolation and the image data respectively collected by the three-color units in the image data includes: According to the position of the infrared unit in the image sensor, the image data of the infrared unit in the image data is replaced with the interpolated green image data to determine the image data for generating a color image.

6. An interpolation device for image data, characterized in that: The device is applied to a color-infrared image sensor, comprising: an acquisition module, which acquires image data collected by the color-infrared image sensor, and determines the image data collected by each three-color unit and each infrared unit in the image data, wherein the infrared unit in the color-infrared image sensor is adjacent to the red unit and the blue unit, and is adjacent to the green unit in a diagonal direction; A positioning module, for at least some infrared units, determining other adjacent infrared units and adjacent three-color units on a diagonal line of the infrared unit as adjacent units according to the positional relationship of each unit in the image sensor; A first interpolation determination module determines, based on the image data collected by the infrared unit and the adjacent units, a component interpolation of green light at the infrared unit when collecting the image data; A gradient determination module, used to determine a main first-order gradient according to a difference of green light units on a main diagonal of the infrared unit; determine a main second-order gradient according to a difference of image data between the infrared unit and other adjacent infrared units on the main diagonal; determine a secondary first-order gradient according to a difference of green light units on a secondary diagonal of the infrared unit; determine a secondary second-order gradient according to a difference of image data between the infrared unit and other adjacent infrared units on the secondary diagonal; determine a main diagonal gradient according to the main first-order gradient and the main second-order gradient, and determine a secondary diagonal gradient according to the secondary first-order gradient and the secondary second-order gradient; A size determination module, used to determine whether the absolute value of the difference between the main diagonal gradient and the secondary diagonal gradient is less than or equal to the redundant parameter according to a preset redundant parameter; if so, determine that the main diagonal gradient is equal to the secondary diagonal gradient; if not, determine the size relationship between the main diagonal gradient and the secondary diagonal gradient; a second interpolation determination module, determining green image data interpolated at the infrared unit according to the size relationship and the component interpolation; The data combination module determines the image data used to generate the color image according to the image data obtained by interpolation and the image data respectively collected by the three-color units in the image data.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 5 is implemented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

Images