Smart binning circuit, image sensing device, and method of operating the same

Abstract

To differently allocate an average value obtained by binning with a Bayer pattern, and a binning value obtained by estimating planes of the same color phase and interpolating the planes, according to weighting, and combine the values.SOLUTION: A smart binning circuit 300 includes: an edge information generation unit 310 for generating edge information from plural pieces of pixel data; a weighting allocation unit 320 for providing a weighting according to the edge information generated by the edge information generation unit 310; a binning unit 330 for generating a binning value by performing EDI binning on the basis of the edge information generated by the edge information generation unit 310; a Bayer binning unit 360 for generating an average value obtained by downscaling plural pieces of pixel data formed by a Bayer format, by a 4 sum binning operation; and a combining unit 390 for combining a binning value generated by the binning unit 330 according to the weighting allocated by the weighting allocation unit 320 and the average value generated by the Bayer binning unit 360.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a semiconductor device, and more particularly, to a smart binning circuit, an image sensing device, and an operating method thereof.

Background Art

[0002] In recent years, the paradigm for the computer environment has been switched to ubiquitous computing that enables the use of computer systems anytime and anywhere. As a result, the use of portable electronic devices such as mobile phones, digital cameras, and notebook computers has increased rapidly.

[0003] In particular, due to the rapid development of video equipment, the development of image capturing devices such as cameras and camcorders equipped with image sensors has been accelerated. Such image capturing devices can capture an image and record it on a recording medium, and can be played back at any time, and the number of users is increasing rapidly. As a result, the user's requirements for performance and functions are gradually increasing, and high performance and multi-functionality are being pursued along with miniaturization, weight reduction, and low power consumption.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Embodiments of the present invention can improve the resolution by providing smart binning in which an average value binned by a Bayer pattern and a binned value estimated and interpolated for the same hue plane are assigned and combined to be different from each other according to weights.

[0005] Also, at high illuminance, by bypassing and immediately outputting image data, there is no image quality degradation, and at low illuminance, by outputting an image via smart binning, noise can be reduced and the resolution can be improved.

[0006] The technical problem to be solved in the present invention is not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those with ordinary knowledge in the technical field to which the present invention pertains from the following description.

Means for Solving the Problem

[0007] The present invention provides a smart binning circuit, an image sensing device, and an operating method thereof.

[0008] A smart binning circuit according to an embodiment of the present invention includes an edge information generation unit that generates edge information from a plurality of pixel data output from a pixel array, a weighting assignment unit that assigns weighting according to the edge information generated by the edge information generation unit, a binning unit that performs EDI (Edge detection interpolation) binning based on the edge information generated by the edge information generation unit to generate a binning value, a Bayer binning unit that generates an average value obtained by downsizing the plurality of pixel data formed in a Bayer format through a 4-sum binning operation, and a combining unit that combines the binning value generated by the binning unit and the average value generated by the Bayer binning unit according to the weighting assigned by the weighting assignment unit.

[0009] Further, the binning unit can include an interpolation block that proceeds with interpolation to estimate a plane of the same hue based on the edge information generated by the edge information generation unit, and a binning block that performs the EDI binning based on the pixel data interpolated by the interpolation block to generate the binning value.

[0010] Further, the interpolation block can estimate red hue pixels and blue hue pixels as green hue based on the edge information generated by the edge information generation unit using a horizontal filter, a vertical filter, and a horizontal / vertical filter.

[0011] In addition, when the minimum edge direction of the pixels is the horizontal direction, the interpolation block uses a horizontal filter; when the minimum edge direction is the vertical direction, the interpolation block uses a vertical filter; and when the minimum edge direction is the diagonal direction, the interpolation block can use the horizontal filter and the vertical filter simultaneously.

[0012] In addition, the combining unit controls to assign weights assigned by the weighting assignment unit to the binning value generated by the binning unit and the average value generated by the Bayer binning unit so that they are different from each other, and can output a combined value by combining the binning value and the average value with the weights assigned to be different from each other.

[0013] In addition, if the edge strength is greater than a set value, the combining unit assigns a weight to the binning value generated by the binning unit; if the edge strength is less than the set value, the combining unit can assign a weight to the average value generated by the Bayer binning unit.

[0014] In addition, the weighting assignment unit can calculate an inclination value using a horizontal filter and a vertical filter, and calculate one weight per 2×2 pixel array based on this.

[0015] An image sensing device according to another embodiment of the present invention includes an image sensor having a plurality of pixels and an image signal processor that processes an output signal of the image sensor. Among the image sensor and the image signal processor, a smart binning circuit is realized inside one of them. The smart binning circuit includes an edge information generation unit that generates edge information from a plurality of pixel data output from a pixel array, a weighting assignment unit that assigns a weight according to the edge information generated by the edge information generation unit, a binning unit that performs EDI (Edge detection interpolation) binning based on the edge information generated by the edge information generation unit to generate a binning value, a Bayer binning unit that generates an average value obtained by downsizing the plurality of pixel data formed in a Bayer format through 4-sum binning, and a combining unit that combines the binning value generated by the binning unit and the average value generated by the Bayer binning unit according to the weight assigned by the weighting assignment unit.

[0016] Further, the binning unit may include an interpolation block that performs interpolation to estimate a green plane based on the edge information generated by the edge information generation unit, and a binning block that performs the EDI binning based on the pixel data of the green plane interpolated by the interpolation block to generate the binning value.

[0017] Further, the interpolation block can estimate red hue pixels and blue hue pixels as green hue using a horizontal filter, a vertical filter, and a horizontal / vertical filter based on the edge information generated by the edge information generation unit.

[0018] Further, the combining unit can control to assign the weights calculated by the weighting assignment unit to the binning value generated by the binning unit and the average value generated by the Bayer binning unit so as to be different from each other, and combine and output the binning value and the average value with the weights assigned to be different from each other.

[0019] Further, if the strength of the edge is greater than a set value, the bonding part can assign a weight to the binning value generated in the binning part, and if the strength of the edge is less than the set value, the bonding part can assign a weight to the average value generated in the Bayer binning part.

[0020] Also, the weight assignment part can calculate an inclination value using a horizontal filter and a vertical filter, and based on this, assign one weight per 2×2 pixel array.

[0021] An image sensing device according to another embodiment of the present invention includes an image sensor having a plurality of pixels and an image signal processor that processes an output signal of the image sensor. A smart binning circuit is implemented inside either one of the image sensor and the image signal processor. The smart binning circuit includes a smart binning part that performs a first binning operation according to edge information generated from a plurality of pixel data output from a pixel array and outputs first pixel information, a Bayer binning part that converts the plurality of pixel data into a Bayer format, performs a second binning operation, and outputs second pixel information, an illuminance information generation part that generates illuminance information including high illuminance and low illuminance, and a binning pixel information selection part that selectively outputs the first pixel information output from the smart binning part or the second pixel information output from the Bayer binning part according to the illuminance information generated by the illuminance information generation part.

[0022] Also, the pixel information selection part can output the first pixel information output from the smart binning part when low illuminance information is provided by the illuminance information generation part, and can output the second pixel information output from the Bayer binning part when high illuminance information is provided by the illuminance information generation part.

[0023] Further, the first binning operation is an operation of performing EDI (Edge detection interpolation) binning based on edge information, and the second binning operation can be an operation of generating an average value obtained by downscaling a plurality of pixels formed in a Bayer format through 4-sum binning.

[0024] Further, the smart binning unit can include an edge information generation unit that generates edge information from a plurality of pixel data output from a pixel array, a weighting assignment unit that assigns weighting according to the edge information generated by the edge information generation unit, and a binning unit that performs EDI (Edge detecton interpolation) binning based on the edge information generated by the edge information generation unit according to the weighting assigned by the weighting assignment unit to generate a binning value.

[0025] Further, an operation method of an image sensing apparatus according to still another embodiment of the present invention includes a step of performing a first binning operation according to edge information generated from a plurality of pixel data output from a pixel array to output first pixel information, a step of converting the plurality of pixel data into a Bayer format and performing a second binning operation to output second pixel information, a step of discriminating a high illuminance state and a low illuminance state based on a preset reference value to generate illuminance information including high illuminance information and low illuminance information, and a step of selectively outputting the first pixel information or the second pixel information according to the illuminance information generated in the step of generating the illuminance information.

[0026] The first binning operation is an operation of performing EDI (Edge detection interpolation) binning based on edge information. The second binning operation is an operation of generating an average value obtained by downscaling a plurality of pixels formed in a Bayer format through 4-sum binning. In the step of selectively outputting, when low illuminance information is provided, the first pixel information can be output, and when high illuminance information is provided, the second pixel information can be output.

[0027] The step of outputting the first pixel information may include a step of generating edge information from a plurality of pixel data output from the pixel array, a step of assigning weights according to the edge information, and a step of performing EDI (Edge detecton interpolation) binning based on the edge information according to the assigned weights to generate a binning value.

[0028] The effects obtained from the present invention are not limited to the effects mentioned above. Other effects not mentioned will be clearly understood by those with ordinary knowledge in the field to which the present invention pertains from the following description.

Effects of the Invention

[0029] The smart binning circuit, image sensing device, and operation method thereof according to an embodiment of the present invention can improve the resolution by providing smart binning in which an average value binned by a Bayer pattern and a binning value estimated and interpolated for the same hue plane are assigned to be different from each other according to weights and combined.

[0030] Also, at high illuminance, by bypassing and immediately outputting image data, there is no image quality degradation. At low illuminance, by outputting an image through smart binning, noise can be reduced and the resolution can be improved.

Brief Description of the Drawings

[0031]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0032] Hereinafter, desirable embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that in the following description, only the parts necessary for understanding the operation according to the present invention will be described, and the description of the other parts will be omitted so as not to obscure the gist of the present invention.

[0033] Hereinafter, embodiments of the present invention will be described more specifically with reference to the drawings.

[0034] FIG. 1 shows a block diagram of an image sensing device according to an embodiment of the present invention.

[0035] As shown in FIG. 1, the image sensing device 10 can include an image sensor 100 and an image signal processor (ISP) 400.

[0036] The image sensing device 10 can be implemented by a PC (personal computer) or a mobile computing device. The image sensing device can be implemented by a laptop computer, a mobile phone, a smart phone, a tablet PC, a PDA (personal digital assistant), an EDA (enterprise digital assistant), a digital still camera, a digital video camera, a PMP (portable multimedia player), a mobile internet device (MID), a wearable computer, an internet of things (IoT) device, or an internet of everything (IoE) device.

[0037] The image sensor 100 can include a pixel array 200 and a smart binning circuit 300.

[0038] The pixel array 200 can include a plurality of pixels. Here, a pixel can mean pixel data and can have an RGB data format, a YUV data format, or a YCbCr data format, but is not limited thereto.

[0039] The smart binning circuit 300 generates edge information from each pixel data output from the pixel array 200 having a plurality of pixels, performs EDI (Edge detection interpolation) binning based on this to generate binning values, and generates an average value downscaled from each pixel data through 4-sum binning of the same hue, and can combine and output the binning value and the average value according to the weighting assigned according to the generated edge information.

[0040] The detailed configuration and operation of the smart binning circuit 300 will be described later with reference to FIGS. 3 to 9.

[0041] The image signal processor 400 is an embodiment of a processor and can be implemented by an integrated circuit, a system on chip (SoC), or a mobile AP. The image signal processor 400 processes the output signal of the image sensor 100. That is, it can receive and process the image output signal combined and output by the smart binning circuit 300 provided in the image sensor 100.

[0042] Specifically, the image signal processor 400 can process a Bayer pattern corresponding to pixel data to generate RGB image data. For example, the image signal processor 400 can process (or, process) the Bayer pattern so that the image data (IDATA) can be displayed on a display, and can transmit the processed image data to an interface.

[0043] According to an embodiment, the image sensor 100 and the image signal processor 400 can each be implemented by a chip and can be implemented in one package, for example, a multi-chip package (MCP). According to other embodiments, the image sensor 100 and the image signal processor 400 can also be implemented in one chip.

[0044] FIG. 2 shows a block diagram of an image sensing apparatus according to another embodiment of the present invention.

[0045] As shown in FIG. 2, the image sensing apparatus 10 can include an image sensor 100 and an image signal processor (ISP) 400.

[0046] Except that the smart binning circuit 300 is not implemented in the image sensor 100 but is implemented in the image signal processor 400, the structure and operation of the image sensing apparatus 10 in FIG. 2 are substantially the same as or similar to those of the image sensing apparatus 10 in FIG. 1, so detailed description thereof will be omitted.

[0047] Hereinafter, a smart binning circuit according to an embodiment of the present invention will be described with reference to FIGS. 3 to 7.

[0048] FIG. 3 shows a block diagram of a smart binning circuit according to an embodiment of the present invention, FIG. 4 is a diagram for explaining the G plane estimation of the binning unit shown in FIG. 3, FIG. 5 is a diagram for explaining the average value of the Bayer binning unit shown in FIG. 3, FIG. 6 is a diagram for explaining the calculation process of the weighting assignment unit shown in FIG. 3, and FIG. 7 is a diagram for explaining an example of a filter used in the calculation process of the weighting assignment unit shown in FIG. 3.

[0049] As shown in FIGS. 3 to 7, the smart binning circuit 300 includes an edge information generation unit 310, a weighting assignment unit 320, a binning unit 330, a Bayer binning unit 360, and a combining unit 390.

[0050] The edge information generation unit 310 can sense edge information from the image data output from the pixel array.

[0051] Based on the edge information sensed by the edge information generation unit 310, the binning unit 330 can perform EDI (Edge detection interpolation) binning to generate a binning value. The EDI binning operation is an operation that advances interpolation within a plane based on the edge information and executes binning based on the interpolated data.

[0052] The binning unit 330 can include an interpolation block 340 and a binning block 350.

[0053] As shown in FIG. 4, in this embodiment, a 6×6 pixel array will be described as an example.

[0054] Based on the sensed edge information, the interpolation block 340 can advance interpolation to estimate the green plane for the edge information generation unit 310.

[0055] For example, pixels of green hue such as G11, G13, G15, G22, G24, G26, G31, G33, G35, G42, G44, G46, G51, G53, G55, G62, G64, G66 are of green hue and thus are maintained as they are without change. Red hue pixels such as R12, R14, R16, R32, R34, R36, R52, R54, R56 are each estimated as G12, G14, G16, G32, G34, G36, G52, G54, G56 of green hue and can be changed. Blue hue pixels such as B21, B23, B25, B41, B43, B45, B61, B63, B65 are each estimated as G21, G23, G25, G41, G43, G45, G61, G63, G65 of green hue and can be changed.

[0056] At this time, based on the sensed edge information, the interpolation block 340 can advance green hue estimation through a horizontal filter, a vertical filter, and a horizontal / vertical filter in a direction that does not oppose the texture.

[0057] If, hypothetically, the minimum edge direction of the red pixel R34 is horizontal, then using a horizontal filter, the red pixel R34 is estimated and interpolated as the green hue pixel value G34 according to the formula G34 = (G33 + G35) / 2.

[0058] If, hypothetically, the minimum edge direction of the red pixel R34 is vertical, then using a vertical filter, the red pixel R34 is estimated and interpolated as the green hue pixel G34 according to the formula G34 = (G24 + G44) / 2.

[0059] If, hypothetically, the minimum edge direction of the red pixel R34 is diagonal (horizontal / vertical), then using a horizontal / vertical filter, the red pixel R34 is estimated and interpolated as the green hue pixel G34 according to the formula G34 = (G33 + G35 + G24 + G44) / 4.

[0060] The binning block 350 can perform binning based on the pixel data of the green plane interpolated in the interpolation block 340 to generate a binning value.

[0061] That is, the 4 pixels included in the 2×2 pixel array can be downscaled to 1 pixel.

[0062] For example, G34 can be downscaled according to the formula (G33 + G34 + G43 + G44) / 4. That is, binning is performed to generate a binning value that is the average value of the green hue pixels G33, G44 and the interpolated pixels G34, G43.

[0063] At this time, the 6×6 pixel array shown in FIG. 4 can be downscaled to 9 2×2 pixel arrays.

[0064] The weighting assignment unit 320 can calculate the weighting according to the edge strength of the texture sensed by the edge information generation unit 310.

[0065] As shown in FIGS. 5 and 6, the weighting assignment unit 320 can calculate the inclination value using a horizontal filter and a vertical filter, and based on this, calculate one weighting per 2×2 pixel array.

[0066] For example, the average value Y00_bin for the 2×2 pixel array G11, R12, B21, G22 can be calculated by the following formula. [Equation 1] Y00_bin = (G11 + R12 + B21 + G22) / 4 That is, for each of the nine 2×2 pixel arrays that collectively form the 6×6 pixel array shown in FIG. 5, it can be calculated in the same manner as Equation 1 so as to generate nine average values.

[0067] Next, as shown in FIG. 6, the inclination value (Gradient) can be calculated for the nine 2×2 representative values using a horizontal filter and a vertical filter, and based on this, a weighting value from 0 to 16 can be obtained.

[0068] At this time, in the embodiment of the present invention, a horizontal filter and a vertical filter using the Prewitt function are used, but in other embodiments, the Sobel function, the Roberts function, etc. may be used.

[0069] Since the specific operations for obtaining the weighting value are well-known to those skilled in the art, detailed descriptions are omitted.

[0070] The Bayer binning unit 360 can generate an average value representing a plurality of down-scaled pixels through a 4-sum (summation) average value binning operation of the same hue for the image data output from the pixel array 200. In the present embodiment, the 4-sum (summation) average value binning operation can be performed on the pixel data in the Bayer pattern.

[0071] In the 2×2 pixel array located at the center in the 6×6 pixel array shown in FIG. 7, the average values for the green hue pixels G33, G44, the red hue pixel R34, and the blue hue pixel B43 can be calculated as the average of the sum of the other adjacent green hue pixels, as shown in Equation 2 below. That is, the average value of the green hue pixels (G OUT ) can be calculated as the average value of the adjacent green pixels G35, G53, G55 and the green pixel G33 of the 2×2 pixel array by Equation 2 below. Also, the average value of the red hue pixels (R OUT ) and the average value of the blue hue pixels (B OUT ) can be calculated in the same way as the average value of the green hue pixels (G OUT ) by Equation 2 below. [Equation 2] G OUT =(G33 + G35 + G53 + G55) / 4 R OUT =(R34 + R36 + R54 + R56) / 4 B OUT =(B43 + B45 + B63 + B65) / 4 The combining unit 390 can assign weights calculated by the weighting assignment unit 320 to the binning value generated by the binning unit 330 and the average value generated by the Bayer binning unit 360 so that they are different from each other, and combine and output the binning value and the average value with different weights assigned to each other.

[0072] For example, the weighting assignment in the combining unit 390 can be assigned as shown in Equation 3 below. [Equation 3] {(Weight * Average value)+(16 - Weight)*Binning value} / 16 At this time, if the intensity of the edge based on the edge information is greater than the set value, a weight is assigned to the binning value generated by the binning block 350 of the binning unit 300, and if the intensity of the edge based on the edge information is less than the set value, a weight can be assigned to the average value generated by the Bayer binning unit 380.

[0073] Hereinafter, a smart binning circuit according to still another embodiment of the present invention will be described with reference to FIG. 8, which shows a block diagram of the smart binning circuit according to still another embodiment of the present invention.

[0074] The smart binning circuit 300 shown in FIG. 8 includes a smart binning unit 305, a Bayer binning unit 360, an illuminance information generation unit 370, and a pixel information selection unit 380.

[0075] The smart binning unit 305 can perform a first binning operation according to edge information generated from a plurality of pixel data output from a pixel array (for example, the pixel array 200 shown in FIG. 1 or FIG. 2) and output first pixel information.

[0076] At this time, the first binning operation indicates an operation of performing EDI (Edge detection interpolation) binning based on the edge information.

[0077] The smart binning unit 305 can include an edge information generation unit 310, a weighting assignment unit 320, and a binning unit 330.

[0078] The edge information generation unit 310 can generate edge information from a plurality of pixel data output from the pixel array.

[0079] The weighting assignment unit 320 can assign a weight according to the edge information generated by the edge information generation unit 310.

[0080] The binning unit 330 can perform EDI (Edge detecton interpolation) binning based on the edge information generated by the edge information generation unit according to the weight assigned by the weighting assignment unit 320 and generate a binning value.

[0081] The binning unit 330 can include an interpolation block 340 and a binning block 350.

[0082] Since the more detailed operations of the edge information generation unit 310, the weighting assignment unit 320, and the binning unit 330 provided in the smart binning unit 305 are the same as those described in FIGS. 3 to 7, they will be omitted.

[0083] The Bayer binning unit 360 can convert the plurality of pixel data into the Bayer format and perform a second binning operation to output second pixel information. At this time, the second binning operation is an operation of generating an average value obtained by downscaling a plurality of pixels formed in the Bayer format through 4-sum binning.

[0084] Since the detailed operation description of the Bayer binning unit 360 is the same as that described in FIGS. 3 to 7, it will be omitted.

[0085] The illuminance information generation unit 370 can generate illuminance information including high illuminance and low illuminance indicating whether the surrounding environmental conditions are in a high illuminance state or a low illuminance state according to a preset standard.

[0086] The pixel information selection unit 380 can selectively output the first pixel information output from the smart binning unit 305 or the second pixel information output from the Bayer binning unit 360 according to the illuminance information generated by the illuminance information generation unit 370.

[0087] At this time, when low illuminance information is provided from the illuminance information generation unit 370, the pixel information selection unit 380 outputs the first pixel information output from the smart binning unit 305, and when high illuminance information is provided from the illuminance information generation unit 370, the pixel information selection unit 380 can output the second pixel information output from the Bayer binning unit 360.

[0088] Hereinafter, with reference to FIG. 9, the operation of the image sensing device according to an embodiment of the present invention will be described. FIG. 9 is a flowchart for explaining the operation of the image sensing device according to an embodiment of the present invention.

[0089] As shown in FIG. 9, the operation of the image sensing device according to an embodiment of the present invention can include a step of providing first pixel information (S1000), a step of providing second pixel information (S2000), a step of determining illuminance information (S3000), and a selective output step (S4000).

[0090] In the step of providing first pixel information (S1000), a first binning operation can be performed according to edge information generated from a plurality of pixel data output from a pixel array to output first pixel information.

[0091] At this time, the first binning operation is an operation of performing EDI (Edge detection interpolation) binning based on the edge information.

[0092] The step of outputting the first pixel information (S1000) can include a step of generating edge information (S1100), a weight assignment step (S1200), and a binning value generation step (S1300).

[0093] In the step of generating edge information (S1100), edge information can be generated from a plurality of pixel data output from the pixel array.

[0094] In the weight assignment step (S1200), weights can be calculated and assigned according to the generated edge information.

[0095] At this time, a tilt value can be calculated using a horizontal filter and a vertical filter, and based on this, one weight can be calculated per 2×2 pixel array. The range of the weight can be between 0 and 16.

[0096] A detailed description of the weight calculation has already been described in FIGS. 5 and 6, so it is omitted.

[0097] In the binning value generation step (S1300), EDI (Edge detection interpolation) binning can be performed based on the generated edge information. At this time, interpolation can be advanced to estimate the same plane based on the generated edge information, and binning can be executed based on the interpolated pixel data to generate a binning value.

[0098] At this time, based on the perceived edge information, green hue estimation can be advanced through a horizontal filter, a vertical filter, and a horizontal / vertical filter in a direction that does not oppose the texture.

[0099] As shown in FIG. 4, when the minimum edge direction is horizontal at the R34 pixel, it can be estimated and interpolated as the green hue pixel value G34 by the formula G34 = (G33 + G35) / 2 using a horizontal filter.

[0100] If, hypothetically, the minimum edge direction is vertical at the R34 pixel, it can be estimated and interpolated as the green hue pixel G34 by the formula G34 = (G24 + G44) / 2 using a vertical filter.

[0101] If, hypothetically, the minimum edge direction is diagonal (horizontal / vertical) at the R34 pixel, it can be estimated and interpolated as the green hue pixel G34 by the formula G34 = (G33 + G35 + G24 + G44) / 4 using a horizontal / vertical filter.

[0102] In the step of providing second pixel information (S2000), a plurality of pixel data can be converted into a Bayer format, and a second binning operation can be performed to provide second pixel information.

[0103] At this time, the second binning operation represents an operation of generating an average value by downscaling a plurality of pixels formed in the Bayer format through 4-sum binning.

[0104] In the step of determining illuminance information (S3000), a high illuminance state and a low illuminance state are determined according to a reference value preset according to the surrounding environmental conditions, and illuminance information including high illuminance information and low illuminance information can be generated.

[0105] In the selective output steps (S4000, S5000), the first pixel information or the second pixel information can be selectively output according to the illuminance information generated in the step of generating illuminance information (S3000).

[0106] At this time, when low illuminance information is provided, the first pixel information can be output (S4000), and when high illuminance information is provided, the second pixel information can be output (S5000).

[0107] Hereinafter, an embodiment of a system to which an image sensing device according to an embodiment of the present invention is applied will be described. FIG. 10 shows a block diagram for explaining an embodiment of a system to which an image sensing device according to an embodiment of the present invention is applied.

[0108] The system shown in FIG. 10 includes, but is not limited to, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a handheld computing device, a cellular phone, a smartphone, a mobile phone, a workstation, a network computer, a consumer device, an application server, a storage device, an intelligent display, a peripheral device, such as a switch, a modem, a router, and others, or generally any type of computing device, and can be any of various types of computing devices.

[0109] According to one embodiment, the system illustrated in FIG. 9 can represent a system-on-a-chip (SOC). As implied by the name, components such as those of the SOC (1000) can be integrated on a single semiconductor substrate, like an integrated circuit “chip”. In some embodiments, components such as these can be implemented on two or more separate chips in the system. The SOC (1000) will be used herein as an example.

[0110] In the illustrated embodiment, components such as those of the SOC (1000) can include a central processing unit (CPU) complex 1020, on-chip peripheral device components 1040A and 1040B (more simply, “peripherals”), a memory controller (MC) 1030, an image signal processor 400, and a communication fabric 1010.

[0111] The SOC (1000) can further be coupled to additional components such as, for example, a memory 1800 and an image sensor 100. All of the components (1020, 1030, 1040A and 1040B, and 200) can be coupled to the communication fabric 1010. The memory controller 1030 can be coupled to the memory 1800 during use, and the peripheral device 1040B can be coupled to an external interface 1900 during use.

[0112] In the illustrated embodiment, the CPU complex 1020 can include one or more processors 1024 and a level 2 (L2) cache 1022. Peripherals 1040A and 1040B can be any set of additional hardware functionality included in the SOC (1000). For example, peripherals 1040A and 1040B can include a display controller configured to display video data on one or more display devices, a graphics processing unit (GPU), a video encoder / decoder, a scaler, a rotator, a blender, and others.

[0113] The image signal processor 400 can process image capture data from the image sensor 100 (or other image sensors). The configurations and operations of the image signal processor 400 and the image sensor 100 shown in FIGS. 1-9 can be applicable.

[0114] The peripherals can further include audio peripherals such as microphones, speakers, interfaces for microphones and speakers, audio processors, digital signal processors, mixers, and others. The peripherals can include a peripheral interface controller (e.g., peripheral 1040B) for various external interfaces 1900 of the SOC (1000) including interfaces such as a universal serial bus (USB), a peripheral component interconnect (PCI) including PCI express (PCIe), serial and parallel ports, and others. The peripherals can further include networking peripherals such as a media access controller (MAC). Generally, any set of hardware can be included by various embodiments and the like.

[0115] The CPU complex 1020 can include one or more CPU processors 1024 that serve as the CPU of the SOC (1000). The CPU of the system can include a processor (etc.) that executes the main control software of the system, such as the operation system. Generally, during use, the software executed by the CPU can control other components of the system to achieve the intended functionality of the system. The processor 1024 can further execute other software, such as application programs. The application programs can provide user functionality and can rely on the operation system for low-level device control. Therefore, the processor 1024 can further be referred to as an application processor.

[0116] The CPU complex 1020 can further include interfaces for other hardware, such as the L2 cache 1022 and / or other components of the system (e.g., an interface to the communication fabric 1010).

[0117] Generally, a processor can include any circuit and / or microcode configured to execute instruction words defined in an instruction set architecture implemented by the processor. The instruction words and data operated on the processor in response to executing the instruction words can generally be stored in the memory 1800, but certain instruction words can also be defined for direct processor access to peripheral devices, etc. The processor can cover processor cores implemented on an integrated circuit together with other components, etc., as a system-on-chip (SOC (1000)) or at other levels of integration. The processor can further cover other microprocessors, processor cores, and / or microprocessors integrated within a multi-chip module implementation, processors implemented as multiple integrated circuits, and others.

[0118] The memory controller 1030 can generally include a circuit that receives memory operations from other components of the SOC (1000) and accesses the memory 1800 to perform the memory operations. The memory controller 1030 can be configured to access any type of memory 1800. For example, the memory 1800 can be SRAM (static random access memory), DRAM (dynamic RAM), such as SDRAM (synchronous DRAM) including double data rate (DDR, DDR2, DDR3, etc.) DRAM. DDR DRAMs such as low power / mobile versions (e.g., LPDDR, mDDR, etc.) can be supported. The memory controller 1030 can include, for example, a queue for memory operations that instructs (and potentially re-instructs) and presents the operations to the memory 1800. The memory controller 1030 can further include a data buffer that stores write data waiting to be written to the memory and read data waiting for a return to the source of the memory operation.

[0119] In some embodiments, etc., the memory controller 1030 can include a memory cache that stores recently accessed memory data. In an SOC implementation example, for example, the memory cache can reduce the power consumption in the SOC by avoiding re-access of data from the memory 1800 when it is expected to be accessed again immediately. In some cases, the memory cache can further be a private cache that assists only certain components, etc., for example, a system cache that is opposite to the L2 cache 1022 or cache of the processor 1024. Additionally, in some embodiments, the system cache need not be located within the memory controller 1030.

[0120] In an embodiment, the memory 1800 can be packaged with the SOC (1000) in a chip-on-chip or package-on-package configuration. A multi-chip module configuration of the SOC (1000) and the memory 1800 can also be used. Such a configuration etc. can be relatively more stable (in terms of data observation aspect) than transmission to other components etc. in the system (e.g., to endpoints 16A and 16B). Thus, while the protected data can reside in the memory 1800 in an unencrypted state, the protected data can be encrypted for exchange between the SOC (1000) and an external endpoint.

[0121] The communication fabric 1010 can be any communication interconnect and protocol for communication among the components of the SOC (1000). The communication fabric 1010 can be based on a bus including a shared bus configuration, a cross bar configuration, and a hierarchical bus having bridges. The communication fabric 1010 can further be packet-based and can be a hierarchical one having bridges, a cross bar, a point-to-point, or other interconnect. There can be more or fewer components / sub-components than those shown in FIG. 9.

[0122] In some embodiments, the methods described herein can be implemented by a computer program product, or software. In some embodiments, a non - transitory, computer - readable storage medium can store instructions that can be used to program a computer system (or other electronic device) to perform some or all of the techniques described herein. A computer - readable storage medium can include any mechanism for storing information in a form readable by a machine (e.g., a computer), such as software, a processing application. Machine - readable media can include, but are not limited to, magnetic storage media (e.g., floppy disk), optical storage media (e.g., CD - ROM), magneto - optical storage media, read - only memory (ROM), random access memory (RAM), erasable and programmable memory (e.g., EPROM and EEPROM), flash memory, and other types of media suitable for storing electrical or program instructions. Further, program instructions can be communicated using light, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.).

[0123] The computer system 1000 can include a processor unit 1020 (including, if possible, multiple processors, single - threaded processors, multi - threaded processors, multi - core processors, etc.) configured to execute a local tom mapping circuit that can exist within one or more modules, e.g., within program instructions stored in the memory 1800 of the same computer system, or within program instructions stored in the memory of still other computer systems that are similar to or different from the computer system 1000.

[0124] On the one hand, in the detailed description of the present invention, specific embodiments have been described, but it goes without saying that various modifications are possible within the scope not departing from the present invention. Therefore, the scope of the present invention should not be determined to be limited to the described embodiments, but should be determined by not only the claims described later but also those equivalent to the claims.

Explanation of Signs

[0125] 10 Image sensing device 100 Image sensor 200 Pixel array 300 Smart binning circuit 310 Edge information generation unit 320 Weight assignment unit 330 Binning unit 340 Interpolation block 350 Binning block 360 Bayer binning unit 390 Combining unit 400 Image signal processor

Claims

1. An edge information generation unit that generates edge information from a plurality of pixel data output from a pixel array; A weighting assignment unit that assigns a weight according to the edge information generated by the edge information generation unit; A binning unit that performs EDI (Edge Detection Interpolation) binning based on the edge information generated by the edge information generation unit to generate a binning value; A Bayer binning unit that generates an average value obtained by downsizing the plurality of pixel data formed in a Bayer format through a 4-sum binning operation; A combining unit that combines the binning value generated by the binning unit and the average value generated by the Bayer binning unit according to the weight assigned by the weighting assignment unit; Comprising The EDI binning is an operation that advances interpolation within a plane based on the edge information and performs binning based on the interpolated data; The combining unit is a smart binning circuit that assigns a weight to the binning value generated by the binning unit if the intensity of the edge is greater than a set value, and assigns a weight to the average value generated by the Bayer binning unit if the intensity of the edge is less than the set value.

2. The binning unit An interpolation block that advances interpolation to estimate a plane of the same hue based on the edge information generated by the edge information generation unit; A binning block that performs the EDI binning based on the pixel data interpolated by the interpolation block to generate the binning value; The smart binning circuit according to claim 1, comprising.

3. The interpolation block estimates red hue pixels and blue hue pixels as green hue using a horizontal filter, a vertical filter, and a horizontal / vertical filter based on the edge information generated by the edge information generation unit. The smart binning circuit according to claim 2.

4. The interpolation block uses a horizontal filter when the minimum edge direction of a pixel is horizontal, uses a vertical filter when the minimum edge direction is vertical, and uses a horizontal filter and a vertical filter simultaneously when the minimum edge direction is diagonal. The smart binning circuit according to claim 3.

5. The combining unit Control is performed so that the weights assigned by the weighting assignment unit to the binning value generated by the binning unit and the average value generated by the Bayer binning unit are assigned to be different from each other, and the binning value and the average value to which the weights are assigned to be different from each other are combined, and the combined value is output. The smart binning circuit according to claim 1.

6. The weighting assignment unit calculates the inclination values of a plurality of pixels provided in the pixel array using a horizontal filter and a vertical filter, and calculates one weighting per 2×2 pixel array based on this. The smart binning circuit according to claim 1.

7. An image sensor including a plurality of pixels, An image signal processor that processes the output signal of the image sensor, Comprising, Among the image sensor and the image signal processor, a smart binning circuit is realized inside any one of them, The smart binning circuit is, An edge information generation unit that generates edge information from a plurality of pixel data output from a pixel array, A weighting assignment unit that assigns a weight according to the edge information generated by the edge information generation unit, A binning unit that performs EDI (Edge detection interpolation) binning based on the edge information generated by the edge information generation unit to generate a binning value, A Bayer binning unit that generates an average value obtained by downsizing the plurality of pixel data formed in a Bayer format through 4-sum binning, A combining unit that combines the binning value generated by the binning unit and the average value generated by the Bayer binning unit according to the weight assigned by the weighting assignment unit, Comprising, The EDI binning is an operation of proceeding with interpolation within a plane based on the edge information and performing binning based on the interpolated data. The combining unit assigns a weight to the binning value generated by the binning unit if the intensity of the edge is greater than a set value, and assigns a weight to the average value generated by the Bayer binning unit if the intensity of the edge is less than the set value. An image sensing device.

8. The binning unit is, An interpolation block that proceeds with interpolation to estimate a green plane based on the edge information generated by the edge information generation unit, A binning block that performs the EDI binning to generate the binning value based on the pixel data of the green plane interpolated by the interpolation block; The image sensing device according to claim 7, comprising: **Claim 9** The interpolation block estimates red hue pixels and blue hue pixels as green hue using a horizontal filter, a vertical filter, and a horizontal / vertical filter based on the edge information generated by the edge information generation unit. The image sensing device according to claim 8. **Claim 10** The combining unit controls to assign weights calculated by the weighting assignment unit to the binning value generated by the binning unit and the average value generated by the Bayer binning unit so that they are different from each other, and combines and outputs the binning value and the average value with weights assigned to be different from each other. The image sensing device according to claim 7. **Claim 11** The weighting assignment unit calculates an inclination value of the plurality of pixels using a horizontal filter and a vertical filter, and assigns one weight per 2×2 pixel array based on this. The image sensing device according to claim 7. **Claim 12** An image sensor including a plurality of pixels; An image signal processor that processes an output signal of the image sensor; comprising a smart binning circuit is implemented inside either one of the image sensor and the image signal processor; the smart binning circuit a smart binning unit that performs a first binning operation according to edge information generated from a plurality of pixel data output from a pixel array and outputs first pixel information; a Bayer binning unit that converts the plurality of pixel data into a Bayer format, performs a second binning operation, and outputs second pixel information; an illuminance information generation unit that generates illuminance information including high illuminance and low illuminance; a binning pixel information selection unit that selectively outputs the first pixel information output from the smart binning unit or the second pixel information output from the Bayer binning unit according to the illuminance information generated by the illuminance information generation unit; comprising The first binning operation is an operation of performing EDI (Edge Detection Interpolation) binning based on edge information, and the second binning operation is an operation of generating an average value obtained by downscaling a plurality of pixels formed in a Bayer format through 4-sum binning. The EDI binning is an operation of advancing interpolation within a plane based on the edge information and performing binning based on the interpolated data. The binning pixel information selection unit outputs the first pixel information output from the smart binning unit when low illuminance information is provided by the illuminance information generation unit, and outputs second pixel information output from the Bayer binning unit when high illuminance information is provided by the illuminance information generation unit. An image sensing device.

13. The smart binning unit An edge information generation unit that generates edge information from a plurality of pixel data output from a pixel array; A weighting assignment unit that assigns weighting according to the edge information generated by the edge information generation unit; A binning unit that performs EDI (Edge Detection Interpolation) binning based on the edge information generated by the edge information generation unit according to the weighting assigned by the weighting assignment unit to generate a binning value; The image sensing device according to claim 12, comprising:

14. Performing a first binning operation according to edge information generated from a plurality of pixel data output from a pixel array to output first pixel information; Converting the plurality of pixel data into a Bayer format and performing a second binning operation to output second pixel information; Determining a high illuminance state and a low illuminance state based on a preset reference value and generating illuminance information including high illuminance information and low illuminance information; Selectively outputting the first pixel information or the second pixel information according to the illuminance information generated in the step of generating the illuminance information; Including The first binning operation is an operation of performing EDI (Edge Detection Interpolation) binning based on edge information, and the second binning operation is an operation of generating an average value obtained by downscaling a plurality of pixels formed in a Bayer format through 4-sum binning. The EDI binning is an operation of advancing interpolation within a plane based on the edge information and performing binning based on the interpolated data. In the step of selectively outputting, a method of driving an image sensing device that outputs the first pixel information when low illuminance information is provided and outputs second pixel information when high illuminance information is provided.

15. The step of outputting the first pixel information includes: generating edge information from a plurality of pixel data output from the pixel array; assigning weights according to the edge information; performing EDI (Edge Detection Interpolation) binning based on the edge information according to the assigned weights to generate a binning value; A method of driving the image sensing device according to claim 14, including the above steps.

Images