Image sensor, imaging system, and image sensor control method

Abstract

The present invention prevents loss of information in an image sensor that compares signals.　The image sensor is equipped with a level control circuit, a comparison unit, and a logic circuit. The level control circuit increases or decreases one of a pair of pixel signals by one of a plurality of gains. In addition, the comparison unit outputs a comparison result obtained by comparing the pair of pixel signals, one of which has been increased or decreased. In addition, the logic circuit generates compressed data by reversibly compressing a bit string in which comparison results corresponding to the plurality of gains are arrayed.

Description

Image sensor, imaging system, and method for controlling image sensor

[0001] The present technology relates to an image sensor, and more particularly to an image sensor that compares signals using a comparator, an imaging system, and a method for controlling the image sensor.

[0002] Conventionally, image sensors and the like have performed various signal processing operations, such as pixel addition and CDS (Correlated Double Sampling) processing, for purposes such as expanding the dynamic range and reducing noise. For example, a device has been proposed that increases or decreases one of a pair of pixel signals using a gain and detects the presence or absence of an edge based on the comparison result (see, for example, Patent Document 1). A comparison circuit in this device outputs four bits based on the comparison result for each window in which multiple pixels are arranged, and an edge determination circuit lossily compresses the four bits to two bits and outputs the edge detection result.

[0003] International Publication No. 2021 / 090538

[0004] The conventional technology described above attempts to achieve edge detection that is independent of the amount of ambient light by increasing or decreasing one of a pair of pixel signals using a gain. However, the edge determination circuit lossily compresses the output of the comparison circuit, which results in the loss of information such as edge direction.

[0005] This technology was developed in light of these circumstances, and aims to prevent information loss in image sensors that compare signals.

[0006] The present technology has been made to solve the above-mentioned problems, and a first aspect thereof is an image sensor and a control method thereof, including: a level control circuit that increases or decreases one of a pair of pixel signals by one of a plurality of gains; a comparator that compares the pair of pixel signals, one of which has been increased or decreased, and outputs a comparison result; and a logic circuit that losslessly compresses a bit string in which the comparison results corresponding to each of the plurality of gains are arranged to generate compressed data, thereby preventing information loss.

[0007] In addition, in the first aspect, a timing control circuit may be further provided that switches between ascending and descending order of gains to be set in the level control circuit from among the plurality of gains, thereby enabling compression into fixed-length compressed data.

[0008] In this first aspect, the compressed data may include fixed-length compressed data, thereby providing an effect that the fixed-length compressed data is output.

[0009] In addition, in this first aspect, the bit string may include a predetermined number of pieces of channel information, and where N is the number of the plurality of gains, the data size of the channel information is N bits, and the logic circuit may losslessly compress each piece of the channel information into F-bit data, where F is the value obtained by truncating the logarithm of (N+1) to base 2. This brings about the effect of compressing N-bit data into F-bits.

[0010] In this first aspect, the compressed data may include variable-length compressed data, thereby improving the compression rate.

[0011] In the first aspect, the variable-length compressed data may include an entropy code, thereby providing an effect that the entropy code is output.

[0012] In addition, in this first aspect, the variable-length compressed data may include gain-direction compressed data compressed based on the gain direction, thereby providing an effect that the gain-direction compressed data is output.

[0013] In addition, in this first aspect, the gain direction compressed data may include a header portion indicating the presence or absence of an edge for each of the gain directions and a data portion indicating the strength of the edge, thereby providing an effect that the header portion and the data portion are output.

[0014] Furthermore, in this first aspect, if the number of gain directions is 2×M, the data size of the header section is 2×M bits, and if the number of the plurality of gains is N and the value obtained by truncating the logarithm of (N+1) to base 2 is F, the maximum data size of the data section may be M×F bits. This brings about the effect that the fixed-length compressed data is compressed from 2×M to 2×M+M×F bits.

[0015] In this first aspect, the logic circuit may further include a compression unit that compresses the bit strings of each of a plurality of pixels in the edge image to generate the compressed data, and a data output unit that outputs addresses of pixels in which edges are detected among the plurality of pixels and the compressed data corresponding to the addresses, thereby providing the effect of efficiently mapping data.

[0016] In addition, in the first aspect, the pixel array unit may further include a pair of shared blocks, each of which includes an array of a plurality of pixels that share the floating diffusion layer, thereby reducing the number of elements per pixel.

[0017] In addition, in this first aspect, one of the pair of pixel signals may be a pixel signal of a pixel summation obtained by summing a plurality of pixels arranged in one of the pair of sharing blocks, and the other of the pair of pixel signals may be a pixel signal of a pixel summation obtained by summing a plurality of pixels arranged in the other of the pair of sharing blocks. This brings about an effect that edges can be detected even in a low-illumination environment.

[0018] According to a second aspect of the present technology, there is provided an imaging system including an image sensor including: a level control circuit that increases or decreases one of a pair of pixel signals by one of a plurality of gains; a comparator that compares the pair of pixel signals, one of which has been increased or decreased, and outputs a comparison result; and a logic circuit that generates compressed data by losslessly compressing a bit string in which the comparison results corresponding to each of the plurality of gains are arranged; and a decompressor that decompresses the compressed data. This provides an effect of preventing information loss in the imaging system.

[0019] 1 is a block diagram showing an example of a configuration of an imaging device according to a first embodiment of the present technology. FIG. 2 is a block diagram showing an example of a configuration of an image sensor according to the first embodiment of the present technology. FIG. 3 is a circuit diagram showing an example of a configuration of a pixel according to the first embodiment of the present technology. FIG. 4 is a block diagram showing an example of a configuration of a column signal processing circuit according to the first embodiment of the present technology. FIG. 5 is a block diagram showing an example of a configuration of a load MOS current source circuit and a column ADC according to the first embodiment of the present technology. FIG. 6 is a block diagram showing an example of a configuration of an edge detection circuit according to the first embodiment of the present technology. FIG. 7 is a block diagram showing an example of a configuration of an edge detection unit according to the first embodiment of the present technology. FIG. 8 is a diagram showing an example of a correspondence between gain directions and comparison results according to the first embodiment of the present technology. FIG. 9 is a circuit diagram showing an example of a gain circuit and a gain offset circuit according to the first embodiment of the present technology. FIG. 10 is a block diagram showing an example of a configuration of a logic circuit according to the first embodiment of the present technology. FIG. 11 is a diagram for explaining buffering of uncompressed data according to the first embodiment of the present technology. FIG. 12 is a diagram showing an example of a correspondence between channel information before compression and channel information after compression according to the first embodiment of the present technology. FIG. 13 is a diagram showing an example of uncompressed data and fixed-length compressed data according to the first embodiment of the present technology. FIG. 14 is a diagram showing an example of a data structure of a frame according to the first embodiment of the present technology. FIG. 1 is a diagram showing an example of a data structure and data type of a packet in the first embodiment of the present technology. FIG. 2 is a diagram showing an example of a packing method for uncompressed data in the first embodiment of the present technology. FIG. 3 is a diagram showing an example of a packing method for fixed-length compressed data in the first embodiment of the present technology. FIG. 4 is a block diagram showing an example of a configuration of an application processor in the first embodiment of the present technology. FIG. 5 is a flowchart showing an example of an operation of an image sensor in the first embodiment of the present technology. FIG. 6 is a block diagram showing an example of a configuration of a logic circuit in the second embodiment of the present technology. FIG. 7 is a block diagram showing an example of a configuration of a gain direction compression unit in the second embodiment of the present technology. FIG. 8 is a diagram for explaining the operation of a data portion generation circuit in the second embodiment of the present technology. FIG. 9 is a diagram showing examples of fixed-length compressed data and gain direction compressed data in the second embodiment of the present technology.10 is a diagram showing an example of a packing method for gain-direction compressed data in a second embodiment of the present technology. FIG. 11 is a flowchart showing an example of a decompression operation of an application processor in the second embodiment of the present technology. FIG. 12 is a block diagram showing an example of a configuration of a logic circuit in a third embodiment of the present technology. FIG. 13 is a diagram showing an example of fixed-length compressed data and an example of a 1- to 7-bit Huffman code in the third embodiment of the present technology. FIG. 14 is a diagram showing an example of a data type added in the third embodiment of the present technology. FIG. 15 is a diagram showing an example of a packing method for Huffman codes in the third embodiment of the present technology. FIG. 16 is a circuit diagram showing an example of a configuration of a pixel array unit in a fourth embodiment of the present technology. FIG. 17 is a circuit diagram showing an example of a configuration of an FD (Floating Diffusion) sharing block in the fourth embodiment of the present technology. FIG. 18 is a diagram showing an example of a pixel pair in a window in a modification of the fourth embodiment of the present technology. FIG. 19 is a diagram showing an example of a pixel pair in a window in a modification of the fourth embodiment of the present technology. FIG. 19 is a diagram showing an example of uncompressed data and fixed-length compressed data in a modification of the fourth embodiment of the present technology. FIG. 19 is a diagram showing an example of fixed-length compressed data and gain-direction compressed data in a modification of the fourth embodiment of the present technology. FIG. 10 is a diagram showing an example of a method for transmitting fixed-length compressed data in a fifth embodiment of the present technology. FIG. 11 is a diagram for explaining a method for receiving fixed-length compressed data in the fifth embodiment of the present technology. FIG. 12 is a diagram for explaining a method for transmitting gain-direction compressed data in the fifth embodiment of the present technology. FIG. 13 is a diagram for explaining a method for receiving gain-direction compressed data in the fifth embodiment of the present technology. FIG. 14 is a diagram showing an example of pixel pairs in a window in a sixth embodiment of the present technology. FIG. 15 is a diagram showing examples of uncompressed data, fixed-length compressed data, and gain-direction compressed data in the sixth embodiment of the present technology. FIG. 16 is a block diagram showing an example of a schematic configuration of a vehicle control system. FIG. 17 is an explanatory diagram showing an example of installation positions of an outside-vehicle information detection unit and an imaging unit.

[0020] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order: 1. First embodiment (example of lossless compression into fixed-length compressed data) 2. Second embodiment (example of lossless compression into gain-direction compressed data) 3. Third embodiment (example of lossless compression into entropy code) 4. Fourth embodiment (example of lossless compression of pixel-added data into fixed-length compressed data) 5. Fifth embodiment (example of lossless compression into fixed-length compressed data and outputting only data of pixels with edges) 6. Sixth embodiment (example of lossless compression of uncompressed data in six or more gain directions) 7. Application example to a moving body

[0021] 1. First Embodiment [Configuration Example of Imaging Device] Fig. 1 is a block diagram showing a configuration example of an imaging device 100 according to a first embodiment of the present technology. The imaging device 100 is a device for capturing image data and includes an optical unit 110, an image sensor 200, and an application processor 120. The imaging device 100 further includes a display unit 130, an operation unit 140, a bus 150, a frame memory 160, a storage unit 170, and a power supply unit 180. The imaging device 100 may be, for example, a digital camera such as a digital still camera, as well as a smartphone, a personal computer, an in-vehicle camera, or the like with an imaging function. Note that a system including the image sensor 200 and the application processor 120 in the imaging device 100 is an example of an imaging system described in the claims.

[0022] The optical unit 110 collects light from a subject and guides it to the image sensor 200. The image sensor 200 generates image data by photoelectric conversion in synchronization with a vertical synchronization signal. Here, the vertical synchronization signal is a periodic signal with a predetermined frequency that indicates the timing of imaging. The image sensor 200 supplies the generated image data as frames to the application processor 120 via a signal line 209.

[0023] The application processor 120 performs predetermined signal processing on frames from the image sensor 200. The application processor 120 outputs the processed frames to a frame memory 160 or the like via a bus 150. Note that instead of the application processor 120, a circuit such as an FPGA (Field-Programmable Gate Array) can also be provided.

[0024] The display unit 130 displays frames. The display unit 130 may be, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The operation unit 140 generates operation signals in accordance with user operations.

[0025] The bus 150 is a common path for the optical unit 110, image sensor 200, application processor 120, display unit 130, operation unit 140, frame memory 160, storage unit 170, and power supply unit 180 to exchange data with one another.

[0026] The frame memory 160 holds frames. The storage unit 170 stores various data such as frames. The power supply unit 180 supplies power to the image sensor 200, the application processor 120, the display unit 130, and the like.

[0027] [Configuration Example of Image Sensor] Fig. 2 is a block diagram showing a configuration example of an image sensor 200 according to the first embodiment of the present technology. The image sensor 200 includes a vertical scanning circuit 210, a DAC (Digital to Analog Converter) 220, a timing control circuit 230, and a pixel array unit 240. The image sensor 200 further includes a column signal processing circuit 300, a horizontal scanning circuit 270, and a logic circuit 280. These circuits are arranged on, for example, a single semiconductor chip. The circuits illustrated in Fig. 2 can also be distributed and arranged on multiple stacked semiconductor substrates.

[0028] In the pixel array section 240, a plurality of pixels 250 are arranged in a two-dimensional lattice pattern.

[0029] The timing control circuit 230 controls the operation timing of the vertical scanning circuit 210, the column signal processing circuit 300, etc. in synchronization with the vertical synchronization signal.

[0030] The timing control circuit 230 also receives a mode signal MODE that indicates the operation mode of the image sensor 200. The operation modes include an edge detection mode that detects the presence or absence of edges, and an imaging mode that only captures image data (in other words, frames) without detecting the presence or absence of edges. The timing control circuit 230 controls each circuit based on the operation mode. Note that the image sensor 200 can also capture frames while detecting the presence or absence of edges.

[0031] The vertical scanning circuit 210 sequentially selects and drives rows to output analog pixel signals. The DAC 220 generates a reference signal by DA (Digital to Analog) conversion and supplies it to the column signal processing circuit 300. For example, a sawtooth ramp signal is used as the reference signal. The pixels 250 generate pixel signals by photoelectric conversion under the control of the vertical scanning circuit 210. Each pixel 250 outputs a pixel signal to the column signal processing circuit 300.

[0032] The column signal processing circuit 300 performs signal processing such as CDS (Correlated Double Sampling) processing and AD (Analog to Digital) conversion processing on pixel signals for each column. The column signal processing circuit 300 supplies image data (frames) made up of digital signals after signal processing to the logic circuit 280 under the control of the horizontal scanning circuit 270.

[0033] The horizontal scanning circuit 270 selects columns in order and causes the column signal processing circuit 300 to output digital signals in order.

[0034] The logic circuit 280 performs various signal processing on frames as necessary. In the imaging mode, the logic circuit 280 performs image processing such as defect correction, and supplies the processed frames to the application processor 120. In the edge detection mode, the logic circuit 280 performs processing such as compression on data from the column signal processing circuit 300. The logic circuit 280 then supplies the processed data to the application processor 120 in packets.

[0035] For example, an interface conforming to the MIPI (Mobile Industry Processor Interface) standard is used as the communication interface between the image sensor 200 and the application processor 120. Note that the communication interface is not limited to that conforming to the MIPI standard.

[0036] 3 is a circuit diagram showing a configuration example of a pixel 250 according to the first embodiment of the present technology. In the pixel array section 240, nine lines, namely, a reset line 219-R, transfer lines 219-Ta, 219-Tb, 219-Tc, and 219-Td, and selection lines 219-Sa, 219-Sb, 219-Sc, and 219-Sd, are wired in the horizontal direction for each row.

[0037] The reset line 219-R transmits a reset signal RST from the vertical scanning circuit 210. The transfer lines 219-Ta, 219-Tb, 219-Tc, and 219-Td transmit transfer signals TRGa, TRGb, TRGc, and TRGd from the vertical scanning circuit 210. The selection lines 219-Sa, 219-Sb, 219-Sc, and 219-Sd transmit selection signals SELa, SELb, SELc, and SELd from the vertical scanning circuit 210.

[0038] The pixels 250 in 4k columns (k is an integer equal to or greater than 0) are connected to a reset line 219-R, a transfer line 219-Ta, and a selection line 219-Sa. The pixels 250 in 4k+1 columns are connected to a reset line 219-R, a transfer line 219-Tb, and a selection line 219-Sb. The pixels 250 in 4k+2 columns are connected to a reset line 219-R, a transfer line 219-Tc, and a selection line 219-Sc. The pixels 250 in 4k+3 columns are connected to a reset line 219-R, a transfer line 219-Td, and a selection line 219-Sd.

[0039] The pixel 250 includes a photoelectric conversion element 251, a transfer transistor 252, a reset transistor 253, a floating diffusion layer 254, an amplification transistor 255, and a selection transistor 256. The pixel 250 illustrated in the figure is connected to a transfer line 219-Ta and a selection line 219-Sa.

[0040] For example, nMOS (n-channel Metal Oxide Semiconductor) transistors are used as various transistors in the pixel 250, such as the transfer transistor 252.

[0041] The photoelectric conversion element 251 generates electric charges by photoelectric conversion. The transfer transistor 252 transfers the electric charges from the photoelectric conversion element 251 to the floating diffusion layer 254 in accordance with a transfer signal TRGa from the vertical scanning circuit 210.

[0042] The reset transistor 253 extracts electric charges from at least one of the photoelectric conversion element 251 and the floating diffusion layer 254 in accordance with a reset signal RST from the vertical scanning circuit 210 to initialize the same.

[0043] Furthermore, a vertical signal line VSL is wired for each column within the pixel array section 240. The vertical signal line VSL for the n-th column will hereinafter be referred to as VSLn.

[0044] The floating diffusion layer 254 accumulates electric charges and converts the amount of electric charges into a voltage. The amplification transistor 255 forms a source follower circuit and outputs a voltage corresponding to the voltage of the floating diffusion layer 254 to the selection transistor 256. The selection transistor 256 supplies a pixel signal of a voltage from the amplification transistor 255 to a vertical signal line VSL in accordance with a selection signal SELa from the vertical scanning circuit 210.

[0045] 4 is a block diagram showing an example of the configuration of a column signal processing circuit 300 according to the first embodiment of the present technology. The column signal processing circuit 300 includes a load MOS current source circuit 310, a column ADC (Analog to Digital Converter) 320, and an edge detection circuit 330. Details of each circuit will be described later.

[0046] [Configuration Example of Column ADC] FIG. 5 is a block diagram showing a configuration example of the load MOS current source circuit 310 and the column ADC 320 according to the first embodiment of the present technology.

[0047] A load MOS current source 311 is arranged for each vertical signal line VSL in the load MOS current source circuit 310. The load MOS current source 311 supplies a load MOS current to the corresponding vertical signal line VSL.

[0048] An ADC 321 is arranged for each vertical signal line VSL in the column ADC 320. The ADC 321 is connected to the corresponding vertical signal line VSL and the DAC 220, and receives the pixel signal ΔVSL from the pixel and the ramp signal RMP from the DAC 220 as input.

[0049] The ADC 321 converts the analog pixel signal of the corresponding column into a digital signal under the control of the timing control circuit 230. The ADC 321 supplies the digital signal DOUT to the logic circuit 280 under the control of the horizontal scanning circuit 270. For example, a single-slope ADC including a comparator and a counter is used as the ADC 321. Note that a successive approximation ADC including a comparator, a successive approximation logic circuit, and a register can also be used.

[0050] 6 is a block diagram showing an example of the configuration of the edge detection circuit 330 according to the first embodiment of the present technology. Here, in the edge detection mode, the vertical scanning circuit 210 drives two pixel pairs within a window to output pixel signals. This window is a rectangular area including two pixel pairs that are targets for edge detection, and each pixel within the pixel pair is arranged along the periphery of the window.

[0051] Furthermore, the line segment connecting one of the two pixel pairs intersects with the line segment connecting the other. In the figure, the line segments with arrows at both ends are line segments connecting pixel pairs. Furthermore, it is preferable that the angle between these line segments and the line (row or column) on which the pixels are arranged is not 0 degrees, 45 degrees, or 90 degrees. By setting the angle between the line segment and the line to a value that is not 0 degrees, 45 degrees, or 90 degrees, it is possible to prevent two pixels to be detected from being connected to a single vertical signal line VSL.

[0052] The size of the window is, for example, 4 rows x 4 columns. To satisfy the angle condition between the line segment and the line, for example, a pixel pair consisting of a pixel at address (1, 0) and a pixel at address (2, 3) and a pixel pair consisting of a pixel at address (3, 1) and a pixel at address (0, 2) are driven as targets for edge detection.

[0053] The vertical scanning circuit 210 simultaneously drives multiple windows arranged in the row direction. The distance between the representative positions (such as the upper left pixel) of two adjacent windows is called the "stride." In the figure, the area surrounded by a thick line corresponds to a window, and the area surrounded by a dashed line corresponds to a window adjacent to the thick-lined window. As illustrated in the figure, the stride is four pixels.

[0054] The edge detection circuit 330 includes an edge detection unit 340 for each pixel pair, which detects the edge of that pixel pair. Each edge detection unit 340 is connected to a pair of vertical signal lines VSL corresponding to the pixel pair to be detected. For example, an edge detection unit 340 that detects a pixel pair consisting of a pixel with address (3, 1) and a pixel with address (0, 2) is connected to vertical signal lines VSL1 and VSL2. An edge detection unit 340 that detects a pixel pair consisting of a pixel with address (1, 0) and a pixel with address (2, 3) is connected to vertical signal lines VSL0 and VSL3.

[0055] Each edge detection unit 340 outputs two comparison results VCO to the logic circuit 280 based on pixel signals from the pixel pair being detected. Since each window includes two pixel pairs, four comparison results are output for each window. These are referred to as comparison results VCO0, VCO1, VCO2, and VCO3.

[0056] 7 is a block diagram showing an example of the configuration of the edge detection unit 340 according to the first embodiment of the present technology. The edge detection unit 340 includes a level control circuit 350 and a comparison circuit 380.

[0057] Here, the voltage of the vertical signal line VSL when the pixel 250 is initialized is called a reset level or P phase. The voltage of the vertical signal line VSL when the charge in the pixel 250 is transferred is called a signal level or D phase. The difference between the P phase and the D phase corresponds to a net pixel signal ΔVSL excluding the noise component at the time of initialization.

[0058] The level control circuit 350 includes gain circuits 360-1 and 360-2, and gain / offset circuits 370-1 and 370-2. The comparison circuit 380 includes comparators 381 and 382.

[0059] The gain circuit 360-1 is connected to the vertical signal line VSL1. The gain circuit 360-1 applies a predetermined gain g １ The signal is increased or decreased by the input voltage Vout and supplied to the inverting input terminal (-) of the comparator 381.

[0060] The gain circuit 360-2 is connected to the vertical signal line VSL2. The gain circuit 360-2 converts the pixel signal ΔVSL2 into a signal g １ Gain g different from ２ The signal is increased or decreased by the value g and supplied to the inverting input terminal (-) of the comparator 382. １ and g ２ is a real number.

[0061] The gain / offset circuit 370-1 is connected to the vertical signal line VSL2. This gain / offset circuit 370-1 applies a gain g １ The voltage is increased or decreased by , and a predetermined offset voltage C is applied as needed, and supplied to the non-inverting input terminal (+) of the comparator 381.

[0062] The gain / offset circuit 370-2 is connected to the vertical signal line VSL1. The gain / offset circuit 370-2 applies a gain g ２ The voltage is increased or decreased by , and an offset voltage C is applied as needed, and supplied to the non-inverting input terminal (+) of the comparator 382.

[0063] g １ / g ２ is the relative gain G, the above-described configuration allows the level control circuit 350 to increase or decrease one of the pair of pixel signals ΔVSL1 and ΔVSL2 using the relative gain G, and to apply an offset voltage C to the other pixel signal as needed. Specifically, the level control circuit 350 increases or decreases the pixel signal ΔVSL1 using the relative gain G, and applies the offset voltage C to the pixel signal ΔVSL2 as needed, and supplies the pixel signal to the comparator 381. The level control circuit 350 also increases or decreases the pixel signal ΔVSL2 using the relative gain G, and applies the offset voltage C to the pixel signal ΔVSL1 as needed, and supplies the pixel signal to the comparator 382.

[0064] The comparator 381 compares the pair of pixel signals after the increase or decrease, and supplies the comparison result VCO0 to the logic circuit 280. The comparator 382 compares the pair of pixel signals after the increase or decrease, and supplies the comparison result VCO1 to the logic circuit 280.

[0065] The edge detection unit 340 connected to the vertical signal lines VSL0 and VSL3 also generates comparison results VCO2 and VCO3 using a circuit similar to that described above, and supplies them to the logic circuit 280.

[0066] Therefore, four bits consisting of the comparison results VCO0, VCO1, VCO2 and VCO3 are output to the logic circuit 280 for each window.

[0067] Although the level control circuit 350 performs both the increase / decrease by gain and the application of an offset voltage, it is also possible to perform only the increase / decrease by gain without applying an offset voltage.

[0068] The timing control circuit 230 (not shown) switches between the multiple gains to be set in the level control circuit 350 in ascending or descending order. For example, when the gain is switched in N (N is an integer) steps, 4 bits that arrange the comparison result VCO are output for each gain, and therefore a bit string of 4 × N bits is output to the logic circuit 280 for each window. This bit string is called "uncompressed data."

[0069] The edge detection mode described above includes a non-compression mode and a compression mode. In the non-compression mode, the logic circuit 280 outputs the non-compressed data of each window in packets. On the other hand, in the compression mode, the logic circuit 280 losslessly compresses the non-compression mode for each window to generate compressed data, and outputs the compressed data in packets. Details of the compression process will be described later.

[0070] Next, an example of the values ​​of the comparison results VCO0 and VCO1 actually output in the circuit of the same figure will be described. For simplicity of explanation, it is assumed that the offset voltage C is not applied.

[0071] For example, it is assumed that a low-level comparison result VCO0 is output when the following equation is satisfied: ΔVSL2<ΔVSL1*G Equation 1 In the above equation, "*" indicates multiplication.

[0072] On the other hand, it is assumed that a high-level comparison result VCO0 is output when the following equation is satisfied: ΔVSL2>ΔVSL1*G (Equation 2)

[0073] Also, it is assumed that a high level comparison result VCO1 is output when the following equation is satisfied: ΔVSL1>ΔVSL2*G (Equation 3)

[0074] On the other hand, it is assumed that a low-level comparison result VCO1 is output when the following equation is satisfied: ΔVSL1≦ΔVSL2*G (Equation 4)

[0075] When both Equation 1 and Equation 3 are true, the following equation is true: ΔVSL1<<ΔVSL2 Equation 5 In the above equation, "<<" indicates the magnitude relationship of the degree to which an edge is determined to exist.

[0076] On the other hand, when both Equation 2 and Equation 4 are true, the following equation is true: ΔVSL1>>ΔVSL2 Equation 6 In the above equation, “>>” indicates the magnitude relationship of the degree to which an edge is determined to exist.

[0077] When the comparison results VCO0 and VCO1 have different values, either Equation 5 or Equation 6 holds. These Equations 5 and 6 indicate that the ratio of one of the pixel signals ΔVSL0 and ΔVSL1 to the other is equal to or greater than a certain value, indicating that there is an edge in the corresponding pixel pair. On the other hand, when the comparison results VCO0 and VCO1 have the same value, neither Equation 5 nor Equation 6 holds, indicating that there is no edge in the pixel pair.

[0078] Next, the advantages of the control in which one of the pair of pixel signals ΔVSL1 and ΔVSL2 is increased or decreased by the relative gain G and compared will be described.

[0079] The pixel that generates the pixel signal ΔVSL1 receives light that is ambient light reflected from the surface of object A with reflectance Ra, and the pixel that generates the pixel signal ΔVSL2 receives light that is ambient light reflected from the surface of object B with reflectance Rb. The amount of ambient light is AL, and the relative gain G is less than 1.

[0080] Here, a first comparative example is considered in which the difference between pixel signals ΔVSL1 and ΔVSL2 is compared with a threshold value Th to detect the presence or absence of an edge. In this first comparative example, it is determined that an edge exists when the following formula is satisfied: ΔVSL1-ΔVSL2=Ra*AL-Rb*AL>Th (Formula 7)

[0081] Equation 7 can be transformed into the following equation: (Ra-Rb)>Th / AL Equation 8

[0082] As exemplified by Equations 7 and 8, in the first comparative example, the edge determination result may change depending on the amount of ambient light AL. This reduces the accuracy of edge detection. For example, the smaller the amount of ambient light AL, the smaller the difference, making edge detection more difficult. Changing the threshold value depending on the amount of ambient light can suppress the reduction in detection accuracy, but it is difficult to accurately measure the amount of light and fine-tune the threshold value accordingly.

[0083] Also, consider a second comparative example in which the difference between the logarithmic values ​​of the pixel signals ΔVSL1 and ΔVSL2 is compared with a threshold value Th to detect the presence or absence of an edge. In this second comparative example, it is determined that an edge exists when the following equation is satisfied: log(ΔVSL1) - log(ΔVSL2) = log(Ra * AL / Rb * AL) = log(Ra / Rb) > Th (Equation 9)

[0084] As shown in Equation 9, the second comparative example can suppress the influence of ambient light, but requires logarithmic calculations and is difficult to implement.

[0085] In contrast to these comparative examples, by introducing a method of determining a threshold value based on a signal rather than based on ambient light, the influence of ambient light can be easily suppressed.

[0086] Whether only pixel signal ΔVSL2 is attenuated by relative gain G or only pixel signal ΔVSL1 is attenuated by relative gain G, the magnitude relationship between the attenuated pixel signals remains unchanged. In this case, the comparison results VCO1 and VCO2 will have different values ​​due to the connection relationship of the input terminals of comparators 381 and 382. In this case, the following equation holds: |Ra*AL-Rb*AL|>G*max(Ra, Rb)*AL ...Equation 10 In the above equation, max() is a function that returns the maximum value of the multiple input values ​​in (). Note that if the relative gain G is greater than 1, the function min(), which returns the minimum value, is used instead of max().

[0087] When Ra is greater than Rb, Equation 10 can be transformed into the following equation: (Ra-Rb)*AL>G*Ra*AL Equation 11

[0088] Equation 11 can be transformed into the following equation: (1-G)*Ra*AL-Rb*AL>0 Equation 12

[0089] Furthermore, if 1-G is g, then Equation 12 can be transformed into the following equation: g*(Ra*AL)-(Rb*AL)>0 Equation 13

[0090] Dividing both sides of Equation 13 by the amount of light AL gives the following equation: g*Ra-Rb>0 Equation 14

[0091] That is, when the comparison results VCO0 and VCO1 are different values, Equation 14 holds. Equation 14 indicates that the ratio between pixel signals ΔVSL1 and ΔVSL2 is equal to or greater than a certain value (g in the above equation). In other words, the comparison results VCO0 and VCO1 are values ​​indicating that the ratio between pixel signals ΔVSL1 and ΔVSL2 is equal to or greater than a certain value. Because Equation 14 does not include a term that includes the light amount AL, the image sensor 200 can determine the presence or absence of an edge without depending on the light amount AL. This allows for improved edge detection accuracy compared to the first comparative example.

[0092] When the illuminance is low and ΔVSL1 and ΔVSL2 are both low, noise can cause the comparison results VCO0 and VCO2 to have different values, resulting in an erroneous edge detection. The edge detection unit 340 can suppress such erroneous edge detection by applying the aforementioned offset voltage C.

[0093] 8 is a diagram illustrating an example of a correspondence relationship between a gain direction and a comparison result according to the first embodiment of the present technology. Here, the “gain direction” refers to a direction from one pixel to the other pixel of a pixel pair for which an edge is to be detected.

[0094] In the figure, "a" indicates the gain direction corresponding to the comparison result VCO0. According to Equations 1 and 2, the pixel signal at address (3, 1) connected to vertical signal line VSL1 is increased or decreased by relative gain G. The gain direction in this case is from address (0, 2) to address (3, 1).

[0095] In the figure, b indicates the gain direction corresponding to the comparison result VCO1. According to Equations 3 and 4, the pixel signal at address (0, 2) connected to vertical signal line VSL2 is increased or decreased by relative gain G. The gain direction in this case is from address (3, 1) to address (0, 2).

[0096] In the figure, c indicates the gain direction corresponding to the comparison result VCO2. The pixel signal at address (2, 3) connected to vertical signal line VSL3 is increased or decreased by relative gain G. The gain direction in this case is from address (1, 0) to address (2, 3).

[0097] In the figure, d indicates the gain direction corresponding to the comparison result VCO3. The pixel signal at address (1, 0) connected to vertical signal line VSL0 is increased or decreased by relative gain G. The gain direction in this case is from address (2, 3) to address (1, 0).

[0098] 9 is a circuit diagram showing an example configuration of a gain circuit 360-1 and a gain / offset circuit 370-1 according to the first embodiment of the present technology. The gain circuit 360-1 includes selectors 361, 362, 363, and 364, and capacitors 365, 366, 367, and 368. The gain / offset circuit 370-1 includes selectors 371, 372, 373, and 374, and capacitors 375, 376, 377, and 378. The circuit configurations of the gain circuit 360-2 and the gain / offset circuit 370-2 are similar to those of the gain circuit 360-1 and the gain / offset circuit 370-1.

[0099] The gain circuit 360-1 receives a 4-bit control signal diff2 from the timing control circuit 230. Within the gain circuit 360-1, a selector 361 switches the connection destination of one end of a capacitor 365 between a ground terminal and a vertical signal line VSL1 in accordance with the first bit of the control signal diff2. A selector 362 switches the connection destination of one end of a capacitor 366 between a ground terminal and a vertical signal line VSL1 in accordance with the second bit of the control signal diff2.

[0100] The selector 363 switches the connection destination of one end of the capacitor 367 between the ground terminal and the vertical signal line VSL1 in accordance with the third bit of the control signal diff2. The selector 364 switches the connection destination of one end of the capacitor 368 between the ground terminal and the vertical signal line VSL1 in accordance with the fourth bit of the control signal diff2.

[0101] The other ends of the capacitors 365 to 368 are commonly connected to the inverting input terminal (−) of a comparator 381 .

[0102] The gain / offset circuit 370-1 receives a 4-bit control signal diff1 and an offset voltage C from the timing control circuit 230. Within the gain / offset circuit 370-1, a selector 371 switches the connection destination of one end of a capacitor 375 between the offset voltage C and the vertical signal line VSL2 in accordance with the first bit of the control signal diff1. A selector 372 switches the connection destination of one end of a capacitor 376 between the ground terminal and the vertical signal line VSL2 in accordance with the second bit of the control signal diff1.

[0103] The selector 373 switches the connection destination of one end of the capacitor 377 between the ground terminal and the vertical signal line VSL2 in accordance with the third bit of the control signal diff1. The selector 374 switches the connection destination of one end of the capacitor 378 between the ground terminal and the vertical signal line VSL2 in accordance with the fourth bit of the control signal diff1.

[0104] The other ends of the capacitors 375 to 378 are commonly connected to the non-inverting input terminal (+) of a comparator 381 .

[0105] With the circuit configuration illustrated in the figure, the timing control circuit 230 can control the gain g1 in 16 steps using the control signal diff1. Furthermore, the timing control circuit 230 can control whether or not to apply the offset voltage C and can control the gain g2 in 8 steps using the control signal diff2.

[0106] 10 is a block diagram showing an example of the configuration of the logic circuit 280 according to the first embodiment of the present technology. The logic circuit 280 includes a data buffer 281, a fixed-length compression unit 282, an image processing unit 283, and a data output unit 284.

[0107] The data buffer 281 stores uncompressed data. The edge detection circuit 330 outputs four bits of comparison results VCO0 to VCO3 for each window. The timing control circuit 230 (not shown) sequentially switches between N gain levels in ascending or descending order. The data buffer 281 stores four bits of comparison results VCO0 to VCO3 for each gain for each window. For example, if N is 3, a total of 12 (= 4 × 3) bits of uncompressed data are stored for each window. Of these, three bits with the same comparison result VCO (in other words, gain direction) are referred to as "channel information." The channel information corresponding to the comparison results VCO0, VCO1, VCO2, and VCO3 are designated CH0, CH1, CH2, and CH3.

[0108] In compression mode, the fixed-length compressor 282 losslessly compresses uncompressed data consisting of channel information CH0, CH1, CH2, and CH3 for each window into fixed-length compressed data. For example, each 3-bit piece of channel information CH is compressed into 2-bit data. The compressed channel information is assumed to be ch0, ch1, ch2, and ch3. The fixed-length compressor 282 then supplies 8-bit fixed-length compressed data consisting of channel information ch0, ch1, ch2, and ch3 to the data output unit 284 for each window.

[0109] In the imaging mode, the image processing unit 283 performs various image processing such as white balance correction and demosaic processing on a frame in which the digital signal DOUT from the column ADC 320 is arranged. The image processing unit 283 supplies the processed frame to the data output unit 284.

[0110] The data output unit 284 outputs frames in packet units to the application processor 120. In the imaging mode, the data output unit 284 outputs frames from the image processing unit 283.

[0111] In addition, in the uncompressed mode, the data output unit 284 reads uncompressed data for each window from the data buffer 281. Each piece of uncompressed data is assigned an XY address on the edge image corresponding to the position of the window, and the data obtained by arranging this uncompressed data corresponds to a frame of the edge image. The data output unit 284 outputs this frame of the edge image. In the compressed mode, the data output unit 284 outputs a frame of the edge image obtained by arranging fixed-length compressed data.

[0112] 11 is a diagram illustrating buffering of uncompressed data according to the first embodiment of the present technology. For example, assume that a relative gain G of 0.25 is initially set. At this time, as illustrated in FIG. 11A, the data buffer 281 stores the comparison results VCO0, VCO1, VCO2, and VCO3 as the 0th bit of the channel information CH0, CH1, CH2, and CH3.

[0113] Next, when the relative gain G is set to 0.50, as shown in FIG. 11B, the data buffer 281 holds the comparison results VCO0, VCO1, VCO2 and VCO3 as the first bit of the channel information CH0, CH1, CH2 and CH3.

[0114] When a relative gain G of 0.25 is set, the data buffer 281 stores the comparison results VCO0, VCO1, VCO2, and VCO3 as the second bit of the channel information CH0, CH1, CH2, and CH3, as shown in Fig. 16c, thereby storing 12 bits of uncompressed data for each window.

[0115] In the figure, the timing control circuit 230 switches the relative gain in ascending order, but it can also switch in descending order.

[0116] 12 is a diagram showing an example of the correspondence relationship between the channel information CH before compression and the channel information ch after compression in the first embodiment of the present technology. According to Equation 1 and Equation 2, the higher the relative gain G, the more likely the comparison result VCO becomes high level. For example, when the gain is switched in the order of 0.25, 0.5, 0.75 (or in the reverse order), the second bit of the channel information CH indicates whether the following equation is established: ΔVSL2>ΔVSL1*0.75 ...Equation 15

[0117] The first bit of the channel information CH indicates whether the following formula is true: ΔVSL2>ΔVSL1*0.5 (Formula 16)

[0118] The 0th bit of the channel information CH indicates whether the following formula is true: ΔVSL2>ΔVSL1*0.25 (Formula 17)

[0119] When formula 15 holds, there exists a combination of ΔVSL1 and ΔVSL2 such that formula 16 also holds, but there does not exist a combination such that formula 16 or formula 17 holds even though formula 15 does not hold. Similarly, when formula 15 and formula 16 hold, formula 17 can also hold, but formula 17 does not hold even though formula 16 does not hold.

[0120] Therefore, when the relative gain G is switched between ascending and descending order, the possible values ​​of the 3 bits of the channel information CH are limited to four binary values: (000), (100), (110), and (111). By allocating 2 bits to each of these combinations, it is possible to compress the channel information to 2 bits. This compression is lossless compression because there is no loss of information.

[0121] For example, if the channel information CH is (000), it is compressed to two bits (00). If the channel information CH is (100), it is compressed to two bits (01). If the channel information CH is (110), it is compressed to two bits (10). If the channel information CH is (111), it is compressed to two bits (11).

[0122] To generalize, when the relative gain G has N levels, the data size of the channel information CH before compression is N bits. The combinations of the channel information CH are limited to N+1 ways. Therefore, if the data size of the channel information ch after compression is F bits (F is an integer), F can be expressed by the following formula: F=f or {log ２ (N+1)} ...Equation 18 In the above equation, fl or () is a function that returns the argument by rounding down the fraction, and log ２ () is a function that returns the logarithm to the base 2.

[0123] As a comparative example, consider a configuration described in Patent Document 1, in which an OR operation is performed every two bits to compress four bits to two bits for each window. This compression is lossy compression, in which information such as the gain direction is lost. This loss of information can degrade the image quality of the edge image, which can degrade performance such as the recognition accuracy of image recognition.

[0124] In contrast, the logic circuit 280 performs lossless compression for each channel information as described above, eliminating information loss and improving the image quality of edge images compared to the comparative example, thereby improving performance such as the recognition accuracy of image recognition.

[0125] 13 is a diagram illustrating an example of uncompressed data and fixed-length compressed data according to the first embodiment of the present technology. When the gain is switched between three levels, the data buffer 281 holds 12 bits of uncompressed data including channel information CH0 to CH3 for each window.

[0126] The fixed-length compression unit 282 then reads channel information CH0 to CH3 from the data buffer 281 and compresses each piece of channel information to 2 bits. For example, channel information CH0 of (100) is compressed to (01), and channel information CH1 of (000) is compressed to (00). Channel information CH2 of (110) is compressed to (10), and channel information CH3 of (000) is compressed to (00). These 8-bit fixed-length compressed data are then output.

[0127] Fig. 14 is a diagram showing an example of a data structure of a frame in the first embodiment of the present technology. In the figure, "a" indicates a data structure in the MIPI standard. In the figure, "a" indicates "Frame Start", and "PH" indicates "Packet Header". "ED" indicates "Embedded Data", and "PF" indicates "Packet Header". Information such as packet size, gain, and offset can be stored in "ED". "PD" indicates "Payload Data", and "FE" indicates "Frame End".

[0128] The PH stores information about whether the stored data is compressed or not, and the compression method.

[0129] In the figure, b shows an example of a packet transmission procedure in accordance with the MIPI standard. SoT in b in the figure indicates "Start of Transmission," EoT indicates "End of Transmission," and LPS indicates "Low Power State."

[0130] 15 is a diagram showing an example of a data structure and data type of a packet according to the first embodiment of the present technology. In the figure, "a" shows an example of a data structure of a long packet in accordance with the MIPI standard, "b" shows an example of a data structure of a short packet in accordance with the MIPI standard, and "c" shows an example of a data type in accordance with the MIPI standard.

[0131] In the figure, DI in a indicates "Data ID" and WC indicates "Word Count." DI stores the data type. ECC indicates "Error detection and Correction Code."

[0132] In FIG. 1B, FN indicates "Frame Number" and LN indicates "Lane Number."

[0133] FIG. 16 is a diagram showing an example of a packing method for uncompressed data in the first embodiment of the present technology. In uncompressed mode, pixels on an edge image are assigned to each window. Payload data is transmitted in long packets, and data is packed in 1-byte units. Therefore, the data output unit 284 packs 24 (= 12 × 2) bits of data for every two pixels into a 3-byte PD for transmission. How data is packed into the 3 bytes can be set arbitrarily. Two packing methods are shown here.

[0134] For example, as illustrated in FIG. 11A, the data output unit 284 stores bits 11 to 4 of uncompressed data P1 for pixel 1 in the first byte of PD. The data output unit 284 also stores bits 11 to 4 of uncompressed data P2 for pixel 2 in the second byte of PD. The data output unit 284 then stores bits 3 to 0 of uncompressed data P2 and bits 3 to 0 of uncompressed data P1 in the third byte of PD.

[0135] Alternatively, as shown in FIG. 1B, the data output unit 284 stores bits 11 through 4 of the uncompressed data P1 in the first byte of PD. The data output unit 284 also stores bits 3 through 0 of the uncompressed data P1 and bits 11 through 8 of the uncompressed data P2 in the second byte of PD. The data output unit 284 then stores bits 7 through 0 of the uncompressed data P2 in the third byte of PD.

[0136] It is not necessary to pack every two pixels, but it is sufficient to pack every number of pixels that is a multiple of two, and the data may span multiple packets.

[0137] The data output unit 284 also stores and outputs one line of uncompressed data in one packet from PH to PF. The data output unit 284 can also store and output data up to the middle of a line in one packet from PH to PF. Alternatively, the data output unit 284 can store and output multiple lines of uncompressed data in one packet from PH to PF.

[0138] 17 is a diagram showing an example of a packing method for fixed-length compressed data according to the first embodiment of the present technology. As described above, 12-bit uncompressed data is losslessly compressed into 8-bit fixed-length compressed data. How the fixed-length compressed data for each pixel is packed can be set arbitrarily.

[0139] Payload data is transmitted in long packets, and data is packed in 1-byte units. Therefore, as shown in the example of a in the figure, the data output unit 284 can store uncompressed data for each pixel directly in the payload in byte units.

[0140] In the figure, b shows an example of a mapping method to a frame. For example, one packet from PH to PF stores one line of data. ０ The Y address of the edge pixel is stored in the first byte PD. ０ The fixed-length compressed data is stored, and the X address is X in the second byte.１ The fixed-length compressed data is stored. Then, the X address is X at the K-1th byte (K is an integer). Ｋ－１ The data output unit 284 can also store the Y address in the first byte PD instead of PH.

[0141] The data output unit 284 also stores one line of fixed-length compressed data in one packet from PH to PF and outputs the data. The data output unit 284 can also store data up to the middle of a line in one packet from PH to PF and output the data. Alternatively, the data output unit 284 can store multiple lines of fixed-length compressed data in one packet from PH to PF and output the data.

[0142] The gray area in "b" in the figure indicates fixed-length compressed data for pixels with edges. Binary data (00000000) indicates fixed-length compressed data for pixels without edges. As shown in "b" in the figure, in the first embodiment, data for all pixels is output regardless of whether there is an edge or not.

[0143] 18 is a block diagram showing an example of the configuration of the application processor 120 according to the first embodiment of the present technology. The application processor 120 includes a MIPI packet disassembly unit 121, a data decompression unit 122, a received data output unit 123, and an application processing unit 124. The application processor 120 is an example of the decompression unit described in the claims.

[0144] The MIPI packet decomposing unit 121 analyzes the packets and decomposes them into data for each pixel. The MIPI packet decomposing unit 121 supplies the uncompressed data for each pixel to the received data output unit 123, and supplies the fixed-length compressed data for each pixel to the data decompressing unit 122.

[0145] The data decompression unit 122 decompresses the fixed-length compressed data. Using the table shown in FIG. 12, the data decompression unit 122 converts the 8-bit fixed-length compressed data for each pixel into 12-bit uncompressed data, and supplies the uncompressed data to the received data output unit 123.

[0146] Returning to FIG. 18, the received data output unit 123 supplies a frame in which uncompressed data for each pixel is arranged to the application processing unit 124.

[0147] The application processing unit 124 performs various image processing operations on the frames, such as image recognition, etc. The application of the application processing unit 124 can also decompress fixed-length compressed data.

[0148] [Image Sensor Operation Example] Fig. 19 is a flowchart showing an example of the operation of the image sensor 200 according to the first embodiment of the present technology. The operation in Fig. 19 is started, for example, when the edge detection mode is set. The image sensor 200 selects and drives an unselected row (step S901) and sets one of N gain levels (step S902). The image sensor 200 then generates pixel signals ΔVSL0 to ΔVSL3 for each window (step S903) and generates comparison results VCO0 to VCO3 (step S904). The image sensor 200 then determines whether comparisons have been performed at all gain levels (step S905).

[0149] If there is any gain that has not been compared (step S905: No), the image sensor 200 returns to step S902 and switches the gain.

[0150] On the other hand, if the comparison is performed for all gains (step S905: Yes), the image sensor 200 determines whether the compression mode is set (step S906). If the compression mode is set (step S906: Yes), the image sensor 200 losslessly compresses the uncompressed data for each pixel to generate fixed-length compressed data (step S907).

[0151] If the non-compression mode is set (step S906: No), or after step S907, the image sensor 200 determines whether all rows have been selected (step S908).

[0152] If any row is not selected (step S908: No), the image sensor 200 repeatedly executes steps S901 and thereafter. On the other hand, if all rows have been selected (step S908: Yes), the image sensor 200 ends the operation for capturing one frame.

[0153] As described above, according to the first embodiment of the present technology, the logic circuit 280 performs lossless compression on the uncompressed data in which the comparison results for each gain are arranged to generate fixed-length compressed data, thereby preventing information loss.

[0154] 2. Second Embodiment In the first embodiment described above, the image sensor 200 compresses 12-bit uncompressed data into 8-bit fixed-length compressed data, but it is preferable to further improve the compression ratio. The image sensor 200 in this second embodiment differs from the first embodiment in that it compresses data into variable-length data.

[0155] 20 is a block diagram showing an example configuration of a logic circuit 280 according to a second embodiment of the present technology. The logic circuit 280 according to the second embodiment differs from the first embodiment in that it further includes a data buffer 285 and a variable-length compression unit 290. The variable-length compression unit 290 includes a gain-direction compression unit 291.

[0156] In the second embodiment, the edge detection modes include a non-compression mode, a fixed length compression mode, and a gain direction compression mode.

[0157] When the fixed length compression mode is set, the fixed length compression unit 282 supplies the fixed length compressed data of the channel information ch0 to ch3 to the data output unit 284. On the other hand, when the gain direction compression mode is set, the fixed length compression unit 282 causes the data buffer 285 to hold the fixed length compressed data of the channel information ch0 to ch3.

[0158] When the gain direction compression mode is set, the gain direction compression unit 291 generates variable-length compressed data by losslessly compressing the fixed-length compressed data based on the gain direction. This variable-length compressed data includes a 4-bit header section and a data section whose data size is 0, 2, or 4 bits. The gain direction compression unit 291 supplies the generated variable-length compressed data to the data output unit 284.

[0159] Here, the header section Hv is data indicating the presence or absence of an edge for each gain direction, and the data section is data indicating the strength of the edge.

[0160] 21 is a block diagram showing an example of the configuration of a gain direction compressor 291 according to the second embodiment of the present technology. The gain direction compressor 291 includes a header portion generating circuit 292 and a data portion generating circuit 297.

[0161] The header part generation circuit 292 generates a 4-bit header part Hv based on the channel information ch0 to ch3. For example, the header part generation circuit 292 includes OR (logical sum) gates 293, 294, 295, and 296.

[0162] The OR gate 293 calculates the logical sum of the 1st bit and the 0th bit of the channel information ch0. The OR gate 293 supplies the logical sum to the data portion generation circuit 297 and the data output portion 284 as the 0th bit of the header portion Hv.

[0163] The OR gate 294 calculates the logical sum of the 1st bit and the 0th bit of the channel information ch1. The OR gate 294 supplies the logical sum to the data portion generation circuit 297 and the data output portion 284 as the 1st bit of the header portion Hv.

[0164] The OR gate 295 calculates the logical sum of the 1st bit and the 0th bit of the channel information ch2. The OR gate 295 supplies the logical sum to the data portion generation circuit 297 and the data output portion 284 as the second bit of the header portion Hv.

[0165] The OR gate 296 calculates the logical sum of the 1st bit and the 0th bit of the channel information ch3. The OR gate 296 supplies the logical sum to the data portion generation circuit 297 and the data output portion 284 as the 3rd bit of the header portion Hv.

[0166] The data portion generation circuit 297 generates the data portion Dv by reading the channel information ch from the data buffer 285 based on the header portion Hv. The operation of the data portion generation circuit 297 will be described in detail later. The data portion generation circuit 297 supplies the generated data portion Dv to the data output unit 284.

[0167] In general, if the number of gain directions (in other words, channels) per window is 2×M (M is an integer), the data size of the header section is 2×M bits. Also, if the data size of the channel information ch is F bits expressed by Equation 18, the maximum data size of the header section is M×F bits.

[0168] FIG. 22 is a diagram for explaining the operation of the data portion generation circuit 297 according to the second embodiment of the present technology.

[0169] When the header section Hv is binary (0000), it means that there are no edges in any gain direction. In this case, the data section generation circuit 297 stops operating and does not output the data section Dv.

[0170] If any of the four bits of the header section Hv is a logical value "1," it means that an edge exists in one of the four gain directions. In this case, the data section generation circuit 297 reads the channel information ch corresponding to the direction in which the edge was detected from the data buffer 285 and outputs it as the data section Dv. For example, if the 0th bit of the header section Hv is a logical value "1," the channel information ch0 corresponding to that bit is read out.

[0171] If two of the four bits in the header section Hv have a logical value of "1," this means that there are edges in two of the four gain directions. In this case, the data section generation circuit 297 reads channel information ch corresponding to each of the two directions in which the edge was detected from the data buffer 285 and outputs it as the data section Dv. For example, if the first and third bits of the header section Hv have a logical value of "1," the channel information ch1 and ch3 corresponding to those bits are read.

[0172] The gain direction compression illustrated in FIGS. 21 and 22 does not cause any loss of information and corresponds to lossless compression.

[0173] FIG. 23 is a diagram illustrating an example of fixed-length compressed data and gain-direction compressed data according to the second embodiment of the present technology.

[0174] As shown in the example of a in the figure, when the logical sum of channel information ch0 and channel information ch2 is logically "1," this means that edges have been detected in the two directions corresponding to those channels. In this case, the gain direction compressor 291 outputs a 4-bit header section and a 4-bit data section consisting of the channel information ch0 and ch2 in which edges have been detected. Note that, because edges are never detected in three or four directions, the data section is a maximum of 4 bits.

[0175] As shown in Fig. 1B, when only the logical sum of channel information ch0 is logically equal to 1, this means that an edge has been detected in one direction corresponding to that channel. In this case, the gain direction compressor 291 outputs a 4-bit header section and a 2-bit data section consisting of channel information ch0 in which the edge has been detected.

[0176] As shown in Fig. 1C, when the logical sum of the channel information ch0 to ch3 is logically "0," this means that no edge was detected in any gain direction. In this case, the gain direction compressor 291 outputs a 4-bit header section but does not output a data section.

[0177] As shown in a, b, and c in the figure, the header section is 4 bits and the data section is 0, 2, or 4 bits, so the data size of the gain-direction compressed data consisting of the header section and the data section is 4 to 8 bits. The compression rate can be further improved by compressing 8-bit fixed-length compressed data to 4-bit to 8-bit gain-direction compressed data.

[0178] The number of pixels in which edges are detected in an edge image is at most about 30% of all pixels. For example, in a certain edge image, edges are detected in 15% of all pixels, and of those 15%, 5% of the pixels have edges detected in two directions, and 10% have edges detected in one direction.

[0179] In this case, the average data size for each pixel can be calculated using the following formula: 8 x 0.05 + 6 x 0.05 x 2 + 4 x 0.85 = 4.4... Formula 19

[0180] Therefore, the compression ratio of the gain-direction compressed data for 12-bit lossy data is 36.6 (≈4.4 / 12×100)%, and the compression ratio of the gain-direction compressed data for fixed-length compressed data is 55 (=4.4 / 8×100)%.

[0181] 24 is a diagram showing an example of a packing method for gain-direction compressed data according to a second embodiment of the present technology. In the second embodiment, the data size of the gain-direction compressed data for each pixel is variable length and consists of a 4-bit header section and a 0- to 4-bit data section. The bit arrangement of the header section can be set arbitrarily. Here, the corresponding bits are arranged in ascending order of channel number, starting from the most significant digit of the header section.

[0182] As shown in FIG. 11A, the data output unit 284 stores the header section and data section consecutively in the PD for each pixel and outputs them. For example, the header section and data section of pixel 1 are both 4 bits, and these 8 bits are stored in the first byte of PD. Similarly, the header section of pixel 2 is 4 bits and the data section is 2 bits, and these 6 bits, along with 2 bits of the header section of the next pixel 3, are stored in the second byte of PD. Pixel 3 has no data section, and the remaining 2 bits of its header section are stored in the third byte of PD.

[0183] Alternatively, as shown in FIG. 1B, the data output unit 284 can separate the header section and the data section, store them in the PD, and output them. For example, the 8 bits comprising the header sections of pixel 1 and pixel 2 are stored in the first byte of the PD. The header section of pixel 3 is stored in the second byte of the PD. The data section of pixel 1 is 4 bits, and the data section of pixel 2 is 2 bits. Furthermore, pixel 3 has no data section. The 6 bits comprising these data sections are stored in a predetermined PD from the third byte onwards.

[0184] The data output unit 284 also stores and outputs one line of gain-direction compressed data in one packet from PH to PF. The data output unit 284 can also store and output data up to the middle of a line in one packet from PH to PF. Alternatively, the data output unit 284 can store and output multiple lines of gain-direction compressed data in one packet from PH to PF.

[0185] 25 is a flowchart showing an example of a decompression operation of the application processor 120 according to the second embodiment of the present technology. This operation is started, for example, every time the application processor 120 receives a packet.

[0186] The application processor 120 analyzes the packet header to determine whether uncompressed data is stored (step S951). If uncompressed data is stored, the application processor 120 reads the uncompressed data from the payload and ends packet processing.

[0187] If compressed data is stored (step S951: No), the application processor 120 determines whether the data is fixed-length compressed data (step S952).

[0188] If the data is fixed-length compressed data (step S952: Yes), the application processor 120 decompresses the fixed-length compressed data for each pixel (step S953).

[0189] If the data is not fixed-length compressed data but gain-direction compressed data (step S952: No), the application processor 120 reads the 4-bit header portion from the payload (step S954).The application processor 120 then counts the number of logical "1"s in the header portion (step S955) and determines whether the count is greater than 0 (step S956).

[0190] If the count is greater than 0 (step S956: Yes), the application processor 120 reads the data portion having a data size of the count multiplied by 2 bits (step S957) and decompresses the header portion and the data portion (step S958). Decompression is performed using a decompression method corresponding to the compression method illustrated in Figures 21 and 22.

[0191] Returning to FIG. 25, if the count value is 0 (step S956: No), there is no data portion, so the application processor 120 decompresses only the header portion (step S959).

[0192] After step S958 or S959, the application processor 120 determines whether decompression of the payload is complete (step S960). For example, whether decompression is complete is determined based on whether the packet footer has been read or whether the number of decompressed pixels has reached the number of pixels for one line.

[0193] If decompression of the payload is not complete (step S960: No), the application processor 120 repeats step S954 and subsequent steps. If decompression of the payload is complete (step S960: Yes), or after step S953, the application processor 120 ends packet processing.

[0194] As described above, according to the second embodiment of the present technology, the logic circuit 280 losslessly compresses the uncompressed data into variable-length gain-direction compressed data, thereby making it possible to further improve the compression ratio.

[0195] 3. Third Embodiment In the second embodiment described above, the logic circuit 280 generates variable-length gain-direction compressed data based on the gain direction, but it is also possible to generate entropy codes instead of this gain-direction compressed data. The logic circuit 280 in this third embodiment differs from the second embodiment in that it compresses uncompressed data to generate entropy codes.

[0196] 26 is a block diagram showing an example configuration of a logic circuit 280 according to the third embodiment of the present technology. The logic circuit 280 according to the third embodiment differs from the second embodiment in that an entropy coding unit 298 is further arranged within the variable-length compression unit 290.

[0197] In the third embodiment, the edge detection modes include a non-compression mode, a fixed length compression mode, a gain direction compression mode, and an entropy coding mode.

[0198] In entropy coding mode, the entropy coding unit 298 encodes (i.e., lossless-compresses) fixed-length compressed data for each window to generate an entropy code. For example, a Huffman code is used as the entropy code. For example, 8-bit fixed-length compressed data is encoded into a 1-bit, 5-bit, 6-bit, or 7-bit Huffman code Hf. This Huffman code Hf is supplied to the data output unit 284.

[0199] 27 is a diagram showing an example of fixed-length compressed data and 1- to 7-bit Huffman codes according to the third embodiment of the present technology. In the diagram, each of the 2-bit channel information in the fixed-length compressed data is represented by a quaternary value ranging from 0 to 3. The Huffman code is represented by a binary number.

[0200] For example, fixed-length compressed data of quaternary (0000) is encoded into a 1-bit Huffman code of logical value "0".

[0201] Fixed-length compressed data in which an edge is detected in only one direction, such as (1000) in quaternary, is coded into a 5-bit Huffman code.

[0202] Fixed length compressed data of quaternary numbers (1010), (1020), (1030) and (2010) are coded into 6-bit Huffman codes.

[0203] Fixed-length compressed data from (2020) to (0330) in quaternary notation is coded into a 7-bit Huffman code.

[0204] As shown in FIG. 28, fixed-length compressed data from (0101) to (0303) in quaternary notation is coded into a 7-bit Huffman code.

[0205] As shown in FIGS. 27 and 28, the compression ratio can be further improved by encoding (lossless compression) 8-bit fixed-length compressed data into 1- to 7-bit Huffman codes.

[0206] In a certain edge image, edges are detected in 15% of all pixels, and of those 15%, 5% of the pixels have edges detected in two directions, and 10% have edges detected in one direction. Also, assume that all pixels with edges detected in two directions are coded to 6 bits. In this case, the average data size per pixel can be calculated using the following formula: 1 x 0.85 + 5 x 0.10 + 6 x 0.5 = 1.65 ...Equation 20

[0207] Therefore, the compression rate of Huffman coding for fixed-length compressed data is 20.625 (=1.65 / 8×100)%.

[0208] Instead of Huffman codes, arithmetic codes or range codes can also be used.

[0209] Fig. 29 is a diagram showing an example of an added data type in the third embodiment of the present technology. In the third embodiment, in addition to the various data types exemplified in b in Fig. 15 , the data type exemplified in Fig. 29 is newly defined.

[0210] For example, data types "0x30", "0x31", and "0x32" are added. "0x30" indicates that the data stored in the packet is fixed-length compressed data. "0x31" indicates that the data stored in the packet is gain-direction compressed data. "0x32" indicates that the data stored in the packet is Huffman coded.

[0211] The downstream application processor 120 can identify the type of data in the payload by referring to the data type in the packet header.

[0212] 30 is a diagram showing an example of a packing method for Huffman code data according to the third embodiment of the present technology. The data size of the Huffman code for each pixel is variable, being 1 bit, 5 bits, 6 bits, or 7 bits. For example, the uncompressed data for pixel 1 is encoded into a 5-bit Huffman code, and the uncompressed data for pixels 2 to 11 is encoded into a 1-bit Huffman code. The uncompressed data for pixel 12 is encoded into a 6-bit Huffman code.

[0213] In this case, the Huffman code P1 for pixel 1 and the Huffman codes P2, P3, and P4 for pixels 2 to 4 are stored in the first byte of PD. The Huffman codes P5 to P11 for pixels 5 to 11 and the fifth bit of the Huffman code P12 for pixel 12 are stored in the second byte of PD. The remaining bits for pixel 12 are stored in the third byte of PD.

[0214] The data output unit 284 also stores and outputs one line's worth of Huffman codes in one packet from PH to PF. The data output unit 284 can also store and output data up to the middle of a line in one packet from PH to PF. Alternatively, the data output unit 284 can store and output multiple lines' worth of Huffman codes in one packet from PH to PF.

[0215] As described above, according to the third embodiment of the present technology, the logic circuit 280 losslessly compresses the uncompressed data into a variable-length entropy code (such as a Huffman code), thereby further improving the compression ratio.

[0216] 4. Fourth Embodiment In the third embodiment described above, the image sensor 200 compares the pixel signals of each pixel pair within a window, but each pixel pair within a window may be the sum of two or more pixels. The image sensor 200 in this fourth embodiment differs from the third embodiment in that each pixel within a pixel pair is the sum of two or more pixels.

[0217] 31 is a diagram showing an example of a pixel array unit 240 according to a fourth embodiment of the present technology. In the fourth embodiment, the pixel array unit 240 has R (Red) pixels, G (Green) pixels, and B (Blue) pixels arranged in a Bayer array. The R pixels receive red visible light, the G pixels receive green visible light, and the B pixels receive blue visible light. The G pixels include Gr pixels arranged in rows of R pixels and Gb pixels arranged in rows of B pixels.

[0218] The pixel array unit 240 is divided into a plurality of FD shared blocks, such as FD shared blocks 260-1, 260-2, 260-3, and 260-4. Each FD shared block has an R pixel, a Gr pixel, a Gb pixel, and a B pixel arranged in two rows and two columns. These four pixels share a floating diffusion layer. The black squares in the figure indicate the floating diffusion layer. The FD shared blocks 260-1 and 260-2 are an example of a pair of shared blocks described in the claims.

[0219] 32 is a circuit diagram showing a configuration example of an FD shared block 260-1 according to the fourth embodiment of the present technology. The FD shared block 260-1 includes photoelectric conversion elements 261, 262, 263, and 264, and transfer transistors 265, 266, 267, and 268. The FD shared block 260-1 further includes a reset transistor 253, a floating diffusion layer 254, an amplification transistor 255, and a selection transistor 256. For example, nMOS transistors are used as the transfer transistors 265, 266, 267, and 268. The circuit configurations of the FD shared blocks 260-2, 260-3, and 260-4 are the same as that of the FD shared block 260-1.

[0220] The transfer transistor 265 transfers charges from the photoelectric conversion element 261 to the floating diffusion layer 254 in accordance with a transfer signal TRGa1 from the vertical scanning circuit 210. The transfer transistor 266 transfers charges from the photoelectric conversion element 262 to the floating diffusion layer 254 in accordance with a transfer signal TRGa2 from the vertical scanning circuit 210. The transfer transistor 267 transfers charges from the photoelectric conversion element 263 to the floating diffusion layer 254 in accordance with a transfer signal TRGa3 from the vertical scanning circuit 210. The transfer transistor 268 transfers charges from the photoelectric conversion element 264 to the floating diffusion layer 254 in accordance with a transfer signal TRGa4 from the vertical scanning circuit 210.

[0221] The connection configuration of the reset transistor 253, the floating diffusion layer 254, the amplification transistor 255, and the selection transistor 256 in the FD shared block 260-1 is the same as that in the first embodiment.

[0222] FIG. 33 is a diagram showing an example of pixel pairs in a window according to the fourth embodiment of the present technology. The vertical scanning circuit 210 simultaneously supplies transfer signals TRGa1, TRGa2, TRGa3, and TRGa4 to the FD shared block 260-1 at the timing of charge transfer. This causes the R pixel, Gr pixel, Gb pixel, and B pixel in the FD shared block 260-1 to be added together. Similarly, four pixels are added together in each of the FD shared blocks 260-2, 260-3, and 260-4. The new pixel generated by pixel addition is referred to as a "sum pixel." The gray areas in the diagram indicate the sum pixels.

[0223] As shown in the figure, pixel addition results in two pairs of pixel sums. The downstream edge detection unit 340 increases or decreases one of the pixel signals in each pixel sum pair and compares them. Because the pixel sum is brighter than the pixel before addition, edge detection accuracy can be improved even in low-light environments.

[0224] Note that the vertical scanning circuit 210 adds together an R pixel, a Gr pixel, a Gb pixel, and a B pixel for each FD shared block, but it is also possible to add together three pixels: an R pixel, a Gr pixel or a Gb pixel, and a B pixel.

[0225] Alternatively, pixels may be arranged in a quad-bayer arrangement, in which four pixels of the same color are arranged in each FD shared block. For each FD shared block, the vertical scanning circuit 210 adds two, three, or four pixels in that block.

[0226] As described above, according to the fourth embodiment of the present technology, the edge detection unit 340 compares the pixel signals of each pair of summed pixels, thereby improving the accuracy of edge detection even in a low-illumination environment.

[0227] In the fourth embodiment described above, the image sensor 200 adds up and compares the R, Gr, B, and Gb pixels, but this pixel addition results in a loss of color information. The image sensor 200 in this modification of the fourth embodiment differs from the fourth embodiment in that it compares pairs of pixel signals for each color.

[0228] 34 is a diagram illustrating an example of pixel pairs in a window in a modification of the fourth embodiment of the present technology. In the modification of the fourth embodiment, a four-pixel addition mode or an RGB mode is set in the image sensor 200.

[0229] When the RGB mode is set, the vertical scanning circuit 210 drives each of the FD shared blocks 260-1, 260-2, 260-3, and 260-4 to output pixel signals for each color. For example, the vertical scanning circuit 210 first drives only the R pixels in each FD shared block, then drives only the Gr and Gb pixels in each FD shared block. Then, the vertical scanning circuit 210 drives only the B pixels in each FD shared block. Note that the vertical scanning circuit 210 can also drive only one of the Gr and Gb pixels without driving both.

[0230] On the other hand, when the four-pixel addition mode is set, four pixels in the FD shared block are added together, as in the fourth embodiment.

[0231] FIG. 35 is a diagram illustrating an example of uncompressed data and fixed-length compressed data according to a modified example of the fourth embodiment of the present technology.

[0232] When RGB mode is set, the edge detection unit 340 outputs 12 bits consisting of four pieces of channel information CH for each of R, G, and B for each window. Therefore, 36 (=12×3) bits of uncompressed data are held in the data buffer 281 for each window. This data includes channel information CH0 to CH11. Channel information CH0 to CH3 is data corresponding to R pixels, and channel information CH4 to CH7 is data corresponding to G pixels. Channel information CH8 to CH11 is data corresponding to B pixels.

[0233] The fixed-length compression unit 282 losslessly compresses each of the 3-bit channel information CH0 to CH11 into 2 bits, thereby generating 24 (=12×2) bits of fixed-length compressed data consisting of the channel information ch0 to ch11.

[0234] As shown in the figure, in the RGB output mode, color information is not lost, so performance such as recognition accuracy in image recognition can be improved.

[0235] FIG. 36 is a diagram illustrating an example of fixed-length compressed data and gain-direction compressed data according to a modified example of the fourth embodiment of the present technology.

[0236] When the gain direction compression mode is set, the gain direction compression unit 291 generates a header section and a data section from 24-bit fixed-length compressed data. Because there are 12 channels, the data size of the header section is 12 bits. The data size of the data section ranges from 0 bits to 12 bits.

[0237] When the entropy coding mode is set, the fixed-length compressed data is coded using Huffman codes or the like.

[0238] In this way, according to the modification of the fourth embodiment of the present technology, the edge detection circuit 330 compares a pair of pixel signals for each of the colors R, G, and B, so that color information is not lost and performance such as recognition accuracy can be improved.

[0239] 5. Fifth Embodiment In the third embodiment described above, the image sensor 200 outputs data of all pixels to the outside, but it is preferable to map that data more efficiently. The image sensor 200 in this fifth embodiment differs from the third embodiment in that it outputs data of only pixels that have edges.

[0240] 37 is a diagram showing an example of a method for transmitting fixed-length compressed data according to the fifth embodiment of the present technology. In the fixed-length compression mode, the data output unit 284 associates the addresses of pixels having edges among a plurality of pixels in an edge image with the fixed-length compressed data corresponding to the addresses, stores them in packets, and outputs the packets.

[0241] For example, if the Y address is Y ０ In the line, X ０ and X １１Assume that an edge is detected at an X address such as: In the figure, the gray portion indicates fixed-length compressed data of the pixel where the edge is detected.

[0242] The data output unit 284 outputs Y to PH. ０ and a data type indicating fixed-length compressed data. The data output unit 284 also stores X in the first byte of PD. ０ Store 8-bit data indicating the ０ Then, the data output unit 284 stores 8-bit fixed-length compressed data corresponding to the third byte of PD. １１ Store 8-bit data indicating the １１ The data output unit 284 then outputs the packet. In the figure, the number of pixels in which an edge is detected within a row is three or more, but if an edge is detected in only two pixels, PF is output after PD in the fourth byte. １ The same process is carried out for each subsequent row. The data output unit 284 can also store the Y address in the first byte PD instead of PH.

[0243] Fixed-length compressed data for pixels without edges is not stored in packets and is not output. By outputting only fixed-length compressed data for pixels with edges, data can be mapped more efficiently than when fixed-length compressed data for all pixels is output. However, if there are a large number of pixels with edges, X addresses must be assigned, which may result in overhead.

[0244] FIG. 38 is a diagram illustrating a method for receiving fixed-length compressed data according to the fifth embodiment of the present technology.

[0245] The MIPI packet disassembler 121 in the application processor 120 reads the Y address and data type from PH. If the data type is fixed-length compressed data, the MIPI packet disassembler 121 reads the data of the X address from an odd-numbered PD byte, such as the first byte, and reads the fixed-length compressed data corresponding to the X address from an even-numbered PD byte, such as the second byte. The MIPI packet disassembler 121 also inserts fixed-length compressed data of binary (00000000) for each address without an edge. The MIPI packet disassembler 121 then supplies the fixed-length compressed data to the data decompressor 122 (not shown) in address order.

[0246] 39 is a diagram showing an example of a method for transmitting gain direction compressed data according to the fifth embodiment of the present technology. In the gain direction compression mode, the data output unit 284 associates the addresses of pixels having edges with the gain direction compressed data corresponding to the addresses, stores the packets in the packets, and outputs the packets.

[0247] For example, if the Y address is Y ０ In the line, X ０ and X １１ Assume that an edge is detected at an X address such as:

[0248] The data output unit 284 outputs Y to PH. ０ The data output unit 284 stores the data type indicating the gain direction compressed data. ０ Store 8-bit data indicating the ０ The XY address must have a number of bits that can achieve a predetermined resolution.

[0249] Then, the data output unit 284 adds X to the third byte of PD. １１ Store 8-bit data indicating the １１ The remaining two bits of the PD have an arbitrary value. The data output unit 284 then outputs the packet. １The same process is carried out for each subsequent row. The data output unit 284 can also store the Y address in the first byte PD instead of PH.

[0250] Furthermore, to reduce address overhead, the data output unit 284 may divide the X address by S most significant bits (S is an integer) and store only the least significant bits in PD. In this case, the data output unit 284 may store the S most significant bits in PH, or store a newly defined data type as a special signal in PH and transmit it to the outside.

[0251] FIG. 40 is a diagram for explaining a method of receiving gain direction compressed data according to the fifth embodiment of the present technology.

[0252] The MIPI packet disassembler 121 in the application processor 120 reads the Y address and data type from PH. If the data type is gain direction compressed data, the MIPI packet disassembler 121 reads the data of the X address from an odd-numbered PD, such as the first byte, and reads the gain direction compressed data corresponding to the X address from an even-numbered PD, such as the second byte. The MIPI packet disassembler 121 also inserts gain direction compressed data of binary (0000) for each address without an edge. The MIPI packet disassembler 121 then supplies the gain direction compressed data to the data decompressor 122 (not shown) in address order.

[0253] Similarly, in the non-compression mode and the entropy coding mode, the data output unit 284 can output only the data of pixels that have edges.

[0254] The fourth embodiment can be applied to the fifth embodiment.

[0255] As described above, according to the fifth embodiment of the present technology, the data output unit 284 stores only data of pixels having edges in packets and outputs the packets, thereby enabling more efficient data mapping.

[0256] 6. Sixth Embodiment In the third embodiment described above, the image sensor 200 has four gain directions and generates four channels of compressed data corresponding to the four directions for each window, but the number of gain directions is not limited to four. The image sensor 200 in this sixth embodiment differs from the third embodiment in that it generates compressed data for six or more channels.

[0257] FIG. 41 is a diagram showing an example of pixel pairs in a window according to the sixth embodiment of the present technology.

[0258] For example, as shown in Fig. 1A, four pixel pairs within a window are targeted for edge detection. In this case, there are eight gain directions, and the edge detection unit 340 compares the pixel signals for each pixel pair and outputs eight comparison result VCOs.

[0259] As shown in Fig. 1B, the number of pixel pairs to be detected can be three. In this case, there are six gain directions, and the edge detection unit 340 compares the pixel signals for each pixel pair and outputs six comparison result VCOs.

[0260] By increasing the number of gain directions to six or more, the image quality of the edge image can be improved more than when there are four gain directions.

[0261] 42 is a diagram showing an example of uncompressed data, fixed-length compressed data, and gain-direction compressed data according to the sixth embodiment of the present technology, where "a" in the figure shows an example of the uncompressed data and fixed-length compressed data, and "b" in the figure shows an example of the fixed-length compressed data and gain-direction compressed data.

[0262] As shown in the example of a in the figure, when there are eight gain directions and three gain levels, the number of channels is eight, and 24 (=3×8) bits of uncompressed data are held in the data buffer 281. The fixed-length compression unit 282 compresses each channel to 2 bits, generating 16 (=2×8) bits of fixed-length compressed data.

[0263] As shown in Fig. 1B, in the gain direction compression mode, 16-bit fixed-length compressed data is compressed into gain direction compressed data consisting of an 8-bit header section and a data section of bits from 0 to 8. In the entropy coding mode, the fixed-length compressed data is coded using Huffman codes or the like.

[0264] Moreover, the fourth and fifth embodiments can be applied to the sixth embodiment.

[0265] As described above, according to the sixth embodiment of the present technology, the number of gain directions is increased to six or more, and therefore the image quality of the edge image can be improved.

[0266] 7. Application Examples of Mobile Objects The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile object, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0267] FIG. 43 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0268] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 43, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053.

[0269] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0270] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0271] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0272] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0273] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0274] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0275] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0276] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0277] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 43, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0278] FIG. 44 is a diagram showing an example of the installation position of the imaging unit 12031.

[0279] In FIG. 44, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0280] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0281] 44 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0282] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0283] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0284] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0285] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0286] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 of the above-described configuration. Specifically, the imaging device 100 in FIG. 1 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, it is possible to prevent information loss and improve the image quality of edge images.

[0287] Note that the above-described embodiment shows an example for realizing the present technology, and the matters in the embodiment and the matters specifying the invention in the claims correspond to each other. Similarly, the matters specifying the invention in the claims and the matters in the embodiment of the present technology having the same name correspond to each other. However, the present technology is not limited to the embodiment, and can be realized by applying various modifications to the embodiment within the scope of the gist thereof.

[0288] The effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0289] The present technology may also be configured as follows: (1) An image sensor comprising: a level control circuit that increases or decreases one of a pair of pixel signals by one of a plurality of gains; a comparison unit that compares the pair of pixel signals, one of which has been increased or decreased, and outputs a comparison result; and a logic circuit that generates compressed data by losslessly compressing a bit string in which the comparison results corresponding to each of the plurality of gains are arranged. (2) The image sensor according to (1), further comprising: a timing control circuit that switches between ascending and descending gains to be set in the level control circuit from among the plurality of gains. (3) The image sensor according to (2), wherein the compressed data includes fixed-length compressed data. (4) The image sensor according to (3), wherein the bit string includes information on a predetermined number of channels, wherein the data size of the channel information is N bits, where N is the number of the plurality of gains, and the logic circuit losslessly compresses each of the channel information into F-bit data, where F is the value obtained by truncating the logarithm of (N+1) to base 2. (5) The image sensor according to any one of (1) to (4), wherein the compressed data includes variable-length compressed data. (6) The image sensor according to (5), wherein the variable-length compressed data includes an entropy code. (7) The image sensor according to (5) or (6), wherein the variable-length compressed data includes gain direction compressed data compressed based on a gain direction. (8) The image sensor according to (7), wherein the gain direction compressed data includes a header section indicating the presence or absence of an edge for each gain direction and a data section indicating the strength of the edge. (9) The image sensor according to (8), wherein the data size of the header section is 2×M bits when the number of gain directions is 2×M, and the maximum data size of the data section is M×F bits when the number of the multiple gains is N and the value obtained by truncating the logarithm of (N+1) to base 2 is F. (10) The image sensor described in any one of (1) to (9), wherein the logic circuit includes: a compression unit that compresses the bit strings of each of a plurality of pixels in an edge image to generate the compressed data; and a data output unit that outputs the address of a pixel in which an edge is detected among the plurality of pixels and the compressed data corresponding to the address.(11) The image sensor according to any one of (1) to (10), further comprising a pixel array unit including a pair of shared blocks in each of which a plurality of pixels sharing a floating diffusion layer are arranged. (12) The image sensor according to (11), wherein one of the pair of pixel signals is a pixel signal of an additive pixel obtained by adding together a plurality of pixels arranged in one of the pair of shared blocks, and the other of the pair of pixel signals is a pixel signal of an additive pixel obtained by adding together a plurality of pixels arranged in the other of the pair of shared blocks. (13) An imaging system comprising: an image sensor including: a level control circuit that increases or decreases one of the pair of pixel signals by one of a plurality of gains; a comparison unit that compares the pair of pixel signals in which one has been increased or decreased and outputs the comparison result; and a logic circuit that losslessly compresses a bit string in which the comparison results corresponding to each of the plurality of gains are arranged to generate compressed data; and a decompression unit that decompresses the compressed data. (14) A control method for an image sensor, comprising: a level control step of increasing or decreasing one of a pair of pixel signals by one of a plurality of gains; a comparison step of comparing the pair of pixel signals, one of which has been increased or decreased, and outputting a comparison result; and a compression step of losslessly compressing a bit string in which the comparison results corresponding to each of the plurality of gains are arranged, to generate compressed data.

[0290] 100 Imaging device 110 Optical unit 120 Application processor 121 MIPI packet disassembly unit 122 Data decompression unit 123 Received data output unit 124 Application processing unit 130 Display unit 140 Operation unit 150 Bus 160 Frame memory 170 Storage unit 180 Power supply unit 200 Image sensor 210 Vertical scanning circuit 220 DAC 230 Timing control circuit 240 Pixel array unit 250 Pixel 251, 261 to 264 Photoelectric conversion element 252, 265 to 268 Transfer transistor 253 Reset transistor 254 Floating diffusion layer 255 Amplification transistor 256 Selection transistor 260-1, 260-2, 260-3, 260-4 FD shared block 270 Horizontal scanning circuit 280 Logic circuit 281, 285 Data buffer 282 Fixed length compression unit 283 Image processing unit 284 Data output unit 290 Variable length compression unit 291 Gain direction compression unit 292 Header part generation circuit 293 to 296 OR (logical sum) gates 297 Data part generation circuit 298 Entropy coding unit 300 Column signal processing circuit 310 Load MOS current source circuit 311 Load MOS current source 320 Column ADC 321 ADC 330 Edge detection circuit 340 Edge detection unit 350 Level control circuit 360-1, 360-2 Gain circuits 361 to 364, 371 to 374 Selectors 365 to 368, 375 to 378 Capacitors 370-1, 370-2 Gain offset circuit 380 Comparison circuit 381, 382 Comparator 12031 Imaging unit

Claims

1. An image sensor comprising: a level control circuit that increases or decreases one of a pair of pixel signals by one of a plurality of gains; a comparison unit that compares the pair of pixel signals, one of which has been increased or decreased, and outputs the comparison result; and a logic circuit that losslessly compresses a bit string in which the comparison results corresponding to each of the plurality of gains are arranged, to generate compressed data.

2. The image sensor according to claim 1, further comprising a timing control circuit that switches the gains to be set in said level control circuit from among said plurality of gains in ascending or descending order.

3. The image sensor of claim 2, wherein said compressed data includes fixed-length compressed data.

4. The image sensor of claim 3, wherein the bit string includes a predetermined number of pieces of channel information, the data size of the channel information is N bits, where N is the number of the plurality of gains, and the logic circuit losslessly compresses each piece of channel information into F bits of data, where F is the value obtained by truncating the logarithm of (N+1) to base 2.

5. The image sensor of claim 1, wherein said compressed data includes variable length compressed data.

6. The image sensor according to claim 5, wherein said variable length compressed data includes an entropy code.

7. The image sensor according to claim 5, wherein the variable-length compressed data includes gain-direction compressed data compressed based on a gain direction.

8. The image sensor according to claim 7, wherein the gain direction compressed data includes a header portion indicating the presence or absence of an edge for each of the gain directions and a data portion indicating the strength of the edge.

9. The image sensor of claim 8, wherein the number of gain directions is 2×M, the data size of the header section is 2×M bits, and the number of the plurality of gains is N, and the value obtained by truncating the logarithm of (N+1) to base 2 is F, the maximum data size of the data section is M×F bits.

10. The image sensor according to claim 1, wherein the logic circuit comprises: a compression unit that compresses the bit strings of each of a plurality of pixels in an edge image to generate the compressed data; and a data output unit that outputs the address of a pixel in which an edge has been detected among the plurality of pixels and the compressed data corresponding to the address.

11. The image sensor according to claim 1, further comprising a pixel array section including a pair of shared blocks in each of which a plurality of pixels sharing a floating diffusion layer are arranged.

12. An image sensor as described in claim 11, wherein one of the pair of pixel signals is a pixel signal of an additive pixel obtained by adding together multiple pixels arranged in one of the pair of shared blocks, and the other of the pair of pixel signals is a pixel signal of an additive pixel obtained by adding together multiple pixels arranged in the other of the pair of shared blocks.

13. An imaging system comprising: an image sensor including: a level control circuit that increases or decreases one of a pair of pixel signals by one of a plurality of gains; a comparison unit that compares the pair of pixel signals, one of which has been increased or decreased, and outputs the comparison result; a logic circuit that losslessly compresses a bit string in which the comparison results corresponding to each of the plurality of gains are arranged, to generate compressed data; and a decompression unit that decompresses the compressed data.

14. A control method for an image sensor comprising: a level control step of increasing or decreasing one of a pair of pixel signals by one of a plurality of gains; a comparison step of comparing the pair of pixel signals, one of which has been increased or decreased, and outputting the comparison result; and a compression step of losslessly compressing a bit string in which the comparison results corresponding to each of the plurality of gains are arranged, to generate compressed data.

Images