Imaging device

Abstract

An imaging device includes: an image sensing unit, which is constructed to include image sensing pixels, which are operable to sense incident light received from a scene and generate image data carrying image information of the scene; a lens module, which is positioned to project incident light from the scene onto the image sensing pixels of the image sensing unit; a line buffer, which is connected to communicate with the image sensing unit and is configured to store the image data received from the image sensing unit; an optical distortion corrector (ODC), which is configured to perform lens distortion correction for output pixel coordinates based on input pixel coordinates of the image data stored in the line buffer to correct distortion caused by the lens module; and a line buffer controller, which is connected to the line buffer and is configured to control a reading speed of the image data from the line buffer to the optical distortion corrector according to the output pixel coordinates.

Description

Technical Field

[0001] Technologies and implementations disclosed in this patent document generally relate to imaging devices capable of generating image data by sensing light. Background Art

[0002] Image sensing devices are devices that capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer, and communication industries, the demand for high-performance image sensing devices is increasing in various fields such as smartphones, digital cameras, game consoles, IoT (Internet of Things), robots, security cameras, and medical micro cameras.

[0003] Image sensing devices can be roughly divided into CCD (Charge Coupled Device) image sensing devices and CMOS (Complementary Metal Oxide Semiconductor) image sensing devices. CCD image sensing devices provide better image quality, but they tend to consume more power and are larger than CMOS image sensing devices. CMOS image sensing devices are smaller in size and consume less power than CCD image sensing devices. In addition, CMOS manufacturing technology is used to manufacture CMOS sensors, so photosensors and other signal processing circuits can be integrated into a single chip, enabling miniaturized image sensing devices to be produced at a lower cost. For these reasons, CMOS image sensing devices are being developed for many applications including mobile devices. Summary of the invention

[0004] Various embodiments of the disclosed technology relate to an imaging device capable of correcting lens distortion.

[0005] According to one embodiment of the disclosed technology, an imaging device may include: an image sensing unit, which is constructed to include image sensing pixels, which are operable to sense incident light received from a scene and generate image data carrying image information of the scene; a lens module, which is positioned to project incident light from the scene onto the image sensing pixels of the image sensing unit; a line buffer, which is connected to communicate with the image sensing unit and is configured to store the image data received from the image sensing unit; an optical distortion corrector (ODC), which is configured to perform lens distortion correction for output pixel coordinates based on input pixel coordinates of the image data stored in the line buffer to correct distortion caused by the lens module; and a line buffer controller, which is connected to the line buffer and is configured to control the reading speed of the image data from the line buffer to the optical distortion corrector according to the output pixel coordinates.

[0006] According to another embodiment of the disclosed technology, an imaging device may include: a line buffer configured to store image data generated by sensing incident light; an optical distortion corrector (ODC) configured to receive the image data from the line buffer and perform lens distortion correction for output pixel coordinates; and a line buffer controller configured to control a reading speed of the image data from the line buffer based on a reference row corresponding to an output row including the output pixel coordinates.

[0007] It is to be understood that both the foregoing general description and the following detailed description of the disclosed technology are exemplary and explanatory and are intended to provide further explanation of the disclosure as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features and advantageous aspects of the disclosed technology will become readily apparent with reference to the following detailed description considered in conjunction with the accompanying drawings.

[0009] Figure 1 is a block diagram illustrating an example of an imaging system based on some implementations of the disclosed technology.

[0010] Figure 2 It is an example Figure 1 A block diagram of an example of an image sensing unit is shown.

[0011] Figure 3 is a conceptual diagram illustrating an example of lens distortion correction.

[0012] Figure 4 is a diagram illustrating that the range of lines to be referred to during lens distortion correction varies depending on the position of each pixel.

[0013] Figure 5 is a graph illustrating a relationship between output pixel coordinates according to one example of lens distortion and input pixel coordinates required for lens distortion correction of a pixel corresponding to the output pixel coordinates.

[0014] Figure 6 is an example of Figure 5 The graph of FIG. 1 is a graph of the capacity of the line buffer required when image data is read at a relatively low read speed.

[0015] Figure 7 is an example of Figure 5 The graph of FIG. 1 is a graph of the capacity of the line buffer required when image data is read at a relatively high read speed.

[0016] Figure 8 (a) and Figure 8 (b) in FIG. 1 shows a diagram illustrating an example of a method for adjusting the read speed of a line buffer.

[0017] Fig. 9 (a) and Fig. 9 (b) in FIG. 1 shows a diagram illustrating another example of a method for adjusting the read speed of a line buffer.

[0018] Fig.10 is a graph illustrating a relationship between output pixel coordinates according to another example of lens distortion and input pixel coordinates required for lens distortion correction of pixels corresponding to the output pixel coordinates.

[0019] Fig.11 is an example of Figure 7 The capacity and read speed of the same capacity and the same read speed of the line buffer with Fig.10 A graph showing the relationship between lens distortion.

[0020] Fig.12 is an example of Fig.10 The graph of FIG. 1 is a graph of the capacity of the line buffer required when reading image data at a constant reading speed.

[0021] Fig.13 is an example of Fig.10 The graph of FIG. 1 is a graph of the capacity of the line buffer required when reading image data at a variable reading speed.

[0022] Fig.14 is an example of reading at a constant reading speed Figure 3 FIG. 5 is a diagram of distorted image data and the capacity of a line buffer required when performing lens distortion correction on the distorted image data.

[0023] Fig.15 is an example of reading at a variable reading speed Figure 3 FIG. 4 is a diagram showing the capacity of a line buffer required when distorted image data is displayed and lens distortion correction is performed on the distorted image data. DETAILED DESCRIPTION

[0024] This patent document provides implementations and examples of imaging devices capable of generating image data by sensing light, which can be used in construction to substantially solve one or more technical or engineering problems and alleviate limitations or disadvantages encountered in some other imaging devices. Some implementations of the disclosed technology relate to imaging devices capable of correcting lens distortion. The disclosed technology provides various implementations of imaging devices that can minimize the required capacity of a line buffer by changing the read speed of the line buffer.

[0025] Reference will now be made in detail to embodiments of the disclosed technology, examples of which are illustrated in the accompanying drawings. As much as possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. Although various modifications and alternative forms are susceptible to being made to the disclosure, specific embodiments thereof are illustrated by way of example in the accompanying drawings. However, the disclosure should not be construed as being limited to the embodiments set forth herein.

[0026] In the following, various embodiments will be described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to a specific embodiment, but includes various modifications, equivalents and / or substitutes of the embodiments. The embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized by the disclosed technology.

[0027] Figure 1 is a block diagram illustrating an example of an imaging system based on some implementations of the disclosed technology. Figure 2 It is an example Figure 1 A block diagram of an example of an image sensing unit is shown. Figure 3 is a conceptual diagram illustrating an example of lens distortion correction. Figure 4 is a diagram illustrating that the range of lines to be referred to during lens distortion correction varies depending on the position of each pixel.

[0028] Reference Figure 1 , the imaging system 1 may refer to a device such as a digital camera for capturing still images or a digital video camera for capturing moving images. For example, the imaging device 10 may be implemented as a digital single-lens reflex (DSLR) camera, a mirrorless camera, or a smart phone, etc. The imaging device 10 may include a device having both a lens and an image pickup element, so that the device can capture (or capture) a target object, and thereby can create an image of the target object.

[0029] The imaging system 1 may include an imaging device 10 and a host device 20 .

[0030] The imaging device 10 may include an image sensing unit 100, a timing controller 200, a line buffer 300, a write controller 400, a line buffer read controller including a read controller 500 and a read speed controller 800, an optical distortion corrector (ODC) 600, a distortion correction value storage 700, an image signal processor (ISP) 900, and an input / output (I / O) interface 1000. The read controller 500 and the read speed controller 800 may be collectively referred to as a line buffer controller.

[0031] The image sensing unit 100 may be a complementary metal oxide semiconductor image sensor (CIS) for converting an optical signal into an electrical signal. The overall operation of the image sensing unit 100, such as on / off, operation mode, operation timing, sensitivity, etc., may be controlled by the timing controller 200. The image sensing unit 100 may provide the line buffer 300 with image data obtained by converting an optical signal into an electrical signal based on the control of the timing controller 200.

[0032] Reference Figure 2 , the image sensing unit 100 may include a lens module 110 , a pixel array 120 , a pixel driving circuit 130 , and a pixel readout circuit 140 .

[0033] The lens module 110 may collect incident light received from a scene image, and may converge the collected light onto pixels of the pixel array 120. The lens module 110 may include a plurality of lenses aligned with an optical axis. The lens module 110 may have a predetermined curvature so that the pixel array 120 may sense a scene corresponding to a predetermined field of view (FOV). However, due to such curvature, lens distortion may occur due to a difference between a scene and a frame sensed by the pixel array 120.

[0034] The pixel array 120 may include a plurality of unit pixels arranged in N rows (where N is an integer of 2 or greater) and M columns (where M is an integer of 2 or greater). In one example, the plurality of unit pixels may be arranged in a two-dimensional (2D) pixel array including rows and columns. In another example, the plurality of unit pixels may be arranged in a three-dimensional pixel array.

[0035] The plurality of unit pixels may convert an optical signal into an electrical signal on a unit pixel or pixel group basis, wherein the unit pixels in a pixel group share at least some internal circuit.

[0036] Each of the plurality of unit pixels may sense incident light (IL) to generate a pixel signal corresponding to the intensity of the incident light (IL). The pixel array 120 may receive a drive signal including a row selection signal, a pixel reset signal, and a transmission signal from the pixel driving circuit 130. Upon receiving the drive signal, the corresponding unit pixel of the pixel array 120 may be activated to perform an operation corresponding to the row selection signal, the pixel reset signal, and the transmission signal.

[0037] The pixel drive circuit 130 may activate the pixel array 120 to perform certain operations on the imaging pixels in the corresponding row based on the command and control signals provided by the controller circuit such as the timing controller 200. In some implementations, the pixel drive circuit 130 may select one or more imaging pixels arranged in one or more rows of the pixel array 120. The pixel drive circuit 130 may generate a row selection signal in response to the row address signal of the timing controller 200 to select one or more rows among the plurality of rows. The pixel drive circuit 130 may sequentially enable a pixel reset signal for resetting the imaging pixels corresponding to at least one selected row, and a transmission signal for the pixels corresponding to at least one selected row. Therefore, a reference signal and an image signal, which are analog signals generated by each imaging pixel of the selected row, may be sequentially transmitted to the pixel readout circuit 140. The reference signal may be an electrical signal provided to the pixel readout circuit 140 when a sensing node (e.g., a floating diffusion node) of the imaging pixel is reset, and the image signal may be an electrical signal provided to the pixel readout circuit 140 when the photocharge generated by the imaging pixel is accumulated at the sensing node. The reference signal indicating the unique reset noise of each pixel and the image signal indicating the intensity of incident light may be collectively referred to as a pixel signal as needed.

[0038] The pixel readout circuit 140 may use correlated double sampling (CDS) to remove undesired pixel offset values, known as fixed pattern noise, by sampling the pixel signal twice to remove the difference between the two samples (i.e., the reference signal and the image signal). In one example, correlated double sampling (CDS) may remove undesired pixel offset values ​​by comparing the pixel output voltages obtained before and after accumulating photocharges generated by incident light in the sensing node, so that only the pixel output voltage based on the incident light may be measured. In some embodiments of the disclosed technology, the pixel readout circuit 140 may sequentially sample and maintain the voltage levels of the reference signal and the image signal provided from the pixel array 120 to each of the plurality of column lines. That is, the pixel readout circuit 140 may sample and maintain the voltage levels of the reference signal and the image signal corresponding to each column of the pixel array 120.

[0039] The pixel readout circuit 140 may include an analog-to-digital converter (ADC) for converting the correlated double sampling signal into a digital signal. In some implementations, the ADC may be implemented as a ramp comparison ADC. The ramp comparison ADC may include: a comparator circuit for comparing the analog pixel signal with a reference signal such as a ramp signal that ramps up or down; and a timer for performing counting until the voltage of the ramp signal matches the analog pixel signal.

[0040] The pixel readout circuit 140 may include an output buffer that temporarily holds the column-based image data provided from the ADC to output the image data. In one example, the output buffer may temporarily store the image data output from the ADC based on a control signal of the timing controller 200. The output buffer may provide an interface to compensate for a data rate difference or a transmission rate difference between the image sensing unit 100 and other devices.

[0041] The pixel readout circuit 140 may include a column driver. The column driver may select a column of the output buffer upon receiving a control signal from the timing controller 200, and sequentially output image data temporarily stored in the selected column of the output buffer. In some implementations, upon receiving a column address signal from the timing controller 200, the column driver may select a column of the output buffer based on the column address signal, so that image data from the selected column of the output buffer may be output to the row buffer 300.

[0042] Return to reference Figure 1 The timing controller 200 may provide the image sensing unit 100 with clock signals required for the operation of the various components of the image sensing unit 100, control signals for timing control, row address signals for selecting rows, and column address signals for selecting columns. In an embodiment of the disclosed technology, the timing controller 200 may include a logic control circuit, a phase-locked loop (PLL) circuit, a timing control circuit, a communication interface circuit, and the like.

[0043] The timing controller 200 may provide the write controller 400 with a row address signal and a column address signal transmitted to the image sensing unit 100 .

[0044] The line buffer 300 may write (i.e., store) image data received from the image sensing unit 100 based on the control of the write controller 400, may read the image data based on the control of the read controller 500, and may transmit the read image data to the ODC 600. The write operation and the read operation of the line buffer 300 will be described later with reference to the write controller 400 and the read controller 500.

[0045] The row buffer 300 may include a volatile memory (e.g., DRAM, SRAM, etc.) and / or a non-volatile memory (e.g., flash memory). The row buffer 300 may have a capacity capable of storing image data corresponding to a predetermined number of rows. Each line may refer to a row (raw) of the pixel array 120, and the predetermined number of rows may be less than the total number of rows of the pixel array 120. Therefore, the row buffer 300 is not a frame memory capable of storing image data corresponding to a frame captured by the pixel array 120 at one time, but a row memory capable of storing image data corresponding to some rows (or lines) of the pixel array 120. The capacity of the row buffer 300 may be determined by the degree of lens distortion of the lens module 110, the generation and processing speed of image data, etc. The capacity of the row buffer 300 discussed in the patent document may refer to a capacity that can be allocated to store image data.

[0046] The write controller 400 may generate input pixel coordinates based on the row address signal and the column address signal received from the timing controller 200, and may transmit the input pixel coordinates to each of the row buffer 300 and the ODC 600. The input pixel coordinates may refer to coordinates of pixels corresponding to image data input from the image sensing unit 100 to the row buffer 300.

[0047] The coordinates of each pixel included in the pixel array 120 may be determined by the row and column to which the corresponding pixel belongs. For example, the coordinates of the pixel belonging to the fifth row and the tenth column may be (10, 5). When image data corresponding to the pixel corresponding to the coordinates (10, 5) is input to the row buffer 300, the write controller 400 may receive a row address signal indicating the fifth row and a column address signal indicating the tenth column from the timing controller 200, and may generate input pixel coordinates corresponding to the coordinates (10, 5) based on the row address signal and the column address signal.

[0048] Therefore, the row address signal may indicate the Y coordinate of the pixel corresponding to the image data input to the row buffer 300 , and the column address signal may indicate the X coordinate of the pixel corresponding to the image data input to the row buffer 300 .

[0049] The line buffer 300 may map the image data received from the image sensing unit 100 to the input pixel coordinates received from the write controller 400, and may store the mapped image data. Here, the mapping and storage operations of the line buffer 300 may indicate that the image data and the input pixel coordinates are stored corresponding to each other, so that the line buffer 300 may identify the image data corresponding to the specific input pixel coordinates.

[0050] The ODC 600 may determine whether to start lens distortion correction based on the input pixel coordinates received from the write controller 400. A more detailed description thereof will be given later in this patent document.

[0051] The read controller 500 may receive the reference pixel coordinates from the distortion correction value storage 700, and may control the line buffer 300 to output the image data corresponding to the reference pixel coordinates to the ODC 600. In some implementations, the read controller 500 sends the reference pixel coordinates to the line buffer 300, and the line buffer 300 reads the image data corresponding to the reference pixel coordinates and outputs the read image data to the ODC 600. The line buffer 300 stores the image data mapped to the input pixel coordinates, thereby reading the image data corresponding to the input pixel coordinates that are the same as the reference pixel coordinates. Here, the reference pixel coordinates may refer to the pixel coordinates required for the lens distortion correction of the ODC 600.

[0052] In addition, the read controller 500 may send the reference pixel coordinates to the line buffer 300 in response to a time point determined by a line spacing control value of the read speed controller 800. Here, the line spacing may refer to a time interval between a time point at which image data corresponding to any one line of the pixel array 120 is completely read and another time point at which image data corresponding to the next line is started to be read. That is, the line spacing may refer to a time section between read times of adjacent lines of the pixel array 120.

[0053] The line spacing control value may be information for determining the line spacing. As the line spacing control value increases, the line spacing may increase. As the line spacing control value decreases, the line spacing may decrease. If it is assumed that a time period from a time point at which image data corresponding to any one row of the pixel array 120 is started to be read to another time point at which image data corresponding to the next row is started to be read is defined as the output time, then the reading speed of the line buffer 300 (i.e., the amount of image data read from the line buffer 300 per unit time) may be inversely proportional to the output time. That is, as the output time is shortened due to a smaller line spacing, the reading speed of the line buffer 300 may be increased.

[0054] The read controller 500 may send the reference pixel coordinates to the line buffer 300 at a time determined by the line interval control value, so that the line buffer 300 may read the image data at intervals of a predetermined line interval corresponding to the line interval control value of the read speed controller 800. For example, after the reading of the image data corresponding to the current line ends, the read controller 500 may send the reference pixel coordinates to the line buffer 300 at a corresponding timing point, so that after 100 cycles have passed from the reading end point of the image data corresponding to the current line, the reading of the image data corresponding to the next line to be read may start. In this case, the cycle may refer to a clock cycle used in the imaging device 10. In addition, the cycle may refer to the time taken to generate and write the image data corresponding to one pixel, or may refer to the time taken to read the image data corresponding to one pixel and perform lens distortion correction on the read image data.

[0055] The ODC 600 may perform lens distortion correction on the image data, so that the ODC 600 may transmit the corrected image data to the ISP 900 .

[0056] Reference Figure 3 , together with the scene to be photographed (SC) and the original image data (OI) corresponding to the frame photographed by the image sensing unit 100. The lens module 110 may have a predetermined curvature to transmit light of a scene corresponding to a predetermined FOV to the pixel array 120, so that lens distortion caused by the difference between the scene (SC) and the original image data (OI) may occur due to such curvature of the lens module 110.

[0057] An image corresponding to a specific position included in the scene (SC) may not be sensed at a pixel disposed at the same position as the specific position but may be detected at another pixel disposed at a position different from the specific position.

[0058] For example, an image corresponding to an upper left vertex position within a scene (SC) may not be sensed at a first pixel (P1) corresponding to the same upper left vertex position within the original image data (OI), but may be sensed at a second pixel (P2) set at a position different from the upper left vertex position within the original image data (OI).

[0059] Alternatively, an image corresponding to a lower right vertex position within the scene (SC) may not be sensed at a third pixel (P3) corresponding to the same lower right vertex position within the original image data (OI), but may be sensed at a fourth pixel (P4) set at a position different from the lower right vertex position.

[0060] like Figure 3As shown, the scene (SC) is divided into a plurality of square regions, and the light from these different regions of the scene is Figure 2 110 and projected onto image sensing pixels in pixel array 120. When lens module 110 has no lens distortion, light patterns carried by light from different areas of the scene are projected by lens module 110 as projected light patterns on pixel array 120, which have the same relative position as the areas in the scene and maintain the same portion of the areas in the scene at pixel array 120. This undistorted projected light pattern is captured by image sensing pixels of pixel array 120 to generate image data representing multiple square areas as original image data (OI). However, in an actual device, different light rays from different areas of the scene pass through lens module 110 along different paths through lens module 110, so that lens distortion may occur in the light received at pixel array 120 due to the curvature and thickness distribution of one or more lenses in lens module 110. As a result, due to such lens distortion, the geometric shape of the light pattern projected by the lens module 110 onto the pixel array 120 is distorted, so that the plurality of square areas included in the scene (SC) may not correspond to the original image data (OI), but may correspond to the distorted image data (DI) including a plurality of distorted square areas, each of which is formed into a distorted shape. Due to the lens distortion, the square areas of the distorted image data (DI) may not perfectly correspond to the square areas of the scene (SC), respectively. For example, assuming that the lens module 110 has a lens distortion distribution that is radially symmetric with respect to the center of the lens module 110, the closer the square area of ​​the distorted image data (DI) is to the center of the lens module 110, the smaller the degree of distortion of the square area of ​​the distorted image data (DI). As the square area of ​​the distorted image data (DI) is positioned closer to the center of the distorted image data (DI), the degree of lens distortion becomes weaker, and as the square area of ​​the distorted image data (DI) is positioned farther from the center of the distorted image data (DI), the degree of lens distortion becomes stronger.

[0061] The position and shape of the distorted image data (DI) may vary depending on the curvature of each lens in the lens module 110, etc. Figure 3 The barrel distortion patterns illustrated in FIG. 1 are examples only.

[0062] Lens distortion correction may refer to image processing for correcting distortion caused by the lens module 110. Lens distortion correction for a specific pixel may refer to an operation for reading image data of a pixel corresponding to a reference pixel coordinate that matches the coordinates of the specific pixel, and processing the read image data. In some implementations, the processing operation for reading image data may refer to an operation for calculating / processing predetermined correction parameters for the read image data.

[0063] For example, lens distortion correction of the first pixel (P1) may include reading image data of the second pixel (P2) corresponding to reference pixel coordinates matching the coordinates of the first pixel (P1), and calculating / processing predetermined correction parameters using the read image data.

[0064] Reference Figure 4 , a first output row (OL1), a second output row (OL2), and a third output row (OL3) are illustrated. The first output row (OL1) may refer to a group of pixels corresponding to the first row in the pixel array 120, the second output row (OL2) may refer to a group of pixels corresponding to a row disposed closer to the center of the pixel array 120 (e.g., the 500th row among a total of 1080 rows), and the third output row (OL3) may refer to a group of pixels corresponding to the last row in the pixel array 120.

[0065] A group of pixels required for lens distortion correction of the first output line (OL1) may be represented by a first reference line (RL1) as part of the distorted image data (DI). The first reference line (RL1) may refer to a group of pixels corresponding to reference pixel coordinates that match coordinates of each pixel belonging to the first output line (OL1). The pixels included in the first reference line (RL1) may have a Y coordinate that is less than or equal to the first upper end coordinate (Yiu1) and greater than or equal to the first lower end coordinate (Yib1).

[0066] A group of pixels required for lens distortion correction of the second output line (OL2) may be represented by a second reference line (RL2) as part of the distorted image data (DI). The second reference line (RL2) may refer to a group of pixels corresponding to reference pixel coordinates that match coordinates of each pixel belonging to the second output line (OL2). The pixels included in the second reference line (RL2) may have a Y coordinate that is less than or equal to the second upper end coordinate (Yiu2) and greater than or equal to the second lower end coordinate (Yib2).

[0067] A group of pixels required for lens distortion correction of the third output line (OL3) may be represented by a third reference line (RL3) as part of the distorted image data (DI). The third reference line (RL3) may refer to a group of pixels corresponding to reference pixel coordinates that match coordinates of each pixel belonging to the third output line (OL3). The pixels included in the third reference line (RL3) may have a Y-axis coordinate that is less than or equal to the third upper end coordinate (Yiu3) and greater than or equal to the third lower end coordinate (Yib3).

[0068] Reference Figure 4, the range (Yiu2-Yib2) of the second reference line (RL2) may be smaller in size than the range (Yiu1-Yib1) of the first reference line (RL1) or the range (Yiu3-Yib3) of the third reference line (RL3). As the position of the lens distortion is closer to the center of the distorted image data (DI), the lens distortion becomes weaker. As the position of the lens distortion is farther from the center of the distorted image data (DI), the lens distortion becomes stronger.

[0069] Return to reference Figure 1 , the ODC 600 may determine whether to initiate lens distortion correction based on the input pixel coordinates received from the write controller 400. In the examples disclosed herein, a row corresponding to a target row reflecting the lens distortion correction to be performed by the ODC 600 will be referred to as an output row hereinafter.

[0070] The ODC 600 may pre-store a threshold lower end coordinate corresponding to the lower end coordinate of the reference row required to perform the lens distortion correction of the output row. The threshold lower end coordinate may be determined by experiment according to the distortion characteristics of the lens module 110. The ODC 600 may compare the threshold lower end coordinate of the output row with the Y coordinate of the input pixel coordinate received from the write controller 400, and may determine whether to start the lens distortion correction of the output row according to the comparison result. If the Y coordinate of the input pixel coordinate is greater than the threshold lower end coordinate, the ODC 600 may start the lens distortion correction of the output row. If the Y coordinate of the input pixel coordinate is less than or equal to the threshold lower end coordinate, the ODC 600 may not start the lens distortion correction of the output row, but may continue to monitor the value of the Y coordinate and wait until the Y coordinate of the input pixel coordinate becomes greater than the threshold lower end coordinate.

[0071] For example, the ODC 600 may store the first lower end coordinate (Yib1) of the first reference line (RL1) required for lens distortion correction of the first output line (OL1) as the threshold lower end coordinate of the first output line (OL1). The image sensing unit 100 may sequentially transmit image data corresponding to the Nth line from the image data corresponding to the first line of the pixel array 120 to the line buffer 300 based on the control of the timing controller 200. As the transmission of the image data is in progress, the Y coordinate of the input pixel coordinate may increase from 1 to N in sequence. Assuming that the threshold lower end coordinate of the first output line (OL1) is set to 30, the lens distortion correction of the first output line (OL1) may require image data corresponding to the first to third lines. The ODC 600 may wait without starting the lens distortion correction of the first output line (OL1) until the Y coordinate of the input pixel coordinate exceeds the coordinate value 30. When the Y coordinate of the input pixel coordinate exceeds the coordinate value 30, the ODC 600 may start the lens distortion correction of the first output line (OL1).

[0072] When starting the lens distortion correction of the output row, the ODC 600 may generate output pixel coordinates, which are coordinates of each pixel included in the corresponding output row reflecting the distortion correction performed by the ODC 600. The output pixel coordinates generated in this way are sent or transmitted to the distortion correction value storage 700 and stored in the distortion correction value storage 700. In some implementations, the ODC 600 may sequentially perform lens distortion correction for some pixels ranging from one pixel corresponding to the first column to another pixel corresponding to the Mth column within a specific output row. For example, when starting the lens distortion correction of the first output row (OL1), the ODC 600 may send the coordinates (1, 1) of the pixel corresponding to the first column within the first output row (OL1) to the distortion correction value storage 700 as the output pixel coordinates. Thereafter, when the lens distortion correction of the output pixel coordinate (1, 1) is completed, the ODC 600 may send the coordinate (2, 1) of the pixel corresponding to the second column within the first output row (OL1) as the output pixel coordinate to the distortion correction value storage 700. The above operation may be repeatedly performed until the lens distortion correction of the pixel corresponding to the Mth column is completed, so that the lens distortion correction of the first output row (OL1) may be completed.

[0073] The ODC 600 may receive image data corresponding to reference pixel coordinates from the line buffer 300, the reference pixel coordinates corresponding to the output pixel coordinates sent to the distortion correction value storage 700. The ODC 600 may receive correction parameters from the distortion correction value storage 700 in response to the output pixel coordinates sent to the distortion correction value storage 700. The ODC 600 may perform arithmetic processing on the image data received from the line buffer 300 using the correction parameters received from the distortion correction value storage 700, may generate corrected image data, and may send the corrected image data to the ISP 900. In some implementations, the arithmetic processing may be an operation for multiplying the image data by the correction parameters, but is not limited thereto.

[0074] The distortion correction value storage 700 may select reference pixel coordinates corresponding to the output pixel coordinates received from the ODC 600 and may transmit the selected reference pixel coordinates to the read controller 500. To this end, the distortion correction value storage 700 may store a first table in which the output pixel coordinates and the reference pixel coordinates are mapped to each other.

[0075] In addition, the distortion correction value storage 700 may select one or more correction parameters corresponding to the output pixel coordinates received from the ODC 600, and may transmit the selected correction parameters to the ODC 600. To this end, the distortion correction value storage 700 may store a second table in which the output pixel coordinates and the correction parameters are mapped to each other.

[0076] The first table and the second table may be determined through experiments based on the lens distortion of the lens module 110 .

[0077] In addition, the distortion correction value storage 700 may transmit the output pixel coordinates received from the ODC 600 to the reading speed controller 800 .

[0078] The reading speed controller 800 is a circuit that can select one or more line interval control values ​​corresponding to the output pixel coordinates received from the distortion correction value storage 700, and can transmit the selected line interval control values ​​to the reading controller 500. To this end, the reading speed controller 800 can store a third table in which the output pixel coordinates and the line interval control values ​​are mapped to each other.

[0079] The third table may be determined through experiments based on the lens distortion of the lens module 110 and the capacity of the line buffer 300 .

[0080] As described above, the read speed controller 800 can control the read speed of the line buffer 300 by adjusting the line spacing control value. In some implementations, the write speed of image data input to the line buffer 300 can be constant, and the read speed of image data output from the line buffer 300 can vary depending on the line spacing control value.

[0081] The ISP 900 may perform image processing of the corrected image data received from the ODC 600. The image signal processor 900 may reduce the noise of the image data, and may perform various types of image signal processing (e.g., gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, etc.) for image quality improvement of the image data. In addition, the ISP 900 may compress the image data (IDATA) that has been created by performing image signal processing for image quality improvement, so that the ISP 900 may create an image file using the compressed image data. Alternatively, the ISP 900 may restore the image data from the image file. In this case, the scheme for compressing such image data may be a reversible format or an irreversible format. As a representative example of such a compression format, in the case of using a still image, a Joint Photographic Experts Group (JPEG) format, a JPEG 2000 format, etc. may be used. In addition, in the case of using a moving image, a plurality of frames may be compressed according to the Moving Picture Experts Group (MPEG) standard, so that a moving image file may be created. For example, an image file may be created according to the Exchangeable Image File Format (Exif) standard.

[0082] The ISP 900 may transmit image data obtained through such image signal processing (hereinafter referred to as ISP image data) to the I / O interface 1000 .

[0083] The I / O interface 1000 may perform communication with the host device 20 and may transmit ISP image data to the host device 20. In some implementations, the I / O interface 1000 may be implemented as a Mobile Industry Processor Interface (MIPI), but is not limited thereto.

[0084] The host device 20 may be a processor (e.g., an application processor) for processing ISP image data received from the imaging device 10, a memory (e.g., a non-volatile memory) for storing ISP image data, or a display device (e.g., a liquid crystal display (LCD)) for visually displaying ISP image data.

[0085] Figure 5 is a graph illustrating a relationship between output pixel coordinates according to one example of lens distortion and input pixel coordinates required for lens distortion correction of a pixel corresponding to the output pixel coordinates. Figure 6 It is an example of Figure 5 The graph of FIG. 1 is a graph of the capacity of the line buffer required when image data is read at a relatively low read speed. Figure 7 It is an example of Figure 5 The graph of FIG. 1 is a graph of the capacity of the line buffer required when image data is read at a relatively high read speed.

[0086] Reference Figure 5 , the X-axis of the graph may represent the Y-coordinate (Yout) of the output pixel coordinate, and the Y-axis of the graph may represent the Y-coordinate (Yin) of the input pixel coordinate. In the following description, it is assumed that the pixel array 120 includes 1080 rows and 1920 columns. Therefore, each of the Y-coordinate (Yin) of the input pixel coordinate and the Y-coordinate (Yout) of the output pixel coordinate may have a range of 0 to 1080.

[0087] From an example showing lens distortion Figure 5 It can be seen that Figure 5 A graph illustrating how the Y coordinate (Yin) of the input pixel coordinates required for lens distortion correction of each pixel having the Y coordinate (Yout) of the output pixel coordinates changes according to the change of the Y coordinate (Yout) of the output pixel coordinates. In this case, pixels having the same Y coordinate (Yout) of the output pixel coordinates can be defined as Figure 4 The output row shown, and a set of pixels corresponding to the input pixel coordinates required for lens distortion correction of the output row can be defined as Figure 4 The reference line shown. Figure 4The example illustrates that the reference row is a group of pixels corresponding to reference pixel coordinates that match the coordinates of each pixel belonging to the output row, but through the operation of the read controller 500 and the row buffer 300, image data corresponding to input pixel coordinates that are the same as the reference pixel coordinates are read and sent to the ODC 600, so for ease of description, the terms "reference pixel coordinates" and "input pixel coordinates" in the following description will be used interchangeably.

[0088] exist Figure 5 In the embodiment of the present invention, as the upper end coordinate (Yiu-A) and the lower end coordinate (Yib-A) of the reference row corresponding to the output row change, the Y coordinate (Yin) of the input pixel coordinate required for lens distortion correction of each pixel having the Y coordinate (Yout) of the output pixel coordinate (i.e., the pixel belonging to the output row) may change. That is, the pixels included in the reference row corresponding to each output row may have a Y coordinate corresponding to the range between the upper end coordinate (Yiu-A) and the lower end coordinate (Yib-A), and the upper end coordinate (Yiu-A) and the lower end coordinate (Yib-A) may change depending on the Y coordinate of the output row.

[0089] exist Figure 5 In one example of lens distortion shown, as the Y coordinate of the output row gradually increases, the difference between the upper end coordinate (Yiu-A) and the lower end coordinate (Yib-A) gradually decreases until it reaches the central portion of the pixel array 120, and then gradually increases after passing through the central area of ​​the pixel array 120 (e.g., the portion where the Y coordinate of the output row is set to 540).

[0090] Reference Figure 6 , illustrates an upper storage coordinate (LBu-A1) and a lower storage coordinate (LBb-A1) of the line buffer 300 when image data is read from the line buffer 300 at a relatively low reading speed. The upper storage coordinate (LBu-A1) may refer to the minimum Y coordinate among the Y coordinates of the pixels stored in the line buffer 300 when the distortion of the output line is corrected, and the lower storage coordinate (LBb-A1) may refer to the maximum Y coordinate among the Y coordinates of the pixels stored in the line buffer 300 when the distortion of the output line is corrected.

[0091] The reading speed of the line buffer 300 may correspond to the slope of the upper storage coordinate (LBu-A1) or the lower storage coordinate (LBb-A1) in the X-axis direction relative to the Y-axis direction. That is, the reading speed of the line buffer 300 may correspond to the increasing speed of the upper storage coordinate (LBu-A1) or the lower storage coordinate (LBb-A1) relative to the Y coordinate (Yin) of the input pixel coordinate increasing at a constant speed. Figure 6The relatively low reading speed may refer to the same speed as the speed of inputting the Y coordinate (Yin) of the pixel coordinates. Therefore, the speed at which the image data is input from the image sensing unit 100 to the line buffer 300 may be equal to the speed at which the image data is read from the line buffer 300 and sent to the ODC 600 (or the speed of lens distortion correction of the ODC 600).

[0092] The capacity of the line buffer 300 may be determined within the range in which the line buffer 300 can store image data corresponding to a reference line that matches each output line. Figure 6 As shown, when the read speed of the line buffer 300 is relatively low, the minimum capacity of the line buffer 300 may correspond to a capacity capable of storing 64 rows (or 64 lines) of image data.

[0093] Reference Figure 7 , which illustrates the Figure 6 The upper storage coordinate (LBu-A2) and the lower storage coordinate (LBb-A2) of the line buffer 300 are compared when the image data is read from the line buffer 300 at a relatively low reading speed. The upper storage coordinate (LBu-A2) may refer to the minimum Y coordinate among the Y coordinates of the pixels stored in the line buffer 300 when the distortion of the output line is corrected, and the lower storage coordinate (LBb-A2) may refer to the maximum Y coordinate among the Y coordinates of the pixels stored in the line buffer 300 when the distortion of the output line is corrected.

[0094] Figure 7 The read speed of the line buffer 300 shown can be faster than Figure 6 The reading speed of the line buffer 300 shown is relatively faster. Therefore, the slope of the upper storage coordinate (LBu-A2) or the lower storage coordinate (LBb-A2) in the X-axis direction relative to the Y-axis direction can be greater than the slope of the upper storage coordinate (LBu-A1) or the lower storage coordinate (LBb-A1) in the X-axis direction relative to the Y-axis direction.

[0095] Figure 7 The relatively high reading speed may refer to a speed higher than the Y coordinate (Yin) of the input pixel coordinates. That is, the speed of reading image data from the line buffer (or the speed of lens distortion correction by ODC 600) may be higher than the speed of inputting image data from the image sensing unit 100 to the line buffer 300.

[0096] like Figure 7 As shown, when the read speed of the line buffer 300 is relatively high, the minimum capacity of the line buffer 300 may correspond to a capacity capable of storing 32 raw (or 32 lines) of image data.

[0097] exist Figure 5 In one example of lens distortion shown, the minimum capacity of line buffer 300 can be significantly reduced by adjusting the read speed of line buffer 300 .

[0098] exist Figure 5 In one example of the lens distortion shown, the minimum capacity of the line buffer 300 can be reduced by increasing the read speed of the line buffer 300, but is not limited thereto. As another example, the minimum capacity of the line buffer 300 can be reduced by reducing the read speed of the line buffer 300 according to the shape of the lens distortion.

[0099] Figure 8 (a) and Figure 8 (b) in FIG. 3 is a diagram illustrating an example of a method for adjusting the read speed of the line buffer 300 .

[0100] In more detail, Figure 8 (a) in FIG. 1 illustrates the Figure 6 An example of an output time achieved in a case where the writing speed at which the image data is written into the line buffer 300 is equal to the reading speed at which the image data is read from the line buffer 300 is shown. Here, the output time may be 2200 cycles, which corresponds to the sum of the reading time of 1920 cycles and the line interval of 280 cycles. In this case, the reading time may refer to the time taken for the image data corresponding to one row (or one output row) to be read from the line buffer 300 and processed by the ODC 600 for lens distortion correction. Since the reading and lens distortion correction of the image data corresponding to one pixel (hereinafter referred to as the reading and lens distortion correction operation) are performed during one cycle, the reading and lens distortion correction of the image data corresponding to the row (or output row) including 1920 pixels may be performed using the reading time of 1920 cycles.

[0101] Figure 8 (b) in FIG. 1 shows an example of Figure 7 The illustrated example is an example of the output time in which the reading speed of the image data from the line buffer 300 is made faster than the writing speed of the image data into the line buffer 300 .

[0102] In order to make the line buffer 300 have Figure 7 The upper storage coordinate (LBu-A2) and the lower storage coordinate (LBb-A2) shown in the curve graph should complete the reading of image data corresponding to 1080 lines and lens distortion correction at the same time as the image data corresponding to (1080-32) lines is input to the line buffer 300.

[0103] In this case, the output time can be calculated as "2200 (cycle) × (1080-32) / 1080 ≒ 2135 (cycle)". That is, after the output time is adjusted to 2135 cycles, when the image data corresponding to (1080-32) lines is input to the line buffer 300, the reading of the image data corresponding to 1080 lines and the lens distortion correction can be completed.

[0104] Here, since it is impossible to reduce the reading time of 1920 cycles in which the reading of image data corresponding to a row (or output row) including 1920 pixels and the lens distortion correction operation are performed, the row interval can be reduced from 280 cycles to 215 cycles, as shown in FIG. Figure 8 As shown in (b) in .

[0105] That is, the reading speed can be increased by reducing the line interval of the output time.

[0106] Fig. 9 (a) and Fig. 9 (b) in FIG. 3 shows a diagram illustrating another example of a method for adjusting the read speed of the line buffer 300 .

[0107] Fig. 9 (a) in FIG. 1 illustrates the Figure 6 Another example of output time achieved when the writing speed of writing image data to the line buffer 300 is equal to the reading speed of reading image data from the line buffer 300 is shown. The output time may be 2000 cycles, which corresponds to the sum of the reading time of 1920 cycles and the line interval of 80 cycles. Therefore, Figure 8 The row spacing shown in (b) can be relatively smaller than Figure 8 The row spacing shown in (a) in .

[0108] Fig. 9 (b) in FIG. 1 shows an example of Figure 7 Another example of the output time is shown in which the reading speed of the image data from the line buffer 300 is made faster than the writing speed of the image data to the line buffer 300 .

[0109] Given that Figure 8 (a) and Figure 8 The implementation method of (b) can improve the reading speed by reducing the line spacing. Fig. 9 The output time of (a) has a relatively small line interval, so that Figure 8 It is difficult to reduce the output time in the implementation method.

[0110] Fig. 9 (a) and Fig. 9The implementation of (b) in allows the reading speed to be increased by reducing the line interval. The line interval can be reduced by increasing the clock frequency on the output side. In this case, the output frequency on the output side may refer to the frequency of a clock signal for reading image data from the line buffer 300 and performing lens distortion correction on the read image data. The clock frequency on the input side may refer to the frequency of a clock signal for generating image data by the image sensing unit 100 and writing the generated image data to the line buffer 300.

[0111] The read speed controller 800 in which the line interval control value is stored can control the clock frequency of the output side. For example, the read speed controller 800 can control a clock signal generator (not shown) for providing a clock signal to each of the read controller 500, the ODC 600, and the distortion correction value storage 700, thereby changing the clock frequency.

[0112] exist Fig. 9 In (b), when the clock frequency on the output side is doubled (for example, from 80 MHz to 160 MHz), the corresponding Fig. 9 The time of one cycle shown in (b) can be reduced to the corresponding Fig. 9 The time of one cycle of (a) is half. Therefore, Fig. 9 The read time shown in (b) can be reduced to Fig. 9 The reading time is half of that shown in (a).

[0113] Therefore, the line interval can be reduced by as much as an amount corresponding to 2080 cycles. By reducing the line interval, the reading speed controller 800 can increase the reading speed.

[0114] Figure 8 (a) and Figure 8 The implementation method of (b) and Fig. 9 (a) and Fig. 9 The embodiments of (b) in the above may be combined with each other and are not mutually exclusive. For example, the read speed controller 800 can be configured according to Figure 8 (a) and Figure 8 The implementation method (b) in the embodiment reduces the line spacing to increase the reading speed, and can also be implemented according to Fig. 9 (a) and Fig. 9 The implementation method (b) in FIG. 1 increases the clock frequency on the output side and then reduces the line interval to control the line interval. Figure 8 (a) Figure 8 (b) Fig. 9 (a) and Fig. 9The combined configuration of the embodiment of (b) in FIG. 1 can be used to solve the limitation of controlling the reading speed using only the line interval. By increasing the clock frequency on the output side and then reducing the line interval, according to Figure 8 (a) Figure 8 (b) Fig. 9 (a) and Fig. 9 The combined configuration of the embodiments of (b) can provide even higher reading speeds.

[0115] Fig.10 is a graph illustrating a relationship between output pixel coordinates according to another example of lens distortion and input pixel coordinates required for lens distortion correction of pixels corresponding to the output pixel coordinates. Fig.11 is an example of Figure 7 The capacity and read speed of the same capacity and the same read speed of the line buffer with Fig.10 A graph showing the relationship between lens distortion. Fig.12 It is an example of Fig.10 The graph of FIG. 1 is a graph of the capacity of the line buffer required when reading image data at a constant reading speed. Fig.13 It is an example of Fig.10 The graph of FIG. 1 is a graph of the capacity of the line buffer required when reading image data at a variable reading speed.

[0116] From an example showing lens distortion Fig.10 It can be seen that Fig.10 A graph illustrating how the Y coordinate (Yin) of the input pixel coordinates required for lens distortion correction of each pixel having the Y coordinate (Yout) of the output pixel coordinates changes according to the change of the Y coordinate of the output pixel coordinates (Yout).

[0117] In with Figure 5 Different ways Fig.10 In one example of lens distortion, as the Y coordinate of the output row gradually increases, the difference between the upper end coordinate (Yiu-B) and the lower end coordinate (Yib-B) remains constant and then gradually decreases until it reaches the central portion of the pixel array 120. Thereafter, the difference between the upper end coordinate (Yiu-B) and the lower end coordinate (Yib-B) gradually increases after passing through the central area of ​​the pixel array 120, and then remains constant. In the section where the difference between the upper end coordinate (Yiu-B) and the lower end coordinate (Yib-B) remains constant, it is assumed that the difference between the upper end coordinate (Yiu-B) and the lower end coordinate (Yib-B) is set to 32. In addition, as shown in FIG. Fig.10 It can be seen from the curve graph that the maximum value between the upper coordinate (Yiu-B) and the lower coordinate (Yib-B) can be set to 32.

[0118] Reference Fig.11 , illustrating the Figure 7 The upper storage coordinate (LBu-A2) and the lower storage coordinate (LBb-A2) of the line buffer 300 having the same capacity and the same reading speed are also illustrated according to Fig.10 The upper end coordinates (Yiu-B) and the lower end coordinates (Yib-B) of another example of lens distortion are stored.

[0119] Since there is a segment in which the difference between the upper end coordinate (Yiu-B) and the lower end coordinate (Yib-B) remains constant, the upper end coordinate (Yiu-B) within the first segment 1110 may deviate from the range between the upper end storage coordinate (LBu-A2) and the lower end storage coordinate (LBb-A2) of the line buffer 300, or the lower end coordinate (Yib-B) within the second segment 1120 may deviate from the range between the upper end storage coordinate (LBu-A2) and the lower end storage coordinate (LBb-A2) of the line buffer 300.

[0120] In the first section 1110 or the second section 1120 , the line buffer 300 may not store image data of a reference line required for lens distortion correction of a corresponding output line.

[0121] Reference Fig.12 , illustrating the Figure 7 The upper storage coordinate (LBu-B1) and the lower storage coordinate (LBb-B1) of the line buffer 300 capable of storing image data of a reference line required for lens distortion correction of an output line at the same reading speed are also illustrated. Fig.10 The upper coordinates (Yiu-B) and lower coordinates (Yib-B) of another example of lens distortion are shown.

[0122] In order for the line buffer 300 to store image data of reference lines required for lens distortion correction of output lines within all segments of the Y coordinate (Yout) of the output pixel coordinates, the minimum capacity of the line buffer 300 can be set to a capacity capable of storing K lines (raw) (or K lines) of image data (where K is an integer of 32 or greater).

[0123] Depending on the type of lens distortion, in order for the line buffer 300 to store image data of reference lines required for lens distortion correction of output lines within all segments of the Y coordinate (Yout) of the output pixel coordinates, the line buffer 300 may have a capacity greater than the maximum difference between the upper coordinate (Yiu-B) and the lower coordinate (Yib-B) of the lens distortion.

[0124] Reference Fig.13, illustrating a lens having the same capacity as the maximum value between the upper end coordinate (Yiu-B) and the lower end coordinate (Yib-B) of the lens distortion and with Figure 7 Different ways have an upper storage coordinate (LBu-B2) and a lower storage coordinate (LBb-B2) of the line buffer 300 with a variable reading speed, and also illustrate the Fig.10 The upper coordinates (Yiu-B) and lower coordinates (Yib-B) of another example of lens distortion are shown.

[0125] exist Fig.13 , as the Y coordinate of the output line gradually increases, the reading speed of the line buffer 300 can be changed in sequence in the order of the first speed→the second speed→the first speed. Here, the first speed can be higher than the second speed. The slope between the upper storage coordinate (LBu-B2) and the lower storage coordinate (LBb-B2) of the line buffer 300 remains equal to the slope between the upper coordinate (Yiu-B) and the lower coordinate (Yib-B) of the lens distortion (i.e., in the section where the reading speed is set to the first speed), changes to be less than the slope between the upper coordinate (Yiu-B) and the lower coordinate (Yib-B) of the lens distortion (i.e., in the section where the reading speed is set to the second speed), and then changes to be equal to the slope between the upper coordinate (Yiu-B) and the lower coordinate (Yib-B) of the lens distortion (i.e., in the section where the reading speed is set to the first speed).

[0126] The reading speed of the line buffer 300 is as follows Fig.13 In the case where the line buffer 300 can be changed, even when the line buffer 300 has the same capacity as the maximum difference between the upper end coordinate (Yiu-B) and the lower end coordinate (Yib-B) of the lens distortion (i.e., 32), the line buffer 300 can store the image data of the reference line required for the lens distortion correction of the output line in all sections of the Y coordinate (Yout) of the output pixel coordinate. That is, by changing the reading speed of the line buffer 300, the capacity of the line buffer 300 can be minimized.

[0127] like Figures 5 to 13 As shown, the reading speed of the line buffer 300 can be constant or variable, but the reading speed controller 800 can determine the reading speed in the following manner: the image data of the reference line required for lens distortion correction of the output line within all segments of at least the Y coordinate (Yout) of the output pixel coordinate can be maintained in the line buffer 300.

[0128] Fig.14 is an example of reading at a constant reading speed Figure 3 FIG. 5 is a diagram of distorted image data and the capacity of a line buffer required when performing lens distortion correction on the distorted image data.

[0129] Reference Fig.14 , exemplifies Figure 3 The original image data (OI) and the distorted image data (DI) are plotted in FIG. Associated with each of the five output lines (Ola to OLe), Fig.14 , the image data stored in the line buffer 300, the capacity of the line buffer 300 required for lens distortion correction, and the operation of the I / O interface 1000 are illustrated. In this case, the output line (OLa) and the output line (OLe) may correspond to the first output line (OL1) and the third output line (OL3), respectively.

[0130] First, the reference lines corresponding to the output lines (OLa) and the output lines (OLe) may be distributed over 65 lines (corresponding to about 6% of 1080 lines). Therefore, the capacity of the line buffer 300 required for lens distortion correction of the output lines (OLa) and the output lines (OLe) may correspond to a capacity capable of storing 65 lines (or 65 rows) of image data.

[0131] The reference lines corresponding to the output lines (OLb) and the output lines (OLd) may be distributed over 30 lines (corresponding to about 2.8% of 1080 lines). Therefore, the capacity of the line buffer 300 required for lens distortion correction of the output lines (OLb) and the output lines (OLd) may correspond to a capacity capable of storing image data of 30 lines (or 30 rows).

[0132] The reference lines corresponding to the output lines (OLc) may be distributed over 7 lines (corresponding to about 0.6% of 1080 lines). Therefore, the capacity of the line buffer 300 required for lens distortion correction of the output lines (OLc) may correspond to a capacity capable of storing image data of 7 lines (or 7 rows).

[0133] Therefore, the capacity of the line buffer 300 required for lens distortion correction gradually decreases as the line buffer 300 is disposed closer to the center of the pixel array 120 , and gradually increases as the line buffer 300 is disposed farther from the center of the pixel array 120 .

[0134] exist Fig.14 , the image data written into the line buffer 300 is displayed as input image data (input), and the image data read from the line buffer 300 is displayed as output image data (output). It is assumed that the reading speed of the data from the line buffer 300 is equal to the writing speed of the data written into the line buffer 300 and is constant. In addition, the speed at which the ODC 600 performs lens distortion correction can also be constant, so that the length of the output section of the I / O interface 1000 can also be kept constant. Fig.14 An example of an output section in which the I / O interface 1000 is a Mobile Industry Processor Interface (MIPI) is illustrated. Fig.14 In the mobile industry processor interface (MIPI), the high speed mode (HS) indicates a time period for outputting ISP image data corresponding to one line, and may correspond to Figure 8 The reading time is shown. Fig.14 In the mobile industry processor interface (MIPI), the low power mode (LP) represents the time period from one HS of any row to another HS of the next row, and may correspond to Figure 8 That is, in Fig.14 In the embodiment of the present invention, the speed of reading image data from the line buffer 300 and performing lens distortion correction on the read image data is constant, so that the low power mode (L) corresponding to the line interval can be kept constant.

[0135] like Fig.14 As shown, if the speed at which data is read from the line buffer 300 and lens distortion correction is performed on the read image data is constant, the line buffer 300 should have a capacity (e.g., 130 lines, approximately 12% of 1080 lines) corresponding to twice the maximum capacity value of the line buffer 300 (e.g., 65 lines) required for the lens distortion correction.

[0136] Fig.15 is an example of reading at a variable reading speed Figure 3 FIG. 4 is a diagram of the capacity of the line buffer required when distorted image data is displayed and lens distortion correction is performed on the distorted image data.

[0137] Reference Fig.15 , exemplifies Figure 3 The original image data (OI) and the distorted image data (DI) are plotted in FIG. Associated with each of the five output lines (Ola to OLe), Fig.15 2 illustrates image data stored in the line buffer 300 , the capacity of the line buffer 300 required for lens distortion correction, and the operation of the I / O interface 1000 .

[0138] exist Fig.15 In, with Fig.14 Differently, such as Fig.15 The speed at which image data is read from line buffer 300 and lens distortion correction is performed on the read image data is shown to be variable rather than constant.

[0139] In some implementations, the speed at which image data is read from the line buffer 300 and the speed at which the ODC 600 performs lens distortion correction may vary according to the capacity of the line buffer 300 required for lens distortion correction (i.e., the number of one or more reference lines corresponding to output lines). As the capacity of the line buffer 300 required for lens distortion correction decreases (or as the number of one or more reference lines corresponding to output lines decreases), the speed at which image data is read from the line buffer 300 and the speed at which the ODC 600 performs lens distortion correction may increase. Thereafter, as the capacity of the line buffer 300 required for lens distortion correction increases (or as the number of one or more reference lines corresponding to output lines increases), the speed at which image data is read from the line buffer 300 and the speed at which the ODC 600 performs lens distortion correction may decrease.

[0140] like Figure 8 As described in the above, by adjusting the length of the line interval, the speed of reading image data from the line buffer 300 and the speed of performing lens distortion correction by the ODC 600 can be controlled. Fig.15 The speed of reading image data from the line buffer 300 and performing lens distortion correction for reading the image data is shown to be capable of being changed, so that the length of the low power mode (LP) (i.e., LP length) decreases as the capacity of the line buffer 300 required for the lens distortion correction decreases, and then increases as the capacity of the line buffer 300 required for the lens distortion correction increases.

[0141] For example, assuming that the number of one or more reference lines required for lens distortion correction of the output lines corresponding to the Y coordinates (Yout) of 300 to 310 is set to 20 (i.e., 20 lines), the number of one or more reference lines required for lens distortion correction of the output lines corresponding to the Y coordinates (Yout) of 311 to 326 is set to 19 (i.e., 19 lines), and the number of one or more reference lines required for lens distortion correction of the output lines corresponding to the Y coordinates (Yout) of 327 to 344 is set to 18 (i.e., 18 lines). In addition, assuming that the output time of the output lines corresponding to the Y coordinates (Yout) of 300 to 310 is represented by 2200 cycles.

[0142] While performing lens distortion correction for output lines corresponding to 16 Y coordinates (Yout) of 311 to 326, in order to increase the reading speed in response to a decrease in the capacity of the line buffer 300 required for indicating lens distortion correction by one line, the output time of the lens distortion correction for one output line can be reduced by approximately 138 cycles (2200 / 16=137.5), thereby forming a final output time represented by 2062 cycles (2200-138=2062 cycles).

[0143] When performing lens distortion correction for output lines corresponding to 18 Y coordinates (Yout) of 327 to 344, in order to increase the reading speed in response to a decrease in the capacity of the line buffer 300 required for indicating lens distortion correction by one line, the output time of the lens distortion correction for one output line can be reduced by approximately 123 cycles (2200 / 18=122.2), thereby forming a final output time represented by 2077 cycles (2200-123=2077 cycles).

[0144] like Fig.15 As shown, when the speed of reading image data from the line buffer 300 and performing lens distortion correction is changed in response to the capacity of the line buffer 300 required for lens distortion correction, the use efficiency of the line buffer 300 can be maximized.

[0145] In this case, if Figure 7 or Fig.13 As shown, the line buffer 300 has the same capacity as the maximum capacity of the line buffer 300 required for lens distortion correction (e.g., 65 lines), so that the reading speed of the line buffer 300 can be appropriately determined considering the shape of the lens distortion, the capacity of the line buffer 300, the performance of the ODC 600, etc.

[0146] It is apparent from the above description that imaging devices based on some implementations of the disclosed technology can minimize the capacity required for the line buffer by changing the reading speed of the line buffer.

[0147] Embodiments of the disclosed technology can provide various effects that can be directly or indirectly identified through the above-mentioned patent documents.

[0148] Although a number of exemplary embodiments have been described, it should be understood that modifications and improvements of the disclosed embodiments and other embodiments may be designed based on what is described and / or illustrated in this patent document.

[0149] CROSS-REFERENCE TO RELATED APPLICATIONS

[0150] This patent document claims priority to and the benefit of Korean Patent Application No. 10-2021-0109534, filed on August 19, 2021, the disclosure of which is incorporated herein by reference in its entirety as a part of the disclosure of this patent document.

Claims

1. An imaging device, comprising: an image sensing unit comprising image sensing pixels operable to sense incident light received from a scene and generate image data carrying image information of the scene; a lens module positioned to project the incident light from the scene onto the image sensing pixels of the image sensing unit; a line buffer coupled to communicate with the image sensing unit and storing the image data received from the image sensing unit; an optical distortion corrector ODC that performs lens distortion correction for output pixel coordinates based on input pixel coordinates of the image data stored in the line buffer to correct distortion caused by the lens module; as well as A line buffer controller includes a read controller and is coupled to the line buffer, and controls a read speed of the image data from the line buffer to the optical distortion corrector according to the output pixel coordinates.

2. The imaging device according to claim 1, further comprising: a distortion correction value storage that selects reference pixel coordinates corresponding to the output pixel coordinates; and The line buffer controller controls the line buffer to read image data corresponding to the reference pixel coordinates.

3. The imaging device according to claim 2, wherein: The distortion correction value storage sends correction parameters corresponding to the output pixel coordinates to the optical distortion corrector ODC.

4. The imaging device according to claim 3, wherein: The optical distortion corrector ODC receives image data corresponding to the reference pixel coordinates corresponding to the output pixel coordinates from the line buffer, calculates the image data corresponding to the reference pixel coordinates and correction parameters corresponding to the output pixel coordinates, and performs the lens distortion correction based on the calculation result.

5. The imaging device according to claim 2, wherein: The line buffer controller controls the reading speed using a line interval between reading times of adjacent lines of a pixel array included in the image sensing unit.

6. The imaging device according to claim 5, wherein: The line buffer controller increases the reading speed by reducing the line interval, or decreases the reading speed by increasing the line interval.

7. The imaging device according to claim 5, wherein: The line buffer controller determines the reading speed so that the image data corresponding to a reference line corresponding to an output line including the output pixel coordinates is held in the line buffer.

8. The imaging device according to claim 5, wherein: The line buffer controller is operable to decrease the line interval as the number of one or more reference lines corresponding to an output line including the output pixel coordinate decreases.

9. The imaging device according to claim 5, wherein: The line buffer controller is operable to increase the line interval as the number of one or more reference lines corresponding to an output line including the output pixel coordinate increases.

10. The imaging device according to claim 5, wherein: The reading speed is higher than a writing speed of writing the image data into the line buffer.

11. The imaging device according to claim 5, wherein: The line buffer controller controls the reading speed using the line interval after a clock frequency of each of the reading controller and the optical distortion corrector (ODC) is increased.

12. The imaging device according to claim 2, further comprising: A write controller sends the input pixel coordinates to each of the line buffer and the optical distortion corrector (ODC).

13. The imaging device according to claim 12, wherein: The line buffer stores the input pixel coordinates mapped to the image data; and The line buffer reads image data corresponding to input pixel coordinates that are the same as the reference pixel coordinates received from the read controller.

14. The imaging device according to claim 12, wherein: The optical distortion corrector ODC compares lower end coordinates of a reference line corresponding to an output line including the output pixel coordinates with the input pixel coordinates, and determines whether to start lens distortion correction of the output pixel coordinates based on the comparison result.

15. An imaging device, comprising: a line buffer storing image data generated by sensing incident light; an optical distortion corrector (ODC) that receives image data from the line buffer and performs lens distortion correction for output pixel coordinates; as well as A line buffer controller controls a reading speed of the image data from the line buffer based on a reference line corresponding to an output line including the output pixel coordinates.

16. The imaging device according to claim 15, wherein: The line buffer controller causes the reading speed to be based on the capacity of the line buffer.

17. The imaging device according to claim 15, wherein: The row buffer controller controls the reading speed using a row interval between reading times of adjacent rows of the pixel array.

18. The imaging device according to claim 15, wherein: The line buffer controller changes the reading speed by changing the line interval.

19. The imaging device according to claim 15, wherein: The line buffer controller controls the line buffer to read image data corresponding to a reference pixel.

20. The imaging device according to claim 19, wherein The row buffer controller is operable to change the read speed after a clock frequency associated with the row buffer controller is increased.

Images