Image signal processor and imaging system including the same

Abstract

The invention relates to an image signal processor and an imaging system including the same. An imaging system includes a frame memory component configured to perform a read operation or a write operation of image data. The imaging system also includes an image signal processor configured to write input image data into the frame memory component in a predetermined order, process data read from the frame memory component, and output the processed image data. The image signal processor converts an order in which the image data is processed into a direction different from a direction in which the image data is input, processes the result data, aligns the order of the processed image data in a predetermined order, and outputs the result data.

Description

Technical Field

[0001] The techniques and implementations disclosed in this patent document generally relate to an imaging system capable of generating image data by sensing light. Background Art

[0002] Image sensors are devices that capture optical images by converting light into electrical signals using photosensitive semiconductor materials that react to light. With the development of the automotive, medical, computer and communication industries, the demand for high-performance image sensing devices is increasing in various devices such as smartphones, digital cameras, game consoles, IOT (Internet of Things), robots, security cameras and medical micro cameras.

[0003] As image sensing device technology develops, the amount of data captured by the image sensing device increases rapidly. As the amount of image data increases, more buffers are required to buffer the image data, and the speed of processing the image data may increase. Therefore, there is a need for image signal processing technology that not only reduces the number of buffers configured to process image data, but also increases the speed of processing image data. Summary of the invention

[0004] According to an embodiment of the disclosed technology, an imaging system may include a frame memory component configured to perform a read operation or a write operation of image data. The imaging system may also include an image signal processor configured to write input image data into the frame memory component in a predetermined order, process the data read from the frame memory component, and output the processed image data. The image signal processor may convert the order of processing the image data into a direction different from the direction in which the image data is input, may process the result data, may align the order of the processed image data in a predetermined order, and may output the result data.

[0005] According to an embodiment of the disclosed technology, an image signal processor may include a frame memory component configured to store image data. The image signal processor may also include a buffer circuit configured to write input image data into the frame memory component in a predetermined order, process data read from the frame memory component, write the processed data into the frame memory component, and output the processed image data read from the frame memory component. When image data is input in a first direction, the buffer circuit may convert the order of processing the image data in a second direction, may process the result data, may align the order of the processed image data in a predetermined order, and may output the result data.

[0006] According to an embodiment of the disclosed technology, an image signal processor includes: a buffer circuit, which is configured to receive image data in a first orientation direction in which the image data is raster scanned; a frame memory component, which is configured to receive image data from the buffer circuit in the first orientation direction and store the image data; an image signal processor, which is configured to read the image data in the first orientation direction through the buffer circuit, change the orientation direction of the image data to a second orientation direction different from the first orientation direction, change the processed image data back to the first orientation direction, and output the processed image data in the first orientation direction through the buffer circuit. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other features and advantageous aspects of the disclosed technology will become readily apparent with reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0008] Figure 1 is a block diagram illustrating an example of an imaging system based on some implementations of the disclosed technology.

[0009] Figure 2 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of the operation of an image signal processor is shown.

[0010] Figure 3 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example case where data input is performed vertically in an image signal processor shown.

[0011] Figure 4 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example case where data input is performed horizontally in an image signal processor shown.

[0012] Figure 5 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of the operation of a data processor in an image signal processor is shown.

[0013] Figure 6 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of an access operation of an image signal processor is shown.

[0014] Figure 7 and Figure 8 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of a write operation of an image signal processor is shown.

[0015] Figures 9 to 11Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of a read operation of an image signal processor is shown.

[0016] Fig.12 and Fig.13 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of a data alignment process during read and write operations in an image signal processor is shown.

[0017] Fig.14 is an access method showing some implementations based on the disclosed technology Figure 1 A conceptual diagram of an example of the operation of outputting data of a frame memory component is shown. DETAILED DESCRIPTION

[0018] This patent document provides implementations and examples of imaging systems capable of generating image data by sensing light, which can be used in a configuration that substantially solves one or more technical or engineering problems and alleviates limitations or disadvantages encountered in some imaging systems in the art. Some implementations of the disclosed technology relate to an imaging system capable of reducing the size of a buffer of an image signal processor and reducing the data processing time of the image signal processor. Recognizing the above problems, an imaging system based on some implementations of the disclosed technology can reduce the size of a buffer of an image signal processor and reduce the data processing time of the image signal processor.

[0019] Reference will now be made in detail to some embodiments of the disclosed technology, examples of which are shown in the accompanying drawings. Whenever possible, the same reference numerals will be used throughout the accompanying drawings to represent the same or similar parts. Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are shown as examples in the accompanying drawings. However, the present disclosure should not be construed as being limited to the embodiments set forth herein.

[0020] Hereinafter, various embodiments are described with reference to the accompanying drawings. However, it should be understood that the disclosed technology is not limited to a specific embodiment, but includes various modifications, equivalents and / or substitutions of the embodiments. The embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized by the disclosed technology.

[0021] It will be understood that both the above general description and the following detailed description of the disclosed technology are exemplary and illustrative and are intended to provide further explanation of the claimed disclosure. Various embodiments of the disclosed technology are directed to an imaging system capable of reducing the size of a buffer of an image signal processor and the data processing time of the image signal processor.

[0022] Figure 1is a block diagram illustrating an example of an imaging system 1 based on some implementations of the disclosed technology.

[0023] Reference Figure 1 , the imaging system 1 may refer to a device such as a digital still camera for capturing still images or a digital video camera for capturing moving images. For example, the imaging device 10 may be implemented as a digital single-lens reflex (DSLR) camera, a mirrorless camera, or a smart phone, etc. The imaging device 10 may include a device having both a lens and an image pickup element, so that the device can capture (or photograph) a target object, and thus can create an image of the target object.

[0024] The imaging system 1 may include an imaging device 10 , a frame memory component 20 including a frame memory, and a host device 30 .

[0025] Here, the imaging device 10 may include an image sensor 100 , a timing controller 110 , a readout circuit 120 , an image signal processor 200 , and an input / output (I / O) interface 300 .

[0026] The image sensor 100 may be or may include a complementary metal oxide semiconductor image sensor (CIS) for converting an optical signal into an electrical signal. The image sensor 100 may control overall operations such as on / off, operation mode, operation timing, sensitivity, etc. through the timing controller 110. The image sensor 100 may provide image data obtained by converting the optical signal into an electrical signal to the readout circuit 120 under the control of the timing controller 110.

[0027] The timing controller 170 may provide the image sensor 100 with clock signals required for the operation of the various components of the image sensor 100, control signals for timing control, and address signals for selecting rows or columns. In an embodiment of the disclosed technology, the timing controller 110 may include a logic control circuit, a phase-locked loop (PLL) circuit, a timing control circuit, a communication interface circuit, and the like.

[0028] The readout circuit 120 may use correlated double sampling (CDS) to remove the undesired offset value of the pixel, which is called fixed pattern noise, by sampling the pixel signal twice to remove the difference between the two samples (i.e., the reference signal and the image signal). The readout circuit 120 may include an analog-to-digital converter (ADC) for converting the correlated double sampling (CDS) signal into a digital signal. The readout circuit 120 may include an output buffer that receives image data of each column from the ADC and temporarily holds and outputs the received image data of each column. The readout circuit 120 may include a column driver that selects a column from the output buffer based on a control signal of the timing controller 110, and controls the image data temporarily stored in the selected column of the output buffer to be output in sequence.

[0029] The image signal processor 200 may perform image processing of the image data received from the readout circuit 120. The image signal processor 200 may reduce the noise of the image data, and may perform various types of image signal processing (e.g., gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, lens distortion correction, etc.) for image quality improvement of the image data. In addition, the image signal processor 200 may compress the image data created by performing the image signal processing for image quality improvement, so that the image signal processor 200 may create an image file using the compressed image data. Alternatively, the image signal processor 200 may restore the image data from the image file. In this case, the scheme for compressing such image data may be a reversible format or an irreversible format. As a representative example of such a compression format, in the case of using a still image, a Joint Photographic Experts Group (JPEG) format, a JPEG 2000 format, etc. may be used. In addition, in the case of using a moving image, a plurality of frames may be compressed according to the Moving Picture Experts Group (MPEG) standard, so that a moving image file may be created. For example, an image file may be created according to the Exchangeable Image File Format (Exif) standard.

[0030] The image signal processor 200 may include a buffer circuit 210 , a data processor 220 , and a memory interface 230 .

[0031] Here, the buffer circuit 210 may temporarily store the image data received from the readout circuit 120, and may write the image data into the frame memory component 20 through the memory interface 230. In addition, the buffer circuit 210 may read the image data written into the frame memory component 20 through the memory interface 230, may temporarily store the read image data, and may transmit the stored image data to the data processor 220.

[0032] The buffer circuit 210 may temporarily store data processed by the data processor 220, and may write the data into the frame memory component 20 through the memory interface 230. In addition, the buffer circuit 210 may read image data written into the frame memory component 20 through the memory interface 230, may temporarily store the read image data, and may transmit the stored image data to the input / output (I / O) interface 300. However, the operation scope of the buffer circuit 210 is not limited thereto. For example, the buffer circuit 210 may perform a series of operations by temporarily storing and transmitting predetermined data using the readout circuit 120, the data processor 220, the frame memory component 20, the host device 30, and other components. The configuration of transmitting or receiving data to or from the buffer circuit 210 is not limited to Figure 1 The constituent elements shown.

[0033] The buffer circuit 210 may include a plurality of buffers (211-215). Each of the plurality of buffers (211-215) may include a flip-flop or a volatile memory (eg, SRAM), but the type of such memory is not limited thereto.

[0034] Here, the buffer 211 may be connected between the readout circuit 120 and the memory interface 230. The buffer 211 may perform an operation T1 of storing the image data received from the readout circuit 120 in the frame memory component 20 through the memory interface 230. The buffers (212 to 214) may be connected between the memory interface 230 and the data processor 220. The buffers (212 to 214) may perform an operation T2 of storing the data received from the memory interface 230 and transmitting the data to the multiscale processors (221 to 223) corresponding to the respective buffers. In addition, the buffers (212 to 214) may perform an operation T3 of storing the data received from the corresponding multiscale processors (221 to 223) and transmitting the data to the memory interface 230 again. The buffer 215 may be connected between the memory interface 230 and the input / output (I / O) interface 300. The buffer 215 may perform an operation T4 of outputting the image data received from the frame memory component 20 to the input / output (I / O) interface 300 through the memory interface 230. More detailed information about the buffer circuit 210 having the above-described constituent elements will be described later.

[0035] In addition, the data processor 220 may perform multi-scale processing based on the image data received from the buffer circuit 210. The data processor 220 may transmit the image data on which the multi-scale processing has been performed to the buffer circuit 210. Here, the multi-scale processing may mean an operation of processing a plurality of images having different scales by rescaling the original image into images having different resolutions through a pre-processing operation.

[0036] The data processor 220 may include a plurality of multiscale processors (221-223). The multiscale processors (221-223) may be connected to the buffers (212-214) in correspondence. The multiscale processors (221-223) may process the data received from the frame memory component 20 through the corresponding buffers (212-214), and may send the processed data to the buffers (212-214) again. More detailed information about the data processor 220 having the above-mentioned constituent elements will be described later.

[0037] The memory interface 230 may perform a data interface between the frame memory component 20 and the buffer circuit 210. Image data from the buffer circuit 210 may be written into the frame memory component 20 through the memory interface 230. Image data read from the frame memory component 20 may be transferred to the buffer circuit 210 through the memory interface 230.

[0038] The image signal processor 200 may transmit ISP (image signal processed) image data to an input / output (I / O) interface 300. The input / output (I / O) interface 300 may perform communication with a host device 30, and may transmit the ISP image data to the host device 30. In some implementations, the input / output (I / O) interface 300 may be a mobile industry processor interface (MIPI) interface, but is not limited thereto.

[0039] The frame memory component 20 may write (i.e., store) image data received from the image signal processor 200, and the image data stored in the frame memory component 20 may be read and transmitted to the image signal processor 200. The frame memory component 20 may include a volatile memory (e.g., DRAM), but the type of such memory is not limited thereto. The frame memory component 20 may store image data corresponding to a frame captured at one time by the pixel array of the image sensor 100.

[0040] although Figure 1 The embodiment discloses the frame memory component 20 located outside the imaging device 10, but the scope or spirit of the disclosed technology is not limited thereto. For example, the frame memory component 20 may be located inside the imaging device 10, or may be located inside the image signal processor 200.

[0041] The host device 30 may be a processor (e.g., an application processor) for processing ISP image data received from the imaging device 10, a memory device (e.g., a device including a non-volatile memory) for storing ISP image data, or a display device (e.g., a device including a liquid crystal display (LCD)) for visually displaying ISP image data.

[0042] Figure 2 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of the operation of an image signal processor is shown.

[0043] Figure 2 The embodiment of may represent an example of a method of processing data when the pixels to be processed are 8 horizontal channels.

[0044] Reference Figure 2(A), the input image data can be processed within one cycle (i.e., P1 processing), and then the next image data needs to be processed (i.e., P2 processing). In the case where filtering processing of adjacent pixels is required, it is necessary to complete the calculation of all pixels at the same time. That is, for processing with high resolution and fixed frame rate, a throughput of one cycle is required. In order to reduce the clock frequency in the image signal processor 200, data input can be performed at intervals of 8 pixels arranged in the horizontal direction, so that 8 horizontal pixels should be processed in one cycle. When 8 pixels are processed in parallel in one cycle, the processing speed can be reduced, but 8 identical circuits are required, so that the scale or number of necessary circuits can be increased compared to the processing of only one pixel.

[0045] Reference Figure 2 (B), the processing direction of the image data can be changed and processed in a vertically parallel direction to be perpendicular to the direction of the image data input (i.e., the raster scanning direction). The order of processing pixels can be changed by changing the direction of processing the image data to a vertical direction so that time division processing of parallel data can be performed. Here, time division processing may refer to an operation of processing multiple input images in a time division multiplexing (TDM) scheme. The direction of the image data is also referred to as the orientation direction of the image data.

[0046] The filtering of the pixel to be processed can be performed using the data of the neighboring pixels among many pixels included in the image data. Even if the filtering processing of the neighboring pixels is required, one horizontal pixel can be processed in one cycle, so that the time division processing of parallel data can be performed.

[0047] For example, if the processing order of pixels is arbitrarily changed, the amount of data used to perform parallel processing can also be changed so that the data of 8 horizontal channels can be increased to the data of 12 channels and then processed. The image data can be processed in one cycle (P3 processing), and then the processing of the image data corresponding to the parallel data is required (i.e., P4 processing).

[0048] That is, the image signal processor based on some implementations of the disclosed technology may store the input image data in the frame memory component 20, may read the stored image data, may change the order of the pixel data in a direction perpendicular to the direction in which the image data is input, and may process the resulting image data. Thereafter, the image signal processor may store the processed pixel data again in the frame memory component 20, may read the data stored in the frame memory component 20 again, may return the order of the pixel data, and may then output aligned image data.

[0049] In some implementations, it is assumed that the image signal processor 200 inputs and outputs image data in a raster scan order. Here, the raster scan order may refer to, for example, an operation of sequentially scanning all rows of a plurality of pixels arranged in the left-right direction (ie, the horizontal direction).

[0050] However, when the buffer circuit 210 (eg, Figure 1 When the order of processing pixels in the buffers 211 to 214 of the buffer circuit 210 (e.g., Figure 1 The buffers 211 to 214) may align the order of pixels in the same order as that of input pixels and may output the aligned pixels.

[0051] In some implementations, the following will be Figure 1 The operation of the buffer circuit 210 is described by taking the operations of T1 to T4 described in the foregoing as an example. The buffer 211 may receive data in a horizontal direction during the T1 operation, and may send the received data to the memory interface 230. The buffers 212 to 214 may process the data in a vertical direction while receiving the data horizontally from the memory interface 230, and may send the processed data to the data processor 220 during the operation T2. ​​In addition, the buffers 212 to 214 may process the data in a horizontal direction while receiving the data vertically from the data processor 220, and may send the processed data to the memory interface 230 during the operation T3. The buffer 215 may receive data in a horizontal direction during the T4 operation, and may send the data to the input / output (I / O) interface 300. The write operation and the read operation of the buffer circuit 210 will be referred to. Figures 7 to 14 Detailed description.

[0052] Therefore, in order to quickly process data using a parallel time division scheme, the data processor 220 located inside the image signal processor 200 may process data in an arbitrary order of pixels, but the order of pixels input and output to the image signal processor 200 included in the imaging device 10 does not change.

[0053] For example, an image signal processor based on some implementations of the disclosed technology and an imaging system including the same may be applied to a device equipped with a two-dimensional (2D) digital filter having a large number of taps, but the scope or spirit of the disclosed technology is not limited thereto. For example, an embodiment of the disclosed technology may be applied to an image signal processor that does not use all pixels in an operation mode (e.g., a still image mode or a video mode) in which the processing time of each frame is limited, and other implementations are also possible.

[0054] When the image data has multiple channels in the horizontal direction, an overlapping circuit is required to process the image data in the horizontal direction, and it is necessary to obtain results from each channel at the same time, so that the number of overlapping circuits cannot be reduced. However, when the image data has multiple channels in the horizontal direction, the image signal processor may process the image data in the vertical direction, or when the image data has multiple channels in the vertical direction, the image signal processor may process the image data in the horizontal direction, so that the number of overlapping circuits can be reduced. Therefore, an image signal processor based on some implementations of the disclosed technology may process data using a buffer having a number less than the number of pixels corresponding to the image data. That is, an image signal processor based on some implementations of the disclosed technology may reduce the number of buffers required for data processing and the processing time required to process such data.

[0055] Figure 3 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example case where data input is performed vertically in the image signal processor 200 is shown.

[0056] Reference Figure 3 , it is assumed that the image data to be processed by the image signal processor 200 has a Bayer pattern as shown in (A). Figure 3 In the embodiment, when processing data of 8 channels arranged in the horizontal direction in a two-dimensional (2D) digital filter, (20 pixels×13 rows) of pixels are required for calculation. The target pixels (TPX) of the filtered 8 channels can be horizontally included in the seventh row among the total 14 rows.

[0057] For example, in the green pixel shown in (B), when data is input in the vertical direction, the data may be processed using a time division scheme using 10 green pixels or 9 green pixels arranged in the horizontal direction. For example, the image signal processor 200 may repeatedly perform an operation (e.g., a pipeline processing operation) of calculating one horizontal line (i.e., 10 green pixels or 9 green pixels) per clock, and thus may perform calculations for 13 lines every 13 clocks.

[0058] For example, in the red pixel shown in (C), when data is input in the vertical direction, the data can be processed using a time division scheme using 10 red pixels arranged in the horizontal direction. For example, the image signal processor 200 can repeatedly perform an operation of calculating one horizontal line (i.e., 10 red pixels) per clock (e.g., a pipeline processing operation), and thus can perform calculations for 13 lines every 13 clocks.

[0059] For example, in the blue pixel shown in (D), when data is input in the vertical direction, the data can be processed using a time division scheme using 10 blue pixels arranged in the horizontal direction. For example, the image signal processor 200 can repeatedly perform an operation of calculating one horizontal line (i.e., 10 blue pixels) per clock (e.g., a pipeline processing operation), and thus can perform calculations for 13 lines every 13 clocks.

[0060] For example, in order to output the data of 8 channels in a two-dimensional (2D) digital filter at the same time, 10 pixels can be calculated in the horizontal direction, and the result of the calculation performed after 13 cycles can be used. For example, it is assumed that there is an operation of squaring the data of each pixel. When the last calculation is performed while leaving the data of each pixel, a buffer (e.g., a trigger) as many as the number of delay units may be required. However, when the data of each pixel is added or another calculation is required, the number of circuits can be reduced by storing only the calculation results in the buffer without storing all the calculation results in the buffer.

[0061] Figure 4 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example case where data input is performed horizontally in the image signal processor 200 shown.

[0062] Reference Figure 4 , assuming that the image data to be processed by the image signal processor 200 has a Bayer pattern as shown in (E). Figure 4 In the embodiment, when processing data of 8 channels arranged in the vertical direction in a two-dimensional (2D) digital filter, (13 pixels×20 rows) of pixels are required for calculation. The target pixel (TPX) of the filtered 8 channels can be vertically included in the seventh pixel among the total 14 pixels.

[0063] For example, in the green pixel shown in (F), when data is input in the horizontal direction, the data may be processed using a time division scheme using 10 green pixels or 9 green pixels arranged in the vertical direction. For example, the image signal processor 200 may repeatedly perform an operation (e.g., a pipeline processing operation) of calculating one vertical line (i.e., 10 green pixels or 9 green pixels) per clock, and thus may perform calculations for 13 lines every 13 clocks.

[0064] For example, in the red pixel shown in (G), when data is input in the horizontal direction, data processing can be performed using 10 red pixels arranged in the vertical direction. For example, the image signal processor 200 can repeatedly perform an operation (e.g., a pipeline processing operation) of calculating one horizontal line (i.e., 10 red pixels) per clock, and thus can perform calculation for 13 lines every 13 clocks.

[0065] For example, in the blue pixel shown in (H), when data is input in the horizontal direction, data processing can be performed using 10 blue pixels arranged in the vertical direction. For example, the image signal processor 200 can repeatedly perform an operation (e.g., a pipeline processing operation) of calculating one horizontal line (i.e., 10 blue pixels) per clock, and thus can perform calculation for 13 lines every 13 clocks.

[0066] For example, in order to simultaneously output data of 8 channels in a two-dimensional (2D) digital filter, 10 pixels may be calculated in the vertical direction, and the result of the calculation performed after 13 cycles may be used. Therefore, the image signal processor may calculate data using a time division scheme in a direction different from the direction in which the target pixels are arranged (i.e., the horizontal direction), so that the image signal processor may reduce the number of circuits required for calculation without changing the throughput.

[0067] Figure 5 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of the operation of a data processor in an image signal processor shown. Although multi-scale processing among various processing operations requiring a frame memory will be described as an example for the sake of convenience of description, the scope or spirit of the embodiments of the disclosed technology is not limited thereto.

[0068] Reference Figure 5 , the data processor 220 may perform multi-scale processing on the image data written in the frame memory component 20. That is, when the image data read from the frame memory component 20 through the buffers 212 to 214 is input to the data processor 220, the data processor 220 may perform multi-scale processing and may rewrite the processed image data into the frame memory component 20 through the buffers 212 to 214.

[0069] For example, each of the plurality of multi-scale processors 221-223 may downscale the input image so that the plurality of multi-scale processors 221-223 may respectively generate images with different resolutions (each having half the size). Here, each of the plurality of multi-scale processors 221-223 may store the images generated for each downscale in the frame memory component 20.

[0070] For example, the multiscale processors 221 to 223 may downscale image data (having a size of 3840×2160) to generate image data (having a size of 1920×1080), and may store the image data having a size of (1920×1080) in the frame memory component 20. In this way, the multiscale processors 221 to 223 may downscale the input image data to a specific size (i.e., half the size), and may store respective image data (having a size of 960×540, a size of 480×270, a size of 240×135, etc.) having a size smaller than the input image data in the frame memory component 20.

[0071] In addition, each of the plurality of multiscale processors 221 to 223 may compare the downscaled image with the respective images having previous sizes, and thus may output image data processed according to the comparison result. Here, each of the plurality of multiscale processors 221 to 223 may store the processing result of each image in the frame memory component 20. For example, the image processing operation performed by each of the multiscale processors 221 to 223 may refer to an operation of performing noise reduction by comparing images having two different sizes with each other.

[0072] For example, the multiscale processors 221 to 223 may compare the downscaled image data (size of 240×135) with the previous image data (size of 480×270), and may store the processed image data in the frame memory component 20. In this way, the multiscale processors 221 to 223 may compare and process the downscaled image data, and may store respective image data (size of 960×540, size of 1920×1080, size of 3840×2160, etc.) having a size greater than the input image data in the frame memory component 20.

[0073] The multiscale processing to be performed by the multiscale processors 221-223 may be processing capable of performing common operations on each scale, rather than dedicated processing for one scale. In addition, the multiscale processors 221-223 may be repeatedly used as many times as the number of multiscales while processing one frame.

[0074] Figure 6 and Figure 7 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of a write operation in the image signal processor 200. Figure 6 and Figure 7 2 is a conceptual diagram showing an example of an operation ( T1 ) in which the buffer 211 transmits image data received from the readout circuit 120 to the frame memory component 20 through the memory interface 230 .

[0075] Reference Figure 6, image data in units of 8 channels (hereinafter referred to as "image data based on 8 channels") can be transmitted from the readout circuit 120 to the buffer 211, as shown in (A). The buffer 211 can transmit the horizontally received image data based on 8 channels to the memory interface 230, as shown in (A). Figure 1 Therefore, image data of a size of (10 bits×8 channels) can be stored in the frame memory component 20.

[0076] For example, Figure 6 As shown in (B), the buffer 211 can align the data so that the data of 8 channels (i.e., Lane0 to Lane7) can be written into the frame memory component 20 in a cycle of once every 6 clocks. For example, when the memory interface 230 accesses the frame memory component 20 in units of 512 bits, the buffer 211 can store 48 pixels (i.e., 480 bits) of data in the frame memory component 20 using the data of the 8 channels in a cycle of 6 clocks.

[0077] Figure 6 (C) may show an example of a storage state of image data in the frame memory component 20. For example, assuming that the width of data stored in the frame memory component 20 is 16 bits, 512 bits corresponding to a size of (16 bits×32 bursts) may be stored during one access operation. For example, 480 bits among the 512 bits may represent a bit value of a pixel (i.e., 48 pixels), and the remaining 32 bits may be used as reserved bits.

[0078] Reference Figure 7 When image data (i.e., image data in units of (10 bits×8 channels)) is input from the readout circuit 120, the buffer 211 may write the data into the frame memory component 20 through the memory interface 230. For example, the buffer 211 may include a plurality of flip-flops (FFs). The buffer 211 may write a total of 512 bits of data into the frame memory component 20 using data of 8 pixels (i.e., data of (10 bits×8 channels)) as a cycle of 6 clocks.

[0079] Figures 8 to 10 Some implementations based on the disclosed technology are shown in Figure 1 A conceptual diagram of an example of a read operation in an image signal processor. Figures 8 to 10 2 is a conceptual diagram showing an example of an operation ( T2 ) in which the buffers 212 ˜ 214 read data from the frame memory unit 20 through the memory interface 230 .

[0080] Figure 8An example is shown of a state in which data corresponding to 1 access (i.e., 480 bits (data of 48 pixels)) is stored in the frame memory component 20. For example, 0x0000 to 0x09FF may be stored as pixel data of the first line (Line 1), 0x0A00 to 0x13FF may be stored as pixel data of the second line (Line 2), 0x1400 to 0x1DFF may be stored as pixel data of the third line (Line 3), and 0x1E00 to 0x27FF may be stored as pixel data of the fourth line (Line 4).

[0081] Fig. 9 It is shown Figure 1 A conceptual diagram of an example of an access operation in the image signal processor 200. Fig. 9 An example operation of accessing data stored in the frame memory component 20 through the buffers 212 to 214 is shown. Figure 1 The buffers 211 to 214 can align the data input in the horizontal direction vertically.

[0082] When a raster scan order is limited during access between the frame memory component 20 and the buffer circuit 210 , the image signal processor 200 may read data of the frame memory component 20 in a row-by-row manner and may store the read data in the buffer circuit 210 .

[0083] However, if Fig. 9 As shown, according to some embodiments of the disclosed technology, buffers 212-214 (e.g., buffer 212) may change the order in which image data is processed, and data processor 220 may perform data processing in a random order so that the raster scan order is not restricted during access between frame memory component 20 and buffers 212-214.

[0084] The buffers 212 to 214 can collect some data (I) located at the beginning of each row used for calculation from the data (e.g., data of the first to fourth rows) written in the frame memory component 20, and can use the collected data (I) for calculation processing. In addition, since data of the unused area (RNR) in the buffers 212 to 214 is not read, the buffer is not required.

[0085] Fig.10 Shown as Figure 12. (T2) shows an example operation of reading data stored in buffers 212 to 214. Buffers 212 to 214 can read image data received from frame memory component 20 through memory interface 230, and can send the read image data to data processor 220. For example, buffers 212 to 214 may include multiple triggers (FF). Buffers 212 to 214 can read (10 bits × 48 pixels) (i.e., 480 bits) of data. Buffers 212 to 214 can read (2 pixels × 24 channels) of data at intervals of one clock. Since write operations and read operations may overlap with each other, the number of rows used in buffers 212 to 214 may be doubled. That is, each of buffers 212 to 214 may include a first buffer group corresponding to rows 1 to 24 and a second buffer group corresponding to rows 25 to 48.

[0086] The frame memory component 20 may store data in a raster scan order. However, as described above, an image signal processor based on some implementations of the disclosed technology may process data in parallel in a direction perpendicular to the direction of data input. Therefore, in order to facilitate data access, the buffer circuit 210 may include at least one flip-flop (FF). Although for ease of description, the embodiment proposed by the disclosed technology discloses an example device including a flip-flop, the configuration of the buffer circuit 210 is not limited thereto, and it should be noted that the buffer circuit 210 may include an SRAM or other volatile memory.

[0087] Fig.11 It is shown in Figure 1 A conceptual diagram of an example of a method of performing a data alignment process during a read operation in an image signal processor. Fig.11 2 is a conceptual diagram showing an example of an operation ( T2 ) in which the buffers 212 ˜ 214 receive data from the frame memory assembly 20 through the memory interface 230 , and the received data is then aligned.

[0088] like Fig.11 As shown in (A), the target data to be read can be set by processing pixels of size (2×12) (i.e., parallel pixels) within one clock cycle. For example, the data storage capacity for parallel processing can be set to 23040 bits corresponding to (10 bits×48 pixels×24 rows×2 sides).

[0089] Fig.11 (B) shows the storage state of the frame memory component 20. For example, 480 bits corresponding to (10 bits×8 channels×6 cycles) can be stored in a memory area of ​​one row having a size of "48 (horizontal)×1 (vertical)", and can be stored within 24 cycles.

[0090] in addition, Fig.11(C) shows an example in which the buffers 212 to 214 can align data stored in the frame memory component 20. For example, the buffers 212 to 214 can divide 48 pixels read from the frame memory component 20 into 24 rows, can read the divided data 24 times, and can store the read data.

[0091] Fig.11 (D) shows the result of completing parallel processing in one clock in buffers 212-214. An image signal processor based on some implementations of the disclosed technology can perform vertical parallel processing, so that the image signal processor can store as much data as can be obtained in 24 clocks by processing only once.

[0092] Fig.12 It is shown in Figure 1 A conceptual diagram of example operations for processing data during a read operation in an image signal processor. Fig.12 is a conceptual diagram illustrating an example of an operation ( T2 ) of allowing the buffers 212 ˜ 214 to process data and transmit the processed data to the multiscale processors 221 ˜ 223 .

[0093] Fig.12 A pattern with a pixel size to be processed is shown, which corresponds to a pixel size of "96 pixels (48 pixels×2) and 12 rows". For the (7×7) filtering process, the image signal processor 200 may mirror some rows at the upper and lower ends, and may read data including not only the pixel to be processed but also 24 rows of neighboring pixels.

[0094] In order to process multiple rows of data, data including pixels located on the left side of each row may be read. For example, the buffers 212 to 214 may read data from the first row address (0x0000000) of the frame memory component 20, and may read data from the second row address (0x0001400) of the frame memory component 20, so that the buffers 212 to 214 may eventually read data up to the twelfth row.

[0095] For example, Fig.12 The implementation of may indicate the operation of the buffer required when performing (7×7) filtering on image data of a 4K (data of size 3840×2160) Bayer array. Fig.12 In the embodiment of the present invention, the data processed by the buffers 212 to 214 and output to the multi-scale processors 221 to 223 may have a size of (2 pixels×12 lines). Therefore, the multi-scale processors 221 to 223 may process (2 pixels×12 lines) of data at the same time, may compare the processing results with 8 channels, and may process the data at a higher processing speed that is three times the processing speed of the 8 channels.

[0096] Fig.13is a conceptual diagram illustrating an example method of aligning data during a write operation. Fig.13 1 is a conceptual diagram for explaining an operation ( T3 ) in which the buffers 212 to 214 write the processed data received from the multiscale processors 221 to 223 into the frame memory unit 20 through the memory interface 230 .

[0097] like Fig.13 As shown in (A), the target data input to the buffer 211 can be transferred to the frame memory component 20 by processing 24 (=2×12) pixels with one clock. For example, the data holding capacity for data alignment processing can be set to 11520 bits corresponding to (10 bits×48 pixels×12 rows×2 sides).

[0098] Fig.13 (B) shows an example operation of converting vertical data input to the buffers 212 to 214 into horizontal data and performing data alignment processing to write the data into the frame memory component 20. For example, in order to process input data with as much data as 12 lines, the buffer circuit 210 can divide the input data into a plurality of data segments each having a size of "2 (= horizontal size) × 1 (= vertical size)", and thus can store result data corresponding to 24 cycles. Fig.13 (C) may indicate a storage state of the frame memory component 20. For example, 480 bits corresponding to (10 bits×2 pixels×24 cycles) may be stored in a memory area of ​​one row having a size of “48 (= horizontal size)×1 (= vertical size)” within 24 cycles.

[0099] Fig.14 Is shown to access from Figure 1 A conceptual diagram of an example of an operation of outputting data of the frame memory component 20. Fig.14 2 is a conceptual diagram showing an example of an operation ( T4 ) in which the buffer 214 transfers data received from the frame memory component 20 to the I / O interface 300 through the memory interface 230 .

[0100] Fig.14 It is a diagram for describing an operation (T4') of transferring data applied from a frame memory component (20) to an input / output (I / O) interface (300) through a memory interface (230) in a buffer (215). Fig.14 (A) shows an example of a storage state of image data in the frame memory component 20. For example, assuming that the width of data stored in the frame memory component 20 is 16 bits, 512 bits corresponding to (16 bits×32 bursts) can be stored during one access operation. For example, 480 bits among the 512 bits can represent the bit value of a pixel (i.e., 48 pixels), and the remaining 32 bits can be used as reserved bits.

[0101] like Fig.14 As shown in (B), the buffer 215 can read the data of 8 channels (Lane0 to Lane7) in a cycle of 6 clocks. For example, when the memory interface 230 accesses the frame memory component 20 at intervals of 512 bits, the buffer 215 can read the data of 48 pixels (i.e., 480 bits) using the data of 8 channels in a cycle of 6 clocks. Fig.14 As shown in (C), the buffer 215 can read image data of a size of (10 bits×8 channels) from the frame memory component 20 in sequence.

[0102] According to the implementation scheme proposed by the disclosed technology, the data width, number of bits, data size, etc. used during data read operations, write operations, and access operations are merely examples for the convenience of description and better understanding of the disclosed technology, but are not limited thereto.

[0103] As apparent from the above description, an imaging system including an image signal processor based on some implementations of the disclosed technology can reduce the size of a buffer of the image signal processor and the data processing time of the image signal processor.

[0104] Various embodiments of the disclosed technology can provide various effects that can be directly or indirectly recognized through the above-mentioned patent documents.

[0105] Although illustrative embodiments have been described, it should be understood that modifications and enhancements to the disclosed embodiments and other embodiments are contemplated based on what is described or illustrated in this patent document.

[0106] CROSS-REFERENCE TO RELATED APPLICATIONS

[0107] This patent document claims priority to and the benefit of Korean Patent Application No. 10-2023-0091693, filed on Jul. 14, 2023, which is hereby incorporated by reference in its entirety as a part of the disclosure of this patent document.

Claims

1. An imaging system, comprising: A frame memory component that performs a read operation or a write operation of image data; as well as an image signal processor that writes the input image data into the frame memory component in a predetermined order, processes the data read from the frame memory component, and outputs the processed image data, The image signal processor converts the order of processing the image data into a direction different from the direction in which the image data is input, processes the result data so that the order of the processed image data is aligned with the predetermined order, and outputs the result data.

2. The imaging system according to claim 1, wherein: The image signal processor: The image data is processed in a direction perpendicular to a direction in which the image data is input.

3. The imaging system according to claim 1, further comprising an input / output (I / O) interface, wherein: The image signal processor comprises: a data processor that processes the image data; and A buffer circuit writes the image data into the frame memory component, reads the data stored in the frame memory component, converts the order of processing the read data into an arbitrary order, sends the result data to the data processor, aligns the order of the image data processed by the data processor in the predetermined order, writes the aligned data into the frame memory component, and outputs the data written in the frame memory component to the input / output I / O interface.

4. The imaging system according to claim 3, wherein: The image signal processor further comprises: A memory interface performs communication of the image data between the frame memory component and the buffer circuit.

5. The imaging system according to claim 3, wherein: The buffer circuit comprises: a first buffer that sends the image data to the frame memory component; a second buffer that converts an order of processing the image data received from the frame memory component into another order, sends the resulting image data to the data processor, aligns the data processed by the data processor, and sends the aligned data to the frame memory component; and A third buffer outputs the data stored in the frame memory component.

6. The imaging system according to claim 3, wherein: The buffer circuit: Only some data among the data read from the frame memory component are stored in at least one flip-flop, and the stored data are used for calculation processing.

7. The imaging system according to claim 3, wherein: The data processor comprises: A plurality of multi-scale processors are provided to perform multi-scale processing based on the image data.

8. The imaging system according to claim 7, wherein: The multi-scale processing includes: A plurality of images having different scales are processed by rescaling the image data into images having different resolutions.

9. The imaging system of claim 1, wherein: The predetermined order is a raster scan order.

10. The imaging system of claim 1, wherein: When the number of target pixels to be filtered among the image data is equal to or greater than a predetermined number, the image signal processor processes the target pixels using a parallel time division scheme.

11. The imaging system according to claim 10, wherein: The image signal processor: increasing the number of target pixels to exceed the number of input pixels; and Processing increases the target pixel.

12. The imaging system according to claim 10, in, When the target pixels are arranged in the horizontal direction, the image signal processor performs data processing in the vertical direction, Wherein, when the target pixels are arranged in a vertical direction, the image signal processor performs data processing in a horizontal direction.

13. The imaging system of claim 1, wherein: The image signal processor: When the image data has a plurality of channels in the horizontal direction, performing data processing in the vertical direction; and When the image data has a plurality of channels in the vertical direction, data processing is performed in the horizontal direction.

14. An image signal processor, the image signal processor comprising: a frame memory component that stores image data; as well as a buffer circuit that writes the input image data into the frame memory component in a predetermined order, processes the data read from the frame memory component, writes the processed data into the frame memory component, and outputs the processed image data read from the frame memory component, Wherein, the buffer circuit further comprises: When the image data is input in the first direction, the order of processing the image data is converted in the second direction, the resultant data is processed so that the order of the processed image data is aligned in the predetermined order, and the resultant data is output.

15. The image signal processor according to claim 14, wherein: The first direction and the second direction are perpendicular to each other.

16. The image signal processor according to claim 14, wherein: The predetermined order is a raster scan order.

17. The image signal processor according to claim 14, further comprising: a data processor that processes the image data received from the buffer circuit; as well as A memory interface performs communication of the image data between the frame memory component and the buffer circuit.

18. The image signal processor according to claim 17, wherein: The buffer circuit comprises: a first buffer that sends the image data to the frame memory component; a second buffer that converts an order of processing the image data received from the frame memory component into another order, sends the resulting image data to the data processor, aligns the data processed by the data processor, and sends the aligned data to the frame memory component; and A third buffer outputs the data stored in the frame memory component.

19. The image signal processor according to claim 17, wherein: The data processor comprises: A plurality of multi-scale processors are provided to perform multi-scale processing based on the image data.

20. An image signal processor, the image signal processor comprising: a buffer circuit that receives the image data in a first orientation direction in which the image data is raster scanned; a frame memory component that receives the image data from the buffer circuit in the first orientation direction and stores the image data; An image signal processor reads the image data in the first orientation direction through the buffer circuit, changes the orientation direction of the image data to a second orientation direction different from the first orientation direction, changes the processed image data back to the first orientation direction, and outputs the processed image data in the first orientation direction through the buffer circuit.

Images