Image signal processing device and imaging system including the same

Abstract

To provide an imaging system capable of reducing the buffer size of an image signal processor and reducing the processing time.SOLUTION: An imaging system 1 includes a frame memory 20 in which a reading operation or writing operation of image data is performed, and an image signal processing device 200 for writing image data input in an already-set order into the frame memory 20, processing data read from the frame memory 20, and outputting the processed image data. The image signal processing device 200 processes the image data while converting the processing order of the image data in a direction different from an input proceeding direction of the image data, and outputting the processed image data while rearranging the order of the processed image data to the already set order.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging system capable of sensing light and generating image data. [Background technology]

[0002] An image sensor is a device that captures an optical image using the properties of light-sensitive semiconductor materials that react to light. With the development of industries such as automobiles, medicine, computers, and communications, the demand for high-performance image sensing devices is increasing in various fields such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical micro cameras.

[0003] As the technology of image sensing devices develops, the amount of data captured by image sensing devices increases. When the amount of image data increases, more buffers are required to buffer the image data, and the speed at which the image data is processed may increase. Therefore, an image signal processing technique is required to reduce the number of buffers and the processing speed for processing image data. Summary of the Invention [Problem to be solved by the invention]

[0004] SUMMARY OF THE PRESENTLY PREFERRED EMBODIMENTS An embodiment of the present invention provides an imaging system that allows for a reduced buffer size in an image signal processor, thereby reducing processing time. [Means for solving the problem]

[0005] An imaging system according to an embodiment of the present invention includes a frame memory in which image data is read or written, and an image signal processing device that writes image data input in a preset order to the frame memory, processes data read from the frame memory, and outputs the processed image data. The image signal processing device can convert and process the processing order of the image data in a direction different from the direction in which the image data is input, and can output the processed image data by arranging the order of the processed image data in the preset order.

[0006] An image signal processing device according to another embodiment of the present invention includes a frame memory for storing image data, and a buffer circuit for writing image data input in a preset order to the frame memory, processing data read from the frame memory and writing the processed image data read from the frame memory to the frame memory, and outputting the processed image data read from the frame memory, in which when the input of image data proceeds in a first direction, the buffer circuit converts and processes a processing order of the image data in a second direction, and outputs the processed image data by arranging the order of the processed image data in a preset order. Effect of the Invention

[0007] Embodiments of the present invention may provide the advantage of reducing the buffer size of the image signal processor, thereby enabling reduced processing time.

[0008] Furthermore, the embodiments of the present invention are for illustrative purposes only, and those skilled in the art can make various modifications, changes, substitutions, and additions within the technical spirit and scope of the appended claims, and such modifications, changes, etc. should be considered to fall within the scope of the following claims. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an imaging system according to an embodiment of the present invention; [Diagram 2]FIG. 2 is a diagram for explaining the operation of the image signal processing device of FIG. [Diagram 3] 2 is a diagram for explaining a case where data input progresses vertically in the image signal processing device of FIG. 1; FIG. [Figure 4] 2 is a diagram for explaining a case where data input progresses horizontally in the image signal processing device of FIG. 1; FIG. [Diagram 5] 2 is a diagram for explaining the operation of a data processing unit in the image signal processing device of FIG. 1; [Figure 6] 2 is a diagram for explaining an access operation in the image signal processing device of FIG. 1; [Figure 7] 2 is a diagram for explaining a write operation in the image signal processing device of FIG. 1; FIG. [Figure 8] 2 is a diagram for explaining a write operation in the image signal processing device of FIG. 1; FIG. [Figure 9] 2 is a diagram for explaining a reading operation in the image signal processing device of FIG. 1; [Figure 10] 2 is a diagram for explaining a reading operation in the image signal processing device of FIG. 1; [Figure 11] 2 is a diagram for explaining a reading operation in the image signal processing device of FIG. 1; [Figure 12] 1. FIG. 4 is a diagram for explaining data alignment processing during read and write operations in the image signal processing device of FIG. [Figure 13] 1. FIG. 4 is a diagram for explaining data alignment processing during read and write operations in the image signal processing device of FIG. [Figure 14] 2 is a diagram for explaining an output access operation from the frame memory of FIG. 1; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Various embodiments will be described below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to specific embodiments, and includes various modifications, equivalents, and / or alternatives of the embodiments. The embodiments of the present disclosure can provide various effects that are directly or indirectly recognizable by the present disclosure.

[0011] FIG. 1 is a block diagram showing an imaging system according to an embodiment of the present invention. 1, the photographing system 1 may refer to a device such as a digital still camera for taking still images or a digital video camera for taking moving images. For example, the photographing device 10 may be realized by, but is not limited to, a digital single lens reflex (DSLR) camera, a mirrorless camera, or a mobile phone (particularly, a smartphone). The photographing device 10 may be a concept including a device that includes a lens and an image sensor to photograph a subject and generate an image.

[0012] The imaging system 1 may include an imaging device 10 , a frame memory 20 , and a host device 30 .

[0013] Here, the photographing device 10 may include an image sensor 100, a timing controller 110, a readout circuit 120, an image signal processor (ISP) 200, and an input / output interface (I / O interface) 300.

[0014] The image sensing unit 100 may be a complementary metal oxide semiconductor image sensor (CIS) that converts an optical signal into an electrical signal. The image sensing unit 100 may be controlled in its overall operation, such as on / off, operation mode, operation timing, and sensitivity, by a timing controller 110. The image sensing unit 100 may transmit image data, which is obtained by converting an optical signal into an electrical signal, to a readout circuit 120 according to the control of the timing controller 110.

[0015] The timing controller 110 may provide the image sensing unit 100 with a clock signal, a control signal for timing control, a row address signal for selecting a row, and a column address signal for selecting a column, which are required for the operation of each component of the image sensing unit 100. According to an embodiment, the timing controller 110 may include a logic control circuit, a Phase Lock Loop (PLL) circuit, a timing control circuit, and a communication interface circuit.

[0016] The readout circuit 120 may use correlated double sampling to remove unwanted offset values ​​of pixels such as fixed pattern noise by sampling the pixel signal twice to remove the difference between two samples (reference signal and image signal). The readout circuit 120 may include an analog-to-digital converter (ADC) that converts the correlated double sampled signal into a digital signal. The readout circuit 120 may include an output buffer that temporarily holds and outputs image data in units of columns provided from the ADC. The readout circuit 120 may include a column driver that selects columns of the output buffer based on a control signal from the timing controller 110 and controls the image data temporarily stored in the selected columns of the output buffer to be output sequentially.

[0017] The image signal processing device 200 may perform image signal processing on image data input from the readout circuit 120. The image signal processing device 200 may reduce noise on the corrected image data and perform image signal processing for improving image quality, such as gamma correction, color filter array interpolation, color matrix, color correction, and color enhancement. The image signal processing device 200 may compress image data generated by performing image signal processing for improving image quality to generate an image file, or may restore image data from the image file. The image compression format may be a lossless format or a lossy format. As an example of the compression format, in the case of a still image, a JPEG (Joint Photographic Experts Group) format or a JPEG2000 format may be used. In the case of a moving image, a moving image file may be generated by compressing a plurality of frames according to the MPEG (Moving Picture Experts Group) standard. The image file may be generated according to, for example, the Exif (Exchangeable image file format) standard.

[0018] The image signal processing device 200 may include a buffer circuit 210 , a data processing unit 220 , and a memory interface 230 . Here, the buffer circuit 210 can temporarily store image data applied from the read circuit 120 and write the image data to the frame memory 20 via the memory interface 230. The buffer circuit 210 can read the image data written to the frame memory 20 via the memory interface 230, temporarily store the image data, and transmit the image data to the data processor 220.

[0019] In addition, the buffer circuit 210 may temporarily store data processed by the data processor 220 and write the data to the frame memory 20 via the memory interface 230. The buffer circuit 210 may read image data written to the frame memory 20 via the memory interface 230, temporarily store the data, and transmit the data to the input / output interface 300. However, the operation of the buffer circuit 210 is not limited thereto. For example, the buffer circuit 210 may perform a series of operations to temporarily store and transmit predetermined data between the read circuit 120, the data processor 220, the frame memory 20, the host device 30, and other components. The configuration for transmitting data to the buffer circuit 210 or receiving data from the buffer circuit 210 is not limited to the configuration shown in FIG. 1.

[0020] Such a buffer circuit 210 can include a plurality of buffers 211 to 215. Each of the plurality of buffers 211 to 215 can include a flip-flop or a volatile memory (for example, an SRAM), but the type of memory is not limited to this.

[0021] Here, the buffer 211 may be connected between the read circuit 120 and the memory interface 230. The buffer 211 may perform an operation (T1) of storing image data input from the read circuit 120 in the frame memory 20 through the memory interface 230. The buffers 212 to 214 may be connected between the memory interface 230 and the data processing unit 220. The buffers 212 to 214 may perform an operation (T2) of storing data applied from the memory interface 230 and transmitting the data to the corresponding multi-scale processing units 221 to 223. Furthermore, the buffers 212 to 214 may perform an operation (T3) of storing data applied from the corresponding multi-scale processing units 221 to 223 and transmitting the data to the memory interface 230 again. The buffer 215 may be connected between the memory interface 230 and the input / output interface 300. The buffer 215 may perform an operation (T4) of outputting image data applied from the frame memory 20 through the memory interface 230 to the input / output interface 300. The buffer circuit 210 having such a configuration will be described in more detail later.

[0022] The data processor 220 may perform multi-scale processing based on the image data applied from the buffer circuit 210. The data processor 220 may transmit the image data subjected to the multi-scale processing to the buffer circuit 210. Here, the multi-scale processing may mean processing a plurality of images each having a different scale by re-adjusting an original image to images having different resolutions through a pre-processing process.

[0023] The data processing unit 220 may include a plurality of multi-scale processing units 221-223. The multi-scale processing units 221-223 may be connected to the buffers 212-214 in a corresponding manner. The multi-scale processing units 221-223 may process data transferred from the frame memory 20 via the corresponding buffers 212-214, and then transfer the processed data back to the buffers 212-214. The data processing unit 220 having such a configuration will be described in more detail below.

[0024] The memory interface 230 can perform data interfacing between the frame memory 20 and the buffer circuit 210. Image data from the buffer circuit 210 can be written to the frame memory 20 via the memory interface 230. And image data read from the frame memory 20 can be transferred to the buffer circuit 210 via the memory interface 230.

[0025] The image signal processing device 200 may transmit the image data that has been subjected to image signal processing to the input / output interface 300. The input / output interface 300 may communicate with the host device 30 and transmit the image data that has been subjected to image signal processing to the host device 30. According to an embodiment, the input / output interface 300 may be a mobile industry processor interface (MIPI) interface, but the scope of the present invention is not limited thereto.

[0026] The frame memory 20 writes (i.e., stores) image data received from the image signal processing device 200, and the image data stored in the frame memory 20 can be read and transferred to the image signal processing device 200. The frame memory 20 can include a volatile memory (e.g., DRAM), but the type of memory is not limited thereto. The frame memory 20 can store image data corresponding to a frame captured at one time by the pixel array of the image sensing unit 100.

[0027] 1 illustrates the frame memory 20 as being external to the image capture device 10, embodiments of the present invention are not limited thereto. As an example, the frame memory 20 may be internal to the image capture device 10 or internal to the image signal processor 200.

[0028] The host device 30 may be a processor (e.g., an application processor) that processes image signal-processed image data received from the photographing device 10, a memory (e.g., non-volatile memory) that stores the image data, or a display device (e.g., a liquid crystal display (LCD)) that visually outputs the image data.

[0029] FIG. 2 is a diagram for explaining the operation of the image signal processing device of FIG. The embodiment of FIG. 2 may show an example of a process of processing data when pixels to be processed are 8 horizontal lanes.

[0030] As shown in FIG. 2A, input image data is processed for one cycle (P1 processing), and then the next image data needs to be processed (P2 processing). In the case of filter processing that requires surrounding pixels, calculations for all pixels need to be completed simultaneously. That is, processing with a high resolution and a fixed frame rate requires a throughput of one cycle. In order to reduce the clock frequency in the image signal processing device 200, data is input in units of eight pixels in the horizontal direction, so eight horizontal pixels must be processed as one cycle. When eight pixels are processed in parallel as one cycle, the processing speed can be reduced, but eight identical circuits must be provided, which may result in a larger circuit size than in single pixel processing.

[0031] In an embodiment of the present invention, as shown in FIG. 2B, the direction of image data can be changed to a vertical parallel form so that the direction of image data input is perpendicular to the direction in which the image data input proceeds (raster scan order) and processed. By changing the processing direction of image data vertically and changing the processing order of pixels, time-division processing of parallel data is possible. Here, time-division processing can mean processing multiple input images in a time-division multiplexing (TDM) manner.

[0032] Among the many pixels contained in the image data, the data of the surrounding pixels can be used to filter the pixel being processed. Even in the case of filter processing that requires the surrounding pixels, one horizontal pixel is processed as one cycle, so time-division processing of parallel data is possible.

[0033] For example, if the pixel processing order is arbitrarily changed, the amount of data to be processed in parallel is also changed, so that 8 horizontal lanes of data can be increased to 12 lanes for processing. The image data is processed for one cycle (P3 processing), and then the image data, which is parallel data, needs to be processed (P4 processing).

[0034] That is, in an embodiment of the present invention, input image data is stored in frame memory 20, the stored image data is read, the order of the pixel data is changed in a direction perpendicular to the direction in which the image data is input, the pixel data is processed, the processed pixel data is stored again in frame memory 20, the data stored in frame memory 20 is read again, the order of the pixel data is restored, and the aligned image data is output.

[0035] In the embodiment of the present invention, it is assumed that image data is input and output to the image signal processing device 200 in a raster scan order, which may mean, for example, sequentially scanning all rows of pixels from left to right.

[0036] However, when the pixel processing order is changed in the buffer circuit 210 (for example, buffers 211-214 in FIG. 1) as described above, the data processing unit 220 can process data in any pixel order regardless of the raster order. Then, in the buffer circuit 210 (for example, buffers 211-214 in FIG. 1), the pixels can be output in the original order, the same as the input pixel order.

[0037] According to an embodiment, the operation of the buffer circuit 210 will be described by taking the operations (T1) to (T4) described in FIG. 1 as an example. The buffer 211 can receive data input in the horizontal direction during the (T1) operation and transmit it to the memory interface 230. The buffers 211 to 214 can receive data input in the horizontal direction from the memory interface 230 during the (T2) operation, process the data in the vertical direction, and transmit it to the data processing unit 220. The buffers 211 to 214 can receive data input in the vertical direction from the data processing unit 220 during the (T3) operation, process the data in the horizontal direction, and transmit it to the memory interface 230. The buffer 215 can receive data input in the horizontal direction during the (T4) operation and transmit it to the input / output interface 300. The write and read operations of the buffer circuit 210 will be described in more detail with reference to FIGS. 7 to 14, which will be described later.

[0038] Therefore, in order to quickly process data in parallel time division, the data is processed in an arbitrary pixel order in the data processing unit 220 inside the image signal processing device 200, but the order of pixels input / output to / from the image signal processing device 200 inside the photographing device 10 is not changed.

[0039] For example, but not limited to, the present invention may be applied to an image signal processing device that does not use all pixels, such as a still image or image mode that has a processing time constraint for one frame.

[0040] When image data has multiple lanes in the horizontal direction, overlapping circuits are required to process data in the horizontal direction, and results must be output simultaneously in each lane, making it impossible to reduce the overlapping circuits. However, the overlapping circuits can be reduced by processing data vertically when multiple lanes are in the horizontal direction, or processing data horizontally when multiple lanes are in the vertical direction. As a result, the image signal processing device according to the embodiment of the present invention can process data using only a smaller number of buffers than the number of pixels corresponding to the image data. That is, according to the embodiment of the present invention, the number of buffers and time required for data processing can be reduced.

[0041] FIG. 3 is a diagram for explaining a case where data input progresses vertically in the image signal processing device of FIG. 3, it is assumed that image data to be processed in the image signal processing device 200 is a Bayer pattern as shown in (A). In the embodiment of FIG. 3, when processing data of 8 lanes arranged in the horizontal direction in a two-dimensional digital filter, 20 pixels×13 lines of pixels are required for calculation. The 7th line of the total 14 lines may include target pixels (TPX) of 8 lanes to be filtered in the horizontal direction.

[0042] For example, in the case of a green pixel as shown in (B), when data input proceeds vertically, data processing can be performed in a time-division manner using 10 or 9 green pixels arranged horizontally. For example, the image signal processing device 200 can perform operations on 13 lines in 13 clocks by repeating operations on one horizontal line (10 or 9 green pixels) per clock (pipeline processing, for example).

[0043] For example, in the case of a red pixel as shown in (C), when data is input vertically, data processing can be performed in a time-division manner using 10 red pixels arranged horizontally. For example, the image signal processing device 200 can perform operations on 13 lines in 13 clocks by repeating operations on one horizontal line (10 red pixels) per clock (pipeline processing, for example).

[0044] For example, in the case of blue pixels as shown in (D), when data is input vertically, data processing can be performed in a time-division manner using 10 blue pixels arranged horizontally. For example, the image signal processing device 200 can perform operations on 13 lines in 13 clocks by repeating operations on one horizontal line (10 blue pixels) per clock (pipeline processing, for example).

[0045] As an example, to output eight lanes of data simultaneously from a two-dimensional digital filter, ten pixels are calculated horizontally, and the calculation results after 13 cycles can be used. For example, assume that there is an operation to square the data of each pixel. If the operation is performed at the end while leaving the data of each pixel, a buffer (e.g., a flip-flop) is required for the delay unit. However, if the data of each pixel is added or other operations are required, the number of circuits can be reduced by storing only the calculation results rather than storing all the calculation results in a buffer.

[0046] FIG. 4 is a diagram for explaining a case where data input progresses horizontally in the image signal processing device of FIG. 4, it is assumed that image data to be processed in the image signal processing device 200 is a Bayer pattern as shown in (E). In the embodiment of FIG. 4, when processing data of 8 lanes arranged vertically in a two-dimensional digital filter, 13 pixels×20 lines of pixels are required for calculation. The target pixel (TPX) of the 8 lanes to be filtered may be included in the vertical direction at the 7th pixel of the total 14 pixels.

[0047] For example, in the case of a green pixel as shown in (F), when data is input in the horizontal direction, data processing can be performed in a time-division manner using 10 or 9 green pixels arranged in the vertical direction. For example, the image signal processing device 200 can perform operations on 13 lines in 13 clocks by repeating (for example, pipeline processing) an operation on one vertical line (10 or 9 green pixels) in one clock.

[0048] For example, in the case of red pixels as shown in (G), when data is input in the horizontal direction, 10 red pixels arranged in the vertical direction can be calculated. For example, the image signal processing device 200 can perform calculations on 13 lines in 13 clocks by repeating (e.g., pipeline processing) a calculation for one vertical line (10 red pixels) per clock.

[0049] For example, in the case of a blue pixel as shown in (H), when data input proceeds in the horizontal direction, 10 blue pixels arranged in the vertical direction can be calculated. For example, the image signal processing device 200 can perform calculations on 13 lines in 13 clocks by repeating (e.g., pipeline processing) a calculation on one vertical line (10 blue pixels) per clock.

[0050] As an example, to simultaneously output eight lanes of data from a two-dimensional digital filter, ten pixels are calculated in the vertical direction, and the calculation results after 13 cycles can be used. This allows the number of circuits required for calculation to be reduced without changing the throughput by calculating data in a time-division multiplexing manner in a direction (horizontal direction) different from the direction in which the target pixels are arranged.

[0051] Fig. 5 is a diagram for explaining the operation of the data processing unit in the image signal processing device of Fig. 1. In the embodiment of the present invention, the multi-scale processing is explained as an example of the processing operation requiring a frame memory, but the embodiment of the present invention is not limited thereto.

[0052] 5, the data processing unit 220 can perform multi-scale processing on image data written in the frame memory 20. That is, when image data read from the frame memory 20 via the buffers 212-214 is input to the data processing unit 220, the data processing unit 220 can perform multi-scale processing and write the processed image data back into the frame memory 20 via the buffers 212-214.

[0053] For example, each of the multiple multi-scale processing units 221 to 223 can downscale input image data to generate images with different resolutions (half the size). Here, each of the multiple multi-scale processing units 221 to 223 can store the image generated for each downscaling in the frame memory 20.

[0054] For example, the multi-scale processing units 221 to 223 can downscale image data (3840×2160 size) to generate image data (1920×1080 size) and store the generated image data in the frame memory 20. In this way, the multi-scale processing units 221 to 223 can downscale input image data to a specific size (half the size) and store in the frame memory 20 each image data (960×540 size, 480×270 size, 240×135 size, etc.) having a size smaller than the input image data.

[0055] Each of the multi-scale processors 221-223 may compare the downscaled image with an image of a previous size and output processed image data. Here, each of the multi-scale processors 221-223 may store a processing result in the frame memory 20 every time an image is processed. As an example, the image processing operation performed by each of the multi-scale processors 221-223 may refer to an operation of comparing two images having different sizes to perform noise reduction.

[0056] For example, the multi-scale processing units 221 to 223 can compare the downscaled image data (240×135 size) with the previous image data (480×270 size) and store the processed image data in the frame memory 20. In this manner, the multi-scale processing units 221 to 223 can compare and process other scaled image data and store in the frame memory 20 each image data (960×540 size, 1920×1080 size, 3840×2160 size, etc.) having a size larger than the input image data.

[0057] The multi-scale processing performed by the multi-scale processing units 221 to 223 may not be dedicated processing for one scale, but may be processing capable of performing common calculations for each scale. The multi-scale processing units 221 to 223 can be used repeatedly multiple times, the number of times being equal to the number of multi-scales, while processing one frame.

[0058] Figures 6 and 7 are diagrams for explaining a write operation in the image signal processing device of Figure 1. Figures 6 and 7 are diagrams for explaining an operation (T1) in which the buffer 211 transfers image data applied from the read circuit 120 to the frame memory 20 via the memory interface 230.

[0059] 6, as shown in (A), image data in units of 8 lanes may be applied from the read circuit 120 to the buffer 211. The buffer 211 may transfer the image data in units of 8 lanes applied in the horizontal direction to the memory interface 230, as shown in (T1) operation of FIG. 1. Thus, image data in units of 10 bits×8 lanes may be stored in the frame memory 20.

[0060] For example, as shown in (B), the buffer 211 can align data so that data of eight lanes (Lane 0 to Lane 7) can be written to the frame memory 20 once every six clocks. For example, when the memory interface 230 accesses the frame memory 20 in units of 512 bits, the buffer 211 can store data of 48 pixels (480 bits) in the frame memory 20 with eight lanes of data in a six clock cycle.

[0061] 6C shows an example of the storage state of image data in the frame memory 20. As an example, assuming that the width of data stored in the frame memory 20 is 16 bits, in one access operation, 512 bits can be stored by 16 bits x 32 bursts. For example, 480 bits of the 512 bits indicate a bit value of a pixel (48 pixels), and the remaining 32 bits can be used as reserved.

[0062] 7, when image data (image data in units of 10 bits×8 lanes) is input from the read circuit 120, the buffer 211 can write the data to the frame memory 20 via the memory interface 230. For example, the buffer 211 can include a plurality of flip-flops (FFs). The buffer 211 can write a total of 512 bits of data of 8 pixels (10 bits×8 lanes) in a period of 6 clocks to the frame memory 20.

[0063] Figures 8 to 10 are diagrams for explaining the reading operation in the image signal processing device of Figure 1. Figures 8 to 10 are diagrams for explaining the operation (T2) in which the buffers 212 to 214 read data from the frame memory 20 via the memory interface 230.

[0064] 8 may illustrate an example of a state in which data (480 bits (48 pixel data)) for one access is stored in the frame memory 20. For example, 0x0000 to 0x09FF may be stored as pixel data of the first line, 0x0A00 to 0x13FF may be stored as pixel data of the second line, 0x1400 to 0x1DFF may be stored as pixel data of the third line, and 0x1E00 to 0x27FF may be stored as pixel data of the fourth line.

[0065] Fig. 9 is a diagram for explaining an access operation in the image signal processing device of Fig. 1. Fig. 9 can show an operation in which data stored in the frame memory 20 is accessed by the buffers 212 to 214, like the (T2) operation of Fig. 1 described above. The buffers 211 to 214 can align data input in the horizontal direction in the vertical direction.

[0066] If there is a restriction on the raster scan order when accessing between the frame memory 20 and the buffer circuit 210, the data in the frame memory 20 can be read line by line and stored in the buffer circuit 210.

[0067] However, referring to FIG. 9, an embodiment of the present invention changes the processing order of image data in buffers 212-214 (e.g., buffer 212) and processes data in any order in data processing unit 220, so that there is no restriction on the raster scan order when accessing between frame memory 20 and buffers 212-214.

[0068] The buffers 212-214 can collect and use for calculation processing a portion of data (I) from the start of each line used for calculation among the data (for example, data of the first to fourth lines) written to the frame memory 20. And, since data in the area (RNR) that has not yet been used in the buffers 212-214 is not read, no buffer is required.

[0069] FIG. 10 may illustrate an operation of reading data stored in the buffers 212 to 214, as in the operation (T2) of FIG. 1 described above. The buffers 212 to 214 may read image data applied from the frame memory 20 via the memory interface 230 and transmit the image data to the data processing unit 220. For example, the buffers 212 to 214 may include a plurality of flip-flops (FFs). The buffers 212 to 214 may read data of 10 bits x 48 pixels (480 bits). The buffers 212 to 214 may read data of 2 pixels x 24 lanes for each clock. The buffers 212 to 214 may have twice the number of lines because the write operation and the read operation may overlap. That is, the buffers 212 to 214 may include a first buffer group of lines 1 to 24 and a second buffer group of lines 25 to 48.

[0070] The frame memory 20 can store data in raster order. However, as described above, the embodiment of the present invention can process data in parallel in a direction perpendicular to the direction in which the input of data proceeds. This allows the buffer circuit 210 to include flip-flops (FFs) that facilitate easy access to data. In the embodiment of the present invention, the buffer circuit 210 is described as including flip-flops as an example, but the configuration of the buffer circuit 210 is not limited thereto, and may include SRAM or other volatile memory.

[0071] Fig. 11 is a diagram for explaining data alignment processing during a reading operation in the image signal processing device of Fig. 1. Fig. 11 is a diagram for explaining an operation (T2) in which the buffers 212 to 214 receive data input from the frame memory 20 via the memory interface 230 and align the data.

[0072] As shown in Fig. 11A, the target data to be read can be set to process 2x12 size pixels (parallel pixels) as one clock period. For example, the data storage capacity for parallel processing can be set to 23,040 bits in the case of 10 bits x 48 pixels x 24 lines x 2 planes.

[0073] 11B shows the storage state of the frame memory 20. For example, in the case of 10 bits x 8 lanes x 6 cycles in a one-line memory area having a size of 48 horizontally and 1 vertically, 480 bits can be stored over 24 cycles.

[0074] 11C may show that the buffers 212-214 align the data stored in the frame memory 20. As an example, the buffers 212-214 may divide the 48 pixels read from the frame memory 20 into 24 lines, and read and store the divided data 24 times.

[0075] 11(D) shows the result of completing parallel processing in one clock in buffers 212 to 214. Since the embodiment of the present invention can perform vertical parallel processing, it is possible to store as much data as can be obtained in 24 clocks in one processing.

[0076] Fig. 12 is a diagram for explaining the operation of processing data during a reading operation in the image signal processing device of Fig. 1. Fig. 12 is a diagram for explaining the operation (T2) of processing data in the buffers 212-214 and transmitting the data to the multi-scale processing units 221-223.

[0077] 12 shows a pattern having a pixel size of 96 pixels (48 pixels×2) and 12 lines to be processed. For 7×7 filtering, the image signal processing device 200 can mirror some lines at the top and bottom and read data of 24 lines including not only the pixel to be processed but also the surrounding pixels.

[0078] To process data for multiple lines, the buffers 212 to 214 can read data including pixels on the left side of each line. For example, the buffers 212 to 214 can read data up to the 12th line by reading data from the first line address (0x0000000) of the frame memory 20, then reading data from the second line address (0x0001400).

[0079] For example, the embodiment of Fig. 12 may show a buffer required when performing 7x7 filtering processing on 4K (3840x2160 size data) Bayer array image data. According to the embodiment of Fig. 12, the data processed by the buffers 212-214 and output to the multi-scale processing units 221-223 may be 2 pixels x 12 lines. Thus, the multi-scale processing units 221-223 can process 2 pixels x 12 lines of data simultaneously, and process data at three times the processing speed compared to 8 lanes.

[0080] Fig. 13 is a diagram for explaining data alignment during a write operation. Fig. 13 is a diagram for explaining an operation (T3) in which the processing data applied from the multi-scale processing units 221-223 in the buffers 212-214 is written to the frame memory 20 via the memory interface 230.

[0081] 13A, the target data input to the buffer 211 can process pixels of size 2×12 in one clock and be transferred to the frame memory 20. For example, the data storage capacity for the alignment process can be set to 11,520 bits in the case of 10 bits×48 pixels×12 lines×2 planes.

[0082] FIG. 13B shows that vertical data input to the buffers 212-214 is converted into horizontal data and aligned in order to write data to the frame memory 20. As an example, the buffer circuit 210 divides the input data into 2 horizontal and 1 vertical data in order to process data for only 12 lines, and stores data for only 24 cycles. FIG. 13C shows the storage state of the frame memory 20. For example, in the case of 10 bits x 2 pixels x 24 cycles in a memory area of ​​one line having a size of 48 horizontal and 1 vertical, 480 bits can be stored over 24 cycles.

[0083] Fig. 14 is a diagram for explaining an output access operation from the frame memory of Fig. 1. Fig. 14 is a diagram for explaining an operation (T4) in which the buffer 215 transmits data applied from the frame memory 20 via the memory interface 230 to the input / output interface 300.

[0084] 14A shows an example of the storage state of image data in the frame memory 20. As an example, assuming that the width of data stored in the frame memory 20 is 16 bits, in one access operation, 512 bits can be stored by 16 bits x 32 bursts. For example, 480 bits of the 512 bits indicate a bit value of a pixel (48 pixels), and the remaining 32 bits can be used as reserved.

[0085] The buffer 215 can read data from 8 lanes (Lane 0 to Lane 7) once every 6 clock cycles as in (B). For example, when the memory interface 230 accesses the frame memory 20 in units of 512 bits, the buffer 215 can read data of 48 pixels (480 bits) from 8 lanes of data in a period of 6 clock cycles. As in (C), the buffer 215 can sequentially read image data in units of 10 bits x 8 lanes from the frame memory 20.

[0086] In the embodiment of the present invention, the width, number of bits, size, and the like of data used in data read, write, and access operations are merely examples for facilitating understanding, and are not limited to these.

Claims

1. a frame memory for reading or writing image data; an image signal processing device for writing the image data input in a preset sequence into the frame memory, processing data read from the frame memory, and outputting the processed image data; Including, The image signal processing device comprises: a processing order of the image data being converted and processed in a direction different from a direction in which the image data is input, and the processed image data being output in a predetermined order;

2. The image signal processing device comprises:

2. The imaging system of claim 1, further comprising: a processor for processing the image data in a direction perpendicular to a direction along which the image data input proceeds.

3. The image signal processing device comprises: a data processing unit for processing the image data; a buffer circuit that writes the image data to the frame memory, reads the data stored in the frame memory, converts the processing order into an arbitrary order, and transmits the converted order to the data processing unit, aligns the order of the image data processed by the data processing unit to the predetermined order, writes the image data to the frame memory, and outputs the data written to the frame memory to an input / output interface.

4. The image signal processing device comprises:

4. The imaging system according to claim 3, further comprising a memory interface for transferring the image data between the frame memory and the buffer circuit.

5. The buffer circuit includes: a first buffer for transmitting the image data to the frame memory; a second buffer for converting a processing order of the image data applied from the frame memory and transmitting the converted data to the data processor, and for arranging the data processed by the data processor and transmitting the data to the frame memory; 4. The imaging system according to claim 3, further comprising: a third buffer for outputting the data stored in the frame memory.

6. The buffer circuit includes:

4. The imaging system according to claim 3, wherein only a portion of the data read from said frame memory is stored in a flip-flop and used for arithmetic processing.

7. The data processing unit includes: The imaging system according to claim 3 , further comprising a plurality of multi-scale processing units for performing multi-scale processing based on the image data.

8. The multi-scale processing includes:

8. The imaging system of claim 7, further comprising: rescaling the image data into images having different resolutions; and processing a plurality of images each having a different scale.

9. The previously set order is: The imaging system according to claim 1 , wherein the imaging order is a raster scan order.

10. The image signal processing device comprises:

2. The imaging system according to claim 1, wherein when the number of target pixels to be filtered in the image data is equal to or greater than a specific number, the image data is processed in a parallel time-division manner.

11. The image signal processing device comprises: The imaging system according to claim 10 , wherein the number of the target pixels is increased from the number of the input pixels for processing.

12. The image signal processing device comprises: If the target pixel is horizontally aligned, 11. The imaging system according to claim 10, wherein data processing is performed in a vertical direction.

13. The image signal processing device comprises: If the target pixel is vertically aligned, 11. The imaging system according to claim 10, wherein data processing is performed in a horizontal direction.

14. The image signal processing device comprises: When the image data has a plurality of lanes in a horizontal direction, data processing is performed in a vertical direction; 2. The imaging system according to claim 1, wherein data processing proceeds horizontally when the image data has a plurality of lanes in the vertical direction.

15. A frame memory for storing image data; a buffer circuit for writing image data input in a preset sequence into the frame memory, processing data read from the frame memory and writing the processed image data into the frame memory, and outputting the processed image data read from the frame memory; Including, The buffer circuit includes:

1. An image signal processing device, comprising: an image signal processing device for processing image data in a second direction by converting a processing order of the image data when the image data is input in a first direction; arranging the processed image data in the predetermined order and outputting the processed image data.

16. 16. The image signal processing apparatus of claim 15, wherein the first direction and the second direction are perpendicular to each other.

17. The previously set order is:

16. An image signal processing apparatus according to claim 15, in a raster scan order.

18. a data processing unit for processing the image data applied from the buffer circuit; 16. The image signal processing apparatus according to claim 15, further comprising: a memory interface for transferring said image data between said frame memory and said buffer circuit.

19. The buffer circuit includes: a first buffer for transmitting the image data to the frame memory; a second buffer for converting a processing order of the image data applied from the frame memory and transmitting the converted data to the data processor, and for arranging the data processed by the data processor and transmitting the data to the frame memory; 20. The image signal processing apparatus according to claim 18, further comprising: a third buffer for outputting data stored in said frame memory.

20. The data processing unit includes: The image signal processing apparatus according to claim 18, comprising a plurality of multi-scale processing units for performing multi-scale processing on the image data.

Images