Scan conversion controller, image processing circuit including same, and method of operating image processing circuit

Abstract

An image processing circuit, a scan conversion controller, and a method of operating the image processing circuit are provided. The image processing circuit includes: a scan conversion controller configured to receive images from a plurality of channels connected to a plurality of imaging devices and generate output data; and an image signal processor configured to receive and process the output data from the scan conversion controller. The scan conversion controller is configured to determine a scan method for generating the output data according to a first operation mode or a second operation mode.

Description

Technical Field

[0001] The inventive concept relates to a scan conversion controller, an image processing circuit including the scan conversion controller, and a method of operating the image processing circuit. Background Art

[0002] Due to the development of mobile devices such as smartphones and tablet personal computers (PCs), electronic devices including multiple image sensors are widely used in these types of devices. An image processing apparatus may include an image signal processor that processes input images generated by multiple image sensors.

[0003] At this time, in order to process a plurality of input images provided through a plurality of channels by using one image signal processor, the image signal processor may process the plurality of input images by a time-division multiplexing (TDM) method. Summary of the invention

[0004] An exemplary problem to be solved by the technical idea of ​​the present disclosure is to provide an image processing circuit that supports line interleaving and can easily switch a scanning method.

[0005] According to one aspect of the inventive concept, there is provided an image processing circuit, the image processing circuit comprising: a scan conversion controller, the scan conversion controller being configured to receive images from a plurality of channels connected to a plurality of imaging devices and generate output data; and an image signal processor, the image signal processor being configured to receive and process the output data from the scan conversion controller. The scan conversion controller is configured to determine a scanning method for generating the output data according to a first operating mode or a second operating mode.

[0006] According to another aspect of the inventive concept, a scan conversion controller is provided, the scan conversion controller comprising: a first data packing circuit, the first data packing circuit being configured to receive first line data from a first channel and pack the first line data into a predetermined data size to generate first packed data; a second data packing circuit, the second data packing circuit being configured to receive second line data from a second channel and pack the second line data into a predetermined data size to generate second packed data; an arbitration circuit, the arbitration circuit being configured to receive the first packed data and the second packed data, and sequentially output the first packed data and the second packed data; a memory, the memory being configured to store the first packed data and the second packed data; a row index controller, the row index controller being configured to manage a first index of a first area in the memory storing the first packed data and a second index of a second area in the memory storing the second packed data, and determine a scanning method of the first packed data and the second packed data respectively assigned to the first index and the second index; a write controller, the write controller being configured to control a write operation of the memory; a read controller, the read controller being configured to control a read operation of the memory; and an unpacking circuit, the unpacking circuit being configured to generate output data by unpacking the read data output from the read controller.

[0007] According to another aspect of the inventive concept, there is provided a method for operating an image processing circuit including a scan conversion controller, the scan conversion controller including an internal memory, the method including: packing line data received through each of a plurality of channels into a predetermined data size to generate packed data; assigning an index corresponding to each of the plurality of channels, and writing the packed data to the internal memory based on an address included in the assigned index; managing the index assigned to each of the plurality of channels; determining a scanning method for the packed data corresponding to the channel that has completed the input of line data based on an index assigned to the channel that has completed the input of line data; outputting index information corresponding to the determined scanning method; reading the packed data from the internal memory based on the index information; and unpacking the read data read from the internal memory and generating output data. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a block diagram of a data processing system according to an embodiment of the inventive concept;

[0010] Figure 2 It is shown Figure 1 A diagram of an example of an image processing circuit;

[0011] Figure 3A and Figure 3B is a diagram illustrating a method of operating a scan conversion controller according to an embodiment of the inventive concept;

[0012] Figure 4A and Figure 4B are diagrams respectively illustrating a scan conversion method according to an embodiment of the inventive concept;

[0013] Figure 5A and Figure 5B is a block diagram illustrating a scan conversion controller included in an image processing circuit according to an embodiment of the inventive concept;

[0014] Figure 6 is a diagram illustrating data processing in a data packing circuit and an arbitration circuit according to an embodiment of the inventive concept;

[0015] Figure 7 is a diagram illustrating a state in which an index is written or read according to an exemplary embodiment of the inventive concept;

[0016] Figure 8 is a diagram illustrating a memory of a scan conversion controller according to an embodiment of the inventive concept, and is a diagram illustrating a write operation;

[0017] Fig. 9 is a diagram illustrating a memory of a scan conversion controller according to an embodiment of the inventive concept, and is a diagram illustrating a read operation;

[0018] Fig.10 is a diagram illustrating index information stored in a plurality of channel index first-in first-out (FIFO) memories according to packed data written to a memory according to an embodiment of the inventive concept;

[0019] Fig.11 is a diagram illustrating index information stored in a write index FIFO memory according to packed data read from a memory according to an embodiment of the inventive concept;

[0020] FIG. 12A to FIG. 12C is a diagram illustrating an index result output according to a scanning operation mode of image data according to an embodiment of the inventive concept;

[0021] Fig.13 is a diagram illustrating a method of outputting a plurality of multi-context data items when a plurality of multi-context data items having different forms are input according to an embodiment of the inventive concept;

[0022] Fig.14A and Fig. 14B is a diagram illustrating an example of storing multiple context data items in a channel index FIFO and reading the multiple context data items in an example method according to an embodiment of the inventive concept;

[0023] Figure 15 to Figure 16 is a flowchart illustrating a method of operating an image processing circuit according to an embodiment of the inventive concept. DETAILED DESCRIPTION

[0024] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent the same elements, and therefore their description will be omitted. The term "and / or" as used herein includes any and all combinations of one or more items in the relevant listed items. Note that the various aspects described with respect to one embodiment can be combined in different embodiments, although not specifically described with respect to it. That is, all embodiments and / or the features of any embodiment can be combined in any manner and / or combination.

[0025] The term "circuit" used in the embodiments of the present invention is conceived to represent a software or hardware component, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and the "circuit" is configured to perform certain functions. However, the "circuit" is not limited to software or hardware. The "circuit" can be configured in an addressable storage medium and reproduce one or more processors. Therefore, as an example, the "circuit" can include components (such as software components, object-oriented software components, class components, and task components), processes, functions, attributes, processes, subroutines, program code segments, drivers, firmware, microcodes, circuit systems, data, databases, data structures, tables, arrays, and variables.

[0026] Figure 1 is a block diagram of a data processing system 100 according to an embodiment of the inventive concept. Figure 1 , the data processing system 100 may include a processor 200 , a first imaging device 310 and a second imaging device 320 , an external memory 330 , and a display 340 .

[0027] The data processing system 100 may be implemented as a personal computer (PC), an Internet of Things (IoT) device, and / or a mobile computing device. The mobile computing device may be implemented as a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital camera, a digital video camera, a portable multimedia player (PMP), a personal navigation device or a portable navigation device (PND), a handheld game console, a mobile Internet device (MID), a wearable computer, an IoT device, an Internet of Everything (IoE) device, and / or an e-book. In addition, the data processing system 100 may be installed in an electronic device such as a drone, an advanced driver assistance system (ADAS), or an electronic device provided as a component in a vehicle, furniture, a manufacturing facility, a door, and / or various measuring devices. According to an example, the data processing system 100 may be implemented as an electronic device that captures an image, displays the captured image, and / or performs operations based on the captured image.

[0028] The processor 200 may be implemented as an integrated circuit (IC), a mainboard, a system on chip (SoC), an AP, and / or a mobile AP. However, embodiments of the processor 200 are not limited thereto.

[0029] The processor 200 may include a bus structure (or bus) 201 , a central processing unit (CPU) 210 , first and second interfaces 220 - 1 and 220 - 2 , an image processing circuit 230 , a memory controller 250 , and a display controller 270 .

[0030] The CPU 210 , the image processing circuit 230 , the memory controller 250 , and the display controller 270 may send or receive commands and / or data through the bus structure 201 .

[0031] The bus structure 201 can be implemented as a bus using an Advanced Microcontroller Bus Architecture (AMBA) protocol, a bus using an Advanced High-Performance Bus (AHB) protocol, a bus using an Advanced Peripheral Bus (APB) protocol, a bus using an AMBA Scalable Interconnect (AXI) protocol, or a combination thereof. However, the embodiments of the bus structure 201 are not limited thereto.

[0032] The CPU 210 may generally control operations of the processor 200. For example, the CPU 210 may control the first and second interfaces 220-1 and 220-2, the image processing circuit 230, the memory controller 250, and the display controller 270. The CPU 210 may include one or more cores.

[0033] The first interface 220-1 may receive a first image and a first control signal output from the first imaging device 310, and may send the first image and the first control signal to the image processing circuit 230. The second interface 220-2 may receive a second image and a second control signal output from the second imaging device 320, and may send the second image and the second control signal to the image processing circuit 230. According to an example, each of the first interface 220-1 and the second interface 220-2 may include direct memory access (DMA). The first image may refer to a first picture, a first image data, a first data stream, or a first frame data. The second image may refer to a second picture, a second image data, a second data stream, or a second frame data.

[0034] The first imaging device 310 may be implemented as a complementary metal oxide semiconductor (CMOS) image sensor chip or a camera module. The first imaging device 310 may be implemented as a complementary metal oxide semiconductor (CMOS) image sensor chip or a camera module. The camera serial interface (CSI) sends the first image and the first control signal to the first interface 220-1. The second imaging device 320 may be implemented as a CMOS image sensor chip or a camera module. The second imaging device 320 may be implemented as a CMOS image sensor chip or a camera module. The CSI sends the second image and the second control signal to the second interface 220-2. For example, when the data processing system 100 includes a dual camera, the first imaging device 310 may include a rear camera, and the second imaging device 320 may include a front camera. For example, the resolution of the first image may be different from the resolution of the second image.

[0035] The first image and the second image outputted from the first imaging device 310 and the second imaging device 320, respectively, may each include a plurality of row data items. According to an embodiment, the first image outputted from the first imaging device 310 may include a plurality of first row data items. The second image outputted from the second imaging device 320 may include a plurality of second row data items. The image processing circuit 230 may perform TDM on the first image and / or the second image in units of row data rather than in units of frame data. In this document, "data" and "data item" may be used interchangeably.

[0036] For example, the image processing circuit 230 may include a plurality of image signal processor (ISP) cores to process images output from the first imaging device 310 and the second imaging device 320 in units of line data simultaneously, in parallel, or on the fly according to the TDM method. Therefore, the image processing circuit 230 may be implemented as a multi-core ISP.

[0037] For example, each of the plurality of ISP cores may perform at least one of automatic dark level compensation, bad pixel replacement, noise reduction, lens shading compensation, color correction, RGB gamma correction, edge enhancement, hue control, and color suppression.

[0038] The image processing circuit 230 of an embodiment conceived according to the present invention can perform row interleaving to process the first input data among the first image and the second image, and can be controlled to output the first image and the second image in a desired scanning method, regardless of the scanning method of the input first image and the second image. The image processing circuit 230 of an embodiment conceived according to the present invention can determine the scanning method for generating output data according to the first operating mode or the second operating mode. The image processing circuit 230 of an embodiment conceived according to the present invention can control the first image and the second image to be scanned in one of the first operating mode and the second operating mode. According to an example, the first operating mode may include a raster scanning method, and the second operating mode may include a tile scan method. The following will refer to Figure 4A and Figure 4B Describe the raster scanning method and the tile scanning method.

[0039] When processing multi-context data (i.e., various different images), the image processing circuit 230 according to an embodiment of the inventive concept may perform scan conversion, such as tile to raster (T2R), raster to tile (R2T), and tile to tile (T2T), through an internal static random access memory (SRAM). Figure 2 The image processing circuit 230 according to an embodiment of the inventive concept is described in detail.

[0040] Under control performed by the CPU 210 , the memory controller 250 may store line data processed by the image processing circuit 230 in units of line data according to the TDM method in the external memory 330 .

[0041] The display controller 270 may transmit data (eg, frame data) output from the external memory 330 to the display 340 under the control performed by the CPU 210. For example, the display controller 270 may transmit the data (eg, frame data) output from the external memory 330 to the display 340. A display serial interface (DSI) or an embedded display port (eDP) transmits data (eg, frame data) output from the external memory 330 to the display 340 .

[0042] Although in Figure 14 and 5. The first imaging device 310 and the second imaging device 320 are shown in FIG. 4, but the number of imaging devices is not limited thereto.

[0043] Figure 2 It is shown Figure 1 FIG. 2 is a diagram of an example of the image processing circuit 230 .

[0044] refer to Figure 2 , the image processing circuit 230 may receive a plurality of images. According to an example, the plurality of images received by the image processing circuit 230 may be output from the first to nth imaging devices 31_1 to 31_n, respectively. According to an example, each of the first to nth imaging devices 31_1 to 31_n may be implemented as an image sensor chip or a camera module. Each of the first to nth imaging devices 31_1 to 31_n may convert a light signal reflected from an object into an electrical signal, and may generate and output image data based on the electrical signal.

[0045] In an embodiment, one of the first to nth imaging devices 31_1 to 31_n may include a vertical depth sensor that extracts depth information, for example by using infrared (IR). In this case, the imaging device may combine the image data values ​​received from the depth sensor with the image data values ​​received from another image sensor to generate a three-dimensional (3D) depth image.

[0046] In an embodiment, at least two image sensors among the first to nth imaging devices 31_1 to 31_n may have different fields of view. In this case, for example, the optical lenses of the at least two imaging devices may be different from each other. However, embodiments of the inventive concept are not limited thereto.

[0047] refer to Figure 2 , the image processing circuit 230 may include a scan conversion controller 231 and an image signal processor 232. In an embodiment, the scan conversion controller 231 and the image signal processor 232 may each be included in an AP.

[0048] The scan conversion controller 231 may receive a first image from the first imaging device 31_1, and an nth image from the nth imaging device 31_n. For example, the first image may be divided into line data units, and the scan conversion controller 231 may receive first line data LD1 corresponding to the first image through the first channel CH1. The nth image may be divided into line data units, and the scan conversion controller 231 may receive nth line data LDn through the nth channel CHn. In this case, n may be a natural number of 2 or more.

[0049] The scan conversion controller 231 may perform TDM in units of line data. As a result of performing TDM, the scan conversion controller 231 may transmit output data OD to the image signal processor 232.

[0050] The scan conversion controller 231 according to an embodiment of the present invention may perform a first-finish first-out (FFFO) interleaving operation instead of a first-in first-out (FIFO) interleaving operation. In some embodiments, the first-finish first-out (FFFO) interleaving operation outputs data on a channel that has completed data input first among the first to n-th line data items LD1 to LDn, and the first-in first-out interleaving operation outputs the first to n-th line data items LD1 to LDn provided through the first to n-th channels CH1 to CHn in the order in which the first to n-th line data items LD1 to LDn are input. Therefore, the capacity of the memory 2311 in the scan conversion controller 231 can be effectively used, and the scan conversion controller 231 can be implemented to include a relatively small amount of memory. According to an example, the memory 2311 may include an SRAM.

[0051] The scan conversion controller 231 according to an embodiment of the inventive concept may include a memory 2311 configured to store packed data items obtained by packing images received from the first to nth channels CH1 to CHn according to an index assigned to each channel, and a plurality of channel index memories (not shown), wherein the memory 2311 is configured to store index information of the memory 2311 in which the packed data items are stored according to each of the first to nth channels CH1 to CHn. According to an example, the channel index memory may operate in a FIFO method. The channel index memory may be referred to as a channel index FIFO memory.

[0052] When the scan conversion controller 231 according to an embodiment of the inventive concept desires to output the first to nth line data items LD1 to LDn, the scan conversion controller 231 may apply a scanning method according to a desired format to output the first to nth line data items LD1 to LDn. The scan conversion controller 231 may output the first to nth line data items LD1 to LDn having different formats by using different scanning methods. The scan conversion controller 231 may output image data by converting a data scanning method by a raster scanning method or a tile scanning method to match the data format for processing the first to nth line data items LD1 to LDn having different formats by the image signal processor 232. Therefore, an interleaving operation may be performed on a plurality of image data items in the FFFO method, and the plurality of image data items may be easily processed by converting the data scanning method to suit the characteristics of the image signal processor.

[0053] The scan conversion controller 231 may pack the first to nth line data items LD1 to LDn into a constant data size in the memory 2311, and may check the free space of the memory 2311 to allocate indexes to the data items arbitrated as the interleaved first to nth line data items LD1 to LDn. When data is written to the memory 2311 by the allocated indexes, the indexes of the data may be stored in the channel index FIFO memory by channel and row, and the scanning method may be converted according to the required data format to output data having the required data format.

[0054] The image signal processor 232 may process the output data OD output from the scan conversion controller 231. In an embodiment, the image signal processor 232 may include a plurality of cores. However, embodiments of the inventive concept are not limited thereto.

[0055] For example, the image signal processor 232 can reduce noise about the output data OD output from the first to nth imaging devices 31_1 to 31_n through the scan conversion controller 231, and can perform image signal processing to improve image quality, such as gamma correction, color filter array interpolation, color matrix, color correction, and color enhancement. In addition, an image file can be generated by compressing the image data generated by the image signal processing to improve the image quality, or the image data can be restored from the image file.

[0056] In addition to the aforementioned image processing operations, the image signal processor 232 may perform an operation of converting the format of image data into complete image data in red, green, and blue.

[0057] Figure 3A and Figure 3B is a diagram illustrating a method of operating the scan conversion controller 231 a according to an embodiment.

[0058] refer to Figure 3A , the data processing device 100a may include: an ISP chain 232a including a plurality of ISP cores, a scan conversion controller 231a, a DMA 280a, and a bus 201a. The ISP chain 232a may correspond to Figure 2 The image signal processor 232, the scan conversion controller 231a may correspond to Figure 2 The scan conversion controller 231, bus 201a may correspond to Figure 1 Since the descriptions of the corresponding components are the same, their repeated descriptions are omitted.

[0059] refer to Figure 3ADMA 280a may be connected between scan conversion controller 231a and bus 201a. DMA 280a may send data to and receive data from a storage device connected to bus 201a. According to an example, data input to scan conversion controller 231a may include data from, for example, Figure 2 The data is input by the imaging device shown, or the data is sent and received through DMA 280a.

[0060] refer to Figure 3B , the data processing device 100b may include: an ISP chain 232b including a plurality of ISP cores, a first scan conversion controller 231b_1, a second scan conversion controller 231b_2, a read direct memory access (RDMA) 280b_1, a write direct memory access (WDMA) 280b_2, compressors 290b_1 and 290b_2, and a bus 201b. Because included in Figure 3B Configuration and reference of the ISP chain 232b, the first scan conversion controller 231b_1, the second scan conversion controller 231b_2 and the bus 201b in the data processing device 100b Figure 3A The descriptions are the same, so their repeated descriptions are omitted.

[0061] According to an example, the RDMA 280b_1 and the compressor 290b_1 may be connected between the first scan conversion controller 231b_1 and the bus 201b. The RDMA 280b_1 may read data stored in an external storage device and may directly send the data to an internal memory provided in the first scan conversion controller 231b_1. The compressor 290b_1 may compress the data sent through the RDMA 280b_1 at a constant rate.

[0062] According to an example, the first scan conversion controller 231b_1 may convert the scanning method into the T2R method. For example, the compressor 290b_1 may compress the data received through the bus 201b in the tile scanning method, and the data compressed by the compressor 290b_1 may be received by the first scan conversion controller 231b_1 via the RDMA 280b_1. In the ISP chain 232b, when the processing is performed in the raster scanning method, the first scan conversion controller 231b_1 may convert the received tile type data into raster scanning type data.

[0063] According to an example, WDMA280b_2 and compressor 290b_2 may be connected between the second scan conversion controller 231b_2 and the bus 201b. WDMA280b_2 may directly send data stored in the internal memory of the second scan conversion controller 231b_2 to an external memory device, and may write the data in the external memory device. Compressor 290b_2 may compress data sent by WDMA280b_2 at a constant rate. According to an example, the second scan conversion controller 231b_2 may convert a scanning method into an R2T method. According to an example, in the ISP chain 232b, when data is to be processed in a raster scanning method, the second scan conversion controller 231b_2 may receive data processed in a raster scanning method, may convert the data into tile-type data, and may send the tile-type data to WDMA280b_2. The data sent to WDMA280b_2 may be processed by compressor 290b_2 and sent to bus 201b.

[0064] refer to Figure 3A and Figure 3B In the ISP chains 232a and 232b, when processing is to be performed in a raster scan method or a tile scan method, a T2R or R2T conversion module may be required depending on the type of input data. According to an example, the scan conversion controllers 231a, 231b_1, and 231b_2 according to the present invention can be applied to tile-based processing in the case of a neural network (NN), a local motion (LM), and a compressor.

[0065] The scan conversion controller according to an embodiment of the present invention can support multi-context data processing in the same area, and can be used as a row interleaving controller and a throttle buffer. According to an example, the role of the row interleaving controller can be performed by a scan conversion controller connected to RDMA, and the role of the throttle buffer can be performed by a scan conversion controller connected to WDMA. According to an example, the compressor can compress data in units of tiles.

[0066] A high bandwidth (BW) is required to store images obtained by multiple image devices in a memory so that the images can be utilized in various ways. However, because the BW of the bus used for communication is limited, it is necessary to reduce the BW of the image to be stored by a compressor. Because compressing tiled images has a much higher compression rate than compressing images processed by a raster scanning method, it may be desirable to perform tile processing to improve the compression rate of the compressor. According to a comparative example, because a high BW is desired, processing under a limited BW may not be performed according to the bus configuration. However, in the case of a scan conversion controller according to an embodiment of the present invention, although multiple images are input simultaneously, multiple images can be processed by using one module instead of using modules corresponding to multiple images one by one. Therefore, it may be effective in terms of area reduction, and the number of WDMAs can be reduced. In addition, power can be reduced by reducing the area.

[0067] Figure 4A and Figure 4B are diagrams respectively illustrating a scan conversion method according to an embodiment of the inventive concept.

[0068] The raster scanning method can refer to scanning the image data sequentially line by line. Figure 4A , when the total width of the input image data is W, by using a raster scanning method, the first line data as much as W as the total width is read, and then the second line data as much as W is continued to be read, so that the image data can be scanned.

[0069] The tile scanning method may mean: setting a tile area having a size smaller than a specific area of ​​the image data, and sequentially scanning each row in the tile area. Figure 4A , the width of the set tile area may be the tile width (tilewidth, TW), and the height of the tile area may be the tile height (tile height, TH). By using the tile scanning method, the image data may be scanned by reading the first row of data as much as the width TW of the tile area, and then reading the second row of data as much as TW until the row corresponding to the height TH of the tile area.

[0070] In the present disclosure, a “raster scanning method” and a “raster scanning mode” may be described with the same meaning, and a “tile scanning method” and a “tile scanning mode” may also be described with the same meaning.

[0071] Figure 4A An example of switching from a raster scanning method to a tile scanning method is shown. Figure 4BAn example of switching from a tile scanning method to a raster scanning method is shown. As described above, it is shown that although the same image data is scanned, the data scanning method can be switched according to the method to be processed. According to an example of the present invention, Figure 4A and Figure 4B The R2T and T2R shown in FIG. 1 may correspond to a scan conversion controller according to the concept of the present invention. Figure 4A and Figure 4B Only R2T or T2R is described, but it should be noted that the T2T method of conversion between different tile regions may also be performed by the scan conversion controller.

[0072] Figure 5A and Figure 5B is a block diagram illustrating scan conversion controllers 1000 and 1000a included in an image processing circuit according to an embodiment. Figure 5A 2 shows that the scan conversion controller 1000 receives image data from four (ie, n=4) image sensors. However, the scan conversion controller according to an embodiment of the inventive concept is not limited thereto, and n may have various values.

[0073] refer to Figure 5A , the scan conversion controller 1000 may include a plurality of data packing circuits (e.g., first to fourth data packing circuits 110_1 to 110_4), an arbitration circuit and write controller 1200, a row index controller 1300, a memory 1400, and a read controller and unpacking circuit 1500. The scan conversion controller 1000 may receive first to fourth row data items LD1 to LD4 from the first to fourth image sensors 300_1 to 300_4 through first to fourth channels CH1 to CH4, respectively.

[0074] The first data packing circuit 110_1 may receive the first row data LD1 from the first channel CH1, and may pack the first row data LD1 into a predetermined data size to generate the first packed data PD1. The second data packing circuit 110_2 may receive the second row data LD2 from the second channel CH2, and may pack the second row data LD2 into a predetermined data size to generate the second packed data PD2. The third data packing circuit 110_3 may receive the third row data LD3 from the third channel CH3, and may pack the third row data LD3 into a predetermined data size to generate the third packed data PD3. The fourth data packing circuit 110_4 may receive the fourth row data LD4 from the fourth channel CH4, and may pack the fourth row data LD4 into a predetermined data size to generate the fourth packed data PD4.

[0075] In an embodiment, a predetermined data size of the packed data as a packing unit of the packing operation may be determined according to at least one of the number of channels connected to the scan conversion controller 1000, the maximum number of bits received during one clock signal cycle in the line data received from the plurality of channels, and the number of pixels per clock signal cycle (PPC) of the line data. For example, the data size of the packed data may be determined by multiplying the number of channels connected to the scan conversion controller 1000, the maximum number of bits received during one clock signal cycle in different line data items received from the plurality of channels, and the number of PPCs of the line data.

[0076] The arbitration circuit and write controller 1200 may include an arbitration circuit that receives the first to fourth packed data items PD1 to PD4 from the first to fourth data packing circuits 110_1 to 110_4 and sequentially outputs the first to fourth packed data items PD1 to PD4 as packed data, and a write controller that controls a write operation of the memory 1400. Because the arbitration circuit performs arbitration on the first to fourth packed data items PD1 to PD4 as packed data items, although any data among the first to fourth packed data items PD1 to PD4 is selected as packed data output, the data may be selected and output only once in every at least 4 cycles of the clock signal. That is, although the first to fourth row data items LD1 to LD4 are continuously input from the first to fourth channels CH1 to CH4, although there is no separate storage of the first to fourth packed data items PD1 to PD4, the arbitration circuit may arbitrate the first to fourth packed data items PD1 to PD4.

[0077] The write controller may receive the packed data from the arbitration circuit, and may store (write) the packed data in the memory 1400. The write controller may allocate indexes according to the channels corresponding to the packed data, and may send index information INDI to the row index controller 1300 according to the allocated indexes. The write controller may convert the allocated indexes into a plurality of write addresses WR_ADDR, and may send the packed data as write data WR_DT to the memory 1400. Therefore, the first to fourth packed data items PD1 to PD4 may be sequentially provided to the memory 1400 as write data WR_DT. In an embodiment, the data size corresponding to one index may be a multiple of the data size of the packed data, for example, 8 times or 16 times.

[0078] The row index controller 1300 may manage a plurality of channels, that is, indexes corresponding to the first to fourth channels CH1 to CH4. For example, the row index controller 1300 may include first to fourth memories storing index information corresponding to the first to fourth channels CH1 to CH4, and may store the index information INDI in a corresponding memory among the first to fourth memories according to the index information INDI received from the arbitration circuit and the write controller 1200. The row index controller 1300 may determine a scanning method of any one of the first to fourth memories storing the index information corresponding to the first to fourth channels CH1 to CH4.

[0079] In addition, the row index controller 1300 may provide the read controller and unpacking circuit 1500 with index information INDO about the index to be read to perform a read operation on the memory 1400. That is, the read controller and unpacking circuit 1500 may be provided with index information INDO about the index corresponding to the channel for which the image data input is completed among the first to fourth channels CH1 to CH4.

[0080] When transmitting index information INDO about an index to be read, the row index controller 1300 may transmit the index information INDO by applying a scanning method corresponding to an image processing method in an image processing circuit. According to an example, the row index controller 1300 may transmit index information INDO for reading an index under a raster scanning method or a tile scanning method.

[0081] For example, the memory 1400 may include, for example, an SRAM. However, in other embodiments, the memory 1400 may be implemented as another type of memory. The memory 1400 may store (write) the packed data in the memory area according to the assigned indexes based on the plurality of write addresses WR_ADDR and write data DT provided by the arbitration circuit and the write controller 1200.

[0082] In addition, the memory 1400 may output (read) read data RD_DT according to a plurality of read addresses RD_ADDR provided by the read controller and the unpacking circuit 1500. The plurality of read addresses RD_ADDR may correspond to a specific index, and the packed data items of the channels corresponding to the specific index may be read as the read data RD_DT. Figures 8 to 9 A description is given of a write operation and a read operation of the memory 1400 .

[0083] The read controller and unpacking circuit 1500 may include a read controller that controls a read operation of the memory 1400, and an unpacking circuit that outputs output data OD by unpacking the read data RD_DT output from the read controller. The unpacking circuit may unpack the output data OD to have the same data form as each of the first to fourth row data items LD1 to LD4.

[0084] The read controller may provide a plurality of read addresses RD_ADDR to the memory 1400, and may receive read data RD_DT from the memory 1400. The read controller may receive index information INDO about an index to be read (i.e., an index corresponding to a channel among the first to fourth channels CH1 to CH4 for which image data input is completed) from the row index controller 1300, and may generate a plurality of read addresses RD_ADDR based on the index information INDO. Therefore, the read controller may not read the read data RD_DT among the first to fourth row data items LD1 to LD4 provided through the first to fourth channels CH1 to CH4 from the memory 140 in the order of first input, but may read the read data RD_DT of the channel among the first to fourth channels CH1 to CH4 for which image data input is completed first based on the index information INDO.

[0085] The scan conversion controller 1000 according to an embodiment of the inventive concept does not output the first to nth line data items LD1 to LDn provided through the first to nth channels CH1 to CHn as output data OD in the order of first input. In the controller of the comparative example that performs the interleaving operation in a first-in-first-out manner, when the speed of the first image sensor 300_1 is low and the speed of the second image sensor 300_2 is high, when the first line data LD1 is first received from the first image sensor 300_1, although a larger amount of second line data LD2 is provided from the second image sensor 300_2, the second line data LD2 is accumulated in the memory of the scan conversion controller of the comparative example until the first line data LD1 of the first image sensor 300_1 is output as output data. Therefore, the throughput of the scan conversion controller in the comparative example is reduced, and the memory becomes full.

[0086] On the other hand, according to an embodiment of the present invention, the scan conversion controller 1000 does not perform an interleaving operation in a FIFO manner, but performs an FFFO interleaving operation in which the image data of the channel that first completes the image data input among the first to fourth image data items is first output. In addition, the scan conversion controller 1000 can convert the output data OD to scan the image data in a desired method by sending index information INDO to the reading controller differently according to the method used for scanning the image data. Therefore, the processing rate of the scan conversion controller 1000 is improved, and the memory 1400 can be effectively used.

[0087] Figure 5B is a block diagram of a scan conversion controller 1000a configured according to an embodiment. Figure 5B The scan conversion controller 1000a is Figure 5A An example of a scan conversion controller. Figure 5B In Figure 5A The same reference numerals as those for the components in FIG. Figure 5A The description is the same as the description of .

[0088] refer to Figure 5A and Figure 5B , the scan conversion controller 1000a may include an arbitration circuit 1210, a write controller 1220a, a row index controller 1300a, a memory 1400, a read controller 1510, and an unpacking circuit 1520. Figure 5B Although not shown in the figure, the scan conversion controller 1000a may also include a plurality of data packing circuits.

[0089] The arbitration circuit 1210 may receive the first to fourth packed data items PD1 to PD4, and may sequentially output the first to fourth packed data items PD1 to PD4 as packed data PD. When two or more of the first to fourth packed data items PD1 to PD4 are simultaneously received, the arbitration circuit 1210 may select the received packed data items one by one, and may sequentially output the packed data items.

[0090] The write controller 1220a may include an address generator 1220, separation circuits 1230 and 1260, a write index FIFO memory 1240, and a register slice circuit 1250. The register slice circuit 1250 may receive the packed data PD from the arbitration circuit 1210 and may send the packed data to the address generator 1220.

[0091] The separation circuit 1260 may receive a plurality of write addresses WR_ADDR and write data WR_DT from the address generator 1220, and may output channel information ICH about a channel through which image data is transmitted to the row index controller 1300a when the write data WR_DT is the last packed data of the image data. That is, the channel information ICH about a channel through which the row data input is completed may be output. The separation circuit 1260 may transmit the plurality of write addresses WR_ADDR and write data WR_DT to the memory 1400.

[0092] The separation circuit 1230 may receive the write index information WR_FOUT including the null index information and may provide the row index controller 1300a with index information INDI about the index in which the packed data PD is to be stored based on the write index information WR_FOUT. The separation circuit 1230 may generate an address ADDR corresponding to the index information INDI based on the write index information WR_FOUT.

[0093] The write index FIFO memory 1240 may store index information about an index according to an area where a data write operation may be performed, that is, empty index information in the memory 1400. The write index FIFO memory 1240 may receive index information WR_FIN about an index on which a read operation is performed from the read controller 1510, and may update the empty index information. That is, the write index FIFO memory 1240 may be updated when a write operation is performed on the memory 1400 and when a read operation is performed on the memory 1400.

[0094] The row index controller 1300a may include a demultiplexer 1310, a plurality of channel index FIFO memories (e.g., first to fourth channel index FIFO memories 1320_1 to 1320_4), and a multiplexer and index controller 1330. The demultiplexer 1310 may receive index information INDI from the write controller 1220a, and may send the index information INDI to one of the first to fourth channel index FIFO memories 1320_1 to 1320_4. For example, when the index information INDI corresponds to the first channel CH1, the index information INDI may be provided to the first channel index FIFO memory 1320_1 as the first index information IND1. Alternatively, for example, when the index information INDI corresponds to one of the second to fourth channels CH2 to CH4, one of the second to fourth index information IND2 to IND4 may be provided to the corresponding channel index FIFO memory among the second to fourth channel index FIFO memories 1320_2 to 1320_4. Each of the first to fourth index information IND1 to IND4 may include index information about an area in which packed data according to a corresponding channel among the first to fourth channels CH1 to CH4 is written to the memory 1400 .

[0095] According to an embodiment of the present invention, the channel index FIFO memory may be a channel index memory operated in a FIFO method. According to an embodiment of the present invention, on the premise that the channel index memory operates in a FIFO method, the channel index memory included in the row index controller 1300a is referred to as a channel index FIFO memory. However, the embodiment of the present invention is not limited thereto. According to an embodiment of the present invention, the channel index memory included in the row index controller 1300a may not operate in a FIFO method. According to an embodiment of the present invention, for convenience, the channel index memory included in the row index controller 1300a is referred to as a channel index FIFO memory.

[0096] The number of channel index FIFO memories included in the row index controller 1300a may be the same as the number of channels connected to the scan conversion controller 1000a. When the number of channels connected to the scan conversion controller 1000a changes, the number of channel index FIFO memories included in the row index controller 1300a may also change. The first channel index FIFO memory 1320_1 may store index information of the memory 1400 in which the first packed data PD1 received from the first channel CH1 is stored, the second channel index FIFO memory 1320_2 may store index information of the memory 1400 in which the second packed data PD2 received from the second channel CH2 is stored, the third channel index FIFO memory 1320_3 may store index information of the memory 1400 in which the third packed data PD3 received from the third channel CH3 is stored, and the fourth channel index FIFO memory 1320_4 may store index information of the memory 1400 in which the fourth packed data PD4 received from the fourth channel CH4 is stored. That is, information about indexes allocated to the first to fourth channels CH1 to CH4 may be stored in the first to fourth channel index FIFO memories 1320_1 to 1320_4, respectively.

[0097] The multiplexer and index controller 1330 may receive first to fourth index information IND1 to IND4 from the first to fourth channel index FIFO memories 1320_1 to 1320_4. The multiplexer and index controller 1330 may provide the read controller 1510 with index information INDO regarding an index corresponding to a channel among the first to fourth channels CH1 to CH4 for which image data input is completed. For example, when image data input is completed on the third channel CH3, the multiplexer and index controller 1330 may provide the third index information IND3 received from the third channel index FIFO memory 1320_3 as the index information INDO to the read controller 1510.

[0098] The multiplexer and index controller 1330 may output index information INDO for a read operation based on information stored in a channel index FIFO memory corresponding to a channel that has completed inputting packed data first among the packed data items.

[0099] According to an example, in the first operating mode, the multiplexer and index controller 1330 can output index information INDO, thereby sequentially reading the indexes stored in the channel index FIFO memory corresponding to the channel that first completed the input of the packed data among the packed data items row by row starting from the first row with the entire width of the channel index FIFO memory.

[0100] According to an example, in the second operation mode, the multiplexer and index controller 1330 may output the index information INDO by sequentially reading the index stored in the channel index FIFO memory corresponding to the channel of the packed data item that has completed the input of the packed data first, row by row, starting from the first row with the width of the set tile area, until as many rows as the number corresponding to the height of the set tile area, which will be referred to below. FIG. 12A to FIG. 12C A specific example thereof is described.

[0101] According to an example, when a plurality of packed data items are randomly input, the multiplexer and index controller 1330 may output the index information immediately when there is no need to convert the scanning method of the input index information and there is no first completed packed data input, and may output the obtained index information after outputting the packed data when there is a first completed packed data input, which will be referred to below. Fig.13 A specific example thereof is described.

[0102] The multiplexer and index controller 1330 may transmit index information WR_FIN about the index on which the read operation is performed to the write controller 1220a. The index information WR_FIN about the index on which the read operation is performed may include the same information as the index information INDO about the index corresponding to the channel for which the image data input is completed. For example, in the multiplexer and index controller 1330, the index according to which data is written in the memory 1400 may be displayed as "1", and the index according to which data is read from the memory 1400 (e.g., an empty index) may be displayed as "0" for storage.

[0103] The row index controller 1300a may include at least one priority memory for storing priority information about a plurality of channels (e.g., first to fourth channels CH1 to CH4) according to the channel information ICH received from the write controller 1220a. For example, the row index controller 1300a may include a first priority FIFO memory 1340_1 and a second priority FIFO memory 1340_2. The first priority FIFO memory 1340_1 may store information about a channel having a relatively high data processing priority, and the second priority FIFO memory 1340_2 may store information about a channel having a relatively low data processing priority. Although in Figure 5B Only the first priority FIFO memory 1340_1 and the second priority FIFO memory 1340_2 are shown in FIG. 1 , but embodiments of the inventive concept are not limited thereto, and the row index controller 1300 a may include one priority FIFO memory or three or more priority FIFO memories.

[0104] For example, row data input may be completed for both the third channel CH3 and the second channel CH2, and in this case, the processing priority of the third channel CH3 may be higher than the processing priority of the second channel CH2. Information about the third channel CH3 may be stored in the first priority FIFO memory 1340_1 according to the channel information ICH of the third channel CH3, and information about the second channel CH2 may be stored in the second priority FIFO memory 1340_2 according to the channel information ICH of the second channel CH2.

[0105] The multiplexer and index controller 1330 may provide the index information INDO to the read controller 1510 for reading the packed data of the channel corresponding to the first priority FIFO memory 1340_1 in priority to the packed data of the channel corresponding to the second priority FIFO memory 1340_2. For example, when information about the third channel CH3 is stored in the first priority FIFO memory 1340_1 and information about the second channel CH2 is stored in the second priority FIFO memory 1340_2, the index information INDO may be provided to the read controller 1510 to read the packed data of the second channel CH2 from the memory 1400 in priority to the packed data of the third channel CH3. Therefore, the scan conversion controller 1000a may output the output data OD corresponding to the third line data provided through the third channel CH3, and then may output the output data OD corresponding to the second line data provided through the second channel CH2.

[0106] The read controller 1510 may provide a read address RD_ADDR to the memory 1400 based on the index information INDO, and may receive read data RD_DT from the memory 1400. The unpacking circuit 1520 may unpack the read data RD_DT and output it as output data OD so that the output data OD has the same data format as each of the first to fourth line data LD1 to LD4.

[0107] Figure 6 is a schematic diagram illustrating data processing in a data packing circuit and an arbitration circuit according to an embodiment of the inventive concept.

[0108] Figure 6 Four different input data items, the result of packing the four different input data items, and the result of arbitrating the packed data items in order are shown.

[0109] Figure 6The four different input data items INPUT_CTX0, INPUT_CTX1, INPUT_CTX2 and INPUT_CTX3 in can represent the input data items input to the scan conversion controller through different channels. According to the example, the input data items INPUT_CTX0, INPUT_CTX1, INPUT_CTX2 and INPUT_CTX3 can be row data items input through different channels. Different input data items INPUT_CTX0, INPUT_CTX1, INPUT_CTX2 and INPUT_CTX3 can be input according to the cycle of the clock signal with a specific period of time switched (toggle). Different input data items INPUT_CTX0, INPUT_CTX1, INPUT_CTX2 and INPUT_CTX3 can have different sizes.

[0110] Different input data items INPUT_CTX0, INPUT_CTX1, INPUT_CTX2, and INPUT_CTX3 may be packed into a predetermined size by a data packing circuit. In an embodiment, the data sizes of the first to fourth packed data items PK_CTX0, PK_CTX1, PK_CTX2, and PK_CTX3 may be determined by multiplying the number of channels connected to the scan conversion controller, the maximum number of bits received during one clock signal cycle in different row data items received by the plurality of channels, and the number of pixels per clock signal cycle of the row data by each other.

[0111] refer to Figure 6 , the output timing and output interval of the first packed data PK_CTX0, the second packed data PK_CTX1, the third packed data PK_CTX2 and the fourth packed data PX_CTX3 may be the same or different. The first packed data PK_CTX0, the second packed data PK_CTX1, the third packed data PK_CTX2 and the fourth packed data PX_CTX3 may include information about the corresponding input data items INPUT_CTX0, INPUT_CTX1, INPUT_CTX2 and INPUT_CTX3.

[0112] The arbitration circuit may arbitrate the first, second, third, and fourth packed data PK_CTX0, PK_CTX1, PK_CTX2, and PX_CTX3 at each time point of output, and may control the first, second, third, and fourth packed data PK_CTX0, PK_CTX1, PK_CTX2, and PX_CTX3 to be sequentially output.

[0113] refer to Figure 6The output data Arbitration Out of the arbitration circuit, 0 represents the first packaged data PK_CTX0, 1 represents the second packaged data PK_CTX1, 2 represents the third packaged data PK_CTX2, and 3 represents the fourth packaged data PK_CTX3. The first packaged data PK_CTX0 and the third packaged data PK_CTX2 outputted simultaneously may be outputted sequentially, and the second packaged data PK_CTX1 outputted thereafter may be outputted sequentially. Therefore, 0, 2, and 1 may be outputted sequentially. Next, the third packaged data PK_CTX2 may be outputted first, the first packaged data PK_CTX0 and the second packaged data PK_CTX1 outputted simultaneously may be outputted sequentially, and then the fourth packaged data PK_CTX3 may be outputted. Therefore, 2, 0, 1, and 3 may be outputted sequentially.

[0114] refer to Figure 6 When data items are input through different channels, the data items may be packed by data packing, and the packed data items may be sequentially output through an arbitration circuit to be arbitrated. The packed data items may be addressed through the channels. According to an example, one index may include a plurality of addresses.

[0115] Figure 7 is a diagram illustrating a state in which an index is written or read according to an exemplary embodiment of the inventive concept. Figure 7 , showing the index in the initial state and the index state when the empty index is checked, written or read.

[0116] Figure 7 The index assignment can be done by Figure 5B The write controller 1220a performs the reference Figure 5B and Figure 7 In order to allocate the index to the memory, the free space of the index may be checked and stored in the write index FIFO memory 1240. At this time, the empty index may be indicated as "0", and the index to which writing is performed to be filled may be indicated as "1". In addition, the index that has been read may be indicated as "0" again.

[0117] exist Figure 7 In the initial state of , all indexes may be empty. Therefore, all corresponding bits are indicated as 0. According to an example, the size of data corresponding to one index may be a constant multiple of the size of the packed data.

[0118] When checking Figure 7When an empty index is received, bits corresponding to indexes 0 to 9 are changed to 1, and bits of the remaining indexes are 0. In this case, the write index FIFO memory WR_FIFO may store information about which bits' indexes are changed to 1 and which bits' indexes remain at 0.

[0119] exist Figure 7 In the example of index allocation for performing writing or reading in , the bits of the indexes (index 1, index 5, and index 8) for which reading has been completed are changed to 0. That is, for the indexes for which reading has been completed, the bits are changed from 1 to 0 again to become empty indexes, which can be stored in the write index FIFO memory WR_FIFO.

[0120] refer to Figure 7 , showing that in order to allocate a data item arbitrated by an arbitration circuit, the free space of the index is checked, and after the index is updated to reflect the check, the index space for writing can be ensured. Figure 7 , you can determine where is the index to write to and which index to read from.

[0121] Figure 8 is a diagram illustrating a memory 1400 of a scan conversion controller according to an embodiment of the inventive concept, and is a diagram illustrating a write operation. Fig. 9 is a diagram illustrating a memory 1400 of a scan conversion controller according to an embodiment of the inventive concept, and is a diagram illustrating a read operation. Fig.10 is a diagram showing index information stored in a plurality of channel index FIFO memories according to packed data written into the memory. Fig.11 is a diagram showing index information stored in a write index FIFO memory according to packed data read from a memory.

[0122] refer to Figures 8 to 10 , the area of ​​memory 1400 can be divided into multiple indexes. For example, 192 indexes (e.g., index 0 to index 191) can be specified for memory 1400. However, embodiments of the inventive concept are not limited thereto. At this time, the data size of the index can be a multiple of the data size of the packed data, such as 8 times or 16 times. For example, when the data size of the packed data is 448 bits, the data size of the index can be 3584 bits, which is 8 times the data size of 448 bits.

[0123] The first index INDEX0, the second index INDEX1, the fifth index INDEX4, the ninth index INDEX8, and the thirteenth index INDEX12 may be assigned to the first channel CH1. That is, the first packed data PD1 in which the first line data LD1 provided by the first channel CH1 is packed may be stored in the area indicated by the first index INDEX0, the second index INDEX1, the fifth index INDEX4, the ninth index INDEX8, and the thirteenth index INDEX12. Therefore, index information about the first index INDEX0, the second index INDEX1, the fifth index INDEX4, the ninth index INDEX8, and the thirteenth index INDEX12 may be stored in the first channel index FIFO memory 1320_1.

[0124] The third index INDEX2, the seventh index INDEX6, the tenth index INDEX9, the eleventh index INDEX10, and the fourteenth index INDEX13 may be assigned to the second channel CH2. That is, the second packed data PD2 in which the second line data LD2 provided by the second channel CH2 is packed may be stored in the area indicated by the third index INDEX2, the seventh index INDEX6, the tenth index INDEX9, the eleventh index INDEX10, and the fourteenth index INDEX13. Therefore, index information about the third index INDEX2, the seventh index INDEX6, the tenth index INDEX9, the eleventh index INDEX10, and the fourteenth index INDEX13 may be stored in the second channel index FIFO memory 1320_2.

[0125] The sixth index INDEX5, the eighth index INDEX7, the twelfth index INDEX11, the fifteenth index INDEX14, the sixteenth index INDEX15, and the eighteenth index INDEX17 may be allocated to the third channel CH3. That is, the third packed data PD3 in which the third line data LD3 provided by the third channel CH3 is packed may be stored in the area indicated by the sixth index INDEX5, the eighth index INDEX7, the twelfth index INDEX11, the fifteenth index INDEX14, the sixteenth index INDEX15, and the eighteenth index INDEX17. Therefore, index information about the sixth index INDEX5, the eighth index INDEX7, the twelfth index INDEX11, the fifteenth index INDEX14, the sixteenth index INDEX15, and the eighteenth index INDEX17 may be stored in the third channel index FIFO memory 1320_3.

[0126] The fourth index INDEX3, the seventeenth index INDEX16, the nineteenth index INDEX18, and the twentieth index INDEX19 may be assigned to the fourth channel CH4. That is, the fourth packed data PD4 in which the fourth line data LD4 provided by the fourth channel CH4 is packed may be stored in the area indicated by the fourth index INDEX3, the seventeenth index INDEX16, the nineteenth index INDEX18, and the twenty-first index INDEX19. Therefore, index information about the fourth index INDEX3, the seventeenth index INDEX16, the nineteenth index INDEX18, and the twenty-first index INDEX19 may be stored in the fourth channel index FIFO memory 1320_4.

[0127] At this time, the image data input from the third channel CH3 may be completed first. The third packed data PD3 written at the eighteenth index INDEX17 may be the last packed data. Therefore, all third packed data PD3 corresponding to the third channel CH3 may be read from the memory 1400.

[0128] refer to Figures 8 to 11 , the image data input of the third channel CH3 from among the first to fourth channels CH1 to CH4 may be completed first. The third packed data PD3 written in the eighteenth index INDEX17 may be the last packed data, and all the third packed data PD3 corresponding to the third channel CH3 may be read from the areas indicated by the sixth index INDEX5, the eighth index INDEX7, the twelfth index INDEX11, the fifteenth index INDEX14, the sixteenth index INDEX15, and the eighteenth index INDEX17 of the memory 1400.

[0129] Since new packed data may be written in the areas indicated by the sixth index INDEX5, the eighth index INDEX7, the twelfth index INDEX11, the fifteenth index INDEX14, the sixteenth index INDEX15, and the eighteenth index INDEX17, the sixth index INDEX5, the eighth index INDEX7, the twelfth index INDEX11, the fifteenth index INDEX14, the sixteenth index INDEX15, and the eighteenth index INDEX17 may be classified as empty indexes. In the write index FIFO memory, index information about the sixth index INDEX5, the eighth index INDEX7, the twelfth index INDEX11, the fifteenth index INDEX14, the sixteenth index INDEX15, and the eighteenth index INDEX17 may be newly stored and updated. Based on the index information about the sixth index INDEX5, the eighth index INDEX7, the twelfth index INDEX11, the fifteenth index INDEX14, the sixteenth index INDEX15, and the eighteenth index INDEX17 stored in the write index FIFO memory, the write controller may store new packed data in areas corresponding to the sixth index INDEX5, the eighth index INDEX7, the twelfth index INDEX11, the fifteenth index INDEX14, the sixteenth index INDEX15, and the eighteenth index INDEX17. Since the scan conversion controller 1000 according to an embodiment of the inventive concept prioritizes output of image data of channels for which image data input has been completed, the memory 1400 may be effectively used.

[0130] At this point, you can delete the Fig.10 Index information about a sixth index INDEX5, an eighth index INDEX7, a twelfth index INDEX11, a fifteenth index INDEX14, a sixteenth index INDEX15, and an eighteenth index INDEX17 in the third channel index FIFO memory 1320_3.

[0131] FIG. 12A to FIG. 12C is a diagram illustrating index results output according to a scan operation mode of image data according to some embodiments of the inventive concept.

[0132] Fig. 12A 1 is a diagram showing indexes to which first packed data ctx0 is assigned row by row according to an example. For convenience, a description is given based on the first packed data ctx0. However, it should be noted that embodiments of the inventive concept can be equally applied to a plurality of packed data items.

[0133] refer to Fig. 12A, the row data corresponding to the first row L0 included in the first packed data ctx0 may be allocated to the first index INDEX0, the fourteenth index INDEX13, the seventeenth index INDEX16, and the twentieth index INDEX19. The row data corresponding to the second row L1 included in the first packed data ctx0 may be allocated to the fourth index INDEX3, the tenth index INDEX9, the twelfth index INDEX11, and the twenty-first index INDEX22. The row data corresponding to the third row L2 included in the first packed data ctx0 may be allocated to the sixth index INDEX5, the eighth index INDEX7, the nineteenth index INDEX18, and the thirty-first index INDEX30. The row data corresponding to the fourth row L3 included in the first packed data ctx0 may be allocated to the sixteenth index INDEX15, the twenty-eighth index INDEX27, the thirty-third index INDEX32, and the thirty-fourth index INDEX33.

[0134] Fig. 12B and Fig. 12C It is shown that by using Fig. 12A An example of allocated index information determines a map of index outputs for data reading differently according to a scan operation mode.

[0135] refer to Fig. 12B , showing that according to Fig. 12A The example allocates the index information in rows. Fig. 12C , a table for explaining a method of reading an index when the scanning operation mode is determined to be a raster scanning mode and when the scanning operation mode is determined to be a tile scanning mode is shown.

[0136] According to an example, when the scanning operation mode is a raster scanning mode, the scanning method may correspond to T2R. When the scanning operation mode is a raster scanning mode, an amount corresponding to the entire width is read from the first line of data, and then the next line of data may be sequentially read. Fig. 12B and Fig. 12C , when the scanning operation mode is the raster scanning mode, indexes 0, 13, 16, and 19 corresponding to the entire width may be output from the first row L0, and indexes 3, 9, 11, and 22 corresponding to the entire width may be output from the second row L1. Then, indexes 5, 7, 18, and 30 corresponding to the entire width may be output from the third row L2, and indexes 15, 27, 32, and 33 corresponding to the entire width may be output from the fourth row L3.

[0137] According to an example, when the scanning operation mode is a tile scanning mode, the scanning method may correspond to R2T. When the scanning operation mode is a tile scanning mode, the output data may vary according to the size of the tile area to be scanned. In the tile scanning mode, the first row L0, the second row L1, the third row L2, and the fourth row L3 may be sequentially scanned up to the tile width of the corresponding tile area, and may be sequentially scanned until a row corresponding to the tile height of the tile area.

[0138] According to the example, it is assumed that the size of the tile area to be scanned is 64×4. At this time, it is assumed that the number of data items stored in one index is 64. Fig. 12B , when the size of the tile area to be scanned is 64×4, the corresponding tile area is shown. When the tile area is 64×4, the first row L0, the second row L1, the third row L2, and the fourth row L3 can be read sequentially up to the tile width. At this time, because the height of the tile area is 4, the first row L0, the second row L1, the third row L2, and the fourth row L3 can be read sequentially. Therefore, the indexes can be output in the order of indexes 0, 3, 5, and 15 corresponding to the first tile area, indexes 13, 9, 7, and 27 corresponding to the second tile area, indexes 16, 11, 18, and 32 corresponding to the third tile area, and indexes 19, 22, 30, and 33 corresponding to the fourth tile area.

[0139] According to the example, it is assumed that the size of the tile area to be scanned is 128×4. At this time, it is assumed that the number of data items stored in one index is 64. Fig. 12B , when the size of the tile area to be scanned is 128×4, the corresponding tile area is shown. When the tile area is 128×4, the first row L0, the second row L1, the third row L2, and the fourth row L3 can be read sequentially up to the tile width. At this time, because the height of the tile area is 4, the first row L0, the second row L1, the third row L2, and the fourth row L3 can be read sequentially. Therefore, the indexes can be output in the order of indexes 0, 13, 3, 9, 5, 7, 15, and 27 corresponding to the first tile area and indexes 16, 19, 11, 22, 18, 30, 32, and 33 corresponding to the second tile area.

[0140] In this way, packed data items can be stored using indexes assigned by row and channel, and the scan conversion controller can control the data reading method to suit the data method to be processed by the image processing circuit. The scan conversion controller can output the indexes assigned and stored by channel and row, and can send the indexes to the reading controller so that the data can be read by a raster scanning method or a tile scanning method. According to an embodiment of the present invention, after assigning the index corresponding to the memory, the conversion to T2R or R2T can be easily performed according to the form in which the index corresponding to the memory is output.

[0141] Fig.13 is a diagram illustrating a method of outputting a plurality of multi-context data items when a plurality of multi-context data items having different forms are input according to an embodiment of the inventive concept.

[0142] refer to Fig.13 , the first multi-context data ctx0, the second multi-context data ctx1, and the third multi-context data ctx2 are randomly input. At this time, it is assumed that the first multi-context data ctx0 and the second multi-context data ctx1 are input as row data, and can be output only after being input to the fourth row L3. The third multi-context data ctx2 may be in a bypass form, that is, in a data form that can be output immediately without being input until any row. According to the example, the first multi-context data ctx0 and the second multi-context data ctx1 may be input by a tile scanning method, and the third multi-context data ctx2 may be input by a raster scanning method. When multi-context data items are randomly input, the write / read points of the multi-context data items are different, so that bypass storage is also required instead of immediate output. According to an embodiment of the present invention, "multi-context" or "multi-context data" may represent multiple image data items input from different imaging devices.

[0143] refer to Fig.13 , the first line data ctx0 L0 of the first multi-context data ctx0 is input, and the third line data ctx1 L2 of the second multi-context data ctx1 is subsequently input. Then, the third multi-context data ctx2 is input in a bypass form ctx2 Bypass. At this time, the first multi-context data ctx0 and the second multi-context data ctx1 may not be output because all the line data items have not been input. However, since the third multi-context data ctx2 is input in a bypass form, the third multi-context data ctx2 may be output in real time.

[0144] Then, the second row data ctx0 L1 of the first multi-context data ctx0 and the third row data ctx0 L2 of the first multi-context data ctx0 are sequentially input, and the fourth row data ctx1 L3 of the second multi-context data ctx1 is input. Therefore, all row data items of the second multi-context data ctx1 can be input and immediately output as the ctx1 tile. The data in the bypass form of the third multi-context data ctx2 input next can be output immediately after the output of the ctx1 tile of all row data items of the second multi-context data ctx1. In addition, by the input of the fourth row data ctx0L3 of the first multi-context data ctx0, all row data items of the first multi-context data ctx0 can be input and immediately output as ctx0 L3.

[0145] refer to Fig.13 Although multi-context data items in different forms, that is, multi-context data items input in different scanning methods, are input in a random order, data in a bypass form can be output immediately, and multi-context data in which all row data are input can also be output immediately.

[0146] refer to Fig.13 , data in bypass form can be input, and different sizes and different rows can be input according to multi-context data. At this time, because the output can be performed sequentially, the role of the throttling buffer can be performed, and the inactive buffer included in the DMA can also be reduced.

[0147] Fig.14A and Fig. 14B is a diagram illustrating an example of storing multiple context data items in a channel index FIFO and reading the multiple context data items in an example method according to an embodiment of the inventive concept.

[0148] refer to Fig.14A , provides multiple context data items with three different sizes. Fig.14A , the first multi-context data CTX0, the second multi-context data CTX1, and the third multi-context data CTX2 are input. The first multi-context data CTX0, the second multi-context data CTX1, and the third multi-context data CTX2 may have different tile areas. The first multi-context data CTX0 may have a tile area size of 64×4, the second multi-context data CTX1 may have a tile area size of 128×4, and the third multi-context data CTX2 may have a tile area size of 32×4. Indexes may be assigned to the first multi-context data CTX0, the second multi-context data CTX1, and the third multi-context data CTX2 according to channels and rows. Fig.14A In the example, 32 bits can be assigned to an index. The index is assigned to Fig.14A The tile areas included in each of the first multi-context data CTX0, the second multi-context data CTX1, and the third multi-context data CTX2 shown in the figure can be marked with tile area sizes. According to the example, the second multi-context data CTX1 can have two 128×4 tile areas, the left tile area of ​​the two tile areas can be marked as B1, B2, B3 and B4 row by row, and the right tile area of ​​the two tile areas can be marked as B5, B6, B7 and B8 row by row.

[0149] The assigned indexes can be stored in the channel index FIFO memory per channel and row. Fig. 14B , index information corresponding to the first multi-context data CTX0 may be stored in the first channel index FIFO memory 1320_1 according to the rows FIFO_CH0_T0, FIFO_CH0_T1, FIFO_CH0_T2, and FIFO_CH0_T3. Index information corresponding to the second multi-context data CTX1 may be stored in the second channel index FIFO memory 1320_2 according to the rows FIFO_CH1_T0, FIFO_CH1_T1, FIFO_CH1_T2, and FIFO_CH1_T3. Index information corresponding to the third multi-context data CTX2 may be stored in the third channel index FIFO memory 1320_3 according to the rows FIFO_CH2_T0, FIFO_CH2_T1, FIFO_CH2_T2, and FIFO_CH2_T3.

[0150] Through the information stored in the first channel index FIFO memory, the second channel index FIFO memory, and the third channel index FIFO memory, the index output may be determined differently according to each scanning mode.

[0151] According to an example, when data is to be output in a raster scan mode through information stored in the first channel index FIFO memory, data can be read in rows up to the entire width. The output result can be (0, 1, 8, 9, 16, 17, 24, 25, 32, and 33), (2, 3, 10, 11, 18, 19, 26, 27, 34, and 35), .... According to an example, the size of the data read according to the index can be optimized and selected according to the image size, the tile area size, the number of channels, and the bit.

[0152] According to an example, when data is to be output in a tile scan mode through information stored in the first channel index FIFO memory, data is read in the first row up to the tile width, and then data can be read in the next row. When the tile area size is 64×4, the output results can be (0, 1, 2, 3, 4, 5, 6, and 7) and (8, 9, 10, 11, 12, 13, 14, and 15).

[0153] Figure 15 to Figure 16 is a flowchart illustrating a method of operating an image processing circuit according to an embodiment of the inventive concept. Fig.16 Operations S1541, S1542, and S1543 may be Fig.15 An example of operation S1540, and operations S1551 and S1552 may be Fig.15 An example of operation S1550.

[0154] refer to Figure 15 to Figure 16 In operation S1510, the image processing circuit may pack the row data received through each of the plurality of channels into a predetermined data size to generate packed data (i.e., packed data). At this time, the predetermined data size may be determined according to the number of channels connected to the scan conversion controller and the maximum number of bits received during one clock signal cycle in different row data items received from the plurality of channels. In addition, the predetermined data size may be determined according to the pixel per cycle (PPC) of each of the plurality of channels.

[0155] In an embodiment, the image processing circuit may receive a first line of data from the first image sensor through a first channel among the plurality of channels, and may receive a second line of data from the second image sensor through a second channel among the plurality of channels. In other embodiments, the image processing circuit may receive a first line of data from the image sensor through a first channel among the plurality of channels, and may receive a second line of data from the memory access controller through a second channel among the plurality of channels.

[0156] In operation S1520, the image processing circuit may allocate an index corresponding to each of the plurality of channels, and may write the packed data to a memory included in the image processing circuit based on the address included in the index. Each address may correspond to the packed data, and the data size of the index may be a multiple (e.g., 8 times) of the data size of the packed data. Thus, one index may correspond to a plurality of addresses.

[0157] In operation S1530, the image processing circuit may manage an index assigned to each of the plurality of channels. For example, the image processing circuit may include: a first memory (eg, Figure 5B A first channel index FIFO memory 1320_1) for managing a second index assigned to a second channel among the plurality of channels (eg, Figure 5B The second channel index FIFO memory 1320_2).

[0158] In operation S1540, the image processing circuit may determine a scanning method for the packed data corresponding to the channel for which the row data input has been completed based on the index assigned to the channel for which the row data input has been completed. According to one example, it may be determined whether the packed data corresponding to the channel for which the row data input has been completed is scanned by a raster scanning method or a tile scanning method. According to an example, the scanning method may be determined according to the operation of the image processing circuit or may be predetermined.

[0159] In operation S1541, the image processing circuit may determine a method of scanning the packet data according to the first operation mode or the second operation mode.

[0160] In operation S1550, the image processing circuit may output index information corresponding to the determined scanning method. When the scanning operation mode is determined to be the first operation mode in operation S1542, in operation S1551, the index information may be output to read data in the raster scanning mode. In operation S1551, according to the first operation mode, the index information may be output so that the index assigned to the channel that has completed the input is sequentially read row by row starting from the first row with the entire width of the channel index FIFO memory storing the index. When the scanning operation mode is determined to be the second operation mode in operation S1543, in operation S1552, the index information may be output to read data in the tile scanning mode. In operation S1552, according to the second operation mode, the index information may be output by the following operation: starting from the first row, the index assigned to the channel that has completed the input is sequentially read row by row with the width of the set tile area of ​​the channel index FIFO memory storing the index, until there are as many rows as the number of heights corresponding to the set tile area.

[0161] In operation S1560, the image processing circuit may read the packed data based on the output index information, and unpack the packed data to generate output data. For example, when the input of the row data of the first channel among the plurality of channels is completed, the packed data corresponding to the first channel may be read from the memory based on the index assigned to the first channel. The image processing circuit may generate output data by unpacking the read data to have the same data form as the row data.

[0162] The image processing circuit of an embodiment conceived according to the present invention can be applied to various technical fields, such as mobile, XR, automotive and / or CIS. The image processing circuit of an embodiment conceived according to the present invention can simultaneously process multiple images received from multiple sensors, and can maximize memory efficiency and security area efficiency by appropriately converting the scanning method of multiple images. The image processing circuit of an embodiment conceived according to the present invention can check the free space of the SRAM memory device, and can convert the scanning method through the FIFO memory index to easily classify the stored data.

[0163] While embodiments of the inventive concept have been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. An image processing circuit, the image processing circuit comprising: a scan conversion controller configured to receive images from a plurality of channels connected to a plurality of imaging devices and generate output data; as well as An image signal processor is configured to receive and process the output data from the scan conversion controller, wherein the scan conversion controller is configured to determine a scanning method for generating the output data according to a first operation mode or a second operation mode.

2. The image processing circuit according to claim 1, wherein: The scan conversion controller is configured to generate the output data using a raster scanning method in the first operation mode, and is configured to generate the output data using a tile scanning method in the second operation mode.

3. The image processing circuit according to claim 2, wherein: The scan conversion controller comprises: a memory configured to: store packed data items obtained by packing images received from each of the plurality of channels according to an index assigned to each of the plurality of channels; and A plurality of channel index memories are configured to store index information of the memories storing the packed data items according to the plurality of channels.

4. The image processing circuit according to claim 3, wherein: The scan conversion controller outputs index information for a read operation based on information stored in a channel index memory corresponding to a channel of a packed data item whose input has been completed earlier among the packed data items.

5. The image processing circuit according to claim 4, wherein: In the first operation mode, the scan conversion controller outputs index information to sequentially read indexes stored in each of the plurality of channel index memories row by row starting from a first row with a total width of the channel index memories.

6. The image processing circuit according to claim 4, wherein: In the second operation mode, the scan conversion controller outputs index information by the following operation: starting from the first row, sequentially reading the indexes stored in each of the multiple channel index memories row by row with the width of the set tile area until the number of rows is the same as the number of rows corresponding to the height of the set tile area.

7. The image processing circuit according to claim 2, wherein: The scan conversion controller selects the first operation mode or the second operation mode to scan an image received from each of the plurality of channels.

8. The image processing circuit according to claim 2, wherein: The scan conversion controller is configured to first output an image of a channel whose input is completed first among images received from the plurality of channels.

9. The image processing circuit according to claim 3, wherein: The memory includes static random access memory, and Each of the plurality of channel index memories operates in a first-in-first-out method.

10. A scan conversion controller, the scan conversion controller comprising: a first data packing circuit configured to receive a first line of data from a first channel and pack the first line of data into a predetermined data size to generate first packed data; a second data packing circuit configured to receive the second line data from the second channel and pack the second line data into a predetermined data size to generate second packed data; an arbitration circuit, the arbitration circuit being configured to receive the first packaged data and the second packaged data, and sequentially output the first packaged data and the second packaged data; a memory configured to store the first packaged data and the second packaged data; a row index controller configured to: manage a first index of a first area in the memory storing the first packaged data and a second index of a second area in the memory storing the second packaged data, and determine scanning methods of the first packaged data and the second packaged data respectively assigned to the first index and the second index; a write controller configured to control a write operation of the memory; a read controller configured to control a read operation of the memory; as well as An unpacking circuit is configured to generate output data by unpacking the read data output from the read controller.

11. The scan conversion controller according to claim 10, wherein: The row index controller comprises: a first channel index memory, in which index information of the first area of ​​the memory is stored; a second channel index memory, in which index information of the second area of ​​the memory is stored; and An index controller is configured to obtain index information from the first channel index memory and the second channel index memory, determine a scanning method for the obtained index information, and output the obtained index information for a read operation.

12. The scan conversion controller according to claim 11, wherein: The index controller is further configured to output the obtained index information for reading a channel index memory corresponding to a channel in which the packed data input is completed first among the first channel index memory and the second channel index memory.

13. The scan conversion controller according to claim 12, wherein: The index controller is configured to: when the scanning method of the obtained index information is a raster scanning method, output the index information so that the index stored in the channel index memory corresponding to the channel in which the packed data input is completed first is read sequentially row by row starting from the first row along the entire width of the channel index memory.

14. The scan conversion controller according to claim 12, wherein: The index controller is configured to: when the scanning method of the obtained index information is a tile scanning method, output the index information by the following operation: starting from the first row, sequentially reading the indexes stored in the channel index memory row by row with the width of the set tile area of ​​the channel index memory corresponding to the channel in which the packed data input is completed first, until there are as many rows as the number of rows corresponding to the height of the set tile area.

15. The scan conversion controller according to claim 12, wherein: The index controller is configured to: output the obtained index information immediately when there is no previously completed packaged data input without converting the scanning method of the obtained index information, and output the obtained index information after outputting the packaged data when there is previously completed packaged data input.

16. The scan conversion controller according to claim 11, wherein: The memory is a static random access memory, and Wherein, the first channel index memory and the second channel index memory are operated in a first-in-first-out method.

17. A method of operating an image processing circuit including a scan conversion controller, the scan conversion controller including an internal memory, the method comprising: packing the row data received through each of the plurality of channels into a predetermined data size to generate packed data; assigning an index corresponding to each of the plurality of channels, and writing the packed data to the internal memory based on an address included in the assigned index; managing an index assigned to each of the plurality of channels; Determine, based on an index assigned to a channel among the plurality of channels that has completed input of row data, a scanning method for the packed data corresponding to the channel that has completed input of row data; Outputting index information corresponding to the determined scanning method; Reading the packaged data from the internal memory based on the index information; as well as The read data read from the internal memory is unpacked and output data is generated.

18. The method according to claim 17, wherein: Determining the scanning method for the packaged data includes: determining a scanning method for generating the packaged data according to a first operation mode or a second operation mode.

19. The method according to claim 18, wherein: Outputting the index information corresponding to the determined scanning method includes: according to the first operating mode, outputting the index information to sequentially read the indexes assigned to the channels that have completed the input of row data, and the sequential reading is performed row by row starting from the first row with the total width of the channel index memory that stores the indexes.

20. The method according to claim 18, wherein: Outputting the index information corresponding to the determined scanning method includes outputting the index information according to the second operating mode through the following operation: sequentially reading the indexes assigned to the channels that have completed the input of row data, and the sequential reading is performed row by row starting from the first row with the width of the set tile area of ​​the channel index memory storing the index until the number of rows is the same as the height of the tile area corresponding to the set.

Images