Row interleaving controller, image signal processor including the same, and application processor

Abstract

A line interleaving controller, an image signal processor including the same, and an application processor are provided. The image signal processor includes a line interleaving controller and an image signal processor core. The line interleaving controller receives a plurality of image data lines included in an image frame, generates one or more dummy data lines corresponding to the image frame, and outputs the plurality of image data lines and the dummy data lines in order line by line. The image signal processor core includes at least one pipeline circuit. The pipeline circuit includes a plurality of processing modules connected in series in succession to sequentially process the data lines received from the line interleaving controller. The line interleaving controller processes one or more end image data lines included in an end portion of the image frame based on the dummy data lines. Interference or collision between channels is reduced or prevented by processing the end image data lines in synchronization with the dummy data lines.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2020-0067473, filed on June 4, 2020, with the Korean Intellectual Property Office (KIPO), the full disclosure of which is incorporated herein by reference. Technical Field

[0003] The example embodiments relate generally to semiconductor integrated circuits, and more specifically, to a line interleaving controller and an image signal processor including the line interleaving controller. Background Technology

[0004] An image signal processor (ESP) can utilize a frame memory to process multiple input images from multiple channels using a time-division multiplexing (TDM) scheme. When an ESP uses an external frame memory for TDM processing, the ESP stores the input images frame by frame in the external frame memory. In some embodiments, the input images are stored in the external frame memory and subsequently read from it via a bus system, and the power consumption of the system including the ESP increases due to bus transactions. Including the frame memory in the ESP also increases the size and / or cost of the ESP. Summary of the Invention

[0005] Some example embodiments may provide a line interleaving controller capable of efficiently supporting time division multiplexing (TDM) on a line-by-line basis (i.e., line-by-line), and an image signal processor including the line interleaving controller.

[0006] According to an example embodiment, an image signal processor includes a line interleaving controller and an image signal processor core. The line interleaving controller receives a plurality of image data lines included in an image frame, generates one or more virtual data lines corresponding to the image frame, and outputs the plurality of image data lines and virtual data lines sequentially, line by line. The image signal processor core includes at least one pipeline circuit. The pipeline circuit includes a plurality of processing modules connected in series to sequentially process the data lines received from the line interleaving controller. The line interleaving controller processes one or more end image data lines included in the ends of the image frame based on the virtual data lines.

[0007] According to an example embodiment, a line interleaving controller includes a buffer circuit, a virtual line generator, a multiplexer, and control logic circuitry. The buffer circuit buffers multiple first image data lines included in a first image frame received via a first channel and multiple second image data lines included in a second image frame received via a second channel, selectively outputting either the individual first image data lines or the individual second image data lines. The virtual line generator generates first virtual data lines corresponding to the first image frame and second virtual data lines corresponding to the second image frame. The multiplexer outputs a streaming signal by selecting the output of the buffer circuit and the output of the virtual line generator. The control logic circuitry controls the buffer circuit, the virtual line generator, and the multiplexer. The first virtual data lines are used to process one or more end image data lines included in the ends of the first image frame, and the second virtual data lines are used to process one or more end image data lines included in the ends of the second image frame.

[0008] According to an example embodiment, an application processor includes a bus, an image signal processor connected to the bus, and a central processing unit (CPU) connected to the bus to control the image signal processor. The image signal processor includes a line interleaving controller configured to receive a plurality of image data lines included in an image frame, generate one or more virtual data lines corresponding to the image frame, and output the plurality of image data lines and virtual data lines line by line in sequence; and an image signal processor core including at least one pipeline circuit including a plurality of processing modules connected in series to sequentially process the data lines received from the line interleaving controller, the line interleaving controller being configured to process one or more end image data lines included in the ends of the image frame based on the virtual data lines.

[0009] The line interleaving controller according to the example embodiment can reduce or prevent interference or collisions between channels and more effectively support TDM on a line-by-line basis by generating virtual data lines for processing end-image data lines included in the end of an image frame and providing the virtual data lines to an image signal processor core including a pipeline structure.

[0010] An image signal processor including a line interleaving controller according to an example embodiment can reduce or minimize access to external memory and reduce or prevent interference between channels by performing pipeline processing with efficient TDM on a line-by-line basis, thereby reducing power consumption and / or improving the performance of the image signal processor and systems including the image signal processor, such as application processors. Attached Figure Description

[0011] Exemplary embodiments of this disclosure will become clearer from the following detailed description taken in conjunction with the accompanying drawings.

[0012] Figure 1 This is a block diagram illustrating an image signal processor according to an example embodiment.

[0013] Figure 2A and Figure 2B This is a diagram used to describe the line-by-line processing of the delay of the image signal processor according to an example embodiment.

[0014] Figure 3 and Figure 4 This is a diagram illustrating an example embodiment of a data stream processed by an image signal processor according to an example embodiment.

[0015] Figure 5 and Figure 6 This is a diagram illustrating an example embodiment of a data stream processed by an image signal processor according to an example embodiment.

[0016] Figure 7 This is a block diagram illustrating an image signal processor according to an example embodiment.

[0017] Figure 8 It is shown Figure 7 Timing diagram of example operation of the image signal processor.

[0018] Figure 9A This is a block diagram illustrating an example of a delay processing module, and Figure 9B It is shown Figure 9A The timing diagram of the operation of the delay processing module.

[0019] Figure 10A This is a block diagram illustrating an example of a delay processing module included in an image signal processor according to an example embodiment, and Figure 10B It is shown Figure 10A The timing diagram of the operation of the delay processing module.

[0020] Figure 11 and Figure 12 This is a block diagram illustrating an image signal processor according to an example embodiment.

[0021] Figure 13 This is a block diagram illustrating a data processing system according to an example embodiment.

[0022] Figure 14 This is a block diagram illustrating a row interleaving controller according to an example embodiment.

[0023] Figure 15 It is shown Figure 14 The timing diagram of the operation of the row interleaving controller.

[0024] Figures 16 to 19 This is a diagram illustrating an example embodiment of a buffer circuit included in a row interleaving controller according to an example embodiment.

[0025] Figure 20This is a block diagram illustrating an image signal processor according to an example embodiment.

[0026] Figure 21 This is a block diagram illustrating a computing system according to an example embodiment. Detailed Implementation

[0027] Hereinafter, various exemplary embodiments will be described more fully with reference to the accompanying drawings, which illustrate some exemplary embodiments. In the drawings, the same reference numerals always refer to the same elements. Repeated descriptions may be omitted.

[0028] Figure 1 This is a block diagram illustrating an image signal processor according to an example embodiment.

[0029] Reference Figure 1 The image signal processor 10 may include a line interleaving controller (LIC) 100 and an image signal processor core (ISPCR) 200.

[0030] The line interleaving controller 100 can receive multiple image data lines included in an image frame, generate one or more virtual data lines corresponding to the image frame, and output the multiple image data lines and virtual data lines line by line in sequence. The line interleaving controller 100 may include a virtual line generator 300 to generate virtual data lines. (See also...) Figure 14 An example embodiment describing the detailed configuration of the line interleaving controller 100.

[0031] The line interleaving controller 100 can output the plurality of image data lines and virtual data lines as a streaming signal STRIN. In other words, as referenced... Figure 3 and Figure 4 As described, the stream signal STRIN output from the line interleaving controller 100 can sequentially include each of the plurality of image data lines and each of the virtual data lines. The stream signal STRIN is provided as an input to the image signal processor core 200, and the stream signal STRIN can be referred to as the input stream signal STRIN.

[0032] In some example embodiments, the line interleaving controller 100 can receive multiple image frames FRM1, FRM2, and FRM3 through multiple channels CH1, CH2, and CH3 respectively. The resolution, frame rate, etc. of the multiple image frames FRM1, FRM2, and FRM3 can be determined independently, and they can be the same or different from each other. Figure 1 A non-limiting example of three image frames received from three channels is shown, and the line interleaving controller 100 can receive two or four or more image frames through a corresponding number of channels.

[0033] The virtual line generator 300 can generate virtual data lines corresponding to multiple image frames FRM1, FRM2, and FRM3, respectively, and the line interleaving controller 100 can sequentially transmit the multiple image data lines and virtual data lines included in the multiple image frames FRM1, FRM2, and FRM3 to the image signal processor core 200 line by line using a time division multiplexing (TDM) scheme. In other words, the input stream signal STRIN output from the line interleaving controller 100 can sequentially include each of the multiple image data lines received through multiple channels CH1, CH2, and CH3, and each of the virtual data lines generated by the virtual line generator 300.

[0034] For example, if referencing Figure 5 and Figure 6 As described, the line interleaving controller 100 can receive a plurality of first image data lines included in a first image frame FRM1 via a first channel CH1 and a plurality of second image data lines included in a second image frame FRM2 via a second channel CH2. In some embodiments, the virtual line generator 300 in the line interleaving controller 100 can generate one or more first virtual data lines corresponding to the first image frame FRM1 and one or more second virtual data lines corresponding to the second image frame FRM2. The line interleaving controller 100 can use a TDM scheme to sequentially pass the input stream signal STRIN to the image signal processor core 200 line by line, such that the input stream signal STRIN includes the plurality of first image data lines, the plurality of second image data lines, the first virtual data lines, and the second virtual data lines. According to an example embodiment, the virtual line generator 300 may not generate virtual data lines corresponding to a specific image frame.

[0035] The image signal processor core 200 may include at least one pipeline circuit PLC. The pipeline circuit PLC may include multiple processing modules M1 to M3 connected in series to sequentially process data lines received from the line interleaving controller 100.

[0036] The pipeline circuit PLC contains multiple processing modules M1 to M3 with different processing functions to process each data row sequentially. The processing results of previous processing modules can be passed to the next processing module in sequence. The last processing module can output the final result of the pipeline processing. Multiple processing modules M1 to M3 can process each data row simultaneously, thus the overall processing time of the multiple data rows can be reduced by utilizing the structure of the pipeline circuit PLC.

[0037] Additionally, the image signal processor core 200 can process one or more end image data lines included in the end of each image frame based on virtual data lines. The image signal processor core 200 can generate an output stream signal STROUT that includes the processed image data lines in sequence.

[0038] Thus, the line interleaving controller 100 according to the example embodiment can reduce or prevent interference or collisions between channels, and efficiently support TDM on a line-by-line basis by generating virtual data lines for processing end-image data lines included in the end of an image frame and providing the virtual data lines to the image signal processor core including a pipeline structure. The image signal processor core 200 can reduce or minimize access to external memory and / or reduce or prevent interference between channels by performing pipeline processing on a line-by-line basis using an efficient TDM scheme, thereby reducing the power consumption and / or improving the performance of the image signal processor 10 and the system including the image signal processor 10, such as an application processor.

[0039] Figure 2A and Figure 2B This is a diagram used to describe the line-by-line processing of the delay of the image signal processor according to an example embodiment.

[0040] Figure 2A Only nine pixel data points P1 to P9 used in the 3x3 box filtering of the many pixel data included in a single image frame are shown.

[0041] The example processing module in the image signal processor core can use the neighboring pixel data P1 to P4 and P6 to P9 to process the central pixel data P5, so as to output the processed pixel data P5'.

[0042] When image frames are provided line by line, in order to process the center pixel data P5 in the i-th data line DLi, in addition to the pixel data P1, P2, and P3 received before the current i-th data line DLi-1, the example processing module also needs the pixel data P7, P8, and P9 received after the i-th data line DLi+1. In other words, after receiving a delayed data line DLi+1 received after the target data line DLi, the example processing module can perform a delayed filtering operation on the target data line DLi.

[0043] Thus, in the case of 3*3 box filtering, the delay size DSZ, which indicates the number of data rows DLi+1 required to process the target data row DLi, can be one.

[0044] Figure 2B Only twenty-five pixel data points P1 to P25 used in the 5x5 box filtering of the many pixel data included in a single image frame are shown.

[0045] An example processing module in the image signal processor core can use neighboring pixel data P1~P12 and P14~P25 to process the center pixel data P13, so as to output the processed pixel data P13'.

[0046] When image frames are provided line by line, in order to process the center pixel data P13 in the i-th data line DLi, in addition to the pixel data P1~P5 in the (i-2)-th data line DLi-2 and the pixel data p6~P10 in the (i-1)-th data line DLi-1 received before the current i-th data line DLi, the example processing module also needs the pixel data P16~P20 in the (i+1)-th data line DLi+1 and the pixel data P21~P25 in the (i+2)-th data line DLi+2 received sequentially after the i-th data line DLi. In other words, after receiving the two delayed data lines DLi+1 and DLi+2 received after the target data line DLi, the example processing module can perform a delayed filtering operation on the target data line DLi.

[0047] Thus, in the case of 5x5 box filtering, the delay size DSZ, which indicates the number of data rows DLi+1 and DLi+2 requiring delay to process the target data row DLi, can be two.

[0048] Reference Figure 2A and Figure 2B The 3x3 box filtering and 5x5 box filtering are described as examples, and the example embodiments are not limited thereto. The example embodiments can be applied to vertical filtering with an n*1 window and box filtering with an n*m window, where each of n and m is an integer greater than one. In the following, the example processing module performing the delayed filtering operation can be referred to as the delayed processing module. The delay size DSZ of the delayed processing module can be referred to as the module delay size, and the overall delay size of the image signal processor core can be referred to as the core delay size.

[0049] Figure 3 and Figure 4 This is a diagram illustrating an example embodiment of a data stream processed by an image signal processor according to an example embodiment.

[0050] Reference Figure 1 and Figure 3 The first image frame FRM1 can be provided to the image signal processor 10 via the first channel CH1. The first image frame FRM1 may include multiple first image data lines IL11 to IL1n, and the multiple first image data lines IL11 to IL1n can be provided to the line interleaving controller 100 in line unit (i.e., line by line).

[0051] The virtual line generator 300 in the line interleaving controller 100 can generate and output first virtual data lines VL1n+1 to VL1n+k, such that the number k of the first virtual data lines VL1n+1 to VL1n+k corresponds to the core delay size CDSZ. (Refer to...) Figure 7 and Figure 8As described, the core delay size CDSZ can indicate the sum of the module delay sizes of the processing modules included in the pipeline path from the input to the output of the image signal processor core 200.

[0052] The line interleaving controller 100 can generate an input stream signal STRIN by sequentially outputting multiple first image data lines IL11 to IL1n and first virtual data lines VL1n+1 to VL1n+k in line units.

[0053] The image signal processor core 200 can perform processing in real time based on multiple first image data lines IL11 to IL1n and first virtual data lines VL1n+1 to VL1n+k in the input stream signal STRIN, and generate the output stream signal STROUT by processing and outputting the first processed image data lines PIL11 to PIL1n.

[0054] Figure 4 An example is shown where a first image frame FRM1 includes five first image data lines IL11 to IL15, and one or more end image data lines included at the ends of the first image frame FRM1 are processed using three virtual data lines VL16 to VL18. However, this example embodiment is not limited to this. The number of first image data lines included in the first image frame FRM1 can be determined differently depending on the resolution of the image received by the image signal processor 10, and the number of first virtual data lines can be determined differently depending on the configuration of the image signal processor core 200.

[0055] exist Figure 4 In this context, tB represents the buffer time of the line interleaving controller 100, tLL represents the line time interval between two adjacent data lines output sequentially from the line interleaving controller 100, tCD represents the line delay time of the image signal processor core 200, and tCP represents the core processing time of the image signal processor core 200. The line delay time tCD can be represented by the product of the core delay size CDSZ and the line time interval tLL, CDSZ*tLL.

[0056] like Figure 4 As shown, the kernel latency size CDSZ can be three, and the number of the first virtual data rows VL16 to VL18 can also be three. In other words, the number of the first virtual data rows VL16 to VL18 can be equal to the kernel latency size CDSZ.

[0057] The first processed image data lines PIL11 to PIL15 correspond to the first image data lines IL11 to IL15 of the first image frame FRM1. Virtual data lines VL16 to VL18 are used and consumed when processing the end image data lines; therefore, the data lines corresponding to virtual data lines VL16 to VL18 are not included in the output stream signal STROUT.

[0058] Figure 5 and Figure 6 This is a diagram illustrating an example embodiment of a data stream processed by an image signal processor according to an example embodiment.

[0059] Reference Figure 1 and Figure 5 The first image frame FRM1 and the second image frame FRM2 can be provided to the image signal processor 10 via the first channel CH1 and the second channel CH2, respectively. The first image frame FRM1 may include multiple first image data lines IL11 to IL1n, which can be provided to the line interleaving controller 100 line by line (i.e., line by line). The second image frame FRM2 may include multiple second image data lines IL21 to IL2m, which can be provided to the line interleaving controller 100 line by line. Depending on the resolution of the first image frame FRM1 and the second image frame FRM2, the number n of the first image data lines IL11 to IL1n and the number m of the second image data lines IL21 to IL2m may be equal to or different from each other.

[0060] The virtual line generator 300 in the line interleaving controller 100 can generate and output first virtual data lines VL1n+1 to VL1n+k and second virtual data lines VL2m+1 to VL2m+k, such that the quantity k of each of the first virtual data lines VL1n+1 to VL1n+k and the second virtual data lines VL2m+1 to VL2m+k corresponds to the core delay size CDSZ. (Refer to...) Figure 7 and Figure 8 The core delay size CDSZ, as described, can represent the module delay size of the processing module included in the pipeline path from the input to the output of the image signal processor core 200.

[0061] The line interleaving controller 100 can generate the input stream signal STRIN by sequentially outputting multiple first image data lines IL11~IL1n, multiple second image data lines IL21~IL2m, first virtual data lines VL1n+1~VL1n+k, and second virtual data lines VL2m+1~VL2m+k in line units.

[0062] The image signal processor core 200 can perform processing in real time based on multiple first image data lines IL11~IL1n, multiple second image data lines IL21~IL2m, first virtual data lines VL1n+1~VL1n+k, and second virtual data lines VL2m+1~VL2m+k in the input stream signal STRIN, and generate the output stream signal STROUT by processing and outputting the first processed image data lines PIL11~PIL1n and the second processed image data lines PIL21~PIL2m.

[0063] Figure 6 An example is shown where a first image frame FRM1 includes three first image data lines IL11-IL13, and a second image frame FRM2 includes five second image data lines IL21-IL25. One or more end image data lines included at the ends of the first image frame FRM1 and the second image frame FRM2 are processed using two first virtual data lines VL14 and VL15 and two second virtual data lines VL26 and VL27. However, this example embodiment is not limited to this. The number of first image data lines included in the first image frame FRM1 and the number of second image data lines included in the second image frame FRM2 can be determined differently depending on the resolution of the image received by the image signal processor 10, and the number of first virtual data lines and the number of second virtual data lines can be determined differently depending on the configuration of the image signal processor core 200.

[0064] exist Figure 6 In this context, tB represents the buffer time of the line interleaving controller 100, tLL represents the line time interval between two adjacent data lines of the first image frame output sequentially from the line interleaving controller 100, tCD represents the line delay time of the image signal processor core 200, and tCP represents the core processing time of the image signal processor core 200.

[0065] The line delay time tCD can be represented by the multiplication of the kernel delay size CDSZ and the line time interval tLL, CDSZ*tLL, but the example embodiment is not limited thereto. The line time interval tLL of the first image data line included in the first image frame FRM1 and the line time interval of the second image data line included in the second image frame FRM2 can be determined differently. For example, the input stream signal STRIN may include two or more sequential image data lines included in the same image frame, so the line delay time can be variable.

[0066] TI1 and TI2 represent the time points when the input of the first image frame FRM1 and the second image frame FRM2 to the line interleaving controller 100 is completed. T01 and T02 represent the time points when the output of the first processed image data lines PIL11 to PIL13 and the second image data lines PIL21 to PIL25 from the image signal processor core 200 is completed.

[0067] like Figure 6 As shown, the kernel latency size CDSZ can be two, and the number of each of the first virtual data rows VL14 and VL15 and the second virtual data rows VL26 and VL27 can be two. In other words, the number of each of the first virtual data rows VL14 and VL15 and the second virtual data rows VL26 and VL27 can be equal to the kernel latency size CDSZ.

[0068] The first processed image data lines PIL11 to PIL13 correspond to the first image data lines IL11 to IL13 of the first image frame FRM1, and the second processed image data lines PIL21 to PIL15 correspond to the second image data lines IL21 to IL25 of the second image frame FRM2. The first virtual data lines VL14 and VL15 and the second virtual data lines VL26 and VL27 are used and consumed in the processed image data lines; therefore, the data lines corresponding to the first virtual data lines VL14 and VL15 and the second virtual data lines VL26 and VL27 are not included in the output stream signal STROUT.

[0069] Figure 7 This is a block diagram illustrating an image signal processor according to an example embodiment.

[0070] Reference Figure 7 The image signal processor 11 may include a line interleaving controller (LIC) 100 and an image signal processor core (ISPCR) 201.

[0071] The line interleaving controller 100 can receive multiple first image data lines included in a first image frame FRM1 via a first channel CH1, and multiple second image data lines included in a second image frame FRM2 via a second channel CH2. The line interleaving controller 100 may include a virtual line generator (VLG) 300, which is used to generate first virtual data lines corresponding to the first image frame FRM1 and second virtual data lines corresponding to the second image frame FRM2.

[0072] The line interleaving controller 100 can use a TDM scheme to sequentially transmit the plurality of first image data lines, the plurality of second image data lines, the first virtual data line, and the second virtual data line to the image signal processor core 201, line by line. The line interleaving controller 100 can also transmit the plurality of first image data lines, the plurality of second image data lines, the first virtual data line, and the second virtual data line as an input stream signal STRIN to the image signal processor core 201.

[0073] The image signal processor core 201 may include a pipelined circuit PLC. The pipelined circuit PLC may include multiple processing modules MA to MC connected in series to sequentially process data lines received from the line interleaving controller 100.

[0074] For example, such as Figure 7 As shown, the pipelined circuit PLC may include a first processing module MA, a second processing module MB, and a third processing module MC. The first processing module MA processes the data rows in the input stream signal STRIN to generate a first stream signal STRA, the second processing module MB processes the data rows in the first stream signal STRA to generate a second stream signal STRB, and the third processing module MC processes the data rows in the second stream signal STRB to generate an output stream signal STROUT. Thus, the processing modules MA, MB, and MC in the pipelined circuit PLC can perform pipelined processing of the data rows included in the input stream signal STRIN.

[0075] The processing modules MA, MB, and MC can be executed as shown in the reference. Figure 2A and Figure 2B The description describes the delay of the filtering operation in the processing modules. For example, the first module delay size MDSZA of the first processing module MA can be one, the second module delay size MDSZB of the second processing module MB can be two, and the third module delay size MDSZC of the third processing module MC can be one. As a result, the core delay size CDSZ of the pipeline circuit PLC can be four, which corresponds to the sum of the module delay sizes MDSZA, MDSZB, and MDSZC of the processing modules MA, MB, and MC included in the pipeline path from the input to the output of the image signal processor core 201.

[0076] Figure 7 The illustrated pipelined PLC is a non-limiting example; for example, the image signal processor core included in the image signal processor according to the example embodiment may have, as will be referred to... Figure 11 and Figure 12 The various configurations described.

[0077] Figure 8 It is shown Figure 7 Timing diagram of example operation of the image signal processor.

[0078] Figure 8 An example is shown where a first image frame FRM1 includes three first image data lines IL11 to IL13, and a second image frame FRM2 includes eight second image data lines IL21 to IL28. The number of first image data lines in the first image frame FRM1 and the number of second image data lines in the second image frame FRM2 can be determined differently based on the resolution of the image received by the image signal processor 11. For ease of illustration and description, Figure 8 The second virtual data line corresponding to the second image frame FRM2 is omitted, and the processing example embodiment of the first image frame FRM1 is described with reference to the description.

[0079] exist Figure 8 In this context, tB represents the initial buffer time of the line interleaving controller 100, tLL represents the line time interval between two adjacent data lines of the first image frame output sequentially from the line interleaving controller 100, tMDA, tMDB and tMDC represent the respective line delay times of the processing modules MA, MB and MC, and tMPA, tMPB and tMPC represent the respective processing times of the processing modules MA, MB and MC.

[0080] The first line delay time tMDA of the first processing module MA can be represented by the product of the first module delay size MDSZA = 1 and the line time interval tLL, which is 1*tLL. The second line delay time tMDB of the second processing module MB can be represented by the product of the second module delay size MDSZB = 2 and the line time interval tLL, which is 2*tLL. Furthermore, the third line delay time tMDC of the third processing module MC can be represented by the product of the third module delay size MDSZC = 1 and the line time interval tLL, which is 1*tLL. However, the example embodiment is not limited to this. The line time interval of the first image data line included in the first image frame FRM1 and the line time interval of the second image data line included in the second image frame FRM2 can be determined differently. For example, the input stream signal STRIN can include two or more sequential image data lines included in the same image frame, so the line delay time can be variable.

[0081] TI1 represents the time point when the input of the first image frame FRM1 to the line interleaving controller 100 is completed. TO1 represents the time point when the first processed image data lines PIL11 to PIL13 are output from the image signal processor core 201.

[0082] Reference Figure 7 and Figure 8The virtual line generator 300 can generate first virtual data lines VL14 to VL17 corresponding to the first image frame FRM1 and second virtual data lines (not shown) corresponding to the second image frame FRM2. The line interleaving controller 100 can generate the input stream signal STRIN by outputting the first image data lines IL1 to IL3, the second image data lines IL21 to IL28, the first virtual data lines VL14 to VL17, and the second virtual data line in line units using a TDM scheme.

[0083] The first processing module MA can perform a delayed filtering operation on the data lines in the input stream signal STRIN by a delay of MDSZA = 1, to generate a first stream signal STRA including the image data lines AL11-AL13 and AL21-AL27 processed by the first processing module MA. In some example embodiments, the first processing module MA can perform a delayed filtering operation on a first end image data line IL13 using a first virtual data line VL14, and output three first virtual data lines VL15-VL17 that are not used in the delayed filtering operation of the first processing module MA without processing.

[0084] The second processing module MB can perform a delayed filtering operation on the data lines in the first stream signal STRA by a delay of MDSZB = 2, to generate a second stream signal STRB comprising the image data lines BL11-BL13 and BL21-BL25 processed by the second processing module MB. In some example embodiments, the second processing module MB can perform a delayed filtering operation on the two end image data lines AL12 and AL13 using two first virtual data lines VL15 and VL16, and output a first virtual data line VL17 that is not used in the delayed filtering operation of the second processing module MB, without processing it.

[0085] The third processing module MC can perform a delayed filtering operation on the data lines in the second stream signal STRB, with a delay of MDSZC = 1, to generate an output stream signal STROUT comprising the image data lines PL11-PL13 and PL21-PL24 processed by the third processing module MC. In some example embodiments, the third processing module MC can perform a delayed filtering operation on an end image data line BL13 using a first virtual data line VL17.

[0086] As a result, the entire line delay time tCD of the image signal processor core 201 can be represented by the product of the core delay size CDSZ = 4 and the line time interval tLL: 4 * tLL. The core processing time tCP can be represented by the sum of the processing times of the first module tMPA, the second module tMPB, and the third module tMPC. As mentioned above, the line time interval can be variable.

[0087] The number of virtual data lines can be determined differently depending on the configuration of the image signal processor core. The line interleaving controller 100 can determine the number of virtual data lines based on the module delay size of the delayed processing module, as described above, where each module delay size represents the number of delayed data lines for each delayed processing module. The time interval between virtual data lines can be determined based on the input rate of the image data lines, or it can be determined as a predetermined (or alternatively, desired) time interval.

[0088] In some example embodiments, when the image signal processor core includes a single pipeline circuit, the line interleaving controller 100 can generate and output virtual data lines such that the number of virtual data lines is equal to the sum of the module delays of the processing modules included in the single pipeline circuit. For example, as referenced Figure 7 and Figure 8 As described, the line interleaving controller 100 includes a pipelined circuit PLC, and the core delay size CDSZ, which is the sum of the module delay sizes MDSZA, MDSZB, and MDSZC of the processing modules MA, MB, and MC included in the pipelined circuit PLC, can be four. In some embodiments, the line interleaving controller 100 can generate and output four virtual data lines VL14 to VL17 corresponding to the first image frame FRM1 and four second virtual data lines (not shown) corresponding to the second image frame FRM2.

[0089] For reference Figure 8 As described, each delay processing module can perform a delay filtering operation on the end image data lines using a first number of received virtual data lines, and output a second number of received virtual data lines without processing. Here, the first number corresponds to the module delay size of each delay processing module, and the second number corresponds to the received virtual data lines that are not used in the delay filtering operation of each delay processing module.

[0090] Figure 9A This is a block diagram illustrating an example of a delay processing module. Figure 9B It is shown Figure 9A The timing diagram shows the operation of the delay processing module. As a non-limiting example, Figure 9A and Figure 9B The module delay size MDSZ is shown to be one.

[0091] Reference Figure 9A The delay processing module 50 may include a line buffer control circuit (LBC) 51 and a filter circuit FLT.

[0092] The row buffer control circuit 51 can buffer the data rows in the first stream signal STR1 provided in the previous stage (another delayed processing module or the input channel itself) to output the second stream signal STR2. The filter circuit FLT can perform a filtering operation based on the data rows in the second stream signal STR2 to generate a third stream signal STR3 that includes the processed or filtered data rows.

[0093] The line buffer control circuit 51 may include two line buffers LNM1 and LNM2 for storing two previously received image data lines, and a line drain circuit LDC for processing one end image data line of an image frame.

[0094] The row buffer control circuit 51 can output the (i-2)th image data row ILi-2 and the (i-1)th image data row ILi-1 stored in the two row buffers LNM1 and LNM2 in sync with the i-th image data row ILi in the first stream signal STR1, and simultaneously output the i-th image data row ILi.

[0095] The filtering circuit FLT can be performed based on the simultaneously received (i-2), (i-1), and i-th image data rows, as shown in the reference. Figure 2A The description describes a 3x3 box filter, and outputs the (i-1)th filtered image data row FIL1-1.

[0096] For ease of illustration and description, Figure 9B An example of an image frame comprising five image data lines IL1 to IL5 is shown. tLL represents the line time interval between two adjacent data lines, tMD represents the line delay time of the delayed processing module 50, and tMP represents the processing time of the delayed processing module 50. The line delay time tMD can be represented by the product of the module delay size MDSZ-1 = 1 and the line time interval tLL, which is 1 * tLL.

[0097] like Figure 9B As shown, the delayed processing module 50 can perform a delayed filtering operation on the (i-1)th image data line ILi-1 synchronously with the i-th image data line ILi received via the first stream signal STR1. In some embodiments, the delayed filtering operation on the end image data line IL5 is performed by the data line IL6 provided from the line consumption circuit LDC, since no data line for processing the end image data line IL5 was received from the previous stage.

[0098] While performing a delayed filtering operation on the end image data line IL5, the delayed processing module 50 may not receive data lines from the previous stage. Therefore, the line buffer control circuit 51 must activate the stall signal STL provided to the previous stage to indicate that the delayed processing module cannot receive data lines.

[0099] When the delayed processing module 50 itself controls the processing timing of the end image data lines, several problems arise for TDM schemes relative to multiple inputs from multiple channels. While the delayed processing module processes the end image data lines, the pause signal STL needs to be activated to reduce or prevent the next data line from being passed from the previous stage. In some embodiments, the buffer of the previous stage may be undesirably filled because image data lines are continuously passed through the channel. Additionally, end image data lines may not be processed in a timely manner due to rushing image data lines from a particular channel. The problem becomes more severe if multiple delayed processing modules exist in the pipeline path. Thus, end image data lines may not be processed correctly because the state of other processing modules and the buffers storing data from the channels may be unavailable when the line buffer control circuit 51 determines the timing for processing the end image data lines.

[0100] Figure 10A This is a block diagram illustrating an example of a delay processing module included in an image signal processor according to an example embodiment. Figure 10B It is shown Figure 10A The timing diagram of the operation of the delay processing module.

[0101] Reference Figure 10A The delay processing module 60 may include a line buffer control circuit LBC 61 and a filter circuit FLT.

[0102] and Figure 9A The row buffer control circuit 51 in the middle compares, Figure 10A The line buffer control circuit 61 in the diagram does not include the line consumption circuit LDC. Apart from filtering operations to account for delays in the end image data lines, Figure 10A and Figure 10B Configuration and operation of delay processing module 60 Figure 9A and Figure 9B The delay processing modules 50 are the same or substantially the same, therefore, repeated descriptions are omitted.

[0103] like Figure 10BAs shown, the delay processing module 60 can synchronously perform a delay filtering operation on the end image data line IL5 in real time with the virtual data line VL6 received from the line interleaving controller 100. In some embodiments, the delay filtering operation for the end image data line IL5 does not require the delay processing module to stop receiving data lines from the previous line interleaving controller 100 or the previous processing module. Therefore, during the delay filtering operation for the end image data line IL5, Figure 10A The row buffer control circuit 61 in the middle does not need to be like Figure 9A The pause signal STL is activated in the same way as the line buffer control circuit 61 in the image signal processor core. In other words, when the delay processing module in the image signal processor core performs a delay filtering operation on the end image data line IL5, the image signal processor core including the delay processing module 60 can receive data lines from the line interleaving controller 100.

[0104] Thus, the line interleaving controller 100 according to the example embodiment can generate additional virtual data lines to reduce or prevent interference or collisions between channels and effectively support TDM on a line-by-line basis. The generation and order of data lines can be determined by the line interleaving controller 100, which precedes the pipeline circuitry in the image signal processor core, and operational uncertainties in the pipeline circuitry can be eliminated. Delayed processing modules in the pipeline circuitry can operate passively and synchronously with the image data lines and virtual data lines provided by the line interleaving controller 100, thereby improving pipeline processing performance.

[0105] Figure 11 and Figure 12 This is a block diagram illustrating an image signal processor according to an example embodiment.

[0106] Reference Figure 11 The image signal processor 12 may include a line interleaving controller LIC 100 and an image signal processor core 202. The line interleaving controller 100 may include a virtual line generator as described above.

[0107] The image signal processor core 202 may include processing modules MA, MB1, MB2, MC1, MC2, and MC3. Two processing modules MB1 and MB2 may form a first pipeline circuit PLC1 to generate a first output stream signal STROUT1. The three processing modules MC1, MC2, and MC3 may be connected in parallel with the first pipeline circuit PLC1 to form a second pipeline circuit PLC2 to generate a second output stream signal STROUT2.

[0108] like Figure 11As shown, the first pipeline circuit PLC1 may include a first delay processing module MB1 that performs the delay filtering operation as described above, and the second pipeline circuit PLC2 may include two second delay processing modules MC2 and MC3. For example, the module delay size MDSZB1 of the first delay processing module MB1 may be one, the module delay size MDSZC2 of one second delay processing module MC2 may be three, and the module delay size MDSZC2 of another second delay processing module MC3 may be one. In some embodiments, the first pipeline delay size PDSZ1 of the first pipeline circuit PLC1 may be one, which corresponds to the first module delay size MDSZB1 of the first delay processing module MB1 included in the first pipeline circuit PLC1; and the second pipeline delay size PDSZ2 of the second pipeline circuit PLC2 may be four, which corresponds to the sum of the two second module delay sizes MDSZC2 and MDSZC3 of the second delay processing modules MC2 and MC3 included in the second pipeline circuit PLC2.

[0109] The line interleaving controller 100 can generate and output virtual data lines such that the number of virtual data lines is equal to the larger of the first pipeline delay size PDSZ1 = 1 and the second pipeline delay size PDSZ1 = 4. That is, the line interleaving controller 100 can generate and output four virtual data lines. The line interleaving controller 100 can receive image frames FRM with a resolution of w*n on a line-by-line basis, where n represents the number of lines in the image frame FRM and w represents the number of pixels per line in the image frame FRM. In other words, the image frame FRM can include n image data lines. The line interleaving controller 100 can generate an input stream signal STRIN on a line-by-line basis such that the input stream signal STRIN includes n image data lines and four virtual data lines. In other words, the input stream signal STRIN can include an extended image frame with a resolution of w*(n+4).

[0110] The image signal processor 202 can passively perform delayed processing synchronously with n rows of image data and four virtual rows of data.

[0111] The first processing module MB1 in the first pipeline circuit PLC1 can perform a delayed filtering operation on one end image data line of the image frame FRM using one of the four virtual data lines to generate a first output stream signal STROUT1 that includes the image frame with a resolution of w*n. The first processing module MB1 can ignore three unused virtual data lines out of the four virtual data lines.

[0112] The second processing module MC2 in the second pipeline circuit PLC2 can perform delayed filtering on the three end image data lines of the image frame FRM using three of the four virtual data lines. The third processing module MC3 can perform delayed filtering on one end image data line using one of the four virtual data lines to generate the second output stream signal STROUT2, which includes the image frame with a resolution of w*n. The number of end image data lines can correspond to the maximum value of the module delay sizes MDSZC2 and MDSZC3 of the second delayed processing modules MC2 and MC3.

[0113] Figure 12 Image signal processor 13 and Figure 11 The image signal processor 12 is similar, and repeated descriptions can be omitted.

[0114] and Figure 11 Compared to the image signal processor 12, Figure 12 The image signal processor 13 may also include a mixer MXR and a processing module MD, wherein the mixer MXR is configured to mix the output of the first pipeline circuit PLC1 and the output of the second pipeline circuit PLC2, and the processing module MD is configured to process the output of the mixer MXR.

[0115] For the mixing operation of the mixer MXR, it may be necessary to synchronize the outputs of the first pipeline circuit PLC1 and the second pipeline circuit PLC2 line by line. For this synchronization, the first pipeline circuit PLC1 may also include a delay buffer DBF on its pipeline path. The delay buffer DBF can delay the received data lines without processing them, allowing the first pipeline delay size PDSZ1 and the second pipeline delay size PDSZ2 to become equal. Figure 12 In the example, the delay size BDSZ of the delay buffer DBF can be set to three, and the first pipeline delay size PDSZ1 can be set to four, which is equal to the second pipeline delay size PDSZ2.

[0116] although Figure 11 and Figure 12 Not shown, but image signal processors 202 and 203 may further include at least one processing module configured to perform scaling and / or cropping operations to change the resolution of the image frame. In some embodiments, the resolution-changing module may output a virtual data line without processing, as shown in reference [reference needed]. Figure 7 and Figure 8 Described.

[0117] Figure 13 This is a block diagram illustrating a data processing system according to an example embodiment.

[0118] Reference Figure 13The data processing system 500 may include an application processor (AP) 600, multiple imaging devices IMD1 and IMD2, external memory (MEM) 700, and / or a display device (DIS) 800. For example, the data processing system 500 may be implemented as a personal computer (PC) or a mobile computing device. For example, a mobile computing device may be a laptop computer, cellular phone, smartphone, tablet PC, personal digital assistant (PDA), enterprise digital assistant (EDA), digital camera, digital camcorder, portable multimedia player (PMP), personal navigation device or portable navigation device (PND), handheld game console, mobile internet device (MID), wearable computer, Internet of Things (IoT) device, Internet of Everything (IoE) device, or e-book.

[0119] AP 600 can be implemented as an integrated circuit (IC), motherboard, system-on-a-chip (SoC), mobile AP, etc. AP 600 may include a bus architecture (or bus) 640, a central processing unit (CPU) 610, multiple interfaces INT1 and INT2, an image signal processor (ISP) 10, a memory controller (MC) 620, and / or a display controller (DC) 630. CPU 610, ISP 10, memory controller 620, and / or display controller 630 can exchange commands and / or data with each other via bus architecture 640. For example, bus architecture 640 can be implemented as a bus utilizing the Advanced Microcontroller Bus Architecture (AMBA) protocol, a bus utilizing the Advanced High-Performance Bus (AHB) protocol, a bus utilizing the Advanced Peripheral Bus (APB) protocol, a bus utilizing the AMBA Scalable Interconnect (AXI) protocol, or a combination thereof.

[0120] The CPU 610 can control the overall operation of the AP 600. For example, the CPU 610 can control interfaces INT1 and INT2, ISP 10, memory controller 620, and display controller 630. The CPU 610 may include at least one core.

[0121] For example, the first interface INT1 can receive a first image and a first control signal from the first imaging device IMD1 and send them to the ISP 10. For example, the second interface INT2 can receive a second image and a second control signal from the second imaging device IMD2 and send them to the ISP 10. For example, the first image and the second image can be referred to as a picture, image data, image data stream, or image frame.

[0122] For example, the first imaging device IMD1 and the second imaging device IMD2 can be implemented as complementary metal-oxide-semiconductor (CMOS) image sensor chips or camera modules. The first imaging device IMD1 and the second imaging device IMD2 can use, for example, a Mobile Industrial Processor Interface (MIPI) Camera Serial Interface (CSI) to send the first image and the second image, as well as the first control signal and the second control signal, to the first interface INT1 and the second interface INT2, respectively. For example, the resolution of the first image can be different from the resolution of the second image.

[0123] ISP 10 can perform time-division multiplexing (TDM) on the first image and / or the second image. TDM can be performed not on an image frame basis. Alternatively, TDM can be performed on a line basis without using external memory 700.

[0124] For example, ISP 10 can execute image frames output from imaging devices IMD1 and IMD2 on a line-by-line basis using a TDM scheme. According to an example embodiment, ISP 10 may include a line interleaving controller 100 and an image signal processor core as described above, such that the line interleaving controller 100 generates virtual data lines, and the image signal processor core uses the virtual data lines to process one or more end image data lines included in the ends of the frame image.

[0125] For example, the image signal processor core can perform at least one of automatic dark level compensation, bad pixel replacement, noise reduction, lens shading compensation, color correction, RGB gamma correction, edge enhancement, tone control, and color suppression.

[0126] The memory controller 620, under the control of the CPU 610, stores data processed by the ISP 10 in TDM mode in the external memory 700. The display controller 630, under the control of the CPU 610, transmits data (e.g., frame data) from the external memory 700 to the display device 800. For example, the display controller 630 may transmit data (e.g., frame data) from the external memory 700 to the display device 800 using, for example, a MIPI display serial interface (DSI) or an embedded display port (eDP).

[0127] Figure 14 This is a block diagram illustrating a line interleaving controller 100 according to an example embodiment.

[0128] Reference Figure 14 The line interleaving controller 100 may include a virtual line generator (VLG) 300, a control logic circuit (CLG) 110, a buffer circuit (BFC) 130 and / or a multiplexer (MUX) 150.

[0129] The buffer circuit 130 can buffer multiple first image data lines in the first image frame FRM1 provided from the first imaging device IMD1 through the first channel CH1 and the first interface INT1, and multiple second image data lines in the second image frame FRM2 provided from the second imaging device IMD2 through the second channel CH2 and the second interface INT2, to generate a buffered output signal BOUT including the first image data line IL1 and the second image data line IL2.

[0130] The virtual line generator 300 can generate a first virtual data line VL1 corresponding to the first image frame FRM1 and a second virtual data line VL2 corresponding to the second image frame FRM2, and selectively outputs either the first virtual data line or the second virtual data line each time. The multiplexer 150 can output the input stream signal STRIN by selecting the output of the buffer circuit 130 and the output of the virtual line generator 300.

[0131] The control logic circuit 110 can control the overall operation of the buffer circuit 130, the virtual line generator 300, and / or the multiplexer 150. The control logic circuit 110 can generate a first control signal CTRL1 for controlling the virtual line generator 300, a second control signal CTRL2 for controlling the buffer circuit 130, and a selection signal SEL for controlling the multiplexer 150 based on a first synchronization SYNC1 provided via a first channel CH1 and a second synchronization SYNC2 provided via a second channel CH2. Additionally, the control logic circuit 110 can generate a third synchronization signal SYNC1', a fourth synchronization signal SYNC2', and a virtual line identification signal VLID for operation configured adjacent to the image signal processor core of the line interleaving controller 100.

[0132] Synchronization signals may include vertical synchronization signals indicating the start and end order of transmitting each image frame, horizontal synchronization signals indicating the start and end order of transmitting each data line, clock signals, data enable signals, etc.

[0133] As described above, the first virtual data line VL1 can be used to process the end image data line of the first image frame FRM1, and the second virtual data line VL2 can be used to process the end image data line of the second image frame FRM2.

[0134] Figure 15 It is shown Figure 14 Timing diagram of the operation of the line interleaving controller 100.

[0135] First image frame FRM1, second image frame FRM2, and input stream signal STRIN are compared with reference. Figure 6 The descriptions are identical, and duplicate descriptions are omitted.

[0136] Reference Figure 14 and Figure 15 Simultaneously with activating the first vertical synchronization signal VSYNC1, the first horizontal synchronization signal HSYNC1 can be repeatedly activated, and the first image data lines IL11 to IL13 can be transmitted from the first imaging device IMD1 to the line interleaving controller 100 synchronously with the activation of the first horizontal synchronization signal HSYNC1. Similarly, simultaneously with activating the second vertical synchronization signal VSYNC2, the second horizontal synchronization signal HSYNC2 can be repeatedly activated, and the second image data lines IL21 to IL25 can be transmitted from the second imaging device IMD2 to the line interleaving controller 100 synchronously with the activation of the second horizontal synchronization signal HSYNC2.

[0137] The control logic circuit 110 in the line interleaving controller 100 can generate a third vertical synchronization signal VSYNC1' and a third horizontal synchronization signal HSYNC1' that are synchronized with the first image data lines IL11 to IL13 and the first virtual data lines VL14 and VL15 included in the input stream signal STRIN, and generate a fourth vertical synchronization signal VSYNC2' and a fourth horizontal synchronization signal HSYNC2' that are synchronized with the second image data lines IL21 to IL25 and the second virtual data lines VL26 and VL27 included in the input stream signal STRIN, which are provided to the image signal processor core.

[0138] The third vertical synchronization signal VSYNC1', the third horizontal synchronization signal HSYNC1', the fourth vertical synchronization signal VSYNC2', and the fourth horizontal synchronization signal HSYNC2' can correspond to channel identification signals indicating whether the current data in the input stream signal STRIN corresponds to the first image frame FRM1 or the second image frame FRM2. The image signal processor core can independently perform processing of the first image frame FRM1 and the second image frame FRM2 based on the channel identification signals.

[0139] Additionally, the control logic circuit 110 in the line interleaving controller 100 can generate a virtual line identification signal (VLID) indicating whether the current data line in the input stream signal STRIN corresponds to an image data line or a virtual data line. The image signal processor core can perform processing of the end image data lines of the first image frame FRM1 and the second image frame FRM2 based on the virtual line identification signal VLID.

[0140] Figures 16 to 19 This is a diagram illustrating an example embodiment of the buffer circuitry included in a line interleaving controller 100 according to an example embodiment.

[0141] Reference Figure 16The buffer circuit 131 may include a first multiplexer MUX1, a line buffer LNM, and / or a second multiplexer MUX2. The first multiplexer MUX1 may selectively output either a first image data line of a first image frame FRM1 or a second image data line of a second image frame FRM2 based on a first selection signal SEL1. The line buffer LNM may store an image data line output from the first multiplexer MUX1 based on a buffer control signal LCTRL, and output the stored image data line. The second multiplexer MUX2 may selectively output the first image data line of the first image frame FRM1, the second image data line of the second image frame FRM2, and an image data line output from the line buffer LNM based on a second selection signal SEL2, to generate a buffered output signal BOUT. The first control signal CTRL1 generated by the control logic circuit 110 may include the first selection signal SEL1, the second selection signal SEL2, and the buffer control signal LCTRL.

[0142] When the transmission time interval of the first image data line overlaps with the transmission time interval of the second image data line, Figure 16 The buffer circuit 131 can temporarily store an image data line in the line buffer LNM to implement TDM of the first image frame FRM1 and the second image frame FRM2 in real time.

[0143] Reference Figure 17 The buffer circuit 132 may include a first row buffer LNM1, a second row buffer LNM2, and / or a multiplexer MUX. The first row buffer LNM1 may store a first image data line of a first image frame FRM1 based on a first buffer control signal LCTRL1, and output the stored first image data line. The second row buffer LNM2 may store a second image data line of a second image frame FRM2 based on a second buffer control signal LCTRL2, and output the stored second image data line. The multiplexer MUX may selectively output the first image data line of the first image frame FRM1, the second image data line of the second image frame FRM2, the first image data line output from the first row buffer LNM1, and the second image data line output from the second row buffer LNM2 each time based on a selection signal MSEL, to generate a buffered output signal BOUT. The first control signal CTRL1 generated by the control logic circuit 110 may include the selection signal MSEL, the first buffer control signal LCTRL1, and the second buffer control signal LCTRL2.

[0144] When the transmission time interval of the first image data line overlaps with the transmission time interval of the second image data line, Figure 17The buffer circuit 132 can temporarily store an image data line in the first row buffer LNM1 or the second row buffer LNM2 to implement TDM of the first image frame FRM1 and the second image frame FRM2 in real time.

[0145] Reference Figure 18 The buffer circuit 133 may include a common memory area CMREG, which is configured to store the first image data lines IL11 and IL12 and the second image data lines IL21, IL22 and IL23 according to the reception order of the first image data lines IL11 and IL12 and the second image data lines IL21, IL22 and IL23 received by the buffer circuit 133. The control logic circuit 110 may sequentially change the write pointer WPTR, which indicates the position of the currently received image data lines stored by the buffer circuit 133.

[0146] Control logic circuit 110 can control buffer circuit 133, causing buffer circuit 133 to generate a buffered output signal BOUT by outputting the first image data lines IL11 and IL12 and the second image data lines IL21, IL22 and IL23 stored in the common memory area CMREG according to the receiving order. Control logic circuit 110 can sequentially change the read pointer RPTR, which indicates the position of the image data lines output from buffer circuit 133.

[0147] Can be used Figure 18 The buffer circuit 133 executes the TDM of the first image frame FRM1 and the second image frame FRM2 in a row-by-row manner according to the first-in-first-out (FIFO) scheme.

[0148] Reference Figure 19 The buffer circuit 134 may include a first memory area MREG1 and a second memory area MREG2. The buffer circuit 134 can store the first image data lines IL11 and IL12 in the first memory area MREG1 in a sequential manner according to the first receiving order of the buffer circuit receiving the first image data lines IL11 and IL12, and store the second image data lines IL21, IL22, and IL23 in the second memory area MREG2 in a second receiving order according to the second receiving order of the buffer circuit 134 receiving the second image data lines IL21, IL22, and IL23. The control logic circuit 110 can sequentially change the first write pointer WPTR1, which indicates the location of the currently received first image data line through the buffer circuit 134, and sequentially change the second write pointer WPTR2, which indicates the location of the currently received second image data line through the buffer circuit 134.

[0149] The control logic circuit 110 can determine the priority order of the first image data lines IL11 and IL12 stored in the first memory area MREG1 and the second image data lines IL21, IL22 and IL23 stored in the second memory area MREG2, and control the buffer circuit 134 such that the buffer circuit 134 generates a buffered output signal BOUT by outputting the first image data lines IL11 and IL12 stored in the first memory area MREG1 and the second image data lines IL21, IL22 and IL23 stored in the second memory area MREG2 according to the priority order. The control logic circuit 110 can sequentially change the first read pointer RPTR1, which indicates the position of the first image data line output from the buffer circuit 134, and sequentially change the second read pointer RPTR2, which indicates the position of the second image data line output from the buffer circuit 134.

[0150] Can be used Figure 19 The buffer circuit 134 adaptively determines the processing order of the first image frame FRM1 and the second image frame FRM2 provided through different channels. The control logic circuit 110 can determine the priority order based on the frame rate, importance, etc. of the first image frame FRM1 and the second image frame FRM2.

[0151] Figure 20 This is a block diagram illustrating an image signal processor according to an example embodiment. Figure 20 The diagram also shows a first imaging device IMD1, a second imaging device IMD2, a first interface INT1 and a second interface INT2, a bus 640, and a memory device 700.

[0152] Reference Figure 20 The image signal processor 14 may include a line interleaving controller LIC, an image signal processor core ISPCR, a scaling and formatting circuit SFC, a multimedia processing circuit MPC, and / or direct memory access controllers DMAC1 to DMAC4. The line interleaving controller LIC may include a virtual line generator VLC.

[0153] In some example embodiments, the line interleaving controller LIC can receive a first image frame from the first imaging device IMD1 and the second imaging device IMD2 via a first channel CH1 and a second channel CH2, respectively. In some example embodiments, the line interleaving controller LIC can receive the first image frame from the memory device 700 via a third channel CH3 and a fourth channel CH4, respectively, under the control of the direct memory access controllers DMAC1 and DMAC2.

[0154] The scaling and formatting circuit SFC can perform scaling operations relative to data output from the image signal processor core ISPCR, changing the format of the scaled data and generating formatted data. The scaling operation may include zoom-in and / or zoom-out operations. The direct memory access controller DMAC3 can store the data processed by the scaling and formatting circuit SFC in the memory device 700 via bus 640.

[0155] The scaling and formatting circuit SFC can pass the processed data to the multimedia processing circuit MPC. The multimedia processing circuit MPC can also process the received data. The multimedia processing circuit MPC can read or retrieve data stored in the memory device 700, and use the direct memory access controller DMAC4 to store the processed data in the memory device 700.

[0156] Figure 21 This is a block diagram illustrating a computing system according to an example embodiment.

[0157] Reference Figure 21 The computing system 1000 may include a processor 1010, a memory device 1020, a storage device 1030, an input / output (I / O) device 1040, an image signal processor (ISP) 10, and / or multiple image sensors 900. Although Figure 21 The device is not shown in the diagram, but the computing system 1000 may also include a port configured to communicate with an external device.

[0158] Processor 1010 can perform various calculations and tasks. According to some embodiments, processor 1010 may be a microprocessor or a central processing unit (CPU). Memory device 1020 can store data used to operate computing system 1000. For example, memory device 1020 may be implemented using dynamic random access memory (DRAM), mobile DRAM, static random access memory (SRAM), phase random access memory (PRAM), ferroelectric random access memory (FRAM), resistive random access memory (RRAM), and / or magnetic random access memory (MRAM). Storage device 1030 may include solid-state drives (SSDs), hard disk drives (HDDs), compact disc read-only memory (CD-ROMs), etc. Input / output device 1040 may include input devices such as a keyboard, keypad, mouse, etc., and output devices such as a printer, display device, etc.

[0159] For example, (but not limited to) any of the elements disclosed above in the line interleaving controller, image signal processor, and application processor may include processing circuitry (separately or in combination) or implemented in processing circuitry, such as hardware including logic circuitry; hardware / software combinations such as processor-executing software; or combinations thereof. More specifically, the processing circuitry may include (but is not limited to) a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc.

[0160] The image signal processor 10 may include a line interleaving controller LIC for generating virtual data lines and an image signal processor core including the pipeline structure described above. Virtual data lines can be provided to the image signal processor core to process end-image data lines included in the ends of image frames synchronously with the virtual data lines. Therefore, interference between channels can be effectively reduced or prevented.

[0161] This invention can be applied to any electronic device and system that requires processing image signals. For example, it can be applied to systems such as mobile phones, smartphones, personal digital assistants (PDAs), portable multimedia players (PMPs), digital cameras, camcorders, personal computers (PCs), server computers, workstations, laptops, digital TVs, set-top boxes, portable game consoles, navigation systems, wearable devices, Internet of Things (IoT) devices, Internet of Things (IoE) devices, e-books, virtual reality (VR) devices, augmented reality (AR) devices, in-vehicle infotainment (IVI) systems, and drones.

[0162] The foregoing description is an illustration of exemplary embodiments and should not be construed as limiting them. Although some exemplary embodiments have been described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without substantially departing from the inventive concept.

Claims

1. An image signal processor, comprising: A line interleaving controller is configured to receive multiple image data lines included in an image frame, generate one or more virtual data lines corresponding to the image frame, and output the multiple image data lines and the virtual data lines line by line in sequence. as well as An image signal processor core includes at least one pipeline circuit comprising a plurality of processing modules connected in series to sequentially process data lines received from the line interleaving controller, the line interleaving controller being configured to process one or more end image data lines included in the end of the image frame based on the virtual data lines.

2. The image signal processor according to claim 1, wherein, The plurality of processing modules include: One or more delay processing modules are each configured to perform a delay filtering operation on the target data line based on one or more delayed data lines received after the target data line to be processed.

3. The image signal processor according to claim 2, wherein, The image signal processor core synchronously performs the delayed filtering operation on the end image data line in real time with the virtual data line received from the line interleaving controller, and the image signal processor receives data lines from the line interleaving controller at the same time that the image signal processor core performs the delayed filtering operation on the end image data line.

4. The image signal processor according to claim 2, wherein, The row interleaving controller determines the number of virtual data rows based on the module delay size of the processing module with the delay, where each module delay size indicates the number of delayed data rows for each processing module with the delay.

5. The image signal processor according to claim 4, wherein, The image signal processor core includes a single pipeline circuit, and The row interleaving controller generates and outputs the virtual data rows such that the number of virtual data rows is equal to the sum of the module delays of the processing modules included in the single pipeline circuit.

6. The image signal processor according to claim 4, wherein, Each delay processing module performs the delay filtering operation on the end image data line using a first number of received virtual data lines, and outputs a second number of received virtual data lines without processing. The first number corresponds to the module delay size of each delay processing module, and the second number corresponds to the received virtual data lines that are not used in the delay filtering operation of each delay processing module.

7. The image signal processor according to claim 4, wherein, The image signal processor core includes: A first pipeline circuit, comprising one or more processing modules with a first delay; and The second pipeline circuit includes one or more processing modules with a second delay, and the first pipeline circuit is connected in parallel with the second pipeline circuit.

8. The image signal processor according to claim 7, wherein, The row interleaving controller generates and outputs the virtual data rows such that the number of virtual data rows is equal to the larger of a first pipeline delay size and a second pipeline delay size, wherein the first pipeline delay size corresponds to the sum of the module delay sizes of the delay processing modules included in the first pipeline circuit, and the second pipeline delay size corresponds to the sum of the module delay sizes of the delay processing modules included in the second pipeline circuit.

9. The image signal processor according to claim 7, wherein, The image signal processor core also includes: A mixer configured to mix the outputs of the first pipeline circuit and the second pipeline circuit.

10. The image signal processor according to claim 9, wherein, One of the first pipeline circuit and the second pipeline circuit includes: A delay buffer is configured to delay received data lines without processing them, such that the delay size of the first pipeline and the delay size of the second pipeline become equal to each other.

11. The image signal processor according to claim 1, wherein, The line interleaving controller receives multiple first image data lines included in a first image frame through a first channel and multiple second image data lines included in a second image frame through a second channel, generates one or more first virtual data lines corresponding to the first image frame and one or more second virtual data lines corresponding to the second image frame, and transmits the multiple first image data lines, the multiple second image data lines, the first virtual data lines and the second virtual data lines to the image signal processor core line by line in sequence using a time-division multiplexing scheme.

12. The image signal processor according to claim 11, wherein, The line interleaving controller transmits the plurality of first image data lines, the plurality of second image data lines, the first virtual data line, and the second virtual data line as input stream signals to the image signal processor core.

13. The image signal processor according to claim 12, wherein, The line interleaving controller transmits the input stream signal to the image signal processor core while the image signal processor core processes the end image data lines of the first image frame or the second image frame.

14. The image signal processor according to claim 12, wherein, The line interleaving controller generates a channel identification signal indicating whether the current data in the input stream signal corresponds to the first image frame or the second image frame, and The image signal processor core independently performs processing of the first image frame and processing of the second image frame based on the channel identification signal.

15. The image signal processor according to claim 12, wherein, The line interleaving controller generates a virtual line identification signal indicating whether the current data line in the input stream signal corresponds to the image data line or the virtual data line, and The image signal processor core performs processing of the end image data rows of the first image frame and the second image frame based on the virtual line recognition signal.

16. The image signal processor according to claim 12, wherein, The row interleaving controller includes: A buffer circuit is configured to buffer the plurality of first image data lines and the plurality of second image data lines to selectively output each of the first image data lines or each of the second image data lines; A virtual row generator is configured to generate the first virtual data row and the second virtual data row, and to selectively output either the first virtual data row or the second virtual data row; A multiplexer configured to output the input stream signal by selecting the output of the buffer circuit and the output of the virtual line generator; and A control logic circuit is configured to control the operation of the buffer circuit, the virtual line generator, and the multiplexer.

17. The image signal processor according to claim 16, wherein, The buffer circuit includes: A common memory area is configured to store the first image data line and the second image data line according to the receiving order of the first image data line and the second image data line received by the buffer circuit, and The control logic circuit is configured to control the buffer circuit, such that the buffer circuit outputs the first image data line and the second image data line stored in the common memory area according to the receiving order.

18. The image signal processor according to claim 16, wherein, The buffer circuit includes: A first memory region is configured to store the first image data lines sequentially according to a first reception order of the first image data lines received by the buffer circuit; and A second memory area is configured to store the second image data lines sequentially according to the second reception order of the buffer circuit, and The control logic circuit is configured to determine the priority order of a first image data line stored in the first memory area and a second image data line stored in the second memory area, and to control the buffer circuit so that the buffer circuit outputs the first image data line stored in the first memory area and the second image data line stored in the second memory area according to the priority order.

19. A line interleaving controller, comprising: A buffer circuit is configured to buffer multiple first image data lines included in a first image frame received through a first channel and multiple second image data lines included in a second image frame received through a second channel, so as to selectively output each first image data line or each second image data line. A virtual line generator is configured to generate a first virtual data line corresponding to the first image frame and a second virtual data line corresponding to the second image frame; A multiplexer configured to output a streaming signal by selecting the output of the buffer circuit and the output of the virtual line generator; as well as A control logic circuit configured to control the buffer circuit, the virtual row generator, and the multiplexer. Wherein, the first virtual data line is used to process one or more end image data lines included in the end of the first image frame, and the second virtual data line is used to process one or more end image data lines included in the end of the second image frame.

20. An application processor, comprising: bus; An image signal processor is connected to the bus; as well as A central processing unit, connected to the bus, is configured to control the image signal processor. The image signal processor includes: A line interleaving controller is configured to receive multiple image data lines included in an image frame, generate one or more virtual data lines corresponding to the image frame, and output the multiple image data lines and the virtual data lines line by line in sequence. as well as An image signal processor core includes at least one pipeline circuit comprising a plurality of processing modules connected in series to sequentially process data lines received from the line interleaving controller, the line interleaving controller being configured to process one or more end image data lines included in the end of the image frame based on the virtual data lines.

Images