Image signal processor, operation method thereof, and image processing system

Abstract

An image signal processor and an operation method thereof and an image processing system are provided. The image processing system includes: a control processor configured to generate and output setting information corresponding to N (where N is an integer of 2 or greater) image frames; and an image signal processor configured to: perform image processing on the N image frames received from an image sensor based on the setting information, and generate an interrupt signal based on completion of the image processing performed on the N image frames and send the interrupt signal to the control processor.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0068267 filed on June 10, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0002] The inventive concept relates to image signal processing, and more particularly, to an image signal processor for performing image processing on image data received from an image sensor, an operating method of the image signal processor, and / or an image processing system including the image signal processor. Background Art

[0003] An image signal processor included in an imaging device such as a camera or a smart phone may perform image processing such as converting a data format of image data provided from an image sensor into a data format such as RGB or YUV, or removing noise and adjusting brightness from the image data. The image signal processor may process image data output from the image sensor in units of frames. Since the imaging device already supports an operation mode such as a slow motion mode or an ultra slow motion mode for providing an image at a high frame rate, the image sensor may generate and output image data at a high frame rate. Therefore, an image signal processor for normally performing image processing on image data having a high frame rate may be beneficial. Summary of the invention

[0004] The inventive concept provides an image signal processor capable of rapid readout to normally perform image processing on image data received at a high frame rate, an operating method of the image signal processor, and an image processing system including the image signal processor.

[0005] According to aspects of the inventive concept, an image processing system is provided, comprising: a control processor configured to generate and output setting information corresponding to N (where N is an integer equal to or greater than 2) image frames; and an image signal processor configured to perform image processing on the N image frames received from an image sensor based on the setting information, and based on the completion of the image processing performed on the N image frames, generate an interrupt signal and send the interrupt signal to the control processor.

[0006] According to other aspects of the inventive concept, an image signal processor is provided, comprising: an image processing engine configured to sequentially perform image processing on image frames received from an image sensor; a direct memory access (DMA) controller configured to store processed data generated by the image processing engine in a memory; and a fast readout circuit configured to: receive setting information including N setting values ​​corresponding to N (where N is an integer equal to or greater than 2) image frames from a control processor, and based on the N image frames being sequentially image processed, provide the setting values ​​of the image frames being image processed to the image processing engine or the DMA controller.

[0007] According to other aspects of the inventive concept, a method for operating an image signal processor is provided, the method comprising: receiving N (wherein N is an integer equal to or greater than 2) setting values ​​from a control processor; storing the N setting values ​​in a storage area; receiving an image frame from an image sensor; sequentially performing image processing on N image frames in the image frame based on the N setting values; and generating an end interrupt signal based on completion of the image processing performed on the N image frames.

[0008] According to other aspects of the inventive concept, an application processor is provided, comprising: a main processor, configured to: generate and output setting information including N (where N is an integer equal to or greater than 2) setting values; and an image signal processor, configured to: receive and store the setting information, sequentially perform image processing on N image frames received from an image sensor based on the setting information, and based on completion of the image processing performed on the N image frames, generate an end interrupt signal and send the end interrupt signal to the image signal processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the inventive concept will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a block diagram illustrating an image processing apparatus according to an example embodiment;

[0011] Figure 2 is a block diagram illustrating an image signal processor according to an example embodiment;

[0012] Figure 3A is a timing diagram illustrating transmission / reception signals and operations of an image signal processor according to an example embodiment, Figure 3B is a timing chart showing transmission / reception signals and operations of an image signal processor according to a comparative example;

[0013] Figure 4 It is shown Figure 1Block diagram of the fast readout circuit;

[0014] Figure 5 is a diagram exemplarily illustrating setting information received by an image signal processor according to an example embodiment;

[0015] Figure 6 is a flowchart illustrating a method of operating an image processing system according to an example embodiment;

[0016] Figure 7 is a flowchart illustrating a method of operating an image processing system according to an example embodiment;

[0017] Figure 8 is a timing diagram illustrating an operation method based on an operation mode of an image signal processor according to an example embodiment;

[0018] Fig. 9 is a block diagram illustrating an image signal processor according to an example embodiment;

[0019] Fig.10 is a block diagram illustrating an image signal processor according to an example embodiment;

[0020] Fig.11 is a block diagram illustrating an image signal processor according to an example embodiment;

[0021] Fig.12 is a block diagram illustrating an image processing system according to an example embodiment;

[0022] Fig.13 is a block diagram illustrating an image processing system according to an example embodiment; and

[0023] Fig.14 is a block diagram illustrating an image processing apparatus according to an example embodiment. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0025] Figure 1 is a block diagram illustrating an image processing apparatus 1000 according to an example embodiment.

[0026] The image processing device 1000 may be implemented as an electronic device that captures an image and displays the captured image or performs an operation based on the captured image. The image processing device 1000 may be implemented as, for example, a personal computer (PC), an Internet of Things (IoT) device, and / or a portable electronic device. Examples of portable electronic devices may include laptop computers, mobile phones, smart phones, tablet PCs, personal digital assistants (PDAs), enterprise digital assistants (EDAs), digital still cameras, digital video cameras, audio devices, portable multimedia players (PMPs), personal navigation devices (PNDs), MP3 players, handheld game consoles, e-books, wearable devices, and the like. In addition, the image processing device 1000 may be equipped in electronic devices such as drones or advanced driver assistance systems (ADAS) and / or in electronic devices provided as components in vehicles, furniture, manufacturing facilities, and various measuring machines.

[0027] Reference Figure 1 , the image processing device 1000 may include an image sensor 1100 and an image processing system 1200. The image processing device 1000 may also include other elements such as a display and a user interface. The image processing system 1200 may include an image signal processor 100, a control processor 200, and a memory 300. The image signal processor 100, the control processor 200, and the memory 300 may be implemented as a single semiconductor chip or multiple semiconductor chips. For example, the image signal processor 100 and the control processor 200 may be integrated into one semiconductor chip.

[0028] The image processing system 1200 may include a processing circuit (such as hardware including a logic circuit, a hardware / software combination (such as a processor that executes software), or a combination thereof). For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), etc.

[0029] As disclosed herein, the terms "memory", "storage medium", "computer-readable storage medium" or "non-transitory computer-readable storage medium" may refer to one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices and / or other tangible machine-readable media for storing information. The term "computer-readable medium" may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other media capable of storing, containing or carrying one or more instructions and / or data.

[0030] The image sensor 1100 may convert an optical signal input through the optical lens LS and corresponding to the object into an electrical signal, and may generate and output image data IDT based on the electrical signal. The image sensor 1100 may include, for example, a pixel array including a plurality of pixels arranged in two dimensions and a readout circuit, and the pixel array may convert the received optical signal into an electrical signal. The pixel array may be implemented, for example, with a photoelectric conversion device (such as a charge coupled device CCD or a complementary metal oxide semiconductor (CMOS)), and further, the pixel array may be implemented with various other photoelectric conversion devices. The readout circuit may generate raw data based on the electrical signal provided from the pixel array, and may output the generated raw data or raw data that has been pre-processed (such as removing bad pixels) as image data IDT. The image sensor 1100 may be implemented as a semiconductor chip or package including a pixel array and a readout circuit.

[0031] The image signal processor 100 may perform image processing on the image data IDT provided from the image sensor 1100. For example, the image signal processor 100 may perform image processing such as converting the data format of the image data IDT (e.g., converting image data having a Bayer pattern into a YUV or RGB format), removing noise, adjusting brightness, and adjusting definition to enhance image quality. The image signal processor 100 may configure the hardware of the image processing system 1200.

[0032] The image signal processor 100 may include an image signal processing core 110 (hereinafter referred to as an ISP core) and a fast readout circuit 120 (hereinafter referred to as an FRO circuit). The ISP core 110 may perform image processing on the image data IDT output from the image sensor 1100 in units of frames. The ISP core 110 may be referred to as an image processing engine. Processing data PDT generated by image processing (e.g., image processing frames (hereinafter referred to as converted image data) generated by image processing and / or result data (statistical data, histogram, etc.)) may be stored in the memory 300.

[0033] The FRO circuit 120 may store setting information IF_N including setting values ​​of a plurality of image frames (hereinafter referred to as N (where N is a positive integer equal to or greater than two) number of frames) provided from the control processor 200, and in the case where image processing is performed on a specific image frame, the FRO circuit 120 may provide a setting value (current setting value) of the corresponding image frame (current image frame). For example, the setting information (e.g., the setting value of the frame) may include a register value for adjusting the image quality of the image frame (e.g., a register value used by the ISP core 110 during image processing) and an address register value representing an area of ​​the memory 300 where the processed data PDT corresponding to each frame is to be stored. Such setting values ​​may be set (or changed) per frame.

[0034] The FRO circuit 120 may receive and store setting information IF_N about N frames from the control processor 200, and in a case where image processing is performed on each of the N frames, the FRO circuit 120 may provide a setting value of the corresponding frame to the ISP core 110 or another circuit (e.g., Figure 2 Therefore, image processing can be performed on N frames in units of frames.

[0035] When the image processing performed on the N frames is completed, the FRO circuit 120 may generate an interrupt signal INT indicating that the image processing performed on the N frames is completed, or may issue a request to the ISP core 110 to generate the interrupt signal INT.

[0036] For example, the FRO circuit 120 may store setting information IF_N about N frames received from the control processor 200 through a one-time reception process. In the case where image processing is performed on each of the N frames, the FRO circuit 120 may provide a setting value of the corresponding frame, and when the image processing performed on the N frames is completed, the FRO circuit 120 may generate an interrupt signal INT.

[0037] The control processor 200 may control the image signal processor 100 to perform image processing. The control processor 200 may include a processing circuit (such as hardware including a logic circuit, a hardware / software combination (such as a processor that executes software), or a combination thereof), for example, to configure software of the image processing system 1200. The control processor 200 may be a central processing unit (CPU), a microprocessor, an ARM processor, an X86 processor, a microprocessor without an interlocked pipeline stage (MIPS) processor, a graphics processing unit (GPU), a general-purpose GPU, and / or other processors configured to execute instructions stored in a memory. The control processor 200 may process or execute data and an instruction code (or program) including an execution algorithm of the image signal processor 100 to generate a control signal CONS for controlling the control processor 200. The control signal CONS may include setting information IF_N about N frames.

[0038] The control processor 200 may generate setting information IF_N about N frames in advance, and before image processing starts to be performed on the N frames, the control processor 200 may transmit the setting information IF_N to the image signal processor 100. The control processor 200 may generate setting information about the next N frames while the image signal processor 100 is performing image processing on the N frames, and before the image processing performed on the N frames is completed (i.e., before receiving an interrupt signal INT indicating that the image processing performed on the N frames is completed from the image signal processor 100), the control processor 200 may transmit the generated setting information about the next N frames to the image signal processor 100. The control processor 200 may generate and transmit the setting information about the next N frames while the image signal processor 100 is performing image processing on the N frames.

[0039] The memory 300 may store the processed data PDT received from the image signal processor 100 , and may provide the processed data PDT to the image signal processor 100 , the control processor 200 , or other elements of the image processing apparatus 1000 .

[0040] The memory 300 may be implemented as a volatile memory or a nonvolatile memory. Examples of volatile memories may include dynamic random access memory (DRAM), static random access memory (SRAM), etc., and examples of nonvolatile memories may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), flash memory, phase change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc.

[0041] Figure 2 is a block diagram illustrating an image signal processor 100 according to an example embodiment. Figure 2 Shown in detail Figure 1 The image signal processor 100 is provided.

[0042] Reference Figure 2 , the image signal processor 100 may include an ISP core 110, a FRO circuit 120, and a DMA controller 130. The ISP core 110 may include a controller 111 and a plurality of intellectual property (IP) blocks 112. Figure 2 , the plurality of IP blocks 112 are illustrated as including first to third IP blocks 11 to 13 (eg, IP_B1, IP_B2, and IP_B3), but are not limited thereto, and the plurality of IP blocks 112 may include two or more IP blocks.

[0043] The controller 111 may receive a control signal CONS from the control processor 200, and may control the overall operation of the image signal processor 100 based on the control signal CONS. The control signal CONS may include setting information IF_N regarding N frames, and the controller 111 may provide the setting information IF_N regarding the N frames to the FRO circuit 120. The FRO circuit 120 may store the received setting information IF_N, and when image processing is performed based on a setting value included in the setting information IF_N per frame, the FRO circuit 120 may provide the stored setting information IF_N to the plurality of IP blocks 112 or the DMA controller 130.

[0044] A plurality of IP blocks 112 (e.g., the first IP block 11 to the third IP block 13) set as image processing blocks may perform image processing, and the first IP block 11 to the third IP block 13 may perform different image processing. In some example embodiments, the first IP block 11 may convert the data format of the image data IDT, the second IP block 12 may adjust the brightness, and the third IP block 13 may adjust the contrast. The first IP block 11 to the third IP block 13 may sequentially perform image processing on the frame. The image processing completed frame and / or result data (e.g., converted image data) obtained by each of the first IP block 11 to the third IP block 13 based on the image processing may be stored in the memory 300. The plurality of IP blocks 112 may receive a setting value corresponding to the processed frame from the FRO circuit 120, and may perform image processing on the corresponding frame based on the setting value.

[0045] The DMA controller 130 may store the processed data PDT received from at least one of the first IP block 11 to the third IP block 13 in the memory 300. The processed data PDT may include result data based on image processing and / or converted image data. At this time, the DMA controller 130 may receive an address ADDR (or an address register value) from the FRO circuit 120, and may store the processed data PDT in a storage area of ​​the memory 300 corresponding to the address ADDR.

[0046] For example, when the first frame is image-processed (or when the image processing is completed), the FRO circuit 120 may generate a first address based on the address register value of the area where the converted image data of the first frame (e.g., the first converted image data) is to be stored in the stored setting information IF_N, and may provide the first address (or the first address register value) to the DMA controller 130, and the DMA controller 130 may store the first converted image data in the area corresponding to the first address. Subsequently, when the second frame is image-processed, the FRO circuit 120 may generate a second address based on the address register value of the area where the converted image data of the second frame (e.g., the second converted image data) is to be stored in the stored setting information IF_N, and may provide the second address (or the second address register value) to the DMA controller 130, and the DMA controller 130 may store the second converted image data in the area corresponding to the second address.

[0047] When the image processing performed on the N frames is completed, the FRO circuit 120 may generate an interrupt signal INT indicating that the image processing is completed. The FRO circuit 120 may directly transmit the interrupt signal INT to the control processor 200 or the controller 111. In some example embodiments, the FRO circuit 120 may issue a request to the controller 111 to generate an interrupt signal, and in response to the request, the controller 111 may generate and transmit the interrupt signal INT.

[0048] Figure 3A is a timing diagram illustrating transmission / reception signals and operations of the image signal processor 100 according to an exemplary embodiment, Figure 3B is a timing chart illustrating transmission / reception signals and operations of an image signal processor according to a comparative example. Figure 3A Show Figure 1 The operation of the image signal processor 100 is shown in FIG.

[0049] Reference Figure 3A The image signal processor 100 may receive image data in units of frames. For example, the image signal processor 100 may receive the first frame F1 to the n+1th frame Fn+1. The image signal processor 100 may perform image processing on the received frames.

[0050] Before performing image processing, the image signal processor 100 may receive a signal from the control processor ( Figure 1The image signal processor 100 may receive a control signal for controlling the processing of a frame. The image signal processor 100 may receive setting information IF_N1 including setting values ​​of N frames (e.g., the first frame to the Nth frame) F1 to Fn. When a frame (e.g., the first frame F1) is received after receiving the setting information IF_N1, the image signal processor 100 may start performing image processing, and may send a start interrupt signal INT_S1 indicating the start of image processing to the control processor 200.

[0051] The image signal processor 100 may perform image processing on the N frames F1 to Fn. As described above, the FRO circuit ( Figure 1 The image signal processor 100 may store the setting information IF_N1, and when each frame starts to be image processed, the FRO circuit 120 may provide the setting value of the corresponding frame. In addition, while the N frames F1 to Fn are image processed, the image signal processor 100 may receive the setting information IF_N2 about the next N frames (e.g., the n+1th frame to the 2nth frame) Fn+1 to F2n from the control processor 200.

[0052] When the N frames F1 to Fn are image-processed, the image signal processor 100 may transmit an end interrupt signal INT_E1 indicating completion of the image processing performed on the N frames to the control processor 200. Subsequently, when a frame is received again (for example, when the n+1th frame Fn+1 is received), the image signal processor 100 may start image processing and may transmit a start interrupt signal INT_S2 indicating the start of image processing to the control processor 200.

[0053] As described above, the image signal processor 100 according to some example embodiments may receive the setting information IF_N1 and IF_N2 in units of N frames, and may perform image processing on the N frames based on the received setting information, and when the image processing performed on the N frames is completed, the image signal processor 100 may send the end interrupt signals INT_E1 and INT_E2 indicating the completion of the image processing to the control processor 200. At this time, the FRO circuit 120 included in the image signal processor 100 may store the setting information IF_N1 and IF_N2, and may provide a corresponding setting value when each frame is image processed. The control processor 200 may generate and send the setting information IF_N2 about the next N frames (e.g., the n+1th frame Fn+1 to the 2nth frame F2n) during the first period T1 in which the N frames (e.g., the first frame F1 to the nth frame Fn) are image processed. The first period T1 may be ensured as a setting margin of the control processor 200 (i.e., a time margin for controlling the setting).

[0054] The following will refer to Figure 3BTransmission / reception signals and operations of an image signal processor according to a comparative example are described. The image signal processor according to the comparative example does not include a FRO circuit.

[0055] The image signal processor according to the comparative example may receive a corresponding one of the setting information IF1 to IFn+2 from the control processor per frame, and may perform image processing on the corresponding frame based on the received setting information, and then, when the image processing is completed, the image signal processor may send an interrupt signal to the control processor. For example, the image signal processor may receive the setting information IF1 including the setting value of the first frame F1 from the control processor, and may perform image processing on the received first frame F1 based on the setting information IF1. When the first frame F1 starts to be image-processed, the image signal processor may send a start interrupt signal INT_S1 to the control processor, and when the image processing is completed, the image signal processor may send an end interrupt signal INT_E1 to the control processor.

[0056] When the first frame F1 is image-processed (ie, after receiving the start interrupt signal INT_S1 until receiving the end interrupt signal INT_E1 ), the control processor may generate and transmit setting information IF2 about the next frame (eg, the second frame F2 ).

[0057] As described above, according to the operation of the image signal processor according to the comparative example, the control processor may generate and transmit setting information about the next frame during the second period T2 during which one frame is image processed. The second period T2 may be ensured as a setting margin of the control processor.

[0058] When image data is received at a high frame rate (e.g., a frame rate equal to or greater than a predetermined threshold, such as 120fps (frames per second), 240fps, 960fps, etc.), the image processing time of one frame can be shortened. In the case where the image signal processor according to the comparative example performs image processing, since the control processor must generate and transmit setting information about the next frame during the image processing time of one frame (e.g., the second period T2), the setting margin of the control processor may be insufficient, and due to this, the image signal processor may not be normally controlled and may operate abnormally.

[0059] However, as described above, in the case where the image signal processor 100 according to some example embodiments performs image processing, the control processor 200 may ensure N number of image processing periods (e.g., the first period T1) as a setting margin, and thus the setting margin may be sufficiently ensured. Therefore, abnormal control of the control processor 200 may be prevented, and the image signal processor 100 may normally perform image processing on image data having a high frame rate.

[0060] Figure 4 It is shown Figure 1 1 is a block diagram of the FRO circuit 120.

[0061] Reference Figure 4 , the FRO circuit 120 may include a storage area 21 storing setting information about N frames and a logic circuit 22.

[0062] The storage area 21 may store setting values ​​(e.g., first to n-th setting values ​​IF1 to IFn) of each of the N frames. For example, the storage area 21 may include N registers (e.g., first to n-th registers) REG1 to REGn, and the first to n-th registers REG1 to REGn may store the first to n-th setting values ​​IF1 to IFn, respectively.

[0063] In response to the command CMD for requesting the information FIF indicating the frame on which the image processing is performed or the setting value of the corresponding frame, the logic circuit 22 may read the setting value IF corresponding to the corresponding frame from the storage area 21, and may output the setting value IF of the corresponding frame or the address ADDR of the corresponding frame. For example, the logic circuit 22 may send the setting value IF of the corresponding frame to the ISP core ( Figure 1 110) provides a register value for adjusting the image quality of the corresponding frame, or can provide a register value for adjusting the image quality of the corresponding frame to the DMA controller ( Figure 2 130) provides an address register value of a corresponding frame or an address ADDR generated based on the address register value.

[0064] Figure 5 is a diagram exemplarily illustrating setting information received by an image signal processor according to an example embodiment.

[0065] As mentioned above, the image signal processor ( Figure 1 The control processor 100 may receive setting information IF_N about N frames from the control processor 200. The setting information IF_N about the N frames may include the size 211 of the received data, the address register values ​​212 of the N frames (e.g., ADD_F1, ADD_F2, ..., ADD_Fn), and frame information 213 about which frame the interruption occurs (e.g., F INT ). When the image processing performed on the corresponding frame is completed, the image signal processor 100 may generate an interrupt signal based on the frame information 213 about the position where the interruption occurs.

[0066] In some example embodiments, the setting information IF_N may be received in a packet data format, and may further include a header bit indicating the start of the packet data and a tail bit indicating the end of the packet data.

[0067] Figure 6 is a flowchart illustrating an operating method of an image processing system according to an example embodiment. Figure 6 Show Figure 1 An operating method of each of the image signal processor 100 and the control processor 200 is disclosed.

[0068] Reference Figure 6 , the control processor 200 may generate setting information IF_N1 about N frames in operation S110, and may transmit the setting information IF_N1 to the image signal processor 100 in operation S120. Figure 1 As described above, the control processor 200 may execute instruction codes and data for controlling the image signal processor 100 to generate the setting information IF_N1.

[0069] In detail, in operation S130, the FRO circuit 120 of the image signal processor 100 may store the received setting information IF_N1. Subsequently, in operation S140, an image frame may be received from the image sensor, and the image signal processor 100 may send a start interrupt signal to the control processor 200 in operation S150, and may start performing image processing.

[0070] In operation S160, the image signal processor 100 may perform image processing on N frames based on the setting information IF_N1. When the frames are image processed, the FRO circuit 120 may provide a setting value for each frame based on the setting information IF_N1, and the image signal processor 100 may perform image processing in units of frames based on the setting value.

[0071] At this time, the control processor 200 may generate setting information IF_N2 about the next N frames in operation S170, and may transmit the setting information IF_N2 to the image signal processor 100 in operation S180. In detail, in operation S190, the FRO circuit 120 of the image signal processor 100 may store the received setting information IF_N2.

[0072] When the image processing performed on the N frames is completed, the image signal processor 100 may transmit an end interrupt signal to the control processor 200 .

[0073] Subsequently, the image signal processor 100 and the control processor 200 may repeatedly perform operations S150 to S200, so that the control processor 200 may generate and send setting information in units of N frames, and the image signal processor 100 may perform image processing in units of frames (i.e., an interrupt signal may be generated in units of N frames and the interrupt signal may be sent to the control processor 200).

[0074] Figure 7 is a flowchart illustrating an operating method of an image processing system according to an example embodiment. Figure 7 The operation method can be Figure 1The image signal processor performs

[0075] Reference Figure 7 , the image processing system 1200 may determine the operation mode in operation S210. The image processing system 1200 may determine the operation mode based on the user's setting, or may detect the frame rate of the image data IDT received from the image sensor 1100 and may determine the operation mode based on the detected frame rate. In some example embodiments, when a frame is received from the image sensor 1100 at a high speed (e.g., 240fps or higher), the image processing system 1200 may determine the operation mode as a high-speed operation mode.

[0076] The image processing system 1200 may determine whether the determined operation mode is a high-speed operation mode in operation S220, and when the determined operation mode is the high-speed operation mode, as described above with reference to FIG. Figures 1 to 6 As described above, the image processing system 1200 may perform image processing in units of N frames in operation S230. In operation S231, the control processor 200 may generate setting information about the N frames, and may transmit the setting information to the image signal processor 100. The image signal processor 100 may perform image processing on the N frames in operation S232, and when the image processing is completed, the image signal processor 100 may generate an interrupt signal in operation S233. In operation S231, the image signal processor 100 may perform image processing substantially in units of frames based on a setting value of a corresponding frame provided from the FRO circuit 120 included therein, per frame.

[0077] When the determined operation mode is not the high-speed operation mode (i.e., when the determined operation mode is the normal operation mode or the low-speed operation mode), the image processing system 1200 may perform image processing in units of one frame in operation S240. The control processor 200 may generate setting information about one frame in operation S241, and may transmit the setting information to the image signal processor 100. The image signal processor 100 may perform image processing on one frame in operation S242, and when the image processing is completed, the image signal processor 100 may generate an interrupt signal in operation S243.

[0078] Figure 8 is a timing diagram illustrating an operation method based on an operation mode of an image signal processor according to an example embodiment. Figure 8 Show Figure 1 The operating method of the image signal processor 100 is shown in FIG.

[0079] Reference Figure 1 and Figure 8, the first operation mode may be a low-speed operation mode or a normal operation mode, and the second operation mode may be a high-speed operation mode. When the image processing system 1200 operates in the second operation mode, the frame rate of the image data (e.g., the receiving speed of the image data) may be higher than the frame rate of the image data when the image processing system 1200 operates in the first operation mode. The period T4 for receiving and processing N frames (e.g., the first frame to the nth frame) in the second operation mode may be relatively shorter than the period T3 for receiving and processing N frames in the first operation mode.

[0080] In the first operation mode, the image signal processor 100 may receive setting information IF1 to IFn+2 in units of frames from the control processor 200. The image signal processor 100 may perform image processing in units of frames based on the received setting information, and may transmit an interrupt signal indicating the start and end of the image processing (e.g., start interrupt signals INT_S1 to INT_Sn+1 and end interrupt signals INT_E1 to INT_En+1) to the control processor 200 in each frame.

[0081] In the second operation mode, the image signal processor 100 may receive setting information (e.g., first to third setting information) IF_N1, IF_N2, and IF_N3 from the control processor 200 in units of N frames. For example, the first setting information IF_N1 may include setting values ​​of the first frame F1 to the nth frame Fn. The image signal processor 100 may perform image processing on N frames, and may generate and transmit interrupt signals (e.g., start interrupt signals INT_S1 and INT_S2 and end interrupt signals INT_E1 and INT_E2) in units of N frames. In this case, the image signal processor 100 may perform image processing substantially in units of frames based on the setting values ​​of each frame provided from the FRO circuit 120 storing the setting information IF_N1, IF_N2, and IF_N3.

[0082] When the image signal processor 100 operates in the first operation mode, the image processing period T5 of one frame may be sufficient for the control processor 200 to generate setting information about the next frame. Therefore, in the first operation mode, the image processing period T5 of one frame may be ensured as a setting margin of the control processor 200, and the control processor 200 may generate and send setting information IF1 to IFn+2 in units of one frame. The image signal processor 100 may perform image processing in units of frames based on the received setting information. At this time, the FRO circuit 120 may be deactivated. When the image processing performed on one frame starts, the image signal processor 100 may generate and send a corresponding one of the interrupt signals INT_S1 to INT_Sn+1. When the image processing performed on one frame is completed, the image signal processor 100 may generate and send a corresponding one of the interrupt signals INT_E1 to INT_En+1.

[0083] When the image signal processor 100 operates in the second operation mode (i.e., when the frame rate of the image data is high), the image processing period T7 of one frame may be very short, and therefore, the control processor 200 may not be sufficient to generate the setting information about the next frame. Therefore, the control processor 200 may generate and transmit the setting information IF_N1 to IF_N3 in units of N frames, and therefore, the processing period T6 of the N frames may be ensured as a setting margin. The image signal processor 100 may store the received setting information in the FRO circuit 120, and when performing image processing on each frame, the image signal processor 100 may perform image processing based on the setting value of the corresponding frame provided from the FRO circuit 120, and may store the converted image data in the memory 300. When the image processing performed on the N frames starts, the image signal processor 100 may generate and transmit interrupt signals INT_S1 and INT_S2. When the image processing performed on the N frames is completed, the image signal processor 100 may generate and transmit interrupt signals INT_E1 and INT_E2.

[0084] As described above, the image signal processor 100 can change the operation mode based on the frame rate of the image data to adaptively change the frame setting method and the operation method. Therefore, even when the frame rate of the image data changes, the setting margin of the control processor 200 can be fully ensured.

[0085] Fig. 9 is a block diagram illustrating an image signal processor 100 a according to an example embodiment. Fig. 9 The configuration and operation of the image signal processor 100a are similar to Figure 2 The configuration and operation of the image signal processor 100 are similar, and therefore, the differences therebetween will be mainly described.

[0086] Reference Fig. 9, the image signal processor 100a may include an ISP core 110a, a FRO circuit 120, and a DMA controller 130, and at least one IP block (e.g., the first IP block 11a) among a plurality of IP blocks 112a (e.g., the first IP block 11a to the third IP block 13a) included in the ISP core 110a may include a FRO circuit 121. The controller 111a may provide some setting information about the first IP block 11a among a plurality of setting information IF_N about N frames received from the control processor 200 to the FRO circuit (FRO1) 121 of the first IP block 11a. In the case of performing image processing on each of the N frames, the first IP block 11a may perform image processing based on the setting value of the corresponding frame in the setting information stored in the FRO circuit 121.

[0087] Fig.10 is a block diagram illustrating an image signal processor 100 b according to an example embodiment.

[0088] Fig.10 The image signal processor 100b may include an ISP core 110b and a DMA controller 130. The ISP core 110b may include a plurality of IP blocks 112b (e.g., first to third IP blocks 11b to 13b), and the first to third IP blocks 11b to 13b may include FRO circuits (FRO1 to FRO3) 121 to 123, respectively. In other words, in Fig.10 In the image signal processor 100 b , the FRO circuits FRO1 to FRO3 may be included in the plurality of IP blocks 112 b , respectively.

[0089] The controller 111b may provide each of the plurality of IP blocks 112b with setting information about each of the plurality of IP blocks 112b (e.g., setting values ​​of N frames of each of the plurality of IP blocks 112b) among the plurality of setting information IF_N about N frames received from the control processor 200. Each of the plurality of IP blocks 112b may store the received setting information in a corresponding FRO circuit included therein, and in the case of performing image processing on each frame, each of the plurality of IP blocks 112b may perform image processing based on the setting value of the corresponding frame in the setting information stored in the corresponding FRO circuit. In addition, each of the plurality of IP blocks 112b may generate an address of the corresponding frame and may provide the address to the DMA controller 130, and thus, result data based on the image processing and / or converted image data may be stored in the memory 300.

[0090] Fig.11 is a block diagram illustrating an image signal processor 100 c according to an example embodiment.

[0091] Fig.11The image signal processor 100c may include an ISP core 110c, a FRO circuit 120, a first DMA controller 140 and a second DMA controller 150. The ISP core 110c may include a controller 111c, a plurality of IP blocks 112c, and a post-processing block 113c.

[0092] Fig.11 The configuration and operation of the image signal processor 100c are similar to Figure 2 The configuration and operation of the image signal processor 100 are similar. Therefore, the differences will be mainly described.

[0093] Reference Fig.11 The post-processing block 113c included in the ISP core 110c may perform post-processing on the converted image data generated by each of the plurality of IP blocks 112c. For example, the post-processing block 113c may include a scaler, a Joint Photographic Coding Experts Group (JPEG) circuit, and the like.

[0094] Each of the plurality of IP blocks 112c may directly transmit the converted image data to the post-processing block 113c, or may store the converted image data in the memory 300 through the first DMA controller 140. At this time, as described above with reference to Figure 2 As described above, the first DMA controller 140 may store the converted image data in the memory 300 based on the address ADDR received from the FRO circuit 120 per frame.

[0095] The post-processing block 113c may receive the converted image data from each of the plurality of IP blocks 112c, or may receive the converted image data stored in the memory 300 through the second DMA controller 150. The post-processing block 113c may store the post-processed image data IDT′ in the memory 300 through the second DMA controller 150, or may output the post-processed image data IDT′ to a processor included in the image processing device ( Figure 1 1000) in other elements (e.g., a display).

[0096] Fig.12 is a block diagram illustrating an image processing system 20 according to an example embodiment.

[0097] Reference Fig.12, the image processing system 20 may include a main processor 210, a ROM 220, a RAM 230, an image signal processor 240, a non-volatile memory interface 250, a camera interface 260, a memory interface 270, and a display interface 280. Elements of the image processing system 20 (e.g., the main processor 210, the ROM 220, the RAM 230, the image signal processor 240, the non-volatile memory interface 250, the camera interface 260, the memory interface 270, and the display interface 280) may send or receive data through a system bus 290. In some example embodiments, the image processing system 20 may be implemented as a system on chip (SoC). In some example embodiments, the image processing system 20 may be an application processor.

[0098] The main processor 210 may control the overall operation of the image processing system 20. The main processor 210 may be implemented, for example, with a CPU, a microprocessor, an ARM processor, an X86 processor, or a MIPS processor. According to some embodiments, the main processor 210 may be implemented with a computing component (e.g., a multi-core processor) including two or more independent processors (or cores). The main processor 210 may process or execute data and instruction codes (or programs) stored in the ROM 220 or the RAM 230, respectively.

[0099] The ROM 220 may store programs and / or data that are continuously used. The ROM 220 may be implemented as an EPROM or an EEPROM.

[0100] The RAM 230 may temporarily store programs, data and / or instructions. According to some embodiments, the RAM 230 may be implemented as a DRAM or an SRAM. The RAM 230 may temporarily store image data input / output through the interfaces 250 to 280 or generated by the image signal processor 240 through image processing.

[0101] The nonvolatile memory interface 250 may interface data input from or output to the nonvolatile memory device 255. The nonvolatile memory device 255 may be implemented with, for example, a memory card (e.g., a multimedia card (MMC), an embedded multimedia card (eMMC), a secure digital (SD) card, or a micro SD card).

[0102] The camera interface 260 may interface image data (e.g., raw image data) input from a camera 265 disposed outside the image processing system 20. The camera 265 may generate data corresponding to an image captured by using a plurality of light sensing devices. The image data received through the camera interface 260 may be provided to the image signal processor 240, or may be stored in the memory 275 through the memory interface 270.

[0103] The memory interface 270 may interface data input from or output to the memory 275. According to some embodiments, the memory 275 may be implemented as a volatile memory such as DRAM or SRAM or a nonvolatile memory such as ReRAM, PRAM, or NAND flash memory.

[0104] The display interface 280 may interface data (eg, image data) output to the display device 285. The display device 285 may output an image signal based on the image data through a display such as a liquid crystal display (LCD) or an active matrix organic light emitting diode (AMOLED).

[0105] The image signal processor 240 may perform image processing on the image data provided from the camera 265 to generate conversion image data, and may store the conversion image data in the memory 275 , or may scale the conversion image data to provide a scaled image to the display device 285 .

[0106] The above references Figures 1 to 11 The control processor and the image signal processor described above may be applied to the main processor 210 and the image signal processor 240, respectively. In the high-speed operation mode, the main processor 210 may generate setting information in units of N frames and may send the setting information to the image signal processor 240, and when the image processing performed on the N frames is completed, the image signal processor 240 may send an interrupt signal to the main processor 210. The image signal processor 240 may include a FRO circuit ( Figure 1 120), and the FRO circuit can store setting information. When each frame is image processed, the FRO circuit can provide the setting value of the corresponding frame. Therefore, even in the high-speed operation mode, the image signal processor 240 can normally perform image processing.

[0107] Fig.13 is a block diagram illustrating an image processing system 30 according to an example embodiment.

[0108] Reference Fig.13 , the image processing system 30 may include a CPU 310, a ROM 320, a post-processing block 330, a sensor interface 340, an ISP core 350, a FRO circuit 360, and a DMA controller 370. The CPU 310, the ROM 320, the post-processing block 330, the sensor interface 340, the ISP core 350, the FRO circuit 360, and the DMA controller 370 may send or receive data through a system bus 380.

[0109] The CPU 310 may control the overall operation of the image processing system 30 , and may process or execute a program stored in the ROM 320 to control the image processing operation.

[0110] The ROM 320 may store instruction codes (eg, programs) including image processing algorithms and / or data.

[0111] The post-processing block 330 may perform post-processing (eg, adjust the size of the data or compress the data) on the converted image data generated by the ISP core 350 . The post-processed image data may be stored in the memory 375 through the DMA controller 370 .

[0112] In some example embodiments, the image processing system 30 may further include a display interface, and the post-processed image data may be provided to a display device through the display interface. Alternatively, the image data stored in the memory 375 may be read through the DMA controller 370 and may be provided to a display device through the display interface.

[0113] The sensor interface 340 may communicate with the image sensor 345 and may receive image data (eg, raw image data) from the image sensor 345 .

[0114] The above references Figure 2 , Fig. 9 , Fig.10 and Fig.11 The described control processor, ISP core, FRO circuit, and DMA controller may be applied as the CPU 310, the ISP core 350, the FRO circuit 360, and the DMA controller 370, respectively. In the high-speed operation mode, the CPU 310 may generate setting information in units of N frames and may send the setting information to the ISP core 350, and when the ISP core 350 completes the image processing performed on the N frames, the FRO circuit 360 may send an interrupt signal to the CPU 310. The FRO circuit 360 may store the setting information, and when each frame is image-processed, the FRO circuit 360 may provide the setting value of the corresponding frame to the ISP core 350 and / or the DMA controller 370. Therefore, the image processing may be performed in units of one frame, and the processing data may be stored in the memory 375. Even in the high-speed operation mode, the image processing system 30 may normally perform the image processing, and may store the image-processed image data (e.g., the converted image data or the post-processed image data) in the memory 375.

[0115] Fig.14 is a block diagram illustrating an image processing apparatus 2000 according to an example embodiment. Fig.14 The image processing apparatus 2000 may be a portable terminal.

[0116] Reference Fig.14According to some example embodiments, the image processing device 2000 may include an application processor (AP) 2100, an image sensor 2200, a display device 2400, a working memory 2500, a storage device 2600, a user interface 2700, and a wireless transceiver 2800, and the application processor 2100 may include an image signal processor (ISP) 2300. Figure 1 The image signal processor 100 may be applied as the image signal processor 2300. In some embodiments, the image signal processor 100 may be implemented as a separate integrated circuit independent of the application processor 2100.

[0117] The application processor 2100 may control overall operations of the image processing apparatus 2000 , and may be provided as a SoC driving an application program and an operating system (OS).

[0118] The application processor 2100 may control the operation of the image signal processor 2300 , and may provide the conversion image data generated by the image signal processor 2300 to the display device 2400 or store the conversion image data generated by the image signal processor 2300 in the storage device 2600 .

[0119] The image sensor 2200 may generate image data (eg, raw image data) based on the received light signal, and may provide the image data to the image signal processor 2300 .

[0120] Reference above Figures 1 to 11 The described image signal processor may be applied as the image signal processor 2300. The image signal processor 2300 may receive setting information about the Nth frame from the processor included in the application processor 2100, and may perform image processing on the Nth frame based on the setting information. When the image processing performed on the Nth frame is completed, the image signal processor 2300 may send an interrupt signal to the processor.

[0121] The working memory 2500 may be implemented as a volatile memory such as DRAM or SRAM or a nonvolatile resistive memory such as FeRAM, RRAM, or PRAM, etc. The working memory 2500 may store programs and / or data each processed or executed by the application processor 2100 .

[0122] The storage device 2600 may be implemented as a nonvolatile memory device such as a NAND flash memory or a resistive memory, and may be provided as a memory card such as an MMC, an eMMC, an SD, or a micro SD, for example. The storage device 2600 may store data and / or a program corresponding to an execution algorithm for controlling an image processing operation of the image signal processor 2300, and when the image processing operation is performed, the data and / or the program may be loaded into the working memory 2500. In some embodiments, the storage device 2600 may store image data generated by the image signal processor 2300 (e.g., converted image data or post-processed image data).

[0123] The user interface 2700 may be implemented with various devices such as a keyboard, a subtitle key panel, a touch panel, a fingerprint sensor, and a microphone to receive user input. The user interface 2700 may receive user input and may provide a signal corresponding to the received user input to the application processor 2100.

[0124] The wireless transceiver 2800 may include a transceiver 2810 , a modem 2820 , and an antenna 2830 .

[0125] Although described with reference to specific examples and drawings, those skilled in the art may make various modifications, additions and substitutions to the exemplary embodiments according to the description. For example, the described techniques may be performed in a different order than the order of the described methods, and / or components such as the described systems, architectures, devices, circuits, etc. may be connected or combined in a different manner than the above methods, or the results may be appropriately achieved by other components or equivalents.

[0126] While the inventive concept has been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. An image processing system, comprising: a control processor configured to generate and output setting information corresponding to N image frames in units of N image frames, wherein N is an integer of 2 or more, wherein the N image frames have a frame rate equal to or greater than a predetermined threshold; and An image signal processor configured to: performing image processing on the N image frames received from the image sensor based on the setting information, and Based on completion of the image processing performed on the N image frames, an interrupt signal is generated in units of the N image frames and is transmitted to a control processor.

2. The image processing system according to claim 1, wherein: The setting information includes a setting value for each of the N image frames.

3. The image processing system according to claim 2, wherein: The image signal processor includes: a plurality of image processing blocks, each of the plurality of image processing blocks being configured to perform a set image processing; a direct memory access controller configured to store processed data generated by each of the plurality of image processing blocks in a memory; and The fast readout circuit is configured as: storing the setting information, and Based on the N image frames being sequentially image-processed, setting values ​​of the image frames being image-processed are provided to the plurality of image processing blocks or the direct memory access controller.

4. The image processing system according to claim 3, wherein: At least one image processing block among the plurality of image processing blocks is configured to store setting information related to the at least one image processing block among the setting information.

5. The image processing system according to claim 3, wherein: The setting values ​​include an address register value indicating an area where conversion image data generated by performing image processing is to be stored.

6. The image processing system according to claim 3, wherein: The direct memory access controller is configured as: receiving the set value or address from the fast readout circuit, and Processing data is stored in an area of ​​a memory corresponding to the setting value or the address, the processing data being received from each of the plurality of image processing blocks.

7. The image processing system according to claim 3, wherein: The fast readout circuit includes: N registers configured to store a setting value for each of the N image frames included in the setting information.

8. The image processing system according to claim 1, wherein: The control processor is configured to: generate setting information related to the next N image frames based on the image signal processor performing image processing on the N image frames, and send the generated setting information to the image signal processor.

9. The image processing system according to claim 1, wherein: The setting information includes: N address register values ​​representing areas where the plurality of pieces of converted image data of the N image frames are to be stored, and Information about the image frame that will generate an interrupt signal.

10. The image processing system according to claim 1, wherein: The image signal processor includes: a plurality of image processing blocks, each image processing block being configured to perform a set image processing; and a direct memory access controller configured to store the processed data generated by each of the plurality of image processing blocks in a memory, and Each of the plurality of image processing blocks comprises: The fast readout circuit is configured as: storing corresponding setting information among the setting information, and Based on the N image frames being sequentially image-processed, setting values ​​of the image frames being image-processed are provided to the direct memory access controller.

11. An image signal processor, comprising: an image processing engine configured to sequentially perform image processing on image frames received from the image sensor; a direct memory access controller configured to store processed data generated by the image processing engine in the memory; as well as A fast readout circuit is configured to: receive setting information including N setting values ​​corresponding to the N image frames in units of N image frames from a control processor, and provide the setting values ​​of the image frames on which the image processing is performed to an image processing engine or a direct memory access controller based on the N image frames being sequentially image processed, wherein N is an integer of 2 or more, and wherein the N image frames have a frame rate equal to or greater than a predetermined threshold, The fast readout circuit is configured to generate an interrupt signal provided to the control processor in units of the N image frames based on completion of image processing performed on the N image frames.

12. The image signal processor according to claim 11, wherein: The setting values ​​include address register values ​​indicating an area where an image processing result of each of the image frames based on the image processing or a converted image frame is to be stored.

13. The image signal processor according to claim 12, wherein: The fast readout circuit is configured as: Based on the image signal processor operating in the first mode, being in an activated state, and storing and outputting the N setting values, and Based on the image signal processor operating in the second mode, it is in a deactivated state.

14. The image signal processor according to claim 13, wherein: In the first mode, a frame rate of each of the N image frames received from the image sensor is 240 frames per second or greater.

15. An operating method of an image signal processor, the operating method comprising: receiving N setting values ​​from a control processor in units of N image frames, wherein N is an integer of 2 or greater, wherein the N image frames have a frame rate equal to or greater than a predetermined threshold; storing the N setting values ​​in a storage area; receiving an image frame from an image sensor; sequentially performing image processing on the N image frames from among the image frames based on the N setting values; and Based on the completion of the image processing performed on the N image frames, an end interrupt signal is generated in units of the N image frames, wherein the end interrupt signal indicates the completion of the image processing performed on the N image frames.

16. The operating method according to claim 15, further comprising: Receive N other setting values ​​from the control processor, Wherein, receiving the N other setting values ​​is performed simultaneously with performing the image processing.

17. The operating method according to claim 16, further comprising: generating a start interrupt signal, wherein the start interrupt signal indicates the start of image processing performed on the N image frames, The receiving of the N other setting values ​​is performed between the moment when the start interrupt signal is generated and the moment when the end interrupt signal is generated.

18. The operating method according to claim 15, wherein: The step of performing image processing includes storing an image processing result of each of the image frames in an area of ​​the memory represented by a setting value corresponding to the image frame on which the image processing is performed, among the N setting values.

19. The operating method according to claim 15, wherein: The steps to perform image processing include: receiving, by the image processing engine, a current setting value among the N setting values ​​from a storage area, and Perform image processing based on the current setting values, The current setting value corresponds to the image frame currently being processed.

20. An application processor, comprising: a main processor configured to: generate and output setting information including N setting values ​​in units of N image frames, wherein N is an integer of 2 or more, wherein the N image frames have a frame rate equal to or greater than a predetermined threshold; and An image signal processor configured to: receiving and storing the setting information, sequentially performing image processing on the N image frames received from the image sensor based on the setting information, and Based on the completion of the image processing performed on the N image frames, an end interrupt signal is generated in units of the N image frames and sent to a main processor, wherein the end interrupt signal indicates the completion of the image processing performed on the N image frames.

21. The application processor according to claim 20, wherein: The image signal processor includes: a plurality of image processing blocks, each image processing block being configured to perform a set of image processing; a direct memory access controller configured to store processed data generated by each of the plurality of image processing blocks in a memory; and The fast readout circuit is configured as: storing the setting information, and Based on the N image frames being sequentially image-processed, setting values ​​of the image frames being image-processed are provided to the plurality of image processing blocks or the direct memory access controller.

22. The application processor according to claim 20, wherein: Each of the N setting values ​​includes an address at which a corresponding converted image frame among N converted image frames respectively corresponding to the N image frames is to be stored, and Each of the N setting values ​​includes a setting value of a corresponding conversion image frame among the N conversion image frames.

23. The application processor according to claim 20, wherein: The image signal processor performs image processing on the N image frames at a frame rate equal to or greater than a predetermined threshold.

24. The application processor according to claim 20, wherein: The main processor is configured to generate setting information including N other setting values ​​before receiving the end interrupt signal, and transmit the generated setting information to the image signal processor.

Images