Image processing system, image signal processor and operation method thereof

Abstract

An image signal processor includes a shared circuit, a first circuit, and a second circuit. The shared circuit processes input image data. The first circuit provides a first algorithm for processing the input image data, and the first algorithm is used by the shared circuit to process the input image data in a first operating mode in a first shooting environment. The second circuit provides a second algorithm for processing the input image data, and the second algorithm is used by the shared circuit to process the input image data in a second operating mode different from the first operating mode in a second shooting environment different from the first shooting environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority of Korean Patent Application No. 10-2019-0002570 filed on January 9, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to an image signal processor, an image processing system, and an operating method of the image signal processor. Background Art

[0004] The image sensor may capture an object and transmit image data according to a specific format to the image signal processor. For example, the image sensor may send the captured result as Bayer format image data to the image signal processor, or may send the captured result as Tetra (or Quad) format image data to the image signal processor. Generally, the Bayer format is widely used, but in a weak light environment where the light intensity is insufficient, the noise-resistant Tetra format may be used.

[0005] When the image signal processor supports only the Bayer format, in order for the image signal processor to process the image data in the Tetra format, Remosaic calculation is required to convert the image data in the Tetra format into the Bayer format. This is because the pixel arrangement is different between the Bayer format and the Tetra format. However, since the Remosaic calculation performs the work of moving the pixels to another position to change the pixel arrangement in the Tetra format to the pixel arrangement in the Bayer format, there is a problem of information loss and resolution degradation. Summary of the invention

[0006] Aspects of the present disclosure provide an image signal processor that processes image data of different formats and is capable of directly processing the image data without converting one format into another format while being capable of not separately configuring hardware for each format.

[0007] According to another aspect of the present disclosure, an image signal processor includes a shared circuit, a first circuit, and a second circuit. The shared circuit processes input image data. The first circuit provides a first algorithm to the shared circuit, and the first algorithm is used by the shared circuit to process the input image data in a first operating mode in a first shooting environment. The second circuit provides a second algorithm to the shared circuit, and the second algorithm is used by the shared circuit to process the input image data in a second operating mode different from the first operating mode in a second shooting environment different from the first shooting environment.

[0008] According to another aspect of the present disclosure, an image processing system includes an image signal processor and a mode determination module. The image signal processor includes a shared circuit and a plurality of algorithm circuits. The shared circuit is configured to process input image data. The plurality of algorithm circuits are configured to provide the shared circuit with an algorithm for processing the input image data in different operation modes according to a shooting environment. The mode determination module detects the shooting environment based on information provided from an image sensor or an ambient light sensor and determines the operation mode of the image signal processor.

[0009] According to another aspect of the present disclosure, an operating method of an image signal processor includes: setting an operating mode for processing input image data, and when the operating mode is a first operating mode, selecting first format image data provided from an image sensor as input image data. The operating method also includes: when the operating mode is a second operating mode different from the first operating mode, selecting second format image data provided from the image sensor as input image data. When the operating mode is the first operating mode, the input image data is processed using a first circuit, and when the operating mode is the second operating mode, the input image data is processed using a second circuit.

[0010] However, the aspects of the present disclosure are not limited to those set forth herein. For those of ordinary skill in the art to which the present disclosure pertains, the above and other aspects of the present disclosure will become more apparent by referring to the detailed description of the present disclosure given below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other aspects and features of the present disclosure will become more apparent by describing in detail example embodiments thereof with reference to the accompanying drawings, in which:

[0012] Figure 1 is a diagram illustrating an image processing system according to an embodiment of the present disclosure;

[0013] Figure 2 is a diagram illustrating an image processing system according to an embodiment of the present disclosure;

[0014] Figure 3 is a diagram illustrating an image processing system according to an embodiment of the present disclosure;

[0015] Figure 4 is a diagram illustrating an image processing system according to an embodiment of the present disclosure;

[0016] Figure 5 is a diagram illustrating an operation example of the image processing system according to an embodiment of the present disclosure;

[0017] Figure 6 is a diagram illustrating another operation example of the image processing system according to an embodiment of the present disclosure;

[0018] Figure 7 is a diagram illustrating another image processing system according to an embodiment of the present disclosure;

[0019] Figure 8 is a diagram illustrating another image processing system according to an embodiment of the present disclosure;

[0020] Fig. 9 is a diagram illustrating another image processing system according to an embodiment of the present disclosure;

[0021] Fig.10 is a diagram illustrating an image processing system according to an embodiment of the present disclosure;

[0022] Fig.11 is a flowchart illustrating an operation method of an image signal processor according to an embodiment of the present disclosure; and

[0023] Fig.12 is a flowchart illustrating another operating method of an image signal processor according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Various embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0025] Figure 1 is a diagram illustrating an image processing system according to an embodiment of the present disclosure.

[0026] Reference Figure 1 , the image processing system 1 according to an embodiment of the present disclosure includes an image sensor 10, a multiplexing circuit 15, a SoC 20 (System on Chip), and a mode selection module 30. Here, the SoC 20 may be implemented as an AP (Application Processor), but the scope of the present disclosure is not limited thereto.

[0027] The image sensor 10 may operate under the control of the SoC 20. For example, the image sensor 10 may receive control data CTL_DATA from the SoC 20 and may start or end photographing according to the control data CTL_DATA.

[0028] The image sensor 10 photographs a subject using a lens and generates image data (B_DATA, T_DATA) Also, the image sensor 10 may transmit one of the image data (B_DATA, T_DATA) generated as a result of the photographing to the SoC 20 as input image data I_DATA.

[0029] The image sensor 10 may generate image data (B_DATA, T_DATA) by capturing an object in various ways in various shooting environments. Specifically, the image sensor 10 may capture an object in a first shooting environment / under a first shooting environment to generate first format image data B_DATA, and may capture an object in a second shooting environment / under a second shooting environment different from the first shooting environment to generate second format image data T_DATA.

[0030] In the present embodiment, the first shooting environment is a general environment, and the first format image data B_DATA may be Bayer format image data. Here, the general environment is a concept in contrast to the low light environment to be described later. The general environment refers to a shooting environment with a relatively sufficient amount of light, and this may be regarded as a strong light environment, a default environment, a normal environment, a standard environment, a main environment, an expected operating environment, etc., such as a daytime shooting environment and / or an outdoor shooting environment.

[0031] On the other hand, in the present embodiment, the second shooting environment is a low light environment, and the second format image data T_DATA may be Tetra format image data. Here, the low light environment refers to a shooting environment where the amount of light is relatively insufficient and the degree of noise is relatively higher than that of a general environment, such as a night shooting environment and / or an indoor shooting environment.

[0032] As is well known, the pixel configurations of the Bayer format and the Tetra format are different from each other. Specifically, in the case of the Bayer format, the pixel arrangement is performed so that G (green), R (red), and B (blue) are not adjacent to each other. Specifically, in a 2×2 pixel area, the pixels may be arranged so that G, which includes more brightness information, occupies 2 pixels, and each of R and B occupies 1 pixel. On the contrary, in the case of the Tetra format, the pixels may be arranged so that four pixels of the same color are adjacent to each other, and the pixels may be arranged so that four pixel areas are set as one unit area, and the unit area has an arrangement similar to the Bayer format. The Tetra format has anti-noise characteristics due to such a pixel arrangement, and can be usefully used in a low-light environment.

[0033] The multiplexing circuit 15 selects one of the image data (B_DATA, T_DATA) generated by the image sensor 10, outputs it as the input image data I_DATA, and transmits the input image data I_DATA to the SoC 20. The multiplexing circuit 15 may receive an input of a selection signal SEL from a mode selection module 30 (described later for the above selection).

[0034] The image sensor 10 may transmit or cause the input image data I_DATA selected by the multiplexing circuit 15 to be transmitted to the SoC 20 in various ways. For example, the image sensor 10 may transmit the input image data I_DATA to the camera interface 210, and for this purpose, the input image data I_DATA may be serialized in various forms according to any of various strategies. The serialized input image data I_DATA may be transmitted to the image signal processor 200 of the SoC 20 via the camera interface 210.

[0035] The SoC 20 may include an image signal processor 200 , a camera interface 210 , a processor 220 , a buffer 230 , and the like.

[0036] The image signal processor 200 processes the input image data I_DATA provided from the image sensor 10 via the multiplexing circuit 15. For example, the image signal processor 200 may perform at least one calculation on the input image data I_DATA to convert the input image data I_DATA into an image having, for example, RGB channels. Here, the at least one calculation may include crosstalk compensation, bad pixel correction, merging or reconstruction of a plurality of exposed pixels, demosaicing, noise reduction, image sharpening, image stabilization, color space conversion, compression, etc., but the scope of the present disclosure is not limited thereto.

[0037] The camera interface 210 supports data movement between the image sensor 10 and the image signal processor 200. In particular, when the image sensor 10 and the image signal processor 200 are implemented as separate chips rather than a single chip, the camera interface 210 is provided between the image sensor 10 and the image signal processor 200 to support data transmission.

[0038] The processor 220 may generally control the SoC 20. Specifically, the processor 220 may execute a program including instructions for operating not only the image signal processor 200 but also several elements of the SoC 20. For example, the processor 220 may be implemented as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., but the scope of the present disclosure is not limited thereto.

[0039] The buffer 230 provides a space in which data can be temporarily stored. For example, the image signal processor 200 can temporarily store image data in the buffer 230 as needed. In addition, a program executed by the processor 220 can be loaded into the buffer 230, and data used by the program can be stored in the buffer 230. For example, the buffer 230 can be implemented as an SRAM (static random access memory), a DRAM (dynamic random access memory), etc. However, the buffer 230 is not limited to the SRAM or the DRAM, and the buffer 230 can also be implemented as, for example, a non-volatile memory as needed.

[0040] The internal bus 290 serves as a path that enables elements in the SoC 20 (i.e., the image signal processor 200, the camera interface 210, the processor 220, the buffer 230, etc.) to exchange data with each other. Here, for example, the internal bus 290 may be implemented as AXI (Advanced eXtensible Interface) compliant with AMBA (Advanced Microcontroller Bus Architecture), but the scope of the present disclosure is not limited thereto.

[0041] Such a configuration of the SoC 20 is merely an example embodiment, and the scope of the present disclosure is not limited thereto. For example, the SoC 20 may further include additional elements such as an I / O interface that provides data exchange with an external device located outside the SoC 20 and a register that provides a data storage space, and Figure 1 Some of the elements shown may be omitted.

[0042] The mode selection module 30 selects the first format image data B_DATA provided from the image sensor 10 as the input image data I_DATA in the first operation mode. The mode selection module 30 selects the second format image data T_DATA provided from the image sensor 10 as the input image data I_DATA in the second operation mode.

[0043] Specifically, the mode selection module 30 receives the operation mode signal MODE and provides the selection signal SEL to the multiplexing circuit 15 according to the operation mode signal MODE. For example, when the value of the operation mode signal MODE means the first operation mode, the mode selection module 30 provides the selection signal SEL having a first value to the multiplexing circuit 15, thereby enabling the multiplexing circuit 15 to select the first format image data B_DATA from the image data (B_DATA, T_DATA) generated by the image sensor 10. On the contrary, when the value of the operation mode signal MODE means the second operation mode different from the first operation mode, the mode selection module 30 provides the selection signal SEL having a second value different from the first value to the multiplexing circuit 15, thereby enabling the multiplexing circuit 15 to select the second format data T_DATA from the image data (B_DATA, T_DATA) generated by the image sensor 10.

[0044] In this embodiment, the first operation mode may be a Bayer format processing mode. On the other hand, in this embodiment, the second operation mode may be a Tetra format processing mode.

[0045] The operation mode signal MODE is provided not only to the mode selection module 30 but also to the image signal processor 200. When the value of the operation mode signal MODE means the first operation mode, the image signal processor 200 recognizes the first format to process the input image data I_DATA. When the value of the operation mode signal MODE means the second operation mode, the image signal processor 200 recognizes the second format to process the input image data I_DATA.

[0046] In particular, it is noted that the image signal processor 200 does not perform format conversion to process the input image data I_DATA in each operation mode. That is, the image signal processor 200 processes or can process the input image data I_DATA without performing format conversion. For example, when operating in the Bayer format processing mode, the image signal processor 200 uses the input image data I_DATA in the Bayer format as is without format conversion. When operating in the Tetra format processing mode, the image signal processor 200 uses the input image data I_DATA in the Tetra format as is without format conversion.

[0047] Therefore, since the image processing system 1 of the present disclosure does not perform Remosiac calculation for moving pixels to another position for changing the pixel arrangement of the Tetra format to the pixel arrangement of the Bayer format, a phenomenon of information loss and / or resolution degradation can be prevented.

[0048] Figure 2 is a diagram illustrating an image processing system according to an embodiment of the present disclosure.

[0049] Reference Figure 2 The image processing system 2 according to the embodiment of the present disclosure includes an image sensor 10, a multiplexing circuit 15, a mode selection module 30, and an image signal processor 200. Here, since the image sensor 10, the multiplexing circuit 15, and the mode selection module 30 are similar to the reference Figure 1 The contents of the description are substantially the same, so a repeated description will not be provided.

[0050] In this embodiment, the image signal processor 200 includes a shared circuit 2000 , a first algorithm circuit 2010 (first circuit), and a second algorithm circuit 2020 (second circuit).

[0051] The shared circuit 2000 is a circuit for processing the input image data I_DATA provided from the image sensor 10 through the multiplexing circuit 15. That is, the shared circuit 2000 performs the above-referenced processing on the input image data I_DATA. Figure 1The shared circuit 2000 converts the input image data I_DATA into an image having, for example, RGB channels. The shared circuit 2000 is used in both the first operation mode and the second operation mode, so that the first operation mode and the second operation mode share the shared circuit 2000.

[0052] The first algorithm circuit 2010 is a first circuit and is used by the shared circuit 2000 to process the input image data I_DATA in a first operation mode in a first shooting environment. In other words, in the first operation mode, the image signal processor 200 may use the shared circuit 2000 to process or be used to process the input image data I_DATA based on a first algorithm provided by the first algorithm circuit 2010 (first circuit).

[0053] For example, in the Bayer format processing mode, the image signal processor 200 may use the shared circuit 2000 to process the input image data I_DATA in the Bayer format based on the Bayer format algorithm provided by the first algorithm circuit 2010 (first circuit).

[0054] The second algorithm circuit 2020 is a second circuit and is used by the shared circuit 2000 to process the input image data I_DATA in the second operation mode in the second shooting environment. In other words, in the second operation mode, the image signal processor 200 may use the shared circuit 2000 to process or be used to process the input image data I_DATA based on the second algorithm provided by the second algorithm circuit 2020 (second circuit).

[0055] For example, in the Tetra format processing mode, the image signal processor 200 may process the input image data I_DATA in the Tetra format using the shared circuit 2000 based on the Tetra format algorithm provided by the second algorithm circuit 2020 (second circuit).

[0056] That is, the shared circuit 2000 equally performs calculations such as crosstalk compensation, bad pixel correction, merging or reconstruction of multiple exposure pixels, demosaicing, noise reduction, image sharpening, image stabilization, color space conversion, and compression in the first operation mode and the second operation mode. However, the shared circuit 2000 may receive only algorithms that support the format of the input image data I_DATA from the algorithm circuits 2010 and 2020 according to each operation mode.

[0057] Therefore, since the image signal processor 200 according to the present embodiment can directly process the input image data I_DATA in each operation mode without performing format conversion, it has the advantage of being able to avoid information loss and image quality degradation, and at the same time has a small chip size when implemented by using a shared circuit without configuring a separate image processing circuit for each format.

[0058] Figure 3 is a diagram illustrating an image processing system according to an embodiment of the present disclosure.

[0059] Reference Figure 3 The image processing system 3 according to the embodiment of the present disclosure includes an image sensor 10, a multiplexing circuit 15, a mode selection module 30 and an image signal processor 200. Here, since the image sensor 10, the multiplexing circuit 15 and the mode selection module 30 are similar to the reference Figure 1 The contents of the descriptions are substantially the same, so no duplication will be provided.

[0060] In this embodiment, the image signal processor 200 includes a pre-processing module 200A and a core processing module 200B.

[0061] The pre-processing module 200A includes a pre-processing shared logic 2000A, a first pre-processing algorithm logic 2010A, and a second pre-processing algorithm logic 2020A. In addition, the core processing module 200B includes a core shared logic 2000B, a first core algorithm logic 2010B, and a second core algorithm logic 2020B. Each or any element of the pre-processing module 200A or the core processing module 200B marked as "logic" can be a software instantiation stored in and executed by the image signal processor 200. Alternatively, each or any element of the pre-processing module 200A or the core processing module 200B marked as "logic" can be implemented as a circuit such as an application-specific integrated circuit (ASIC).

[0062] On the other hand, the shared circuit 2000 includes a pre-processing shared logic 2000A and a core shared logic 2000B. In addition, the first algorithm circuit 2010 (first circuit) includes a first pre-processing algorithm logic 2010A and a first core algorithm logic 2010B. The second algorithm circuit 2020 (second circuit) includes a second pre-processing algorithm logic 2020A and a second core algorithm logic 2020B. Each or any element of the shared circuit 2000, the first algorithm circuit 2010, or the second algorithm circuit 2020 marked as "logic" can be a software instantiation stored in and executed by the image signal processor 200. Alternatively, each or any element of the shared circuit 2000, the first algorithm circuit 2010, or the second algorithm circuit 2020 marked as "logic" can be implemented as a circuit such as an application-specific integrated circuit (ASIC).

[0063] The pre-processing shared logic 2000A performs pre-processing on the input image data I_DATA by using the first pre-processing algorithm logic 2010A in the first operation mode, and performs pre-processing on the input image data I_DATA by using the second pre-processing algorithm logic 2020A in the second operation mode. Here, the pre-processing may include processes such as crosstalk compensation, bad pixel correction, and merging or reconstruction of multiple exposure pixels, but the scope of the present disclosure is not limited thereto.

[0064] For example, in the Bayer format processing mode, the image signal processor 200 performs or can perform preprocessing of the input image data I_DATA in the Bayer format based on the Bayer format algorithm provided by the first preprocessing algorithm logic 2010A using the preprocessing shared logic 2000A. In addition, in the Tetra format processing mode, the image signal processor 200 performs or can perform preprocessing of the input image data I_DATA in the Tetra format based on the Tetra format algorithm provided by the second preprocessing algorithm logic 2020A using the preprocessing shared logic 2000A.

[0065] The core sharing logic 2000B performs core processing on the input image data I_DATA by using the first core algorithm logic 2010B in the first operation mode, and performs core processing on the input image data I_DATA by using the second core algorithm logic 2020B in the second operation mode. Here, the core processing may include, for example, demosaic processing, but the scope of the present disclosure is not limited thereto.

[0066] For example, in the Bayer format processing mode, the image signal processor 200 uses the core shared logic 2000B to process the input image data I_DATA in the Bayer format based on the Bayer format algorithm provided by the first core algorithm logic 2010B. In addition, in the Tetra format processing mode, the image signal processor 200 can use the core shared logic 2000B to process the input image data I_DATA in the Tetra format based on the Tetra format algorithm provided by the second core algorithm logic 2020B.

[0067] As a result, since the image signal processor 200 according to the present embodiment can directly process the input image data I_DATA in each operation mode without performing format conversion, it has the advantage of being able to avoid information loss and image quality degradation, while having a small chip size when implemented by using a shared circuit without configuring a separate image processing circuit for each format.

[0068] Figure 4 is a diagram illustrating an image processing system according to an embodiment of the present disclosure. Figure 5 is a diagram illustrating an operation example of the image processing system according to an embodiment of the present disclosure. Figure 6 is a diagram illustrating another operation example of the image processing system according to an embodiment of the present disclosure.

[0069] Reference Figure 4 According to the embodiment of the present disclosure, the image signal processor 200 of the image processing system 4 includes a first preprocessing submodule 200C, a second preprocessing submodule 200D, a third preprocessing submodule 200E and a core processing module 200B. In the present embodiment, for the convenience of explanation, the first preprocessing submodule 200C, the second preprocessing submodule 200D and the third preprocessing submodule 200E are shown, but the present disclosure is not limited thereto, and more or less than three preprocessing submodules may be provided.

[0070] The first pre-processing submodule 200C includes a first pre-processing shared sub-logic 2000C, a first pre-processing algorithm sub-logic 2010C, and a second pre-processing algorithm sub-logic 2020C. Each or any element of the first pre-processing submodule 200C labeled as “logic” may be a software instantiation stored in and executed by the image signal processor 200. Alternatively, each or any element of the first pre-processing submodule 200C labeled as “logic” may be implemented as a circuit such as an application specific integrated circuit (ASIC).

[0071] The second pre-processing submodule 200D includes a second pre-processing shared sub-logic 2000D, a third pre-processing algorithm sub-logic 2010D, and a fourth pre-processing algorithm sub-logic 2020D. Each or any element of the second pre-processing submodule 200D labeled as “logic” may be a software instantiation stored in and executed by the image signal processor 200. Alternatively, each or any element of the second pre-processing submodule 200D labeled as “logic” may be implemented as a circuit such as an application specific integrated circuit (ASIC).

[0072] The third pre-processing submodule 200E includes a third pre-processing shared sub-logic 2000E, a fifth pre-processing algorithm sub-logic 2010E, and a sixth pre-processing algorithm sub-logic 2020E. Each or any element of the third pre-processing submodule 200E labeled as “logic” may be a software instantiation stored in and executed by the image signal processor 200. Alternatively, each or any element of the third pre-processing submodule 200E labeled as “logic” may be implemented as a circuit such as an application specific integrated circuit (ASIC).

[0073] In addition, the core processing module 200B includes a core shared logic 2000B, a first core algorithm logic 2010B, and a second core algorithm logic 2020B. Each or any element of the core processing module 200B labeled as "logic" may be a software instantiation stored in and executed by the image signal processor 200. Alternatively, each or any element of the core processing module 200B labeled as "logic" may be implemented as a circuit such as an application specific integrated circuit (ASIC).

[0074] On the other hand, the shared circuit 2000 includes pre-processing shared sub-logic 2000C, 2000D and 2000E and core shared logic 2000B. Each or any element of the shared circuit 2000 marked as "logic" can be a software instantiation stored in and executed by the image signal processor 200. Alternatively, each or any element of the shared circuit 2000 marked as "logic" can be implemented as a circuit such as an application-specific integrated circuit (ASIC). In addition, the first algorithm circuit 2010 (first circuit) includes a first pre-processing algorithm sub-logic 2010C, a third pre-processing algorithm sub-logic 2010D, a fifth pre-processing algorithm sub-logic 2010E and a first core algorithm logic 2010B. The second algorithm circuit 2020 (second circuit) includes a second pre-processing algorithm sub-logic 2020C, a fourth pre-processing algorithm sub-logic 2020D, a sixth pre-processing algorithm sub-logic 2020E and a second core algorithm logic 2020B. Each or any element of the first algorithm circuit 2010 (first circuit) and / or the second algorithm circuit 2020 (second circuit) labeled as “logic” may be a software instantiation stored in and executed by the image signal processor 200. Alternatively, each or any element of the first algorithm circuit 2010 (first circuit) and / or the second algorithm circuit 2020 (second circuit) labeled as “logic” may be implemented as a circuit such as an application specific integrated circuit (ASIC).

[0075] The first pre-processing shared sub-logic 2000C performs a first pre-processing on the input image data I_DATA by using the first pre-processing algorithm sub-logic 2010C in the first operation mode, and performs a first pre-processing on the input image data I_DATA by using the second pre-processing algorithm sub-logic 2020C in the second operation mode. Here, the first pre-processing may include a process such as crosstalk compensation, but the scope of the present disclosure is not limited thereto.

[0076] The second pre-processing shared sub-logic 2000D performs second pre-processing on the input image data I_DATA by using the third pre-processing algorithm sub-logic 2010D in the first operation mode, and performs second pre-processing on the input image data I_DATA by using the fourth pre-processing algorithm sub-logic 2020D in the second operation mode. Here, the second pre-processing may include a process such as bad pixel correction, but the scope of the present disclosure is not limited thereto.

[0077] The third pre-processing shared sub-logic 2000E performs a third pre-processing on the input image data I_DATA by using the fifth pre-processing algorithm sub-logic 2010E in the first operation mode, and performs a third pre-processing on the input image data I_DATA by using the sixth pre-processing algorithm sub-logic 2020E in the second operation mode. Here, the third pre-processing may include, for example, a process of merging or reconstructing a plurality of exposure pixels, but the scope of the present disclosure is not limited thereto.

[0078] The core sharing logic 2000B performs core processing on the input image data I_DATA by using the first core algorithm logic 2010B in the first operation mode, and performs core processing on the input image data I_DATA by using the second core algorithm logic 2020B in the second operation mode. Here, the core processing may include, for example, demosaic processing, but the scope of the present disclosure is not limited thereto.

[0079] Next, refer to Figure 5 In the Bayer format processing mode, the image signal processor 200 may use the first pre-processing shared sub-logic 2000C to perform the first pre-processing of crosstalk compensation on the Bayer format input image data I_DATA based on the Bayer format algorithm provided by the first pre-processing algorithm sub-logic 2010C. As a result, the image data R1 may be generated.

[0080] In addition, the image signal processor 200 may use the second preprocessing shared sub-logic 2000D to perform the second preprocessing of bad pixel correction on the Bayer format input image data I_DATA having completed the first preprocessing based on the Bayer format algorithm provided by the third preprocessing algorithm sub-logic 2010D. As a result, image data R2 may be generated.

[0081] In addition, the image signal processor 200 may use the third pre-processing shared sub-logic 2000E to perform a third pre-processing of merging or reconstructing a plurality of exposure pixels on the Bayer format input image data I_DATA that has completed the second pre-processing based on the Bayer format algorithm provided by the fifth pre-processing algorithm sub-logic 2010E. As a result, image data R3 may be generated.

[0082] In addition, the image signal processor 200 may use the core sharing logic 2000B to perform a core process of demosaicing on the Bayer format input image data I_DATA that has completed the first to third preprocessing based on the Bayer format algorithm provided by the first core algorithm logic 2010B. As a result, the image data R4 may be generated.

[0083] Next, refer to Figure 6In the Tetra format processing mode, the image signal processor 200 may use the first pre-processing shared sub-logic 2000C to perform the first pre-processing of crosstalk compensation on the input image data I_DATA in the Tetra format based on the Tetra format algorithm provided by the second pre-processing algorithm sub-logic 2020C. As a result, the image data R1 may be generated.

[0084] In addition, the image signal processor 200 may use the second preprocessing shared sub-logic 2000D to perform second preprocessing of bad pixel correction on the input image data I_DATA in the Tetra format that has completed the first preprocessing based on the Tetra format algorithm provided by the fourth preprocessing algorithm sub-logic 2020D. As a result, image data R2 may be generated.

[0085] In addition, the image signal processor 200 may use the third pre-processing shared sub-logic 2000E to perform a third pre-processing of merging or reconstructing a plurality of exposure pixels on the input image data I_DATA in the Tetra format that has completed the second pre-processing based on the Tetra format algorithm provided by the sixth pre-processing algorithm sub-logic 2020E. As a result, image data R3 may be generated.

[0086] In addition, the image signal processor 200 performs a core process of demosaicing the input image data I_DATA in the Tetra format that has completed the first to third preprocessing using the core shared logic 2000B based on the Tetra format algorithm provided by the second core algorithm logic 2020B. As a result, image data R4 can be generated.

[0087] As a result, since the image signal processor 200 according to the present embodiment can directly process the input image data I_DATA in each operation mode without performing format conversion, it has the advantage of being able to avoid information loss and image quality degradation, while having a small chip size when implemented by using a shared circuit without configuring a separate image processing circuit for each format.

[0088] Figure 7 is a diagram illustrating another image processing system according to an embodiment of the present disclosure.

[0089] Reference Figure 7 , and according to Figure 1 Unlike the image processing system 1 of the embodiment of FIG. 1 , in the image processing system 5 according to the embodiment of the present disclosure, the multiplexing circuit 15 and the mode selection module 30 are implemented inside the SoC 20.

[0090] That is, in addition to the image signal processor 200, the camera interface 210, the processor 220, and the buffer 230, the SoC 20 also includes a multiplexing circuit 15 and a mode selection module 30. The SoC 20 is configured so that the image signal processor 200, the camera interface 210, the processor 220, the buffer 230, the multiplexing circuit 15, and the mode selection module 30 can exchange data with each other via the internal bus 290.

[0091] Specifically, the selection signal SEL output by the mode selection module 30 for controlling the multiplexing circuit 15 may be transmitted to the multiplexing circuit 15 via the internal bus 290 of the SoC 20 .

[0092] However, this embodiment is only an example of various embodiments of the present disclosure. That is, both the multiplexing circuit 15 and the mode selection module 30 may be as follows Figure 1 As shown, the multiplexing circuit 15 is implemented outside the SoC 20. Alternatively, the multiplexing circuit 15 may be implemented as shown. Figure 7 In the embodiment of the present invention, the mode selection module 30 is implemented inside the SoC 20, while the mode selection module 30 can be implemented outside the SoC 20. In another embodiment, the mode selection module 30 can be implemented as Figure 7 In this embodiment, the multiplexing circuit 15 is implemented inside the SoC 20, while the multiplexing circuit 15 can be implemented outside the SoC 20.

[0093] Figure 8 is a diagram illustrating another image processing system according to an embodiment of the present disclosure.

[0094] Reference Figure 8 , and according to Figure 1 Unlike the image processing system 1 of the embodiment of the present disclosure, the image processing system 6 according to the embodiment of the present disclosure further includes a mode determination module 40. For convenience of explanation, in the present embodiment, the mode determination module 40 is shown as being implemented outside the SoC 20, but the scope of the present disclosure is not limited thereto, and the mode determination module 40 may also be implemented inside the SoC 20.

[0095] The mode decision module 40 detects a photographing environment based on information provided from the image sensor 10. The mode decision module 40 may decide the operation mode of the image signal processor 200 to be any one of the first operation mode and the second operation mode.

[0096] Specifically, the image sensor 10 may detect characteristics of the photographing environment, and generate and collect information about the characteristics of the photographing environment. For example, the image sensor 10 may collect data obtained by sensing characteristics such as the amount of light of the photographing environment.

[0097] In this case, based on the light amount information collected by the image sensor 10, when the light amount is sufficient, the mode determination module 40 may determine the operation mode as the Bayer format processing mode. Conversely, when the light amount is insufficient, the mode determination module 40 may determine the operation mode as the Tetra format processing mode. In an embodiment, the mode determination module 40 may compare a predetermined threshold with the data value of the light amount information and / or the data value of each other characteristic provided from the image sensor 10 to determine whether the light amount is insufficient. The mode determination module 40 may accordingly determine the operation mode as the determined operation mode.

[0098] In the present embodiment, the information provided from the image sensor 10 has been described as light amount information, but the scope of the present disclosure is not limited thereto, and the information provided from the image sensor 10 may include any type of sensing data that enables photographing information to be recognized.

[0099] The mode determination module 40 may change the operation mode between the first operation mode and the second operation mode during the operation of the image processing system 6. For example, when the shooting environment switches from outdoor during the day to indoor at night, the mode determination module 40 may dynamically change the operation mode from the first operation mode to the second operation mode during the operation of the image processing system 6.

[0100] As a result, since the image signal processor 200 according to the present embodiment can directly process the input image data I_DATA in each operation mode without performing format conversion, there is an advantage that information loss and image quality degradation can be avoided, while having a small chip size when implemented by using a shared circuit without configuring a separate image processing circuit for each format. In addition, a delay that may occur when the operation mode is changed can also be minimized.

[0101] Fig. 9 is a diagram illustrating another image processing system according to an embodiment of the present disclosure.

[0102] Reference Fig. 9 , and according to Figure 1 Unlike the image processing system 1 of the embodiment of the present disclosure, the image processing system 7 according to the embodiment of the present disclosure further includes a mode determination module 40 and an ambient light sensor 50. For convenience of description, in the present embodiment, the mode determination module 40 is shown as being implemented outside the SoC 20, but the scope of the present disclosure is not limited thereto, and the mode determination module 40 may also be implemented inside the SoC 20.

[0103] The mode decision module 40 detects the photographing environment based on the information provided from the ambient light sensor 50. The mode decision module 40 may decide that the operation mode of the image signal processor 200 is one of the first operation mode and the second operation mode.

[0104] Specifically, the ambient light sensor 50 may detect characteristics of a photographing environment, such as the amount of light of the photographing environment. The ambient light sensor 50 may collect data obtained by sensing the amount of light of the photographing environment.

[0105] In this case, based on the light amount information collected by the ambient light sensor 50, when the light amount is sufficient, the mode determination module 40 may determine the operation mode to be the Bayer format processing mode. Conversely, when the light amount is insufficient, the mode determination module 40 may determine the operation mode to be the Tetra format processing mode. In an embodiment, the mode determination module 40 may compare a predetermined threshold with the data value of the light amount information provided from the ambient light sensor 50 to determine whether the light amount is sufficient or insufficient. The mode determination module 40 may determine the operation mode accordingly.

[0106] The mode determination module 40 may change the operation mode between the first operation mode and the second operation mode during the operation of the image processing system 7. For example, when the shooting environment switches from outdoor during the day to indoor at night, the mode determination module 40 may dynamically change the operation mode from the first operation mode to the second operation mode during the operation of the image processing system 7.

[0107] Therefore, since the image signal processor 200 according to the present embodiment can directly process the input image data I_DATA in each operation mode without performing format conversion, it is advantageous in that information loss and image quality degradation can be avoided while having a small chip size in its implementation by using a shared circuit without configuring a separate image processing circuit for each format. In addition, a delay that may occur when the operation mode is changed can also be minimized.

[0108] Fig.10 is a diagram illustrating an image processing system according to an embodiment of the present disclosure.

[0109] Reference Fig.10 The image processing system 8 according to one embodiment of the present disclosure may be implemented as a computing device including an image sensor 10, an SoC 20, a mode selection module 30, a mode determination module 40, an ambient light sensor 50, a memory 60, a storage 70, and a display 80. The image sensor 10, the SoC 20, the mode selection module 30, the mode determination module 40, the ambient light sensor 50, the memory 60, the storage 70, and the display 80 may exchange data with each other via a bus 90.

[0110] Fig.11 is a flowchart illustrating an operation method of an image signal processor according to an embodiment of the present disclosure.

[0111] Reference Fig.11, the operating method of the image signal processor according to an embodiment of the present disclosure includes reading a predetermined operation mode signal MODE to set an operation mode for processing input image data I_DATA at S1101.

[0112] Furthermore, the method includes determining whether the operation mode is the second operation mode (low light mode) at S1103 .

[0113] When the operation mode is not the second operation mode (low light mode) (S1103=No), that is, when the operation mode is the first operation mode (normal mode), Fig.11 The method includes selecting first format image data B_DATA provided from the image sensor 10 as input image data I_DATA at S1105.

[0114] also, Fig.11 The method includes preprocessing input image data I_DATA in a first format using a preprocessing shared logic 2000A based on an algorithm of a first format (Bayer format) provided by a first preprocessing algorithm logic 2010A at S1107.

[0115] in addition, Fig.11 The method includes performing core processing on the first format input image data I_DATA that has been pre-processed at S1107 using the core shared logic 2000B based on the first format algorithm provided by the first core algorithm logic 2010B at S1109.

[0116] When the operation mode is the second operation mode (low light mode) (S1103=Yes), Fig.11 The method includes selecting second format image data T_DATA provided from the image sensor 10 as input image data I_DATA at S1113.

[0117] in addition, Fig.11 The method includes preprocessing the input image data I_DATA in the second format using the preprocessing shared logic 2000A based on an algorithm in the second format (Tetra format) provided by the second preprocessing algorithm logic 2020A at S1115.

[0118] in addition, Fig.11 The method includes performing core processing on the second format input image data I_DATA that has been pre-processed at S1115 using the core shared logic 2000B based on the second format algorithm provided by the second core algorithm logic 2020B at S1117.

[0119] in addition, Fig.11 The method includes outputting, at S1111 , an image for which core processing has been completed at S1109 or S1117 .

[0120] Fig.12 is a flowchart illustrating another operating method of an image signal processor according to an embodiment of the present disclosure.

[0121] Reference Fig.12 , the operating method of the image signal processor according to an embodiment of the present disclosure includes determining an operating mode for processing input image data I_DATA based on information provided from the image sensor 10 or the ambient light sensor 50 at S1201 .

[0122] also, Fig.12 The method includes determining whether the operation mode is the second operation mode (low light mode) at S1203.

[0123] When the operation mode is not the second operation mode (low light mode) (S1203=No), that is, when the operation mode is the first operation mode (normal mode), Fig.12 The method includes selecting first format image data B_DATA provided from the image sensor 10 as input image data I_DATA at S1205.

[0124] also, Fig.12 The method includes preprocessing the input image data I_DATA in the first format using the preprocessing shared logic 2000A based on the algorithm of the first format (Bayer format) provided by the first preprocessing algorithm logic 2010A at S1207.

[0125] also, Fig.12 The method includes performing core processing on the first format input image data I_DATA that has been pre-processed at S1207 using the core shared logic 2000B based on the first format algorithm provided by the first core algorithm logic 2010B at S1209.

[0126] When the operation mode is the second operation mode (low light mode) (S1203=Yes), Fig.12 The method includes selecting the second format image data T_DATA provided from the image sensor 10 as the input image data I_DATA at S1213.

[0127] in addition, Fig.12 The method includes preprocessing the input image data I_DATA in the second format using the preprocessing shared logic 2000A based on an algorithm in the second format (Tetra format) provided by the second preprocessing algorithm logic 2020A at S1215.

[0128] in addition, Fig.12The method includes performing core processing on the second format input image data I_DATA that has been pre-processed at S1215 using the core shared logic 2000B based on the second format algorithm provided by the second core algorithm logic 2020B at S1217.

[0129] also, Fig.12 The method includes outputting, at S1211 , an image for which core processing has been completed at S1209 or at S1217 .

[0130] Summarizing the detailed description, those skilled in the art will appreciate that many changes and modifications may be made to the preferred embodiments without substantially departing from the principles of the present disclosure. Therefore, the preferred embodiments of the present disclosure disclosed are used only in a general and descriptive sense, and not for limitation.

Claims

1. An image signal processor, include: a shared circuit that processes input image data; a first circuit that provides a first algorithm to the shared circuit, the first algorithm being used by the shared circuit to process the input image data in a first format in a first operating mode in a first capture environment; as well as a second circuit that provides a second algorithm to the shared circuit, the second algorithm being used by the shared circuit to process the input image data of a second format in a second capture environment that is different from the first capture environment and in a second operating mode that is different from the first operating mode, The pixel arrangement of the first format is different from the pixel arrangement of the second format.

2. The image signal processor according to claim 1, in, The first shooting environment is a normal environment, and the first operation mode is a Bayer format processing mode.

3. The image signal processor according to claim 1, in, The second shooting environment is a low-light environment, and the second operation mode is a Tetra format processing mode.

4. The image signal processor according to claim 1, further comprising: include: A mode selection module selects first format image data provided from an image sensor as the input image data in the first format in the first operation mode, and selects second format image data provided from the image sensor as the input image data in the second format in the second operation mode.

5. The image signal processor according to claim 4, in, The first format image data is Bayer format image data.

6. The image signal processor according to claim 4, in, The second format image data is Tetra format image data.

7. The image signal processor according to claim 1, in, The shared circuit includes a pre-processing shared logic and a core shared logic. The pre-processing shared logic performs pre-processing on the input image data of the first format by using a first pre-processing algorithm logic in the first operation mode, and performs pre-processing on the input image data of the second format by using a second pre-processing algorithm logic in the second operation mode, and The core sharing logic performs core processing on the input image data in the first format by using a first core arithmetic logic in the first operation mode, and performs core processing on the input image data in the second format by using a second core arithmetic logic in the second operation mode.

8. The image signal processor according to claim 7, in, The pre-processing shared logic, the first pre-processing algorithm logic and the second pre-processing algorithm logic are included in a pre-processing module, and The core shared logic, the first core algorithm logic, and the second core algorithm logic are included in a core processing module.

9. The image signal processor according to claim 7, in, The pre-processing shared logic includes a first pre-processing shared sub-logic and a second pre-processing shared sub-logic, The first preprocessing algorithm logic includes a first preprocessing algorithm sub-logic and a third preprocessing algorithm sub-logic, The second preprocessing algorithm logic includes a second preprocessing algorithm sub-logic and a fourth preprocessing algorithm sub-logic, The first pre-processing shared sub-logic performs a first pre-processing on the input image data in the first format by using the first pre-processing algorithm sub-logic in the first operation mode, and performs a first pre-processing on the input image data in the second format by using the second pre-processing algorithm sub-logic in the second operation mode, and The second preprocessing shared sub-logic performs second preprocessing on the input image data in the first format by using the third preprocessing algorithm sub-logic in the first operation mode, and performs second preprocessing on the input image data in the second format by using the fourth preprocessing algorithm sub-logic in the second operation mode.

10. An image processing system, include: an image signal processor including a shared circuit configured to process input image data and a plurality of algorithm circuits configured to provide an algorithm to the shared circuit, the algorithm being used to process the input image data in different formats having different pixel arrangements in different operation modes according to a photographing environment; as well as A mode determination module detects the photographing environment based on information provided from an image sensor or an ambient light sensor, and determines an operation mode of the image signal processor as a determined operation mode.

11. The image processing system according to claim 10, further comprising: include: A mode selection module selects one of first format image data having a first pixel arrangement and second format image data having a second pixel arrangement different from the first pixel arrangement provided from the image sensor as the input image data according to the determined operation mode.

12. A method of operating an image signal processor, include: Setting the operation mode for processing input image data; When the operation mode is the first operation mode, selecting first format image data provided from an image sensor as the input image data; when the operation mode is a second operation mode different from the first operation mode, selecting second format image data provided from the image sensor as the input image data, wherein a pixel arrangement of the first format image data is different from a pixel arrangement of the second format image data; when the operation mode is the first operation mode, processing the input image data using a first circuit, wherein the first circuit provides a first algorithm for processing the input image data; as well as When the operation mode is the second operation mode, the input image data is processed using a second circuit, wherein the second circuit provides a second algorithm for processing the input image data.

13. The method for operating the image signal processor according to claim 12, in, When the operation mode is the first operation mode, processing the input image data using the first circuit includes: performing pre-processing on the input image data using a first pre-processing algorithm logic, and Core processing is performed on the input image data using a first core algorithm logic.

14. The method for operating the image signal processor according to claim 13, in, The first preprocessing algorithm logic includes a first preprocessing algorithm sub-logic and a third preprocessing algorithm sub-logic, and Using the first pre-processing algorithm logic to perform pre-processing on the input image data includes: Preliminarily performing preprocessing on the input image data using the first preprocessing algorithm sub-logic, and The input image data is pre-processed a second time using the third pre-processing algorithm sub-logic.

15. The method for operating the image signal processor according to claim 12, in, When the operation mode is the second operation mode, processing the input image data using the second circuit includes: performing pre-processing on the input image data using a second pre-processing algorithm logic, and Core processing is performed on the input image data using a second core algorithm logic.

16. The method for operating the image signal processor according to claim 15, in, The second preprocessing algorithm logic includes a second preprocessing algorithm sub-logic and a fourth preprocessing algorithm sub-logic, and Performing preprocessing on the input image data using the second preprocessing algorithm logic includes: Preliminarily performing preprocessing on the input image data using the second preprocessing algorithm sub-logic, and The input image data is pre-processed a second time using the fourth pre-processing algorithm sub-logic.

17. The method for operating the image signal processor according to claim 12, in, The first operation mode is a Bayer format processing mode, and the first format image data is Bayer format image data.

18. The method for operating the image signal processor according to claim 12, in, The second operation mode is a Tetra format processing mode, and the second format image data is Tetra format image data.

19. The method for operating the image signal processor according to claim 12, in, Setting the operating mode for processing the input image data further comprises: One of the first operation mode and the second operation mode is determined based on information provided from the image sensor.

20. The method for operating the image signal processor according to claim 12, in, Setting the operating mode for processing the input image data further comprises: One of the first operating mode and the second operating mode is determined based on information provided from an ambient light sensor.

Images