Image processing device and control method for image processing device

Abstract

To provide a synthesizing method suitable for a subject.SOLUTION: An image processing device has an imaging element that outputs a plurality of images with different gains on a single exposure and outputs an image in which the capacitance of a charge storage unit that stores the charge is the same for all images. first synthesizing means for acquiring the plurality of images and synthesizing the darkest image of the plurality of images by matching the brightness of the other images, second synthesizing means for synthesizing the brightest image of the plurality of images by matching the brightness of the other images, and synthesis selection means for selecting the first synthesis means and the second synthesis means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus that synthesizes a plurality of images output from an image pickup device and performs image processing for enlarging the dynamic range of an image, and a control method for the image processing apparatus.

Background Art

[0002] Conventionally, there is a synthesis processing technique for synthesizing a plurality of images taken with different exposure amounts. For example, by connecting appropriate exposure signals of each image having different exposure amounts, an image without overexposure or underexposure can be obtained. In recent years, due to improvements in sensor performance, etc., an image pickup device (Dual Gain Output DGO) having two column circuits for an output signal from a unit pixel, having different gains for amplification units in the column circuits, and capable of outputting images with different gains is used. This image pickup device can output two images (High gain and Low gain images) with different gains by a single exposure. Comparing the synthesis of two images by DGO and the synthesis of two images obtained by time division exposure, DGO has the advantages of not requiring alignment processing and being strong against moving objects. Therefore, it is compatible with image synthesis. Here, as a technique for synthesizing images with different exposure amounts, for example, Patent Document 1 describes a technique for enlarging the dynamic range by synthesizing a low exposure image and a high exposure image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art disclosed in Patent Document 1 mentioned above, the synthesis method is fixed, such as using a low-exposure image for the bright areas and a high-exposure image for the dark areas, which may not be suitable for the subject. Therefore, the object of the present invention is to provide a processing method that makes it possible to select a synthesis method that is more suitable for the subject. [Means for solving the problem]

[0005] To achieve the above objective, the present invention provides a method that allows for a single exposure The obtained image Multiple images with different gain settings of acquisition means of acquisition and The system comprises a first combining means that combines the darkest image of the plurality of images by matching the brightness of the other images, a second combining means that combines the brightest image of the plurality of images by matching the brightness of the other images, and a combining selection means that selects the first combining means and the second combining means. The second synthesis means comprises a synthesis means for synthesizing linear images and a synthesis means for synthesizing nonlinear images. It is characterized by the following: [Effects of the Invention]

[0006] According to the present invention, a synthesis method more suitable for the subject can be provided. [Brief explanation of the drawing]

[0007] [Figure 1] This is a general block diagram of the imaging device according to the embodiment of the present invention. [Figure 2] This diagram shows the internal blocks of a solid-state image sensor. [Figure 3] This is a block diagram showing the processing of the image synthesis unit 104. [Figure 4] This is an illustrative diagram of the distribution of luminance values. [Figure 5] This is an illustrative image of dark area priority compositing. [Figure 6] This is an illustrative diagram of highlight-prioritized image compositing. [Figure 7] This is an illustrative diagram of a nonlinear composite image prioritizing highlights. [Figure 8] This is a flowchart showing the processing of the synthesis method selection unit 1041. [Figure 9] Block diagram of the imaging measure according to the second embodiment [Figure 10] Flowchart showing the processing of the composition method selection unit 1002 **Embodiments for Carrying Out the Invention**

[0008] (First Embodiment) Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings.

[0009] FIG. 1 is a block diagram of an imaging device 100 as an example of an image processing device according to the present embodiment.

[0010] The optical lens 101 is an optical lens for taking in the light of a subject and forming an image of the light on the imaging element 102 to enter the light.

[0011] The imaging element 102 receives incident light from the optical lens 101, converts it into an electrical signal, and outputs it. Typical examples include a CCD image sensor (Charge Coupled Device) and a CMOS image sensor.

[0012] The video signals output from these imaging elements 102 include those that directly output analog video signals and those that perform AD (analog-digital) conversion processing inside the imaging element 102 and output digital data such as LVDS (Low voltage differential signaling).

[0013] Here, as an example of the imaging element used in the present invention, each block of the imaging element is shown in FIG. 2.

[0014] The timing pulse control unit 201 supplies an operation CLK to each block of this imaging element and supplies a timing signal to each block to control the operation of the imaging element.[[ID=​​The vertical scanning circuit 202 performs timing control for sequentially reading out pixel signal voltages of the pixel unit 203 arranged two-dimensionally. Generally, a video signal is sequentially read out row by row from the upper row to the lower row within one frame. In this embodiment, it is possible to select whether to output phase difference information from the pixel unit.

[0016] The pixel unit 203 is a photoelectric conversion element that performs photoelectric conversion according to the incident light amount and outputs it as a voltage. The column amplifier 204 is used to electrically amplify the signal read out from the pixel unit 203. By amplifying the signal with the column amplifier 204, the signal level of the pixel is amplified with respect to the noise output by the subsequent column ADC 205 (Analog Digital Convertor), equivalently improving the SN. Also, it has a structure that allows the gain of the column amplifier 204 to be changed from the timing pulse control unit.

[0017] In the imaging device 102 in this embodiment, for image synthesis, the column amplifier 204 has two input memories, and it is possible to change the column amplifier gain and output two types of gains. By having two input memories, two gains can be applied to the signal at a certain time read out from the photodiode and output. Although the data amount increases, two images with different gains and simultaneity can be obtained.

[0018] The column ADC 205 performs AD conversion on the readout signal from the column amplifier 204. The digitized signal is sequentially read out by the horizontal transfer circuit 206. The output of the horizontal transfer circuit 206 is input to the signal processing circuit 207. The signal processing circuit 207 is a circuit that performs digital signal processing. In addition to adding a certain amount of offset value by digital processing, it can perform a simple gain operation by performing a shift operation and a multiplication. Also, the pixel unit 203 may have an intentionally shaded pixel region, and a digital black level clamp operation using this can be performed.

[0019] The output of the signal processing circuit 207 is passed to the external output circuit 208. The external output circuit 208 has a serializer function and converts the multi-bit input parallel signal from the signal processing circuit 207 into a serial signal. It also converts this serial signal into, for example, an LVDS signal and outputs it for the exchange of image information with an external device.

[0020] The image acquisition unit 103 consists of blocks that capture the video signal output from the image sensor 102 and perform various processing steps. If AD conversion is not performed within the image sensor 102, it also includes an analog front end that performs analog-to-digital conversion. The image acquisition unit 103 performs tasks such as removing fixed pattern noise from the image sensor 102 and black level clamping. It also has the role of separating the image used for video signal recording from the evaluation signal for image sensor control.

[0021] The image synthesis unit 104 synthesizes an image from the image signal output from the image sensor using an arbitrary synthesis method. The image processing unit 105 performs various image processing functions, including pixel addition, noise reduction, gamma correction, knee correction, digital gain correction, and scratch correction, which are typical image processing functions of an imaging device. Furthermore, the image sensor 102, image acquisition unit 103, and image synthesis unit 104 may be provided as multiple different chips within the imaging device 100, or they may be integrated as an image sensor with a stacked structure.

[0022] The signal recording unit 106 records the video signal received from the signal processing unit 105 to a storage device or storage medium.

[0023] The exposure control unit 107 can calculate the optimal exposure amount from the video signal information received from the image acquisition unit 103. Then, it determines the operation of the image sensor control unit 108 and transmits the control.

[0024] The display unit 109 displays images and image-related information processed by the image processing unit 105 on a display device such as a liquid crystal.

[0025] The operation unit 110 consists of various operating components that act as an input unit, receiving user input and issuing operation instructions to the control unit 111. These components include a shutter button, electronic dial, power switch, directional keys, buttons, and a touch panel. The touch panel is configured by being superimposed on the liquid crystal of the display unit 109 and is an input device that outputs coordinate information corresponding to the position of contact.

[0026] Next, we will describe the operation of the image sensor 102 and the image synthesis unit 104 during image generation.

[0027] As mentioned above, the image sensor 102 of this embodiment can change the column amplifier gain and output it in order to generate an image. Figure 3 is a detailed block diagram of the processing performed by the image synthesis unit 104 on two images with two different gains output from the image sensor 102. Of the two images, the image with a relatively high gain is referred to as the High Gain image (hereinafter referred to as the H image), and the image with a relatively low gain is referred to as the Low Gain image (hereinafter referred to as the L image).

[0028] The image synthesis method selection unit 1041 is a processing unit that selects the image synthesis method. In this embodiment, the image synthesis selection unit 1041 automatically selects the synthesis method from the distribution of brightness values ​​of the L image. However, it is not limited to this, and the synthesis method may also be selected from the distribution of brightness values ​​of the H image, or the user may manually select the synthesis method in the image synthesis method selection unit 1041 via the operation unit 110 provided on the imaging device 100. The system may also be configured so that the user can select which of these methods to use for synthesis in the mode settings.

[0029] The synthesis method selection unit 1041 first calculates the brightness value of each pixel in the L image. Here, CCD sensors and CMOS sensors often adopt a Bayer array structure in which each pixel has only one type of color information. Therefore, in this embodiment, when calculating the brightness value of each pixel, the color information that the pixel does not have is interpolated using information from neighboring pixels. Then, the brightness value is calculated by multiplying each color information by a predetermined coefficient. However, the algorithm for calculating brightness can be any algorithm and is not limited to the above means. From the brightness of each pixel calculated, the synthesis method selection unit 1041 calculates the distribution of brightness values ​​(brightness value histogram) of the L image. Figure 4 is an illustrative diagram of the brightness value distribution, where the horizontal axis of the figure shows the brightness value and the vertical axis shows the count of each brightness value in the image.

[0030] Next, the synthesis method selection unit 1041 determines the luminance distribution range. In this embodiment, the distribution range is defined as the distance (difference in luminance values) between the lowest and highest luminances with counts exceeding the count threshold. Figure 4 shows an image of the count threshold and the distribution range. Here, the count threshold is the threshold for the lowest count used in the calculation when determining the distribution range. The reason for excluding luminance regions with low counts from the distribution range using the count threshold is to eliminate areas such as noise.

[0031] Furthermore, the synthesis method selection unit 1041 selects one synthesis method from at least two options—dark area priority synthesis, bright area priority synthesis, and bright area priority nonlinear synthesis—based on the determined luminance distribution range.

[0032] Here, we will explain each synthesis method. Figures 5(a) to (c) are conceptual diagrams of dark-area-priority synthesis, Figures 6(a) to (c) are conceptual diagrams of bright-area-priority synthesis, and Figures 7(a) to (d) are conceptual diagrams of bright-area-priority nonlinear synthesis. Here, each figure from 5 to 7 shows the correspondence between the input light intensity and the digital data. The dashed lines in Figures 5 to 7 also indicate the maximum value that can be represented by a single image before synthesis.

[0033] Dark-area-priority blending is a blending method that matches the exposure of the H image to that of the L image. As shown in Figure 5(b), an image (referred to as the H2 image) is generated by applying a gain with a small value to the H image, and during blending, the H2 image is used for dark areas and the L image is used for bright areas. Here, since a gain with a small value is applied to the H2 image, the noise value is also small. In this embodiment, the dark areas and bright areas are assumed to be predetermined. In this embodiment, the H2 image is used for dark areas and the L image is used for bright areas, but the H image and L2 image may be blended and used depending on the brightness. By using such a blending method, it is possible to generate a blended image with reduced noise in the dark areas compared to the original image. In this embodiment, the H image is used for low brightness areas and the L2 image is used for high brightness areas, but the H image and L2 image may be blended and used depending on the brightness.

[0034] Brightness-prioritizing composite is a composite method in which the exposure of the L image is matched to that of the H image. As shown in Figure 6(b), an image (referred to as the L2 image) is generated by applying a gain with a large value to the L image, and during composite, the H image is mainly used for the dark areas and the L2 image is mainly used for the bright areas. In this embodiment, the H image is used for the dark areas and the L2 image is used for the bright areas, but the H image and L2 image may be combined and used depending on the brightness. By using such a composite method, it is possible to generate an image that can express the gradation of the bright areas that could not be expressed before composite. However, as shown in Figure 6(c), if the digital data exceeds the maximum value that can be expressed by a single image, the number of bits of digital data required to express that data will be larger than before composite.

[0035] Brightness-prioritized nonlinear compositing is a compositing method that, like brightness-prioritized compositing, adjusts the exposure of the L image to match that of the H image. However, as shown in Figure 7(b), unlike brightness-prioritized compositing, it converts the L and H images into nonlinear images (represented as Ln and Hn images, respectively) before compositing. As mentioned earlier, in brightness-prioritized compositing, the number of bits in the data after compositing may be larger than that before compositing. In this case, if the bit width becomes too large, it may exceed the data width that the image processing device can handle. Therefore, by converting the H and L images into nonlinear images before compositing, it is possible to suppress the expansion of the bit width of the data after compositing while generating an image that can express the gradation of the bright areas that could not be expressed before compositing.

[0036] Figure 8 shows a flowchart illustrating the synthesis method selection process of the synthesis method selection unit 1041. Range threshold 1 and range threshold 2 in Figure 8 are values ​​of a pre-set distribution range. Range threshold 1 is the threshold for whether to match the exposure of the L image to the H image or the exposure of the H image to the L image. Range threshold 2 is the threshold for whether or not to convert to nonlinear before synthesis.

[0037] (STEP 8-1) If the luminance distribution range is smaller than the distribution range threshold 1, the synthesis method selection unit 1041 proceeds to STEP 8-2 and selects dark area priority synthesis as the synthesis method.

[0038] (STEP 8-3) The synthesis method selection unit 1041 proceeds to STEP 8-4 and selects bright area priority synthesis if the luminance distribution range is greater than or equal to the distribution range threshold 1 and less than the distribution range threshold 2. The synthesis method selection unit 1041 proceeds to STEP 8-5 and selects bright area priority nonlinear synthesis if the luminance distribution range is greater than or equal to the distribution range threshold 2. Furthermore, the system is not limited to the above embodiment; dark area priority synthesis and bright area priority nonlinear synthesis may be selected with distribution range threshold 1 as the boundary, or bright area priority synthesis and bright area priority nonlinear synthesis may be selected with distribution range threshold 2 as the boundary.

[0039] After selecting a synthesis method, the synthesis method selection unit 1041 outputs the synthesis method information to the dark area priority synthesis unit 1042, the bright area priority synthesis unit 1043, and the bright area priority nonlinear synthesis unit 1044. The synthesis method information includes information necessary for synthesis and synthesis flag information indicating whether or not to perform synthesis.

[0040] Upon receiving the synthesis information, the dark area-priority synthesis unit 1042, the bright area-priority synthesis unit 1043, and the bright area-priority nonlinear synthesis unit 1044 perform image synthesis if the synthesis flag information indicates that synthesis is set. The bright area-priority synthesis unit 1042 synthesizes the images using the bright area-priority synthesis method based on the input H image, L image, and image information, and outputs the synthesized image. The dark area-priority synthesis unit 1043 synthesizes the images using the dark area-priority synthesis method based on the input H image, L image, and image information, and outputs the synthesized image. The bright area-priority nonlinear synthesis unit 1044 synthesizes the images using the bright area-priority nonlinear synthesis method based on the input H image, L image, and image information, and outputs the synthesized image.

[0041] As described above, in this embodiment, by providing a configuration that allows adaptive selection of the image synthesis method for images generated with multiple different gains, a synthesis method more suitable for the subject can be provided.

[0042] (Second embodiment) Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0043] Figure 9 is a block diagram of an imaging device as an example of the image processing apparatus of this embodiment. Here, the components numbered the same as in Figure 1 are the same processing units as in the first embodiment, so their explanation is omitted.

[0044] The imaging device of this embodiment is capable of capturing both still images and videos. In this embodiment, settings made via the operation unit 110 allow the user to choose whether to retain only one of the captured images (a linear image, a RAW image) or a non-linear image (a JPEG image), or to retain both, when capturing still images. In addition to the above selection, settings made via the operation unit 110 also allow the user to set the frame rate of the recorded video when capturing videos.

[0045] The control unit 110 has an interface for receiving instructions from the user regarding the synthesis method and allows setting of items that can be set on the imaging device. In the imaging device of this embodiment, at least the following settings are possible. • Still image recording settings (You can choose to record RAW images, JPEG images, or both.) • Setting the frame rate during video recording (In this embodiment, 30FPS, 60FPS, and 240FPS can be set). The operation unit 110 outputs the received setting item information to the control unit 111, and the control unit 111 sets it in the synthesis method selection unit 1002.

[0046] The image synthesis method selection unit 1002 is a processing unit that selects the image synthesis method. In this embodiment, the image synthesis method selection unit 1002 selects the image synthesis method from the image input from the image acquisition unit 103 and the setting item information input from the operation unit 110. The selectable synthesis methods are RAW synthesis and YUV synthesis. RAW synthesis is a synthesis process performed in linear space with a linear image, while YUV synthesis is a synthesis process performed in non-linear space with a non-linear image (YUV format). The RAW synthesis in this embodiment is a process aimed at reducing noise in dark areas compared to before synthesis. The synthesized RAW can also be saved as a file. The YUV synthesis in this embodiment is a process aimed at enriching the tonal expression of bright areas compared to before synthesis. Since synthesis is not performed in RAW, the synthesized RAW cannot be saved as a file, but the processing speed is faster than RAW synthesis. If the image synthesis method selection unit 1002 selects RAW synthesis as the synthesis method, it outputs the image to the RAW image synthesis unit 1003, and if it selects YUV synthesis, it outputs the image to the YUV image synthesis unit 1004.

[0047] Here, Figure 10 shows the flow of the synthesis method selection process of the synthesis method selection unit 1002, and each process is explained below.

[0048] (STEP10-1) The synthesis method selection unit 1002 determines whether it is still image shooting based on the setting item information. If it is still image shooting, proceed to STEP 10-2. If it is not still image shooting, proceed to STEP 10-4.

[0049] (STEP10-2) The synthesis method selection unit 1002 calculates the dynamic range of the image from the input L image. In this embodiment, the dynamic range is defined as the difference between the lowest and highest brightness in the image, but the method of calculating the dynamic range is not limited to this. The synthesis method selection unit 1002 compares the calculated dynamic range with a preset dynamic range threshold, and if it is less than the dynamic threshold, proceeds to STEP 10-6. If the dynamic range is equal to or greater than the dynamic threshold, proceeds to STEP 10-7.

[0050] (STEP10-3) The synthesis method selection unit 1002 determines from the setting item information whether the setting is to retain RAW. If the setting is to retain RAW, proceed to STEP 10-6. If the setting is not to retain RAW, proceed to STEP 10-7.

[0051] (STEP10-4) The synthesis method selection unit 1002 calculates the dynamic range of the image from the input L image. In this embodiment, the dynamic range is defined as the difference between the lowest and highest brightness in the image, but the method for calculating the dynamic range is not limited to this. The synthesis method selection unit 1002 compares the calculated dynamic range with a preset dynamic range threshold, and proceeds to STEP 10-5 if the dynamic range is less than or equal to the dynamic threshold. If the dynamic range is greater than or equal to the dynamic threshold, proceeds to STEP 10-7.

[0052] (STEP10-5) The synthesis method selection unit 1002 compares the video's frame rate with a pre-set frame rate threshold. If the video's frame rate is less than the frame rate threshold, the synthesis method selection unit 1002 proceeds to STEP 10-6. If the video's frame rate is equal to or greater than the frame rate threshold, it proceeds to STEP 10-7.

[0053] (STEP10-6) The synthesis method selection unit 1002 selects RAW synthesis as the synthesis method.

[0054] (STEP10-7) The synthesis method selection unit 1002 selects YUV synthesis as the synthesis method.

[0055] The RAW image compositing unit 1003 is a processing unit that performs image compositing and image processing using RAW images. During compositing, gain is applied to the H image to adjust the exposure of the L image. The image with gain applied to the H image is called the H2 image. Compositing is then performed using the H2 image for dark areas and the L image for bright areas. The RAW image compositing unit also converts the combined RAW image to a YUV image. The RAW image compositing unit 1003 can perform various image processing on the RAW image before compositing, the combined RAW image, and the YUV image. The RAW image compositing unit outputs the combined RAW image and YUV image to the image processing unit 3.

[0056] The YUV image compositing unit 1004 is a processing unit that performs image compositing using YUV and also processes other image data. The YUV image compositing unit converts the H image and L image into YUV images (which are then named HY and LY images, respectively). It then applies gain to the LY image to match the exposure of the HY image (which is then named LY2 image). Finally, it performs compositing using the LH image for dark areas and the LY2 image for bright areas. The YUV image compositing unit 1004 can perform various image processing operations on the RAW image before compositing, the YUV image before compositing, and the YUV image after compositing. The YUV image compositing unit 1004 outputs the combined YUV image to the image processing unit 3.

[0057] The image processing unit 1005 is a processing unit that performs image processing on the input YUV image. The image processing unit 1005 performs various image processing on the input YUV image and outputs the YUV image and, if a RAW image is input, the RAW image to the signal recording unit 106.

[0058] As described above, in this embodiment, by selecting a synthesis method according to the settings at the time of shooting, it is possible to provide a synthesis method that is more suitable for the subject and to provide a synthesized image with good image quality.

[0059] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0060] (Other embodiments) The object of the present invention can also be achieved as follows: a storage medium containing program code for software describing the procedures for realizing the functions of each embodiment described above is supplied to a system or device. The computer (or CPU, MPU, etc.) of that system or device then reads and executes the program code stored on the storage medium (recording medium).

[0061] In this case, the program code read from the storage medium itself realizes the novel function of the present invention, and the storage medium and program that store that program code constitute the present invention.

[0062] Furthermore, storage media for supplying program code include, for example, flexible disks, hard disks, optical disks, and magneto-optical disks. CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, and ROMs can also be used.

[0063] Furthermore, the functions of each of the embodiments described above are realized by making the program code read by the computer executable. In addition, this also includes cases in which the OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program code, and the functions of each of the embodiments described above are realized through that processing.

[0064] Furthermore, the following cases are also included: First, program code read from a storage medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions of that program code, the CPU or other components of that function expansion board or function expansion unit perform some or all of the actual processing.

[0065] Furthermore, the present invention is not limited to devices primarily intended for photography, such as digital cameras, but is applicable to any device that incorporates or externally connects an imaging device, such as mobile phones, personal computers (laptops, desktops, tablets, etc.), and game consoles. Accordingly, the term "imaging device" in this specification is intended to encompass any electronic device equipped with an imaging function. [Explanation of symbols]

[0066] 100 Imaging device 101 Optical Lenses 102 Image sensor 103 Image acquisition unit 104 Image Synthesis Unit 105 Signal Processing Unit 106 Signal Recording Unit 107 Exposure Control Unit 108 Image sensor control unit 1041 Synthesis Method Selection Section 1042 Bright area priority composition section 1043 Dark area priority compositing section 1044 Bright area priority non-linear composite section

Claims

1. An acquisition means for acquiring multiple images with different gains applied to an image obtained in a single exposure, A first combining means for combining the first image with the brightness of the other images from among the aforementioned multiple images, A second combining means for combining the multiple images to create a second image that is brighter than the first image, by adjusting the brightness of the other images accordingly. The acquisition means includes a synthesis selection means that selects whether to synthesize the plurality of images acquired by the acquisition means using the first synthesis means or the second synthesis means, The image processing apparatus is characterized in that the second synthesis means comprises a synthesis means for synthesizing linear images and a synthesis means for synthesizing nonlinear images.

2. The image processing apparatus according to claim 1, wherein the composite selection means calculates the dynamic range using at least one of the plurality of images and performs the selection according to the calculated dynamic range.

3. The image processing apparatus according to claim 2, characterized in that the synthesis selection means performs synthesis using the first synthesis means when the calculated dynamic range is smaller than a first threshold, and performs synthesis using the second synthesis means when the dynamic range is larger than the first threshold.

4. The image processing apparatus according to claim 2 or 3, characterized in that the composite selection means performs the selection while the plurality of images are linear images when the calculated dynamic range is smaller than a second threshold, and performs the selection while the plurality of images are nonlinear images when the dynamic range is larger than a second threshold.

5. The system further includes an input means for receiving a synthesis method entered by the user, The image processing apparatus according to any one of claims 1 to 4, characterized in that the synthesis selection means performs the selection according to the synthesis method received by the input means.

6. The image processing apparatus according to claim 5, characterized in that the linear image is a RAW image.

7. The image processing apparatus according to claim 5 or 6, characterized in that the nonlinear image is in YUV format.

8. The image processing apparatus according to any one of claims 1 to 7, characterized in that the synthesis selection means makes the selection based on whether or not to record the linear image after synthesis.

9. The image processing apparatus according to any one of claims 1 to 7, characterized in that the composite selection means performs the selection based on the frame rate captured by the imaging device.

10. The imaging means for capturing the aforementioned multiple images, The image processing apparatus according to any one of claims 1 to 9, further comprising: a signal recording means for recording a composite image generated by the first and second synthesis means onto a recording medium.

11. The image processing apparatus according to claim 10, wherein the imaging means has a stacked structure, generates a plurality of images by applying different gains to an image obtained in a single exposure, and outputs them to the acquisition means.

12. An acquisition step to obtain multiple images with different gains applied to an image obtained in a single exposure, A first synthesis step involves combining the multiple images by adjusting the brightness of the other images to match the first image, A second synthesis step involves combining the multiple images by adjusting the brightness of the other images to create a second image that is brighter than the first image, The system includes a selection step for selecting whether to combine the multiple images acquired in the acquisition step using the first combination step or the second combination step, A control method for an image processing apparatus, characterized in that the second synthesis step involves either synthesizing using linear images or synthesizing using non-linear images.

13. A computer-executable program describing the procedure for controlling the image processing apparatus described in claim 12.

14. A computer-readable storage medium in which a program is stored that causes the computer to perform each step of the control method for the image processing apparatus described in claim 12.

Images