Imaging device, imaging system, method for controlling the imaging device, and program

Abstract

The present invention provides an imaging device capable of suppressing image quality degradation that occurs during image synthesis processing. [Solution] An imaging device (100) that can communicate with an illumination device (300) comprises an image sensor (102) that acquires a first RAW image and a processing means (101) that instructs the illumination device to emit light and performs image synthesis processing. The processing means generates a second RAW image by removing information related to the image sensor or imaging settings from the first RAW image, performs synthesis processing using the second RAW image to show the effect of light emission from the illumination device, and generates a third RAW image by adding information to the second RAW image after synthesis processing.

Description

Technical Field

[0001] The present invention relates to an imaging device, an imaging system, a control method for an imaging device, and a program.

Background Art

[0002] Conventionally, as an exposure simulation function of an imaging device, there is known a function that can pseudo-reproduce a shooting result before performing actual shooting and allow a user to check an image reproduced before shooting. Also, in a case where shooting is performed in synchronization with an illumination device that emits flash light, since the illumination light is not constant light, the exposure simulation does not function well, which has been a factor that the shooting result cannot be known until actual shooting.

[0003] Patent Document 1 discloses an imaging system that calculates respective light components from images shot under each of a plurality of illumination units and ambient light and synthesizes the images. Patent Document 2 discloses a camera system that continuously controls a plurality of illumination devices in a series of sequences, emits light at a ratio with respect to constant light, and stores an image in which light is emitted and each non-emitting image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The imaging system disclosed in Patent Document 1 performs a process to convert the captured developed image into a luminance-linear color space by applying an inverse function of the gamma curve. As a result, the processing time until the completion of the synthesis process is extended due to the development processing time that occurs for each developed image, and the image quality of the final synthesis result may deteriorate when the inverse function is applied to a lossily compressed developed image.

[0006] The camera system disclosed in Patent Document 2 performs processing on developed images. As a result, the processing time until the completion of the synthesis process is extended due to the development processing time that occurs for each developed image, and there is a possibility of image quality degradation such as brightness shift or color shift caused by the synthesis of compressed color spaces.

[0007] Therefore, the present invention aims to provide an imaging device capable of suppressing image quality degradation that occurs during image synthesis processing. [Means for solving the problem]

[0008] One aspect of the present invention is an imaging device capable of communicating with an illumination device, comprising: an image sensor for acquiring a first RAW image; and processing means for instructing the illumination device to emit light and performing image synthesis processing, wherein the processing means generates a second RAW image by removing information related to the image sensor or imaging settings from the first RAW image; performs synthesis processing using the second RAW image to show the effect of the light emitted by the illumination device; and generates a third RAW image by adding the information to the second RAW image after the synthesis processing.

[0009] Other objects and features of the present invention are described in the following embodiments. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an imaging device that can suppress image quality degradation that occurs during image synthesis processing. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram of the imaging system according to the first embodiment. [Figure 2] This is a flowchart showing the imaging operation according to the first embodiment. [Figure 3] This flowchart shows the synthesis process according to each embodiment. [Figure 4] These are illustrative diagrams showing the first to third RAW images according to each embodiment. [Figure 5] This is an illustrative diagram showing the synthesis operation according to the first embodiment. [Figure 6] This is a block diagram of the imaging system according to the second embodiment. [Figure 7] This is a flowchart showing the imaging operation according to the second embodiment. [Figure 8] This is an illustrative diagram showing the synthesis operation according to the second embodiment. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0013] (First Embodiment) First, with reference to Figure 1, the imaging system 10 in the first embodiment of the present invention will be described. Figure 1 is a block diagram of the imaging system 10. The imaging system 10 comprises a camera body (imaging device) 100, a lens device 200, and a strobe device (illumination device) 300.

[0014] In the present embodiment, the lens device 200 is an interchangeable lens configured to be detachable from the camera body 100, but the present invention is not limited to this. The present embodiment is also applicable to an imaging device in which the lens device and the camera body are integrally configured. Further, in the present embodiment, a strobe device 300 is used as the lighting device, but the present invention is not limited to this, and other lighting devices such as an organic EL light or an LED light may be used. Further, in the present embodiment, the strobe device 300 is configured to be detachable from the camera body 100, but the present invention is not limited to this. The strobe device 300 may be integrally configured with the camera body 100, or the strobe device 300 may be wirelessly connected to the camera body 100 and not mechanically connected to the camera body 100. That is, the camera body 100 only needs to be configured to be able to communicate with the strobe device 300 (able to control the strobe device 300).

[0015] In FIG. 1, a lens device 200 is mounted on the front surface of the camera body 100. The lens device 200 is interchangeable, and the camera body 100 and the lens device 200 are electrically connected via a mount contact group 103. A strobe device 300 is mounted on the upper surface of the camera body 100. The strobe device 300 is interchangeable, and the camera body 100 and the strobe device 300 are electrically connected via a lighting contact group 109.

[0016] First, the configuration of the camera body 100 will be described. The camera control unit (processing means, control means) 101 is a microcomputer that controls the operations of each part of the camera body 100. The camera control unit 101 also includes a built-in memory for storing various adjustment values and programs for executing various controls. The built-in memory also serves as a buffer memory for temporarily storing various data processed at various locations.

[0017] The imaging element 102 is a CMOS sensor, a CCD sensor, or the like, and converts light from a subject incident through a lens (imaging optical system) 202 into an electrical signal, generates an image signal including a still image or a moving image, and outputs the image signal to the camera control unit 101.

[0018] The focal plane shutter 104 is disposed between the imaging device 102 and the lens 202 and operates according to an instruction from the camera control unit 101. The focal plane shutter 104 is composed of a front curtain and a rear curtain. When the front curtain travels and the shutter opens, the exposure of the imaging device 102 starts, and when the rear curtain travels and the shutter closes, the exposure of the imaging device 102 ends.

[0019] The camera operation unit (setting means) 105 includes operation members operated by the user, detects operations performed by the user via buttons, switches, dials, connected devices, etc. attached to the camera body 100, and sends a signal corresponding to the operation instruction to the camera control unit 101. When shooting a still image, the camera operation unit 105 outputs an instruction signal (SW1 signal) issued when the user half-presses the release button and an instruction signal (SW2 signal) issued when the user performs a full-press operation of deeply pressing the release button to the camera control unit 101. When shooting a moving image, the camera operation unit 105 outputs an instruction signal (REC signal) issued when the user operates the recording button to the camera control unit 101.

[0020] The camera display unit (display means) 106 displays shooting information and shooting images according to an instruction from the camera control unit 101.

[0021] The camera control unit 101 controls the operation of the camera body 100 based on the output signal of the camera operation unit 105. When the output signal of the camera operation unit 105 is the SW1 signal, the imaging device 102 is driven to perform imaging, and focus information such as the defocus amount at each distance measurement point is output. Also, the main subject is detected from the imaging result, and photometric control (AE operation) for measuring the luminance of the main subject is repeated, and the shutter speed, aperture value, and ISO sensitivity used at the time of shooting are determined from the photometric result. Here, the shutter speed, aperture value, and ISO sensitivity used at the time of shooting are collectively referred to as exposure control values. Note that the user may determine the above exposure control values manually from the camera operation unit 105. The determined exposure control values are displayed on the screen of the camera display unit 106.

[0022] When the output signal of the camera operation unit 105 is the SW2 signal, the camera control unit 101 drives the aperture 203 in the lens 202, sets the sensitivity (ISO sensitivity) of the image sensor 102, and controls the focal plane shutter 104 to illuminate the image sensor 102 with light. When the output signal of the camera operation unit 105 is the REC signal, the camera control unit 101 sets the sensitivity (ISO sensitivity) and frame rate of the image sensor 102, drives the image sensor 102 to take an image, and outputs focus information such as the amount of defocus at each metering point. The camera control unit 101 also detects the main subject from the imaging result and illuminates the image sensor 102 with light while repeatedly performing photometering control (AE operation) to measure the brightness of the main subject.

[0023] The lens control unit 201, described later, drives a focus lens (not shown) for adjusting the focus within the lens 202 to perform autofocus repeatedly, in accordance with instructions from the camera control unit 101. The camera control unit 101 displays the captured image on the screen of the camera display unit 106 according to the image data acquired from the image sensor 102, and also controls the writing of image data (including sound information) to the storage unit (storage means) 107.

[0024] The camera wireless communication unit 108 performs wireless communication between the camera body 100 and external devices, for example, by transmitting and receiving data such as image signals, audio signals, compressed image data, and compressed audio data. The camera wireless communication unit 108 also transmits and receives control signals related to shooting, such as start and end commands, as well as other setting and operation command information. The camera wireless communication unit 108 is a wireless communication module such as an infrared communication module, a Bluetooth® communication module, a wireless LAN communication module, or a WirelessUSB module.

[0025] Next, the configuration of the lens device 200 will be described. The lens control unit 201 is a microcomputer that controls the operation of each part of the lens device 200. The lens 202 is composed of multiple lenses and forms an image of the subject on the image sensor 102. Inside the lens 202, there is an aperture 203 for adjusting the amount of light and a focus lens (not shown) for adjusting the focus. The lens control unit 201 adjusts the amount of light taken into the camera and the focus according to instructions from the camera control unit 101 via control through the mount contact group 103, and sends distance information and other data to the camera control unit 101.

[0026] Next, the configuration of the strobe device 300 will be described. The lighting control unit 301 is a microcomputer that controls the operation of each part of the strobe device 300. The lighting control unit 301 can communicate with the camera control unit 101 via the lighting contact group 109 and can receive light emission control instructions and camera information from the camera, and transmit strobe device information. The light emission unit 302 is configured to include a light emission circuit and a light emission optical system.

[0027] The lighting control unit 303 is equipped with user-operated controls and detects user operations via buttons, dials, etc., attached to the strobe device 300, sending signals corresponding to the operation instructions to the lighting control unit 301. The lighting display unit 304 displays the light emission mode and other information according to the instructions of the lighting control unit 301. The power supply unit 305 uses the power of a battery (not shown) mounted on the strobe device 300 to supply energy for generating illumination light to illuminate the subject being photographed. Information regarding power supply (including battery level, etc.) is controlled by the lighting control unit 301 and transmitted to the camera control unit 101 via the lighting contact group 109.

[0028] The light-emitting unit 302 drives the light-emitting circuit to emit light from the xenon tube according to the instructions of the lighting control unit 301, and illuminates the subject at a predetermined illumination angle via the light-emitting optical system. The settings for the light emission amount and illumination angle of the light-emitting unit 302 may be set by the lighting operation unit 303, or may be obtained by communication from the camera control unit 101 via the lighting contact group 109 or the camera wireless communication unit 108 and lighting wireless communication unit 306. The lighting control unit 301 receives a control signal from the camera control unit 101 via the lighting contact group 109 and can make the light-emitting unit 302 emit light at a predetermined light emission amount and illumination angle in conjunction with the shooting operation of the camera body 100. When obtained by communication from the camera control unit 101, automatic setting by the camera control unit 101 or setting by operation from the camera operation unit 105 is possible.

[0029] The lighting wireless communication unit 306, like the camera wireless communication unit 108, performs wireless communication between the strobe device 300 and an external device (camera body 100), for example, to send and receive various settings and operation commands such as light output and illumination angle. The lighting wireless communication unit 306 is a wireless communication module such as an infrared communication module, a Bluetooth® communication module, a wireless LAN communication module, or a WirelessUSB module.

[0030] Next, with reference to Figure 2, the operation (imaging operation) of the imaging system 10 in this embodiment will be described. Figure 2 is a flowchart showing an example of an imaging operation using the camera body 100 and the strobe device 300. When the main power of the camera body 100 is turned on in the camera operation unit 105, power is supplied to each block of the camera body 100 from a battery (not shown), initialization and loading of various settings are performed, and preparation for shooting is completed. When the main power of the strobe device 300 is turned on in the lighting operation unit 303, power is supplied to each block of the strobe device 300 from the power supply unit 305, initialization and loading of various settings are performed, and preparation for shooting is completed. The strobe device 300 may also be configured to receive power from the camera body 100 via the lighting contact group 109.

[0031] In step S200, the user operates the camera control unit 105 according to the mode selection screen displayed on the camera display unit 106 to select the strobe simulation mode.

[0032] In step S201, the camera control unit 101 monitors the state of the SW1 signal and determines whether SW1 is ON or OFF. If it is determined that SW1 is ON, the process proceeds to step S202. In step S202, the camera control unit 101 notifies the strobe device 300 of the camera settings. In step S218, the lighting control unit 301 acquires various information about the strobe device 300 via the lighting contact group 109. The information acquired here includes information indicating the maximum light output, illumination angle range, and battery status of the strobe device 300.

[0033] In step S203, the camera control unit 101 issues a light emission instruction to the lighting control unit 301 via the lighting contact group 109 of the strobe device 300 to perform a pre-flash for acquiring the source image. In step S219, the lighting control unit 301 controls the light emission unit 302 based on the light emission instruction from the camera control unit 101 in step S203, and performs a pre-flash to acquire the source image to be used in the strobe simulation synthesis process.

[0034] In step S204, the camera control unit 101 acquires the pre-flash information of the strobe device 300 from step S219 via the illumination contact group 109, and drives the image sensor 102 to expose the pre-flash coming from the lens device 200. The camera control unit 101 then calculates the amount of light emitted by the light-emitting unit 302 that will be needed in the next step S206.

[0035] In step S205, the camera control unit 101 issues a light emission instruction to the lighting control unit 301 via the lighting contact group 109 of the strobe device 300, instructing it to emit light for acquiring the source image using the amount of light emission calculated in step S204. In step S220, the lighting control unit 301 controls the light emission unit 302 based on the light emission instruction from the camera control unit 101 in step S205, and emits light to acquire the source image to be used in the strobe simulation synthesis process.

[0036] In step S206, the camera control unit 101 acquires the light emission information from the strobe device 300 in step S220 via the illumination contact group 109, and drives the image sensor 102 to expose the light entering from the lens device 200. The camera control unit 101 drives the image sensor 102 and stores the acquired RAW image (image data that directly records the output of the image sensor 102) in its internal memory. Hereafter, the RAW image acquired from the image sensor 102 will be referred to as the first RAW image.

[0037] In step S206, the camera control unit 101 acquires two source images (first RAW images). One is a non-strobe image (second image data acquired in a second state where the strobe device 300 is not fired), and in addition to using exposure control values ​​set manually by the user, it may also be acquired by substituting it with an AE (Auto Exposure) image or the like. At this time, the lens control unit 201 performs autofocus by driving the focus lens (not shown) in the lens 202 according to instructions from the camera control unit 101 via the mount contact group 103.

[0038] Another type is a strobe-fired image (first image data acquired in the first state when the strobe device 300 is fired) using the aforementioned strobe device 300. It is preferable that the conditions other than the presence or absence of strobe flashing (for example, the position of the focus lens (not shown) in the lens 202, shutter speed, aperture value, ISO sensitivity, and other exposure control values) are the same for the strobe-fired image as for the non-strobe-fired image. This is because it may affect brightness shifts, color shifts, or deterioration of image resolution during the composite processing described later. Thus, the first RAW image includes first image data acquired in the first state when the strobe device 300 is fired and second image data acquired in the second state when the strobe device 300 is not fired.

[0039] In step S207, the camera control unit 101 uses the first RAW image stored in the camera control unit 101's internal memory in step S206 to perform a composite processing for strobe simulation, which shows an image obtained when the strobe device 300 is firing. The composite processing will be described later.

[0040] In step S208, the camera control unit 101 displays the result obtained by applying a gamma curve to the third RAW image synthesized in step S207 on the camera display unit 106. Alternatively, the camera control unit 101 may display the result obtained by applying a gamma curve to the fourth RAW image obtained by encoding (lossless compression) the third RAW image on the camera display unit 106. Or, the camera control unit 101 may display an image obtained by developing the third RAW image or the fourth RAW image. Note that by not performing development processing on each source image obtained in step S206, and instead performing development processing on the result after synthesis in step S207, the processing load can be reduced and the overall processing time when performing strobe simulation can be shortened.

[0041] In step S209, the user operates the camera control unit 105 to confirm whether the strobe simulation result displayed on the camera display unit 106 is the intended result. If the intended result is obtained, the user proceeds to step S211. On the other hand, if the result is different from the intended result and the user wants to change it, the user proceeds to step S210.

[0042] In step S210, the user operates the camera control unit 105 to change the settings of the strobe simulation results displayed on the camera display unit 106. These settings are mainly used for exposure compensation and affect the adjustment of the flash output of the strobe device 300. Once the settings are complete, the process proceeds to step S207, where the synthesis process is repeated, allowing for repeated fine-tuning until the user achieves their preferred result.

[0043] In step S211, the camera control unit 101 calculates the amount of light emitted by the strobe device 300 for this shooting based on the composite processing result determined in step S209.

[0044] In step S212, the camera control unit 101 monitors the state of the SW2 signal and determines whether SW2 is ON or OFF. If it is determined that SW2 is ON, the process proceeds to step S214. On the other hand, if SW2 is OFF, the camera control unit 101 proceeds to step S213 and repeats the monitoring in step S212 until SW2 is ON, as long as the ON state of SW1 is maintained. If SW1 is OFF in step S213, the process proceeds to step S201.

[0045] In step S214, the camera control unit 101 issues a light emission instruction to the lighting control unit 301 via the lighting contact group 109 of the strobe device 300, instructing it to emit light for the actual shooting at the amount of light emission calculated in step S211. In step S221, the lighting control unit 301 controls the light emission unit 302 based on the light emission instruction from the camera control unit 101 in step S214, and emits light for the actual shooting.

[0046] In step S215, the camera control unit 101 acquires the light emission information from the strobe device 300 in step S221 via the illumination contact group 109, and drives the image sensor 102 to perform imaging processing, exposing the light entering from the lens device 200. During imaging processing, the exposure control value or the amount of light emitted from the light-emitting unit 302 is controlled so that the acquired image has the user's preferred exposure determined in step S209. The camera operation unit 105 may be configured to allow separate settings for acquiring the first RAW image during the composite processing and for acquiring the first RAW image during the actual shooting. The camera control unit 101 drives the image sensor 102 and stores the acquired first RAW image of the actual shooting in the built-in memory of the camera control unit 101. After the imaging processing is completed, the process proceeds to step S216.

[0047] In step S216, the camera control unit 101 displays on the camera display unit 106 the result obtained by applying a gamma curve to the first RAW image acquired from the image sensor 102 in step S215. Alternatively, the camera control unit 101 may display on the camera display unit 106 the result obtained by applying a gamma curve to the fifth RAW image obtained by encoding (lossless compression) the first RAW image. Alternatively, the camera control unit 101 may display an image obtained by developing the first RAW image or the fifth RAW image.

[0048] In step S217, the camera control unit 101 saves the image developed in step S216 to the storage unit 107. At this time, information such as that of the camera body 100, lens device 200, and strobe device 300 is embedded in the image. In addition to the developed image of this shooting, the first to fifth RAW images and display images, including the source images generated in the series of flows of the strobe simulation mode, may also be saved to the storage unit 107 (the second RAW image will be described later). Furthermore, in order to compress the capacity used by the built-in memory in the camera control unit 101, the first to fifth RAW images and display images may be saved again from the built-in memory in the camera control unit 101 to the storage unit 107 after the processing of each of them is completed in a step prior to step S217. After each image is saved, the series of imaging operations is completed.

[0049] Next, the synthesis process in step S207 will be described with reference to Figures 3 to 5. Figure 3 is a flowchart showing an example of the synthesis process in step S207. Figure 4 is an image diagram showing the first to third RAW images. Figure 5 is an image diagram showing the synthesis calculation. When the first RAW image of the source images is stored in the built-in memory of the camera control unit 101 in step S206, the camera control unit 101 starts the flowchart in Figure 3.

[0050] In step S300, the camera control unit 101 detects the offset amount added to the first RAW image of the source image acquired in step S206. The offset amount will now be explained with reference to Figure 4.

[0051] The first RAW image obtained from the image sensor 102 can be represented as a model, as shown in the left-hand diagram of Figure 4. The offset amount shown here corresponds to information regarding the imaging conditions and is added to the data of the first RAW image. The offset amount is, for example, a value that changes for each image sensor 102 or a value that changes according to settings such as ISO sensitivity (imaging settings) (information regarding the image sensor and imaging settings). Therefore, if multiple images are directly combined using the first RAW image, a shift of the offset amount will occur from the RGB values ​​that were originally desired, and the image output after the combined processing will be a factor in image quality degradation such as brightness shift or color shift.

[0052] Therefore, in this embodiment, the second RAW image (the image on the right in Figure 4), obtained by temporarily removing the offset amount from the first RAW image, is used for the synthesis calculation, and after the synthesis process, the removed offset amount is added again to generate the third RAW image. This makes it possible to suppress the luminance shift or color shift that occurs during the synthesis process compared to when the offset amount is not removed.

[0053] From step S301 onward, the flowchart follows the above-described procedure. In step S301, the camera control unit 101 subtracts the offset amount detected in step S300 from the first RAW image of each source image, generates a second RAW image from which the offset amount has been removed, and stores it in the built-in memory of the camera control unit 101.

[0054] In step S302, the camera control unit 101 determines whether the color data of the second RAW image of the stored source image is outside a predetermined threshold range (predetermined range). This is done to prevent color shifts, as using an image with blown-out highlights, crushed blacks, or a similar exposure state as a source image for synthesis results in a breakdown of the RGB relationship.

[0055] If the color data falls outside a predetermined threshold range, the camera control unit 101 sets a warning flag and, when displaying on the camera display unit 106, displays a warning (warning process) such as zebra marking or highlighting in the corresponding image range. In step S302, the judgment is performed on the second RAW image of each source image, but this embodiment is not limited to this. The warning flag may be carried over along with the warning range information for any state of the first to fifth RAW images, or it may be performed each time for the first to fifth RAW images or the image displayed on the camera display unit 106 at a timing not shown. In addition, a predetermined subject detection may be performed and the warning judgment process may be changed according to the detection result of the main subject. For example, if a person is detected as the main subject, the warning may be set to be displayed only in the area around the face, or in the case of a landscape, the entire sky may be set to be the warning range. As a method for detecting the predetermined main subject, known techniques such as face detection, pupil detection, motion detection, pattern matching, and distance mapping can be used.

[0056] Thus, the camera control unit 101 can perform a warning process if at least one of the first RAW image, second RAW image, third RAW image, fourth RAW image, and fifth RAW image contains data that falls outside a predetermined range. Alternatively, the camera control unit 101 may perform a correction process if at least one of the first to fifth RAW images contains data that falls outside a predetermined range.

[0057] In step S303, the camera control unit 101 performs calculations for synthesis using the second RAW image of each source image, and stores the result in the built-in memory of the camera control unit 101.

[0058] Here, the synthesis calculation will be explained with reference to Figure 5. In this embodiment, the strobe light is extracted by subtracting the non-strobe image (4th image data) from the strobe-emitting image (3rd image data). Next, a gain equivalent to the required exposure compensation amount is applied based on the amount of light emitted when the strobe-emitting image was acquired (gain processing is performed). This reproduces what happens when the amount of light emitted is actually changed by exposure compensation. Then, by adding the exposure-compensated strobe light component to the non-strobe image, the result of the actual shooting can be simulated.

[0059] Thus, in this embodiment, the second RAW image includes a third image data generated by subtracting an offset amount from the first image data, and a fourth image data generated by subtracting an offset amount from the second image data. The camera control unit 101 performs a synthesis process using the data obtained by performing gain processing on the difference between the third image data and the fourth image data, and the second image data. Preferably, the camera control unit 101 performs gain processing during the synthesis process based on the amount of light emitted for each individual or group of strobe devices 300.

[0060] According to this method, although there is only one strobe device 300 in this embodiment, it is possible to acquire as source images the number of strobe devices 300 (or the number of groups that emit light) and one non-strobe image in step S206. Therefore, the optimal light output setting for the strobe device 300 can be found without having to repeat the actual shooting each time.

[0061] Furthermore, unless the user has made prior settings, it is preferable to set the default strobe simulation settings so that the exposure of the main subject is appropriate after the composite calculation. In other words, in the case of an imaging system using multiple strobe devices 300, it is preferable to set the default settings so that the amount of light emitted is adjusted according to how the strobe light hits the main subject from each strobe device 300 and the number of devices.

[0062] In this embodiment, it is preferable that the camera control unit 101 instructs the flash unit 300 to emit light at an appropriate amount for the subject to acquire a first RAW image. It is also preferable that the camera control unit 101 instructs the flash amount of the strobe device 300 so that at least one of the first to fifth RAW images does not fall outside a predetermined range, and performs gain processing according to the appropriate amount of light for the subject during the synthesis process to generate a third RAW image.

[0063] In step S304, the camera control unit 101 determines whether the color data of the composite second RAW image is outside the range of a predetermined threshold. If this determination has already been performed in step S302, this determination may be omitted. Also, if a warning flag is set, the camera display unit 106 may issue a warning according to the warning flag.

[0064] In step S305, the camera control unit 101 performs data correction processing on the composite second RAW image. This is because if an image with blown-out highlights, crushed blacks, or a similar exposure state is used as the source image for the composite, the RGB relationship will be disrupted. For example, if the source image is close to being blown out, problems such as the composite image turning pink may occur.

[0065] Therefore, in the case of the overexposed state mentioned as an example, a process may be performed to interpolate the data so that the relevant area is displayed in white, or a process may be performed to blur it, such as creating a white gradient. Alternatively, depending on the predetermined subject detection result described above, the user may be given the option to, for example, issue a warning in step S302 or step S304 if the interpolation range is a person, and only perform white interpolation in step S305 for the background sky.

[0066] In step S306, the camera control unit 101 adds the offset amount removed in step S301 to the composite second RAW image to generate a third RAW image with the offset amount added, and stores it in the camera control unit 101's built-in memory or storage unit 107. After that, the flowchart in Figure 3 ends and the process proceeds to step S208.

[0067] In this embodiment, by performing the synthesis process using linear RAW image data without a gamma curve, it is possible to suppress brightness shifts, color shifts, or image quality degradation that occur during the synthesis process, and to obtain a synthesis result that is close to the shooting result obtained in the actual shooting. Furthermore, in this embodiment, the first RAW image is stored in the built-in memory of the camera control unit 101 and the synthesis process shown in Figure 3 is performed, but it is not limited to this. Instead of storing the source image in the built-in memory of the camera control unit 101 in the state of the first RAW image, it may be encoded into a fifth RAW image and then stored. In that case, the fifth RAW image should be decoded back to the state of the first RAW image before performing the synthesis process shown in Figure 3, and then the synthesis process shown in Figure 3 should be performed. This increases the encoding and decoding processes, but it is possible to compress the built-in memory capacity of the camera control unit 101.

[0068] In this embodiment, the above-mentioned various processes are performed in conjunction with the imaging operation of the camera body 100, lens device 200, and strobe device 300. According to this embodiment, by performing a composite processing using the first to third RAW images of the source images, brightness shifts, color shifts, or image quality degradation that occur during the composite processing can be suppressed, and a composite processing result close to the shooting result obtained in the actual shooting can be obtained.

[0069] Note that the flowcharts described in this embodiment are merely examples, and various processes may be executed in a different order than those described in this embodiment.

[0070] (Second Embodiment) Next, with reference to Figure 6, the operation (imaging operation) of the imaging system 10a in the second embodiment of the present invention will be described. Figure 6 is a block diagram of the imaging system 10a. The imaging system 10a is composed of a camera body (imaging device) 100, a lens device 200, a strobe device (illumination device) 300, and a transmitter (transmitting device) 400.

[0071] Figure 6 shows the state in which the strobe device 300 in Figure 1 has been removed from the illumination contact group 109 of the camera body 100, and the transmitter 400 has been attached to the illumination contact group 109 instead. Figure 6 also shows the state in which the transmitter wireless communication unit 402 of the transmitter 400 and the illumination wireless communication unit 306 of each strobe device 300 are wirelessly connected. In this embodiment, there are multiple strobe devices 300 wirelessly connected to the transmitter 400, and they can be controlled individually by changing the wireless communication group settings.

[0072] This embodiment differs from the first embodiment in that the strobe device 300 is not physically connected to the camera body 100 and constitutes the imaging system as an independent lighting device. This embodiment also differs from the first embodiment in that the imaging system includes multiple strobe devices 300 connected wirelessly. In this embodiment, an example in which a transmitter 400 is wirelessly connected to multiple strobe devices 300 is described, but it is also possible to use the camera wireless communication unit 108 of the camera body 100 to wirelessly connect to multiple strobe devices 300 without using the transmitter 400.

[0073] In Figure 6, the camera body 100, lens device 200, and strobe device 300, other than the transmitter 400, are the same as in the first embodiment, so their descriptions are omitted.

[0074] The transmitter control unit 401 is a microcomputer that controls the operation of each part of the transmitter 400. The transmitter control unit 401 can communicate with the camera control unit 101 via the lighting contact group 109 and can receive flash control instructions and camera information from the camera to the strobe device 300, as well as transmit strobe device information.

[0075] The transmitter wireless communication unit 402, like the camera wireless communication unit 108 and the lighting wireless communication unit 306, performs wireless communication between the transmitter 400 and an external device (camera body 100 or strobe device 300). The transmitter wireless communication unit 402 performs various settings such as light output and illumination angle, as well as sending and receiving operation commands. The transmitter wireless communication unit 402 is a wireless communication module such as an infrared communication module, a Bluetooth® communication module, a wireless LAN communication module, or a WirelessUSB module.

[0076] The transmitter operation unit 403 is equipped with user-operated controls and detects user operations via buttons, dials, etc., attached to the transmitter 400, sending signals corresponding to the operation instructions to the transmitter control unit 401. The transmitter display unit 404 displays the illumination mode and other information according to instructions from the transmitter control unit 401. The transmitter 400 does not have a power supply unit such as a battery, and is configured to be driven by power supplied from the camera body 100 via the illumination contact group 109.

[0077] Next, the operation of the imaging system 10a (imaging operation) will be explained with reference to Figure 7. Figure 7 is a flowchart showing an example of an imaging operation using the camera body 100 and the strobe device 300. Note that in Figure 7, the parts that are the same as the operations in steps S200 to S221 in Figure 2 will be omitted from the explanation.

[0078] In step S701, the camera control unit 101 monitors the state of the SW1 signal and determines whether SW1 is ON or OFF. If it is determined that SW1 is ON, the process proceeds to step S702. In step S702, the camera control unit 101 notifies the strobe device 300 of the camera settings via wireless communication from the transmitter 400 through the lighting wireless communication unit 306 and the transmitter wireless communication unit 402. In step S718, the lighting control unit 301 of each of the multiple strobe devices 300 acquires various information from the transmitter 400 via wireless communication through the lighting wireless communication unit 306 and the transmitter wireless communication unit 402. The information acquired here includes information indicating the maximum light output, illumination angle range, battery status, number of wirelessly connected strobe devices 300, light emission preparation status, light emission group, ID, and channel settings of the strobe device 300.

[0079] In step S703, the camera control unit 101 sends a flashing instruction to the lighting control units 301 of each of the multiple strobe devices 300 via the transmitter 400 to perform a pre-flash for acquiring source images. In step S719, the lighting control units 301 of each of the multiple strobe devices 300 control the flashing unit 302 based on the flashing instruction from the camera control unit 101 in step S703 to perform a pre-flash for acquiring source images to be used in the strobe simulation synthesis process.

[0080] In step S704, the camera control unit 101 acquires pre-flash information from each of the multiple strobe devices 300 via the transmitter 400 in step S719. The camera control unit 101 also drives the image sensor 102 to expose the pre-flash coming from the lens device 200 and calculates the amount of light emitted by the light-emitting unit 302 that will be needed in the next step S706.

[0081] In step S705, the camera control unit 101 sends a light emission command to the lighting control units 301 of each of the multiple strobe devices 300 via the transmitter 400, instructing them to emit light for acquiring source images with the amount of light emission calculated in step S704. In step S720, the lighting control units 301 of each of the multiple strobe devices 300 control the light emission unit 302 based on the light emission command from the camera control unit 101 in step S705, and emit light for acquiring source images to be used in the strobe simulation synthesis process.

[0082] In step S706, the camera control unit 101 acquires the light emission information from each of the multiple strobe devices 300 in step S720 via the transmitter 400, and drives the image sensor 102 to expose the light entering from the lens device 200. The camera control unit 101 stores the first RAW image acquired by driving the image sensor 102 into the built-in memory of the camera control unit 101.

[0083] In steps S706 and S720, adjustments may be made to suppress overexposure in the strobe flash images of the source images in order to acquire source images that are easy to composite in step S707, in order to prevent the reacquisition of source images as described later. For example, adjustments may be made to control the flash output of each strobe device 300 so that the exposure is slightly underexposed, and then acquire the source images. Alternatively, exposure control for the strobe simulation mode may be performed, for example, by increasing the shutter speed to suppress subject blur and lowering the ISO sensitivity to suppress noise.

[0084] In this embodiment, in step S706, the camera control unit 101 acquires, in addition to the non-flash image, source images corresponding to the number of wireless groups of strobe devices 300 wirelessly connected to the transmitter 400. If flash control is not performed for each wireless group, but rather for each individual strobe device 300, then, in addition to the non-flash image, source images corresponding to the number of wirelessly connected strobe devices 300 are acquired.

[0085] In step S709, the user operates the camera control unit 105 to confirm whether the strobe simulation result displayed on the camera display unit 106 is the intended result. If the intended result is obtained, the user proceeds to step S711. On the other hand, if the result is different from the intended result and the user wants to change it, the user proceeds to step S722.

[0086] In step S722, the user operates the camera control unit 105 to select whether or not to reacquire the source images. If the user chooses to reacquire the source images, the process proceeds to step S704. On the other hand, if the user chooses not to reacquire the source images and only changes the settings, the process proceeds to step S710. Note that if the source images acquired in step S706 include images that fall outside a predetermined threshold range, such as overexposure or underexposure, an automatic determination process may be implemented to automatically reacquire the source images. Furthermore, when reacquiring source images, simply firing each strobe device 300 at a similar flash output to before reacquisition is unlikely to produce the desired result for the user. For this reason, for example, if overexposure occurred before reacquisition, the flash output of the strobe device 300 that fired for that source image may be changed to -2 stops, and if underexposure occurred before reacquisition, the flash output of the strobe device 300 that fired for that source image may be changed to +2 stops. In this way, the settings may be changed so that the reacquired source images fall within a predetermined threshold range.

[0087] In step S710, the user operates the camera control unit 105 to change the settings of the strobe simulation results displayed on the camera display unit 106. These settings can be changed for each strobe device 300 wirelessly connected via the transmitter 400, and are applied to the strobe devices 300 or wireless groups that the user wishes to modify to achieve their preferred results. Once the settings are complete, the process proceeds to step S707, where the synthesis process is repeated, allowing for repeated fine-tuning until the user achieves their desired result.

[0088] In step S711, the camera control unit 101 calculates the amount of light emitted from each strobe device 300 for the actual shooting based on the composite processing result determined in step S709.

[0089] In step S714, the camera control unit 101 sends a flashing instruction to the lighting control units 301 of each of the multiple strobe devices 300 via the transmitter 400, instructing them to flash for the actual shooting with the flash amount calculated in step S711. In step S721, the lighting control units 301 of each of the multiple strobe devices 300 control the flashing unit 302 based on the flashing instruction from the camera control unit 101 in step S714, and flash for the actual shooting.

[0090] In step S715, the camera control unit 101 acquires the light emission information from each of the multiple strobe devices 300 from step S721 via the transmitter 400, and drives the image sensor 102 to perform imaging processing, exposing the light entering from the lens device 200. During imaging processing, the exposure control value or the amount of light emitted from the light-emitting unit 302 is controlled so that the acquired image has the user's preferred exposure determined in step S709. The camera control unit 101 stores the first RAW image of this shoot, acquired by driving the image sensor 102, in the built-in memory of the camera control unit 101. After the imaging processing is completed, the process proceeds to step S716.

[0091] Finally, referring to Figure 8, the composite calculation when there are multiple strobe devices 300 will be explained. Figure 8 is an illustrative diagram showing the composite calculation in this embodiment. This embodiment differs from the first embodiment in that it extracts strobe light from each of the multiple wirelessly connected strobe devices 300 and applies a gain equivalent to the required exposure correction amount based on the amount of light emitted when acquiring the strobe flash image for each device. Then, by adding the exposure-corrected strobe light components to the non-strobe flash image, the results of the actual shooting can be simulated. In this method, the number of flash images for each strobe device 300 (or the number of wireless groups that emit light) and one non-strobe flash image are acquired as source images in step S206. This makes it possible to find the optimal light emission setting for the strobe device 300 for the user without having to repeat the actual shooting each time.

[0092] As described above, various processes are performed in conjunction with the imaging operation of the camera body 100, lens device 200, strobe device 300, and transmitter 400. According to this embodiment, in multi-light shooting with multiple strobe devices, by performing a composite processing using the first to third RAW images of the source images, brightness shifts, color shifts, or image quality degradation that occur in the composite processing are suppressed, and composite processing results close to the shooting results obtained in the actual shooting are obtained. Note that each flowchart described in this embodiment is merely an example, and various processes may be executed in a different order than the flowcharts described in this embodiment.

[0093] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0094] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0095] For example, instead of the camera body 100 as the imaging device, a smartphone with a camera function that can wirelessly connect to the strobe device 300, or a head-mounted display with a built-in camera may be used. Alternatively, instead of the strobe device 300, other lighting devices such as LED lights or organic EL lights may be used.

[0096] Each embodiment of the disclosure includes the following configuration and method. (Composition 1) An imaging device capable of communicating with a lighting device, An image sensor that acquires a first RAW image in both a first state where the illumination device is illuminated and a second state where it is not illuminated, It has processing means for performing image synthesis, The processing means is A second RAW image is generated by removing information regarding the imaging conditions from the first RAW image. The synthesis process is performed using the second RAW image. An imaging apparatus characterized by generating a third RAW image by adding the information to the second RAW image after synthesis processing. (Configuration 2) The imaging apparatus according to configuration 1, characterized in that the information relating to the imaging conditions includes information relating to the image sensor and imaging settings. (Composition 3) The first RAW image includes first image data acquired by illuminating the lighting device and second image data acquired without illuminating the lighting device. The second RAW image includes a third image data generated by removing the information from the first image data, and a fourth image data generated by removing the information from the second image. The imaging apparatus according to configuration 1 or 2, characterized in that the processing means performs the synthesis process using data obtained by performing gain processing on the difference between the third image data and the fourth image data, and the second image data. (Composition 4) The imaging apparatus according to any one of configurations 1 to 3, further comprising a storage means for storing the third RAW image. (Composition 5) The imaging apparatus according to configuration 4, characterized in that the storage means stores at least one of the first RAW image, the second RAW image, the third RAW image, a fourth RAW image obtained by lossless compression, and the first RAW image obtained by lossless compression. (Composition 6) The imaging apparatus according to configuration 5, further comprising display means for displaying an image to which a gamma curve has been applied to the third RAW image or the fourth RAW image. (Composition 7) The imaging apparatus according to configuration 5 or 6, characterized in that the processing means performs a warning process if at least one of the first RAW image, second RAW image, third RAW image, fourth RAW image, and fifth RAW image contains data that falls outside a predetermined range. (Composition 8) The imaging apparatus according to configuration 5 or 6, characterized in that the processing means performs a correction process if at least one of the first RAW image, second RAW image, third RAW image, fourth RAW image, and fifth RAW image contains data that falls outside a predetermined range. (Composition 9) The processing means is The amount of light emitted by the illumination device is instructed so that at least one of the first RAW image, second RAW image, third RAW image, fourth RAW image, and fifth RAW image does not fall outside a predetermined range. The imaging apparatus according to any one of configurations 5 to 8, characterized in that it generates the third RAW image by performing gain processing according to the appropriate amount of light for the subject during the synthesis process. (Composition 10) The imaging apparatus according to any one of configurations 1 to 9, further comprising setting means for individually setting a setting for acquiring the first RAW image during the synthesis process and a setting for acquiring the first RAW image during the actual shooting. (Composition 11) The imaging apparatus according to any one of configurations 1 to 10, characterized in that the processing means performs gain processing during the synthesis process based on the amount of light emitted for each individual or group of the lighting devices. (Composition 12) The processing means is An imaging device according to any one of configurations 1 to 11, characterized in that it instructs the emission of light at an amount that is appropriate for the amount of light on the subject, and acquires the first RAW image using the image sensor. (Composition 13) The imaging apparatus according to any one of configurations 1 to 12, wherein the processing means reacquires the first RAW image if the first RAW image acquired during the emission of light by the lighting device contains data outside a predetermined range, or if the first RAW image is an image different from the user's intention. (Composition 14) The imaging apparatus according to any one of configurations 1 to 13, characterized in that the processing means sets the shooting settings based on the settings used when acquiring the first RAW image for performing the synthesis processing. (Composition 15) The imaging device according to any one of configurations 1 to 14, characterized in that the processing means changes the warning determination process according to the detection result of the subject. (Composition 16) An imaging system characterized by comprising an imaging device described in any of configurations 1 to 15 and the illumination device. (Method 1) A control method for an imaging device that can communicate with an illumination device, A step of acquiring a first RAW image using an image sensor in both a first state where the illumination device is illuminated and a second state where it is not illuminated, The steps include generating a second RAW image by removing information regarding imaging conditions from the first RAW image, The steps include: performing a synthesis process using the aforementioned second RAW image, A method for controlling an imaging device, characterized by comprising the step of generating a third RAW image by adding the information to the second RAW image after synthesis processing. (Composition 17) A program characterized by causing a computer to execute the control method of the imaging device described in Method 1. [Explanation of Symbols]

[0097] 100 Camera body (imaging device) 101 Camera control unit (processing means) 102 Image sensor 300 Strobe devices (lighting devices)

Claims

1. An imaging device capable of communicating with a lighting device, An image sensor that acquires a first RAW image in both a first state in which the illumination device is illuminated and a second state in which it is not illuminated, It has processing means for performing image synthesis, The processing means is A second RAW image is generated by removing information regarding the imaging conditions from the first RAW image. The synthesis process is performed using the second RAW image. An imaging apparatus characterized by generating a third RAW image by adding the information to the second RAW image after synthesis processing.

2. The imaging apparatus according to claim 1, characterized in that the information relating to the imaging conditions includes information relating to the image sensor and imaging settings.

3. The first RAW image includes first image data acquired by illuminating the illumination device and second image data acquired without illuminating the illumination device. The second RAW image includes a third image generated by removing the information from the first image data and a fourth image generated by removing the information from the second image. The imaging apparatus according to claim 1, characterized in that the processing means performs the synthesis process using the data obtained by performing gain processing on the difference between the third image data and the fourth image data, and the second image data.

4. The imaging apparatus according to claim 1, further comprising a storage means for storing the third RAW image.

5. The imaging apparatus according to claim 4, characterized in that the storage means stores at least one of the first RAW image, the second RAW image, the third RAW image, a fourth RAW image obtained by lossless compression, and the first RAW image obtained by lossless compression.

6. The imaging apparatus according to claim 5, further comprising a display means for displaying an image in which a gamma curve has been applied to the third RAW image or the fourth RAW image.

7. The imaging apparatus according to claim 5, characterized in that the processing means performs a warning process if at least one of the first RAW image, the second RAW image, the third RAW image, the fourth RAW image, and the fifth RAW image contains data that falls outside a predetermined range.

8. The imaging apparatus according to claim 5, characterized in that the processing means performs a correction process if at least one of the first RAW image, the second RAW image, the third RAW image, the fourth RAW image, and the fifth RAW image contains data that falls outside a predetermined range.

9. The processing means is The amount of light emitted by the illumination device is instructed so that at least one of the first RAW image, the second RAW image, the third RAW image, the fourth RAW image, and the fifth RAW image does not fall outside a predetermined range. The imaging apparatus according to claim 5, characterized in that it generates the third RAW image by performing gain processing according to the appropriate amount of light for the subject during the synthesis process.

10. The imaging apparatus according to claim 1, further comprising setting means for individually setting settings for acquiring the first RAW image during the synthesis process and settings for acquiring the first RAW image during the actual shooting.

11. The imaging apparatus according to claim 1, characterized in that the processing means performs gain processing during the synthesis process based on the amount of light emitted for each individual or group of the lighting devices.

12. The processing means is The imaging apparatus according to claim 1, characterized in that it instructs the emission of light at an amount that is appropriate for the amount of light on the subject, and acquires the first RAW image using the image sensor.

13. The imaging apparatus according to claim 1, wherein the processing means reacquires the first RAW image when the first RAW image acquired during the emission of light by the lighting device contains data outside a predetermined range, or when the first RAW image is an image different from the user's intention.

14. The imaging apparatus according to claim 1, characterized in that the processing means sets the shooting settings based on the settings when the first RAW image for performing the synthesis processing was acquired.

15. The imaging device according to claim 1, characterized in that the processing means changes the warning determination process according to the detection result of the subject.

16. An imaging system characterized by comprising an imaging device according to any one of claims 1 to 15 and the illumination device.

17. A control method for an imaging device that can communicate with an illumination device, A step of acquiring a first RAW image using an image sensor in both a first state where the illumination device is illuminated and a second state where it is not illuminated, The steps include generating a second RAW image by removing information regarding the imaging conditions from the first RAW image, The steps include: performing a synthesis process using the second RAW image; A method for controlling an imaging device, characterized by comprising the step of generating a third RAW image by adding the information to the second RAW image after synthesis processing.

18. A program characterized by causing a computer to execute the control method of the imaging device described in claim 17.

Images