Image pickup apparatus, image pickup system, control method, storage medium, and computer program product

Abstract

The invention relates to an imaging apparatus, an imaging system, a control method, a storage medium, and a computer program product. An image pickup apparatus capable of communicating with a lighting apparatus may include: an image sensor configured to acquire a first original image in each of a first state in lighting by the lighting apparatus and a second state in the absence of lighting by the lighting apparatus; and a processing unit configured to generate a second original image by removing information related to an imaging condition from the first original image, perform a synthesis process using the second original image, and generate a third original image by adding the information to the second original image that has received the synthesis process.

Description

Technical Field

[0001] This disclosure relates to one or more embodiments of a camera device, camera system, control method, and storage medium. Background Technology

[0002] One of the traditional exposure simulation functions used in camera equipment is to simulate the shooting result before the actual shooting, so that the user can check the reproduced image before shooting. However, this function has the following problem: because the illumination light is not constant, the exposure simulation does not work properly when shooting in sync with a lighting device that emits a flash, so the shooting result is unknown until the actual shooting.

[0003] Japanese Patent Application Publication No. 2007-281937 discloses a camera system that calculates individual light components based on images captured under multiple lighting units and ambient light, and then synthesizes the images. Japanese Patent Application Publication No. 2010-114600 discloses a camera system that continuously controls multiple lighting devices in a series of sequences and stores images emitting light at a ratio relative to constant light and images emitting no light.

[0004] The camera system disclosed in Japanese Patent Application Publication No. 2007-281937 performs a conversion process to a linear color space by applying the inverse function of the γ curve to the captured (developed) image. Therefore, since each developed image requires development time, the processing time for completing the compositing process may increase, and the image quality of the final composite result may deteriorate when the inverse function is applied to a lossy compressed developed image.

[0005] The camera system disclosed in Japanese Patent Application Publication No. 2010-114600 processes developed images. Therefore, since each developed image requires developing time, the processing time used to complete the compositing process may increase, and the compositing of the compressed color space may lead to image quality degradation such as brightness shift or color shift. Summary of the Invention

[0006] A camera device capable of communicating with a lighting device may include: an image sensor configured to acquire a first original image in a first state under illumination by the lighting device and a second state without illumination by the lighting device; and a processing unit configured to generate a second original image by removing information related to imaging conditions from the first original image, perform compositing processing using the second original image, and generate a third original image by adding the information to the second original image that has already undergone compositing processing. Other aspects of this disclosure include a camera system having the aforementioned camera device, a control method corresponding to the aforementioned camera device, and a storage medium storing a program that causes a computer to execute the aforementioned control method.

[0007] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of embodiments will be provided by way of example. Attached Figure Description

[0008] Figure 1 This is a block diagram of the camera system according to the first embodiment.

[0009] Figure 2 This is a flowchart illustrating the camera operation according to the first embodiment.

[0010] Figure 3 This is a flowchart illustrating the synthesis process according to various embodiments.

[0011] Figure 4 This is a conceptual diagram showing the first to third raw images according to various embodiments.

[0012] Figure 5 The synthesis calculation according to the first embodiment is conceptually illustrated.

[0013] Figure 6 This is a block diagram of a camera system according to the second embodiment.

[0014] Figure 7 This is a flowchart illustrating the camera operation according to the second embodiment.

[0015] Figure 8 The synthesis calculation according to the second embodiment is conceptually illustrated. Detailed Implementation

[0016] In the following text, the term "unit" may refer to a software context, a hardware context, or a combination of both. In a software context, the term "unit" refers to a function, application, software module, feature, routine, instruction set, or program that can be executed by a programmable processor, such as a microprocessor, central processing unit (CPU), or a specially designed programmable device or controller. Memory contains instructions or programs that, when executed by the CPU, cause the CPU to perform operations corresponding to the unit or function. In a hardware context, the term "unit" refers to a hardware element, circuit, assembly, physical structure, system, module, or subsystem. According to a specific embodiment, the term "unit" may include mechanical, optical, or electrical components, or any combination thereof. The term "unit" may include active (e.g., transistors) or passive (e.g., capacitors) components. The term "unit" may include a semiconductor device having a substrate and other material layers having various conductivity concentrations. It may include a CPU or programmable processor capable of executing programs stored in memory to perform a specified function. The term "unit" may include logic elements (e.g., AND, OR) implemented by transistor circuitry or any other switching circuitry. In the combination of software and hardware contexts, the term "unit" or "circuit" refers to any combination of software and hardware contexts as described above. Additionally, the terms "element," "assembly," "component," or "device" may also refer to a "circuit" integrated with or not integrated with packaging material.

[0017] Embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. First Embodiment

[0018] Now refer to Figure 1 To describe the camera system 10 according to the first embodiment of the present disclosure. Figure 1 This is a block diagram of the camera system 10. The camera system 10 includes a camera body (camera device) 100, a lens device 200, and a flash device (lighting device, strobe device) 300.

[0019] In this embodiment, the lens device 200 is an interchangeable lens that can be attached to and detached from the camera body 100; however, this embodiment is applicable to video recording devices that integrate a lens device and a camera body. In this embodiment, the flash device 300 serves as an illumination device; however, other illumination devices such as organic EL lights or LED lights can also be used. In this embodiment, the flash device 300 can be attached to and detached from the camera body 100; however, the flash device 300 can be integrated with the camera body 100, or it can be configured to be wirelessly connected to the camera body 100 rather than mechanically connected. That is, the camera body 100 can communicate with the flash device 300 (so that the camera body 100 can control the flash device 300).

[0020] exist Figure 1 In this camera, lens device 200 is attached to the front of camera body 100. Lens device 200 is replaceable, and camera body 100 and lens device 200 are electrically connected via mounting contact group 103. Flash device 300 is attached to the top of camera body 100. Flash device 300 is replaceable, and camera body 100 and flash device 300 are electrically connected via illumination contact group 109.

[0021] First, the structure of the camera body 100 will be described. The camera control unit (one or more processors) 101 is a microcomputer that controls the operation of various parts of the camera body 100. The camera control unit 101 also includes an internal memory (one or more memory) that stores various adjustment values ​​and programs (instructions) for executing various controls. The internal memory also serves as a buffer memory for temporarily storing various data being processed at various locations.

[0022] The image sensor 102 is a CMOS sensor or a CCD sensor, etc., and converts the light from the subject incident through the lens (camera optical system) 202 into electrical signals, generates image signals including still images and moving images, and outputs these image signals to the camera control unit 101.

[0023] The focal plane shutter 104 is positioned between the image sensor 102 and the lens 202 and operates according to instructions from the camera control unit 101. The focal plane shutter 104 includes a front curtain and a rear curtain, and exposure of the image sensor 102 begins when the front curtain moves and the shutter opens, and ends when the rear curtain moves and the shutter closes.

[0024] The camera operation unit (setting unit) 105 includes user-operable operating components and detects operations performed by the user via buttons, switches, dials, connecting devices, etc., attached to the camera body 100, and sends signals corresponding to the operation commands to the camera control unit 101. For still image capture, the camera operation unit 105 outputs a command signal (SW1 signal) to the camera control unit 101 when the user half-presses the release button and a command signal (SW2 signal) when the user fully presses the release button. For moving image capture, the camera operation unit 105 outputs a command signal (REC signal) to the camera control unit 101 when the user operates the record button.

[0025] The camera display unit 106 displays video information and captured images according to instructions from the camera control unit 101.

[0026] The camera control unit 101 controls the operation of the camera body 100 based on the output signal from the camera operation unit 105. When the output signal from the camera operation unit 105 is an SW1 signal, the camera control unit 101 drives the image sensor 102 to capture images and outputs focus information such as the amount of defocus for each focus point. The camera control unit 101 also detects the main subject based on the captured image and repeatedly performs metering (light measurement) control (automatic exposure (AE) operation) to measure the brightness of the main subject, and determines the shutter speed, aperture value (F-number), and ISO sensitivity (ISO speed) to be used during image capture based on the metering results. Here, the shutter speed, aperture value, and ISO sensitivity to be used during image capture are collectively referred to as the exposure control value. The user can manually set the exposure control value using the camera operation unit 105. The determined exposure control value is displayed on the screen of the camera display unit 106.

[0027] When the output signal from the camera operation unit 105 is the SW2 signal, the camera control unit 101 drives the aperture 203 in the lens 202, sets the ISO sensitivity of the image sensor 102, and controls the focal plane shutter 104 to illuminate the image sensor 102. When the output signal from the camera operation unit 105 is the REC signal, the camera control unit 101 sets the ISO sensitivity and frame rate of the image sensor 102, drives the image sensor 102 to capture images, and outputs focus information such as the amount of defocus for each focus point. The camera control unit 101 also detects the main subject based on the captured image and illuminates the image sensor 102 while repeatedly performing metering control (AE operation) to measure the brightness of the main subject.

[0028] The lens control unit 201 described below drives a focusing lens (not shown) for focusing within the lens 202, according to instructions from the camera control unit 101, to repeatedly perform autofocus (AF). The camera control unit 101 displays captured images on the screen of the camera display unit 106 based on image data acquired from the image sensor 102, and controls the writing of image data (including sound information) to the memory (storage device, one or more memories) 107.

[0029] The camera wireless communication unit 108 performs wireless communication between the camera body 100 and external devices, thereby sending and receiving data such as image signals, audio signals, compressed image data, and compressed audio data. The camera wireless communication unit 108 also sends and receives camera-related control signals such as commands to start and stop recording, as well as other setting and operation command information. The camera wireless communication unit 108 is a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, or a wireless USB module.

[0030] The structure of the lens device 200 will now be described. The lens control unit 201 is a microcomputer that controls the operation of the various components of the lens device 200. The lens 202 includes multiple lenses and forms a subject image on the image sensor 102. The lens 202 includes an aperture 203 for adjusting the amount of light and a focusing lens (not shown) for focusing. The lens control unit 201, controlled via the mounting contact group 103, adjusts the amount of light entering the camera and the focus according to instructions from the camera control unit 101, and sends the distance information and other data obtained at this time to the camera control unit 101.

[0031] Next, the structure of the flash device 300 will be described. The illumination control unit 301 is a microcomputer that controls the operation of the various components of the flash device 300. The illumination control unit 301 can communicate with the camera control unit 101 via the illumination contact group 109, and can receive illumination control commands and camera information from the camera and transmit flash device information. The light emitter 302 includes a light-emitting circuit and a light-emitting optical system.

[0032] The illumination operation unit 303 has user-operable operating components that detect operations performed by the user via buttons, dials, etc., attached to the flash device 300, and send signals corresponding to the operation commands to the illumination control unit 301. The illumination display unit 304 displays the illumination mode, etc., according to commands from the illumination control unit 301. The power supply unit 305 uses power from a battery (not shown) installed in the flash device 300 to supply energy for generating illumination light to illuminate the subject to be photographed. Power-related information (including remaining battery capacity, etc.) is controlled by the illumination control unit 301 and sent to the camera control unit 101 via the illumination contact group 109.

[0033] According to instructions from the lighting control unit 301, the light emitter 302 drives the light-emitting circuit to emit light from the xenon tube, and illuminates the subject at a predetermined illumination angle via the light-emitting optical system. The light emission level and illumination angle of the light emitter 302 can be set by the lighting operation unit 303, or obtained from the camera control unit 101 via communication through the lighting contact group 109 or the camera wireless communication unit 108 and lighting wireless communication unit 306. The lighting control unit 301 receives control signals from the camera control unit 101 via the lighting contact group 109, and can, in conjunction with the imaging operation of the camera body 100, cause the light emitter 302 to emit light at a predetermined light emission level and illumination angle. When obtaining information from the camera control unit 101 via communication, the settings can be automatically configured by the camera control unit 101, or configured through operation from the camera operation unit 105.

[0034] Similar to the camera wireless communication unit 108, the illumination wireless communication unit 306 performs wireless communication between the flash device 300 and the external device (camera body 100), and sends and receives various settings and operation commands such as light intensity and illumination angle. The illumination wireless communication unit 306 may be, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, or a wireless USB module.

[0035] Now refer to Figure 2 The operation (camera operation) of the camera system 10 according to this embodiment will be described. Figure 2This is a flowchart illustrating an example of video recording operation using the camera body 100 and flash device 300. When the main power supply of the camera body 100 in the camera operation unit 105 is turned on, power is supplied to each component of the camera body 100 from a battery (not shown) for initialization, and various settings are read. This prepares for video recording operation. When the main power supply of the flash device 300 in the illumination operation unit 303 is turned on, power is supplied to each component of the flash device 300 from the power supply unit 305 for initialization, and various settings are read. This prepares for video recording operation. The flash device 300 can be configured to receive power from the camera body 100 via the illumination contact group 109.

[0036] In step S200, the user operates the camera operation unit 105 to select the flash simulation mode according to the mode selection screen displayed on the camera display unit 106.

[0037] In step S201, the camera control unit 101 monitors the status of the SW1 signal and determines whether SW1 is on. If SW1 is on, the process proceeds to step S202. In step S202, the camera control unit 101 notifies the flash device 300 of the camera settings. In step S218, the illumination control unit 301 acquires various information related to the flash device 300 via the illumination contact group 109. This acquired information includes information indicating the maximum light output of the flash device 300, the illumination angle range, battery status, etc.

[0038] In step S203, the camera control unit 101 sends a light emission command to the lighting control unit 301 via the lighting contact group 109 of the flash device 300 to perform a (flash) pre-emission (preparatory emission) for acquiring the source image. In step S219, the lighting control unit 301 controls the light emitter 302 based on the light emission command from the camera control unit 101 in step S203, and performs a pre-emission for acquiring the source image to be used in the compositing process of the flash simulation.

[0039] In step S204, the camera control unit 101 acquires the pre-emission information from the flash device 300 in step S219 via the illumination contact group 109, and drives the image sensor 102 to perform exposure using the pre-emission received from the lens device 200. Then, the camera control unit 101 calculates the amount of light emitted by the light emitter 302 required in the next step S206.

[0040] In step S205, the camera control unit 101 sends a light emission command to the lighting control unit 301 via the lighting contact group 109 of the flash device 300, thereby emitting light for acquiring the source image at the light emission amount calculated in step S204. In step S220, the lighting control unit 301 controls the light emitter 302 based on the light emission command from the camera control unit 101 in step S205, and emits light to acquire the source image to be used in the flash simulation compositing process.

[0041] In step S206, the camera control unit 101 acquires the light emission information related to the flash device 300 from step S220 via the illumination contact group 109, and drives the image sensor 102 to expose the light incident from the lens device 200. The camera control unit 101 drives the image sensor 102 and stores the acquired raw image (image data recorded as is from the output of the image sensor 102) in the internal memory of the camera control unit 101. Hereinafter, the raw image acquired from the image sensor 102 will be referred to as the first raw image.

[0042] In step S206, the camera control unit 101 acquires two source images (first raw images). One of the source images is a non-flashed image (second image data acquired in a second state when the flash device 300 does not fire a flash), which can be acquired using a user-manually set exposure control value or alternatively, an AE image, etc. At this time, the lens control unit 201 performs AF by driving the focusing lens (not shown) in the lens 202 according to the command from the camera control unit 101 via the mounting contact group 103.

[0043] Another source image is a flashed image (first image data acquired in a first state when the flash device 300 fires a flash) using the aforementioned flash device 300. Except for the presence or absence of flash firing, the flashed image has the same conditions as the non-flashed image (e.g., exposure control values ​​such as the position of the focusing lens (not shown) in lens 202, shutter speed, aperture value, and ISO sensitivity). This is because, during the compositing process described below, it may cause brightness shifts, color shifts, or image resolution degradation. Therefore, the first original image includes the first image data acquired in the first state where the flash device 300 fires a flash and the second image data acquired in the second state where the flash device 300 does not fire a flash.

[0044] In step S207, the camera control unit 101 uses the first raw image stored in the internal memory of the camera control unit 101 in step S206 to perform a compositing process for displaying a flash simulation of the image obtained when the flash device 300 fires a flash. The compositing process will be described later.

[0045] In step S208, the camera control unit 101 applies a γ curve (gamma curve) to the third original image synthesized in step S207 and displays the result on the camera display unit 106. The camera control unit 101 may also apply the γ curve to a fourth original image obtained by encoding (lossless compression) the third original image and display the result on the camera display unit 106. Alternatively, the camera control unit 101 may display an image obtained by developing either the third or fourth original image. Developing the synthesized result of step S207, rather than developing the individual source images obtained in step S206, reduces the processing load and the total processing time in the flash simulation.

[0046] In step S209, the user operates the camera operation unit 105 to confirm whether the flash simulation result displayed on the camera display unit 106 is the expected result. If the expected result is obtained, the process proceeds to step S211. On the other hand, if the result is different from the expected result and needs to be changed, the process proceeds to step S210.

[0047] In step S210, the user operates the camera operation unit 105 to change the settings of the flash simulation results displayed on the camera display unit 106. These settings are primarily for flash adjustment (brightness correction) and affect the adjustment of the amount of light emitted by the flash device 300. Upon completion of the settings, the process proceeds to step S207, where the compositing process is repeated for repeated fine-tuning until the user achieves the desired result.

[0048] In step S211, the camera control unit 101 calculates the amount of light emitted by the flash device 300 for actual imaging based on the synthesis processing result determined in step S209.

[0049] In step S212, the camera control unit 101 monitors the status of the SW2 signal and determines whether SW2 is on. If SW2 is on, the process proceeds to step S214. On the other hand, if SW2 is off, the process proceeds to step S213, and as long as SW1 remains on, the camera control unit 101 repeats monitoring step S212 until SW2 is on. If SW1 is off in step S213, the process proceeds to step S201.

[0050] In step S214, the camera control unit 101 sends a light emission command to the lighting control unit 301 via the lighting contact group 109 of the flash device 300, thereby emitting a flash with the light emission amount calculated in step S211 for actual video recording. In step S221, the lighting control unit 301 controls the light emitter 302 to emit a flash for actual video recording based on the light emission command from the camera control unit 101 in step S214.

[0051] In step S215, the camera control unit 101 acquires the light emission information related to the flash device 300 from step S221 via the illumination contact group 109, and drives the image sensor 102 to perform image processing and expose the light from the lens device 200. During image processing, the exposure control value or the amount of light emitted by the light emitter 302 is controlled so that the captured image has the user-desired exposure as determined in step S209. The camera operation unit 105 can be configured to allow separate settings for acquiring the first raw image during the compositing process and for acquiring the first raw image during actual image capture. The camera control unit 101 stores the first raw image for actual image capture acquired by driving the image sensor 102 in its internal memory. After image processing is completed, the process proceeds to step S216.

[0052] In step S216, the camera control unit 101 applies the γ curve to the first raw image acquired from the image sensor 102 in step S215 and displays the result on the camera display unit 106. Alternatively, the camera control unit 101 may apply the γ curve to a fifth raw image obtained by encoding (lossless compression) the first raw image and display the result on the camera display unit 106. The camera control unit 101 may display an image obtained by developing either the first raw image or the fifth raw image.

[0053] In step S217, the camera control unit 101 saves the image developed in step S216 to memory 107. At this time, information related to the camera body 100, lens device 200, and flash device 300 is embedded in the image. In addition to the developed image for actual imaging, the first to fifth original images (including source images generated in the flash simulation mode sequence) and the displayed image can also be stored in memory 107 (the second original image will be described later). Furthermore, to reduce the use of internal memory in the camera control unit 101, in steps prior to S217, once each processing step is completed, the first to fifth original images and the displayed image can be re-stored from the internal memory of the camera control unit 101 to memory 107. After storing each image, the series of imaging operations ends.

[0054] Now refer to Figures 3 to 5 To describe the synthesis process in step S207. Figure 3 This is a flowchart illustrating an example of the synthesis process in step S207. Figure 4 This is a conceptual diagram showing the first to third original images. Figure 5 The compositing calculation is conceptually illustrated. After storing the first original image of the source image in the internal memory of the camera control unit 101 in step S206, the camera control unit 101 begins... Figure 3 The flowchart.

[0055] In step S300, the camera control unit 101 detects the offset added to the first original image of the source image acquired in step S206. Now refer to... Figure 4 Used to describe the offset.

[0056] The first raw image obtained from the image sensor 102 as the model can be represented as follows: Figure 4 As shown on the left. The offset shown here corresponds to information related to the imaging conditions and is added to the data of the first original image. The offset is, for example, a value that changes for each image sensor 102 or a value that changes according to settings such as ISO sensitivity (imaging settings) (information related to the image sensor and imaging settings). Therefore, if multiple images are directly synthesized using the first original image, a deviation in offset will occur relative to the desired RGB values, and the image output after synthesis will suffer image quality degradation such as brightness shift or color shift.

[0057] Therefore, this embodiment uses a second original image obtained by temporarily removing the offset from the first original image. Figure 4 The image on the right is used for compositing calculations, and after compositing, the removed offset is added back to generate an image that serves as the third original image. Therefore, compared to not removing the offset, brightness or color shifts can be suppressed during compositing.

[0058] The flowchart follows the above process starting from step S301. In step S301, the camera control unit 101 subtracts the offset detected in step S300 from the first original image of each source image to generate a second original image with the offset removed, and stores it in the internal memory of the camera control unit 101.

[0059] In step S302, the camera control unit 101 determines whether the color data of the second original image of the stored source image falls within a predetermined threshold range (predetermined range). This is to prevent color misalignment, because the RGB relationship is disrupted when an image with overexposure or crushed shadow, or an image with similar exposure, is used as a source image for compositing.

[0060] If the color data falls outside a predetermined threshold range, the camera control unit 101 sets a warning sign and displays a warning (warning processing) such as zebra stripe display or highlight display on the relevant image area when the image is displayed on the camera display unit 106. In step S302, a judgment is made on the second original image of each source image, but this embodiment is not limited to this example. The warning sign can be inherited along with the warning range information for any of the first to fifth original images, or it can be made each time at a time not shown for the first to fifth original images or the image displayed on the camera display unit 106. A predetermined subject detection can be performed, and the warning judgment processing can be changed according to the detection result of the main subject. For example, if a person is detected as the main subject, the warning can be set only for the area around the face, or in the case of a landscape, the warning range can be set to cover the entire sky. Methods using known techniques such as face detection, pupil detection, moving subject detection, pattern matching, and distance mapping can be used to detect the predetermined main subject.

[0061] Therefore, if at least one of the first, second, third, fourth, and fifth original images contains data outside a predetermined range, the camera control unit 101 may perform a warning. Alternatively, if at least one of the first to fifth original images contains data falling outside the predetermined range, the camera control unit 101 may perform a correction.

[0062] In step S303, the camera control unit 101 performs a composite calculation using the second original images of each source image and stores the result in the internal memory of the camera control unit 101.

[0063] Now refer to Figure 5The synthesis calculation is described below. In this embodiment, the flash is extracted by subtracting the non-flash image (fourth image data) from the flashed image (third image data). Next, a gain corresponding to the necessary flash adjustment amount is applied based on the amount of light emitted when the flashed image was captured (gain processing is performed). This reproduces what would happen if the amount of light emitted were actually changed by flash adjustment. Then, the adjusted flash component is added to the non-flash image, and the result of actual shooting can be simulated.

[0064] Therefore, in this embodiment, the second original image includes third image data generated by subtracting an offset from the first image data and fourth image data generated by subtracting an offset from the second image data. The camera control unit 101 uses the second image data and data obtained by gain processing the difference between the third and fourth image data for compositing. The camera control unit 101 can perform gain processing during compositing based on the emission levels of each individual flash device or group of flash devices for the flash device 300.

[0065] Although this embodiment uses a single flash device 300 in the method, the method can acquire, in step S206, a number of flashed images (or groups of flash devices) and a non-flashed image as source images. This embodiment allows the user of the flash device 300 to find the appropriate light output setting without having to repeat the actual shooting each time.

[0066] Unless the user has pre-configured it, the default flash simulation settings can be settings that cause appropriate exposure of the main subject after compositing calculations. In other words, in a camera system using multiple flash devices 300, the default settings can be set such that the amount of light emitted is adjusted according to the number of flash devices and how the flashes from each flash device 300 illuminate the main subject.

[0067] In this embodiment, the camera control unit 101 can acquire a first raw image by issuing a light emission command suitable for the amount of light emitted by the subject. The camera control unit 101 can instruct the flash device 300 to emit light to prevent at least one of the first to fifth raw images from falling outside a predetermined range, perform gain processing corresponding to the appropriate amount of light for the subject during the compositing process, and generate a third raw image.

[0068] In step S304, the camera control unit 101 determines whether the color data of the received synthetically calculated second original image falls outside a predetermined threshold range. If this determination has already been performed in step S302, it can be omitted. If a warning flag is set, the camera control unit 101 can issue a warning at the camera display unit 106 based on the warning flag.

[0069] In step S305, the camera control unit 101 performs data correction processing on the second original image that has already been received for compositing. This is because the RGB relationship is disrupted when an image with overexposure or missing shadows, or an image with similar exposure, is used as the source image for compositing. For example, if the source image is in a state close to overexposure, the composite image will appear pinkish.

[0070] Therefore, in the case of overexposure, data supplementation can be performed to make the relevant area appear white, or a blurring process such as a white gradient can be performed. Alternatively, based on the predetermined subject detection result described above, the user can choose to issue a warning in step S302 or S304 if the supplementation range includes a person, or to perform white supplementation in step S305 only for the background sky.

[0071] In step S306, the camera control unit 101 adds the offset removed in step S301 to the second original image that has already been received for compositing, generating a third original image with the offset added, and stores it in the internal memory of the camera control unit 101 or in memory 107. Then, Figure 3 The flowchart ends, and the process proceeds to step S208.

[0072] Therefore, this embodiment uses linear raw image data that has not yet received the γ curve for compositing, suppressing brightness shifts, color shifts, or image quality degradation that may occur during compositing, and obtaining a compositing result close to the actual imaging result. This embodiment stores the first raw image in the internal memory of the camera control unit 101 and performs... Figure 3 The compositing process can be varied, but not limited to this example. Instead of storing the source image as a first raw image in the internal memory of the camera control unit 101, the source image can be encoded into a fifth raw image and then stored. In this case, it is possible to... Figure 3 Before the compositing process, the fifth original image is decoded and restored to the state of the first original image, and then the compositing can proceed. Figure 3 The synthesis process requires encoding and decoding, but this can reduce the internal memory capacity of the camera control unit 101.

[0073] This embodiment performs various processes associated with the imaging operations of the camera body 100, lens device 200, and flash device 300. This embodiment uses first to third original images of the source material for compositing, suppressing brightness shifts, color shifts, or image quality degradation that may occur during compositing, and obtaining a compositing result that closely approximates the actual shooting result.

[0074] The flowcharts described in this embodiment are merely illustrative, and various processes can be performed in a different order than those described in this embodiment. Second Embodiment

[0075] Now refer to Figure 6 The operation (camera operation) of the camera system 10a according to the second embodiment of the present disclosure will be described. Figure 6 This is a block diagram of the camera system 10a. The camera system 10a includes a camera body (camera device) 100, a lens device 200, a flash device (lighting device) 300, and a transmitter (transmission device) 400.

[0076] Figure 6 The following state is shown, wherein the illumination contact group 109 of the camera body 100 is removed. Figure 1 The flashing device 300, and instead the transmitter 400 is attached to the lighting contact group 109. Figure 6 The following configuration is also shown, in which the transmitter wireless communication unit 402 of the transmitter 400 and the illumination wireless communication unit 306 of each flash device 300 are wirelessly connected. In this embodiment, multiple flash devices 300 are wirelessly connected to the transmitter 400 and can be individually controlled by changing the wireless communication group settings.

[0077] The difference between this embodiment and the first embodiment is that the flash device 300 is not physically connected to the camera body 100 and constitutes the camera system as an independent lighting device. Another difference is that the camera system includes multiple flash devices 300 via wireless communication. This embodiment will discuss an example of a transmitter 400 wirelessly connected to multiple flash devices 300; however, instead of using the transmitter 400, the camera wireless communication unit 108 of the camera body 100 can be used to wirelessly connect to the multiple flash devices 300.

[0078] exist Figure 6 In this embodiment, the camera body 100, lens device 200, and flash device 300, except for the transmitter 400, are the same as those in the first embodiment, and therefore their description will be omitted.

[0079] The transmitter control unit 401 is a microcomputer that controls the operation of the various components in the transmitter 400. The transmitter control unit 401 can communicate with the camera control unit 101 via the illumination contact group 109, and can receive light emission control commands and camera information from the camera body 100 to the flash device 300, and can transmit flash device information.

[0080] Similar to the camera wireless communication unit 108 and the illumination wireless communication unit 306, the transmitter wireless communication unit 402 performs wireless communication between the transmitter 400 and an external device (camera body 100 or flash device 300). The transmitter wireless communication unit 402 transmits and receives various settings and operation commands, such as light intensity and illumination angle. The transmitter wireless communication unit 402 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, or a wireless USB module.

[0081] The transmitter operation unit 403 has user-operable operating components that detect operations performed by the user via buttons, dials, etc., attached to the transmitter 400, and send signals corresponding to the operation commands to the transmitter control unit 401. The transmitter display unit 404 displays the illumination mode and other information according to instructions from the transmitter control unit 401. The transmitter 400 does not have a power supply unit such as a battery, but is configured to be powered by receiving power from the camera body 100 via the illumination contact group 109.

[0082] Now refer to Figure 7 To describe the operation of the camera system 10a (camera operation). Figure 7 This is a flowchart illustrating an example of video recording operation using a camera body 100 and a flash device 300. Figure 7 In the middle, omission and Figure 2 The description of the parts corresponding to the operations in steps S200 to S221.

[0083] In step S701, the camera control unit 101 monitors the status of the SW1 signal and determines whether SW1 is on. If SW1 is on, the process proceeds to step S702. In step S702, the camera control unit 101 notifies the flash device 300 of camera settings via wireless communication from the transmitter 400 to the flash device 300 via the illumination wireless communication unit 306 and the transmitter wireless communication unit 402. In step S718, the illumination control unit 301 of each of the multiple flash devices 300 wirelessly acquires various information from the transmitter 400 via the illumination wireless communication unit 306 and the transmitter wireless communication unit 402. The acquired information includes the maximum light output of the flash device 300, the illumination angle range and battery status, the number of wirelessly connected flash devices 300, the light preparation status, the light group, ID, channel settings, etc.

[0084] In step S703, the camera control unit 101 sends a light emission command via the transmitter 400 to the illumination control unit 301 of each of the plurality of flash devices 300 to perform a pre-emission for acquiring the source image. In step S719, the illumination control unit 301 of each of the plurality of flash devices 300 controls the light emitter 302 based on the light emission command from the camera control unit 101 in step S703, and performs a pre-emission to acquire the source image to be used for flash simulation compositing processing.

[0085] In step S704, the camera control unit 101 acquires pre-flash information for each of the multiple flash devices 300 from step S719 via the transmitter 400. The camera control unit 101 also drives the image sensor 102 to expose it using the pre-flash received from the lens device 200, and calculates the amount of light emitted by the light emitter 302 required in the next step S706.

[0086] In step S705, the camera control unit 101 sends a light emission command via the transmitter 400 to the illumination control unit 301 of each of the plurality of flash devices 300, so that the light emission amount calculated in step S704 is used to acquire the source image. In step S720, the illumination control unit 301 of each of the plurality of flash devices 300 controls the light emitter 302 based on the light emission command from the camera control unit 101 in step S705, and emits light to acquire the source image to be used in the flash simulation compositing process.

[0087] In step S706, the camera control unit 101 acquires the light emission information related to each of the multiple flash devices 300 in step S720 via the transmitter 400, and drives the image sensor 102 to expose it using light from the lens device 200. The camera control unit 101 stores the first raw image acquired by driving the image sensor 102 in its internal memory.

[0088] In steps S706 and S720, adjustments can be made to suppress overexposure in the flashed image of the source image, obtain a source image that is easy to synthesize in step S707, and prevent the re-acquisition of the source image as described below. For example, adjustments can be made to control the light emission of each flash device 300 for slight underexposure and to acquire the source image. Furthermore, exposure control for the flash simulation mode can be performed, for example, by increasing the shutter speed to suppress subject blur and decreasing the ISO sensitivity to suppress noise.

[0089] In this embodiment, in step S706, in addition to non-flash images, the camera control unit 101 also acquires a number of source images equal to the number of wireless groups of flash devices 300 wirelessly connected to the transmitter 400. When illumination control is performed on each individual flash device 300 instead of each wireless group, a number of source images equal to the number of wirelessly connected flash devices 300 are acquired in addition to non-flash images.

[0090] In step S709, the user operates the camera operation unit 105 to check whether the flash simulation result displayed on the camera display unit 106 is the expected result. If the expected result is obtained, the process proceeds to step S711. On the other hand, if the result is different from the expected result and needs to be changed, the process proceeds to step S722.

[0091] In step S722, the user operates the camera operation unit 105 to select whether to reacquire the source image. If reacquiring the source image is desired, the process proceeds to step S704. On the other hand, if the source image is not reacquired and only the settings are changed, the process proceeds to step S710. If the source image acquired in step S706 includes an image that falls outside a predetermined threshold range (such as overexposure or missing shadows), automatic judgment processing can be performed to automatically reacquire the source image. When reacquiring the source image, even if each flash device 300 emits flash at a light intensity close to that used before reacquisition, the user's expected result may not be achieved. Therefore, for example, if overexposure occurred before reacquisition, the light intensity of the flash device 300 used for that source image can be reduced by two stops; if missing shadows occurred before reacquisition, the light intensity of the flash device 300 used for that source image can be increased by two stops. Therefore, changes can be made so that the reacquired source image falls within the predetermined threshold range.

[0092] In step S710, the user operates the camera operation unit 105 to change the settings of the flash simulation results displayed on the camera display unit 106. The settings are changed for all flash devices 300 wirelessly connected via the transmitter 400, and can be changed for the specific flash device 300 or wireless group to which the user's desired result is desired. Upon completion of the settings, the process proceeds to step S707, where the compositing process is repeated for repeated fine-tuning until the user achieves the desired result.

[0093] In step S711, the camera control unit 101 calculates the amount of light emitted by each flash device 300 for actual imaging based on the composite processing result determined in step S709.

[0094] In step S714, the camera control unit 101 sends a light emission command via the transmitter 400 to the illumination control unit 301 of each of the plurality of flash devices 300, so that the light emission amount calculated in step S711 is emitted for actual imaging. In step S721, the illumination control unit 301 of each of the plurality of flash devices 300 controls the light emitter 302 based on the light emission command from the camera control unit 101 in step S714, and emits a flash for actual imaging.

[0095] In step S715, the camera control unit 101 acquires the light emission information related to each of the multiple flash devices 300 obtained in step S721 via the transmitter 400, and drives the image sensor 102 to perform image processing and expose it using light incident from the lens device 200. During image processing, the exposure control value or the amount of light emitted by the light emitter 302 is controlled so that the captured image has the exposure desired by the user as determined in step S709. The camera control unit 101 stores the first raw image of the actual captured image obtained by driving the image sensor 102 in the internal memory of the camera control unit 101. After the image processing is completed, the process proceeds to step S716.

[0096] Finally, refer to Figure 8 This describes the synthesis calculation in the case of multiple flash devices 300. Figure 8 The synthesis calculation according to this embodiment is conceptually illustrated. This embodiment differs from the first embodiment in that: flashes are extracted from each of the multiple wirelessly connected flash devices 300, and a gain corresponding to the desired flash adjustment amount is applied based on the amount of light emitted when each flashed image is captured. The actual imaging result can then be simulated by adding the adjusted flash components to the non-flashed image. In step S206, this method acquires a non-flashed image and flashed images for each flash device 300 (or for each wireless group to be emitted) as source images. This method allows the user of the flash device 300 to find the appropriate light emission setting without having to repeat the actual imaging process each time.

[0097] As described above, this embodiment performs various processes associated with the imaging operations of the camera body 100, lens device 200, flash device 300, and transmitter 400. In multi-light imaging using multiple flash devices, this embodiment uses first to third original images of the source images for compositing processing, suppressing brightness shifts, color shifts, or image quality degradation that may occur during compositing processing, and obtaining a compositing result that is close to the imaging result obtained in actual imaging. The flowcharts described in this embodiment are merely illustrative, and various processes may be performed in a different order than those described in this embodiment. Other embodiments

[0098] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0099] While exemplary embodiments have been described in this disclosure, it should be understood that this disclosure is not limited to the exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

[0100] For example, instead of the camera body 100 as the video recording device, a smartphone with camera functionality that can wirelessly connect to the flash device 300 and a head-mounted display with a built-in camera can be used. Instead of the flash device 300, other lighting devices such as LED lights or organic EL lights can be used.

[0101] According to various embodiments of this disclosure, a camera device can be provided that can suppress image quality degradation that occurs during image synthesis processing.

Claims

1. A camera device capable of communicating with a lighting device, the camera device comprising: An image sensor is configured to acquire a first raw image in each of a first state under illumination by the lighting device and a second state without illumination by the lighting device. as well as The processing unit is configured as follows: A second original image is generated by removing information related to the shooting conditions from the first original image. The second original image is used for synthesis processing, and A third original image is generated by adding the information to the second original image that has already received the synthesis process.

2. The camera device according to claim 1, characterized in that, Information related to the camera conditions includes information related to the image sensor and camera settings.

3. The camera device according to claim 1, characterized in that, The first original image includes first image data acquired under the illumination of the lighting device and second image data acquired without the illumination of the lighting device. The second original image includes third image data generated by removing the information from the first image data and fourth image data generated by removing the information from the second image data. The processing unit is configured to perform the synthesis process using the second image data and data obtained by applying gain processing to the difference between the third image data and the fourth image data.

4. The camera device according to claim 1 further includes a memory configured to store the third original image.

5. The camera device according to claim 4, further comprising a memory configured to store at least one of the first original image, the second original image, a fourth original image obtained by losslessly compressing the third original image, and a fifth original image obtained by losslessly compressing the first original image.

6. The camera device according to claim 5, further comprising a display unit configured to display an image obtained by applying a γ curve to the third original image or the fourth original image.

7. The camera device according to claim 5, characterized in that, The processing unit is configured to perform a warning if at least one of the first original image, the second original image, the third original image, the fourth original image, and the fifth original image includes data that falls outside a predetermined range.

8. The camera device according to claim 5, characterized in that, The processing unit is configured to perform correction processing if at least one of the first original image, the second original image, the third original image, the fourth original image, and the fifth original image includes data that falls outside a predetermined range.

9. The camera device according to claim 5, characterized in that, The processing unit is configured as follows: Indicate the amount of light emitted by the lighting device such that at least one of the first original image, the second original image, the third original image, the fourth original image, and the fifth original image does not fall outside a predetermined range, and During the compositing process, gain processing is performed based on the appropriate amount of light for the subject to generate the third original image.

10. The camera device according to claim 1, further comprising a setting unit configured to separately set settings for acquiring the first original image during the compositing process and settings for acquiring the first original image during actual recording.

11. The camera device according to claim 1, characterized in that, The processing unit is configured to perform gain processing based on the luminous intensity of each lighting device or group of lighting devices during the synthesis process.

12. The camera device according to claim 1, characterized in that, The processing unit is configured as follows: Instructions to emit light in an appropriate amount for the subject, and The image sensor is used to acquire the first raw image.

13. The camera device according to claim 1, characterized in that, If the first original image acquired in the first state includes data falling outside a predetermined range, or if the first original image is an image different from what the user expected, the processing unit is configured to reacquire the first original image.

14. The camera device according to claim 1, characterized in that, The processing unit is configured to set settings for actual video recording based on settings used when acquiring the first original image for the synthetic processing.

15. The camera device according to any one of claims 1 to 14, characterized in that, The processing unit is configured to change the warning judgment processing based on the subject detection result.

16. A camera system, comprising: Lighting equipment; as well as The camera device according to any one of claims 1 to 15.

17. A control method for a camera device, the camera device being able to communicate with a lighting device, the control method comprising: Using an image sensor, a first raw image is acquired in each of the states: a first state under illumination by the lighting device and a second state without illumination by the lighting device. A second original image is generated by removing information related to the shooting conditions from the first original image. The second original image is used for synthesis processing, and A third original image is generated by adding the information to the second original image that has already received the synthesis process.

18. A non-transitory computer-readable storage medium storing a program that causes a computer to perform the control method according to claim 17.

19. A computer program product comprising a program that causes a computer to perform the control method according to claim 17.

Images