Imaging apparatus

Abstract

To provide an imaging apparatus which obtains an imaging result appropriate to various scenes by expanding an AUTO function to allow a user to click a shutter without care about setting.SOLUTION: In the imaging apparatus, an imaging unit comprises: a first processing unit which generates developing data based on an imaging signal output by an image sensor; and a second processing unit which includes an application processor for processing an application. The second processing unit analyzes the imaging signal output by the image sensor, and the first processing unit performs development based on a result of the analysis of the imaging signal.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] FIELD An embodiment of the present invention relates to an imaging device. [Background technology]

[0002] An imaging device is known that extends the conventional auto function and, when a specified operation is performed, displays multiple sample images corresponding to the shooting scene, and then switches between canceling the display of the sample images or continuing to display them depending on the degree of scene change (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-075631 Summary of the Invention

[0004] One aspect of the present invention is an imaging device comprising a first processing unit that generates development data based on an imaging signal output by an image sensor, and a second processing unit that includes an application processor that processes applications, wherein the second processing unit analyzes the imaging signal output by the image sensor, and the first processing unit develops the imaging signal based on the analysis results. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of an imaging device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an outline of the operation of the imaging device according to the present embodiment. [Figure 3] FIG. 2 is a partial block diagram showing an example of a camera body of the imaging device according to the present embodiment. [Figure 4] FIG. 10 is a diagram showing an example of scene determination information. [Figure 5] FIG. 10 is a diagram illustrating an example of scene recognition information. [Figure 6] 5 is a flowchart illustrating an example of the operation of the imaging device according to the present embodiment. [Figure 7] 10A and 10B are diagrams illustrating an outline of the operation of an imaging device according to a modified example of an embodiment. [Figure 8] FIG. 10 is a partial block diagram showing an example of a camera body of an imaging device according to a modified example of the embodiment. [Figure 9] 10 is a flowchart illustrating an example of the operation of an imaging device according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0006] (Embodiment) The imaging device according to this embodiment will be described below with reference to the drawings. Fig. 1 is a schematic diagram showing an example of the imaging device according to this embodiment. The imaging device 100 comprises a camera body 1 and a lens unit 2. The lens unit 2 is attached to the camera body 1. The camera body 1 comprises a release button 3. The release button 3 is also called a shutter button, and is the button that is pressed to release the shutter of the camera. The lens unit 2 includes an optical system arranged along an optical axis OA within its barrel. The optical system guides incident subject light beams to the camera body 1.

[0007] The imaging device 100 extends the conventional AUTO function, and allows the user to release the shutter without having to worry about settings, thereby obtaining photographic results suitable for various scenes. Fig. 2 is a diagram showing an overview of the operation of the imaging device according to this embodiment. As shown in Fig. 2, the imaging device 100 acquires an image and determines a scene based on the acquired image. Specifically, the imaging device 100 determines what the shooting scene is. For example, the imaging device 100 determines whether the shooting scene is a portrait, a landscape, or a sport. The imaging device 100 performs settings appropriate for the scene based on the determination result of the shooting scene. For example, the imaging device 100 sets the aperture, shutter speed, ISO sensitivity, etc. The imaging device 100 updates the preview image. The imaging device 100 captures (takes an image) when the user releases the shutter.

[0008] 3 is a partial block diagram showing an example of a camera body of an imaging device according to this embodiment. Camera body 1 includes, for example, an imaging unit 16. Imaging unit 16 includes an image sensor 13, a first processing unit 11, and a second processing unit 12. First processing unit 11 is connected to second processing unit 12. The image sensor 13 is configured using, for example, a CMOS (Complementary Metal Oxide Semiconductor). The subject light beam guided by the optical system to the camera body 1 is guided to the imaging unit 16. In the imaging unit 16, the image sensor 13 photoelectrically converts the incident light to generate an imaging signal. The image sensor 13 transmits the generated imaging signal to the first processing unit 11.

[0009] The first processing unit 11 generates developed data based on the imaging signal output by the image sensor 13. The first processing unit 11 outputs the generated developed data to the second processing unit 12. The second processing unit 12 includes an application processor that processes applications. The second processing unit 12 acquires the developed data output by the first processing unit 11 and analyzes the acquired developed data. The second processing unit 12 outputs the analysis results of the developed data to the first processing unit 11. The first processing unit 11 acquires the analysis result of the development data output by the second processing unit 12, and performs development based on the acquired analysis result of the development data.

[0010] The first processing unit 11 includes a development unit 11-1, a scene determination unit 11-2, and a shooting / development setting unit 11-3. The second processing unit 12 includes an image conversion unit 12-1, a scene recognition unit 12-2, a scene processing unit 12-3, and an image recording unit 12-4. The first processing unit 11 receives the imaging signal transmitted from the image sensor 13. The first processing unit 11 develops the received imaging signal and acquires developed data. In the first processing unit 11, the developed data is output to the developing section 11-1.

[0011] An example of developed data generated by the development unit 11-1 is YUV. YUV expresses an image using luminance information Y and two pieces of color difference information (saturation information) U and V. Below, we will continue to explain the case where the developed data is "YUV420 PLANAR."

[0012] In the first processing unit 11, the developed data generated by the development section 11-1 is output to the scene determination section 11-2 and the second processing unit 12. The scene determination unit 11-2 acquires the developed data output by the development unit 11-1. Based on the acquired developed data, the scene determination unit 11-2 determines the scene represented in the developed data. The scene determination unit 11-2 has scene determination information. The scene determination unit 11-2 acquires information on the focal length, shooting distance, and average brightness value input to the first processing unit 11.

[0013] 4 is a diagram showing an example of scene determination information, which is information in a table format having scene names. Examples of scene names are portrait, night portrait, close-up (macro), and landscape. The scene determination unit 11-2 determines whether the acquired developed data is a portrait, a night portrait, a close-up, or a landscape. Returning to FIG.

[0014] In the second processing unit 12, the image conversion unit 12-1 acquires the developed data generated by the development unit 11-1. When acquiring the developed data, the image conversion unit 12-1 uses communication via MIPI-CSI (MIPI Camera Serial Interface). By using such high-speed communication, when the second processing unit 12 performs scene recognition and preview display based on the acquired developed data, it becomes possible to transmit the image captured by the image sensor 13 to the second processing unit with low latency and process it. The image conversion unit 12-1 converts the acquired developed data into an image format that is easy to handle for the scene recognition unit 12-2 of the second processing unit. The conversion is not limited to data format but also includes changing the image resolution and cropping. The scene recognition unit 12-2 acquires the scene determination result, information specifying the focal length, information specifying the shooting distance, and average luminance value from the scene determination unit 11-2. The scene recognition unit 12-2 acquires the result of converting the developed data from the image conversion unit 12-1. The scene recognition unit 12-2 has scene recognition information. The scene recognition unit 12-2 uses communication via USB (Universal Serial Bus) when acquiring information from the scene determination unit 11-2. In this way, by separating the communication method from the MIPI-CSI used to acquire the developed data by the image conversion unit 12-1, the first processing unit and the second processing unit can be loosely coupled, making it possible to achieve a system configuration in which the processing load is distributed.

[0015] FIG. 5 is a diagram showing an example of scene recognition information. Examples of scene names include portrait, night portrait, close-up (macro), landscape, pet, night view, sunset, cooking (food), children's snapshot, candlelight, sea, snow, cherry blossom, autumn leaves, illumination, fireworks, and other (none applicable). Returning to Figure 3, we will continue the explanation.

[0016] The scene recognition unit 12-2 identifies the scene represented in the developed data based on the scene determination result acquired from the scene determination unit 11-2, information specifying the focal length, information specifying the shooting distance, the average luminance value, and the developed data acquired from the image conversion unit 12-1. When identifying the scene, the scene recognition unit 12-2 may use pattern matching or image recognition using deep learning. Specifically, based on the acquired scene determination result, information specifying the focal length, information specifying the shooting distance, average brightness value, and the result of image conversion of the developed data, the scene recognition unit 12-2 determines whether the image represented in the developed data is a scene other than the scene determination result acquired from the scene determination unit 11-2, such as a pet, a night view, a sunset, cooking (food), a child's snapshot, a candle, the sea, snow, cherry blossoms, autumn leaves, illuminations, fireworks, or other (none). Here, scene recognition section 12-2 may again determine whether the scene included in the scene determination result acquired from scene determination section 11-2 is a portrait, a night scene portrait, a close-up (macro), or a landscape.

[0017] The scene processing unit 12-3 acquires the scene determination result from the scene recognition unit 12-2. Based on the acquired scene determination result, the scene processing unit 12-3 determines the shooting and development processing conditions to be applied by the first processing unit 11. The shooting and development processing conditions refer to the conditions necessary for shooting (capturing images) and development, including the aperture value, shutter speed, ISO sensitivity, AWB setting, AF setting, and picture control, that are suitable for the identified scene. The scene processing unit 12-3 may correct the exposure conditions depending on the scene determination result obtained from the scene recognition unit 12-2. For example, if the scene determination result obtained from the scene recognition unit 12-2 identifies a "sunset," the exposure may be corrected by -1 step, and the shooting and development conditions may be changed to obtain a darker shooting result. When an external flash is connected, an external flash operation signal is input to the scene processing unit 12-3. When the external flash operation signal is input to the scene processing unit 12-3, the scene processing unit 12-3 determines shooting and development processing conditions suitable for use with the external flash. The scene processing unit 12-3 outputs the determined shooting and development processing conditions to the first processing unit 11. The scene processing unit 12-3 may output the identified scene instead of the shooting and development processing conditions to the shooting and development setting unit 11-3 of the first processing unit 11. In this case, the shooting and development setting unit 11-3 of the first processing unit 11 determines the shooting and development processing conditions based on the acquired scene information. The scene processing unit 12-3 is connected to receive a release signal output from the release button 3 in FIG. 1. The release button 3 in FIG. 1 is also connected to the development unit 11-1 of the first processing unit 11, and its operation changes depending on whether priority is given to shooting response or scene recognition. For example, when priority is given to shooting response, the release signal from the first processing unit 11 takes priority, resulting in less delay before capturing an image. On the other hand, when priority is given to scene recognition, the release signal from the second processing unit 12 takes priority. Therefore, the scene processing unit 12-3 outputs the shooting and development processing conditions at the time the release signal was input to the shooting and development setting unit 11-3 of the first processing unit 11, and then instructs the development unit 11-1 to capture an image via the shooting and development setting unit 11-3 of the first processing unit 11. Therefore, it takes a slightly longer time to capture an image than when priority is given to shooting response. However, since the development and imaging settings are appropriate for the scene specified immediately before the release, it is easier to obtain imaging and development results that are appropriate for the scene than when priority is given to shooting response.Which of the release signals input to the development section 11-1 of the first processing unit 11 and the scene processing section 12-3 of the second processing unit 12 is given priority is determined in advance by the mode dial or settings of the camera.Alternatively, the camera may automatically determine this based on the exposure, settings, etc.

[0018] In the first processing unit 11, the shooting and development setting section 11-3 acquires the shooting and development conditions output by the scene processing section 12-3. The shooting and development setting section 11-3 sets the shooting conditions and development processing conditions included in the acquired shooting and development shooting processing conditions in the development section 11-1. A release button is connected to the developing unit 11-1. When the release button of the first processing unit 11 has priority, when a signal instructing capture of an image is input from the release button, the developing unit 11-1 drives the image sensor 13 based on the capture conditions set by the capture / development setting unit 11-3, and generates developed data from the image signal acquired from the image sensor 13 based on the development processing conditions set by the capture / development setting unit 11-3. Here, a case will be described in which the developed data generated by the developing unit 11-1 is JPEG. The developing unit 11-1 outputs the generated developed data to the second processing unit 12. In the second processing unit 12, the image recording unit 12-4 acquires the developed data output by the development unit 11-1 of the first processing unit 11. The image recording unit 12-4 uses USB communication when acquiring the developed data. This allows MIPI-CSI to communicate only the developed data used for scene recognition and preview display. Therefore, after capturing an image, operations can be performed immediately to prepare for the next capture. Specifically, scene recognition can be started immediately after capturing an image, and there is no need to reduce the frequency of updating the preview display. The image recording unit 12-4 converts and outputs (records) the acquired developed data based on an image format preset by the user. The image format can be set to specify an image file format such as JPEG, as well as the image resolution and compression rate.

[0019] (Operation of imaging device) 6 is a flowchart showing an example of the operation of the imaging device according to this embodiment. The operations of the first processing unit 11 and the second processing unit 12 of the imaging device 100 will be mainly described with reference to FIG. (Step S1-1) In the first processing unit 11, the developing section 11-1 acquires an imaging signal. (Step S2-1) In the first processing unit 11, the developing section 11-1 processes the acquired imaging signal to generate developed data, an example of which is YUV. (Step S3-1) In the first processing unit 11, the developed data generated by the developing section 11-1 is output to the second processing unit 12. (Step S4-1) In the second processing unit 12, the image conversion unit 12-1 acquires the developed data generated by the development unit 11-1. The image conversion unit 12-1 converts the acquired developed data into an image format that is easy to handle in the second processing unit 12. The conversion is not limited to the data format, but also includes changing the image resolution and cropping.

[0020] (Step S5-1) In the first processing unit 11, the scene determination unit 11-2 acquires the developed data generated by the development unit 11-1. Based on the acquired developed data, the scene determination unit 11-2 determines the scene of the image represented in the developed data. The scene determination unit 11-2 acquires the scene determination result and information input to the first processing unit 11, including the focal length, shooting distance, and average luminance value. (Step S6-1) In the second processing unit 12, the scene recognition section 12-2 acquires the scene determination result, information specifying the focal length, information specifying the shooting distance, and an average luminance value from the scene determination section 11-2. (Step S7-1) In the second processing unit 12, the scene recognition section 12-2 acquires the result of image conversion of the developed data from the image conversion section 12-1. The scene recognition section 12-2 identifies the scene of the image represented in the developed data based on the acquired scene determination result, information specifying the focal length, information specifying the shooting distance, the average luminance value, and the result of image conversion of the developed data.

[0021] (Step S8-1) In the second processing unit 12, the scene processing unit 12-3 acquires the scene identification result from the scene recognition unit 12-2. Based on the acquired scene identification result, the scene processing unit 12-3 determines the shooting and development processing conditions to be applied in the first processing unit 11. When an external flash operation signal is input, the scene processing unit 12-3 determines the shooting and development processing conditions suitable for use with the external flash.

[0022] (Step S9-1) In the second processing unit 12, the scene processing section 12-3 outputs the determined shooting and development processing conditions to the first processing unit 11. (Step S10-1) In the first processing unit 11, the shooting and development setting section 11-3 acquires the shooting and development processing conditions output by the scene processing section 12-3. The shooting and development setting section 11-3 sets the acquired shooting and development processing conditions in the development section 11-1. (Step S11-1) In the first processing unit 11, when an operation to instruct photography is performed using the release button and a photography instruction signal is input, the development section 11-1 drives the image sensor 13 based on the photography and development processing conditions set by the photography and development setting section 11-3, and acquires an image signal. (Step S12-1) In the first processing unit 11, the developing section 11-1 performs development processing based on the shooting and development processing conditions set by the shooting and development setting section 11-3, thereby generating developed data.

[0023] (Step S13-1) In the first processing unit 11, the developing section 11-1 outputs the generated developed data. (Step S14-1) In the second processing unit 12, the image recording section 12-4 acquires the developed data output from the developing section 11-1 in the first processing unit 11. The image recording section 12-4 converts the acquired developed data into an image format to be saved in the second processing unit 12, and records (outputs) the data.

[0024] In the above-described embodiment, the second processing unit 12 acquires the developed data output by the first processing unit 11 and analyzes the acquired developed data, but this is not limiting. For example, the second processing unit 12 may acquire the imaging signal output by the image sensor 13, develop the acquired imaging signal, and acquire the developed data. An example of the developed data is YUV.

[0025] The imaging device 100 according to this embodiment includes a first processing unit 11 that generates developed data based on an imaging signal output by an image sensor 13, and a second processing unit 12 that includes an application processor that processes applications. The second processing unit 12 analyzes the imaging signal output by the image sensor 13, and the first processing unit 11 develops the imaging signal based on the analysis result. By configuring in this manner, the first processing unit can develop images based on the analysis results of the imaging signal by the second processing unit 12, which includes an application processor. This allows the first processing unit to identify various scenes and apply shooting and development conditions suitable for those scenes by utilizing software technology that is widely available for the second processing unit 12, which includes an application processor.

[0026] In the imaging device 100, the first processing unit 11 outputs the development data to the second processing unit 12, the second processing unit 12 analyzes the development data output by the first processing unit 11, and the first processing unit 11 develops the image based on the analysis results of the development data. By configuring in this manner, the first processing unit can perform development based on the analysis results of the development data by the second processing unit 12 including the application processor, and can therefore utilize software technology that is widely available for the second processing unit 12 including the application processor to identify various scenes and apply shooting and development conditions suitable for those scenes.

[0027] In the imaging device 100, the second processing unit 12 performs scene recognition processing based on the developed data output by the first processing unit 11, and the first processing unit 11 performs development based on the results of the scene recognition processing. By configuring in this manner, the first processing unit can perform development based on the results of scene recognition processing by the second processing unit 12, which includes an application processor. This allows the first processing unit to identify various scenes and apply shooting and development conditions suitable for those scenes by utilizing software technology that is widely available for the second processing unit 12, which includes an application processor.

[0028] (Modification of the embodiment) FIG. 1 can be applied to an example of an imaging device 100a according to a modification of the embodiment. The imaging device 100a extends the conventional AUTO function, allowing the user to release the shutter without worrying about settings, thereby obtaining photographic results suitable for various scenes. Furthermore, the imaging device 100a performs HDR (High Dynamic Range) image composition to obtain photographic results that reproduce a wide range of areas, from shadows to highlights, with little blown-out highlights or crushed shadows. Fig. 7 is a diagram showing an overview of the operation of an imaging device according to a modified example of the embodiment. As shown in Fig. 7, the imaging device 100a checks the exposure difference required for HDR shooting and continuously captures images with different exposures in a single shooting session. The imaging device 100a performs HDR image synthesis. When performing HDR image synthesis, the imaging device 100a uses the scene identification results to apply shooting and development processing, thereby obtaining more impressive HDR imaging results.

[0029] 8 is a partial block diagram showing an example of a camera body of an imaging device according to a modified example of the embodiment. The camera body 1 includes, for example, an imaging unit 16a. The imaging unit 16a includes an image sensor 13, a first processing unit 11a, and a second processing unit 12a. The first processing unit 11a is connected to the second processing unit 12a. The first processing unit 11a includes a development unit 11-1, a scene determination unit 11-2, and a shooting / development setting unit 11-3. The second processing unit 12a includes an image conversion unit 12a-1, a scene recognition unit 12-2, an exposure difference determination unit 12a-3, a scene processing unit 12a-4, a face detection unit 12a-5, an image synthesis unit 12a-6, an image selection unit 12a-7, and an image recording unit 12a-8.

[0030] In the second processing unit 12a, the exposure difference determination unit 12a-3 acquires the result of converting the developed data from the image conversion unit 12a-1. The exposure difference determination unit 12a-3 analyzes the result of converting the acquired developed data as a reference for exposure difference and derives the shooting conditions for the exposure difference required for HDR image synthesis (an underexposed image, an overexposed image, or both). The exposure difference determination unit 12a-3 outputs the derived shooting conditions for the exposure difference to the scene processing unit 12a-4. In the second processing unit 12a, the scene processing unit 12a-4 acquires the scene identification result from the scene recognition unit 12-2. The scene processing unit 12a-4 acquires the exposure difference shooting conditions from the exposure difference determination unit 12a-3. The scene processing unit 12a-4 determines the shooting and development processing conditions to be applied by the first processing unit 11a based on the acquired scene identification result and the exposure difference shooting conditions. The shooting and development processing conditions include the number of times images are taken. When an external flash is connected, the scene processing unit 12a-4 receives an external flash operation signal. When the external flash operation signal is received, the scene processing unit 12a-4 determines shooting and development processing conditions suitable for use with the external flash. The scene processing unit 12a-4 outputs the determined shooting and development processing conditions to the first processing unit 11a. The scene processing unit 12a-4 may output the scene identification results acquired from the scene recognition unit 12-2 to the shooting and development setting unit 11-3 of the first processing unit 11a instead of the determined shooting and development processing conditions. In this case, the shooting and development setting unit 11-3 of the first processing unit 11a determines the shooting and development processing conditions based on the acquired scene information. The scene processing unit 12a-4 is connected to receive a release signal output from the release button 3 in FIG. 1. The development unit 11-1 of the first processing unit 11a is also connected to receive a release signal output from the release button 3 in FIG. 1, and its operation changes depending on whether priority is given to shooting response or scene recognition. For example, when priority is given to shooting response, the release signal from the first processing unit 11a is given priority, so there is less delay before capturing an image. On the other hand, when priority is given to scene recognition, the release signal from the second processing unit 12a is given priority. Therefore, the scene processing unit 12a-4 outputs the shooting and development processing conditions at the time the release button is pressed to the shooting and development setting unit 11-3 of the first processing unit 11a, and then instructs the development unit 11-1 to capture an image via the shooting and development setting unit 11-3 of the first processing unit 11a. Therefore, it takes a slightly longer time to capture an image than when priority is given to shooting response. However, since the development and imaging settings are appropriate for the scene specified immediately before the release, it is easier to obtain imaging and development results that are appropriate for the scene than when priority is given to shooting response.Which of the release signal input to the development section 11-1 of the first processing unit 11a and the release signal input to the scene processing section 12a-4 of the second processing unit 12a is given priority is determined in advance by the mode dial or settings of the camera.Alternatively, the camera may automatically determine this based on the exposure, settings, etc. The scene processing unit 12a-4 outputs the scene identification result acquired from the scene recognition unit 12-2 to the image synthesis unit 12a-6.

[0031] In the first processing unit 11a, the shooting and development setting section 11-3 acquires the shooting and development processing conditions output by the scene processing section 12a-4. The shooting and development setting section 11-3 sets the acquired shooting and development processing conditions in the developing section 11-1. The developing section 11-1 drives the image sensor 13 based on the shooting and development processing conditions set by the shooting and development setting section 11-3, and generates developed data for the acquired number of imaging operations based on the development processing conditions set by the shooting and development setting section 11-3 from the imaging signals acquired from the image sensor 13. Here, each of the multiple development data corresponds to a respective development data obtained by developing multiple imaging signals with different exposures acquired based on the number of imaging times included in the imaging and development processing conditions. In a modified embodiment, the development unit 11-1 performs imaging and development once as a preliminary imaging after multiple imaging operations under the imaging and development conditions determined to be appropriate by the first processing unit 11a. The results of this preliminary imaging are output to and used by the image selection unit 12a-7 of the second processing unit 12a, which will be described later. The image conversion unit 12a-1 acquires multiple pieces of developed data corresponding to the number of times an image was captured, generated by the development unit 11-1. The image conversion unit 12a-1 uses USB communication when acquiring the developed data. This allows MIPI-CSI to communicate only the developed data used for scene recognition and preview display. Therefore, after capturing an image, it is possible to perform operations to prepare for the next image capture. Specifically, scene recognition can be started immediately after capturing an image, and there is no need to reduce the frequency of updating the preview display. The image conversion section 12a-1 converts each of the multiple pieces of developed data corresponding to the acquired number of times of imaging into an image format that is easy to handle by the second processing unit 12a.

[0032] In the second processing unit 12a, the face detection unit 12a-5 detects a face from the result of image conversion of each piece of developed data that has an appropriate reference exposure. The face detection unit 12a-5 identifies the coordinates of the face position. The face detection unit 12a-5 outputs the identified face position coordinates to the image synthesis unit 12a-6. In the second processing unit 12a, the image synthesis unit 12a-6 acquires multiple image conversion results from the image conversion unit 12a-1. The image synthesis unit 12a-6 acquires the coordinates of the face position from the face detection unit 12a-5. When synthesizing the multiple acquired images based on the image that serves as the exposure reference and the coordinates of the face position among the multiple acquired image conversion results, the image synthesis unit 12a-6 changes the image to be synthesized so that the area containing the face does not become dark due to the synthesis. Specifically, if the face is likely to become dark after synthesis, the image to be input is selected as the reference image and an image brighter than that is selected as the input image. Based on the scene identification result acquired from the scene processing unit 12a-4, the image synthesis unit 12a-6 changes the image composition (finish) after image synthesis to one that is suitable for the identified scene.

[0033] Furthermore, if the ghosts that appear as a result of combining the acquired multiple images are equal to or greater than a preset threshold, the image combining unit 12a-6 determines that the image combining has failed and outputs the result of the error and the combined image to the image selecting unit 12a-7. If the ghosts that appear in the combined image obtained by combining the multiple images are less than a preset threshold, the image combining unit 12a-6 determines that the image combining has been successful and outputs the result of the error-free combination and the combined image to the image selecting unit 12a-7.

[0034] In the second processing unit 12a, the image selection unit 12a-7 acquires the result of the image composition, whether or not there is an error in the image composition, and the composite image output by the image composition unit 12a-6. The image selection unit 12a-7 acquires the result of the preliminary photographing output by the development unit 11-1 of the first processing unit 11a. The image selection unit 12a-7 checks whether or not there is an error in the image composition acquired from the image composition unit 12a-6, and if an error is detected, it assumes that the image composition has failed and outputs the result of the preliminary photographing acquired from the development unit 11-1 of the first processing unit 11a to the image recording unit 12a-8. The image selection unit 12a-7 checks whether there are any errors in the image synthesis obtained from the image synthesis unit 12a-6, and if there are no errors, it considers the image synthesis to be successful and outputs the synthesized image obtained from the image synthesis unit 12a-6 to the image recording unit 12a-8.

[0035] In the second processing unit 12a, the image recording unit 12a-8 acquires the image output by the image selection unit 12a-7. The image recording unit 12a-8 converts and outputs (records) the acquired image based on an image format preset by the user. The image format setting can specify an image file format such as JPEG, and the image resolution and compression rate may also be specified. When the image recording unit 12a-8 converts the composite image into an image file such as JPEG, Exif information may be added. For example, the image recording unit 12a-8 acquires information specifying the date and time of photography, and adds Exif information including the acquired information specifying the date and time of photography.

[0036] (Operation of imaging device) 9 is a flowchart showing an example of the operation of an imaging device according to a modification of the embodiment. With reference to FIG. 9, the operations of the first processing unit 11a and the second processing unit 12a of the imaging device 100a will be mainly described. Steps S1-2 to S7-2 can be performed using steps S1-1 to S7-1 described with reference to FIG. 6, and therefore a description thereof will be omitted here. (Step S8-2) In the second processing unit 12a, the exposure difference determination unit 12a-3 acquires the result of image conversion of the developed data from the image conversion unit 12a-1. The exposure difference determination unit 12a-3 analyzes the acquired result of image conversion of the developed data using as a reference, and derives the exposure difference required for HDR image synthesis (an underexposed image, an overexposed image, or both). The exposure difference determination unit 12a-3 outputs the derived exposure difference required for HDR image synthesis to the scene processing unit 12a-4. (Step S9-2) In the second processing unit 12a, the scene processing unit 12a-4 acquires the scene identification result from the scene recognition unit 12-2. The scene processing unit 12a-4 acquires the exposure difference required for HDR image synthesis from the exposure difference determination unit 12a-3. The scene processing unit 12a-4 determines the shooting and development processing conditions to be applied by the first processing unit 11 based on the acquired scene identification result and the exposure difference required for HDR image synthesis. The shooting and development processing conditions include the number of images to be captured to obtain the exposure difference required for HDR image synthesis. When an external flash operation signal is input, the scene processing unit 12a-4 determines the shooting and development processing conditions suitable for use with an external flash.

[0037] (Step S10-2) In the second processing unit 12a, the scene processing section 12a-4 outputs the determined shooting and development processing conditions to the first processing unit 11a. (Step S11-2) In the first processing unit 11a, the shooting and development setting section 11-3 acquires the shooting and development processing conditions output by the scene processing section 12a-4. The shooting and development setting section 11-3 sets the acquired shooting and development processing conditions in the development section 11-1. (Step S12-2) In the first processing unit 11a, the developing unit 11-1 receives an operation to instruct shooting by a release signal output from the release button, and when the shooting instruction signal is input, it drives the image sensor 13 and acquires an image signal based on the shooting and development processing conditions set by the shooting and development setting unit 11-3. When driving the image sensor 13, the developing unit 11-1 instructs the image sensor 13 to capture multiple images with changed exposure settings, and generates multiple developed data by acquiring and processing each of the image signals. After capturing multiple images, the developing unit 11-1 captures and develops one preliminary image under normal shooting and development conditions, and generates developed data as the preliminary shooting result. (Step S13-2) In the first processing unit 11a, the developing section 11-1 outputs a plurality of developed data and the developed data of the preliminary photographing result to the second processing unit. In the second processing unit 12a, the image conversion section 12a-1 acquires the plurality of developed data generated by the development section 11-1 and the developed data of the preliminary photographing result. (Step S14-2) In the second processing unit 12a, the image conversion section 12-1 performs image conversion on each of the plurality of pieces of developed data corresponding to the acquired number of times of imaging.

[0038] (Step S15-2) In the second processing unit 12a, the face detection unit 12a-5 identifies the position coordinates of a face from the result with the appropriate reference exposure among the image conversion results of each acquired developed data. The face detection unit 12a-5 outputs the identified position coordinates of the face to the image synthesis unit 12a-6. (Step S16-2) In the second processing unit 12a, the image synthesis unit 12a-6 acquires multiple images from the image conversion unit 12a-1. The image synthesis unit 12a-6 acquires the position coordinates of faces from the face detection unit 12a-5. The image synthesis unit 12a-6 acquires the identified scene name from the scene processing unit 12a-4. The image synthesis unit 12a-6 synthesizes images based on the results of image conversion of each of the acquired development data and the identified scene name. When synthesizing multiple images, the image synthesis unit 12a-6 changes the images to be used for synthesis based on the acquired position coordinates of the faces so that the brightness of the faces does not become dark during synthesis and so that the area where the faces are located does not become dark due to synthesis. If the ghosts that appear as a result of combining the acquired multiple images are equal to or greater than a preset threshold, the image combining unit 12a-6 determines that the image combining has failed and outputs the error result and the combined image to the image selecting unit 12a-7. If the ghosts that appear in the combined image obtained by combining the multiple images are less than a preset threshold, the image combining unit 12a-6 determines that the image combining has been successful and outputs the error-free result and the combined image to the image selecting unit 12a-7.

[0039] (Step S17-2) In the second processing unit 12a, the image selection unit 12a-7 outputs the composite image of the image composition unit 12a-6 to the image recording unit 12a-8 if there is no error, and outputs the preliminary photographing result obtained from the development unit 11-1 of the first processing unit 11a to the image recording unit 12a-8 if there is an error. (Step S18-2) In the second processing unit 12a, the image recording unit 12a-8 acquires the image output by the image selection unit 12a-7. The image recording unit 12a-8 converts the acquired image based on an image format preset by the user and outputs (records) it.

[0040] In the modified example of the embodiment described above, the developing unit 11-1 generates developed data corresponding to the number of times of imaging, which is the sum of multiple imaging sessions and preliminary imaging sessions, but the present invention is not limited to this example. For example, the developing unit 11-1 may generate developed data corresponding to the number of times of imaging without preliminary imaging. In this case, when the image selecting unit 12a-7 receives an erroneous result from the image combining unit 12a-6, the image selecting unit 12a-7 may output one of the images that has already been captured instead of the obtained combined image. In the modified example of the embodiment described above, when an erroneous result is obtained from the image synthesis unit 12a-6, the image selection unit 12a-7 obtains developed data from the development unit 11-1 and outputs the obtained developed data, but this example is not limiting. For example, the image composition unit 12a-6 determines whether a composite image obtained by combining the acquired multiple images is a desired image. If the acquired determination result indicates that the image obtained by combining the multiple images is not a desired image, the image selection unit 12a-7 may acquire developed data from the development unit 11-1 and output the acquired developed data. Furthermore, the image selection unit 12a-7 may output one of the images that has already been captured if the judgment result indicates that the image obtained by combining multiple images is not the desired image, based on the acquired judgment result.

[0041] According to the imaging device 100a according to the modified embodiment, in the imaging device 100, the second processing unit 12a determines the number of times to capture images for compositing based on the preview image output by the first processing unit 11a. The first processing unit 11a captures images a number of times that is the number of times to capture images for compositing determined by the second processing unit 12a, increased by one. If the second processing unit 12a cannot combine images captured the number of times for compositing, it outputs images captured by the number of times that is increased by one. With this configuration, the first processing unit 11a can perform imaging a number of times that is the number of times that is increased by one from the number of times that the second processing unit 12a has determined for imaging for composition. Therefore, when the second processing unit 12a cannot compose images obtained by imaging the number of times that is determined for imaging for composition, it can output images obtained by imaging the number of times that is increased by one.

[0042] In the imaging device 100a, the second processing unit 12a determines the number of images to be taken for compositing based on the developed data (preview image) output by the first processing unit 11a, the first processing unit 11a takes images for the number of images to be taken for compositing determined by the second processing unit 12a, the second processing unit 12a generates a composite image by combining the images taken for the number of images to be taken for compositing, and if the generated composite image is not desired, outputs one of the images that has already been taken. With this configuration, the second processing unit 12a generates a composite image by combining images captured the number of times for composition, and if the generated composite image is not the desired one, it can output one of the images that have already been captured. Because it is possible to output an image that has already been captured, the number of images can be reduced compared to when capturing images the number of times determined by the second processing unit 12a for composition plus one.

[0043] In the imaging device 100a, the second processing unit 12a determines the exposure conditions and the number of times to capture images for compositing based on the developed data (preview image) output by the first processing unit 11a, the first processing unit 11a captures images based on the exposure conditions and the number of times to capture images for compositing determined by the second processing unit 12a, and the second processing unit selects images with appropriate exposure from the images captured with the number of times to capture images for compositing and composites the selected images. With this configuration, the second processing unit can select an image with an appropriate exposure from the images captured the number of times for composition, and can therefore compose the selected images with the appropriate exposure.

[0044] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and the present invention also includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0045] 1...camera body, 3...release button, 11, 11a...first processing unit, 11-1...developing unit, 11-2...scene determination unit, 11-3...shooting and development setting unit, 12, 12a...second processing unit, 12-1, 12a-1...image conversion unit, 12-2...scene recognition unit, 12a-3...exposure difference determination unit, 12-4, 12a-8...image recording unit, 12-3, 12a-4...scene processing unit, 12a-5...face detection unit, 12a-6...image synthesis unit, 12a-7...image selection unit, 13...image sensor, 16, 16a...imaging unit, 100, 100a...imaging device

Claims

1. a first processing unit that generates developed data based on an image signal output by the image sensor; a second processing unit including an application processor for processing an application; Equipped with the second processing unit analyzes the imaging signal output by the image sensor; The first processing unit performs development based on an analysis result of the imaging signal.

2. the first processing unit outputs the developed data to the second processing unit; the second processing unit analyzes the development data output by the first processing unit, The imaging device according to claim 1 , wherein the first processing unit performs development based on an analysis result of the development data.

3. the second processing unit performs a scene recognition process based on the developed data output by the first processing unit; The imaging device according to claim 2 , wherein the first processing unit performs development based on a result of the scene recognition processing.

4. The second processing unit determines development conditions based on a result of the scene recognition processing; The imaging device according to claim 3 , wherein the first processing unit performs development based on the development conditions.

5. An imaging device as described in Claim 4, wherein the development conditions include conditions related to white balance.

6. The imaging device described in Claim 3, wherein the second processing unit performs the scene recognition processing using deep learning.

7. A conversion unit that converts at least one of the format and resolution of the imaging signal or the developed data, The imaging device according to claim 3 , wherein the second processing unit performs the scene recognition processing based on the imaging signal or the developed data converted by the conversion unit.

8. the second processing unit determines an exposure condition based on the preview image output by the first processing unit; The imaging device according to claim 1 , wherein the first processing unit captures an image based on the exposure condition.

9. the second processing unit determines the number of times of imaging for synthesis based on the preview image output by the first processing unit; the first processing unit performs imaging a number of times that is the number of times of imaging for composition determined by the second processing unit plus one, The imaging device according to claim 1 , wherein the second processing unit outputs an image captured one more time when the images captured the number of times for composition cannot be combined.

10. the second processing unit determines the number of times of imaging for synthesis based on the preview image output by the first processing unit; the first processing unit performs the number of imaging operations for combination determined by the second processing unit; 3. The imaging device according to claim 1, wherein the second processing unit generates a composite image by combining images captured for the number of times for compositing, and outputs one of the images already captured if the generated composite image is not desired.

11. the second processing unit determines an exposure condition and the number of times to capture images for composition based on the preview image output by the first processing unit; the first processing unit performs imaging based on the exposure conditions determined by the second processing unit and the number of imaging times for composition; The imaging device according to claim 1 , wherein the second processing unit selects an image with an appropriate exposure from the images captured for the number of times for composition, and composes the selected images.

12. An imaging device as described in claim 1 or 2, wherein the first processing unit and the second processing unit are configured as different processors connected by a communication means.

13. The imaging device described in Claim 12, wherein the first processing unit and the second processing unit are connected by multiple communication means.

Images