Imaging device
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- NIKON CORP
- Filing Date
- 2022-07-12
- Publication Date
- 2026-08-04
Smart Images
Figure 0007899615000001 
Figure 0007899615000002 
Figure 0007899615000003
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to an imaging device.
Background Art
[0002] There is known an imaging device that extends a conventional auto function and, when a predetermined operation is performed, displays a plurality of sample images according to a shooting scene, and then switches whether to stop displaying the sample images or continue displaying them according to the degree of scene change (see, for example, Patent Document 1).
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
[0004] One aspect of the present invention is An imaging unit that captures an image of a subject and outputs an imaging signal, and the a first processing unit that generates Images for display or recording based on an imaging signal, Based on at least one of the imaging signal or the image, the imaging scene is determined. a second processing unit, and an operation unit that receives a user's operation, and the second processing unit Based on the determined imaging scene, development conditions, including conditions related to auto white balance, are determined. , and the first processing unit The aforementioned development conditions based on , generate the above image , When the control unit is operated, the generation of the image is started. is an imaging device.
Brief Description of the Drawings
[0005] [Figure 1] It is a schematic diagram showing an example of an imaging device according to the present embodiment. [Figure 2] It is a diagram showing an outline of the operation of the imaging device according to the present embodiment. [Figure 3] It is a partial block diagram showing an example of a camera body of the imaging device according to the present embodiment. [Figure 4] It is a diagram showing an example of scene determination information. [Figure 5] This figure shows an example of scene recognition information. [Figure 6] This flowchart shows an example of the operation of the imaging device according to this embodiment. [Figure 7] This diagram shows an overview of the operation of the imaging device according to a modified embodiment. [Figure 8] This is a partial block diagram showing an example of a camera body of an imaging device according to a modified embodiment. [Figure 9] This flowchart shows an example of the operation of an imaging device according to a modified embodiment. [Modes for carrying out the invention]
[0006] (Embodiment) The imaging apparatus according to this embodiment will be described below with reference to the drawings. Figure 1 is a schematic diagram showing an example of the imaging apparatus according to this embodiment. The imaging device 100 comprises a camera body 1 and a lens unit 2. The lens unit 2 is attached to the camera body 1. The camera body 1 is equipped with a release button 3. The release button 3, also called the shutter button, is the button pressed when the camera's shutter is released. The lens unit 2 has an optical system arranged along the optical axis OA within its lens barrel. The optical system guides the incident subject light beam to the camera body 1.
[0007] The imaging device 100 extends the conventional AUTO function, allowing users to obtain shooting results suitable for various scenes simply by pressing the shutter without having to be aware of the settings. Figure 2 is a diagram illustrating the overview of the operation of the imaging device according to this embodiment. As shown in Figure 2, the imaging device 100 acquires an image and determines the scene based on the acquired image. Specifically, the imaging device 100 determines what the shooting scene is. For example, the imaging device 100 determines whether the shooting scene is a portrait, landscape, or sports. The imaging device 100 sets the appropriate settings for the scene based on the scene determination result. For example, the imaging device 100 sets the aperture, shutter speed, ISO sensitivity, etc. The imaging device 100 updates the preview image. The imaging device 100 takes a picture (image) when the user presses the shutter button.
[0008] Figure 3 is a partial block diagram showing an example of the camera body of the imaging device according to this embodiment. The camera body 1 includes, for example, an imaging unit 16. The imaging unit 16 includes an image sensor 13, a first processing unit 11, and a second processing unit 12. The first processing unit 11 is connected to the second processing unit 12. The image sensor 13 is constructed using, for example, a CMOS (Complementary Metal Oxide Semiconductor). The subject light beam guided by the optical system to the camera body 1 is guided to the imaging unit 16. In the imaging unit 16, the image sensor 13 converts the incident light into an electric signal to generate an imaging signal. The image sensor 13 transmits the generated imaging signal to the first processing unit 11.
[0009] The first processing unit 11 generates development data based on the imaging signal output by the image sensor 13. The first processing unit 11 outputs the generated development data to the second processing unit 12. The second processing unit 12 includes an application processor that processes the application. The second processing unit 12 acquires the development data output by the first processing unit 11 and analyzes the acquired development data. The second processing unit 12 outputs the analysis results of the development data to the first processing unit 11. The first processing unit 11 acquires the analysis results of the developed data output by the second processing unit 12 and develops the data based on the acquired analysis results of the developed data.
[0010] The first processing unit 11 includes a developing unit 11-1, a scene determination unit 11-2, and a shooting / developing setting unit 11-3. The second processing unit 12 includes an image conversion unit 12-1, a scene recognition unit 12-2, a scene processing unit 12-3, and an image recording unit 12-4. The first processing unit 11 receives the imaging signal transmitted by the image sensor 13. The first processing unit 11 develops the received imaging signal and obtains developed data. In the first processing unit 11, the developed data is output to the developing unit 11-1.
[0011] An example of the developed data generated by the developing unit 11-1 is YUV or the like. YUV uses luminance information Y and two color difference information (chroma information) U and V to represent an image. Hereinafter, the case where the developed data is "YUV420 PLANAR" will be continued to be described.
[0012] In the first processing unit 11, the developed data generated by the developing unit 11-1 is output to the scene determination unit 11-2 and the second processing unit 12. The scene determination unit 11-2 obtains the developed data output by the developing unit 11-1. The scene determination unit 11-2 determines the scene represented by the obtained developed data based on the obtained developed data. The scene determination unit 11-2 has scene determination information. The scene determination unit 11-2 obtains information on the focal length, shooting distance, and average luminance value input to the first processing unit 11.
[0013] FIG. 4 is a diagram showing an example of scene determination information. The scene determination information is information in a table format having scene names. Examples of scene names are portrait, night portrait, close-up (macro), and landscape. The scene determination unit 11-2 determines whether the obtained developed data is any of portrait, night portrait, close-up, and landscape. Returning to FIG. 3, the description will be continued.
[0014] In the second processing unit 12, the image conversion unit 12-1 acquires the development data generated by the development unit 11-1. When acquiring the development data, the image conversion unit 12-1 uses communication via MIPI-CSI (MIPI Camera Serial Interface). By using such high-speed communication, in the second processing unit 12, when performing scene recognition or preview display based on the acquired development data, the video captured by the image sensor 13 can be transmitted to the second processing unit with low latency and processed. The image conversion unit 12-1 converts the acquired development data into an image format that is easy to handle by the scene recognition unit 12-2 of the second processing unit. The conversion includes not only changing the data format but also changing the resolution of the image and trimming, etc. The scene recognition unit 12-2 acquires the scene determination result from the scene determination unit 11-2, the information specifying the focal length, the information specifying the shooting distance, and the average luminance value. The scene recognition unit 12-2 acquires the result of converting the development data from the image conversion unit 12-1. The scene recognition unit 12-2 has scene recognition information. When acquiring information from the scene determination unit 11-2, the scene recognition unit 12-2 uses communication via USB (Universal Serial Bus). By separating the communication methods between MIPI-CSI used for acquiring the development data by the image conversion unit 12-1 in this way, the first processing unit and the second processing unit can be coupled in a loosely coupled manner, so that a system configuration with a distributed processing load can be achieved.
[0015] FIG. 5 is a diagram showing an example of scene recognition information. Examples of scene names include portrait, night portrait, close-up (macro), landscape, in addition to pet, night view, sunset, cuisine (food), kids snap, candlelight, sea, snow, cherry blossoms, autumn leaves, illumination, fireworks display, and others (not applicable). Returning to FIG. 3, the description will continue.
[0016] The scene recognition unit 12-2 identifies the scene represented in the developed data based on the scene determination result obtained from the scene determination unit 11-2, information specifying the focal length, information specifying the shooting distance, the average brightness value, and the developed data obtained from the image conversion unit 12-1. The scene recognition unit 12-2 may use pattern matching or deep learning-based image recognition when identifying the scene. Specifically, the scene recognition unit 12-2 determines, based on the acquired scene determination result, information identifying the focal length, information identifying the shooting distance, the average brightness value, and the result of image conversion of the developed data, which of the following scenes the image represented in the developed data is: pet, night view, sunset, food, children's snapshot, candle, sea, snow, cherry blossoms, autumn leaves, illumination, fireworks, or other (none of the above), whichever is not the scene determination result acquired from the scene determination unit 11-2. Here, the scene recognition unit 12-2 may re-determine whether the scene obtained from the scene determination unit 11-2 is a portrait, a night portrait, a close-up (macro), or a landscape.
[0017] The scene processing unit 12-3 obtains the scene determination result from the scene recognition unit 12-2. Based on the obtained scene determination result, the scene processing unit 12-3 determines the shooting and development processing conditions to be applied by the first processing unit 11. The shooting and development processing conditions refer to the conditions necessary for shooting (imaging) and development, including the aperture value, shutter speed, ISO sensitivity, AWB setting, AF setting, and Picture Control, which are suitable for the identified scene. The scene processing unit 12-3 may correct the exposure conditions depending on the scene determination result obtained from the scene recognition unit 12-2. For example, if the scene determination result obtained from the scene recognition unit 12-2 is identified as "sunset," the exposure may be corrected by -1 stop to change the shooting and development conditions to produce a darker image. If an external flash is connected, the scene processing unit 12-3 receives an external flash operation signal. When the scene processing unit 12-3 receives an external flash operation signal, it determines shooting and development processing conditions suitable for using the external flash. The scene processing unit 12-3 outputs the determined shooting and development processing conditions to the first processing unit 11. The scene processing unit 12-3 may output the identified scene instead of the shooting and development processing conditions to the shooting and development setting unit 11-3 of the first processing unit 11. In that case, the shooting and development setting unit 11-3 of the first processing unit 11 will determine the shooting and development processing conditions based on the acquired scene information. The scene processing unit 12-3 is connected to receive the release signal output from the release button 3 in Figure 1. Similarly, the release button 3 in Figure 1 is also connected to the development unit 11-1 of the first processing unit 11, and its operation changes depending on whether shooting response or scene recognition is prioritized. For example, if shooting response is prioritized, the release signal from the first processing unit 11 is prioritized, resulting in less delay before image capture. On the other hand, if scene recognition is prioritized, the release signal from the second processing unit 12 is prioritized, so the scene processing unit 12-3 outputs the shooting and development processing conditions when the release signal is input to the shooting and development setting unit 11-3 of the first processing unit 11, and then instructs the development unit 11-1 to take an image via the shooting and development setting unit 11-3 of the first processing unit 11. Therefore, it takes a little longer to take an image than when shooting response is prioritized. However, since the development and imaging settings are optimized for the scene identified immediately before the shutter release, it becomes easier to obtain imaging and development results that are more suitable for the scene than when prioritizing shooting response. Note that the priority given to the release signal input to the development unit 11-1 of the first processing unit 11 and the release signal input to the scene processing unit 12-3 of the second processing unit 12 is determined in advance by the camera's mode dial or settings. Alternatively, the camera may automatically determine this based on exposure and other set information.
[0018] In the first processing unit 11, the shooting / development setting unit 11-3 acquires the shooting / development conditions output by the scene processing unit 12-3. The shooting / development setting unit 11-3 sets the shooting conditions and development processing conditions included in the acquired shooting / development processing conditions to the development unit 11-1. The development unit 11-1 is connected to a shutter release button. When the shutter release button of the first processing unit 11 takes priority, the development unit 11-1, upon receiving a signal from the shutter release button to instruct imaging, drives the image sensor 13 based on the shooting conditions set by the shooting / development setting unit 11-3, and generates developed data based on the imaging signal acquired from the image sensor 13 and the development processing conditions set by the shooting / development setting unit 11-3. Here, we will explain the case where the developed data generated by the development unit 11-1 is JPEG. The development unit 11-1 outputs the generated developed data to the second processing unit 12. In the second processing unit 12, the image recording unit 12-4 acquires the developed data output by the development unit 11-1 of the first processing unit 11. The image recording unit 12-4 uses USB communication when acquiring the developed data. This allows MIPI-CSI to communicate only the developed data used for scene recognition and preview display. Therefore, it is possible to perform operations to prepare for the next image capture immediately after imaging. Specifically, scene recognition can be started immediately after imaging, and there is no need to reduce the update frequency of the preview display. The image recording unit 12-4 converts and outputs (records) the acquired developed data based on the image format set in advance by the user. The image format setting can specify the format of the image file, such as JPEG, or it may also specify the image resolution and compression ratio.
[0019] (Operation of the imaging device) Figure 6 is a flowchart illustrating an example of the operation of the imaging device according to this embodiment. Referring to Figure 6, the operation of the first processing unit 11 and the second processing unit 12 of the imaging device 100 will be mainly described. (Step S1-1) In the first processing unit 11, the developing unit 11-1 acquires the imaging signal. (Step S2-1) In the first processing unit 11, the developing unit 11-1 generates developed data by processing the acquired imaging signal. An example of developed data is YUV. (Step S3-1) In the first processing unit 11, the development data generated by the development unit 11-1 is output to the second processing unit 12. (Step S4-1) In the second processing unit 12, the image conversion unit 12-1 acquires the developed data generated by the development unit 11-1. The image conversion unit 12-1 converts the acquired developed data into an image format that is easy for the second processing unit 12 to handle. The conversion includes not only the data format but also changes in image resolution and cropping.
[0020] (Step S5-1) In the first processing unit 11, the scene determination unit 11-2 acquires the development data generated by the development unit 11-1. Based on the acquired development data, the scene determination unit 11-2 determines the scene of the image represented in the development data. The scene determination unit 11-2 acquires the scene determination result and information such as the focal length, shooting distance, and average brightness value input to the first processing unit 11. (Step S6-1) In the second processing unit 12, the scene recognition unit 12-2 obtains the scene determination result, information to identify the focal length, information to identify the shooting distance, and the average brightness value from the scene determination unit 11-2. (Step S7-1) In the second processing unit 12, the scene recognition unit 12-2 obtains the result of image conversion of the developed data from the image conversion unit 12-1. Based on the acquired scene determination result, information that identifies the focal length, information that identifies the shooting distance, the average brightness value, and the result of image conversion of the developed data, the scene of the image represented in the developed data is identified by the scene recognition unit 12-2.
[0021] (Step S8-1) In the second processing unit 12, the scene processing unit 12-3 obtains the scene identification result from the scene recognition unit 12-2. Based on the obtained scene identification result, the scene processing unit 12-3 determines the shooting and development processing conditions to be applied in the first processing unit 11. If an external flash operation signal is input, the scene processing unit 12-3 determines shooting and development processing conditions suitable for the use of the external flash.
[0022] (Step S9-1) In the second processing unit 12, the scene processing unit 12-3 outputs the determined shooting and development processing conditions to the first processing unit 11. (Step S10-1) In the first processing unit 11, the shooting / development setting unit 11-3 acquires the shooting / development processing conditions output by the scene processing unit 12-3. The shooting / development setting unit 11-3 sets the acquired shooting / development processing conditions in the development unit 11-1. (Step S11-1) In the first processing unit 11, the developing unit 11-1 is operated by the release button to instruct shooting, and when a shooting instruction signal is input, the shooting / developing setting unit 11-3 drives the image sensor 13 based on the shooting / developing processing conditions set and acquires an imaging signal. (Step S12-1) In the first processing unit 11, the developing unit 11-1 generates developed data by performing developing processing based on the shooting and developing processing conditions set by the shooting and developing setting unit 11-3.
[0023] (Step S13-1) In the first processing unit 11, the developing unit 11-1 outputs the generated developing data. (Step S14-1) In the second processing unit 12, the image recording unit 12-4 acquires the development data output from the development unit 11-1 in the first processing unit 11. The image recording unit 12-4 converts the acquired development data into an image format to be saved in the second processing unit 12 and records (outputs) it.
[0024] In the embodiment described above, a case was described in which the second processing unit 12 acquires the development data output by the first processing unit 11 and analyzes the acquired development data. However, the invention is not limited to this example. For example, the second processing unit 12 may acquire the imaging signal output by the image sensor 13, develop the acquired imaging signal, and acquire development data. An example of development data is YUV.
[0025] According to the imaging device 100 of this embodiment, the imaging device 100 comprises a first processing unit 11 that generates development data based on the imaging signal output by the image sensor 13, and a second processing unit 12 that includes an application processor for processing applications. The second processing unit 12 analyzes the imaging signal output by the image sensor 13, and the first processing unit 11 develops the image based on the analysis results of the imaging signal. With this configuration, the first processing unit can develop images based on the analysis results of the imaging signal by the second processing unit 12, which includes an application processor. Therefore, by utilizing widely available software technologies for the second processing unit 12, which includes an application processor, it is possible to identify various scenes and apply shooting and development conditions suitable for those scenes.
[0026] In the imaging device 100, the first processing unit 11 outputs development data to the second processing unit 12, the second processing unit 12 analyzes the development data output by the first processing unit 11, and the first processing unit 11 develops based on the analysis results of the development data. By configuring it in this way, the first processing unit can develop the image based on the analysis results of the development data by the second processing unit 12, which includes an application processor. Therefore, by utilizing widely available software technologies for the second processing unit 12, which includes an application processor, it is possible to identify various scenes and apply shooting and development conditions suitable for those scenes.
[0027] In the imaging device 100, the second processing unit 12 performs scene recognition processing based on the development data output by the first processing unit 11, and the first processing unit 11 develops the image based on the results of the scene recognition processing. By configuring it in this way, the first processing unit can develop images based on the results of scene recognition processing by the second processing unit 12, which includes an application processor. Therefore, by utilizing widely available software technologies for the second processing unit 12, which includes an application processor, it is possible to identify various scenes and apply shooting and development conditions suitable for those scenes.
[0028] (Modified examples of the embodiment) An example of an imaging device 100a according to a modified embodiment can be shown in Figure 1. The imaging device 100a extends the conventional AUTO function, allowing users to obtain shooting results suitable for various scenes simply by pressing the shutter without having to think about the settings. Furthermore, by performing HDR (High Dynamic Range) image synthesis, the imaging device 100a obtains shooting results that reproduce a wide range from shadows to highlights with minimal overexposure and underexposure. Figure 7 is a diagram illustrating the operation of an imaging device according to a modified embodiment. As shown in Figure 7, the imaging device 100a checks the exposure difference necessary for HDR shooting and continuously captures images with different exposures in a single shot. The imaging device 100a performs HDR image synthesis. When performing HDR image synthesis, the imaging device 100a can obtain more impressive HDR imaging results by applying shooting and development processing using the scene identification results.
[0029] Figure 8 is a partial block diagram showing an example of a camera body of an imaging device according to a modified embodiment. The camera body 1 includes, for example, an imaging unit 16a. The imaging unit 16a includes an image sensor 13, a first processing unit 11a, and a second processing unit 12a. The first processing unit 11a is connected to the second processing unit 12a. The first processing unit 11a includes a developing unit 11-1, a scene determination unit 11-2, and a shooting / developing setting unit 11-3. The second processing unit 12a includes an image conversion unit 12a-1, a scene recognition unit 12-2, an exposure difference determination unit 12a-3, a scene processing unit 12a-4, a face detection unit 12a-5, an image synthesis unit 12a-6, an image selection unit 12a-7, and an image recording unit 12a-8.
[0030] In the second processing unit 12a, the exposure difference determination unit 12a-3 obtains the result of converting the development data from the image conversion unit 12a-1. The exposure difference determination unit 12a-3 analyzes the result of converting the acquired development data as a reference for exposure difference and derives the shooting conditions for the exposure difference (underexposed image, overexposed image, or both) necessary for HDR image synthesis. The exposure difference determination unit 12a-3 outputs the derived exposure difference shooting conditions to the scene processing unit 12a-4. In the second processing unit 12a, the scene processing unit 12a-4 obtains the scene identification result from the scene recognition unit 12-2. The scene processing unit 12a-4 obtains the exposure difference shooting conditions from the exposure difference determination unit 12a-3. Based on the obtained scene identification result and the exposure difference shooting conditions, the scene processing unit 12a-4 determines the shooting and development processing conditions to be applied in the first processing unit 11a. The shooting and development processing conditions include the number of images taken. If an external flash is connected, the scene processing unit 12a-4 receives an external flash operation signal. When the scene processing unit 12a-4 receives an external flash operation signal, it determines shooting and development processing conditions suitable for using the external flash. The scene processing unit 12a-4 outputs the determined shooting and development processing conditions to the first processing unit 11a. The scene processing unit 12a-4 may output the scene identification result obtained from the scene recognition unit 12-2 to the shooting / development setting unit 11-3 of the first processing unit 11a, instead of the determined shooting / development processing conditions. In that case, the shooting / development setting unit 11-3 of the first processing unit 11a will determine the shooting / development processing conditions based on the acquired scene information. The scene processing unit 12a-4 is connected to receive the release signal output from the release button 3 in Figure 1. Similarly, the development unit 11-1 of the first processing unit 11a is also connected to receive the release signal output from the release button 3 in Figure 1, and its operation changes depending on whether shooting response or scene recognition is prioritized. For example, if shooting response is prioritized, the release signal from the first processing unit 11a is prioritized, resulting in less delay before image capture. On the other hand, if scene recognition is prioritized, the release signal from the second processing unit 12a is prioritized, so the scene processing unit 12a-4 outputs the shooting and development processing conditions when the release button is pressed to the shooting and development setting unit 11-3 of the first processing unit 11, and then instructs the development unit 11-1 to take an image via the shooting and development setting unit 11-3 of the first processing unit 11a. Therefore, it takes a little longer to take an image than when shooting response is prioritized. However, since the development and imaging settings are optimized for the scene identified immediately before the shutter release, it becomes easier to obtain imaging and development results that are more suitable for the scene than when prioritizing shooting response. Note that the priority given to the release signal input to the development unit 11-1 of the first processing unit 11a and the release signal input to the scene processing unit 12a-4 of the second processing unit 12a is determined in advance by the camera's mode dial or settings. Alternatively, the camera may automatically determine this based on exposure and other set information. The scene processing unit 12a-4 outputs the scene identification result obtained from the scene recognition unit 12-2 to the image synthesis unit 12a-6.
[0031] In the first processing unit 11a, the shooting / development setting unit 11-3 acquires the shooting / development processing conditions output by the scene processing unit 12a-4. The shooting / development setting unit 11-3 sets the acquired shooting / development processing conditions in the development unit 11-1. Based on the shooting / development processing conditions set by the shooting / development setting unit 11-3, the development unit 11-1 drives the image sensor 13 and generates development data for the number of times the image was captured, based on the development processing conditions set by the shooting / development setting unit 11-3, using the image acquisition signal from the image sensor 13. Here, each of the multiple developed data corresponds to each of the developed data obtained by developing multiple image signals with different exposures, which were acquired based on the number of images taken included in the shooting and developing processing conditions. In a modified embodiment, the developing unit 11-1 performs one preliminary shot after multiple images using the shooting and developing conditions determined to be appropriate by the first processing unit 11a. The results of this preliminary shot are output to the image selection unit 12a-7 of the second processing unit 12a, which will be described later, and used. The image conversion unit 12a-1 acquires multiple development data corresponding to the number of images captured by the development unit 11-1. The image conversion unit 12a-1 uses USB communication when acquiring development data. This allows MIPI-CSI to communicate only the development data used for scene recognition and preview display. Therefore, it can immediately prepare for the next image capture after imaging. Specifically, scene recognition can be started immediately after imaging, and there is no need to reduce the update frequency of the preview display. The image conversion unit 12a-1 converts each of the multiple development data corresponding to the number of acquired imaging counts into an image format that is easy for the second processing unit 12a to handle.
[0032] In the second processing unit 12a, the face detection unit 12a-5 detects faces from the results of image conversion of each of the developed data, selecting those with appropriate reference exposure. The face detection unit 12a-5 identifies the coordinates of the face's position. The face detection unit 12a-5 outputs the identified face's position coordinates to the image synthesis unit 12a-6. In the second processing unit 12a, the image synthesis unit 12a-6 acquires multiple images resulting from image conversion from the image conversion unit 12a-1. The image synthesis unit 12a-6 acquires the coordinates of the face's position from the face detection unit 12a-5. When combining the acquired multiple images based on the exposure reference image and the face's position coordinates, the image synthesis unit 12a-6 modifies the images to be combined so that the area containing the face does not become dark after the synthesis. Specifically, if the face is likely to become dark after synthesis, the input images are selected as a reference image and an image that is brighter than it. Based on the scene identification result acquired from the scene processing unit 12a-4, the image synthesis unit 12a-6 modifies the image creation (final result) after image synthesis to be suitable for the identified scene.
[0033] Furthermore, if the ghosting that occurs in the composite image resulting from the synthesis of multiple acquired images exceeds a preset threshold, the image synthesis unit 12a-6 considers the image synthesis to have failed and outputs the image synthesized with the error to the image selection unit 12a-7. If the ghosting that occurs in the composite image resulting from the synthesis of multiple images is below a preset threshold, the image synthesis is considered to have succeeded, and outputs the image synthesized with the error-free image to the image selection unit 12a-7.
[0034] In the second processing unit 12a, the image selection unit 12a-7 acquires the result of whether or not there are errors in the image synthesis output by the image synthesis unit 12a-6, and the synthesized image. The image selection unit 12a-7 acquires the result of the preliminary shooting output by the development unit 11-1 of the first processing unit 11a. The image selection unit 12a-7 checks whether or not there are errors in the image synthesis acquired from the image synthesis unit 12a-6, and if there are errors, it considers the image synthesis to have failed and outputs the result of the preliminary shooting acquired from the development unit 11-1 of the first processing unit 11a to the image recording unit 12a-8. The image selection unit 12a-7 checks for any errors in the image synthesis obtained from the image synthesis unit 12a-6. If there are no errors, it considers the image synthesis to have been successful and outputs the synthesized image obtained from the image synthesis unit 12a-6 to the image recording unit 12a-8.
[0035] In the second processing unit 12a, the image recording unit 12a-8 acquires the image output by the image selection unit 12a-7. The image recording unit 12a-8 converts and outputs (records) the acquired image based on the image format set in advance by the user. The image format can be set to an image file format such as JPEG, or it may be possible to specify the image resolution and compression ratio. When the image recording unit 12a-8 converts a composite image to an image file such as JPEG, it may also add Exif information. For example, the image recording unit 12a-8 acquires information that identifies the date and time of shooting, and adds Exif information that includes the acquired information that identifies the date and time of shooting.
[0036] (Operation of the imaging device) Figure 9 is a flowchart showing an example of the operation of an imaging device according to a modified embodiment. Referring to Figure 9, the operation of the first processing unit 11a and the second processing unit 12a of the imaging device 100a will be mainly described. Steps S1-2 to S7-2 can be explained by applying steps S1-1 to S7-1 as described in Figure 6, so their explanation is omitted here. (Step S8-2) In the second processing unit 12a, the exposure difference determination unit 12a-3 obtains the result of image conversion of the developed data from the image conversion unit 12a-1. The exposure difference determination unit 12a-3 analyzes the obtained developed data based on the result of image conversion and derives the exposure difference (underexposed image, overexposed image, or both) necessary for HDR image synthesis. The exposure difference determination unit 12a-3 outputs the derived exposure difference necessary for HDR image synthesis to the scene processing unit 12a-4. (Step S9-2) In the second processing unit 12a, the scene processing unit 12a-4 obtains the scene identification result from the scene recognition unit 12-2. The scene processing unit 12a-4 obtains the exposure difference necessary for HDR image synthesis from the exposure difference determination unit 12a-3. Based on the obtained scene identification result and the exposure difference necessary for HDR image synthesis, the scene processing unit 12a-4 determines the shooting and development processing conditions to be applied in the first processing unit 11. The shooting and development processing conditions include the number of images taken to obtain the exposure difference necessary for HDR image synthesis. If an external flash operation signal is input, the scene processing unit 12a-4 determines shooting and development processing conditions suitable for the use of an external flash.
[0037] (Step S10-2) In the second processing unit 12a, the scene processing unit 12a-4 outputs the determined shooting and development processing conditions to the first processing unit 11a. (Step S11-2) In the first processing unit 11a, the shooting / development setting unit 11-3 acquires the shooting / development processing conditions output by the scene processing unit 12a-4. The shooting / development setting unit 11-3 sets the acquired shooting / development processing conditions in the development unit 11-1. (Step S12-2) In the first processing unit 11a, the developing unit 11-1 is operated to instruct shooting based on the release signal output from the release button. When a shooting instruction signal is input, the developing unit 11-1 drives the image sensor 13 and acquires an imaging signal based on the shooting and developing processing conditions set by the shooting and developing setting unit 11-3. When driving the image sensor 13, the developing unit 11-1 instructs multiple shots with different exposure settings, acquires each imaging signal, and processes them to generate multiple developed data. After multiple shots, the developing unit 11-1 performs one preliminary shot under normal shooting and developing conditions and generates developed data as a preliminary shot result. (Step S13-2) In the first processing unit 11a, the developing unit 11-1 outputs multiple development data and development data of the preliminary shooting results to the second processing unit. In the second processing unit 12a, the image conversion unit 12a-1 acquires multiple development data generated by the development unit 11-1 and development data of the preliminary shooting results. (Step S14-2) In the second processing unit 12a, the image conversion unit 12-1 converts each of the multiple development data corresponding to the number of acquired imaging counts into an image.
[0038] (Step S15-2) In the second processing unit 12a, the face detection unit 12a-5 identifies the position coordinates of the face from the result of image conversion of each acquired development data, based on the result with the appropriate reference exposure. The face detection unit 12a-5 outputs the identified position coordinates of the face to the image synthesis unit 12a-6. (Step S16-2) In the second processing unit 12a, the image synthesis unit 12a-6 acquires multiple images from the image conversion unit 12a-1. The image synthesis unit 12a-6 acquires the position coordinates of faces from the face detection unit 12a-5. The image synthesis unit 12a-6 acquires the identified scene name from the scene processing unit 12a-4. The image synthesis unit 12a-6 synthesizes images based on the image conversion results of each acquired development data and the identified scene name. When synthesizing multiple images, the image synthesis unit 12a-6 changes the image used for synthesis based on the acquired face position coordinates so that the brightness of faces does not become darker during synthesis, so that areas containing faces do not become darker during synthesis. If the image synthesis unit 12a-6 detects that the ghosting in the composite image obtained by combining multiple acquired images exceeds a preset threshold, it considers the image synthesis to have failed and outputs the result with errors and the synthesized image to the image selection unit 12a-7. If the ghosting in the composite image obtained by combining multiple images is below a preset threshold, it considers the image synthesis to have succeeded and outputs the result without errors and the synthesized image to the image selection unit 12a-7.
[0039] (Step S17-2) In the second processing unit 12a, if there are no errors, the image selection unit 12a-7 outputs the composite image from the image synthesis unit 12a-6 to the image recording unit 12a-8. If there are errors, it outputs the preliminary shooting results obtained from the development unit 11-1 of the first processing unit 11a to the image recording unit 12a-8. (Step S18-2) In the second processing unit 12a, the image recording unit 12a-8 acquires the image output by the image selection unit 12a-7. The image recording unit 12a-8 converts the acquired image based on an image format set in advance by the user and outputs (records) it.
[0040] In the modified embodiment described above, the developing unit 11-1 generates developing data corresponding to the total number of imaging sessions, including multiple imaging sessions and preliminary imaging. However, the invention is not limited to this example. For example, the developing unit 11-1 may generate developing data corresponding to the number of images taken without taking a preliminary shot. In this case, if the image selection unit 12a-7 obtains a result with an error from the image synthesis unit 12a-6, it may output one of the already captured images instead of the acquired composite image. In the modified embodiment described above, the image selection unit 12a-7 obtained development data from the development unit 11-1 and output the acquired development data when it obtained an error-filled result from the image synthesis unit 12a-6, but this is not the only example. For example, the image synthesis unit 12a-6 determines whether the composite image obtained by combining multiple acquired images is the desired image. Based on the acquired determination result, the image selection unit 12a-7 may acquire development data from the development unit 11-1 and output the acquired development data if the determination result indicates that the image obtained by combining multiple images is not the desired image. Furthermore, the image selection unit 12a-7 may output one of the already captured images if the judgment result indicates that the image obtained by combining multiple images is not the desired image.
[0041] According to the imaging device 100a, a modified embodiment, in the imaging device 100, the second processing unit 12a determines the number of images to be captured for synthesis based on the preview image output by the first processing unit 11a. The first processing unit 11a captures images one more time than the number of images to be captured for synthesis determined by the second processing unit 12a. If the second processing unit 12a cannot synthesize images captured by the number of images to be captured for synthesis, it outputs an image captured by the additional number of images captured. By configuring it in this way, the first processing unit 11a can perform imaging a number of times that is one step greater than the number of imaging steps for synthesis determined by the second processing unit 12a. Therefore, if the second processing unit 12a is unable to synthesize images obtained from imaging the number of times required for synthesis, it can output images obtained from imaging the number of times increased by one.
[0042] In the imaging device 100a, the second processing unit 12a determines the number of images to be captured for synthesis based on the development data (preview image) output by the first processing unit 11a. The first processing unit 11a then captures the number of images determined by the second processing unit 12a for synthesis. The second processing unit 12a generates a composite image by combining the images captured during the number of images determined for synthesis. If the generated composite image is not desired, it outputs one of the images that has already been captured. With this configuration, the second processing unit 12a generates a composite image by combining images taken a specified number of times for synthesis, and if the generated composite image is not desired, it can output one image that has already been captured. Because it can output an image that has already been captured, the number of images can be reduced compared to when the second processing unit 12a takes images one more time than the number of images determined for synthesis.
[0043] In the imaging device 100a, the second processing unit 12a determines the exposure conditions and the number of images to be taken for synthesis based on the development data (preview image) output by the first processing unit 11a. The first processing unit 11a takes images based on the exposure conditions and the number of images to be taken for synthesis determined by the second processing unit 12a. The second processing unit selects an image with appropriate exposure from the images taken the required number of times for synthesis and synthesizes the selected images. By configuring it in this way, the second processing unit can select an image with the appropriate exposure from the images captured for the number of imaging cycles required for synthesis, and thus synthesize the selected image with the appropriate exposure.
[0044] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of the invention. [Explanation of symbols]
[0045] 1…Camera body, 3…Release button, 11, 11a…First processing unit, 11-1…Developing unit, 11-2…Scene determination unit, 11-3…Shooting / development setting unit, 12, 12a…Second processing unit, 12-1, 12a-1…Image conversion unit, 12-2…Scene recognition unit, 12a-3…Exposure difference determination unit, 12-4, 12a-8…Image recording unit, 12-3, 12a-4…Scene processing unit, 12a-5…Face detection unit, 12a-6…Image synthesis unit, 12a-7…Image selection unit, 13…Image sensor, 16, 16a…Imaging unit, 100, 100a…Imaging device
Claims
1. An imaging unit that captures an image of a subject and outputs an imaging signal, A first processing unit that generates an image for display or recording based on the aforementioned imaging signal, A second processing unit that determines the imaging scene based on at least one of the imaging signal or the image, A control unit that accepts user input and Equipped with, The second processing unit determines development conditions, including conditions related to auto white balance, based on the determined imaging scene. The first processing unit generates the image based on the development conditions, When the control unit is operated, the generation of the image is started. Imaging device.
2. The second processing unit determines the imaging scene using deep learning, The imaging apparatus according to claim 1.
3. The second processing unit determines the exposure conditions based on the display image output by the first processing unit. The imaging unit captures images based on the exposure conditions. The imaging apparatus according to claim 1 or claim 2.
4. A method for combining multiple images to generate a composite image. The imaging apparatus according to claim 1 or 2.
5. The second processing unit determines the number of imaging cycles for synthesis based on the display image output by the first processing unit. The imaging unit performs imaging a number of times that is one more than the number of imaging times for synthesis determined by the second processing unit. The second processing unit outputs an image obtained by increasing the number of imaging attempts by one if it is not possible to synthesize the images obtained by the number of imaging attempts for synthesis. The imaging apparatus according to claim 4.
6. The second processing unit determines the number of imaging cycles for synthesis based on the display image output by the first processing unit. The imaging unit performs imaging for the number of imaging sessions determined by the second processing unit for synthesis, The second processing unit generates a composite image by combining images taken the specified number of times for synthesis, and if the generated composite image is not desired, it outputs one of the images that has already been taken. The imaging apparatus according to claim 4.
7. A conversion unit that converts at least one of the format or resolution of the imaging signal or the image, The second processing unit determines the imaging scene based on the imaging signal or image converted by the conversion unit. The imaging apparatus according to claim 1 or claim 2.
8. The first processing unit and the second processing unit are configured as different processors connected by communication means. The imaging apparatus according to claim 1 or claim 2.
9. The first processing unit and the second processing unit are connected by a plurality of communication means, The imaging apparatus according to claim 8.
10. The first processing unit outputs the image to the second processing unit by the first communication means, The second processing unit outputs the development conditions to the first processing unit via a second communication means. The imaging device according to claim 9.