Camera device, camera method and storage medium

Abstract

A camera device is provided, comprising an image sensor and a processor, wherein the processor performs the following processing: acquiring distance data, wherein the distance is the distance between a plurality of subjects within a camera area captured by the image sensor and the camera device; generating boundary data based on the distance data, wherein the boundary data represents the region of a boundary subject among the plurality of subjects that exists within the boundary of the depth of field; generating dynamic image data including the boundary data based on image data captured by the image sensor; and outputting the dynamic image data.

Description

Technical Field

[0001] The present invention relates to a camera device, camera method, and program. Background Technology

[0002] Japanese Patent Application Publication No. 2008-145465 discloses a display method and a user interface for displaying focus distribution data. The display method is as follows: In a camera device that has a manual focus adjustment function, a function that allows the user to arbitrarily set the aperture F-value before shooting, and a function that detects contrast and determines focus based on its height, the lens is set to the open aperture F-value and contrast data of all focus-drive ranges are read from the camera element. The acquired contrast data table is converted into a focus determination table. Based on the contrast data, the obtained depth of field, and the focus determination table, the focus distribution data is displayed in a real-time preview using a two-color overlay method.

[0003] Japanese Patent Application Publication No. 2007-214845 discloses an electronic camera equipped with a multi-point simultaneous focus frame display mode. The camera includes the following mechanisms: a focus object candidate acquisition mechanism, which, in the multi-point simultaneous focus frame display mode, compares the contrast values ​​of images sequentially obtained from the imaging element and acquires subjects corresponding to the set of contrasts whose differences from the contrast values ​​are within a specified threshold as focus object candidates; a focus point acquisition mechanism, which acquires the focus lens position of each focus object candidate; a focus point similarity determination mechanism, which compares the focus lens positions of each focus object candidate acquired by the focus point acquisition mechanism and determines whether the focus points are similar; and a focus frame display control mechanism, which displays a focus frame of approximately the same color for each image of a focus object candidate determined by the focus point similarity determination mechanism to have similar focus lens positions.

[0004] Japanese Patent Application Publication No. 2010-093422 discloses a camera device for capturing images of a subject, comprising the following mechanisms: a mechanism for detecting distance information of the subject; a mechanism for extracting the subject based on the distance information and distance; a mechanism for calculating focus information based on the focal length and F-number of the aperture obtained from the position of the focusing lens; a display mechanism for displaying an image of the subject and a subject distance chart on a display screen, the subject distance chart being displayed on a chart with distance as the axis and subject position marks indicating the position of the subject made based on the extracted subject being supplemented with focus information; and a mechanism for changing the focus and adjusting the aperture based on the displayed subject distance chart or operations on the subject image.

[0005] Japanese Patent Application Publication No. 10-197938 discloses a camera that displays a focusing distance range, wherein the focusing distance range is configured to display an appropriate range of focusing distances calculated based on the camera distance setting and the aperture setting value, corresponding to a distance scale.

[0006] Japanese Patent Application Publication No. 2010-177741 discloses a camera device, which includes: a camera unit for photographing a camera object; a focus adjustment unit for adjusting the focus; a map image generation unit for generating a map image for displaying a depth position indicating the position of the camera object in the depth direction and a focus position indicating the position of the focus; and a display unit for displaying the generated map image. Summary of the Invention

[0007] One embodiment of the present invention provides a camera device, camera method, and program, wherein the camera device allows a user to, for example, determine the position of the depth-of-field boundary based on an image displayed through dynamic image data.

[0008] means for solving technical problems

[0009] The camera device of the present invention is a camera device equipped with an image sensor and a processor. The processor performs the following processing: acquiring distance data about distance, which is the distance between a plurality of subjects within the camera area captured by the image sensor and the camera device; generating boundary data based on the distance data, which represents the area of ​​the boundary subject existing within the distance of the boundary portion of the depth of field; generating dynamic image data including the boundary data based on image data captured by the image sensor; and outputting the dynamic image data.

[0010] The processor can output dynamic image data as data for displaying a first image on a first display, wherein the first image distinguishes the area of ​​the boundary subject and the area other than the area of ​​the boundary subject.

[0011] Boundary data can be data used for image processing of the area of ​​the subject that is marked on the second image displayed on the second display based on the image data.

[0012] Image processing can be used to assign a predetermined color to the first pixel corresponding to the region of the boundary subject among the multiple first pixels constituting the second image.

[0013] Image processing can apply a predetermined brightness to the second pixel corresponding to the region of the boundary subject among the multiple second pixels constituting the second image.

[0014] Image processing can be used to attach labels to the second image to indicate the boundaries of the subject.

[0015] Image processing can be used to overlay a distance image generated from distance data onto a second image.

[0016] The boundary portion may include: a first boundary portion located on the near point side of the depth of field; and a second boundary portion located on the far point side of the depth of field. The boundary subject may include: a first boundary subject existing within the distance of the first boundary portion; and a second boundary subject existing within the distance of the second boundary portion. The boundary data may include: first boundary data representing the area of ​​the first boundary subject; and second boundary data representing the area of ​​the second boundary subject.

[0017] The first boundary data may be data representing the area of ​​the first boundary subject in a first manner based on the third image displayed on the third display according to the image data, and the second boundary data may be data representing the area of ​​the second boundary subject in the third image in a second manner different from the first manner.

[0018] The boundary can be at least one of the near point and the far point of the depth of field.

[0019] The processor can obtain region data representing the area of ​​the boundary subject based on the distance data. The boundary subject exists within a distance equal to the distance between the multiple subjects and the camera device and the boundary portion, and generates boundary data based on the region data.

[0020] The boundary can be at least one of the range including the near point of the depth of field and the range including the far point of the depth of field.

[0021] The range including the near point of depth of field can be the range extending from the near point of depth of field to the far point of depth of field.

[0022] The range including the far point of the depth of field can be the range extending from the far point of the depth of field to the near point of the depth of field.

[0023] The processor can set a distance range including the distance of the boundary portion, obtain region data representing the area of ​​the boundary subject based on the distance data, and generate boundary data based on the region data. The boundary subject exists within the distance range among multiple subjects and the camera device.

[0024] The width of the boundary can vary depending on the depth of field.

[0025] The processor can widen the boundary as the depth of field increases and narrow the boundary as the depth of field decreases.

[0026] The width of the boundary portion can vary depending on the number of pixels, which are the pixels corresponding to the boundary portion among the multiple pixels constituting the fourth image displayed on the fourth display based on the dynamic image data.

[0027] The camera device is equipped with a camera lens, and the permissible circle of confusion diameter of the image sensor can vary depending on at least one of the object distance, focal length, and aperture value in the camera lens.

[0028] The processor can obtain the first depth of field based on the object distance, focal length, aperture value, and predetermined allowable circle of confusion diameter. When the depth of the first depth of field is shallower than the first predetermined depth, the allowable circle of confusion diameter can be smaller than the first predetermined value.

[0029] The processor can perform the following processing: when displaying an image on the fifth display, it generates display image data representing the display image by including focus position data and boundary data in the dynamic image data based on distance data, and outputs the display image data to the fifth display; when recording an image is displayed on the fifth display, it outputs the dynamic image data to the fifth display.

[0030] The processor can store image data in a non-transitory storage medium.

[0031] An image sensor can have multiple phase difference pixels, and a processor can obtain distance data based on the phase difference pixel data output from the phase difference pixels.

[0032] Phase difference pixels can be pixels that selectively output non-phase difference pixel data and phase difference pixel data. Non-phase difference pixel data can be pixel data obtained by photoelectric conversion of the entire area of ​​the phase difference pixel, while phase difference pixel data can be pixel data obtained by photoelectric conversion of a portion of the phase difference pixel.

[0033] The imaging method of the present invention includes the following steps: acquiring distance data, wherein the distance is the distance between a plurality of subjects within the imaging area captured by an image sensor of the imaging device and the imaging device; generating boundary data based on the distance data, wherein the boundary data represents the area of ​​the boundary subject existing within the distance of the boundary portion of the depth of field; generating dynamic image data including the boundary data based on image data captured by the image sensor; and outputting the dynamic image data.

[0034] The program of the present invention is a program for causing a computer to perform a process comprising the following steps: acquiring distance data relating to distance, the distance being the distance between a plurality of subjects within a camera area captured by an image sensor of a camera device and the camera device; generating boundary data based on the distance data, the boundary data representing the area of ​​a boundary subject existing within the distance of a boundary portion of the depth of field; generating dynamic image data including the boundary data based on image data captured by the image sensor; and outputting the dynamic image data. Attached Figure Description

[0035] Figure 1 This is a schematic structural diagram illustrating an example of the structure of the camera device according to the first embodiment.

[0036] Figure 2 This is a schematic structural diagram illustrating an example of the hardware structure of the optical system and electrical system of the camera device according to the first embodiment.

[0037] Figure 3 This is a schematic structural diagram illustrating an example of the structure of the photoelectric conversion element according to the first embodiment.

[0038] Figure 4 This is an explanatory diagram illustrating an example of the relationship between the camera lens and the depth of field according to the first embodiment.

[0039] Figure 5 This is a block diagram illustrating an example of the functional structure of the CPU according to the first embodiment.

[0040] Figure 6 This is an explanatory diagram illustrating an example of the first operation of the camera device according to the first embodiment.

[0041] Figure 7 This is a front view showing an example of an image obtained through the first operation example of the camera device according to the first embodiment.

[0042] Figure 8 This is an explanatory diagram illustrating an example of a second operation of the camera device according to the first embodiment.

[0043] Figure 9 This is a front view showing an example of an image obtained by the second operation example of the camera device according to the first embodiment.

[0044] Figure 10 This is an explanatory diagram illustrating an example of the third operation of the camera device according to the first embodiment.

[0045] Figure 11 This is a front view showing an example of an image obtained by the third operation example of the camera device according to the first embodiment.

[0046] Figure 12A This is a flowchart illustrating an example of the flow of the first process in the dynamic image generation process performed by the CPU according to the first embodiment.

[0047] Figure 12B This is a flowchart illustrating an example of the flow of the second process in the dynamic image generation process performed by the CPU according to the first embodiment.

[0048] Figure 13 This is an explanatory diagram illustrating an example of the first operation of the camera device according to the second embodiment.

[0049] Figure 14This is an explanatory diagram illustrating an example of the second operation of the camera device according to the second embodiment.

[0050] Figure 15 This is an explanatory diagram illustrating an example of the third operation of the camera device according to the second embodiment.

[0051] Figure 16A This is a flowchart illustrating an example of the flow of the first process in the dynamic image generation process performed by the CPU according to the second embodiment.

[0052] Figure 16B This is a flowchart illustrating an example of the flow of the second process in the dynamic image generation process performed by the CPU according to the second embodiment.

[0053] Figure 17 This is an explanatory diagram illustrating an example of the first operation of the camera device according to the third embodiment.

[0054] Figure 18 This is an explanatory diagram illustrating an example of the second operation of the camera device according to the third embodiment.

[0055] Figure 19 This is a flowchart illustrating an example of a portion of the dynamic image generation process executed by the CPU according to the third embodiment.

[0056] Figure 20 This is an explanatory diagram illustrating an example of the operation of the camera device according to the fourth embodiment.

[0057] Figure 21 This is an explanatory diagram illustrating an example of the relationship between the first rear depth of field and the diameter of the permissible circle of confusion according to the fourth embodiment.

[0058] Figure 22 This is a flowchart illustrating an example of a portion of the dynamic image generation process performed by the CPU according to the fourth embodiment.

[0059] Figure 23 This is an explanatory diagram illustrating an example of the first operation of the camera device according to the fifth embodiment.

[0060] Figure 24 This is a front view showing an example of an image obtained through the first operation example of the camera device according to the fifth embodiment.

[0061] Figure 25A This is a flowchart illustrating an example of the flow of the first process in the dynamic image generation process performed by the CPU according to the fifth embodiment.

[0062] Figure 25BThis is a flowchart illustrating an example of the flow of the second process in the dynamic image generation process performed by the CPU according to the fifth embodiment. Detailed Implementation

[0063] Hereinafter, an example of the imaging device, imaging method and procedure related to the present invention will be described with reference to the accompanying drawings.

[0064] First, let me explain the words and phrases used in the following description.

[0065] CPU stands for Central Processing Unit. GPU stands for Graphics Processing Unit. TPU stands for Tensor Processing Unit. NVM stands for Non-volatile memory. RAM stands for Random Access Memory. IC stands for Integrated Circuit. ASIC stands for Application Specific Integrated Circuit. PLD stands for Programmable Logic Device. FPGA stands for Field-Programmable Gate Array. SoC stands for System-on-a-chip. SSD stands for Solid State Drive. USB stands for Universal Serial Bus. HDD stands for Hard Disk Drive. EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". fps is an abbreviation for "frames per second". MF is an abbreviation for "Manual Focus". AF is an abbreviation for "Auto Focus". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". A / D is an abbreviation for "Analog / Digital". PC is an abbreviation for "Personal Computer".LiDAR is an abbreviation for "Light Detection and Ranging". TOF is an abbreviation for "Time of Flight". EVF is an abbreviation for "Flectronic View Finder".

[0066] In this specification, "parallel" means parallelism, other than complete parallelism, encompassing a degree of error that is generally permissible in the technical field to which the present invention pertains and does not contradict the spirit of the present invention. Similarly, "orthogonal" means orthogonalism, other than complete orthogonality, encompassing a degree of error that is generally permissible in the technical field to which the present invention pertains and does not contradict the spirit of the present invention. Furthermore, "consistent" means consistency, other than complete consistency, encompassing a degree of error that is generally permissible in the technical field to which the present invention pertains and does not contradict the spirit of the present invention. Similarly, "equal" means equality, other than complete equality, encompassing a degree of error that is generally permissible in the technical field to which the present invention pertains and does not contradict the spirit of the present invention. Finally, in the following description, the numerical range indicated by "~" refers to the range including the values ​​before and after "~" as lower and upper limits.

[0067] [First Implementation]

[0068] As an example, such as Figure 1 As shown, the camera device 10 is a device for photographing a subject (not shown), and includes a controller 12, a camera device main body 16, and an interchangeable lens 18. The camera device 10 is an example of a "camera device" according to the technology of this invention, and the controller 12 is an example of a "computer" according to the technology of this invention. The controller 12 is built into the camera device main body 16 and controls the entire camera device 10. The interchangeable lens 18 is interchangeably mounted to the camera device main body 16. The interchangeable lens 18 is provided with a focusing ring 18A. The focusing ring 18A is operated by the user of the camera device 10 (hereinafter referred to as "user") or others when manually adjusting the focus of the subject through the camera device 10.

[0069] exist Figure 1 The example shown illustrates a digital camera with an interchangeable lens as one example of the imaging device 10. However, this is only one example, and the imaging device 10 can also be a digital camera with a fixed lens, or a digital camera built into various electronic devices such as smart devices, wearable terminals, cell observation devices, ophthalmic observation devices, or surgical microscopes.

[0070] The camera device body 16 is equipped with an image sensor 20. The image sensor 20 is an example of an "image sensor" according to the technology of this invention. As an example, the image sensor 20 is a CMOS image sensor. The image sensor 20 captures an image area including at least one subject. When the interchangeable lens 18 is mounted on the camera device body 16, subject light representing the subject is transmitted through the interchangeable lens 18 and imaged onto the image sensor 20, which then generates image data representing the subject.

[0071] In the first embodiment, a CMOS image sensor is exemplified as the image sensor 20, but the technology of the present invention is not limited thereto. For example, the technology of the present invention also applies even if the image sensor 20 is another type of image sensor such as a CCD image sensor.

[0072] The upper surface of the camera device body 16 is provided with a release button 22 and a dial 24. The dial 24 is operated when setting the operation mode of the camera system and the operation mode of the playback system, etc. By operating the dial 24, the camera device 10 can selectively set the shooting mode, playback mode, and setting mode as the operation mode. The shooting mode is the operation mode in which the camera device 10 performs shooting. The playback mode is the operation mode in which the images (e.g., still images and / or moving images) captured in the shooting mode are played back. The setting mode is the operation mode in which the camera device 10 is set when setting various setting values ​​used in the control related to shooting.

[0073] The release button 22 functions as both a camera preparation indicator and a camera indicator, detecting press operations in both the camera preparation indicator state and the camera indicator state. The camera preparation indicator state refers, for example, to the state where the button is pressed from the standby position to the middle position (half-pressed position), and the camera indicator state refers to the state where the button is pressed to the final pressed position (fully pressed position) beyond the middle position. Hereinafter, the state of "pressing from the standby position to the half-pressed position" will be referred to as the "half-pressed state," and the state of "pressing from the standby position to the fully pressed position" will be referred to as the "fully pressed state." Depending on the structure of the camera device 10, the shooting preparation indicator state can also refer to the state where the user's finger touches the release button 22, and the shooting indicator state can also refer to the state where the user's finger moves from the state of touching the release button 22 to the state of releasing it.

[0074] The back of the camera unit 16 is provided with indicator keys 26 and a touch screen display 32. The touch screen display 32 includes a display 28 and a touch panel 30 (see also...). Figure 2 As an example of display 28, an FL display (e.g., an organic FL display or an inorganic FL display) can be cited. Display 28 may also be other types of displays, such as liquid crystal displays, instead of FL displays.

[0075] The display 28 displays images and / or character information, etc. When the camera device 10 is in camera mode, the display 28 is used to display real-time preview images obtained from continuous video recording for real-time preview purposes. Here, "real-time preview image" refers to a dynamic image for display based on image data captured by the image sensor 20. The shooting performed to obtain the real-time preview image (hereinafter also referred to as "shooting for real-time preview image") is performed, for example, at a frame rate of 60 fps. 60 fps is just one example; it can also be a frame rate less than 60 fps or a frame rate greater than 60 fps.

[0076] When the camera device 10 is instructed to capture a still image via the release button 22, the display 28 is also used to display a still image obtained by capturing a still image. Furthermore, the display 28 is also used to display playback images when the camera device 10's operating mode is playback mode. Moreover, when the camera device 10's operating mode is setting mode, the display 28 is also used to display a menu screen where various menus can be selected, and a setting screen for setting various settings used in controls related to shooting.

[0077] Touch panel 30 is a transmissive touch panel that is superimposed on the surface of the display area of ​​display 28. Touch panel 30 receives instructions from the user by detecting contact with an indicator such as a finger or stylus. For ease of explanation, the "full press state" mentioned above also includes the state in which the user presses the soft key for starting photography via touch panel 30.

[0078] In the first embodiment, an example of a touchscreen display 32 is an external touchscreen display in which the touch panel 30 is superimposed on the surface of the display area of ​​the display 28, but this is only one example. For example, an external or internal touchscreen display can also be used as the touchscreen display 32.

[0079] Indicator key 26 receives various instructions. Here, "various instructions" refers to, for example, instructions for displaying the menu screen, instructions for selecting one or more menus, instructions for confirming the selection, instructions for deleting the selection, zooming in, zooming out, and frame forward, etc. Furthermore, these instructions can also be made via touch panel 30.

[0080] As an example, such as Figure 2 As shown, the image sensor 20 includes a photoelectric conversion element 72. The photoelectric conversion element 72 has a light-receiving surface 72A. The photoelectric conversion element 72 is disposed within the camera device body 16 such that the center of the light-receiving surface 72A coincides with the optical axis 0A (see also...). Figure 1The photoelectric conversion element 72 has a plurality of photosensitive pixels 72B arranged in a matrix (reference). Figure 3 The light-receiving surface 72A is formed by multiple photosensitive pixels 72B. Each photosensitive pixel 72B has a microlens 72C (see reference). Figure 3 Each photosensitive pixel 72B is a physical pixel with a photodiode (not shown), which performs photoelectric conversion on the received light and outputs an electrical signal corresponding to the amount of light received.

[0081] Furthermore, multiple photosensitive pixels 72B are arranged in a matrix with red (R), green (G) or blue (B) color filters (illustration omitted) in a predetermined pattern (e.g., Bayer arrangement, RGB stripe arrangement, R / G square arrangement, X-Trans (registered trademark) arrangement or honeycomb arrangement).

[0082] The interchangeable lens 18 includes an imaging lens 40. The imaging lens 40 comprises an objective lens 40A, a focusing lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, focusing lens 40B, zoom lens 40C, and aperture 40D are arranged sequentially along the optical axis OA from the subject side (object side) to the imaging device body 16 side (image side). The imaging lens 40 is an example of a "camera lens" according to the technology of this invention.

[0083] Furthermore, the interchangeable lens 18 includes a control device 36, a first actuator 37, a second actuator 38, a third actuator 39, a first position sensor 42A, a second position sensor 42B, and an aperture sensor 42C. The control device 36 controls the entire interchangeable lens 18 according to instructions from the camera unit 16. The control device 36 is, for example, a computer device including a CPU, an NVM, and RAM. The NVM of the control device 36 is, for example, an EEPROM. However, this is only one example; an HDD and / or SSD, etc., can also be used as the NVM of the control device 36 instead of or together with an EEPROM. The RAM of the control device 36 temporarily stores various information and is used as working memory. In the control device 36, the CPU reads necessary programs from the NVM and executes the read programs on the RAM, thereby controlling the entire interchangeable lens 18.

[0084] Furthermore, as an example of control device 36, a device with a computer is listed here, but this is only one example, and devices including ASICs, FPGAs, and / or PLDs can also be applied. Moreover, as control device 36, a device implemented by a combination of hardware and software structures can also be used, for example.

[0085] The first actuator 37 includes a focusing sliding mechanism (generally shown) and a focusing motor (generally shown). A focusing lens 40B is mounted on the focusing sliding mechanism in a manner that allows it to slide along the optical axis OA. Furthermore, the focusing sliding mechanism is connected to the focusing motor, and the focusing sliding mechanism operates by receiving power from the focusing motor, thereby causing the focusing lens 40B to move along the optical axis OA.

[0086] The second actuator 38 includes a zoom sliding mechanism (generally shown) and a zoom motor (generally shown). The zoom lens 40C is mounted on the zoom sliding mechanism in a manner that allows it to slide along the optical axis OA. Furthermore, the zoom motor is connected to the zoom sliding mechanism, and the zoom sliding mechanism operates by receiving power from the zoom motor, thereby causing the zoom lens 40C to move along the optical axis OA.

[0087] Furthermore, an example is given here where a focusing sliding mechanism and a zoom sliding mechanism are separately provided. However, this is only one example, and an integrated sliding mechanism that can achieve both focusing and zooming can also be used. In this case, the power generated by a single motor can be transmitted to the sliding mechanism without using separate focusing and zoom motors.

[0088] The third actuator 39 includes a power transmission mechanism (generally shown) and an aperture motor (generally shown). The aperture 40D has an opening 40D1, and the size of the opening 40D1 is changeable. The opening 40D1 is formed, for example, by a plurality of blades 40D2. The plurality of blades 40D2 are connected to the power transmission mechanism. Furthermore, the aperture motor is connected to the power transmission mechanism, which transmits power from the aperture motor to the plurality of blades 40D2. The plurality of blades 40D2 operate by receiving power from the power transmission mechanism, thereby changing the size of the opening 40D1. By changing the size of the opening 40D1, the aperture amount based on the aperture 40D is changed, thereby adjusting the exposure.

[0089] The focusing motor, zoom motor, and aperture motor are connected to the control device 36, which controls the driving of each motor. In the first embodiment, a stepper motor is used as an example of the focusing motor, zoom motor, and aperture motor. Therefore, the focusing motor, zoom motor, and aperture motor operate synchronously according to commands and pulse signals from the control device 36. Here, an example is shown where the focusing motor, zoom motor, and aperture motor are located on the interchangeable lens 18; however, this is only one example, and at least one of the focusing motor, zoom motor, and aperture motor may also be located on the imaging device body 16. The configuration and / or operating method of the interchangeable lens 18 can be changed as needed.

[0090] The first position sensor 42A detects the position of the focusing lens 40B on the optical axis 0A. A potentiometer can be used as an example of the first position sensor 42A. The detection result of the first position sensor 42A is acquired by the control device 36. The control device 36 adjusts the position of the focusing lens 40B on the optical axis 0A based on the detection result of the first position sensor 42A.

[0091] The second position sensor 42B detects the position of the zoom lens 40C on the optical axis 0A. A potentiometer can be used as an example of the second position sensor 42B. The detection result of the second position sensor 42B can be acquired by the control device 36.

[0092] The aperture sensor 42C detects the size of the aperture 40D1 (i.e., the aperture amount). A potentiometer can be cited as an example of the aperture sensor 42C. The detection result of the aperture sensor 42C can be acquired by the control device 36.

[0093] In the imaging device 10, when the operating mode is imaging mode, the MF mode and AF mode can be selectively set according to the instructions given to the imaging device main body 16. MF mode is a manual focusing operating mode. In MF mode, for example, by the user operating the focus ring 18A, the focusing lens 40B moves along the optical axis OA by a movement corresponding to the operation amount of the focus ring 18A, thereby adjusting the focus position. AF mode performs AF. AF refers to the process of adjusting the focus position according to the signal obtained from the image sensor 20. For example, in AF mode, the imaging device main body 16 calculates the distance between the imaging device 10 and the subject, and the focusing lens 40B moves along the optical axis OA to a position focused on the subject, thereby adjusting the focus position.

[0094] The main body 16 of the camera device includes an image sensor 20, a controller 12, an image memory 46, a UI system device 48, an external I / F 50, a communication I / F 52, a photoelectric conversion element driver 54, and an input / output interface 70. Furthermore, the image sensor 20 includes a photoelectric conversion element 72 and an A / D converter 74.

[0095] The input / output interface 70 is connected to a controller 12, an image memory 46, a UI system device 48, an external I / F 50, a communication I / F 52, a photoelectric conversion element driver 54, and an A / D converter 74. Furthermore, the input / output interface 70 is also connected to a control device 36 for the interchangeable lens 18.

[0096] The controller 12 controls the entire camera device 10. That is, in Figure 2In the example shown, the image memory 46, UI system device 48, external I / F 50, communication I / F 52, photoelectric conversion element driver 54, and control device 36 are controlled by controller 12. Controller 12 includes CPU 62, NVM 64, and RAM 66. CPU 62 is an example of a "processor" according to the technology of this invention, and NVM 64 and / or RAM 66 are examples of a "memory" according to the technology of this invention.

[0097] CPU62, NVM64, and RAM66 are interconnected via bus 68, which is connected to input / output interface 70. Additionally, Figure 2 In the example shown, for ease of illustration, a single bus, designated as bus 68, is depicted. However, bus 68 can also be multiple buses. Bus 68 can be a serial bus or a parallel bus that includes data buses, address buses, and control buses.

[0098] NVM64 is a non-transitory storage medium that stores various parameters and programs. These programs include program 65 (described later, see reference). Figure 5 NVM64 is, for example, an EEPROM. However, this is just one example; HDDs and / or SSDs can also be used as NVM64, either in place of or in conjunction with an EEPROM. Furthermore, RAM66 temporarily stores various information and is used as working memory. CPU62 reads the necessary programs from NVM64 and executes the read programs on RAM66. CPU62 performs image processing based on the programs executed on RAM66.

[0099] The CPU 62 acquires the detection result of the first position sensor 42A through the control device 36, and controls the control device 36 based on the detection result of the first position sensor 42A, thereby adjusting the position of the focusing lens 40B on the optical axis OA. Furthermore, the CPU 62 acquires the detection result of the second position sensor 42B through the control device 36, and controls the control device 36 based on the detection result of the second position sensor 42B, thereby adjusting the position of the zoom lens 40C on the optical axis OA. Moreover, the CPU 62 acquires the detection result of the aperture sensor 42C through the control device 36, and controls the control device 36 based on the detection result of the aperture sensor 42C, thereby adjusting the size of the aperture 40D1.

[0100] A photoelectric conversion element driver 54 is connected to the photoelectric conversion element 72. The photoelectric conversion element driver 54 supplies a shooting timing signal to the photoelectric conversion element 72 according to an instruction from the CPU 62. This shooting timing signal specifies the timing for shooting performed by the photoelectric conversion element 72. The photoelectric conversion element 72 performs reset, exposure, and outputs electrical signals based on the shooting timing signal supplied from the photoelectric conversion element driver 54. Examples of shooting timing signals include, for example, a vertical synchronization signal and a horizontal synchronization signal.

[0101] With the interchangeable lens 18 mounted on the main body 16 of the imaging device, the subject light incident on the imaging lens 40 is imaged onto the light-receiving surface 72A by the imaging lens 40. Under the control of the photoelectric conversion element driver 54, the photoelectric conversion element 72 performs photoelectric conversion on the subject light received by the light-receiving surface 72A and outputs an electrical signal corresponding to the amount of subject light to the A / D converter 74 as image data 73 representing the subject light. Specifically, the A / D converter 74 reads the image data 73 from the photoelectric conversion element 72 in exposure sequence for each horizontal line, one frame at a time.

[0102] The A / D converter 74 digitizes the analog camera data 73 read from the photoelectric conversion element 72. The camera data 73 digitized by the A / D converter 74 is so-called RAW image data, representing an image in which R pixels, G pixels, and B pixels are arranged in a mosaic pattern. Furthermore, in the first embodiment, as an example, the number of bits (i.e., bit length) of each pixel including R pixels, B pixels, and G pixels in the RAW image data is 14 bits.

[0103] The A / D converter 74 outputs digitized image data 73 to the image memory 46 and stores the image data 73 in the image memory 46. The CPU 62 performs image processing (e.g., white balance processing and / or color correction) on the image data 73 in the image memory 46. The CPU 62 generates motion image data 80 based on the image data 73. The CPU 62 stores the generated motion image data 80 in an NVM 64. Furthermore, the NVM 64 is an example of a "non-transitory storage medium" involved in the technology of this invention.

[0104] UI system device 48 includes a display 28. CPU 62 displays images on the display 28 based on dynamic image data 80. Furthermore, CPU 62 displays various information on the display 28.

[0105] Furthermore, the UI system device 48 includes a receiving device 76 for receiving instructions from the user. The receiving device 76 includes a touch panel 30 and a hard key section 78. The hard key section 78 includes an indicator key 26 (see reference). Figure 1The CPU 62 operates based on various instructions received via the touch panel 30. Furthermore, while the hard key unit 78 is included in the UI system device 48, the technology of the present invention is not limited thereto; for example, the hard key unit 78 may also be connected to an external I / F 50.

[0106] The external I / F50 controls the exchange of various information with devices located outside the camera device 10 (hereinafter also referred to as "external devices"). An example of an external I / F50 is a USB interface. External devices such as smart devices, personal computers, servers, USB storage devices, memory cards, and / or printers (generally shown in the diagram) are connected directly or indirectly to the USB interface.

[0107] The communication I / F52 is connected to a network (not shown). The communication I / F52 controls the exchange of information between a communication device (not shown) such as a server on the network and the controller 12. For example, the communication I / F52 sends information corresponding to a request from the controller 12 to the communication device via the network. Furthermore, the communication I / F52 receives information sent from the communication device and outputs the received information to the controller 12 via the input / output interface 70.

[0108] As an example, such as Figure 3 As shown, multiple photosensitive pixels 72B are arranged in a two-dimensional pattern on the light-receiving surface 72A of the photoelectric conversion element 72. Each photosensitive pixel 72B is equipped with a color filter (not shown) and a microlens 72C. Figure 3 In this configuration, a direction parallel to the light-receiving surface 72A (e.g., the row direction of a plurality of photosensitive pixels 72B arranged in a two-dimensional configuration) is designated as the X direction, and a direction orthogonal to the X direction (e.g., the column direction of a plurality of photosensitive pixels 72B arranged in a two-dimensional configuration) is designated as the Y direction. The plurality of photosensitive pixels 72B are arranged along both the X and Y directions. Each photosensitive pixel 72B includes an independent pair of photodiodes PD1 and PD2. A first beam (e.g., a beam representing the subject (hereinafter also referred to as the "subject beam")) is obtained by the pupil of the transmission imaging lens 40. Figure 2 The light beam from the first pupil region of the image is incident on the photodiode PD1, and the second light beam obtained by the subject beam being split by the pupil (e.g., through the imaging lens 40 (reference)) is also incident on the imagediode PD1. Figure 2 The light beam from the second pupil region of the first beam is incident on photodiode PD2. Photodiode PD1 performs photoelectric conversion of the first beam. Photodiode PD2 performs photoelectric conversion of the second beam.

[0109] As an example, the photoelectric conversion element 72 is a photoelectric conversion element that uses a pair of photodiodes PD1 and PD2 to represent the image plane phase difference for one photosensitive pixel 72B. As an example, in the photoelectric conversion element 72, all photosensitive pixels 72B have the function of outputting data related to image capture and phase difference. The photoelectric conversion element 72 outputs non-phase difference pixel data 73A by combining a pair of photodiodes PD1 and PD2 into one photosensitive pixel 72B. Furthermore, the photoelectric conversion element 72 outputs phase difference pixel data 73B by detecting signals from a pair of photodiodes PD1 and PD2 respectively. That is, all photosensitive pixels 72B provided in the photoelectric conversion element 72 are so-called phase difference pixels.

[0110] Photosensitive pixel 72B is a pixel that selectively outputs non-phase difference pixel data 73A and phase difference pixel data 73B. Non-phase difference pixel data 73A is pixel data obtained by photoelectric conversion across the entire area of ​​photosensitive pixel 72B, and phase difference pixel data 73B is pixel data obtained by photoelectric conversion across a portion of photosensitive pixel 72B. Here, "the entire area of ​​photosensitive pixel 72B" includes the light-receiving areas of photodiodes PD1 and PD2. Furthermore, "a portion of the area of ​​photosensitive pixel 72B" is either the light-receiving area of ​​photodiode PD1 or the light-receiving area of ​​photodiode PD2. Photosensitive pixel 72B is an example of a "phase difference pixel" according to the technology of this invention.

[0111] Furthermore, non-phase difference pixel data 73A can also be generated based on the phase difference pixel data 73B. For example, non-phase difference pixel data 73A can be generated by adding the phase difference pixel data 73B to the pixel signals corresponding to each pair of photodiodes PD1 and PD2. Moreover, the phase difference pixel data 73B may include only the data output from one of the pair of photodiodes PD1 and PD2. For example, when the phase difference pixel data 73B includes only the data output from photodiode PD1, the data output from photodiode PD2 can be generated by subtracting the phase difference pixel data 73B from the non-phase difference pixel data 73A for each pixel.

[0112] The image data 73 includes image data 81 and phase difference pixel data 73B. Image data 81 is generated based on non-phase difference pixel data 73A. For example, image data 81 is obtained by analog A / D conversion of non-phase difference pixel data 73A. That is, image data 81 is data obtained by digitizing non-phase difference pixel data 73A output from photoelectric conversion element 72. CPU 62 obtains digitized image data 73 from A / D converter 74 and obtains distance data 82 based on the obtained image data 73. For example, CPU 62 obtains phase difference pixel data 73B from image data 73 and generates distance data 82 based on the obtained phase difference pixel data 73B. Distance data 82 is distance data, which is the distance between multiple subjects in the image area captured by image sensor 20 and the camera device 10. The distance data represents the distance obtained for each photosensitive pixel 72B (i.e., the distance between objects in the image area and the camera device 10). Distance data is one example of the "distance data" involved in the technology of this invention.

[0113] As an example, Figure 4 An example of the relationship between the camera lens 40 and the depth of field is shown. Figure 4 In the example shown, the imaging lens 40 is schematically shown as a lens. The image distance is the distance along the depth direction from the principal point of the imaging lens 40 to the light-receiving surface 72A of the photoelectric conversion element 72. This is achieved by using a predetermined formula or data matching table, and according to the first position sensor 42A (reference). Figure 2 ) Detected focusing lens 40B (reference) Figure 2 The position of the image is used to obtain the image distance. The depth direction is parallel to the optical axis OA.

[0114] The object distance is the distance along the depth direction from the principal point of the imaging lens 40 to the subject in focus. The subject in focus is located at the focus position. The focus position is the optimal focus position. When the object distance is L, the object distance L is calculated by the following formula (1). However, t is the image distance, which, as described above, is obtained based on the position detection result of the focusing lens 40B of the first position sensor 42A. And f is the focal length of the focusing lens 40B, which is a known fixed value. The focal length f is an example of the "focal length in the imaging lens" involved in the technology of the present invention, and the object distance L is an example of the "object distance in the imaging lens" involved in the technology of the present invention.

[0115] [Formula 1]

[0116]

[0117] The depth of field of the camera device 10 includes a front depth of field and a rear depth of field. When the depth of field is D, the depth of field D is calculated using the following formula (2). Furthermore, when the front depth of field is D1, the front depth of field D1 is calculated using the following formula (3). Furthermore, when the rear depth of field is D2, the rear depth of field D2 is calculated using the following formula (4). Wherein, F is the aperture 40D (reference). Figure 2 The aperture value (F-number) is the aperture value, and δ is the diameter of the circle of confusion. The aperture value F is equivalent to the aperture value passing through the aperture sensor 42C (reference). Figure 2 The detected aperture amount. The permissible circle of confusion diameter δ is a known fixed value. The permissible circle of confusion diameter δ is the photosensitive pixels 72B arranged on the light-receiving surface 72A (reference). Figure 3 The spacing between the arrays is approximately 1 to 2 times that of the array, allowing for a blur of about one pixel. The aperture value F is an example of the "aperture value in a camera lens" involved in the technology of this invention, and the permissible circle of confusion diameter δ is an example of the "permissible circle of confusion diameter in a camera lens" involved in the technology of this invention.

[0118] [Formula 2]

[0119] D = D1 + D2 ... (2)

[0120] [Formula 3]

[0121]

[0122] [Formula 4]

[0123]

[0124] The near point distance is the distance along the depth direction from the principal point of the camera lens 40 to the near point of the depth of field. When the near point distance is L1, the near point distance L1 is calculated using the following formula (5). The far point distance is the distance along the depth direction from the principal point of the camera lens 40 to the far point of the depth of field. When the far point distance is L2, the far point distance L2 is calculated using the following formula (6).

[0125] [Formula 5]

[0126]

[0127] [Formula 6]

[0128]

[0129] The above equations (1) to (6) are used in the dynamic image generation process described below.

[0130] As an example, such as Figure 5As shown, program 65 is stored in NVM 64. Program 65 is an example of a "program" according to the technology of this invention. CPU 62 reads program 65 from NVM 64 and executes the read program 65 in RAM 66. CPU 62 performs operations based on camera data 73 (see reference) according to program 65 executed in RAM 66. Figure 2 Generate 80 dynamic image data (reference) Figure 2 The dynamic image generation process is implemented by the CPU 62 according to program 65, which operates as the following units: first camera control unit 100, first dynamic image data generation unit 102, second camera control unit 104, distance data acquisition unit 106, object distance acquisition unit 108, near point distance acquisition unit 110, far point distance acquisition unit 112, first subject determination unit 114, second subject determination unit 118, second dynamic image data generation unit 120, dynamic image data output unit 122, and dynamic image data storage control unit 124.

[0131] As an example, Figure 6 An example is shown where a first subject 90A, a second subject 90B, and a third subject 92 exist within the imaging area captured by the image sensor 20. One example of the first subject 90A, the second subject 90B, and the third subject 92 is a human. The first subject 90A, the second subject 90B, and the third subject 92 are examples of "multiple subjects" according to the technology of this invention. The first subject 90A, the second subject 90B, and the third subject 92 are staggered along the depth direction of the imaging device 10. Furthermore, when viewed from above, the first subject 90A, the second subject 90B, and the third subject 92 are staggered along a direction orthogonal to the depth direction of the imaging device 10 (i.e., the left-right direction of the imaging device 10). Hereinafter, when it is not necessary to distinguish between the first subject 90A, the second subject 90B, and the third subject 92, the first subject 90A, the second subject 90B, and the third subject 92 will be referred to as subjects.

[0132] A portion of the face of the first subject 90A exists within the near point distance. That is, a portion of the face of the first subject 90A exists at the near point of the depth of field, at a distance from the principal point of the imaging lens 40. Hereinafter, the portion of the face of the first subject 90A existing within the near point distance will be referred to as the first boundary subject 91A. A portion of the face of the second subject 90B exists within the far point distance. That is, a portion of the face of the second subject 90B exists at the far point of the depth of field, at a distance from the principal point of the imaging lens 40. Hereinafter, the portion of the face of the second subject 90B existing within the far point distance will be referred to as the second boundary subject 91B. The third subject 92 exists between the first subject 90A and the second subject 90B. A portion of the face of the third subject 92 exists within the object distance. That is, a portion of the face of the third subject 92 exists at the focus position, at a distance from the principal point of the imaging lens 40. Hereinafter, a portion of the face of the third subject 92, which exists within the object distance, will be referred to as the focus subject 93. Furthermore, when it is not necessary to distinguish between the first boundary subject 91A and the second boundary subject 91B, the first boundary subject 91A and the second boundary subject 91B will be referred to as the boundary subject 91.

[0133] The near and far points of the depth of field are examples of the "depth of field boundary" as described in the present invention. The near point of the depth of field is an example of the "first boundary located on the near point side of the depth of field" as described in the present invention, and the far point of the depth of field is an example of the "second boundary located on the far point side of the depth of field" as described in the present invention. The near point distance is an example of the "distance between boundary portions" and the "distance between the first boundary portions" as described in the present invention, and the far point distance is an example of the "distance between boundary portions" and the "distance between the second boundary portions" as described in the present invention. The first boundary subject 91A is an example of the "first boundary subject" as described in the present invention, and the second boundary subject 91B is an example of the "second boundary subject" as described in the present invention. The focusing subject 93 is an example of the "focusing subject" as described in the present invention.

[0134] The following is based on Figure 6 The following components will be described in the example shown: first camera control unit 100, first motion image data generation unit 102, second camera control unit 104, distance data acquisition unit 106, object distance acquisition unit 108, near point distance acquisition unit 110, far point distance acquisition unit 112, first subject determination unit 114, second subject determination unit 118, second motion image data generation unit 120, motion image data output unit 122, and motion image data storage control unit 124.

[0135] The first camera control unit 100 controls the photoelectric conversion element 72 to output non-phase difference pixel data 73A. Specifically, the first camera control unit 100 outputs a first camera command to the photoelectric conversion element driver 54. This first camera command is used to output a first camera timing signal as a camera timing signal to the photoelectric conversion element 72. The first camera timing signal is a camera timing signal used to cause the photoelectric conversion element 72 to output non-phase difference pixel data 73A. Each photosensitive pixel 72B of the photoelectric conversion element 72 performs photoelectric conversion over its entire area according to the first camera timing signal, outputting non-phase difference pixel data 73A. The photoelectric conversion element 72 outputs the non-phase difference pixel data 73A output from each photosensitive pixel 72B to the A / D converter 74. The A / D converter 74 digitizes the non-phase difference pixel data 73A output from each photosensitive pixel 72B to generate image data 81.

[0136] The first dynamic image data generation unit 102 acquires image data 81 from the A / D converter 74. Image data 81 represents data representing images obtained by the image sensor 20 of the first subject 90A, the second subject 90B, and the third subject 92. Image data 81 is an example of "image data" according to the technology of this invention. Then, the first dynamic image data generation unit 102 generates first dynamic image data (i.e., dynamic image data for one frame component) based on the image data 81.

[0137] The second camera control unit 104 controls the photoelectric conversion element 72 to output non-phase difference pixel data 73B. Specifically, the second camera control unit 104 outputs a second camera command to the photoelectric conversion element driver 54. This second camera command is used to output a second camera timing signal as a camera timing signal to the photoelectric conversion element 72. The second camera timing signal is a camera timing signal used to cause the photoelectric conversion element 72 to output phase difference pixel data 73B. Each photosensitive pixel 72B of the photoelectric conversion element 72 performs photoelectric conversion in a portion of the photosensitive pixel 72B according to the second camera timing signal, outputting phase difference pixel data 73B. The photoelectric conversion element 72 outputs the non-phase difference pixel data 73B obtained from each photosensitive pixel 72B to the A / D converter 74. The A / D converter 74 digitizes the phase difference pixel data 73B and outputs the digitized phase difference pixel data 73B to the distance data acquisition unit 106.

[0138] The distance data acquisition unit 106 acquires distance data 82. Specifically, the distance data acquisition unit 106 acquires phase difference pixel data 73B from the A / D converter 74, and generates distance data 82 corresponding to each photosensitive pixel 72B (that is, data representing the distance between the object in the imaging area and each photosensitive pixel 72B) based on the acquired phase difference pixel data 73B.

[0139] The object distance acquisition unit 108 acquires the image distance t and focal length f. Then, the object distance acquisition unit 108 calculates the object distance L based on the image distance t and focal length f using the above formula (1), and acquires the object distance L. In this case, the object distance acquisition unit 108 acquires the object distance L based on the focusing lens 40B (reference) detected by the first position sensor 42A. Figure 2 The object distance acquisition unit 108 acquires, for example, the focal length f pre-stored in the NVM64.

[0140] The near distance acquisition unit 110 acquires the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ. Then, the near distance acquisition unit 110 calculates the near distance L1 using the above formula (5) based on the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ, and acquires the near distance L1. In this case, the near distance acquisition unit 110 acquires the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ with the following key points: That is, the near distance acquisition unit 110 acquires the object distance L acquired by the object distance acquisition unit 108. Furthermore, the near distance acquisition unit 110 acquires, for example, the focal length f pre-stored in the NVM 64. Furthermore, the near distance acquisition unit 110 acquires the aperture value F corresponding to the aperture amount detected by the aperture amount sensor 42C. Furthermore, for example, when the user assigns the permissible circle of confusion diameter δ to the receiving device 76, the near distance acquisition unit 110 acquires the permissible circle of confusion diameter δ received by the receiving device 76.

[0141] The far-point distance acquisition unit 112 acquires the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ. Then, the far-point distance acquisition unit 112 calculates the far-point distance L2 using the above formula (6) based on the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ, and acquires the far-point distance L2. In this case, the far-point distance acquisition unit 112 acquires the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ with the following key points: That is, the far-point distance acquisition unit 112 acquires the object distance L acquired by the object distance acquisition unit 108. Furthermore, the far-point distance acquisition unit 112 acquires, for example, the focal length f pre-stored in the NVM 64. Furthermore, the far-point distance acquisition unit 112 acquires the aperture value F corresponding to the aperture amount detected by the aperture amount sensor 42C. Furthermore, for example, when the user assigns the permissible circle of confusion diameter δ to the receiving device 76, the far-point distance acquisition unit 112 acquires the permissible circle of confusion diameter δ received by the receiving device 76.

[0142] In addition, the near distance acquisition unit 110 and the far distance acquisition unit 112 can, for example, obtain the allowable dispersion circle diameter δ from information pre-stored in the NVM64.

[0143] The first subject determination unit 114 determines whether a first boundary subject 91A exists within the near distance based on the distance data acquired by the distance data acquisition unit 106 (in other words, whether a first boundary subject 91A exists at the near point). Specifically, the first subject determination unit 114 compares the distance data for each photosensitive pixel 72B (see reference) Figure 3 The first subject determination unit 114 determines whether the distances obtained for each photosensitive pixel 72B include a distance equal to the near point distance, based on the distances and the near point distances. The photosensitive pixel 72B with a distance equal to the near point distance is the photosensitive pixel 72B corresponding to the first boundary subject 91A. Then, when the distances obtained for each photosensitive pixel 72B include a distance equal to the near point distance, the first subject determination unit 114 determines that the first boundary subject 91A exists within the near point distance. On the other hand, when the distances obtained for each photosensitive pixel 72B do not include a distance equal to the near point distance, the first subject determination unit 114 determines that the first boundary subject 91A does not exist within the near point distance.

[0144] As an example, in Figure 6 In the example shown, a first boundary subject 91A exists at a near distance. When a first boundary subject 91A exists at a near distance, the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance.

[0145] The second subject determination unit 118 determines whether a second boundary subject 91B exists within the far point distance (in other words, whether a second boundary subject 91B exists at the far point) based on the distance data acquired by the distance data acquisition unit 106. Specifically, the second subject determination unit 118 compares the distance data for each photosensitive pixel 72B (see reference) Figure 3 The second subject determination unit 118 determines whether the distances obtained for each photosensitive pixel 72B include a distance equal to the distance to the far point, based on the distances obtained and the distance to the far point. The photosensitive pixel 72B with a distance equal to the distance to the far point is the photosensitive pixel 72B corresponding to the second boundary subject 91B. Then, when the distances obtained for each photosensitive pixel 72B include a distance equal to the distance to the far point, the second subject determination unit 118 determines that a second boundary subject 91B exists within the distance to the far point. On the other hand, when the distances obtained for each photosensitive pixel 72B do not include a distance equal to the distance to the far point, the second subject determination unit 118 determines that a second boundary subject 91B does not exist within the distance to the far point.

[0146] As an example, in Figure 6 In the example shown, a second boundary subject 91B exists at a distance from the far point. When a second boundary subject 91B exists at a distance from the far point, the second subject determination unit 118 determines that a second boundary subject 91B exists at a distance from the far point.

[0147] When the first subject determination unit 114 determines that a first boundary subject 91A exists within a near distance, the second dynamic image data generation unit 120 generates first boundary data representing the region of the first boundary subject 91A existing within the near distance based on distance data. Specifically, the second dynamic image data generation unit 120 generates the first boundary data based on the following points.

[0148] That is, the second dynamic image data generation unit 120 acquires first region data representing the region of the first boundary subject 91A based on distance data, wherein the first boundary subject 91A exists for each photosensitive pixel 72B (reference). Figure 3 Within the distance obtained, the distance is equal to the nearest point distance. The first region data is represented, for example, by the address of the photosensitive pixel 72B. That is, the second dynamic image data generation unit 120 obtains the address of the photosensitive pixel 72B that is equal to the nearest point distance from the plurality of photosensitive pixels 72B based on the distance data as the first region data. The address of the photosensitive pixel 72B is a coordinate determined for each photosensitive pixel 72B. The coordinate determined for each photosensitive pixel 72B refers to, for example, two-dimensional coordinates representing the longitudinal and lateral (e.g., row direction and column direction) of the photoelectric conversion element 72.

[0149] Next, the second dynamic image data generation unit 120 generates first boundary data based on the first region data. The first boundary data is data that represents the region of the first boundary subject 91A within the near distance (i.e., the region represented by the first region data) in a first manner on the image displayed on the display 28 according to the first dynamic image data. An example of the data that represents the region of the first boundary subject 91A in the first manner is data used for first image processing, which marks the region of the first boundary subject 91A within the near distance on the image displayed on the display 28 in a manner that distinguishes it from other regions.

[0150] As an example, the first image processing involves assigning a first predetermined color to the pixels (i.e., the pixels corresponding to the address of the photosensitive pixel 72B represented by the first region data) that correspond to the region of the first boundary subject 91A existing within a near distance of the pixels constituting the image displayed on the display 28. The process of assigning the first predetermined color involves, for example, replacing the pixel's signal value with a value corresponding to the first predetermined color. For example, when assigning red to a pixel, the pixel's red (R) value, green (G) value, and blue (B) value are set to 255, 0, and 0, respectively. The first predetermined color can be achromatic or chromatic. The first predetermined color is, for example, red, blue, or yellow. The first boundary data is an example of "boundary data" and "first boundary data" according to the technology of this invention. The first image processing is an example of "image processing" according to the technology of this invention. The first region data is an example of "region data" according to the technology of this invention.

[0151] Furthermore, when the second subject determination unit 118 determines that a second boundary subject 91B exists within a distance from the far point, the second motion image data generation unit 120 generates second boundary data representing the region of the second boundary subject 91B existing within the distance from the far point based on the distance data. Specifically, the second motion image data generation unit 120 generates the second boundary data based on the following points.

[0152] That is, the second dynamic image data generation unit 120 acquires second region data representing the region of the second boundary subject 91B based on distance data, wherein the second boundary subject 91B exists for each photosensitive pixel 72B (reference). Figure 3 The second region data is obtained from the distance data and is located within the distance equal to the distance to the far point. For example, the address of the photosensitive pixel 72B is used to represent this second region data. That is, the second dynamic image data generation unit 120 obtains the address of the photosensitive pixel 72B that is at a distance equal to the distance to the far point from the plurality of photosensitive pixels 72B, as the second region data, based on the distance data.

[0153] Next, the second dynamic image data generation unit 120 generates second boundary data based on the second region data. The second boundary data represents the region of the second boundary subject 91B existing within a distance from the far point (i.e., the region represented by the second region data) in a second manner different from the first manner, representing the image displayed on the display 28 based on the first dynamic image data. One example of this second manner of representing the region of the second boundary subject 91B is data used for second image processing, which marks the region of the second boundary subject 91B existing within a distance from the far point in the image displayed on the display 28 based on the first dynamic image data in a manner distinguishable from other regions.

[0154] As an example, the second image processing involves assigning a second predetermined color to the pixels (i.e., the pixels corresponding to the address of the photosensitive pixel 72B represented by the second region data) that correspond to the region of the second boundary subject 91B existing within a distance from the far point of the multiple pixels constituting the image displayed on the display 28. The second predetermined color is a color different from the first predetermined color. The second predetermined color can be achromatic or chromatic. For example, the second predetermined color can be red, blue, or yellow. Hereinafter, when it is not necessary to distinguish between the first image processing and the second image processing, the first image processing and the second image processing will be referred to as boundary data. Furthermore, when it is not necessary to distinguish between the first predetermined color and the second predetermined color, the first predetermined color and the second predetermined color will be referred to as predetermined colors. The second boundary data is an example of "boundary data" and "second boundary data" according to the technology of this invention. The second image processing is an example of "image processing" according to the technology of this invention. The second region data is an example of "region data" according to the technology of this invention.

[0155] Then, when the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance and the second subject determination unit 118 determines that a second boundary subject 91B exists at a far distance, the second motion image data generation unit 120 generates second motion image data including the first boundary data and the second boundary data based on the first motion image data. Specifically, the second motion image data generation unit 120 performs first image processing on the pixels corresponding to the region of the first boundary subject 91A existing at a near distance among the plurality of pixels constituting the image displayed on the display 28 according to the first motion image data. Similarly, the second motion image data generation unit 120 performs second image processing on the pixels corresponding to the region of the second boundary subject 91B existing at a far distance among the plurality of pixels constituting the image displayed on the display 28 according to the first motion image data. Therefore, the second dynamic image data is generated by the second dynamic image data generation unit 120. The second dynamic image data represents an image in which a first predetermined color is attached to the pixels corresponding to the region of the first boundary subject 91A that exists within the near point distance, and a second predetermined color is attached to the pixels corresponding to the region of the second boundary subject 91B that exists within the far point distance.

[0156] Furthermore, when generating second dynamic image data including first boundary data and second boundary data, the second dynamic image data generation unit 120 can generate the second dynamic image data in stages. That is, for example, the second dynamic image data generation unit 120 can generate temporary dynamic image data including first boundary data based on the first dynamic image data, and then generate second dynamic image data including second boundary data based on the temporary dynamic image data.

[0157] The second dynamic image data is an example of the "dynamic image data" involved in the technology of this invention. Hereinafter, when it is not necessary to distinguish between the first boundary data and the second boundary data, the first boundary data and the second boundary data will be referred to as boundary data.

[0158] When the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance and the second subject determination unit 118 determines that a second boundary subject 91B exists at a far distance, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the first boundary data and the second boundary data) to the display 28 as display motion image data. The display 28 displays an image based on the display motion image data.

[0159] The motion image data storage control unit 124 stores the first motion image data generated by the first motion image data generation unit 102 as recording motion image data in the NVM 64. Furthermore, while an example of storing recording motion image data in the NVM 64 will be described here, the motion image data storage control unit 124 can also be connected to an external I / F 50 (see reference). Figure 2 The dynamic image data is stored in a memory card and / or USB memory, etc., for connection.

[0160] As an example, Figure 7 The text is a jumbled mix of Chinese characters and symbols, making it impossible to translate coherently. It appears to be a collection Figure 6 In the example shown, the generated second dynamic image data is displayed as image 200 on monitor 28. Figure 7 In the image 200 shown, the region of the first boundary subject 91A and the region other than the region of the first boundary subject 91A are shown in a distinct manner. Furthermore, in Figure 7 In the image 200 shown, the region of the second boundary subject 91B and the region other than the region of the second boundary subject 91B are represented in a distinguishing manner. That is, as an example, in Figure 7 In the image 200 shown, the region of the first boundary subject 91A is represented by a first predetermined color, and the region of the second boundary subject 91B is represented by a second predetermined color. Furthermore, in Figure 7 In the example shown, the regions serving as the first boundary subject 91A and the second boundary subject 91B are depicted as annular regions; however, this is merely an example, and other shapes are also possible. Furthermore, the regions of the first boundary subject 91A and the second boundary subject 91B can also be regions of different shapes. Moreover, the same pattern can be applied to the regions of the first boundary subject 91A and the second boundary subject 91B, or different patterns (e.g., dots and grids) can be applied.

[0161] Display 28 is an example of "first display," "second display," "third display," "fourth display," and "fifth display" according to the technology of this invention. In this invention, for convenience, display 28 is referred to as "first display," "second display," "third display," "fourth display," and "fifth display." Image 200 displayed on display 28 is an example of "first image," "second image," and "third image" according to the technology of this invention. In this invention, for convenience, image 200 is referred to as "first image," "second image," and "third image." The plurality of pixels constituting image 200 displayed on display 28 are examples of "first pixel" and "second pixel" according to the technology of this invention. In this invention, for convenience, the plurality of pixels constituting image 200 displayed on display 28 are referred to as "first pixel" and "second pixel."

[0162] As an example, Figure 8 An example is shown where a first subject 90A and a third subject 92 exist within the image area captured by image sensor 20. Figure 8 In the example shown, the positions of the first subject 90A and the third subject 92 are... Figure 6 The example shown is the same.

[0163] The following is based on Figure 8 The following components will be described in the example shown: first camera control unit 100, first motion image data generation unit 102, second camera control unit 104, distance data acquisition unit 106, object distance acquisition unit 108, near point distance acquisition unit 110, far point distance acquisition unit 112, first subject determination unit 114, second subject determination unit 118, second motion image data generation unit 120, motion image data output unit 122, and motion image data storage control unit 124.

[0164] The operation and control of the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near-point distance acquisition unit 110, the far-point distance acquisition unit 112, the first subject determination unit 114, and the dynamic image data storage control unit 124 Figure 6 The example shown is the same. In Figure 8 In the example shown, the operation of the second subject determination unit 118, the second motion image data generation unit 120, and the motion image data output unit 122 is similar to... Figure 6 The examples shown are different. Below, for... Figure 8The example shown illustrates the operation of the second subject determination unit 118, the second motion image data generation unit 120, and the motion image data output unit 122. Figure 6 The examples shown are different.

[0165] As an example, in Figure 8 In the example shown, there is no second boundary subject 91B within the far point distance (reference). Figure 6 When there is no second boundary subject 91B within the far point distance, the second subject determination unit 118 determines that there is no second boundary subject 91B within the far point distance.

[0166] When the first subject determination unit 114 determines that a first boundary subject 91A exists within a near distance and the second subject determination unit 118 determines that a second boundary subject 91B does not exist within a far distance, the second motion image data generation unit 120 generates second motion image data including the first boundary data based on the first motion image data. Specifically, the second motion image data generation unit 120 performs first image processing on the pixels corresponding to the region of the first boundary subject 91A existing within a near distance among the plurality of pixels constituting the image displayed on the display 28 based on the first motion image data. Thus, the second motion image data is generated by the second motion image data generation unit 120, which represents an image in which a first predetermined color is applied to the pixels corresponding to the region of the first boundary subject 91A existing within a near distance.

[0167] When the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance and the second subject determination unit 118 determines that a second boundary subject 91B does not exist at a far distance, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the first boundary data) to the display 28 as display motion image data (e.g., data representing a real-time preview image). The display 28 displays an image (e.g., a real-time preview image) based on the display motion image data.

[0168] As an example, Figure 9 The text is a jumbled mix of Chinese characters and symbols, making it impossible to translate coherently. It appears to be a collection Figure 8 In the example shown, the generated second dynamic image data is displayed as image 200 on monitor 28. Figure 9 In the image 200 shown, the region of the first boundary subject 91A and the region other than the region of the first boundary subject 91A are shown in a distinguishing manner. That is, as an example, in Figure 9 In the image 200 shown, the region of the first boundary subject 91A is represented by a first predetermined color.

[0169] As an example, Figure 10 An example is shown where a second subject 90B and a third subject 92 exist within the image area captured by image sensor 20. Figure 10 In the example shown, the positions of the second subject 90B and the third subject 92 are... Figure 6 The example shown is the same.

[0170] The following is based on Figure 10 The following components will be described in the example shown: first camera control unit 100, first motion image data generation unit 102, second camera control unit 104, distance data acquisition unit 106, object distance acquisition unit 108, near point distance acquisition unit 110, far point distance acquisition unit 112, first subject determination unit 114, second subject determination unit 118, second motion image data generation unit 120, motion image data output unit 122, and motion image data storage control unit 124.

[0171] The operation and control of the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near-point distance acquisition unit 110, the far-point distance acquisition unit 112, the second subject determination unit 118, and the dynamic image data storage control unit 124. Figure 6 The example shown is the same. Figure 10 In the example shown, the operation of the first subject determination unit 114, the second motion image data generation unit 120, and the motion image data output unit 122 is similar to... Figure 6 The examples shown are different. Below, for... Figure 10 The example shown illustrates the operation of the first subject determination unit 114, the second motion image data generation unit 120, and the motion image data output unit 122. Figure 6 The examples shown are different.

[0172] As an example, in Figure 10 In the example shown, there is no first boundary subject 91A within the near point distance (reference). Figure 6 When the first boundary subject 91A is not present within the near point distance, the first subject determination unit 114 determines that the first boundary subject 91A is not present within the near point distance.

[0173] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance and the second subject determination unit 118 determines that there is a second boundary subject 91B within the far distance, the second motion image data generation unit 120 generates second motion image data including the second boundary data based on the first motion image data. Specifically, the second motion image data generation unit 120 performs second image processing on the pixels corresponding to the region of the second boundary subject 91B present within the far distance from the plurality of pixels constituting the image displayed on the display 28 based on the first motion image data. Thus, the second motion image data generated by the second motion image data generation unit 120 represents an image in which a second predetermined color is applied to the pixels corresponding to the region of the second boundary subject 91B present within the far distance.

[0174] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance and the second subject determination unit 118 determines that there is a second boundary subject 91B within the far distance, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the second boundary data) to the display 28 as display motion image data. The display 28 displays an image based on the display motion image data.

[0175] As an example, Figure 11 The text is a jumbled mix of Chinese characters and symbols, making it impossible to translate coherently. It appears to be a collection Figure 10 In the example shown, the generated second dynamic image data is displayed as image 200 on monitor 28. Figure 11 In the image 200 shown, the region of the second boundary subject 91B and the region other than the region of the second boundary subject 91B are represented in a distinguishing manner. That is, as an example, in Figure 11 In the image 200 shown, the region of the second boundary subject 91B is represented by a second predetermined color.

[0176] In particular, although not shown in the figure, when the first subject 90A and the second subject 90B are not present in the image area captured by the image sensor 20, the first subject determination unit 114 determines that the first boundary subject 91A is not present at the near point distance and the second subject determination unit 118 determines that the second boundary subject 91B is not present at the far point distance.

[0177] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance and the second subject determination unit 118 determines that there is no second boundary subject 91B within the far distance, the operation of the second motion image data generation unit 120 is skipped. That is, the generation of second motion image data by the second motion image data generation unit 120 is not performed, and the processing performed by the motion image data output unit 122 is performed instead.

[0178] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance and the second subject determination unit 118 determines that there is no second boundary subject 91B within the far distance, the motion image data output unit 122 outputs the first motion image data generated by the first motion image data generation unit 102 to the display 28 as display motion image data. The display 28 displays an image based on the display motion image data. In this case, the first subject 90A and the second subject 90B are not displayed as images in the image displayed on the display 28.

[0179] Next, refer to Figure 12A and Figure 12B The operation of the camera device 10 according to the first embodiment will be explained. Figure 12A and Figure 12B An example of the flow of the motion image generation process according to the first embodiment is shown. When the operating mode of the camera device 10 is the camera mode, the motion image generation process is performed.

[0180] exist Figure 12A In the dynamic image generation process shown, firstly in step ST10, the first camera control unit 100 causes the photoelectric conversion element 72 to output non-phase difference pixel data 73A. After the processing in step ST10 is performed, the dynamic image generation process proceeds to step ST11.

[0181] In step ST11, the first dynamic image data generation unit 102 acquires image data 81. Then, the first dynamic image data generation unit 102 generates first dynamic image data based on the image data 81. After the processing in step ST11 is performed, the dynamic image generation process proceeds to step ST12.

[0182] In step ST12, the second camera control unit 104 causes the photoelectric conversion element 72 to output phase difference pixel data 73B. After the processing in step ST12 is performed, the motion image generation process is transferred to step ST13.

[0183] In step ST13, the distance data acquisition unit 106 acquires distance data. After the processing in step ST13 is performed, the dynamic image generation process proceeds to step ST14.

[0184] In step ST14, the object distance acquisition unit 108 acquires the object distance L based on the image distance t and the focal length f. After the processing in step ST14 is performed, the image generation process proceeds to step ST15.

[0185] In step ST15, the near distance acquisition unit 110 acquires the near distance L1 based on the object distance L, focal length f, aperture value F, and allowable circle of confusion diameter δ. After the processing in step ST15 is performed, the image generation process proceeds to step ST16.

[0186] In step ST16, the near-point distance acquisition unit 112 acquires the far-point distance L2 based on the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ. After the processing in step ST16 is executed, the dynamic image generation process proceeds. Figure 12B Step ST17 is shown.

[0187] In step ST17, the first subject determination unit 114 determines the subject based on the conditions set in step ST13 (refer to step ST14). Figure 12A The distance data obtained in step ST17 determines whether a first boundary subject 91A exists within the near distance. In step ST17, if the first boundary subject 91A does not exist within the near distance, the determination is rejected, and the motion image generation process proceeds to step ST23. In step ST17, if the first boundary subject 91A exists within the near distance, the determination is affirmative, and the motion image generation process proceeds to step ST18.

[0188] In step ST18, the second subject determination unit 118 determines whether a second boundary subject 91B exists within the far point distance based on the distance data acquired in step ST13. If, in step ST18, the second boundary subject 91B does not exist within the far point distance, the determination is rejected, and the motion image generation process proceeds to step ST22. If, in step ST18, the second boundary subject 91B exists within the far point distance, the determination is affirmed, and the motion image generation process proceeds to step ST19.

[0189] In step ST19, the second dynamic image data generation unit 120 generates first boundary data representing the region of the first boundary subject 91A within the near distance, based on the distance data acquired in step ST13. Furthermore, the second dynamic image data generation unit 120 generates second boundary data representing the region of the second boundary subject 91B within the far distance, based on the distance data acquired in step ST13. Then, the second dynamic image data generation unit 120 generates second dynamic image data including the first boundary data and the second boundary data based on the first dynamic image data. After performing the processing in step ST19, the image generation process proceeds to step ST20.

[0190] In step ST20, the dynamic image data output unit 122 outputs the second dynamic image data generated in step ST19 (i.e., the second dynamic image data including the first boundary data and the second boundary data) to the display 28 as dynamic image data for display. After the processing in step ST20 is performed, the image generation process moves to step ST27.

[0191] In step ST21, the second dynamic image data generation unit 120 generates first boundary data representing the region of the first boundary subject 91A existing within the near distance, based on the distance data acquired in step ST13. Then, the second dynamic image data generation unit 120 generates second dynamic image data including the first boundary data based on the first dynamic image data. After performing the processing in step ST21, the image generation process proceeds to step ST22.

[0192] In step ST22, the dynamic image data output unit 122 outputs the second dynamic image data generated in step ST18 (i.e., the second dynamic image data including the first boundary data) to the display 28 as dynamic image data for display. After the processing in step ST22 is performed, the image generation process proceeds to step ST27.

[0193] In step ST23, the second subject determination unit 118 determines whether a second boundary subject 91B exists within the far point distance based on the distance data acquired in step ST13. If, in step ST23, the second boundary subject 91B does not exist within the far point distance, the determination is rejected, and the motion image generation process proceeds to step ST26. If, in step ST23, the second boundary subject 91B exists within the far point distance, the determination is affirmed, and the motion image generation process proceeds to step ST24.

[0194] In step ST24, the second dynamic image data generation unit 120 generates second boundary data representing the region of the second boundary subject 91B existing within the far point distance, based on the distance data acquired in step ST13. Then, the second dynamic image data generation unit 120 generates second dynamic image data including the second boundary data based on the first dynamic image data. After performing the processing in step ST24, the image generation process proceeds to step ST25.

[0195] In step ST25, the dynamic image data output unit 122 outputs the second dynamic image data generated in step ST24 (i.e., the second dynamic image data including the second boundary data) to the display 28 as dynamic image data for display. After the processing in step ST25 is performed, the image generation process proceeds to step ST27.

[0196] In step ST26, the dynamic image data output unit 122 outputs the first dynamic image data generated in step ST11 to the display 28 as dynamic image data for display. After the processing in step ST26 is performed, the image generation process proceeds to step ST27.

[0197] In step ST27, the motion image data storage control unit 124 stores the first motion image data generated in step ST11 as a recording motion image data in the NVM64. After the processing in step ST27 is performed, the image generation process proceeds to step ST28.

[0198] In step ST28, the CPU 62 determines whether the condition for ending the motion image generation process is met. For example, a condition for ending the motion image generation process could be the switching of the camera device 10's operating mode from camera mode to playback mode or setting mode. In step ST28, if the condition for ending the motion image generation process is not met, the determination is rejected, and the motion image generation process proceeds to... Figure 12A Step ST10 is shown. In step ST28, when the condition for ending the motion image generation process is met, the determination is affirmed, and the motion image generation process ends. Furthermore, the imaging method described for the function of the imaging device 10 described above is an example of the "imaging method" involved in the technology of this invention.

[0199] As explained above, in the imaging device 10 according to the first embodiment, when, for example, a first subject 90A, a second subject 90B, and a third subject 92 are present within the imaging area, the CPU 62 acquires distance data regarding the distances between the plurality of subjects and the imaging device 10. Then, the CPU 62 generates first boundary data representing the region of the first boundary subject 91A present within the near distance, based on the distance data. Similarly, the CPU 62 generates second boundary data representing the region of the second boundary subject 91B present within the far distance, based on the distance data. Further, the CPU 62 generates second dynamic image data including the first boundary data and the second boundary data based on the first dynamic image data. Then, the CPU 62 outputs the second dynamic image data including the first boundary data and the second boundary data to the display 28 as dynamic image data for display. Therefore, for example, a user can confirm the position of the pixel with the first predetermined color and the position of the pixel with the second predetermined color by the image displayed on the monitor 28, thereby being able to determine the position of the near point and the far point of the depth of field.

[0200] Furthermore, in the imaging device 10 according to the first embodiment, when, for example, a first subject 90A and a third subject 92 are present within the imaging area, the CPU 62 acquires distance data regarding the distances between the plurality of subjects and the imaging device 10. Then, the CPU 62 generates first boundary data representing the region of the first boundary subject 91A present within the near distance, based on the distance data. Furthermore, the CPU 62 generates second dynamic image data including the first boundary data based on first dynamic image data captured by the image sensor 20. Then, the CPU 62 outputs the second dynamic image data including the first boundary data to the display 28 as dynamic image data for display. Therefore, for example, a user can confirm the position of the pixel with the first predetermined color by viewing the image displayed on the display 28, thereby determining the position of the near point of the depth of field.

[0201] Furthermore, in the imaging device 10 according to the first embodiment, for example, when a second subject 90B and a third subject 92 are present in the imaging area, the CPU 62 acquires distance data regarding the distances between the plurality of subjects and the imaging device 10. Then, the CPU 62 generates second boundary data representing the region of the second boundary subject 91B existing within the distance of the far point, based on the distance data. Furthermore, the CPU 62 generates second dynamic image data including the second boundary data based on the first dynamic image data captured by the image sensor 20. Then, the CPU 62 outputs the second dynamic image data including the second boundary data to the display 28 as dynamic image data for display. Therefore, for example, a user can confirm the position of the pixel with the second predetermined color by the image displayed on the display 28, thereby being able to determine the position of the far point of the depth of field.

[0202] Furthermore, in the camera device 10 according to the first embodiment, the focus position data indicating the area of ​​the focused subject 93 within the object distance is not included in the display moving image data. Therefore, for example, compared to the case where the area of ​​the focused subject 93 and the area other than the focused subject 93 are displayed on the display 28 in a differentiated manner based on the focus position data, the user can more easily confirm the expression and / or movement of the focused subject 93 based on the image displayed on the display 28.

[0203] Furthermore, the dynamic image data used for displaying the image on the display 28 is data used to distinguish the area of ​​the boundary subject 91 and the area other than the boundary subject 91 in a differentiated manner. Therefore, the user can, for example, distinguish the area of ​​the boundary subject 91 and the area other than the boundary subject 91 based on the image displayed on the display 28.

[0204] Furthermore, the boundary data is image processing data used to mark the area of ​​the boundary subject 91 in the image displayed on the display 28 based on the first dynamic image data. Therefore, the user can identify the area of ​​the boundary subject 91 based on the image processed.

[0205] Furthermore, the image processing involves assigning a predetermined color to the pixels among the multiple pixels constituting the image displayed on the display 28 that correspond to the area of ​​the boundary subject 91. Therefore, the user can identify the area of ​​the boundary subject 91 based on the pixels assigned the predetermined color.

[0206] Furthermore, in the imaging device 10 according to the first embodiment, when a first subject 90A, a second subject 90B, and a third subject 92 are present within the imaging area, the CPU 62 generates second dynamic image data including first boundary data and second boundary data. The first boundary data represents the area of ​​the first boundary subject 91A within the near distance, and the second boundary data represents the area of ​​the second boundary subject 91B within the far distance. Then, the CPU 62 outputs the second dynamic image data including the first boundary data and the second boundary data to the display 28 as dynamic image data for display. Therefore, for example, a user can confirm the position of the pixel with the first predetermined color and the position of the pixel with the second predetermined color by the image displayed on the display 28, thereby being able to grasp the position of the near point and the position of the far point of the depth of field.

[0207] Furthermore, the first boundary data is data representing the area of ​​the first boundary subject 91A in a first manner on the image displayed on the display 28, and the second boundary data is data representing the area of ​​the second boundary subject 91B in a second manner different from the first manner on the image displayed on the display 28. Therefore, for example, compared to the case where the first boundary subject 91A and the second boundary subject 91B are represented in the same manner on the image displayed on the display 28, the user can more easily distinguish the first boundary subject 91A and the second boundary subject 91B.

[0208] Furthermore, the CPU 62 acquires first region data representing the region of the first boundary subject 91A based on distance data. The first boundary subject 91A exists within a distance equal to the nearest point distance among the distances between multiple subjects and the camera device 10. Then, the CPU 62 generates first boundary data representing the region of the first boundary subject 91A existing within the nearest point distance in a first manner based on the first region data. Therefore, it is possible to generate first boundary data based on distance data.

[0209] Similarly, the CPU 62 obtains second region data representing the region of the second boundary subject 91B, which exists within a distance equal to the distance to the far point among the distances between the plurality of subjects and the camera device 10. Then, the CPU 62 represents the second boundary data of the region of the second boundary subject 91B existing within the far point distance in a second manner based on the second region data. Therefore, second boundary data can be generated based on the distance data.

[0210] Furthermore, the CPU 62 stores the first motion image data as a recording motion image data in the NVM 64. Therefore, when an image is displayed on the display 28 based on the recording motion image data stored in the NVM 64, it is possible to prevent the area of ​​the first boundary subject 91A and / or the area of ​​the second boundary subject 91B from being displayed in the image with color applied.

[0211] Furthermore, the photoelectric conversion element 72 of the image sensor 20 has multiple photosensitive pixels 72B, and the CPU 62 acquires distance data based on the phase difference pixel data 73B output from the photosensitive pixels 72B. Therefore, a distance sensor other than the image sensor 20 is not required.

[0212] Furthermore, the photosensitive pixel 72B selectively outputs non-phase difference pixel data 73A and phase difference pixel data 73B. Non-phase difference pixel data 73A is pixel data obtained by photoelectric conversion across the entire area of ​​the photosensitive pixel 72B, while phase difference pixel data 73B is pixel data obtained by photoelectric conversion across a portion of the photosensitive pixel 72B. Therefore, image data 81 and distance data 82 can be acquired from the image data 73.

[0213] Furthermore, in the first embodiment, the first image processing performed by the second dynamic image data generation unit 120 is a process of attaching a first predetermined color to the pixels among the plurality of pixels corresponding to the region of the first boundary subject 91A that exists within a near distance. However, the first image processing may also be a process of attaching a first predetermined brightness to the pixels among the plurality of pixels corresponding to the region of the first boundary subject 91A. The first predetermined brightness may be a brightness such that the brightness of the region of the first boundary subject 91A is higher than the brightness of the regions other than the region of the first boundary subject 91A, or it may be a brightness such that the brightness of the region of the first boundary subject 91A is lower than the brightness of the regions other than the region of the first boundary subject 91A.

[0214] Similarly, in the first embodiment, the second image processing performed by the second dynamic image data generation unit 120 is a process of attaching a second predetermined color to the pixels among the plurality of pixels corresponding to the region of the second boundary subject 91B existing within a distance from the far point. However, the second image processing may also be a process of attaching a second predetermined brightness to the pixels among the plurality of pixels corresponding to the region of the second boundary subject 91B. The second predetermined brightness may be a brightness such that the brightness of the region of the second boundary subject 91B is higher than the brightness of the regions other than the region of the second boundary subject 91B, or it may be a brightness such that the brightness of the region of the second boundary subject 91B is lower than the brightness of the regions other than the region of the second boundary subject 91B.

[0215] Furthermore, the first predetermined brightness can be a different brightness than the second predetermined brightness. The first predetermined brightness and the second predetermined brightness are examples of "determined brightness" involved in the technology of this invention.

[0216] Furthermore, the first image processing may be a process of attaching a first mark to the image displayed on the display 28, indicating the area of ​​the first boundary subject 91A. Similarly, the second image processing may be a process of attaching a second mark to the image displayed on the display 28, indicating the area of ​​the second boundary subject 91B. Examples of the first and second marks include arrows and / or boxes. The shape of the first mark may differ from the shape of the second mark. The first and second marks are examples of "markers" as understood in the present invention.

[0217] Furthermore, the first image processing can be a process of superimposing a first distance image generated based on distance data (i.e., a distance image representing the region of the first boundary subject 91A) onto an image displayed on the display 28. Similarly, the second image processing can be a process of superimposing a second distance image generated based on distance data (i.e., a distance image representing the region of the second boundary subject 91B) onto an image displayed on the display 28. As an example of the first and second distance images, an image after heat mapping of the distance data 82 can be given (e.g., a shadow image, a dot image, a contour image, and / or a contour image, etc.). The manner of the first distance image may differ from that of the second distance image. The first and second distance images are examples of "distance images" according to the technology of the present invention.

[0218] Furthermore, in the first embodiment, as an example of the first image processing to attach a first predetermined color, the red (R) value, green (G) value, and blue (B) value of the pixel are changed. However, as another example of the first image processing to attach a first predetermined color, the brightness (Y) value, blue color difference (Cb) value, and red color difference (Cr) value of the pixel can also be changed. As an example of this, for instance, the brightness (Y) value, blue color difference (Cb) value, and red color difference (Cr) value of the pixel can be set to 128, 128, and 0, respectively.

[0219] Similarly, in the first embodiment, as an example of the second image processing with a second predetermined color, the red (R) value, green (G) value, and blue (B) value of the pixel are changed. However, as an example of the second image processing with a second predetermined color, the brightness (Y) value, blue color difference (Cb) value, and red color difference (Cr) value of the pixel can also be changed.

[0220] Furthermore, the first image processing may be a process of adding a first shadow line to the image displayed on the display 28 to represent the area of ​​the first boundary subject 91A. Similarly, the second image processing may be a process of adding a second shadow line to the image displayed on the display 28 to represent the area of ​​the second boundary subject 91B. The manner in which the first shadow line is added may differ from the manner in which the second shadow line is added.

[0221] Furthermore, when a pixel before the first predetermined color is applied has the same color as the first predetermined color, the second dynamic image data generation unit 120 can set the first predetermined color to a different color than the pixel before the first predetermined color was applied. Similarly, when a pixel before the second predetermined color is applied has the same color as the second predetermined color, the second dynamic image data generation unit 120 can set the second predetermined color to a different color than the pixel before the second predetermined color was applied.

[0222] Furthermore, in the first embodiment, the motion image data storage control unit 124 stores the first motion image data generated by the first motion image data generation unit 102 in the NVM 64 as motion image data for recording. However, for example, when a first subject 90A, a second subject 90B, and a third subject 92 are present in the imaging area, the motion image data storage control unit 124 may store the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the first boundary data and the second boundary data) in the NVM 64 as motion image data for recording.

[0223] Furthermore, for example, when a first subject 90A and a third subject 92 are present in the camera area, the motion image data storage control unit 124 can store the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the first boundary data) as a recording motion image data in the NVM64.

[0224] Furthermore, for example, when a second subject 90B and a third subject 92 are present in the camera area, the motion image data storage control unit 124 can store the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the second boundary data) as a recording motion image data in the NVM64.

[0225] Furthermore, in the first embodiment, although the CPU 62 acquires the distance to a subject corresponding to each of the photosensitive pixels 72B included in the photoelectric conversion element 72, it is not necessary to acquire the distance to a subject corresponding to all photosensitive pixels 72B. That is, the photosensitive pixels 72B whose distances are acquired can be spaced apart.

[0226] Furthermore, in the first embodiment, the photoelectric conversion element 72 is a photoelectric conversion element that provides a pair of photodiodes PD1 and PD2 for each pixel in an image plane phase difference manner. All photosensitive pixels 72B have the function of outputting data related to imaging and phase difference, but it is not limited to all photosensitive pixels 72B having the function of outputting data related to imaging and phase difference. The photoelectric conversion element 72 may include photosensitive pixels 72B that do not have the function of outputting data related to imaging and phase difference. Furthermore, the photoelectric conversion element 72 is not limited to a photoelectric conversion element that provides a pair of photodiodes PD1 and PD2 for each pixel in an image plane phase difference manner. It may also be a photoelectric conversion element that includes a photosensitive pixel 72B for imaging to acquire non-phase difference pixel data 73A and a photosensitive pixel 72B for phase difference detection to acquire phase difference pixel data 73B. In this case, the phase difference pixel is provided with a light-shielding member to receive one of the first pupil portion region and the second pupil portion region.

[0227] Furthermore, in the first embodiment, although distance data is acquired using a phase-difference photoelectric conversion element 72, it is not limited to the phase-difference method. A TOF-type photoelectric conversion element, a stereo camera, or a depth sensor can also be used to acquire distance data. For example, using a LiDAR system can be employed to acquire distance data using a TOF-type photoelectric conversion element. Additionally, distance data can be acquired based on the frame rate of the image sensor 20, or at time intervals longer or shorter than the time interval specified by the frame rate of the image sensor 20.

[0228] [Second Implementation]

[0229] As an example, such as Figure 13 As shown, in the second embodiment, the structure of the camera device 10 is changed as follows compared to the first embodiment.

[0230] That is, in addition to the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near point distance acquisition unit 110, the far point distance acquisition unit 112, the first subject determination unit 114, the second subject determination unit 118, the second dynamic image data generation unit 120, the dynamic image data output unit 122, and the dynamic image data storage control unit 124, the CPU 62 also operates as the first distance range setting unit 130 and the second distance range setting unit 132.

[0231] The operation of the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near-point distance acquisition unit 110, the far-point distance acquisition unit 112, the dynamic image data output unit 122, and the dynamic image data storage control unit 124 is the same as in the first embodiment. In the second embodiment, the operation of the first subject determination unit 114, the second subject determination unit 118, and the second dynamic image data generation unit 120 differs from that in the first embodiment.

[0232] Hereinafter, the differences between the operation of the first distance range setting unit 130, the second distance range setting unit 132, the first subject determination unit 114, the second subject determination unit 118, and the second dynamic image data generation unit 120 of the camera device 10 according to the second embodiment and the camera device 10 according to the first embodiment will be described.

[0233] The first distance range setting unit 130 sets a first distance range including a near point and a first boundary point based on the near point distance acquired by the near point distance acquisition unit 110. The first distance range is the range between the near point and the first boundary point. The first boundary point is located on the far point side relative to the near point. That is, the first distance range is the range extending from the near point to the far point side. The first distance range is a distance range with the near point distance as the lower limit and the first boundary point distance as the upper limit. The first boundary point distance is the distance along the depth direction from the principal point of the imaging lens 40 to the first boundary point. The width of the first distance range (i.e., the difference between the first boundary point distance and the near point distance) can be a predetermined fixed value or a specified value assigned to the receiving device 76 by the user. The first distance range is an example of the "boundary portion," "first boundary portion," and "distance range" involved in the technology of this invention.

[0234] A portion of the face of the first subject 90A exists within the first distance range. That is, a portion of the face of the first subject 90A exists between the near point and the first boundary point. Hereinafter, the portion of the face of the first subject 90A existing within the first distance range will be referred to as the first boundary subject 91A.

[0235] The second distance range setting unit 132 sets a second distance range including the far point and the second boundary point based on the far point distance acquired by the far point distance acquisition unit 112. The second distance range is the range between the far point and the second boundary point. The second boundary point is located on the near point side relative to the far point. That is, the second distance range is the range extending from the far point to the near point side. The second distance range is a distance range with the second boundary point distance as the lower limit and the far point distance as the upper limit. The second boundary point distance is the distance along the depth direction from the principal point of the imaging lens 40 to the second boundary point. The width of the second distance range (i.e., the difference between the far point distance and the second boundary point distance) can be a predetermined fixed value or a value specified by the user to the receiving device 76. The second distance range is an example of the "boundary portion", "second boundary portion" and "distance range" involved in the technology of the present invention.

[0236] A portion of the face of the second subject 90B exists within the second distance range. That is, a portion of the face of the second subject 90B exists between the second boundary point and the far point. Hereinafter, the portion of the face of the second subject 90B existing within the second distance range will be referred to as the second boundary subject 91B.

[0237] The first subject determination unit 114 determines whether a first boundary subject 91A exists within the first distance range (in other words, whether a first boundary subject 91A exists between the near point and the first boundary point) based on the first distance range set by the first distance range setting unit 130. Specifically, the first subject determination unit 114 compares each photosensitive pixel 72B (see reference) based on the first distance range. Figure 3 Based on the distance obtained and the first distance range, it is determined whether the distance obtained for each photosensitive pixel 72B includes distances falling within the first distance range. That is, when considering the distance obtained for each photosensitive pixel 72B (reference...),... Figure 3 When the distance obtained is d, the near point distance is L1, and the width of the first distance range is R1, the first subject determination unit 114 determines whether the distance obtained for each photosensitive pixel 72B includes a distance d that satisfies the following formula (7).

[0238] [Formula 7]

[0239] L1≤d≤L1+R1···(7)

[0240] The photosensitive pixel 72B that falls within the first distance range is the photosensitive pixel 72B corresponding to the first boundary subject 91A. Then, when the distance obtained for each photosensitive pixel 72B includes a distance falling within the first distance range, the first subject determination unit 114 determines that the first boundary subject 91A exists within the first distance range. On the other hand, when the distance obtained for each photosensitive pixel 72B does not include a distance falling within the first distance range, the first subject determination unit 114 determines that the first boundary subject 91A does not exist within the near distance.

[0241] As an example, in Figure 13 In the example shown, a first boundary subject 91A exists within a first distance range. When a first boundary subject 91A exists within a first distance range, the first subject determination unit 114 determines that a first boundary subject 91A exists within a first distance range.

[0242] The second subject determination unit 118 determines whether a second boundary subject 91B exists within a second distance range (in other words, whether a second boundary subject 91B exists between the second boundary point and the far point) based on distance data acquired by the distance data acquisition unit 106. Specifically, the second subject determination unit 118 compares distance data for each photosensitive pixel 72B (see reference). Figure 3 The distance obtained for each photosensitive pixel 72B and the second distance range are used to determine whether the distances obtained for each photosensitive pixel 72B include distances falling within the second distance range. That is, when the distances obtained for each photosensitive pixel 72B (reference...) are... Figure 3 When the distance obtained is d, the distance to the far point is L2, and the width of the second distance range is R2, the second subject determination unit 118 determines whether the distance obtained for each photosensitive pixel 72B includes a distance d that satisfies the following formula (8).

[0243] [Formula 8]

[0244] L²-R²≤d≤L²···(8)

[0245] The photosensitive pixel 72B that falls within the second distance range is the photosensitive pixel 72B corresponding to the second boundary subject 91B. Then, when the distance obtained for each photosensitive pixel 72B includes a distance falling within the second distance range, the second subject determination unit 118 determines that the second boundary subject 91B exists within the second distance range. On the other hand, when the distance obtained for each photosensitive pixel 72B does not include a distance falling within the second distance range, the second subject determination unit 118 determines that the second boundary subject 91B does not exist within the second distance range.

[0246] As an example, in Figure 13In the example shown, a second boundary subject 91B exists within the second distance range. When a second boundary subject 91B exists within the second distance range, the second subject determination unit 118 determines that a second boundary subject 91B exists within the second distance range.

[0247] When the first subject determination unit 114 determines that a first boundary subject 91A exists within a first distance range, the second motion image data generation unit 120 generates first boundary data representing the region of the first boundary subject 91A existing within the first distance range based on the distance data. Specifically, the second motion image data generation unit 120 generates the first boundary data based on the following points.

[0248] That is, the second dynamic image data generation unit 120 acquires first region data representing the region of the first boundary subject 91A based on distance data, wherein the first boundary subject 91A exists for each photosensitive pixel 72B (reference). Figure 3 The distances obtained fall within the first distance range. The first region data is represented, for example, by the address of the photosensitive pixel 72B. That is, the second dynamic image data generation unit 120 obtains the addresses of the photosensitive pixels 72B that fall within the first distance range from among the plurality of photosensitive pixels 72B based on the distance data, as the first region data. The address of the photosensitive pixel 72B is a coordinate determined for each photosensitive pixel 72B, and represents the vertical and horizontal coordinates of the photoelectric conversion element 72.

[0249] Next, the second dynamic image data generation unit 120 generates first boundary data based on the first region data. The first boundary data is data that represents, in a first manner, the region of the first boundary subject 91A existing within a first distance range (i.e., the region represented by the first region data) in the image displayed on the display 28 according to the first dynamic image data. The first boundary data is an example of data that represents the region of the first boundary subject 91A in a first manner, and is data used for first image processing, which marks the region of the first boundary subject 91A existing within a first distance range in the image displayed on the display 28.

[0250] As an example, the first image processing involves assigning a first predetermined color to the pixels among the plurality of pixels constituting the image displayed on the display 28 that correspond to the region of the first boundary subject 91A existing within the first distance range (i.e., the pixel corresponding to the address of the photosensitive pixel 72B represented by the first region data). The process of assigning the first predetermined color is the same as in the first embodiment. The first boundary data is an example of "boundary data" and "first boundary data" according to the technology of this invention. The first image processing is an example of "image processing" according to the technology of this invention.

[0251] Furthermore, when the second subject determination unit 118 determines that a second boundary subject 91B exists within a second distance range, the second motion image data generation unit 120 generates second boundary data representing the region of the second boundary subject 91B existing within the second distance range based on the distance data. Specifically, the second motion image data generation unit 120 generates the second boundary data based on the following points.

[0252] That is, the second dynamic image data generation unit 120 acquires second region data representing the region of the second boundary subject 91B based on distance data, wherein the second boundary subject 91B exists for each photosensitive pixel 72B (reference). Figure 3 The distances obtained fall within the second distance range. The second region data is represented, for example, by the address of the photosensitive pixel 72B. That is, the second dynamic image data generation unit 120 obtains the addresses of the photosensitive pixels 72B that fall within the second distance range from the plurality of photosensitive pixels 72B based on the distance data as the second region data.

[0253] Next, the second dynamic image data generation unit 120 generates second boundary data based on the second region data. The second boundary data is data representing the region of the second boundary subject 91B existing within a second distance range (i.e., the region represented by the second region data) in a second manner, different from the first manner, of the image displayed on the display 28 based on the first dynamic image data. The second boundary data is an example of data representing the region of the second boundary subject 91B in a second manner, and is data used for second image processing, which marks the region of the second boundary subject 91B existing within a second distance range in the image displayed on the display 28 based on the first dynamic image data.

[0254] As an example, the second image processing involves assigning a second predetermined color to the pixels among the plurality of pixels constituting the image displayed on the display 28 that correspond to the region of the second boundary subject 91B existing within the second distance range (i.e., the pixel corresponding to the address of the photosensitive pixel 72B represented by the second region data). The process of assigning the second predetermined color is the same as in the first embodiment. The second boundary data is an example of "boundary data" and "second boundary data" according to the technology of this invention. The second image processing is an example of "image processing" according to the technology of this invention.

[0255] Then, when the first subject determination unit 114 determines that a first boundary subject 91A exists within a first distance range and the second subject determination unit 118 determines that a second boundary subject 91B exists within a second distance range, the second motion image data generation unit 120 generates second motion image data including first boundary data and second boundary data based on the first motion image data. Specifically, the second motion image data generation unit 120 performs first image processing on pixels corresponding to the region of the first boundary subject 91A existing within the first distance range among the plurality of pixels constituting the image displayed on the display 28 based on the first motion image data. Similarly, the second motion image data generation unit 120 performs second image processing on pixels corresponding to the region of the second boundary subject 91B existing within the second distance range among the plurality of pixels constituting the image displayed on the display 28 based on the first motion image data. Therefore, the second dynamic image data is generated by the second dynamic image data generation unit 120. The second dynamic image data represents an image in which a first predetermined color is attached to the pixels corresponding to the region of the first boundary subject 91A existing within the first distance range, and a second predetermined color is attached to the pixels corresponding to the region of the second boundary subject 91B existing within the second distance range.

[0256] When the first subject determination unit 114 determines that a first boundary subject 91A exists within a first distance range and the second subject determination unit 118 determines that a second boundary subject 91B exists within a second distance range, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the first boundary data and the second boundary data) to the display 28 as display motion image data. The display 28 displays an image based on the display motion image data. In this case, the image 200 displayed on the display 28 based on the motion image data is consistent with the image in the first embodiment. Figure 7 The example shown is the same.

[0257] As an example, Figure 14 The image shows an example where a first subject 90A and a third subject 92 exist within the image area captured by the image sensor 20. The positions of the first subject 90A and the third subject 92 are shown in the image. Figure 13 The example shown is the same.

[0258] The following is based on Figure 14The following components will be described in the example shown: first camera control unit 100, first motion image data generation unit 102, second camera control unit 104, distance data acquisition unit 106, object distance acquisition unit 108, near point distance acquisition unit 110, far point distance acquisition unit 112, first distance range setting unit 130, second distance range setting unit 132, first subject determination unit 114, second subject determination unit 118, second motion image data generation unit 120, and motion image data output unit 122.

[0259] The operation and control of the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near-point distance acquisition unit 110, the far-point distance acquisition unit 112, the first subject determination unit 114, the second dynamic image data generation unit 120, and the dynamic image data storage control unit 124 are as follows: Figure 13 The example shown is the same. In Figure 14 In the example shown, the operation of the second subject determination unit 118, the second motion image data generation unit 120, and the motion image data output unit 122 is similar to... Figure 13 The examples shown are different. Below, for... Figure 14 The example shown illustrates the operation of the second subject determination unit 118, the second motion image data generation unit 120, and the motion image data output unit 122. Figure 13 The examples shown are different.

[0260] As an example, in Figure 14 In the example shown, there is no second boundary subject 91B within the second distance range (reference). Figure 13 When there is no second boundary subject 91B within the second distance range, the second subject determination unit 118 determines that there is no second boundary subject 91B within the second distance range.

[0261] When the first subject determination unit 114 determines that a first boundary subject 91A exists within a first distance range and the second subject determination unit 118 determines that a second boundary subject 91B does not exist within a second distance range, the second motion image data generation unit 120 generates second motion image data including the first boundary data based on the first motion image data. Specifically, the second motion image data generation unit 120 performs first image processing on the pixels corresponding to the region of the first boundary subject 91A existing within the first distance range among the plurality of pixels constituting the image displayed on the display 28 based on the first motion image data. Thus, the second motion image data is generated by the second motion image data generation unit 120, which represents an image in which a first predetermined color is applied to the pixels corresponding to the region of the first boundary subject 91A existing within the first distance range.

[0262] When the first subject determination unit 114 determines that a first boundary subject 91A exists within a first distance range and the second subject determination unit 118 determines that a second boundary subject 91B does not exist within a second distance range, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the first boundary data) to the display 28 as display motion image data. The display 28 displays an image based on the display motion image data. In this case, the image 200 displayed on the display 28 based on the motion image data is consistent with the image in the first embodiment. Figure 9 The example shown is the same.

[0263] As an example, Figure 15 The image shows an example where a second subject 90B and a third subject 92 exist within the image area captured by the image sensor 20. The positions of the second subject 90B and the third subject 92 are shown in the image. Figure 13 The example shown is the same.

[0264] The following is based on Figure 15 The following components will be described in the example shown: first camera control unit 100, first motion image data generation unit 102, second camera control unit 104, distance data acquisition unit 106, object distance acquisition unit 108, near point distance acquisition unit 110, far point distance acquisition unit 112, first distance range setting unit 130, second distance range setting unit 132, first subject determination unit 114, second subject determination unit 118, second motion image data generation unit 120, and motion image data output unit 122.

[0265] The operation and control of the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near-point distance acquisition unit 110, the far-point distance acquisition unit 112, the second dynamic image data generation unit 120, the second subject determination unit 118, and the dynamic image data storage control unit 124 Figure 13 The example shown is the same. In Figure 15 In the example shown, the operation of the first subject determination unit 114, the second motion image data generation unit 120, and the motion image data output unit 122 is similar to... Figure 13 The examples shown are different. Below, for... Figure 15 The example shown illustrates the operation of the first subject determination unit 114, the second motion image data generation unit 120, and the motion image data output unit 122. Figure 13 The examples shown are different.

[0266] As an example, in Figure 15 In the example shown, there is no first boundary subject 91A within the first distance range (reference). Figure 13 When there is no first boundary subject 91A within the first distance range, the first subject determination unit 114 determines that there is no first boundary subject 91A within the first distance range.

[0267] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the first distance range and the second subject determination unit 118 determines that there is a second boundary subject 91B within the second distance range, the second motion image data generation unit 120 generates second motion image data including second boundary data based on the first motion image data. Specifically, the second motion image data generation unit 120 performs second image processing on the pixels corresponding to the region of the second boundary subject 91B present within the second distance range among the plurality of pixels constituting the image displayed on the display 28 based on the first motion image data. Thus, the second motion image data generated by the second motion image data generation unit 120 represents an image in which a second predetermined color is applied to the pixels corresponding to the region of the second boundary subject 91B present within the second distance range.

[0268] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the first distance range and the second subject determination unit 118 determines that there is a second boundary subject 91B within the second distance range, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the second boundary data) to the display 28 as display motion image data. The display 28 displays an image based on the display motion image data. In this case, the image 200 displayed on the display 28 based on the motion image data is compared with the image in the first embodiment. Figure 11 The example shown is the same.

[0269] In particular, although not shown in the figure, when the first subject 90A and the second subject 90B are not present in the image area captured by the image sensor 20, the first subject determination unit 114 determines that the first boundary subject 91A is not present in the first distance range and the second subject determination unit 118 determines that the second boundary subject 91B is not present in the second distance range.

[0270] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the first distance range and the second subject determination unit 118 determines that there is no second boundary subject 91B within the second distance range, the operation of the second motion image data generation unit 120 is skipped. That is, the generation of second motion image data by the second motion image data generation unit 120 is not performed, and the processing performed by the motion image data output unit 122 is performed instead.

[0271] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the first distance range and the second subject determination unit 118 determines that there is no second boundary subject 91B within the second distance range, the motion image data output unit 122 outputs the first motion image data generated by the first motion image data generation unit 102 to the display 28 as display motion image data. The display 28 displays an image based on the display motion image data. In this case, the first subject 90A and the second subject 90B are not displayed as images in the image displayed on the display 28.

[0272] Next, refer to Figure 16A and Figure 16B The operation of the camera device 10 according to this second embodiment will be explained. Figure 16A and Figure 16B An example of the flow of dynamic image generation processing according to the second embodiment is shown.

[0273] In the dynamic image generation process according to the second embodiment, steps ST10 to ST16 are the same as in the first embodiment. After performing the processing in step ST16, Figure 16A The dynamic image generation process shown is transferred to step ST30.

[0274] In step ST30, the first distance range setting unit 130 sets a first distance range including the nearest point based on the near point distance obtained in step ST15. After the processing in step ST30 is performed, the image generation process proceeds to step ST31.

[0275] In step ST31, the second distance range setting unit 132 sets a second distance range including the far point based on the far point distance obtained in step ST16. After the processing in step ST31 is performed, the image compositing process begins. Figure 16B Step ST17 is shown.

[0276] In step ST17, the first subject determination unit 114 determines the subject based on the conditions in step ST30 (refer to step ST30). Figure 16AThe system determines whether a first boundary subject 91A exists within the first distance range set in step ST17. If, in step ST17, the first boundary subject 91A does not exist within the first distance range, the determination is rejected, and the motion image generation process proceeds to step ST23. If, in step ST17, the first boundary subject 91A exists within the first distance range, the determination is affirmative, and the motion image generation process proceeds to step ST18.

[0277] In step ST18, the second subject determination unit 118 determines whether a second boundary subject 91B exists within a second distance range based on the distance data obtained in step ST13. If, in step ST18, the second boundary subject 91B does not exist within the second distance range, the determination is rejected, and the motion image generation process proceeds to step ST21. If, in step ST18, the second boundary subject 91B exists within the second distance range, the determination is affirmed, and the motion image generation process proceeds to step ST19.

[0278] In step ST19, the second dynamic image data generation unit 120 generates first boundary data representing the region of the first boundary subject 91A existing within the first distance range, based on the distance data acquired in step ST13. Furthermore, the second dynamic image data generation unit 120 generates second boundary data representing the region of the second boundary subject 91B existing within the second distance range, based on the distance data acquired in step ST13. Then, the second dynamic image data generation unit 120 generates second dynamic image data including the first boundary data and the second boundary data based on the first dynamic image data. After performing the processing in step ST19, the image generation process proceeds to step ST20.

[0279] In the dynamic image generation process according to the second embodiment, step ST20 is the same as in the first embodiment.

[0280] In step ST21, the second dynamic image data generation unit 120 generates first boundary data representing the region of the first boundary subject 91A existing within the first distance range, based on the distance data acquired in step ST13. Then, the second dynamic image data generation unit 120 generates second dynamic image data including the first boundary data based on the first dynamic image data. After performing the processing in step ST21, the image generation process proceeds to step ST22.

[0281] In the dynamic image generation process according to the second embodiment, step ST22 is the same as in the first embodiment.

[0282] In step ST23, the second subject determination unit 118 determines whether a second boundary subject 91B exists within a second distance range based on the distance data acquired in step ST13. If, in step ST23, the second boundary subject 91B does not exist within the second distance range, the determination is rejected, and the motion image generation process proceeds to step ST26. If, in step ST23, the second boundary subject 91B exists within the second distance range, the determination is affirmed, and the motion image generation process proceeds to step ST24.

[0283] In step ST24, the second dynamic image data generation unit 120 generates second boundary data representing the region of the second boundary subject 91B existing within the second distance range, based on the distance data acquired in step ST13. Then, the second dynamic image data generation unit 120 generates second dynamic image data including the second boundary data based on the first dynamic image data. After performing the processing in step ST24, the image generation process proceeds to step ST25.

[0284] In the dynamic image generation process according to the second embodiment, steps ST25 to ST28 are the same as in the first embodiment.

[0285] As explained above, in the imaging device 10 according to the second embodiment, the CPU 62 sets a first distance range including the nearest point. Furthermore, when a first boundary subject 91A exists within the first distance range, the CPU 62 generates first boundary data representing the area of ​​the first boundary subject 91A present within the first distance range. Then, the CPU 62 outputs display dynamic image data including the first boundary data to the display 28. Therefore, for example, compared to generating first boundary data representing the area of ​​the first boundary subject 91A present within the nearest distance, even if jitter occurs on the subject and / or the imaging device 10, the pixels corresponding to the area of ​​the first boundary subject 91A among the multiple pixels constituting the image displayed on the display 28 can be more stably assigned the first predetermined color. That is, even if jitter occurs on the subject and / or the imaging device 10, interruption of pixels assigned the first predetermined color, and / or display or disappearance of pixels assigned the first predetermined color can be suppressed.

[0286] Similarly, in the imaging device 10 according to the second embodiment, the CPU 62 sets a second distance range including the farthest point. When a second boundary subject 91B exists within the second distance range, the CPU 62 generates second boundary data representing the region of the second boundary subject 91B present within the second distance range. Then, the CPU 62 outputs display dynamic image data including the second boundary data to the display 28. Therefore, for example, compared to generating second boundary data representing the region of the second boundary subject 91B present within the farthest point distance, even if jitter occurs on the subject and / or the imaging device 10, the pixels corresponding to the region of the second boundary subject 91B among the plurality of pixels constituting the image displayed on the display 28 can be stably assigned the second predetermined color. That is, even if jitter occurs on the subject and / or the imaging device 10, interruption of pixels assigned the second predetermined color, and / or display or disappearance of pixels assigned the second predetermined color can be suppressed.

[0287] Furthermore, the first distance range is the range extending from the near point of the depth of field to the far point of the depth of field. Therefore, for example, compared to the case where first boundary data representing the region of the first boundary subject 91A existing within the near point distance is generated, even when the first subject 90A moves from the near point to the far point, it is also possible to suppress the immediate disappearance of pixels with the first predetermined color.

[0288] Furthermore, the second distance range is the range extending from the far point of the depth of field to the near point of the depth of field. Therefore, for example, compared to generating second boundary data representing the region of the second boundary subject 91B existing within the far point distance, it is also possible to suppress the immediate disappearance of pixels with the second predetermined color when the second subject 90B moves from the far point to the near point.

[0289] Furthermore, the CPU 62 acquires first region data representing the region of the first boundary subject 91A based on distance data. The first boundary subject 91A exists within a first distance range among the distances between multiple subjects and the camera device 10. Then, the CPU 62 generates first boundary data representing the region of the first boundary subject 91A existing within the first distance range in a first manner based on the first region data. Therefore, it is possible to generate first boundary data based on distance data.

[0290] Similarly, the CPU 62 obtains second region data representing the region of the second boundary subject 91B based on distance data, wherein the second boundary subject 91B exists within a second distance range among the distances between multiple subjects and the camera device 10. Then, the CPU 62 represents the second boundary data of the region of the second boundary subject 91B existing within the second distance range in a second manner based on the second region data. Therefore, the second boundary data can be generated based on the distance data.

[0291] Furthermore, in the camera device 10 according to the second embodiment, the width of the first distance range and the width of the second distance range can be the same.

[0292] [Third Implementation]

[0293] As an example, such as Figure 17 and Figure 18 As shown, in the third embodiment, the structure of the camera device 10 is changed as follows compared to the second embodiment.

[0294] That is, in addition to the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near point distance acquisition unit 110, the far point distance acquisition unit 112, the first distance range setting unit 130, the second distance range setting unit 132, the first subject determination unit 114, the second subject determination unit 118, the second dynamic image data generation unit 120, and the dynamic image data output unit 122, the CPU 62 also operates as the front depth-of-field acquisition unit 140 and the rear depth-of-field acquisition unit 142.

[0295] The operation of the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near-point distance acquisition unit 110, the far-point distance acquisition unit 112, the first subject determination unit 114, the second subject determination unit 118, the second dynamic image data generation unit 120, the dynamic image data output unit 122, and the dynamic image data storage control unit 124 is the same as in the second embodiment. In the third embodiment, the operation of the first distance range setting unit 130 and the second distance range setting unit 132 differs from that in the second embodiment.

[0296] Hereinafter, the differences between the operation of the first distance range setting unit 130, the second distance range setting unit 132, the front depth-of-field acquisition unit 140, and the rear depth-of-field acquisition unit 142 and the camera device 10 according to the third embodiment will be explained. Furthermore, compared with the camera device 10 according to the second embodiment... Figure 17 Compared to the example shown, Figure 18 The example shown is an example of a deeper depth of field.

[0297] The front depth-of-field acquisition unit 140 acquires the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ. Then, the front depth-of-field acquisition unit 140 calculates the front depth of field D1 using the above formula (3) based on the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ, and acquires the front depth of field D1. In this case, the front depth-of-field acquisition unit 140 acquires the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ with the following key points: That is, the front depth-of-field acquisition unit 140 acquires the object distance L acquired by the object distance acquisition unit 108. Furthermore, the front depth-of-field acquisition unit 140 acquires, for example, the focal length f pre-stored in the NVM64. Furthermore, the front depth-of-field acquisition unit 140 acquires the aperture value F corresponding to the aperture amount detected by the aperture amount sensor 42C. Furthermore, for example, when the user assigns the permissible circle of confusion diameter δ to the receiving device 76, the front depth-of-field acquisition unit 140 acquires the permissible circle of confusion diameter δ received by the receiving device 76.

[0298] The rear depth-of-field acquisition unit 142 acquires the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ. Then, the rear depth-of-field acquisition unit 142 calculates the rear depth-of-field D2 using the above formula (4) based on the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ. In this case, the rear depth-of-field acquisition unit 142 acquires the object distance L, focal length f, aperture value F, and permissible circle of confusion diameter δ with the following key points: That is, the rear depth-of-field acquisition unit 142 acquires the object distance L acquired by the object distance acquisition unit 108. Furthermore, the rear depth-of-field acquisition unit 142 acquires, for example, the focal length f pre-stored in the NVM64. Also, the rear depth-of-field acquisition unit 142 acquires the aperture value F, which corresponds to the aperture amount detected by the aperture amount sensor 42C. Furthermore, for example, when the user assigns the allowable circle of confusion diameter δ to the receiving device 76, the rear depth of field acquisition unit 142 acquires the allowable circle of confusion diameter δ received by the receiving device 76.

[0299] When the depth of field is D1, the width of the first distance range is R1, and the first coefficient is P1, the first distance range setting unit 130 calculates the width R1 of the first distance range based on the depth of field D1 calculated by the depth of field acquisition unit 140 using the following formula (9). The first coefficient P1 is a coefficient that defines the ratio of the width R1 of the first distance range to the depth of field D1. The first coefficient P1 is set, for example, in the range of 0.05 to 0.15, and preferably around 0.10. The first coefficient P1 can be stored in the NVM 64 in advance, or it can be assigned to the receiving device 76 by the user.

[0300] [Formula 9]

[0301] R1=D1×P1···(9)

[0302] As an example, such as Figure 17 and Figure 18 As shown, the depth of field in front increases as the depth of field increases and decreases as the depth of field decreases. According to the above formula (9), the first distance range setting unit 130 widens the width R1 of the first distance range as the depth of field in front D1 increases and narrows the width R1 of the first distance range as the depth of field in front D1 decreases.

[0303] Furthermore, when the distance to the first boundary point is L p1 When the nearest point distance is L1, the first distance range setting unit 130 uses the following formula (10) to calculate the first boundary point distance L. p1 .

[0304] [Formula 10]

[0305] L P1 =L1+R1···(10)

[0306] Then, the first distance range setting unit 130 sets the near point distance L1 as the lower limit and sets the first boundary point distance L... p1 The first distance range serves as the upper limit.

[0307] When the rear depth of field is D2, the width of the second distance range is R2, and the second coefficient is P2, the second distance range setting unit 132 calculates the width R2 of the second distance range based on the rear depth of field D2 calculated by the rear depth of field acquisition unit 142 using the following formula (11). The second coefficient P2 is a coefficient that defines the ratio of the width R2 of the second distance range to the rear depth of field D2. The second coefficient P2 is set, for example, in the range of 0.05 to 0.15, and preferably around 0.10. The second coefficient P2 can be stored in the NVM 64 in advance, or it can be assigned to the receiving device 76 by the user.

[0308] [Formula 11]

[0309] R² = D² × P²···(11)

[0310] As an example, such as Figure 17 and Figure 18 As shown, the depth of field behind the subject increases as the depth of field increases and decreases as the depth of field decreases. According to the above formula (11), the second distance range setting unit 132 widens the width of the second distance range as the depth of field behind the subject D2 increases and narrows the width of the second distance range as the depth of field behind the subject D2 decreases.

[0311] Furthermore, when the distance to the second boundary point is L p2 When the distance to the far point is L2, the second distance range setting unit 132 uses the following formula (12) to calculate the distance L of the second boundary point. p2.

[0312] [Formula 12]

[0313] L P2 =D² + R²···(12)

[0314] Then, the second distance range setting unit 132 sets the distance L of the second boundary point. p2 The second distance range is defined as the lower limit and the distance from the far point L2 is defined as the upper limit.

[0315] Next, refer to Figure 19 The operation of the camera device 10 according to the third embodiment will be explained. Figure 19 An example of a portion of the dynamic image generation process involved in the third embodiment is shown.

[0316] In the dynamic image generation process according to the third embodiment, steps ST10 to ST16 are the same as in the second embodiment. After performing the processing in step ST16, Figure 19 The dynamic image generation process shown is transferred to step ST40.

[0317] In step ST40, the foreground depth-of-field acquisition unit 140 acquires the foreground depth-of-field D1 based on the object distance L, focal length f, aperture value F, and allowable circle of confusion diameter δ. After the processing in step ST40 is performed, the image generation process proceeds to step ST41.

[0318] In step ST41, the rear depth-of-field acquisition unit 142 acquires the rear depth-of-field D2 based on the object distance L, focal length f, aperture value F, and allowable circle of confusion diameter δ. After the processing in step ST41 is performed, the image generation process proceeds to step ST30.

[0319] In step ST30, the first distance range setting unit 130 sets a first distance range based on the foreground depth of field obtained in step ST40. In this case, the first distance range setting unit 130 widens the width of the first distance range as the foreground depth of field increases, and narrows the width of the first distance range as the foreground depth of field decreases, based on a first coefficient that specifies the ratio of the width of the first distance range to the foreground depth of field. After performing the processing in step ST30, the image generation process proceeds to step ST31.

[0320] In step ST31, the second distance range setting unit 132 sets a second distance range based on the rear depth of field obtained in step ST41. In this case, the second distance range setting unit 132, based on a second coefficient specifying the ratio of the width of the second distance range to the rear depth of field, widens the width of the second distance range as the rear depth of field increases and narrows the width of the second distance range as the rear depth of field decreases. After the processing in step ST31 is performed, the image compositing process begins. Figure 16B Step ST17 is shown.

[0321] In the motion image generation process according to the third embodiment, steps ST17 to ST28 (see reference) Figure 16B The same as the second implementation.

[0322] As explained above, in the imaging device 10 according to the third embodiment, the CPU 62 changes the width of the first distance range and / or the width of the second distance range according to the depth of field. That is, the width of the first distance range and / or the width of the second distance range differs according to the depth of field. Therefore, for example, even if the depth of field in front and / or the depth of field behind is changed by adjusting the position and / or the aperture of the focusing lens 40B, a predetermined color can be applied to the pixels corresponding to the area of ​​the boundary subject with the same degree of blur (e.g., a blur range of about 0.9 to 1.0 times the pixel size).

[0323] Furthermore, the CPU62 widens the first distance range as the depth of field in front increases and narrows the first distance range as the depth of field in front decreases. Therefore, for example, compared to the case where the width of the first distance range is constant, the visual recognition of the position of the nearest point in the depth of field can be improved even if the depth of field in front changes.

[0324] Similarly, the CPU62 widens the second distance range as the depth of the background increases and narrows the second distance range as the depth of the background decreases. Therefore, for example, compared to the case where the width of the second distance range is constant, the visual recognition of the position of the far point in the depth of field can be improved even if the depth of the background changes.

[0325] Furthermore, the CPU 62 can change the width of the first distance range based on the number of pixels corresponding to the first distance range among the multiple pixels constituting the image displayed on the display 28. For example, the CPU 62 can widen the width of the first distance range as the number of pixels corresponding to the first distance range decreases. In this case, the width of the first distance range varies depending on the number of pixels corresponding to the first distance range. Therefore, for example, compared to the case where the width of the first distance range is constant, even if the number of pixels corresponding to the first distance range changes, the visual recognition of the position of the nearest point in the depth of field can be improved.

[0326] Similarly, the CPU 62 can change the width of the second distance range based on the number of pixels corresponding to the second distance range among the multiple pixels constituting the image displayed on the display 28. For example, the CPU 62 can widen the width of the second distance range as the number of pixels corresponding to the second distance range decreases. In this case, the width of the second distance range varies depending on the number of pixels corresponding to the second distance range. Therefore, for example, compared to the case where the width of the second distance range is constant, even if the number of pixels corresponding to the second distance range changes, the visual recognition of the position of the far point in the depth of field can be improved.

[0327] Furthermore, the CPU 62 can adjust the width of the first distance range based on the object distance. For example, the CPU 62 can widen the first distance range as the object distance increases and narrow the first distance range as the object distance decreases. In this case, for example, compared to the case where the width of the first distance range is constant, the visual recognition of the position of the nearest point in the depth of field can be improved even when the object distance changes.

[0328] Similarly, the CPU 62 can adjust the width of the second distance range based on the object distance. For example, the CPU 62 can widen the second distance range as the object distance increases and narrow the second distance range as the object distance decreases. In this case, for example, compared to the case where the width of the second distance range is constant, the visual recognition of the position of the far point in the depth of field can be improved even when the object distance changes.

[0329] Furthermore, the CPU62 can change the width of the first distance range and / or the width of the second distance range based on at least one of the focal length and aperture value.

[0330] [Fourth Implementation]

[0331] As an example, such as Figure 20 As shown, in the fourth embodiment, the structure of the camera device 10 is changed as follows compared to the first embodiment.

[0332] That is, in addition to the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near point distance acquisition unit 110, the far point distance acquisition unit 112, the first subject determination unit 114, the second subject determination unit 118, the second dynamic image data generation unit 120, the dynamic image data output unit 122, and the dynamic image data storage control unit 124, the CPU 62 also operates as the depth of field acquisition unit 150 and the allowable circle of confusion diameter acquisition unit 152.

[0333] The operation of the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near point distance acquisition unit 110, the far point distance acquisition unit 112, the first subject determination unit 114, the second subject determination unit 118, the second dynamic image data generation unit 120, the dynamic image data output unit 122, and the dynamic image data storage control unit 124 is the same as in the first embodiment.

[0334] Hereinafter, the differences between the operation of the depth-of-field acquisition unit 150 and the permissible circle of confusion diameter acquisition unit 152 of the camera device 10 according to the fourth embodiment and the camera device 10 according to the first embodiment will be explained.

[0335] The depth-of-field acquisition unit 150 acquires the object distance L, focal length f, aperture value F, and predetermined allowable circle of confusion diameter δ. a Then, the depth-of-field acquisition unit 150 determines the depth of field based on the object distance L, focal length f, aperture value F, and a predetermined allowable circle of confusion diameter δ. a And the first rear depth of field D is calculated using the following formula (13). 2a Obtain the first rear depth of field D 2a The first rear depth of field D 2a This is an example of the "first depth of field" involved in the technology of this invention.

[0336] [Formula 13]

[0337]

[0338] In this case, the depth-of-field acquisition unit 150 acquires the object distance L, focal length f, aperture value F, and predetermined allowable circle of confusion diameter δ from the following key points: a That is, the depth-of-field acquisition unit 150 acquires the object distance L acquired by the object distance acquisition unit 108. Furthermore, the depth-of-field acquisition unit 150 acquires, for example, the focal length f pre-stored in the NVM 64. Also, the depth-of-field acquisition unit 150 acquires the aperture value F corresponding to the aperture amount detected by the aperture sensor 42C. And, for example, when the user assigns a predetermined allowable circle of confusion diameter δ to the receiving device 76... aAt that time, the depth-of-field acquisition unit 150 acquires the predetermined allowable circle of confusion diameter δ received by the receiving device 76. a The predetermined permissible circle diameter is an example of the "predetermined permissible circle diameter" involved in the technology of this invention.

[0339] As an example, in Figure 21 The image shows the depth of field D behind the first element. 2a The corresponding allowable circle diameter δ. Figure 21 The first rear depth of field D shown 2a In the relationship between the allowable circle of confusion diameter δ and the first rear depth of field D 2a The depth is divided into four ranges: a first range, a second range, a third range, and a fourth range. The first range is shallower than the first depth. The second range is greater than the first depth but shallower than the second depth. The third range is greater than the second depth but shallower than the third depth. The fourth range is greater than the third depth. The first depth is an example of the "first predetermined depth" as understood in this invention.

[0340] Within the first range, as the first rear depth of field D... 2a As the depth of field becomes shallower, the allowable circle of confusion diameter δ decreases. That is, when the first rear depth of field D... 2a When the depth is shallower than the first depth, the permissible circle of confusion diameter δ is smaller than the first value. In the second range, the permissible circle of confusion diameter δ remains constant at the first value. That is, when the first depth of field D is shallower... 2a When the depth is in the second range, the permissible circle of confusion diameter δ is the first value. The first value is an example of the "first predetermined value" involved in the technology of this invention. In the third range, as the first rear depth of field D... 2a As the depth of field increases, the allowable diameter of the circle of confusion, δ, also increases. That is, when the first rear depth of field, D... 2a When the depth is in the third range, the permissible circle of confusion diameter δ is greater than the first value and less than the second value. In the fourth range, the permissible circle of confusion diameter δ is constant at the second value. That is, when the first rear depth of field D... 2a When the depth is in the fourth range, the allowable diameter δ of the dispersion circle is the second value. Figure 21 The first rear depth of field D shown 2a The relationship between the allowable dispersion circle diameter δ and the data is stored as relational data in [database name missing]. Figure 20 In the NVM64 shown.

[0341] The allowable circle of confusion diameter acquisition unit 152 extracts the distance from the first rear depth of field D based on the relational data. 2a The corresponding allowable circle diameter δ is obtained.

[0342] Next, refer to Figure 22 The operation of the camera device 10 according to the fourth embodiment will be explained. Figure 22An example of a portion of the dynamic image generation process involved in the fourth embodiment is shown.

[0343] In the motion image generation process according to the fourth embodiment, steps ST10 to ST14 are the same as in the first embodiment. After performing the processing in step ST14, Figure 22 The dynamic image generation process shown is transferred to step ST50.

[0344] In step ST50, the depth-of-field acquisition unit 150 determines the depth of field based on the object distance L, focal length f, aperture value F, and predetermined allowable circle of confusion diameter δ. a Obtain the first rear depth of field D 2a After the processing in step ST50 is performed, the image generation process moves to step ST51.

[0345] In step ST51, the allowable circle of confusion diameter acquisition unit 152 acquires the distance from the first rear depth of field D based on the relational data. 2a The corresponding allowable circle of confusion diameter δ. After performing the processing in step ST51, the image generation process proceeds to step ST15.

[0346] In the motion image generation process according to the fourth embodiment, steps ST15 to ST16 are the same as in the first embodiment. After performing the processing in step ST16, the motion image generation process proceeds to... Figure 12B Step ST17 is shown. In the motion image generation process according to the fourth embodiment, steps ST17 to ST28 (see reference) Figure 12B (Same as the first embodiment)

[0347] As explained above, in the imaging device 10 according to the fourth embodiment, the CPU 62 changes the permissible circle of confusion diameter δ based on the object distance L, focal length f, and aperture value F in the imaging lens 40. That is, the permissible circle of confusion diameter δ varies depending on at least one of the object distance L, focal length f, and aperture value F in the imaging lens. Therefore, it is possible to adjust the blur amount to correspond to the object distance L, focal length f, and aperture value F.

[0348] Furthermore, the CPU62 determines the object distance L, focal length f, aperture value F, and predetermined allowable circle of confusion diameter δ. a Obtain the first rear depth of field D 2a Then, when the first rear depth of field D... 2a When the depth is shallower than the first depth, the CPU62 sets the allowable blur circle diameter δ to a value smaller than the first value. Therefore, for example, compared to the case where the allowable blur circle diameter δ is constant, the amount of blur can be limited to a range where the user is not likely to perceive the blur.

[0349] Furthermore, in the camera device 10 according to the fourth embodiment, the CPU 62 can change the allowable circle of confusion diameter δ according to at least one of the object distance L, focal length f, and aperture value F in the camera lens 40.

[0350] Furthermore, in the camera device 10 according to the fourth embodiment, the first rear depth of field D 2a The relationship between the allowable circle of confusion diameter δ and the data is pre-set as relational data. Then, the CPU 62 uses the relational data and the first rear depth of field D to perform the calculations. 2a Set the allowable blur circle diameter δ. However, this is related to the first rear depth of field D. 2a Similar to the relationship between the allowable circle of confusion diameter δ and the first forward depth of field D 1a The relationship between the allowable circle of confusion diameter δ and the relationship data can also be preset as relational data. Then, the CPU62 can determine the relationship data and the depth of field D from the first front view. 1a Set the allowable blur circle diameter δ. In this case, for example, compared to the case where the allowable blur circle diameter δ is constant, it is possible to limit the amount of blur to a range that is not easily perceived by the user.

[0351] Furthermore, when the first rear depth of field D is... 2a and the first foreground depth of field D 1a When the average depth of field is used as the average depth of field, the relationship between the average depth of field and the allowable circle of confusion diameter δ can be preset as relational data. Then, the CPU62 can set the allowable circle of confusion diameter δ from the average depth of field based on the relational data. In this case, for example, compared to the case where the allowable circle of confusion diameter δ is constant, the amount of blur can be limited to a range where the blur is not easily perceived by the user.

[0352] [Fifth Implementation]

[0353] As an example, such as Figure 23 As shown, in the fifth embodiment, the structure of the camera device 10 is changed as follows compared to the first embodiment.

[0354] That is, in addition to the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near point distance acquisition unit 110, the far point distance acquisition unit 112, the first subject determination unit 114, the second subject determination unit 118, the second dynamic image data generation unit 120, the dynamic image data output unit 122, and the dynamic image data storage control unit 124, the CPU 62 also functions as the motion mode determination unit 160.

[0355] The operation of the first camera control unit 100, the first dynamic image data generation unit 102, the second camera control unit 104, the distance data acquisition unit 106, the object distance acquisition unit 108, the near-point distance acquisition unit 110, the far-point distance acquisition unit 112, the first subject determination unit 114, the second subject determination unit 118, and the dynamic image data storage control unit 124 is the same as in the first embodiment. In the fifth embodiment, the operation of the second dynamic image data generation unit 120 and the dynamic image data output unit 122 differs from that in the first embodiment.

[0356] Hereinafter, the differences between the operation of the motion mode determination unit 160, the second motion image data generation unit 120 and the motion image data output unit 122 of the camera device 10 according to the fifth embodiment and the camera device 10 according to the first embodiment will be described.

[0357] The motion mode determination unit 160 determines whether the motion mode of the camera device 10 is either a real-time preview image display mode or a motion image recording mode. The real-time preview image display mode is a mode in which a real-time preview image is displayed on the display 28 based on motion image data. The motion image recording mode is a mode in which an image is displayed on the display 28 based on motion image data, and motion image recording data is stored in the NVM 64.

[0358] When the motion mode determination unit 160 determines that the motion mode of the camera device 10 is the live preview image display mode, the second motion image data generation unit 120 generates focus position data representing the area of ​​the focused subject 93 within the object distance based on distance data. Specifically, the second motion image data generation unit 120 generates focus position data based on the following points.

[0359] That is, the second dynamic image data generation unit 120 acquires focus area data representing the area of ​​the focused subject 93 based on distance data, wherein the focused subject 93 exists for each photosensitive pixel 72B (reference). Figure 3 The focus area data is obtained within the distance equidistant from the distance to the object. For example, the address of the photosensitive pixel 72B is used to represent the focus area data. That is, the second dynamic image data generation unit 120 obtains the address of the photosensitive pixel 72B that is equidistant from the object from among the multiple photosensitive pixels 72B based on the distance data. The address of the photosensitive pixel 72B is a coordinate determined for each photosensitive pixel 72B, and represents the vertical and horizontal coordinates of the photoelectric conversion element 72.

[0360] Next, the second dynamic image data generation unit 120 generates focus position data based on the focus area data. The focus position data is data that, for the image displayed on the display 28 based on the first dynamic image data, represents the area of ​​the focused subject 93 within the object distance in a third manner, different from the first and second methods described above (i.e., the area represented by the third area data). As an example, the focus position data is data used for third image processing, which marks the area of ​​the focused subject 93 within the object distance on the image displayed on the display 28.

[0361] As an example, the third image processing involves assigning a third predetermined color to the pixels corresponding to the area of ​​the focused subject 93 within the object distance (i.e., the pixels corresponding to the address of the photosensitive pixel 72B represented by the focus area data) among the pixels constituting the image displayed on the display 28. The process of assigning the third predetermined color involves, for example, replacing the pixel's signal value with a value corresponding to the third predetermined color. The third predetermined color is a color different from the first and second predetermined colors. The third predetermined color can be achromatic or chromatic. For example, the third predetermined color can be red, blue, or yellow. The focus position data is an example of "focus position data" according to the technology of this invention. The second image processing is an example of "image processing" according to the technology of this invention.

[0362] Then, as an example, such as Figure 23 As shown, there are a first boundary subject 91A and a second boundary subject 91B. When the first subject determination unit 114 determines that there is a first boundary subject 91A at a near distance, and the second subject determination unit 118 determines that there is a second boundary subject 91B at a far distance, and the motion mode determination unit 160 determines that the motion mode of the camera device 10 is the live preview image display mode, the second motion image data generation unit 120 generates second motion image data including first boundary data, second boundary data and focus position data based on the first motion image data.

[0363] Specifically, the second dynamic image data generation unit 120 performs first image processing on pixels corresponding to the region of the first boundary subject 91A located within a near distance from the first dynamic image data displayed on the display 28. Similarly, the second dynamic image data generation unit 120 performs second image processing on pixels corresponding to the region of the second boundary subject 91B located within a far distance from the first dynamic image data displayed on the display 28. Furthermore, the second dynamic image data generation unit 120 performs third image processing on pixels corresponding to the region of the focusing subject 93 located within an object distance from the first dynamic image data displayed on the display 28. Therefore, the second dynamic image data is generated by the second dynamic image data generation unit 120. The second dynamic image data represents an image in which a first predetermined color is attached to the pixels corresponding to the region of the first boundary subject 91A that exists within the near point distance, a second predetermined color is attached to the pixels corresponding to the region of the second boundary subject 91B that exists within the far point distance, and a third predetermined color is attached to the pixels corresponding to the region of the focusing subject 93 that exists within the object distance.

[0364] When the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance, the second subject determination unit 118 determines that a second boundary subject 91B exists at a far distance, and the operation mode determination unit 160 determines that the operation mode of the camera device 10 is a live preview image display mode, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the first boundary data, the second boundary data, and the focus position data) to the display 28 as display motion image data. The display 28 displays an image based on the display motion image data. In this case, the image displayed on the display 28 is a live preview image.

[0365] As an example, Figure 24 The text is a jumbled mix of Chinese characters and symbols, making it impossible to translate coherently. It appears to be a collection Figure 23 In the example shown, the generated second dynamic image data is displayed as image 200 on monitor 28. Figure 24 In the image 200 shown, the area of ​​the focused subject 93 and the area other than the focused subject 93 are clearly distinguished. Furthermore, in Figure 24 In the image 200 shown, the area of ​​the subject 93 in focus is represented by regions distinct from the regions of the first boundary subject 91A and the second boundary subject 91B. That is, as an example, in... Figure 24In the image 200 shown, the area of ​​the first boundary subject 91A is represented by a first predetermined color, the area of ​​the second boundary subject 91B is represented by a second predetermined color, and the area of ​​the in-focus subject 93 is represented by a third predetermined color.

[0366] Furthermore, when the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance, the second subject determination unit 118 determines that a second boundary subject 91B exists at a far distance, and the motion mode determination unit 160 determines that the motion mode of the camera device 10 is a dynamic image recording mode, the second dynamic image data generation unit 120 generates second dynamic image data including the first boundary data and the second boundary data based on the first dynamic image data.

[0367] When the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance, the second subject determination unit 118 determines that a second boundary subject 91B exists at a far distance, and the operation mode determination unit 160 determines that the operation mode of the camera device 10 is a dynamic image recording mode, the dynamic image data output unit 122 outputs the second dynamic image data generated by the second dynamic image data generation unit 120 (i.e., the second dynamic image data including the first boundary data and the second boundary data) to the display 28 as dynamic image data for display.

[0368] Furthermore, when the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance, the second subject determination unit 118 determines that a second boundary subject 91B does not exist at a far distance, and the action mode determination unit 160 determines that the action mode of the camera device 10 is the real-time preview image display mode, the second motion image data generation unit 120 generates second motion image data including first boundary data and focus position data based on the first motion image data.

[0369] When the first subject determination unit 114 determines that a first boundary subject 91A exists at the near distance, the second subject determination unit 118 determines that a second boundary subject 91B does not exist at the far distance, and the motion mode determination unit 160 determines that the motion mode of the camera device 10 is the real-time preview image display mode, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the first boundary data and the focus position data) to the display 28 as motion image data for display.

[0370] Furthermore, when the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance, the second subject determination unit 118 determines that a second boundary subject 91B does not exist at a far distance, and the motion mode determination unit 160 determines that the motion mode of the camera device 10 is a motion image recording mode, the second motion image data generation unit 120 generates second motion image data including the first boundary data based on the first motion image data.

[0371] When the first subject determination unit 114 determines that a first boundary subject 91A exists at a near distance, the second subject determination unit 118 determines that a second boundary subject 91B does not exist at a far distance, and the operation mode determination unit 160 determines that the operation mode of the camera device 10 is a dynamic image recording mode, the dynamic image data output unit 122 outputs the second dynamic image data generated by the second dynamic image data generation unit 120 (i.e., the second dynamic image data including the first boundary data) to the display 28 as dynamic image data for display.

[0372] Furthermore, when the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance, the second subject determination unit 118 determines that there is a second boundary subject 91B within the far distance, and the action mode determination unit 160 determines that the action mode of the camera device 10 is the real-time preview image display mode, the second motion image data generation unit 120 generates second motion image data including second boundary data and focus position data based on the first motion image data.

[0373] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance, the second subject determination unit 118 determines that there is a second boundary subject 91B within the far distance, and the motion mode determination unit 160 determines that the motion mode of the camera device 10 is the real-time preview image display mode, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including the second boundary data and the focus position data) to the display 28 as display motion image data.

[0374] Furthermore, when the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance, the second subject determination unit 118 determines that there is a second boundary subject 91B within the far distance, and the motion mode determination unit 160 determines that the motion mode of the camera device 10 is the motion image recording mode, the second motion image data generation unit 120 generates second motion image data including the second boundary data based on the first motion image data.

[0375] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance, the second subject determination unit 118 determines that there is a second boundary subject 91B within the far distance, and the operation mode determination unit 160 determines that the operation mode of the camera device 10 is the dynamic image recording mode, the dynamic image data output unit 122 outputs the second dynamic image data generated by the second dynamic image data generation unit 120 (i.e., the second dynamic image data including the second boundary data) to the display 28 as dynamic image data for display.

[0376] Furthermore, when the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance, the second subject determination unit 118 determines that there is no second boundary subject 91B within the far distance, and the action mode determination unit 160 determines that the action mode of the camera device 10 is the real-time preview image display mode, the second motion image data generation unit 120 generates second motion image data including focus position data based on the first motion image data.

[0377] When the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance, the second subject determination unit 118 determines that there is no second boundary subject 91B within the far distance, and the action mode determination unit 160 determines that the action mode of the camera device 10 is the real-time preview image display mode, the motion image data output unit 122 outputs the second motion image data generated by the second motion image data generation unit 120 (i.e., the second motion image data including focus position data) to the display 28 as motion image data for display.

[0378] Furthermore, when the first subject determination unit 114 determines that there is no first boundary subject 91A within the near distance, the second subject determination unit 118 determines that there is no second boundary subject 91B within the far distance, and the operation mode determination unit 160 determines that the operation mode of the camera device 10 is the dynamic image recording mode, the dynamic image data output unit 122 outputs the first dynamic image data to the display 28 as dynamic image data for display.

[0379] When the motion mode determination unit 160 determines that the motion mode of the camera device 10 is a dynamic image recording mode, the image displayed on the display 28 based on the second dynamic image data is a recording image (e.g., a backward browsing image).

[0380] A real-time preview image is an example of a "display image" according to the technology of this invention. A recording image is an example of a "recording image" according to the technology of this invention. Focus position data is an example of "focus position data" according to the technology of this invention. Second dynamic image data including focus position data is an example of "display image data" according to the technology of this invention.

[0381] Next, refer to Figure 25A and Figure 25B The operation of the camera device 10 according to this fifth embodiment will be explained. Figure 25A and Figure 25B An example of a portion of the dynamic image generation process involved in the fifth embodiment is shown.

[0382] In the dynamic image generation process according to the fifth embodiment, steps ST10 to ST16 (refer to...) Figure 12A )and Figure 25A Steps ST17 to ST18 shown are the same as in the first embodiment. After performing the process in step ST18, Figure 25A The dynamic image generation process shown is transferred to step ST60.

[0383] In step ST60, the motion mode determination unit 160 determines whether the motion mode of the camera device 10 is a real-time preview image display mode or a motion image recording mode. In step ST60, if the motion mode of the camera device 10 is a real-time preview image display mode, the motion image generation process proceeds to step ST61. In step ST60, if the motion mode of the camera device 10 is a motion image recording mode, the motion image generation process proceeds to step ST19. Figure 25A Steps ST19 and ST20 shown are the same as in the first embodiment.

[0384] In step ST61, the second dynamic image data generation unit 120, according to step ST13 (reference) Figure 12A The distance data obtained from the first dynamic image data is used to generate focus position data representing the area of ​​the focused subject 93 within the object's distance. Then, the second dynamic image data generation unit 120 generates second dynamic image data including first boundary data, second boundary data, and focus position data based on the first dynamic image data. After performing the processing in step ST61, the image generation process proceeds to step ST62.

[0385] In step ST62, the dynamic image data output unit 122 outputs the second dynamic image data generated in step ST61 (i.e., the second dynamic image data including the first boundary data, the second boundary data, and the focus position data) to the display 28 as dynamic image data for display. After the processing in step ST62 is performed, the image generation process moves to step ST27.

[0386] Figure 25A When the dynamic image generation process shown is rejected in step ST18, the process proceeds to step ST63.

[0387] In step ST63, the motion mode determination unit 160 determines whether the motion mode of the camera device 10 is a real-time preview image display mode or a motion image recording mode. In step ST63, if the motion mode of the camera device 10 is a real-time preview image display mode, the motion image generation process proceeds to step ST64. In step ST63, if the motion mode of the camera device 10 is a motion image recording mode, the motion image generation process proceeds to step ST21. Figure 25A Steps ST21 and ST22 shown are the same as in the first embodiment.

[0388] In step ST64, the second dynamic image data generation unit 120 generates focus position data representing the area of ​​the focused subject 93 within the object distance, based on the distance data acquired in step ST13. Then, the second dynamic image data generation unit 120 generates second dynamic image data, including first boundary data and focus position data, based on the first dynamic image data. After performing the processing in step ST64, the image generation process proceeds to step ST65.

[0389] In step ST65, the dynamic image data output unit 122 outputs the second dynamic image data generated in step ST64 (i.e., the second dynamic image data including the first boundary data and focus position data) to the display 28 as dynamic image data for display. After the processing in step ST65 is performed, the image generation process proceeds to step ST27.

[0390] Figure 25A When the dynamic image generation process shown is rejected in step ST1 7, it is transferred to... Figure 25B Step ST23 is shown. Figure 25B Step ST23 shown is the same as in the first embodiment. After performing the process of step ST23, Figure 25B The dynamic image generation process shown is then transferred to step ST66.

[0391] In step ST66, the motion mode determination unit 160 determines whether the motion mode of the camera device 10 is a real-time preview image display mode or a motion image recording mode. In step ST66, if the motion mode of the camera device 10 is a real-time preview image display mode, the motion image generation process proceeds to step ST67. In step ST66, if the motion mode of the camera device 10 is a motion image recording mode, the motion image generation process proceeds to step ST24. Figure 25B Steps ST24 and ST25 shown are the same as in the first embodiment.

[0392] In step ST67, the second dynamic image data generation unit 120, according to step ST13 (reference) Figure 12A The distance data obtained from the first dynamic image data is used to generate focus position data representing the area of ​​the focused subject 93 within the object's distance. Then, the second dynamic image data generation unit 120 generates second dynamic image data including second boundary data and focus position data based on the first dynamic image data. After performing the processing in step ST67, the image generation process proceeds to step ST68.

[0393] In step ST68, the dynamic image data output unit 122 outputs the second dynamic image data generated in step ST67 (i.e., the second dynamic image data including the second boundary data and focus position data) to the display 28 as dynamic image data for display. After the processing in step ST68 is performed, the image compositing process is transferred to the display 28. Figure 25A Step ST27 is shown.

[0394] Figure 25B When the dynamic image generation process shown is rejected in step ST23, the process proceeds to step ST69.

[0395] In step ST69, the motion mode determination unit 160 determines whether the motion mode of the camera device 10 is a real-time preview image display mode or a motion image recording mode. In step ST69, if the motion mode of the camera device 10 is a real-time preview image display mode, the motion image generation process proceeds to step ST70. In step ST69, if the motion mode of the camera device 10 is a motion image recording mode, the motion image generation process proceeds to step ST26. Figure 25B Step ST26 shown is the same as in the first embodiment.

[0396] In step ST70, the second dynamic image data generation unit 120 generates focus position data representing the area of ​​the focused subject 93 within the object distance, based on the distance data acquired in step ST13. Then, the second dynamic image data generation unit 120 generates second dynamic image data including the focus position data based on the first dynamic image data. After the processing in step ST70 is performed, the image generation process proceeds to step ST71.

[0397] In step ST71, the dynamic image data output unit 122 outputs the second dynamic image data generated in step ST70 (i.e., the second dynamic image data including focus position data) to the display 28 as dynamic image data for display. After the processing in step ST71 is performed, the image compositing process is transferred to... Figure 25A Step ST27 is shown.

[0398] In the dynamic image generation process according to the fifth embodiment, Figure 25A Steps ST27 and ST28 shown are the same as in the first embodiment.

[0399] As explained above, in the imaging device 10 according to the fifth embodiment, when a display image is displayed on the display 28, the CPU 62 generates display image data representing the display image by including focus position data representing the area of ​​the focused subject 93 in the moving image data based on distance data. Then, the CPU 62 outputs the display image data to the display 28. Therefore, when the display image is displayed on the display 28, for example, a user can confirm the position of the pixel with the third predetermined color by looking at the image displayed on the display 28, thereby mastering the focus position. On the other hand, when a recording image is displayed on the display 28, the CPU 62 outputs the first moving image data to the display 28. Therefore, when the recording image is displayed on the display 28, it is possible to prevent the area of ​​the focused subject 93 from being displayed in the image with color.

[0400] Furthermore, in the fifth embodiment, the second dynamic image data generation unit 120 can generate the second dynamic image data in stages when generating the second dynamic image data including the first boundary data, the second boundary data, and the focus position data. That is, for example, the second dynamic image data generation unit 120 can generate a first temporary dynamic image data including the first boundary data based on the first dynamic image data, and generate a second temporary dynamic image data including the second boundary data based on the first temporary dynamic image data, and generate a second dynamic image data including the focus position data based on the second temporary dynamic image data.

[0401] The first to fifth embodiments have been described above. However, the above embodiments and modifications can be combined with each other as long as they do not contradict each other. Furthermore, when combining the above embodiments and modifications, if there are multiple repeated steps, priority can be assigned to the multiple steps according to various conditions.

[0402] Furthermore, in the embodiments described above, the display video data is output to the display 28 of the camera device 10. However, the display video data may, for example, be output to an EVF. Also, the display video data may be output to an external display device located outside the camera device 10.

[0403] Furthermore, while CPU 62 is exemplified in the above embodiments, it may also be used in place of CPU 62 or in conjunction with at least one other CPU, at least one GPU, and / or at least one TPU.

[0404] Furthermore, while examples of NVM 64 storing program 65 have been described in the above embodiments, the technology of the present invention is not limited thereto. For example, program 65 may be stored in a portable, non-transitory computer-readable storage medium (hereinafter referred to as "non-transitory storage medium") such as an SSD or USB memory. Program 65 stored in non-transitory storage medium is installed in the controller 12 of the camera device 10. CPU 62 executes dynamic image generation processing according to program 65.

[0405] Furthermore, the program 65 can also be stored in the storage device of other computers or server devices connected to the camera device 10 via a network, and the program 65 can be downloaded according to the request of the camera device 10 and installed on the controller 12.

[0406] In addition, instead of storing all programs 65 in the storage devices of other computers or server devices connected to the camera device 10 or in the NVM 64, a portion of the programs 65 can be stored.

[0407] and, Figure 1 and Figure 2 The camera device 10 shown has a built-in controller 12, but the technology of the present invention is not limited thereto. For example, the controller 12 can also be set outside the camera device 10.

[0408] Furthermore, while the above embodiments illustrate a controller 12 including a CPU 62, an NVM 64, and a RAM 66, the technology of the present invention is not limited thereto. The controller 12 can be used in place of devices including ASICs, FPGAs, and / or PLDs. Moreover, a combination of hardware and software structures can be used instead of the controller 12.

[0409] Furthermore, various processors, as described below, can be used as the hardware resources for performing the motion image generation processing described in the above embodiments. For example, a CPU can be used, which is a general-purpose processor that performs the function of executing the motion image generation processing hardware resources by executing software (i.e., a program). Also, a dedicated circuit can be used, such as an FPGA, PLD, or ASIC, which has a circuit structure specifically designed for performing a particular process. Each processor has a built-in or connected memory, and each processor performs the motion image generation processing using the memory.

[0410] The hardware resources for performing motion image generation processing can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resources for performing motion image generation processing can be a single processor.

[0411] As an example of a single processor, firstly, there is a method where a processor is composed of a combination of one or more CPUs and software, and this processor performs the functions of hardware resources for executing dynamic image generation processing. Secondly, there is a method, such as a System-on-a-Chip (SoC), where a single IC chip implements the overall system functions, including multiple hardware resources for performing dynamic image generation processing. Thus, dynamic image generation processing is implemented using one or more of the aforementioned processors as hardware resources.

[0412] Furthermore, the hardware structure of these various processors, more specifically, can utilize circuits composed of combined semiconductor elements and other circuit components. Moreover, the aforementioned dynamic image generation process is merely one example. Therefore, without departing from the main point, unnecessary steps can certainly be removed, new steps added, or the processing order switched.

[0413] The above description and illustrations are detailed explanations of the parts related to the technology of this invention, and are merely one example of the technology of this invention. For example, the descriptions of the structure, function, effect, and effect described above are examples of the structure, function, effect, and effect of the parts related to the technology of this invention. Therefore, it is of course possible to delete unnecessary parts, add new elements, or replace the above description and illustrations without departing from the spirit of the technology of this invention. Furthermore, in order to avoid trouble and facilitate understanding of the parts related to the technology of this invention, the descriptions of technical common sense that does not require special explanation when implementing the technology of this invention have been omitted in the above description and illustrations.

[0414] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when three or more items are associated and expressed using "and / or", the same meaning as "A and / or B" applies.

[0415] All documents, patent applications and technical standards described in this specification may be referenced in this specification to the same extent as the specific and individually described instances of reference to each document, patent application and technical standard.

Claims

1. A camera device comprising an image sensor and a processor, wherein, The processor performs the following processing: Acquire distance data regarding distance, wherein the distance is the distance between multiple subjects within the camera area captured by the image sensor and the camera device; Boundary data is generated based on the distance data, and the boundary data represents the region of the subject within the distance of the boundary part of the depth of field; Based on image data captured by the image sensor, dynamic image data including the boundary data is generated; and Output the dynamic image data.

2. The camera device according to claim 1, wherein, The processor outputs the dynamic image data as data for displaying the first image on the first display, wherein the first image distinguishes the area of ​​the boundary subject and the area other than the boundary subject.

3. The camera device according to claim 1 or 2, wherein, The boundary data is data used for image processing of the area of ​​the boundary subject marked by the second image displayed on the second display based on the image data.

4. The camera device according to claim 3, wherein, The image processing involves assigning a predetermined color to the first pixel among the plurality of first pixels constituting the second image that corresponds to the region of the boundary subject.

5. The camera device according to claim 3, wherein, The image processing involves applying a predetermined brightness to the second pixel among the plurality of second pixels constituting the second image that corresponds to the region of the boundary subject.

6. The camera device according to claim 3, wherein, The image processing involves attaching a marker to the second image to indicate the region of the boundary subject.

7. The camera device according to claim 3, wherein, The image processing involves overlaying a distance image generated based on the distance data onto the second image.

8. The camera device according to claim 1 or 2, wherein, The boundary portion includes: The first boundary portion is located on the near point side of the depth of field; and The second boundary portion is located on the far side of the depth of field. The boundary subject includes: The first boundary subject exists within the distance of the first boundary portion; and The second boundary subject exists within the distance of the second boundary portion. The boundary data includes: The first boundary data represents the region of the subject at the first boundary; and The second boundary data represents the region of the subject at the second boundary.

9. The camera device according to claim 8, wherein, The first boundary data is data representing the area of ​​the subject at the first boundary in a first manner from the third image displayed on the third display based on the image data. The second boundary data is data representing the region of the subject on the second boundary in the third image in a second manner different from the first manner.

10. The camera device according to claim 1 or 2, wherein, The boundary portion is at least one of the near point and the far point of the depth of field.

11. The camera device according to claim 10, wherein, The processor performs the following processing: Based on the distance data, region data representing the area of ​​the boundary subject is obtained, wherein the boundary subject exists within a distance equal to the distance between the plurality of subjects and the camera device. and The boundary data is generated based on the region data.

12. The camera device according to claim 1 or 2, wherein, The boundary portion is at least one of the range including the near point of the depth of field and the range including the far point of the depth of field.

13. The camera device according to claim 12, wherein, The range including the near point of the depth of field is the range extending from the near point of the depth of field to the far point of the depth of field.

14. The camera device according to claim 12, wherein, The range including the far point of the depth of field is the range extending from the far point of the depth of field to the near point of the depth of field.

15. The camera device according to claim 1 or 2, wherein, The processor performs the following processing: Define a distance range including the distance to the boundary portion; Based on the distance data, region data representing the area of ​​the boundary subject is obtained, wherein the boundary subject exists within the distance range between the plurality of subjects and the camera device. and The boundary data is generated based on the region data.

16. The camera device according to claim 1 or 2, wherein, The width of the boundary varies depending on the depth of the field.

17. The camera device according to claim 16, wherein, The processor widens the boundary portion as the depth of field increases and narrows the boundary portion as the depth of field decreases.

18. The camera device according to claim 1 or 2, wherein, The width of the boundary portion varies depending on the number of pixels, where each pixel is a pixel corresponding to the boundary portion among a plurality of pixels constituting a fourth image displayed on a fourth display based on the dynamic image data.

19. The camera device according to claim 1 or 2, wherein, The camera device is equipped with a camera lens. The permissible circle of confusion diameter of the image sensor varies depending on at least one of the object distance, focal length, and aperture value in the camera lens.

20. The camera device according to claim 19, wherein, The processor obtains the first depth of field based on the object distance, the focal length, the aperture value, and the predetermined allowable circle of confusion diameter. When the depth of the first depth of field is shallower than the first predetermined depth, the diameter of the permissible circle of confusion is smaller than the first predetermined value.

21. The camera device according to claim 1 or 2, wherein, The processor performs the following processing: When the display image is displayed on the fifth display, display image data representing the display image is generated by including the focus position data and the boundary data in the dynamic image data according to the distance data, wherein the focus position data represents the area of ​​the focused subject that exists within the object distance among the plurality of subjects; The image data for display is output to the fifth display. and When the recorded image is displayed on the fifth display, the dynamic image data is output to the fifth display.

22. The camera device according to claim 1 or 2, wherein, The processor stores the image data in a non-transitory storage medium.

23. The camera device according to claim 1 or 2, wherein, The image sensor has multiple phase difference pixels. The processor obtains the distance data based on the phase difference pixel data output from the phase difference pixels.

24. The camera device according to claim 23, wherein, The phase difference pixel is a pixel that selectively outputs non-phase difference pixel data and the phase difference pixel data. The non-phase difference pixel data is pixel data obtained by photoelectric conversion of the entire region of the phase difference pixel. The phase difference pixel data is pixel data obtained by photoelectric conversion of a portion of the phase difference pixel region.

25. A method for taking a photograph, comprising the following steps: Acquire distance data regarding distance, wherein the distance is the distance between multiple subjects within the camera area captured by the image sensor of the camera device and the camera device itself; Boundary data is generated based on the distance data, and the boundary data represents the region of the subject within the distance of the boundary part of the depth of field; Based on image data captured by the image sensor, dynamic image data including the boundary data is generated; and Output the dynamic image data.

26. A storage medium storing a program for causing a computer to perform a process comprising the following steps: Acquire distance data regarding distance, wherein the distance is the distance between multiple subjects within the camera area captured by the image sensor of the camera device and the camera device itself; Boundary data is generated based on the distance data, and the boundary data represents the region of the subject within the distance of the boundary part of the depth of field; Based on image data captured by the image sensor, dynamic image data including the boundary data is generated; and Output the dynamic image data.

Images