Imaging device, information processing method, and program

Abstract

This imaging device comprises an image sensor and a processor. The processor: acquires information regarding distances at a plurality of positions within an imaging area of the image sensor; generates video image data represented with a vertical axis and a horizontal axis on the basis of imaging data acquired by the image sensor; generates distance image data with a first axis corresponding to the vertical axis or the horizontal axis, and a second axis representing information regarding distances; and outputs the video image data and the distance image data.

Description

[Technical Field]

[0001] The technology disclosed herein relates to an imaging device, an information processing method, and a program. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2012-230258 discloses an imaging device that extracts high-frequency components from a video signal of a predetermined area on the screen to generate a TV-AF evaluation value signal, and then compiles and captures the outputs of pixels used for focus detection of the image sensor into two output signals, Image A and Image B. The imaging device described in Japanese Patent Application Laid-Open Publication No. 2012-230258 performs phase difference ranging calculations to determine the amount of focus shift in each part of the imaging area and generates a distance map. When displaying a distance bar for focus adjustment, the maximum value of the scale is determined based on the distance map, and the distance bar is displayed with scale intervals that are equal or logarithmic.

[0003] Japanese Patent Application Laid-Open Publication No. 2019-169769 discloses an image processing device including an imaging unit that captures an image of a subject and a distance map acquisition unit that acquires information related to the distance distribution of the subject as map data. The distance map acquisition unit acquires distance map data or map data of image shift or defocus amount related to the captured image using a time-of-flight (TOF) method or an imaging surface phase difference detection method using a split-pupil image sensor. An image processing unit generates texture image data in which low-frequency components of the captured image are suppressed, and generates image data representing the distance distribution of the subject by combining the texture image data with the map data acquired by the distance map acquisition unit.

[0004] Japanese Patent Laid-Open Publication No. 10-197938 discloses a camera equipped with an aperture position detector, a focus position detector, and a zoom position detector that detect the drive positions of the camera's various drive units, as well as a display element in which small, rectangular display elements are arranged along a distance scale to display the appropriate focus distance range in the viewfinder. In the camera disclosed in Japanese Patent Laid-Open Publication No. 10-197938, a microcomputer controls the display element to display the appropriate distance range calculated based on the detection data. Furthermore, a display mechanism for this appropriate distance range can be located on the lens barrel, and the display of this appropriate distance range can be expressed in perspective. Summary of the Invention

[0005] An embodiment of the technique of the present disclosure provides an imaging device, an information processing method, and a program that allow a user to easily visually recognize the distance to a subject. [Means for solving the problem]

[0006] The imaging device of the present disclosure is an imaging device equipped with an image sensor and a processor, in which the processor acquires information regarding distance at multiple positions within the imaging area of ​​the image sensor, generates moving image data represented by vertical and horizontal axes based on the imaging data obtained by the image sensor, generates distance image data in which the first axis corresponds to the vertical or horizontal axis and the second axis represents information regarding distance, and outputs the moving image data and distance image data.

[0007] The processor preferably outputs the video image data and the depth image data to a display destination.

[0008] The distance image data preferably includes information representing at least one of a first mark indicating the focus distance and a second mark indicating the depth of field.

[0009] It is preferable that the camera is provided with a focus lens, and when the processor receives an operation to change the position of the first mark along the second axis, it controls the movement of the focus lens to a position corresponding to the in-focus distance indicated by the first mark.

[0010] It is preferable that the camera is provided with an aperture, and when the processor receives an operation to change the position of the second mark along the second axis, it controls to change the aperture value of the aperture to a value corresponding to the depth of field represented by the second mark.

[0011] When an operation to change the position of the first mark or the second mark is performed, it is preferable that the processor controls at least one of the focus lens and the aperture so that the target subject in the video image data falls within the depth of field.

[0012] The target subject is preferably a subject extracted by the processor based on the video image data.

[0013] The target subject is preferably a subject obtained by the processor extracting a designated area designated in the video data when an operation to designate an area is performed in the video data.

[0014] It is preferable that the processor performs processing to expand the distance image data in the second axis direction when an operation to increase the depth of field is performed using the second mark.

[0015] It is preferable that the processor performs a process to enlarge the distance image data in the second axis direction when the operation speed remains below a certain speed for a certain period of time during an operation to change the position of the first mark or the second mark.

[0016] When an operation for specifying an area in the depth image data is performed, the processor preferably changes the color of a group of pixels in the video image data that corresponds to the area specified in the depth image data.

[0017] When an operation for specifying an area in the video image data is performed, the processor preferably changes the color of a group of pixels in the depth image data that corresponds to the area specified in the video image data.

[0018] Preferably, the processor extracts a subject area based on the video image data, obtains information relating to the distance corresponding to the extracted subject area, and generates distance image data based on the obtained information relating to the distance.

[0019] When an operation to specify an area in video data is performed, it is preferable that the processor extracts the specified area specified in the video data, obtains information regarding the distance corresponding to the extracted specified area, and generates distance image data based on the obtained information regarding the distance.

[0020] The processor preferably obtains information about the distance based on the imaging data obtained by the image sensor.

[0021] It is preferable that the image sensor includes a plurality of phase difference pixels, and the processor acquires information about distance based on imaging data obtained from the phase difference pixels among the imaging data.

[0022] It is preferable that the phase difference pixel is capable of selectively outputting non-phase difference image data in which photoelectric conversion has been performed by the entire region of the pixel and phase difference image data in which photoelectric conversion has been performed by a partial region of the pixel, and that the processor acquires information regarding distance based on imaging data in the case in which the phase difference pixel outputs phase difference image data.

[0023] The information processing method disclosed herein includes acquiring information regarding distance at multiple positions within an imaging area of ​​an image sensor, generating moving image data represented by a vertical axis and a horizontal axis based on the imaging data obtained by the image sensor, generating distance image data in which a first axis corresponds to the vertical axis or the horizontal axis and a second axis represents information regarding distance, and outputting the moving image data and the distance image data.

[0024] The program disclosed herein causes a computer to perform processes including acquiring information regarding distance at multiple positions within the imaging area of ​​an image sensor, generating moving image data represented by vertical and horizontal axes based on the imaging data obtained by the image sensor, generating distance image data in which a first axis corresponds to the vertical or horizontal axis and a second axis represents information regarding distance, and outputting the moving image data and the distance image data. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic diagram illustrating an example of the overall configuration of an imaging device. [Figure 2] FIG. 1 is a schematic diagram illustrating an example of a hardware configuration of an optical system and an electrical system of an imaging apparatus. [Figure 3] FIG. 2 is a schematic diagram illustrating an example of the configuration of a photoelectric conversion element. [Figure 4] FIG. 2 is a block diagram showing an example of the functions of a processor. [Figure 5] FIG. 2 is a bird's-eye view of an area captured by an imaging device. [Figure 6] FIG. 2 is a diagram showing an example of moving image data. [Figure 7] FIG. 10 is a conceptual diagram illustrating an example of distance image data generation processing. [Figure 8] FIG. 10 is a conceptual diagram illustrating an example of a data output process. [Figure 9] 10 is a flowchart showing an example of the flow of focus assist processing. [Figure 10] FIG. 10 is a block diagram showing an example of the functions of a processor according to a first modified example. [Figure 11] FIG. 10 is a diagram showing an example of distance image data according to a first modified example. [Figure 12] 10 is a flowchart showing an example of the flow of focus assist processing according to a first modified example. [Figure 13] FIG. 10 is a block diagram showing an example of the functions of a processor according to a second modified example. [Figure 14]FIG. 10 is a diagram showing an example of a change operation of the focal distance according to a second modified example. [Figure 15] FIG. 10 is a diagram showing an example of an operation for changing the depth of field according to a second modified example. [Figure 16] 10 is a flowchart showing an example of the flow of a lens control process according to a second modified example. [Figure 17] FIG. 11 is a block diagram showing an example of the functions of a processor according to a third modified example. [Figure 18] 13A and 13B are diagrams illustrating an example of an operation for changing the depth of field according to a third modified example. [Figure 19] 13A and 13B are diagrams illustrating an example of an operation for changing the depth of field according to a fourth modified example. [Figure 20] FIG. 13 is a diagram showing an example of a change operation of the focal distance according to a fourth modified example. [Figure 21] 13 is a flowchart showing an example of the flow of an enlargement process according to a fourth modified example. [Figure 22] FIG. 13 is a diagram showing an example of an area designation operation according to a fifth modified example. [Figure 23] FIG. 13 is a diagram showing an example of an area designation operation according to a sixth modified example. [Figure 24] FIG. 13 is a diagram showing an example of an area designation operation according to a seventh modified example. [Figure 25] FIG. 13 is a diagram illustrating tethered photography according to an eighth modified example. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, examples of an imaging device, an information processing method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". TPU is an abbreviation for "Tensor processing unit". NVM is an abbreviation for "Non-volatile memory". RAM is an abbreviation for "Random Access Memory". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". fps is an abbreviation for "frame per second". MF is an abbreviation for "Manual Focus". AF is an abbreviation for "Auto Focus". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". A / D is an abbreviation for "Analog / Digital". PC is an abbreviation for "Personal Computer".

[0029] As an example, as shown in FIG. 1 , an imaging device 10 is a device that captures an image of a subject, and includes a processor 12, an imaging device body 16, and an interchangeable lens 18. The processor 12 is an example of a "computer" according to the technology of the present disclosure. The processor 12 is built into the imaging device body 16 and controls the entire imaging device 10. The interchangeable lens 18 is interchangeably attached to the imaging device body 16. The interchangeable lens 18 is provided with a focus ring 18A. The focus ring 18A is operated by a user of the imaging device 10 (hereinafter simply referred to as "user") when the user manually adjusts the focus of the imaging device 10 on a subject.

[0030] 1, a digital camera with an interchangeable lens is shown as an example of the imaging device 10. However, this is merely one example, and the imaging device 10 may be a digital camera with a fixed lens, or may be a digital camera built into various electronic devices such as a smart device, a wearable terminal, a cell observation device, an ophthalmic observation device, or a surgical microscope.

[0031] The imaging device body 16 is provided with an image sensor 20. The image sensor 20 is an example of an "image sensor" according to the technology of the present disclosure. The image sensor 20 is a CMOS image sensor. The image sensor 20 captures an image of an imaging area including at least one subject. When an interchangeable lens 18 is attached to the imaging device body 16, subject light representing the subject passes through the interchangeable lens 18 and is focused on the image sensor 20, and image data representing the image of the subject is generated by the image sensor 20.

[0032] In this embodiment, a CMOS image sensor is exemplified as the image sensor 20, but the technology of the present disclosure is not limited to this, and the technology of the present disclosure is also applicable even if the image sensor 20 is another type of image sensor, such as a CCD image sensor.

[0033] A release button 22 and a dial 24 are provided on the top surface of the imaging device body 16. The dial 24 is operated when setting the operation mode of the imaging system and the operation mode of the playback system, and by operating the dial 24, the imaging device 10 is selectively set to one of the imaging mode, playback mode, and setting mode as its operation mode. The imaging mode is an operation mode that causes the imaging device 10 to capture images. The playback mode is an operation mode that plays back images (e.g., still images and / or moving images) obtained by capturing images for recording in the imaging mode. The setting mode is an operation mode that is set for the imaging device 10 when, for example, setting various setting values ​​used in control related to imaging.

[0034] The release button 22 functions as an imaging preparation instruction unit and an imaging instruction unit, and is capable of detecting two stages of pressing operation: an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state in which the button is pressed from a standby position to an intermediate position (half-pressed position), for example, and the imaging instruction state refers to a state in which the button is pressed beyond the intermediate position to a final pressed position (fully-pressed position). Note that, hereinafter, the "state in which the button is pressed from the standby position to the half-pressed position" will be referred to as the "half-pressed state," and the "state in which the button is pressed from the standby position to the fully-pressed position" will be referred to as the "fully-pressed state." Depending on the configuration of the imaging device 10, the imaging preparation instruction state may be a state in which the user's finger is in contact with the release button 22, and the imaging instruction state may be a state in which the operating user's finger has moved from a state in which the button is in contact with the release button 22 to a state in which the finger is released.

[0035] On the rear surface of the imaging device main body 16, command keys 26 and a touch panel display 32 are provided.

[0036] The touch panel display 32 includes a display 28 and a touch panel 30 (see also FIG. 2). An example of the display 28 is an EL display (e.g., an organic EL display or an inorganic EL display). The display 28 may be a different type of display, such as a liquid crystal display, instead of an EL display.

[0037] The display 28 displays images and / or text information, etc. When the imaging device 10 is in imaging mode, the display 28 is used to capture images for live view images, i.e., to display live view images obtained by continuous imaging. Here, a "live view image" refers to a moving image for display based on image data obtained by imaging by the image sensor 20. The imaging performed to obtain a live view image (hereinafter also referred to as "image capture for live view images") is performed at a frame rate of, for example, 60 fps. 60 fps is merely an example, and the frame rate may be less than 60 fps or may be greater than 60 fps.

[0038] The display 28 is also used to display a still image obtained by capturing a still image when an instruction to capture a still image is given to the imaging device 10 via the release button 22. The display 28 is also used to display a playback image when the imaging device 10 is in playback mode. Furthermore, the display 28 is also used to display a menu screen on which various menus can be selected when the imaging device 10 is in setting mode, and a setting screen for setting various setting values ​​used in imaging-related controls.

[0039] The touch panel 30 is a transmissive touch panel that is overlaid on the surface of the display area of ​​the display 28. The touch panel 30 receives instructions from the user by detecting contact with a pointing object such as a finger or a stylus pen. For ease of explanation, the "full press state" described above will hereinafter also include a state in which the user presses the soft key for starting imaging via the touch panel 30.

[0040] In this embodiment, an out-cell type touch panel display in which the touch panel 30 is overlaid on the surface of the display area of ​​the display 28 is given as an example of the touch panel display 32, but this is merely an example. For example, an on-cell type or an in-cell type touch panel display can also be used as the touch panel display 32.

[0041] The instruction keys 26 accept various instructions. Here, "various instructions" refers to, for example, an instruction to display a menu screen, an instruction to select one or more menus, an instruction to confirm a selection, an instruction to erase a selection, an instruction to zoom in, zoom out, and frame-by-frame advance. These instructions may also be given via the touch panel 30.

[0042] As an example, as shown in FIG. 2, the image sensor 20 includes a photoelectric conversion element 72. The photoelectric conversion element 72 has a light-receiving surface 72A. The photoelectric conversion element 72 is disposed within the imaging device body 16 so that the center of the light-receiving surface 72A coincides with the optical axis OA (see also FIG. 1). The photoelectric conversion element 72 has a plurality of photosensitive pixels 72B (see FIG. 3) arranged in a matrix, and the light-receiving surface 72A is formed by the plurality of photosensitive pixels. Each photosensitive pixel 72B has a microlens 72C (see FIG. 3). Each photosensitive pixel 72B is a physical pixel having a photodiode (not shown), which photoelectrically converts received light and outputs an electrical signal according to the amount of received light.

[0043] In addition, the multiple photosensitive pixels 72B have red (R), green (G), or blue (B) color filters (not shown) arranged in a matrix in a predetermined pattern arrangement (e.g., Bayer arrangement, G-stripe R / G complete checkerboard, X-Trans (registered trademark) arrangement, honeycomb arrangement, etc.).

[0044] For ease of explanation, hereinafter, a photosensitive pixel 72B having a microlens 72C and an R color filter will be referred to as an R pixel, a photosensitive pixel 72B having a microlens 72C and a G color filter will be referred to as a G pixel, and a photosensitive pixel 72B having a microlens 72C and a B color filter will be referred to as a B pixel. Also, for ease of explanation, hereinafter, an electrical signal output from an R pixel will be referred to as an "R signal," an electrical signal output from a G pixel will be referred to as a "G signal," and an electrical signal output from a B pixel will be referred to as a "B signal."

[0045] The interchangeable lens 18 includes an imaging lens 40. The imaging lens 40 has an objective lens 40A, a focus lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, the focus lens 40B, the zoom lens 40C, and the aperture 40D are arranged in this order along the optical axis OA from the subject side (object side) to the imaging device main body 16 side (image side).

[0046] The interchangeable lens 18 also includes a control device 36, a first actuator 37, a second actuator 38, and a third actuator 39. The control device 36 controls the entire interchangeable lens 18 in accordance with instructions from the imaging device main body 16. The control device 36 is a device having a computer including, for example, a CPU, an NVM, and RAM. The NVM of the control device 36 is, for example, an EEPROM. However, this is merely one example, and instead of or together with the EEPROM, an HDD and / or an SSD may be used as the NVM of the system controller 44. The RAM of the control device 36 temporarily stores various types of information and is used as a work memory. In the control device 36, the CPU reads necessary programs from the NVM and executes the read programs on the RAM to control the entire imaging lens 40.

[0047] Although a device having a computer is given here as an example of the control device 36, this is merely an example, and devices including ASIC, FPGA, and / or PLD may also be applied. Furthermore, the control device 36 may be, for example, a device realized by a combination of hardware and software configurations.

[0048] The first actuator 37 includes a focusing slide mechanism (not shown) and a focusing motor (not shown). The focusing slide mechanism has a focus lens 40B attached thereto so as to be slidable along the optical axis OA. The focusing motor is also connected to the focusing slide mechanism, and the focusing slide mechanism operates by receiving power from the focusing motor to move the focus lens 40B along the optical axis OA.

[0049] The second actuator 38 includes a zoom slide mechanism (not shown) and a zoom motor (not shown). The zoom lens 40C is attached to the zoom slide mechanism so that it can slide along the optical axis OA. The zoom motor is also connected to the zoom slide mechanism, and the zoom slide mechanism operates by receiving power from the zoom motor to move the zoom lens 40C along the optical axis OA.

[0050] The third actuator 39 includes a power transmission mechanism (not shown) and an aperture motor (not shown). The aperture 40D has an opening 40D1, and the size of the opening 40D1 is variable. The opening 40D1 is formed, for example, by a plurality of aperture blades 40D2. The plurality of aperture blades 40D2 are connected to the power transmission mechanism. The power transmission mechanism is also connected to an aperture motor, and the power transmission mechanism transmits the power of the aperture motor to the plurality of aperture blades 40D2. The plurality of aperture blades 40D2 operate upon receiving power transmitted from the power transmission mechanism, thereby changing the size of the aperture 40D1. The aperture 40D adjusts exposure by changing the size of the opening 40D1.

[0051] The focus motor, zoom motor, and aperture motor are connected to a control device 36, which controls the driving of each of the focus motor, zoom motor, and aperture motor. In this embodiment, a stepping motor is used as an example of the focus motor, zoom motor, and aperture motor. Therefore, the focus motor, zoom motor, and aperture motor operate in synchronization with pulse signals in response to commands from the control device 36. While an example is shown here in which the focus motor, zoom motor, and aperture motor are provided in the interchangeable lens 18, this is merely an example, and at least one of the focus motor, zoom motor, and aperture motor may be provided in the imaging device body 16. The components and / or operation method of the interchangeable lens 18 can be changed as needed.

[0052] In the imaging mode, the imaging device 10 selectively sets MF mode and AF mode in accordance with instructions given to the imaging device body 16. The MF mode is an operating mode in which the focus is adjusted manually. In the MF mode, for example, when the user operates the focus ring 18A or the like, the focus lens 40B moves along the optical axis OA by an amount corresponding to the amount of operation of the focus ring 18A or the like, thereby adjusting the focus.

[0053] In AF mode, imaging device body 16 calculates the in-focus position according to the subject distance, and adjusts the focus by moving focus lens 40B toward the calculated in-focus position. Here, the in-focus position refers to the position on optical axis OA of focus lens 40B when the subject is in focus.

[0054] The imaging device main body 16 includes an image sensor 20, a processor 12, a system controller 44, an image memory 46, a UI device 48, an external I / F 50, a communication I / F 52, a photoelectric conversion element driver 54, and an input / output interface 70. The image sensor 20 also includes a photoelectric conversion element 72 and an A / D converter 74.

[0055] The input / output interface 70 is connected to the processor 12, image memory 46, UI device 48, external I / F 50, photoelectric conversion element driver 54, and A / D converter 74. The input / output interface 70 is also connected to the control device 36 of the interchangeable lens 18.

[0056] The system controller 44 includes a CPU (not shown), an NVM (not shown), and a RAM (not shown). In the system controller 44, the NVM is a non-transitory storage medium that stores various parameters and programs. The NVM of the system controller 44 is, for example, an EEPROM. However, this is merely an example, and instead of or in addition to the EEPROM, a HDD and / or an SSD may be used as the NVM of the system controller 44. The RAM of the system controller 44 temporarily stores various information and is used as a work memory. In the system controller 44, the CPU reads necessary programs from the NVM and executes the read programs on the RAM to control the entire imaging device 10. That is, in the example shown in FIG. 2, the processor 12, the image memory 46, the UI device 48, the external I / F 50, the communication I / F 52, the photoelectric conversion element driver 54, and the control device 36 are controlled by the system controller 44.

[0057] The processor 12 operates under the control of the system controller 44. The processor 12 includes a CPU 62, an NVM 64, and a RAM 66.

[0058] The CPU 62, NVM 64, and RAM 66 are connected via a bus 68, which is connected to an input / output interface 70. Although the example shown in Fig. 2 shows a single bus as the bus 68 for convenience of illustration, multiple buses may be used. The bus 68 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0059] The NVM 64 is a non-transitory storage medium that stores various parameters and programs different from those stored in the NVM of the system controller 44. The various programs include a program 65 (see FIG. 4), which will be described later. The NVM 64 is, for example, an EEPROM. However, this is merely an example, and instead of or together with the EEPROM, an HDD and / or an SSD may be used as the NVM 64. The RAM 66 temporarily stores various information and is used as a work memory.

[0060] The CPU 62 reads out a necessary program from the NVM 64 and executes the read program in the RAM 66. The CPU 62 performs image processing in accordance with the program executed on the RAM 66.

[0061] The photoelectric conversion element 72 is connected to a photoelectric conversion element driver 54. The photoelectric conversion element driver 54 supplies an imaging timing signal that defines the timing of imaging performed by the photoelectric conversion element 72 to the photoelectric conversion element 72 in accordance with an instruction from the CPU 62. The photoelectric conversion element 72 performs resetting, exposure, and output of an electrical signal in accordance with the imaging timing signal supplied from the photoelectric conversion element driver 54. Examples of imaging timing signals include a vertical synchronization signal and a horizontal synchronization signal.

[0062] When the interchangeable lens 18 is attached to the imaging device body 16, subject light incident on the imaging lens 40 is imaged on the light-receiving surface 72A by the imaging lens 40. Under the control of the photoelectric conversion element driver 54, the photoelectric conversion element 72 photoelectrically converts the subject light received by the light-receiving surface 72A and outputs an electrical signal corresponding to the amount of subject light to the A / D converter 74 as imaging data 73 indicating the subject light. Specifically, the A / D converter 74 reads out the imaging data 73 from the photoelectric conversion element 72 in units of one frame and for each horizontal line using an exposure sequential readout method.

[0063] The A / D converter 74 digitizes the analog imaging data 73 read out from the photoelectric conversion element 72. The imaging data 73 digitized by the A / D converter 74 is so-called RAW image data, and represents an image in which R pixels, G pixels, and B pixels are arranged in a mosaic pattern. In this embodiment, as an example, the number of bits for each of the R pixels, B pixels, and G pixels included in the RAW image data, i.e., the bit length, is 14 bits.

[0064] In this embodiment, as an example, the CPU 62 of the processor 12 acquires imaging data 73 from the A / D converter 74 and performs image processing on the acquired imaging data 73. In this embodiment, the processor 12 generates moving image data 80 and distance image data 82 based on the imaging data 73. The image memory 46 stores the moving image data 80 and distance image data 82. In this embodiment, the moving image data 80 is moving image data used to display a live view image. The distance image data 82 is image data for displaying distances to multiple subjects within the imaging area.

[0065] The UI device 48 includes a display 28. The CPU 62 causes the display 28 to display moving image data 80 and depth image data 82 stored in the image memory 46. The CPU 62 also causes the display 28 to display various types of information.

[0066] The UI device 48 also includes a reception device 76. The reception device 76 includes a touch panel 30 and a hard key unit 78. The hard key unit 78 is a plurality of hard keys including the instruction keys 26 (see FIG. 1 ). The CPU 62 operates in accordance with various instructions received by the touch panel 30. Note that, although the hard key unit 78 is included in the UI device 48 here, the technology of the present disclosure is not limited to this; for example, the hard key unit 78 may be connected to the external I / F 50.

[0067] The external I / F 50 controls the exchange of various information with devices (hereinafter also referred to as "external devices") that exist outside the imaging device 10. An example of the external I / F 50 is a USB interface. To the USB interface, external devices (not shown) such as smart devices, personal computers, servers, USB memory, memory cards, and / or printers are directly or indirectly connected.

[0068] The communication I / F 52 is connected to a network (not shown). The communication I / F 52 controls the exchange of information between a communication device (not shown), such as a server on the network, and the system controller 44. For example, the communication I / F 52 transmits information in response to a request from the system controller 44 to the communication device via the network. The communication I / F 52 also receives information transmitted from the communication device and outputs the received information to the system controller 44 via the input / output interface 70.

[0069] 3, in this embodiment, photosensitive pixels 72B, each including a pair of independent photodiodes PD1 and PD2, are two-dimensionally arranged on a light receiving surface 72A of a photoelectric conversion element 72. In FIG. 3, one direction parallel to the light receiving surface 72A is defined as the X direction, and a direction perpendicular to the X direction is defined as the Y direction. The photosensitive pixels 72B are arranged along the X and Y directions.

[0070] The photodiode PD1 performs photoelectric conversion on a light beam passing through a first pupil partial region in the imaging lens 40. The photodiode PD2 performs photoelectric conversion on a light beam passing through a second pupil partial region in the imaging lens 40. A color filter (not shown) and a microlens 72C are arranged in each of the photosensitive pixels 72B.

[0071] The photoelectric conversion element 72 shown in FIG. 3 is an image plane phase difference type photoelectric conversion element in which a pair of photodiodes PD1 and PD2 is provided per pixel. In this embodiment, all photosensitive pixels 72B of the photoelectric conversion element 72 have the function of outputting data related to imaging and phase difference. During imaging, the photoelectric conversion element 72 combines the pair of photodiodes PD1 and PD2 into one pixel to output non-phase difference image data 73A. In AF mode, the photoelectric conversion element 72 detects signals from each of the pair of photodiodes PD1 and PD2 to output phase difference image data 73B.

[0072] That is, all of the photosensitive pixels 72B provided in the photoelectric conversion element 72 of this embodiment are so-called "phase difference pixels." Each photosensitive pixel 72B can selectively output non-phase difference image data 73A that has been photoelectrically converted by the entire region of the pixel, and phase difference image data that has been photoelectrically converted by a partial region of the pixel. Here, "the entire region of the pixel" refers to the combined light-receiving region of the photodiode PD1 and the photodiode PD2. Furthermore, "a partial region of the pixel" refers to the light-receiving region of the photodiode PD1 or the light-receiving region of the photodiode PD2.

[0073] The non-phase-contrast image data 73A can also be generated based on the phase-contrast image data 73B. For example, the non-phase-contrast image data 73A is generated by adding the phase-contrast image data 73B for each pair of pixel signals corresponding to the pair of photodiodes PD1 and PD2. The phase-contrast image data 73B may also include only data output from one of the pair of photodiodes PD1 and PD2. For example, if the phase-contrast image data 73B includes only data output from the photodiode PD1, it is possible to generate data output from the photodiode PD2 by subtracting the phase-contrast image data 73B from the non-phase-contrast image data 73A for each pixel.

[0074] That is, the imaging data 73 read out from the photoelectric conversion element 72 includes non-phase difference image data 73A and / or phase difference image data 73B. In this embodiment, in MF mode, distance image data is generated by measuring the distance to a subject present in the imaging area based on the phase difference image data 73B. In MF mode, the user can easily visually recognize the distance to the subject being imaged by referring to the distance image. As a result, the user can easily adjust the focus on the subject.

[0075] 4, a program 65 is stored in NVM 64 of imaging device 10. CPU 62 reads out program 65 from NVM 64 and executes the read program 65 on RAM 66. CPU 62 performs focus assist processing in accordance with program 65 executed on RAM 66. Distance information display processing is realized by CPU 62 operating as moving image data generation unit 62A, distance information acquisition unit 62B, distance image data generation unit 62C, and data output processing unit 62D in accordance with program 65.

[0076] As an example, as shown in FIG. 5, an image of a subject is captured using the imaging device 10. FIG. 5 is a bird's-eye view of an imaging area 10A captured by the imaging device 10 as seen from above. In the example shown in FIG. 5, a person 11A, a tree 11B, and a building 11C exist as subjects in the imaging area 10A. In FIG. 5, the direction parallel to the optical axis OA is defined as the Z direction. The Z direction is perpendicular to the X direction and the Y direction described above. Furthermore, the distance from the imaging device 10 in the Z direction is defined as D. In the example shown in FIG. 5, of the person 11A, the tree 11B, and the building 11C, the distance D to the person 11A is the shortest, and the distance D to the building 11C is the longest.

[0077] As an example, as shown in FIG. 6, the moving image data generator 62A generates moving image data 80 including a plurality of frames 80A based on non-phase difference image data 73A (see FIG. 4) obtained by the imaging operation of the image sensor 20. Each frame 80A of the moving image data 80 is two-dimensional data represented by a vertical axis and a horizontal axis. For example, the vertical axis is parallel to the Y direction, and the horizontal axis is parallel to the X direction. In the example shown in FIG. 6, each frame 80A includes a person 11A, a tree 11B, and a building 11C.

[0078] The distance information acquisition unit 62B acquires information about distance (hereinafter referred to as distance information) 84 (see FIG. 7) based on the phase difference image data 73B. That is, the distance information acquisition unit 62B acquires the distance information 84 based on the imaging data 73 when the photosensitive pixels 72B (see FIG. 3) serving as phase difference pixels output the phase difference image data 73B.

[0079] Specifically, the distance information acquisition unit 62B detects the phase difference (amount and direction of shift) between an image formed by the signal output from photodiode PD1 and an image formed by the signal output from photodiode PD2 based on the phase difference image data 73B, thereby acquiring distance information 84 at multiple positions within the imaging area 10A. In this embodiment, an image plane phase difference type photoelectric conversion element 72 in which a pair of photodiodes is provided per pixel is used, so that distance D can be acquired for positions corresponding to each of the photosensitive pixels 72B. The distance information 84 represents a two-dimensional distribution of distance D on the XY plane.

[0080] 7, distance image data generation unit 62C generates distance image data 82 based on distance information 84 acquired by distance information acquisition unit 62B. In distance image data 82, a first axis A1 corresponds to the vertical or horizontal axis of video data 80, and a second axis A2 represents distance information 84. In this embodiment, first axis A1 of distance image data 82 corresponds to the X axis (i.e., the horizontal axis of the angle of view), and second axis A2 represents distance D. In this embodiment, first axis A1 and second axis A2 are orthogonal to each other.

[0081] When the distance to a position (X1, Y1) in imaging area 10A is D1, distance image data generator 62C plots a point at a position (X1, D1) on a plane represented by first axis A1 and second axis A2. Distance image data generator 62C similarly plots points for all positions included in distance information 84, thereby generating distance image data 82.

[0082] In the example shown in Fig. 7, distance image data 82 includes a set G1 of points corresponding to person 11A, a set G2 of points corresponding to tree 11B, and a set G3 of points corresponding to building 11C. The distances to person 11A, tree 11B, and building 11C can be determined from the positional relationships of sets G1 to G3 along second axis A2. Note that although Fig. 6 shows the ground, for the sake of simplicity, distance image data 82 shown in Fig. 7 only reflects distance information for person 11A, tree 11B, and building 11C. The same applies to distance image data 82 shown in the other figures.

[0083] For example, each time the moving image data generation section 62A generates one frame 80A, the distance image data generation section 62C generates distance image data 82 using distance information 84 corresponding to the generated frame 80A.

[0084] 8, the data output processing unit 62D outputs the moving image data 80 and the distance image data 82 to the display 28 included in the touch panel display 32, with the distance image data 82 superimposed on a frame 80A of the moving image data 80. Specifically, the data output processing unit 62D superimposes the distance image data 82 in a rectangular window area 85 set within the frame 80A. The data output processing unit 62D may also combine the moving image data 80 and the distance image data 82 and output the combined image to the display 28.

[0085] Display 28 is an example of a "display destination" according to the technology of the present disclosure. Data output processing unit 62D is not limited to directly outputting moving image data 80 and depth image data 82 to a display destination, but may output the data indirectly to the display destination via a relay device or the like.

[0086] The moving image data 80 and the distance image data 82 are temporarily stored in the image memory 46 and then displayed on the display 28. The position and size of the window area 85 within the frame 80A may be changeable based on the user's operation of the receiving device 76. This type of display mode is called a picture-in-picture display (hereinafter referred to as PinP display). By observing the distance image data 82 displayed within the moving image data 80, the user can easily visually recognize the distance of the subject.

[0087] Next, the operation of the imaging device 10 will be described with reference to Fig. 9. Fig. 9 shows an example of the flow of focus assist processing executed by the CPU 62. The focus assist processing shown in Fig. 9 is executed, for example, while a live view image is being displayed in MF mode before an image capture instruction is given by the release button 22.

[0088] 9, first, in step ST100, the moving image data generator 62A determines whether or not the image sensor 20 (see FIG. 2) has generated the imaging data 73 (see FIG. 3). Here, the imaging data 73 includes non-phase difference image data 73A and phase difference image data 73B.

[0089] In step ST100, if the imaging data 73 has not been generated by the image sensor 20, the determination is negative, and the focus assist process proceeds to step ST105. In step ST100, if the imaging data 73 has been generated by the image sensor 20, the determination is positive, and the focus assist process proceeds to step ST101.

[0090] In step ST101, the moving image data generation unit 62A generates moving image data 80 (see FIG. 6) based on non-phase difference image data 73A included in the imaging data 73. After one frame 80A of the moving image data 80 is generated in step ST101, the focus assist process proceeds to step ST102.

[0091] In step ST102, the distance information acquisition unit 62B acquires distance information 84 (see FIG. 7) corresponding to the frame 80A generated in step ST101, based on the phase difference image data 73B included in the imaging data 73. After the processing of step ST102 is executed, the focus assist processing proceeds to step ST103.

[0092] In step ST103, distance image data generation unit 62C generates distance image data 82 (see FIG. 7) based on distance information 84 acquired in step ST102. After the process of step ST103 is executed, the focus assist process proceeds to step ST104.

[0093] In step ST104, data output processing unit 62D outputs moving image data 80 and distance image data 82 to display 28, with distance image data 82 superimposed on window area 85 of frame 80A (see FIG. 8). After the processing of step ST104 is executed, the focus assist processing proceeds to step ST105.

[0094] In step ST105, the CPU 62 determines whether or not a condition for terminating the focus assist process (hereinafter referred to as the "termination condition") has been satisfied. An example of the termination condition is that an image capture instruction has been detected using the release button 22 (see FIG. 1). If the termination condition has not been satisfied in step ST105, the determination is negative, and the focus assist process proceeds to step ST100. If the termination condition has been satisfied in step ST105, the determination is positive, and the focus assist process ends.

[0095] As described above, with imaging device 10, the user can easily visually recognize the distance to the subject by observing distance image data 82 displayed in moving image data 80. Furthermore, the user can manually adjust the focus according to the recognized distance to the subject and then issue an image capture instruction.

[0096] In the above embodiment, the first axis A1 of the distance image data 82 corresponds to the X axis of the video data 80 (i.e., the horizontal axis of the angle of view), and the second axis A2 represents the distance D. Alternatively, the first axis A1 of the distance image data 82 may correspond to the Y axis of the video data 80 (i.e., the vertical axis of the angle of view), and the second axis A2 may represent the distance D.

[0097] Furthermore, in the above embodiment, the distance D is acquired for the position corresponding to each of the photosensitive pixels 72B included in the photoelectric conversion element 72, but it is not necessarily necessary to acquire the distance D from the positions corresponding to all of the photosensitive pixels 72B. In other words, the photosensitive pixels 72B from which the distance D is acquired may be thinned out.

[0098] Furthermore, in the above embodiment, the photoelectric conversion element 72 is an image plane phase difference type photoelectric conversion element in which each pixel has a pair of photodiodes, and all photosensitive pixels 72B have the function of outputting data related to imaging and phase difference. However, this is not limited to the fact that all photosensitive pixels 72B have the function of outputting data related to imaging and phase difference. The photoelectric conversion element 72 may also include photosensitive pixels that do not have the function of outputting data related to imaging and phase difference. Furthermore, the photoelectric conversion element 72 is not limited to an image plane phase difference type photoelectric conversion element in which each pixel has a pair of photodiodes, but may include an imaging photosensitive pixel for acquiring non-phase difference image data 73A and a phase difference detection photosensitive pixel for acquiring phase difference image data 73B. In this case, a light-shielding member is provided for the phase difference pixel so that it receives light from either the first pupil partial region or the second pupil partial region.

[0099] Furthermore, in the above embodiment, the distance information 84 is obtained by the photoelectric conversion element 72 of the phase difference type, but the distance information 84 is not limited to the phase difference type, and may be obtained by using a photoelectric conversion element of the TOF type.

[0100] [First Modification] The imaging device 10 according to the first modification allows the user to easily check not only the distance to the subject but also whether the subject to be imaged is within the depth of field.

[0101] As an example, as shown in FIG. 10, in the first variant, the distance image data generation unit 62C acquires distance information 84 from the distance information acquisition unit 62B, and also acquires focus distance information 90 and depth of field information 91, for example, from the system controller 44.

[0102] The focal distance information 90 is information that indicates the position of the focus lens 40B (i.e., the focal distance). The depth of field information 91 is the rear depth of field L r and the forward depth of field L expressed by the following equation (2): f This is information that represents the following.

[0103]

number

number

[0104] Here, f is the focal length, F is the aperture value (i.e., F-number) of the aperture 40D, L is the focusing distance, and δ is the permissible circle of confusion diameter. The permissible circle of confusion diameter is approximately twice the arrangement pitch of the photosensitive pixels 72B, and allows for blurring of approximately one pixel in size. The permissible circle of confusion diameter may be set or changed by the user using the UI device 48. The focusing distance L is the distance from the light receiving surface 72A of the photoelectric conversion element 72 included in the image sensor 20 to the subject in a focused state.

[0105] For example, the depth of field information 91 includes a rear depth of field L r and front depth of field L f The depth of field information 91 may also include the values ​​of the focal length f, the aperture value F, the focusing distance L, and the permissible circle of confusion diameter δ. In this case, the distance image data generator 62C calculates the rear depth of field L based on the above formulas (1) and (2). r and front depth of field L f can be calculated.

[0106] 11, in this modification, distance image data generator 62C generates a linear first mark M1 indicating the focal distance L based on focal distance information 90, and generates linear second marks M2r and M2f indicating the depth of field based on depth of field information 91. Second mark M2r indicates the rear depth of field L r The second mark M2f indicates the front depth of field L f Distance image data generator 62C superimposes first mark M1 and second marks M2r and M2f on distance image data 82 generated based on distance information 84.

[0107] 12 as an example, in this modification, in the focus assist processing shown in the above embodiment, distance image data generation unit 62C generates distance image data 82 in step ST103, and then executes steps ST400 and ST401. In step ST400, distance image data generation unit 62C acquires focus distance information 90 and depth of field information 91 from system controller 44. In step ST401, distance image data generation unit 62C generates first mark M1 and second marks M2r, M2f based on focus distance information 90 and depth of field information 91, and superimposes the generated first mark M1 and second marks M2r, M2f on distance image data 82.

[0108] In this modified example, after step ST401, in step ST104, the data output processing unit 62D outputs moving image data 80 and distance image data 82 including the first mark M1 and the second marks M2r and M2f to the display 28 included in the touch panel display 32.

[0109] The other steps shown in FIG. 12 are the same as the steps shown in FIG. 9 described in the above embodiment, and therefore will not be described again.

[0110] According to this modification, the user can easily visually determine whether the subject to be imaged is within the depth of field. In the example shown in Fig. 11, the set G1 is between the second mark M2r and the second mark M2f and exists on the first mark M1. Therefore, the user can easily visually confirm that the person 11A (see Fig. 8) is within the depth of field and is in focus.

[0111] In this modified example, the distance image data 82 includes information representing the first mark M1 and the second marks M2r, M2f, but it is sufficient if it includes information representing at least one of the first mark M1 and the second marks M2r, M2f.

[0112] [Second Modification] The imaging device 10 according to the second variant allows the user to change the focus distance and / or depth of field by performing operations on the distance image data 82 displayed on the touch panel display 32.

[0113] 13 , in the second modified example, CPU 62 includes lens control unit 62E in addition to moving image data generation unit 62A, distance information acquisition unit 62B, distance image data generation unit 62C, and data output processing unit 62D. Similar to the first modified example, distance image data generation unit 62C in the second modified example generates first mark M1 and second marks M2r, M2f based on focus distance information 90 and depth of field information 91, and superimposes the generated first mark M1 and second marks M2r, M2f on distance image data 82.

[0114] Lens control unit 62E controls focus lens 40B and / or aperture 40D by issuing instructions to control device 36 of interchangeable lens 18 based on operation signals output from touch panel 30 included in touch panel display 32 (see FIG. 1). In this modification, the user can perform an operation to change the position of first mark M1 or second marks M2r, M2f along second axis A2 in distance image data 82 displayed on touch panel display 32.

[0115] As an example, as shown in FIG. 14, in the distance image data 82 displayed on the touch panel display 32, the user can change the position of the first mark M1 by touching the first mark M1 with the finger of the hand H and performing a drag operation along the second axis A2.

[0116] When the acceptance device 76 (see FIG. 2) accepts an operation to change the position of the first mark M1 along the second axis A2, the lens control unit 62E controls the movement of the focus lens 40B based on the operation signal output from the touch panel 30. Specifically, the lens control unit 62E controls the change of the position of the focus lens 40B so that the distance D indicated by the changed first mark M1 becomes the in-focus distance L.

[0117] The second marks M2r and M2f move in conjunction with the movement of the first mark M1. This is because the rear depth of field L r and front depth of field L f This is because it depends on the focal distance L indicated by the first mark M1 (see the above formulas (1) and (2)).

[0118] In the example shown in FIG. 14, the user drags the first mark M1 from a state in which the person 11A is in focus to a building 11C behind the person 11A.

[0119] 15, for example, the user can widen the space between the second marks M2r and M2f by performing a pinch-out operation while touching the second marks M2r and M2f with the fingers of hand H in distance image data 82 displayed on touch panel display 32. Widening the space between the second marks M2r and M2f changes the positions of each of the second marks M2r and M2f.

[0120] When the touch panel 30 receives an operation to change the positions of the second marks M2r and M2f along the second axis A2, the lens control unit 62E performs control to change the aperture value (F-number) of the diaphragm 40D based on the operation signal output from the touch panel 30. Specifically, the lens control unit 62E controls the rear depth of field L indicated by the second marks M2r and M2f. r and front depth of field L f The aperture value is controlled based on the above formulas (1) and (2) so that the following formula is obtained.

[0121] It should be noted that the user can change the distance between the second marks M2r and M2f not only by pinching out, but also by dragging one of the second marks M2r and M2f along the second axis A2.

[0122] Next, an example of lens control processing will be described with reference to Fig. 16. Fig. 16 shows an example of the flow of lens control processing executed by CPU 62. The lens control processing shown in Fig. 16 is executed, for example, while a live view image is being displayed in MF mode before an image capture instruction is issued by release button 22.

[0123] In the lens control process shown in FIG. 16, first, in step ST200, the lens control unit 62E determines whether or not the first mark M1 has been operated.

[0124] In step ST200, if the first mark M1 has not been operated, the determination is negative and the lens control process proceeds to step ST202. In step ST200, if the first mark M1 has been operated, the determination is positive and the lens control process proceeds to step ST201.

[0125] In step ST201, lens control unit 62E controls focus lens 40B so that distance D indicated by first mark M1 changed by operation becomes in-focus distance L. After focus lens 40B is controlled in step ST201, the lens control process proceeds to step ST202.

[0126] In step ST202, the lens control unit 62E determines whether or not the second marks M2r and M2f have been operated.

[0127] In step ST202, if the second marks M2r and M2f have not been operated, the determination is negative and the lens control process proceeds to step ST204. In step ST202, if the second marks M2r and M2f have been operated, the determination is positive and the lens control process proceeds to step ST203.

[0128] In step ST203, the lens control unit 62E changes the rear depth of field L indicated by the second marks M2r and M2f changed by the operation. r and front depth of field L f After the aperture 40D is controlled in step ST203, the lens control process proceeds to step ST204.

[0129] In step ST204, the CPU 62 determines whether or not a condition for terminating the lens control process (hereinafter referred to as the "termination condition") has been satisfied. An example of the termination condition is that an image capture instruction has been detected using the release button 22 (see FIG. 1). If the termination condition has not been satisfied in step ST204, the determination is negative, and the lens control process proceeds to step ST200. If the termination condition has been satisfied in step ST204, the determination is positive, and the lens control process ends.

[0130] As described above, in imaging device 10 according to this modification, first mark M1 or second marks M2r, M2f can be changed by performing an operation on distance image data 82. This allows the user to intuitively manually adjust the focus and / or aperture value.

[0131] In this modified example, the position of the first mark M1 or the second marks M2r, M2f can be changed by operating the touch panel 30, but it is not limited to the touch panel 30 and may also be changed by an operating member such as the hard key section 78.

[0132] [Third Modification] The imaging device 10 according to the third modified example differs from the imaging device 10 according to the second modified example in that the changeable range of the first mark M1 or the second marks M2r, M2f is limited based on the distance to a specific subject.

[0133] 17, in the third modified example, the CPU 62 includes a moving image data generator 62A, a distance information acquirer 62B, a distance image data generator 62C, a data output processor 62D, and a lens controller 62E, as well as a subject extractor 62F. The subject extractor 62F extracts a main subject from each frame 80A of the moving image data 80. For example, the subject extractor 62F extracts a person 11A as the main subject from each frame 80A using a known technique such as face detection or subject detection. The main subject is an example of a "target subject" according to the technology of the present disclosure.

[0134] In this modified example, lens control unit 62E controls focus lens 40B and / or aperture 40D so that the main subject falls within the depth of field (i.e., between second marks M2r and M2f). As an example, FIG. 18 shows a situation in which, when the main subject is included within the depth of field, the user attempts to narrow the depth of field below the distance range of the main subject by dragging second mark M2r. In this case, lens control unit 62E controls aperture 40D to reduce its F-number, but limits the change in the F-number (i.e., limits the change in the depth of field) so that the main subject falls within the depth of field.

[0135] In addition, when the position of the second marks M2r, M2f is changed in conjunction with the operation of the first mark M1, the lens control unit 62E similarly limits the change in the focus distance so that the main subject falls within the depth of field.

[0136] In this modified example, changes to the depth of field and focus distance are limited so that the main subject falls within the depth of field, allowing the user to intuitively place the main subject appropriately within the depth of field.

[0137] Furthermore, in this modification, lens control unit 62E extracts the main subject based on moving image data 80, and therefore, even if the distance to the main subject changes, focus lens 40B and / or aperture 40D are controlled so that the main subject falls within the depth of field. Therefore, in this modification, the depth of field can be made to track the main subject.

[0138] Furthermore, in this modification, subject extraction unit 62F extracts the main subject based on moving image data 80, but a subject specified by the user using touch panel 30 or the like may be extracted as the main subject. For example, when an operation to specify an area is performed in moving image data 80 displayed on touch panel display 32, subject extraction unit 62F extracts a specified area specified in moving image data 80 as the main subject. In this case, lens control unit 62E controls focus lens 40B and / or aperture 40D so that the main subject specified by the user falls within the depth of field.

[0139] [Fourth Modification] Imaging device 10 according to the fourth modification allows the user to enlarge a portion of distance image data 82 by performing an operation on distance image data 82 displayed on touch panel display 32. In this modification, distance image data generator 62C enlarges a portion of distance image data 82 in the direction of second axis A2 based on an operation signal output from touch panel 30 or the like.

[0140] 19, when the user performs a pinch-out operation to increase the distance between second marks M2r and M2f, distance image data generator 62C performs a process of enlarging the area including second marks M2r and M2f in the direction of second axis A2. More specifically, when the operation speed of the pinch-out operation remains below a certain speed for a certain period of time, distance image data generator 62C performs a process of enlarging the area including second marks M2r and M2f in the direction of second axis A2.

[0141] As an example, as shown in FIG. 20, when the user drags the first mark M1 and the operation speed of the drag operation remains below a certain speed for a certain period of time, the distance image data generation unit 62C performs a process of expanding the area including the second marks M2r and M2f centered on the first mark M1 in the direction of the second axis A2.

[0142] Next, an example of the enlargement process will be described with reference to Fig. 21. Fig. 21 shows an example of the flow of the enlargement process executed by the CPU 62. The enlargement process shown in Fig. 21 is executed, for example, while a live view image is being displayed in MF mode before an image capture instruction is given by the release button 22.

[0143] In the lens control process shown in FIG. 21, first, in step ST300, distance image data generation unit 62C determines whether first mark M1 or second marks M2r, M2f has been operated.

[0144] In step ST300, if the first mark M1 and the second marks M2r, M2f have not been operated, the determination is negative and the enlargement process proceeds to step ST304. In step ST300, if the first mark M1 or the second marks M2r, M2f have been operated, the determination is positive and the enlargement process proceeds to step ST301.

[0145] In step ST301, distance image data generation unit 62C determines whether the operation speed is equal to or less than a certain speed. If the operation speed is not equal to or less than a certain speed in step ST301, the determination is negative, and the enlargement process executes step ST301 again. If the operation speed is equal to or less than a certain speed in step ST301, the determination is positive, and the enlargement process proceeds to step ST302.

[0146] In step ST302, distance image data generation unit 62C determines whether the state in which the operation speed is equal to or less than a certain speed has continued for a certain period of time. If the state in which the operation speed is equal to or less than a certain speed has not continued for a certain period of time in step ST302, the determination is negative, and the enlargement processing proceeds to step ST304. If the state in which the operation speed is equal to or less than a certain speed has continued for a certain period of time in step ST302, the determination is positive, and the enlargement processing proceeds to step ST303.

[0147] In step ST303, distance image data generation unit 62C enlarges, in the direction of second axis A2, the area including second marks M2r and M2f in distance image data 82. After the process of enlarging, in the direction of second axis A2, the area including second marks M2r and M2f in step ST303, the enlargement process proceeds to step ST304.

[0148] In step ST304, the CPU 62 determines whether or not a condition for terminating the enlargement processing (hereinafter referred to as the "termination condition") has been satisfied. An example of the termination condition is that an image capture instruction has been detected using the release button 22 (see FIG. 1). If the termination condition has not been satisfied in step ST304, the determination is negative, and the enlargement processing proceeds to step ST300. If the termination condition has been satisfied in step ST304, the determination is positive, and the enlargement processing ends.

[0149] In this modified example, the user can partially enlarge the distance image data 82 in response to an operation, and can easily adjust and check the depth of field for the subject based on the enlarged image.

[0150] [Fifth Modification] The imaging device 10 of the fifth variant differs from the imaging device 10 of the second variant in that when a user performs an operation to specify an area on the distance image data 82 displayed on the touch panel display 32, the color of the corresponding area in the moving image data 80 is changed.

[0151] As an example, as shown in FIG. 22, the moving image data generation unit 62A of this embodiment identifies a group of pixels in moving image data 80 that corresponds to an area specified in distance image data 82, and changes the color of the identified group of pixels.

[0152] 22, the user specifies an area including set G1 from distance image data 82 by encircling set G1 with the fingers of hand H. In this example, moving image data generator 62A changes the color of the pixel group corresponding to set G1 in moving image data 80 (i.e., the pixel group corresponding to person 11A) to a specific color. Note that changing the color also includes changing the density.

[0153] According to this modification, the user can easily confirm the relationship between the area specified in the distance image data 82 and the subject in the video data 80.

[0154] In this modification, lens control unit 62E may control focus lens 40B and / or aperture 40D so that the area specified in distance image data 82 falls within the depth of field. As a result, first mark M1 or second marks M2r, M2f move according to the specified area. This allows the user to intuitively manually adjust the focus and / or aperture value.

[0155] [Sixth Modification] The imaging device 10 of the sixth variant differs from the imaging device 10 of the second variant in that when a user performs an operation to specify an area in the moving image data 80 displayed on the touch panel display 32, the color of the corresponding area in the distance image data 82 is changed.

[0156] As an example, as shown in FIG. 23, the distance image data generation unit 62C of this embodiment identifies a group of pixels in the distance image data 82 that corresponds to an area specified in the video image data 80, and changes the color of the identified group of pixels.

[0157] 23, the user specifies an area including tree 11B in video data 80 by encircling tree 11B with the fingers of hand H. In this example, distance image data generator 62C changes the color of the pixel group corresponding to tree 11B in distance image data 82 (i.e., the pixel group of set G2) to a specific color. Note that changing the color also includes changing the density.

[0158] According to this modification, the user can easily check the distance of the area specified in the moving image data 80 in the distance image data 82.

[0159] In this modification, the lens control unit 62E may control the focus lens 40B and / or the aperture 40D so that a specified area in the moving image data 80 falls within the depth of field. As a result, the first mark M1 or the second marks M2r, M2f move according to the specified area. This allows the user to intuitively manually adjust the focus and / or aperture value.

[0160] [Seventh Modification] The imaging device 10 of the seventh variant differs from the imaging device 10 of the second variant in that when a user performs an operation to specify an area in the moving image data 80 displayed on the touch panel display 32, distance information is displayed only for the corresponding area in the distance image data 82.

[0161] As an example, as shown in FIG. 24, a distance image data generating section 62C of this embodiment generates distance image data 82 based only on distance information 84 corresponding to a specified area in moving image data 80.

[0162] 24, the user specifies an area in moving image data 80 by drawing a circle around the face of person 11A with the fingers of hand H. In this example, distance image data generator 62C generates distance image data 82 based only on distance information 84 corresponding to the face of person 11A. As a result, distance image data 82 displays only a part of set G1A of set G1 corresponding to person 11A.

[0163] According to this modification, the user can check distance information of the subject only for the area specified in video data 80. Furthermore, distance image data generator 62C generates distance image data 82 based only on distance information 84 corresponding to the specified area, which speeds up the process of generating distance image data 82.

[0164] In this modification, the lens control unit 62E may control the focus lens 40B and / or the aperture 40D so that a specified area in the moving image data 80 falls within the depth of field. As a result, the first mark M1 or the second marks M2r, M2f move according to the specified area. This allows the user to intuitively manually adjust the focus and / or aperture value.

[0165] Furthermore, distance image data generation section 62C may generate distance image data 82 based only on distance information 84 corresponding to the subject region extracted by subject extraction section 62F shown in the third modified example. In this case, the position and / or size of the subject region extracted by subject extraction section 62F changes depending on the movement of the subject or imaging device 10. Because distance image data generation section 62C repeatedly generates distance image data 82 at regular intervals, distance image data 82 changes in response to changes in the position and / or size of the subject region.

[0166] [Eighth Modification] In the above embodiment and each of the above modifications, the distance image data 82 is displayed superimposed on the moving image data 80 (i.e., PinP display) (see, for example, FIG. 8). The display mode of the distance image data 82 is not limited to PinP display. For example, the distance image data 82 and the moving image data 80 may be displayed side by side.

[0167] 25 , in tethered shooting performed by connecting imaging device 10 to PC 100 as an external device, moving image data 80 may be displayed on display 110 of PC 100, and distance image data 82 may be displayed on display 28 of imaging device 10. Imaging device 10 and PC 100 are connected via a wired or wireless connection. Moving image data 80 generated by imaging device 10 is transferred to PC 100, and a live view image is displayed on display 110 of PC 100.

[0168] Furthermore, the moving image data 80 according to the technique of the present disclosure is not limited to moving image data for live view images, but may be moving image data for recording to an internal memory or an external memory in response to operation of the release button 22. As explained in the fifth modified example, when changing the color of part of the moving image data 80, it is preferable not to change the color of the moving image data for recording.

[0169] The above-described embodiment and each of the modifications can be combined with each other as long as no contradiction occurs.

[0170] Furthermore, in the above embodiment, CPU 62 is exemplified, but instead of CPU 62 or together with CPU 62, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0171] In the above embodiment, an example in which the program 65 is stored in the NVM 64 has been described, but the technology of the present disclosure is not limited to this. For example, the program 65 may be stored in a portable non-transitory storage medium such as an SSD or a USB memory. The program 65 stored in the non-transitory storage medium is installed in the processor 12 of the imaging device 10. The CPU 62 executes focus assist processing in accordance with the program 65.

[0172] Alternatively, the program 65 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the program 65 may be downloaded and installed in the processor 12 in response to a request from the imaging device 10.

[0173] It is not necessary to store the entire program 65 in a storage device such as another computer or server device connected to the imaging device 10, or in the NVM 64; only a part of the program 65 may be stored therein.

[0174] Furthermore, although the imaging device 10 shown in FIGS. 1 and 2 has a built-in processor 12, the technology of the present disclosure is not limited to this. For example, the processor 12 may be provided outside the imaging device 10.

[0175] In the above embodiment, the processor 12 includes a CPU 62, an NVM 64, and a RAM 66, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the processor 12. Furthermore, a combination of a hardware configuration and a software configuration may be used instead of the processor 12.

[0176] The hardware resources for executing the focus assist processing described in the above embodiments can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing the focus assist processing by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations designed specifically for executing specific processing. Each processor has a built-in or connected memory, and each processor uses the memory to execute the focus assist processing.

[0177] The hardware resource that executes the focus assist process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the focus assist process may be a single processor.

[0178] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the focus assist process. Second, there is a configuration in which a processor is used that realizes the functions of the entire system, including multiple hardware resources that execute the focus assist process, on a single IC chip, as typified by SoCs. In this way, the focus assist process is realized using one or more of the above-mentioned various processors as hardware resources.

[0179] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor elements. The focus assist process described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the process.

[0180] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0181] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0182] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An imaging device comprising an image sensor and a processor, The processor: acquiring information about distances at a plurality of positions within an imaging area of ​​the image sensor; generating moving image data represented by a vertical axis and a horizontal axis based on the imaging data obtained by the image sensor; generating distance image data in which a first axis corresponds to the vertical axis or the horizontal axis and a second axis represents information related to the distance, the distance image data including information representing a second mark indicating a depth of field; outputting the moving image data and the distance image data; Accepting an operation to change the position of the second mark along the second axis; When an operation for widening the depth of field is performed using the second mark, a process for enlarging the distance image data in the second axis direction is performed. Imaging device.

2. The processor, generating the distance image data including information representing a first mark indicating a focusing distance in addition to the second mark; Accepting an operation to change the position of the first mark or the second mark along the second axis; The imaging device according to claim 1 .

3. An imaging device comprising an image sensor and a processor, The processor: acquiring information about distances at a plurality of positions within an imaging area of ​​the image sensor; generating moving image data represented by a vertical axis and a horizontal axis based on the imaging data obtained by the image sensor; generating distance image data in which a first axis corresponds to the vertical axis or the horizontal axis and a second axis represents information related to the distance, the distance image data including information representing at least one of a first mark indicating a focus distance and a second mark indicating a depth of field; outputting the moving image data and the distance image data; accepting an operation to change the position of the first mark or the second mark along the second axis; performing a process of enlarging the distance image data in the second axis direction when a state in which the operation speed is equal to or lower than a certain speed continues for a certain period of time during an operation to change the position of the first mark or the second mark; Imaging device.

4. Equipped with a focus lens, The processor: When an operation to change the position of the first mark along the second axis is received, movement control of the focus lens is performed to a position corresponding to the in-focus distance indicated by the first mark. The imaging device according to claim 2 or 3.

5. Equipped with aperture, The processor: when an operation to change the position of the second mark along the second axis is accepted, control is performed to change the aperture value of the aperture to an aperture value corresponding to the depth of field represented by the second mark. The imaging device according to claim 1 .

6. The processor outputs the moving image data and the depth image data to a display destination. The imaging device according to claim 1 .

7. The processor: When an operation for designating an area in the distance image data is performed, a color of a pixel group in the moving image data corresponding to the area designated in the distance image data is changed. The imaging device according to claim 1 .

8. The processor: When an operation for designating an area in the moving image data is performed, a color of a pixel group in the distance image data corresponding to the area designated in the moving image data is changed. The imaging device according to claim 1 .

9. The processor: extracting a subject area based on the video image data; obtaining information about the distance corresponding to the extracted subject region; generating the distance image data based on the acquired information about the distance; The imaging device according to claim 1 .

10. The processor: extracting a designated area designated in the video data when an operation for designating an area in the video data is performed; acquiring information about the distance corresponding to the extracted designated area; generating the distance image data based on the acquired information about the distance; The imaging device according to claim 1 .

11. The processor: acquiring information about the distance based on imaging data obtained by the image sensor; The imaging device according to any one of claims 1 to 10.

12. the image sensor includes a plurality of phase difference pixels; The processor: acquiring information about the distance based on the imaging data obtained from the phase difference pixels among the imaging data; The imaging device according to claim 11.

13. the phase difference pixel is capable of selectively outputting non-phase difference image data in which photoelectric conversion is performed by an entire region of the pixel and phase difference image data in which photoelectric conversion is performed by a partial region of the pixel, The processor: acquiring information about the distance based on imaging data when the phase difference pixels output the phase difference image data; The imaging device according to claim 12.

14. obtaining information about distances at a plurality of positions within an imaging area of ​​the image sensor; generating moving image data represented by a vertical axis and a horizontal axis based on the imaging data obtained by the image sensor; generating distance image data in which a first axis corresponds to the vertical axis or the horizontal axis and a second axis represents information related to the distance, the distance image data including information representing a second mark indicating a depth of field; outputting the moving image data and the distance image data; accepting an operation to change the position of the second mark along the second axis; performing a process of enlarging the distance image data in the second axis direction when an operation of widening the depth of field is performed using the second mark; An information processing method including:

15. Obtaining information about distances at a plurality of positions within an imaging area of ​​an image sensor; generating moving image data represented by a vertical axis and a horizontal axis based on the imaging data obtained by the image sensor; generating distance image data in which a first axis corresponds to the vertical axis or the horizontal axis and a second axis represents information related to the distance, the distance image data including information representing at least one of a first mark indicating a focus distance and a second mark indicating a depth of field; outputting the moving image data and the distance image data; accepting an operation to change the position of the first mark or the second mark along the second axis; performing a process of enlarging the distance image data in the second axis direction when a state in which the operation speed is equal to or lower than a certain speed continues for a certain period of time during an operation to change the position of the first mark or the second mark; An information processing method including:

16. obtaining information about distances at a plurality of positions within an imaging area of ​​the image sensor; generating moving image data represented by a vertical axis and a horizontal axis based on the imaging data obtained by the image sensor; generating distance image data in which a first axis corresponds to the vertical axis or the horizontal axis and a second axis represents information related to the distance, the distance image data including information representing a second mark indicating a depth of field; outputting the moving image data and the distance image data; accepting an operation to change the position of the second mark along the second axis; performing a process of enlarging the distance image data in the second axis direction when an operation of widening the depth of field is performed using the second mark; A program for causing a computer to execute a process including the above.

17. Obtaining information about distances at a plurality of positions within an imaging area of ​​an image sensor; generating moving image data represented by a vertical axis and a horizontal axis based on the imaging data obtained by the image sensor; generating distance image data in which a first axis corresponds to the vertical axis or the horizontal axis and a second axis represents information related to the distance, the distance image data including information representing at least one of a first mark indicating a focus distance and a second mark indicating a depth of field; outputting the moving image data and the distance image data; accepting an operation to change the position of the first mark or the second mark along the second axis; performing a process of enlarging the distance image data in the second axis direction when a state in which the operation speed is equal to or lower than a certain speed continues for a certain period of time during an operation to change the position of the first mark or the second mark; A program for causing a computer to execute a process including the above.

Images