Image processing device, image processing method, and program

Abstract

This image processing device comprises a processor, wherein the processor performs a first process for: acquiring distance data pertaining to the distance between an image sensor and a subject; outputting first image data that represents first images obtained by imaging with the image sensor; outputting first brightness data created on the basis of first signals of the first image data with regard to at least a first region among a plurality of regions in which the first images are classified in accordance with the distance data; and causing the content of a first instruction to be reflected in the first image data and / or the first brightness data when the first instruction pertaining to the first brightness data is received by a receiving device.

Description

[Technical field]

[0001] The technology disclosed herein relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Japanese Patent Application Laid-Open No. 2013-135308 discloses an imaging device having an image sensor that generates an image signal, a distance calculation unit that calculates the distance to a subject from the image signal, a reliability calculation unit that calculates the reliability of a result of the calculation by the distance calculation unit, a histogram generation unit that generates multiple histograms according to the distance to the subject based on the result of the reliability calculation and the result of the calculation of the distance to the subject, and a display unit that displays the multiple histograms.

[0003] Japanese Patent Application Laid-Open No. 2013-201701 discloses an photographing device that includes an imaging unit having an image sensor that images a subject, a display unit that displays an image based on image data acquired by the imaging unit, a touch panel arranged on a display surface of the display unit, a light source detection unit that detects a light source in the image being displayed on the display unit, an operation input determination unit that detects and determines coordinate data corresponding to an input operation on the touch panel, a control unit that sets a designated area in the image based on the coordinate data detected by the operation input determination unit, a luminance distribution determination unit that determines the luminance distribution of the area corresponding to the coordinate data determined by the operation input determination unit, a luminance distribution emphasis unit that emphasizes the luminance of a predetermined area in the image, and a recording unit that records image data, where the luminance distribution emphasis unit emphasizes the luminance of the designated area set by the control unit.

[0004] JP 2018-093474 A discloses an image processing device having an image acquisition means for acquiring an image of a subject, a calculation means for calculating the proportion of pixels included in a preset brightness range relative to the entire image, an image processing means for enhancing the contrast of the image when the proportion is equal to or greater than a first threshold, and a control means for changing the brightness range in which the proportion is calculated in accordance with at least one of the illuminance of the subject and a target exposure value when acquiring the image. Summary of the Invention

[0005] One embodiment of the technology disclosed herein provides an image processing device, an image processing method, and a program that can change the aspect of a first image and / or first luminance information in response to an instruction received by a reception device, for example. [Means for solving the problem]

[0006] The image processing device of the present disclosure is an image processing device including a processor, which acquires distance information data relating to distance information between an image sensor and a subject, outputs first image data indicating a first image obtained by capturing an image with the image sensor, outputs first luminance information data indicating first luminance information created based on a first signal of the first image data for at least a first region out of a plurality of regions into which the first image is classified according to the distance information, and, when a first instruction relating to the first luminance information is received by a receiving device, performs a first process of reflecting the content of the first instruction in the first image and / or the first luminance information.

[0007] The first luminance information may be a first histogram.

[0008] The first histogram may show the relationship between signal value and the number of pixels.

[0009] The processor outputs second luminance information data indicating second luminance information created based on a second signal of the first image data for a second region of the multiple regions, and the first processing may include a second processing of prohibiting the content of the first instruction from being reflected in the second region and / or the second luminance information.

[0010] The processor may output third luminance information data indicating third luminance information created based on a third signal of the first image data for a third region among the multiple regions, and the first processing may include a third processing for reflecting the content of the first instruction in the third region and / or the third luminance information.

[0011] The first processing is processing for changing the first signal in accordance with the content of the first instruction, and the third processing is processing for changing the third signal in accordance with the content of the first instruction, and the amount of change in the first signal value contained in the first signal may be different from the amount of change in the second signal value contained in the third signal.

[0012] When a range of distances between a plurality of first pixels corresponding to a first region and the subject is defined as a first distance range, and a range of distances between a plurality of second pixels corresponding to a third region and the subject is defined as a second distance range, the amount of change in the first signal value may be constant in the first distance range, and the amount of change in the second signal value may be constant in the second distance range.

[0013] The first processing is processing for changing the first signal in accordance with the content of the first instruction, and the third processing is processing for changing the third signal in accordance with the content of the first instruction, and the amount of change in the second signal value contained in the third signal may differ depending on the distance between the subject and a plurality of second pixels corresponding to the third region.

[0014] The first process may be a process of changing the first signal in accordance with the content of the first instruction.

[0015] The first instruction may be an instruction to change the form of the first luminance information.

[0016] The first brightness information may be a second histogram having a plurality of bins, and the first instruction may be an instruction to move a bin among the plurality of bins that corresponds to a third signal value selected based on the first instruction.

[0017] The processor may output second image data indicative of a second image in which the regions are partitioned differently depending on the distance information.

[0018] The processor may output third image data indicating a distance map image representing a distribution of distance information with respect to the angle of view of a first imaging device equipped with an image sensor, and output fourth image data indicating a reference distance image representing a reference distance for classifying multiple regions.

[0019] The reference distance image may be an image showing a scale bar and a slider, where the scale bar indicates a plurality of distance ranges corresponding to a plurality of regions, the slider is provided on the scale bar, and the position of the slider indicates the reference distance.

[0020] The scale bar may be a single scale bar that indicates multiple distance ranges together.

[0021] The scale bar may be a plurality of scale bars each indicating a plurality of distance ranges separately.

[0022] The processor receives a second instruction for the reception device to output the third image data and / or the fourth image data. When the instruction is received, fifth image data showing a third image in which a plurality of regions are divided in different manners according to the distance information may be output.

[0023] When the acceptance device accepts a third instruction related to the reference distance, the processor may perform a fourth process of reflecting the content of the third instruction in the reference distance image, and may change the reference distance according to the content of the third instruction.

[0024] The first image data may be video image data.

[0025] The image processing device may be an imaging device.

[0026] The processor may output the first image data and / or the first luminance information data to a display destination.

[0027] The first processing may be processing for changing a display mode of the first image and / or the first luminance information displayed on the display destination.

[0028] The image sensor may have a plurality of phase difference pixels, and the processor may acquire distance information data based on phase difference pixel data output from the phase difference pixels.

[0029] The phase difference pixel is a pixel that selectively outputs non-phase difference pixel data and phase difference pixel data, and the non-phase difference pixel data is pixel data obtained by performing photoelectric conversion by the entire region of the phase difference pixel, and the phase difference pixel data may be pixel data obtained by performing photoelectric conversion by a partial region of the phase difference pixel.

[0030] The image processing method disclosed herein includes acquiring distance information data relating to distance information between an image sensor and a subject, outputting first image data indicating a first image obtained by imaging using the image sensor, outputting first luminance information data indicating first luminance information created based on a first signal of the first image data for at least a first region of a plurality of regions in which the first image is classified according to the distance information, and, when a first instruction relating to the first luminance information is accepted by a accepting device, performing a first process to reflect the content of the first instruction in the first image and / or the first luminance information.

[0031] The program disclosed herein is a program for causing a computer to execute processes including acquiring distance information data relating to distance information between an image sensor and a subject, outputting first image data indicating a first image obtained by capturing an image with the image sensor, outputting first luminance information data indicating first luminance information created based on a first signal of the first image data for at least a first region out of a plurality of regions into which the first image is classified according to the distance information, and, when a first instruction relating to the first luminance information is accepted by a accepting device, performing a first process for reflecting the content of the first instruction in the first image and / or the first luminance information. [Brief description of the drawings]

[0032] [Figure 1] 1 is a schematic diagram illustrating an example of a configuration of an imaging device according to an embodiment. [Diagram 2] 1 is a schematic diagram illustrating an example of a hardware configuration of an optical system and an electrical system of an image capturing apparatus according to an embodiment. [Diagram 3]1 is a schematic diagram illustrating an example of the configuration of a photoelectric conversion element according to an embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of a functional configuration of a CPU according to the embodiment. [Diagram 5] 4 is a block diagram showing an example of a functional configuration of an operation mode setting processing unit according to the embodiment; FIG. [Figure 6] 2 is a block diagram showing an example of a functional configuration of an imaging processing unit according to the embodiment. FIG. [Figure 7] 2 is a block diagram showing an example of a functional configuration of an image adjustment processing unit according to the embodiment. FIG. [Figure 8] FIG. 11 is an operation explanatory diagram showing an example of a first operation of the image adjustment processing unit according to the embodiment. [Figure 9] 13 is an operation explanatory diagram showing an example of a second operation of the image adjustment processing unit according to the embodiment. FIG. [Figure 10] FIG. 13 is an operation explanatory diagram showing an example of a third operation of the image adjustment processing unit according to the embodiment. [Figure 11] FIG. 13 is an operation explanatory diagram showing an example of a fourth operation of the image adjustment processing unit according to the embodiment. [Figure 12] FIG. 13 is an operation explanatory diagram showing an example of a fifth operation of the image adjustment processing unit according to the embodiment. [Figure 13] FIG. 13 is an operation explanatory diagram showing an example of a sixth operation of the image adjustment processing unit according to the embodiment. [Figure 14] 4 is a graph showing a first example of a relationship between a signal value before processing and a signal value after processing. [Figure 15] FIG. 13 is an operation explanatory diagram showing an example of a seventh operation of the image adjustment processing unit according to the embodiment. [Figure 16] FIG. 13 is an operation explanatory diagram showing an example of an eighth operation of the image adjustment processing unit according to the embodiment. [Figure 17] FIG. 13 is an operation explanatory diagram showing an example of a ninth operation of the image adjustment processing unit according to the embodiment. [Figure 18] FIG. 19 is an operation explanatory diagram showing an example of a tenth operation of the image adjustment processing unit according to the embodiment. [Figure 19] 10 is a graph showing a second example of the relationship between the signal value before processing and the signal value after processing. [Figure 20] FIG. 11 is an operation explanatory diagram showing an example of an eleventh operation of the image adjustment processing unit according to the embodiment. [Figure 21] FIG. 23 is an operation explanatory diagram showing an example of a twelfth operation of the image adjustment processing unit according to the embodiment. [Figure 22] 13 is a graph showing a third example of the relationship between the signal value before processing and the signal value after processing. [Diagram 23] FIG. 23 is an operation explanatory diagram showing an example of a thirteenth operation of the image adjustment processing unit according to the embodiment. [Figure 24] 4 is a block diagram showing an example of a functional configuration of a reference distance change processing unit according to the embodiment; FIG. [Diagram 25] 10 is an operation explanatory diagram showing an example of a first operation of a reference distance change processing unit according to the embodiment; FIG. [Figure 26] 13 is an operation explanatory diagram showing an example of a second operation of the reference distance change processing unit according to the embodiment. FIG. [Figure 27] 13 is an operation explanatory diagram showing an example of a third operation of the reference distance change processing unit according to the embodiment. FIG. [Figure 28] 13 is an operation explanatory diagram showing an example of a fourth operation of the reference distance change processing unit according to the embodiment. FIG. [Figure 29] FIG. 13 is an operation explanatory diagram showing an example of a fifth operation of the reference distance change processing unit according to the embodiment. [Diagram 30] FIG. 13 is an operation explanatory diagram showing an example of a sixth operation of the reference distance change processing unit according to the embodiment. [Diagram 31] FIG. 13 is an operation explanatory diagram showing an example of a seventh operation of the reference distance change processing unit according to the embodiment. [Diagram 32] FIG. 13 is an operation explanatory diagram showing an example of an eighth operation of the reference distance change processing unit according to the embodiment. [Diagram 33] FIG. 13 is an operation explanatory diagram showing an example of a ninth operation of the reference distance change processing unit according to the embodiment. [Diagram 34] 10 is a flowchart showing an example of the flow of an operation mode setting process according to the embodiment. [Diagram 35] 10 is a flowchart showing an example of a flow of an imaging process according to the embodiment. [Diagram 36] 10 is a flowchart showing an example of the flow of an image adjustment process according to the embodiment. [Figure 37] 10 is a flowchart showing an example of the flow of a reference distance change process according to the embodiment. [Figure 38] FIG. 11 is a diagram showing a first modified example of processing intensity data according to the embodiment. [Figure 39] 13A and 13B are diagrams illustrating an example of how the forms of a plurality of histograms are changed according to processing strength according to the embodiment. [Diagram 40] FIG. 11 is a diagram showing a second modified example of processing intensity data according to the embodiment. [Diagram 41] FIG. 13 is a diagram showing a third modified example of processing intensity data according to the embodiment. [Diagram 42] 13A and 13B are diagrams illustrating modified examples of the reference distance image according to the embodiment. [Diagram 43] FIG. 13 is a diagram showing a modified example of luminance information according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Hereinafter, an example of an image processing device, an image processing method, and a program according to the technique of the present disclosure will be described with reference to the attached drawings.

[0034] First, the terms used in the following description will be explained.

[0035] CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". EL is an abbreviation for "Electro-Luminescence". fps is an abbreviation for "frame per second". CPU is an abbreviation for "Central Processing Unit". NVM is an abbreviation for "Non-volatile memory". RAM is an abbreviation for "Random Access Memory". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". HDD is an abbreviation for "Hard Disk Drive". SSD is an abbreviation for "Solid State Drive". ASIC is an abbreviation for "Application Specific Integrated Circuit". FPGA is an abbreviation for "Field-Programmable Gate Array". PLD is an abbreviation for "Programmable Logic Device". MF is an abbreviation for "Manual Focus". AF is an abbreviation for "Auto Focus". UI is an abbreviation for "User Interface". I / F is an abbreviation for "Interface". A / D is an abbreviation for "Analog / Digital". USB is an abbreviation for "Universal Serial Bus". LiDAR is an abbreviation for "Light Detection And Ranging". TOF is an abbreviation for "Time of Flight". GPU is an abbreviation for "Graphics Processing Unit". TPU is an abbreviation for "Tensor processing unit". SoC is an abbreviation for "System-on-a-chip". IC is an abbreviation for "Integrated Circuit".

[0036] In this specification, "parallel" refers to parallel in the sense of including, in addition to completely parallel, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and an error that does not go against the spirit of the technology of the present disclosure. In addition, in this specification, "orthogonal" refers to orthogonal in the sense of including, in addition to completely orthogonal, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and an error that does not go against the spirit of the technology of the present disclosure. In addition, in the explanation of this specification, "match" refers to match in the sense of including, in addition to completely matching, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and an error that does not go against the spirit of the technology of the present disclosure. In addition, in the following explanation, a numerical range expressed using "~" means a range that includes the numerical values ​​written before and after "~" as the lower limit and upper limit.

[0037] 1, as an example, an imaging device 10 is a device that captures an image of a subject (not shown), and includes a controller 12, an imaging device body 16, and an interchangeable lens 18. The imaging device 10 is an example of an "image processing device," an "imaging device," and a "first imaging device" according to the technology of the present disclosure, and the controller 12 is an example of a "computer" according to the technology of the present disclosure.

[0038] The controller 12 is built into the imaging device body 16 and controls the entire imaging device 10. The interchangeable lens 18 is attached to the imaging device body 16 in an interchangeable manner.

[0039] 1, a lens-interchangeable digital camera is shown as an example of the imaging device 10. However, this is merely an example, and the imaging device 10 may be a fixed-lens digital camera, or a digital camera built into various electronic devices such as a smart device, a wearable terminal, a cell observation device, an ophthalmic observation device, or a surgical microscope.

[0040] The imaging device body 16 is provided with an image sensor 20. The image sensor 20 is an example of an "image sensor" according to the technology of the present disclosure. The image sensor 20 is, for example, a CMOS image sensor. The image sensor 20 captures an image of an imaging area including at least one subject. When the interchangeable lens 18 is attached to the imaging device body 16, subject light indicative of the subject passes through the interchangeable lens 18 and is focused on the image sensor 20, and image data indicative of the image of the subject is generated by the image sensor 20.

[0041] In this embodiment, a CMOS image sensor is exemplified as the image sensor 20, but the technology of the present disclosure is not limited to this. For example, the technology of the present disclosure can be applied even if the image sensor 20 is another type of image sensor such as a CCD image sensor.

[0042] A release button 22 and a dial 24 are provided on the top surface of the imaging device body 16. The dial 24 is operated when setting an operation mode of the imaging system and an operation mode of the playback system, and by operating the dial 24, the imaging device 10 is selectively set as an operation mode among an imaging mode, a playback mode, and a setting mode. The imaging mode is an operation mode that causes the imaging device 10 to capture an image. The playback mode is an operation mode that plays back an image (e.g., a still image and / or a moving image) obtained by capturing an image for recording in the imaging mode. The setting mode is an operation mode that is set for the imaging device 10 when setting various setting values ​​used in control related to imaging. Furthermore, the imaging device 10 is selectively set as an operation mode among an image adjustment mode and a reference distance change mode. The image adjustment mode and the reference distance change mode will be described in detail later.

[0043] The release button 22 functions as an imaging preparation instruction section and an imaging instruction section, and is capable of detecting two-stage pressing operations, an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state in which the button is pressed from a standby position to an intermediate position (half-pressed position), for example, and the imaging instruction state refers to a state in which the button is pressed beyond the intermediate position to a final pressed position (fully-pressed position). Hereinafter, the "state in which the button is pressed from the standby position to the half-pressed position" is referred to as the "half-pressed state," and the "state in which the button is pressed from the standby position to the fully-pressed position" is referred to as the "fully-pressed state." Depending on the configuration of the imaging device 10, the imaging preparation instruction state may be a state in which the user's finger is in contact with the release button 22, and the imaging instruction state may be a state in which the operating user's finger has moved from a state in which the button is in contact with the release button 22 to a state in which the finger is released.

[0044] The imaging device body 16 is provided on the rear surface with instruction keys 26 and a touch panel display 32. The touch panel display 32 includes a display 28 and a touch panel 30 (see also FIG. 2). An example of the display 28 is an EL display (e.g., an organic EL display or an inorganic EL display). The display 28 may be another type of display, such as a liquid crystal display, instead of an EL display.

[0045] The display 28 displays images and / or characters, etc. For example, when the operation mode of the imaging device 10 is the imaging mode, the display 28 is used for capturing images for live view images, i.e., for displaying live view images obtained by performing continuous imaging. Here, the term "live view image" refers to a moving image for display based on image data obtained by capturing images using the image sensor 20. The display 28 is an example of a "display destination" according to the technology of the present disclosure.

[0046] The instruction keys 26 accept various instructions. Here, "various instructions" refers to, for example, an instruction to display a menu screen, an instruction to select one or more menus, an instruction to confirm the selected content, an instruction to erase the selected content, various instructions such as zooming in, zooming out, and frame-by-frame advance. These instructions may also be given by the touch panel 30.

[0047] As an example, as shown in FIG. 2, the image sensor 20 includes a photoelectric conversion element 72. The photoelectric conversion element 72 has a light receiving surface 72A. The photoelectric conversion element 72 is disposed in the imaging device body 16 so that the center of the light receiving surface 72A coincides with the optical axis OA of the interchangeable lens 18 (see also FIG. 1). The photoelectric conversion element 72 has a plurality of photosensitive pixels 72B (see FIG. 3) arranged in a matrix, and the light receiving surface 72A is formed by the plurality of photosensitive pixels 72B. Each of the photosensitive pixels 72B has a microlens 72C (see FIG. 3). Each of the photosensitive pixels 72B is a physical pixel having a photodiode (not shown), which photoelectrically converts received light and outputs an electrical signal according to the amount of received light.

[0048] In addition, the multiple photosensitive pixels 72B have red (R), green (G), or blue (B) color filters (not shown) arranged in a matrix in a predetermined pattern array (e.g., a Bayer array, an RGB stripe array, an R / G checkerboard array, an X-Trans (registered trademark) array, or a honeycomb array, etc.).

[0049] The interchangeable lens 18 includes an imaging lens 40. The imaging lens 40 has an objective lens 40A, a focus lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, the focus lens 40B, the zoom lens 40C, and the aperture 40D are arranged in this order along the optical axis OA from the subject side (object side) to the imaging device body 16 side (image side).

[0050] The interchangeable lens 18 also includes a control device 36, a first actuator 37, a second actuator 38, and a third actuator 39, a first position sensor 42A, a second position sensor 42B, and an aperture sensor 42C. The control device 36 controls the entire interchangeable lens 18 in accordance with instructions from the imaging device body 16. The control device 36 is a device having a computer including, for example, a CPU, an NVM, and a RAM. The NVM of the control device 36 is, for example, an EEPROM. However, this is merely an example, and instead of the EEPROM, or together with the EEPROM, a HDD and / or an SSD may be applied as the NVM of the control device 36. The RAM of the control device 36 temporarily stores various information and is used as a work memory. In the control device 36, the CPU reads out necessary programs from the NVM and executes the various programs read out on the RAM to control the entire interchangeable lens 18.

[0051] Although a device having a computer is given here as an example of the control device 36, this is merely an example and devices including an ASIC, an FPGA, and / or a PLD may also be applied. Also, a device realized by combining a hardware configuration and a software configuration, for example, may be used as the control device 36.

[0052] The first actuator 37 includes a focusing slide mechanism (not shown) and a focusing motor (not shown). The focusing slide mechanism has a focus lens 40B attached thereto so as to be slidable along the optical axis OA. The focusing motor is also connected to the focusing slide mechanism, and the focusing slide mechanism is actuated by receiving power from the focusing motor to move the focus lens 40B along the optical axis OA.

[0053] The second actuator 38 includes a zoom slide mechanism (not shown) and a zoom motor (not shown). The zoom lens 40C is attached to the zoom slide mechanism so as to be slidable along the optical axis OA. The zoom motor is also connected to the zoom slide mechanism, and the zoom slide mechanism operates by receiving power from the zoom motor to move the zoom lens 40C along the optical axis OA.

[0054] In this embodiment, the focusing slide mechanism and the zooming slide mechanism are provided separately, but this is merely an example, and an integrated slide mechanism capable of both focusing and zooming may be used. focus In this case, instead of using a motor for focusing and a motor for zooming, the power generated by a single motor may be transmitted to the slide mechanism.

[0055] The third actuator 39 includes a power transmission mechanism (not shown) and a diaphragm motor (not shown). The diaphragm 40D has an opening 40D1, and the size of the opening 40D1 is variable. The opening 40D1 is formed by, for example, a plurality of blades 40D2. The plurality of blades 40D2 are connected to the power transmission mechanism. In addition, a diaphragm motor is connected to the power transmission mechanism, and the power transmission mechanism transmits the power of the diaphragm motor to the plurality of blades 40D2. The plurality of blades 40D2 change the size of the opening 40D1 by operating upon receiving the power transmitted from the power transmission mechanism. As the size of the opening 40D1 changes, the amount of aperture by the diaphragm 40D changes, thereby adjusting the exposure.

[0056] The focus motor, zoom motor, and aperture motor are connected to a control device 36, which controls the driving of the focus motor, zoom motor, and aperture motor. BooksIn the embodiment, a stepping motor is used as an example of the focus motor, zoom motor, and aperture motor. Therefore, the focus motor, zoom motor, and aperture motor operate in synchronization with a pulse signal by a command from the control device 36. Here, an example is shown in which the focus motor, zoom motor, and aperture motor are provided in the interchangeable lens 18, but this is merely an example, and at least one of the focus motor, zoom motor, and aperture motor may be provided in the imaging device body 16. The components and / or operation method of the interchangeable lens 18 can be changed as necessary.

[0057] The first position sensor 42A detects the position of the focus lens 40B on the optical axis OA. An example of the first position sensor 42A is a potentiometer. The detection result by the first position sensor 42A is obtained by the control device .

[0058] The second position sensor 42B detects the position of the zoom lens 40C on the optical axis OA. An example of the second position sensor 42B is a potentiometer. The detection result by the second position sensor 42B is obtained by the control device .

[0059] The aperture amount sensor 42C detects the size of the opening 40D1 (that is, the aperture amount). An example of the aperture amount sensor 42C is a potentiometer. The detection result by the aperture amount sensor 42C is acquired by the control device 36.

[0060] In the imaging device 10, when the operation mode is the imaging mode, the MF mode and the AF mode are selectively set according to an instruction given to the imaging device body 16. The MF mode is an operation mode in which the focus is adjusted manually. In the MF mode, for example, the user operates the focus ring 18A (see FIG. 1) or the like, and the focus lens 40B moves along the optical axis OA by an amount corresponding to the amount of operation of the focus ring 18A or the like, thereby adjusting the position of the focus. In the AF mode, AF is performed. AF refers to a process of adjusting the position of the focus according to a signal obtained from the image sensor 20. For example, in the AF mode, the imaging device body 16 calculates the distance between the imaging device 10 and the subject, and the focus lens 40B moves along the optical axis OA to a position where the subject is in focus, thereby adjusting the position of the focus.

[0061] The imaging device main body 16 includes an image sensor 20, a controller 12, an image memory 46, a UI device 48, an external I / F 50, a communication I / F 52, a photoelectric conversion element driver 54, and an input / output interface 70. The image sensor 20 also includes a photoelectric conversion element 72 and a signal processing circuit 74.

[0062] The input / output interface 70 is connected to the controller 12, the image memory 46, the UI device 48, the external I / F 50, the communication I / F 52, the photoelectric conversion element driver 54, and a signal processing circuit 74. In addition, the input / output interface 70 is also connected to the control device 36 of the interchangeable lens 18.

[0063] The controller 12 controls the entire imaging device 10. That is, in the example shown in Fig. 2, the image memory 46, the UI device 48, the external I / F 50, the communication I / F 52, the photoelectric conversion element driver 54, and the control device 36 are controlled by the controller 12. The controller 12 includes a CPU 62, an NVM 64, and a RAM 66. The CPU 62 is an example of a "processor" according to the technology of the present disclosure, and the NVM 64 and / or the RAM 66 are an example of a "memory" according to the technology of the present disclosure.

[0064] The CPU 62 , NVM 64 , and RAM 66 are connected via a bus 68 , which is connected to an input / output interface 70 .

[0065] NVM 64 is a non-transitory storage medium, and stores various parameters and various programs. The various programs include a program 65 (see FIG. 4) described later. NVM 64 is, for example, an EEPROM. However, this is merely an example, and instead of or together with EEPROM, a HDD and / or an SSD may be applied as NVM 64. RAM 66 temporarily stores various information and is used as a work memory. CPU 62 reads out necessary programs from NVM 64, and executes the read programs in RAM 66.

[0066] The CPU 62 acquires the detection result of the first position sensor 42A from the control device 36, and controls the control device 36 based on the detection result of the first position sensor 42A, thereby adjusting the position of the focus lens 40B on the optical axis OA. The CPU 62 also acquires the detection result of the second position sensor 42B from the control device 36, and controls the control device 36 based on the detection result of the second position sensor 42B, thereby adjusting the position of the zoom lens 40C on the optical axis OA. The CPU 62 also acquires the detection result of the aperture amount sensor 42C from the control device 36, and controls the control device 36 based on the detection result of the aperture amount sensor 42C, thereby adjusting the size of the opening 40D1.

[0067] The photoelectric conversion element 72 is connected to the photoelectric conversion element driver 54. The photoelectric conversion element driver 54 supplies an imaging timing signal that specifies the timing of imaging performed by the photoelectric conversion element 72 to the photoelectric conversion element 72 in accordance with an instruction from the CPU 62. The photoelectric conversion element 72 performs resetting, exposure, and output of an electrical signal in accordance with the imaging timing signal supplied from the photoelectric conversion element driver 54. Examples of the imaging timing signal include a vertical synchronization signal and a horizontal synchronization signal.

[0068] When the interchangeable lens 18 is attached to the imaging device body 16, subject light incident on the imaging lens 40 is imaged on the light receiving surface 72A by the imaging lens 40. Under the control of the photoelectric conversion element driver 54, the photoelectric conversion element 72 photoelectrically converts the subject light received by the light receiving surface 72A, and outputs an electrical signal according to the amount of subject light to the signal processing circuit 74 as imaging data 73 indicating the subject light. Specifically, the signal processing circuit 74 reads out the imaging data 73 from the photoelectric conversion element 72 in units of frames and for each horizontal line by using an exposure sequential readout method.

[0069] The signal processing circuit 74 digitizes the analog imaging data 73 read out from the photoelectric conversion element 72. The imaging data 73 digitized by the signal processing circuit 74 is so-called RAW image data. The RAW image data is image data representing an image in which R pixels, G pixels, and B pixels are arranged in a mosaic pattern.

[0070] The signal processing circuit 74 outputs the digitized imaging data 73 to the image memory 46, thereby causing the imaging data 73 to be stored in the image memory 46. The CPU 62 performs image processing (e.g., white balance processing and / or color correction, etc.) on the imaging data 73 in the image memory 46. The CPU 62 generates moving image data 80 based on the imaging data 73. An example of the moving image data 80 is moving image data for display, that is, live view image data showing a live view image. Note that, although live view image data is illustrated here, this is merely an example, and the moving image data 80 may be post view image data showing a post view image.

[0071] The UI device 48 includes a display 28. The CPU 62 causes the display 28 to display an image (here, as an example, a live view image) based on moving image data 80. The CPU 62 also causes the display 28 to display various types of information.

[0072] The UI device 48 also includes a reception device 76. The reception device 76 includes a touch panel 30 and a hard key unit 78, and receives instructions from a user. The hard key unit 78 is a plurality of hard keys including the instruction keys 26 (see FIG. 1). The CPU 62 operates according to various instructions received by the touch panel 30.

[0073] The external I / F 50 controls the exchange of various information with devices (hereinafter also referred to as "external devices") that exist outside the imaging device 10. An example of the external I / F 50 is a USB interface. To the USB interface, external devices (not shown), such as a smart device, a personal computer, a server, a USB memory, a memory card, and / or a printer, are directly or indirectly connected.

[0074] The communication I / F 52 is connected to a network (not shown). The communication I / F 52 is responsible for transmitting information between a communication device (not shown) such as a server on the network and the controller 12. For example, the communication I / F 52 transmits information in response to a request from the controller 12 to the communication device via the network. The communication I / F 52 also receives information transmitted from the communication device and outputs the received information to the controller 12 via the input / output interface 70.

[0075] As an example, as shown in FIG. 3, a plurality of photosensitive pixels 72B are two-dimensionally arranged on a light receiving surface 72A of a photoelectric conversion element 72. A color filter (not shown) and a microlens 72C are arranged in each of the photosensitive pixels 72B. In FIG. 3, one direction parallel to the light receiving surface 72A (for example, the row direction of the plurality of photosensitive pixels 72B arranged two-dimensionally) is defined as an X direction, and a direction perpendicular to the X direction (for example, the column direction of the plurality of photosensitive pixels 72B arranged two-dimensionally) is defined as a Y direction. The plurality of photosensitive pixels 72B are arranged along the X direction and the Y direction. Each of the photosensitive pixels 72B includes a pair of independent photodiodes PD1 and PD2. A first light beam (e.g., a light beam passing through a first pupil partial region in the imaging lens 40 (see FIG. 2)) obtained by pupil division of a light beam indicating a subject (hereinafter also referred to as "subject light beam") that has passed through the imaging lens 40 is incident on the photodiode PD1, and a second light beam (e.g., a light beam passing through a second pupil partial region in the imaging lens 40 (see FIG. 2)) obtained by pupil division of the subject light beam is incident on the photodiode PD2. The photodiode PD1 performs photoelectric conversion on the first light beam. The photodiode PD2 performs photoelectric conversion on the second light beam.

[0076] As an example, the photoelectric conversion element 72 is an image plane phase difference type photoelectric conversion element in which one photosensitive pixel 72B is provided with a pair of photodiodes PD1 and PD2. As an example, the photoelectric conversion element 72 has a function in which all the photosensitive pixels 72B output data related to imaging and phase difference. The photoelectric conversion element 72 outputs non-phase difference pixel data 73A by combining the pair of photodiodes PD1 and PD2 into one photosensitive pixel. In addition, the photoelectric conversion element 72 outputs phase difference pixel data 73B by detecting signals from each of the pair of photodiodes PD1 and PD2. That is, all the photosensitive pixels 72B provided in the photoelectric conversion element 72 are so-called phase difference pixels. The photosensitive pixel 72B is an example of a "phase difference pixel" according to the technology of the present disclosure.

[0077] The photosensitive pixel 72B is a pixel that selectively outputs non-phase difference pixel data 73A and phase difference pixel data 73B. The non-phase difference pixel data 73A is pixel data obtained by performing photoelectric conversion by the entire region of the photosensitive pixel 72B, and the phase difference pixel data 73B is pixel data obtained by performing photoelectric conversion by a partial region of the photosensitive pixel 72B. Here, the "entire region of the photosensitive pixel 72B" refers to the combined light receiving region of the photodiode PD1 and the photodiode PD2. Also, the "partial region of the photosensitive pixel 72B" refers to the light receiving region of the photodiode PD1 or the light receiving region of the photodiode PD2.

[0078] The non-phase difference pixel data 73A can also be generated based on the phase difference pixel data 73B. For example, the phase difference pixel data 73B is added for each pair of pixel signals corresponding to the pair of photodiodes PD1 and PD2, thereby generating the non-phase difference pixel data 73A. The phase difference pixel data 73B may include only data output from one of the pair of photodiodes PD1 and PD2. For example, when the phase difference pixel data 73B includes only data output from the photodiode PD1, the phase difference pixel data 73B is subtracted from the non-phase difference pixel data 73A for each pixel, thereby generating data output from the photodiode PD2.

[0079] The imaging data 73 includes image data 81 and phase difference pixel data 73B. The image data 81 is generated based on the non-phase difference pixel data 73A. For example, the image data 81 is obtained by A / D conversion of the analog non-phase difference pixel data 73A. That is, the image data 81 is data obtained by digitizing the non-phase difference pixel data 73A output from the photoelectric conversion element 72. The CPU 62 acquires the digitized imaging data 73 from the signal processing circuit 74, and acquires distance information data 82 based on the acquired imaging data 73. For example, the CPU 62 acquires the phase difference pixel data 73B from the imaging data 73, and generates the distance information data 82 based on the acquired phase difference pixel data 73B. The distance information data 82 is an example of the "distance information data" according to the technology of the present disclosure. The distance information data 82 is data related to distance information between the photoelectric conversion element 72 and the subject. The distance information is information related to the distance between each photosensitive pixel 72B and the subject. Hereinafter, the distance between the photosensitive pixel 72B and the subject 202 is referred to as the subject distance. Distance information between the photoelectric conversion element 72 and the subject is synonymous with distance information between the image sensor 20 and the subject. Distance information between the image sensor 20 and the subject is an example of "distance information" according to the technology of the present disclosure.

[0080] 4 as an example, a program 65 is stored in the NVM 64 of the imaging device 10. The program 65 is an example of a "program" according to the technology of the present disclosure. The CPU 62 reads out the program 65 from the NVM 64, and executes the read program 65 in the RAM 66. By executing the program 65, the CPU 62 operates as an operation mode setting processing unit 100, an imaging processing unit 110, an image adjustment processing unit 120, and a reference distance change processing unit 140.

[0081] The imaging device 10 has an imaging mode, an image adjustment mode, and a reference distance change mode as operation modes. The operation mode setting processing unit 100 selectively sets the imaging mode, the image adjustment mode, and the reference distance change mode as operation modes of the imaging device 10. When the operation mode setting processing unit 100 sets the operation mode of the imaging device 10 to the imaging mode, the CPU 62 operates as an imaging processing unit 110. Image Adjustment When the operation mode is set to the image adjustment mode, the CPU 62 operates as the image adjustment processing unit 120. Change reference distance When the mode is set, the CPU 62 operates as the reference distance change processing unit 140 .

[0082] As an example, as shown in FIG. 5, the operation mode setting processing unit 100 sets an image capture mode, an image capture mode, and an image capture mode as the operation mode of the image capture device 10. adjustment The operation mode setting processing unit 100 performs an operation mode setting process for selectively setting an image capture mode, an image adjustment mode, and a reference distance change mode. The operation mode setting processing unit 100 has an image capture mode setting unit 101, a first mode switching determination unit 102, an image adjustment mode setting unit 103, a second mode switching determination unit 104, a reference distance change mode setting unit 105, and a third mode switching determination unit 106.

[0083] The image capture mode setting unit 101 sets an image capture mode as an initial setting for the operation mode of the image capture device 10 .

[0084] The first mode switching determination unit 102 determines whether or not a first mode switching condition for switching the operation mode of the imaging device 10 from the imaging mode to the image adjustment mode is satisfied. An example of the first mode switching condition is, for example, a condition in which a first mode switching instruction for switching the operation mode of the imaging device 10 from the imaging mode to the image adjustment mode is accepted by the acceptance device 76. When the first mode switching instruction is accepted by the acceptance device 76, a first mode switching instruction signal indicating the first mode switching instruction is output from the acceptance device 76 to the CPU 62. When the first mode switching instruction signal is input to the CPU 62, the first mode switching determination unit 102 determines that the first mode switching condition for switching the operation mode of the imaging device 10 from the imaging mode to the image adjustment mode is satisfied.

[0085] When the first mode switching determination unit 102 determines that the first mode switching condition is met, the image adjustment mode setting unit 103 sets the image adjustment mode as the operation mode of the imaging device 10.

[0086] The second mode switching determination unit 104 determines whether or not a second mode switching condition for switching the operation mode of the imaging device 10 from the imaging mode or the image adjustment mode to the reference distance change mode is satisfied. An example of the second mode switching condition is, for example, a condition in which a second mode switching instruction for switching the operation mode of the imaging device 10 from the imaging mode or the image adjustment mode to the reference distance change mode is accepted by the acceptance device 76. When the second mode switching instruction is accepted by the acceptance device 76, a second mode switching instruction signal indicating the second mode switching instruction is output from the acceptance device 76 to the CPU 62. When the second mode switching instruction signal is input to the CPU 62, the second mode switching determination unit 104 determines that the second mode switching condition for switching the operation mode of the imaging device 10 from the imaging mode or the image adjustment mode to the reference distance change mode is satisfied.

[0087] When the second mode switching determination unit 104 determines that the second mode switching condition is met, the reference distance change mode setting unit 105 sets the reference distance change mode as the operation mode of the imaging device 10.

[0088] The third mode switching determination unit 106 determines whether the operation mode of the imaging device 10 is in the image adjustment mode or the reference distance change mode. When the third mode switching determination unit 106 determines that the operation mode of the imaging device 10 is in the image adjustment mode or the reference distance change mode, it determines whether a third mode switching condition for switching the operation mode of the imaging device 10 from the image adjustment mode or the reference distance change mode to the imaging mode is established. An example of the third mode switching condition is, for example, a condition in which a third mode switching instruction for switching the operation mode of the imaging device 10 from the image adjustment mode or the reference distance change mode to the imaging mode is accepted by the acceptance device 76. When the third mode switching instruction is accepted by the acceptance device 76, a third mode switching instruction signal indicating the third mode switching instruction is output from the acceptance device 76 to the CPU 62. When the third mode switching instruction signal is input to the CPU 62, the third mode switching determination unit 106 determines that the third mode switching condition for switching the operation mode of the imaging device 10 from the image adjustment mode or the reference distance change mode to the imaging mode is established.

[0089] When the third mode switching determination unit 106 determines that the third mode switching condition is met, the imaging mode setting unit 101 sets the imaging mode as the operation mode of the imaging device 10.

[0090] 6, as an example, the imaging processing unit 110 performs imaging processing for outputting moving image data 80 based on image data 81 obtained by causing the image sensor 20 to capture an image of a subject 202. The imaging processing unit 110 has an imaging control unit 111, an image data acquisition unit 112, a moving image data generation unit 113, and a moving image data output unit 114.

[0091] The imaging control unit 111 controls the photoelectric conversion element 72 to output the non-phase difference pixel data 73A. Specifically, the imaging control unit 111 outputs a first imaging command signal to the photoelectric conversion element driver 54 to cause the photoelectric conversion element 72 to output a first imaging timing signal as an imaging timing signal. The first imaging timing signal is an imaging timing signal to cause the photoelectric conversion element 72 to output the non-phase difference pixel data 73A. Each photosensitive pixel 72B of the photoelectric conversion element 72 outputs the non-phase difference pixel data 73A by performing photoelectric conversion by the entire area of ​​the photosensitive pixel 72B in accordance with the first imaging timing signal. The photoelectric conversion element 72 outputs the non-phase difference pixel data 73A output from each photosensitive pixel 72B to the signal processing circuit 74. The signal processing circuit 74 generates image data 81 by digitizing the non-phase difference pixel data 73A output from each photosensitive pixel 72B.

[0092] The image data acquisition unit 112 acquires image data 81 from the signal processing circuit 74. The image data 81 is data representing an image 200 obtained by the image sensor 20 capturing an image of a plurality of subjects 202.

[0093] The moving image data generator 113 generates moving image data 80 based on the image data 81 acquired by the image data acquisition unit 112 .

[0094] The moving image data output unit 114 outputs the moving image data 80 generated by the moving image data generation unit 113 to the display 28 at a default frame rate (for example, 30 frames / second). The display 28 displays an image based on the moving image data 80.

[0095] In the example shown in FIG. 6, the image 200 is a landscape image. A plurality of subjects 202 (four subjects 202 as an example) are captured in the image 200. Hereinafter, when the plurality of subjects 202 are to be distinguished from one another, the plurality of subjects 202 are referred to as a first subject 202A, a second subject 202B, a third subject 202C, and a fourth subject 202D, respectively. As an example, the first subject 202A is trees, and the second subject 202B, the third subject 202C, and the fourth subject 202D are mountains, respectively. The first subject 202A, the second subject 202B, the third subject 202C, and the fourth subject 202D are located at positions that are distant from the imaging device 10 in the order of the first subject 202A, the second subject 202B, the third subject 202C, and the fourth subject 202D. As an example, a second subject 202B, a third subject 202C, and a fourth subject 202D are covered with a haze 204, and the haze 204 is captured as an image in the image 200. In the example shown in Fig. 6, the haze 204 is represented by hatching.

[0096] 7 as an example, the image adjustment processing unit 120 performs image adjustment processing for adjusting a histogram 208 and an image 200 (described later) based on an adjustment instruction received by the reception device 76. The image adjustment processing unit 120 has a first imaging control unit 121, an image data acquisition unit 122, a second imaging control unit 123, a distance information data acquisition unit 124, a reference distance data acquisition unit 125, an area classification data generation unit 126, a histogram data generation unit 127, an adjustment instruction determination unit 128, an adjustment instruction data acquisition unit 129, a processing intensity setting unit 130, a signal value processing unit 131, a histogram adjustment unit 132, an image adjustment unit 133, a moving image data generation unit 134, and a moving image data output unit 135.

[0097] As an example, as shown in FIG. 8, the first imaging control unit 121 controls the photoelectric conversion element 72 to output the non-phase difference pixel data 73A. Specifically, the first imaging control unit 121 outputs a first imaging command signal to the photoelectric conversion element driver 54 to cause the photoelectric conversion element 72 to output the first imaging timing signal as an imaging timing signal. The first imaging timing signal is an imaging timing signal to cause the photoelectric conversion element 72 to output the non-phase difference pixel data 73A. Each photosensitive pixel 72B of the photoelectric conversion element 72 outputs the non-phase difference pixel data 73A by performing photoelectric conversion by the entire region of the photosensitive pixel 72B according to the first imaging timing signal. The photoelectric conversion element 72 outputs the non-phase difference pixel data 73A output from each photosensitive pixel 72B to the signal processing circuit 74. The signal processing circuit 74 generates image data 81 by digitizing the non-phase difference pixel data 73A output from each photosensitive pixel 72B. The image data 81 is an example of the “first image data” according to the technique of the present disclosure. The image data acquisition unit 122 acquires the image data 81 from the signal processing circuit 74.

[0098] The second imaging control unit 123 controls the photoelectric conversion element 72 to output the phase difference pixel data 73B. Specifically, the second imaging control unit 123 outputs a second imaging command signal to the photoelectric conversion element driver 54 to cause the photoelectric conversion element 72 to output a second imaging timing signal as an imaging timing signal. The second imaging timing signal is an imaging timing signal to cause the photoelectric conversion element 72 to output the phase difference pixel data 73B. Each photosensitive pixel 72B of the photoelectric conversion element 72 outputs the phase difference pixel data 73B by performing photoelectric conversion using a partial region of the photosensitive pixel 72B in accordance with the second imaging timing signal. The photoelectric conversion element 72 outputs the phase difference pixel data 73B obtained from each photosensitive pixel 72B to the signal processing circuit 74. The signal processing circuit 74 digitizes the phase difference pixel data 73B and outputs the digitized phase difference pixel data 73B to the distance information data acquisition unit 124.

[0099] The distance information data acquisition unit 124 acquires the distance information data 82. Specifically, the distance information data acquisition unit 124 acquires the phase difference pixel data 73B from the signal processing circuit 74, and generates distance information data 82 indicating distance information related to the subject distance corresponding to each photosensitive pixel 72B based on the acquired phase difference pixel data 73B.

[0100] The reference distance data acquisition unit 125 acquires reference distance data 83 pre-stored in the NVM 64. The reference distance data 83 is data indicating a reference distance for classifying an image 200 (see FIG. 9) based on the image data 81 into a plurality of regions 206 (see FIG. 9) according to the subject distance.

[0101] As an example, as shown in FIG. 9, the area classification data generation unit 126 generates area classification data 84 for classifying an image 200 into multiple areas 206 (for example, four areas 206) according to the subject distance based on distance information data 82 and reference distance data 83.

[0102] The region classification data 84 is data indicating the multiple regions 206. The multiple regions 206 are classified for each subject distance based on the reference distance (three reference distances as an example) indicated by the reference distance data. Hereinafter, when the multiple regions 206 are to be distinguished from one another, the multiple regions 206 will be referred to as a first region 206A, a second region 206B, a third region 206C, and a fourth region 206D, respectively.

[0103] In the example shown in FIG. 9, the subject distances corresponding to the regions 206 are longer in the order of the first region 206A, the second region 206B, the third region 206C, and the fourth region 206D. The first region 206A and the second region 206B are regions classified based on a first reference distance among a plurality of reference distances. The second region 206B and the third region 206C are regions classified based on a second reference distance among a plurality of reference distances. The third region 206C and the fourth region 206D are regions classified based on a third reference distance among a plurality of reference distances. The first region 206A is a region corresponding to the first subject 202A, the second region 206B is a region corresponding to the second subject 202B, the third region 206C is a region corresponding to the third subject 202C, and the fourth region 206D is a region corresponding to the fourth subject 202D.

[0104] 10, the histogram data generating unit 127 generates histogram data 85 corresponding to each region 206 based on the image data 81 and region classification data 84. The histogram data 85 is data indicating a histogram 208 corresponding to each region 206. The histogram data 85 has first histogram data 85A, second histogram data 85B, third histogram data 85C, and fourth histogram data 85D. Hereinafter, when the histograms 208 corresponding to each region 206 are to be distinguished from one another, the multiple histograms 208 will be referred to as a first histogram 208A, a second histogram 208B, a third histogram 208C, and a fourth histogram 208D, respectively.

[0105] The first histogram data 85A is data representing the first histogram 208A corresponding to the first region 206A. The second histogram data 85B is data representing the second histogram 208B corresponding to the second region 206B. The third histogram data 85C is data representing the third histogram 208C corresponding to the third region 206C. The fourth histogram data 85D is data representing the fourth histogram 208D corresponding to the fourth region 206D.

[0106] Each histogram 208 is a histogram created based on a signal of the image data 81 for each region 206. The signal of the image data 81 is a collection of signal values ​​(i.e., a group of signal values). That is, the first histogram 208A is created based on a first signal (i.e., a first signal group) corresponding to the first region 206A. The second histogram 208B is created based on a second signal (i.e., a second signal group) corresponding to the second region 206B. The third histogram 208C is created based on a third signal (i.e., a third signal group) corresponding to the third region 206C. The fourth histogram 208D is created based on a fourth signal (i.e., a fourth signal group) corresponding to the fourth region 206D.

[0107] As an example, each histogram 208 is a histogram that indicates the relationship between signal value and pixel count for each region 206. The pixel count refers to the number of pixels that form the image 200 (hereinafter referred to as image pixels).

[0108] In the example shown in Fig. 10, the signal value increases from the first region 206A to the fourth region 206D. Therefore, from the multiple histograms 208, it can be confirmed that the first region 206A is not covered with the haze 204 (see Fig. 7), and that the density of the haze 204 increases from the second region 206B to the fourth region 206D.

[0109] 11 shows an example in which an adjustment instruction for adjusting the histogram 208 and the image 200 has not been received by the reception device 76. The adjustment instruction is an instruction to change the form of any of the multiple histograms 208. If an adjustment instruction has been received by the reception device 76, adjustment instruction data 86 (see FIG. 12) is stored in the RAM 66, but if an adjustment instruction has not been received by the reception device 76, the adjustment instruction data 86 is not stored in the RAM 66. The adjustment instruction determination unit 128 determines whether or not the adjustment instruction data 86 is stored in the RAM 66.

[0110] When the adjustment instruction determination unit 128 determines that the adjustment instruction data 86 indicating the adjustment instruction is not stored in the RAM 66, the moving image data generation unit 134 performs a histogram data Generation part 127 and histogram data 85 generated by the above.

[0111] The video data output unit 135 outputs the video data 80 generated by the video data generation unit 134 to the display 28. The display 28 displays an image based on the video data 80. In the example shown in FIG. 11, an image 200 indicated by image data 81 is displayed. and A plurality of histograms 208 represented by the histogram data 85 are displayed on the display 28. The image 200 is an example of a "first image" according to the technique of the present disclosure.

[0112] 12 shows an example in which an adjustment instruction is received by the reception device 76. As an example, the adjustment instruction is an instruction to change the form of the histogram 208 displayed on the display 28 via the touch panel 30. Hereinafter, as an example, a case in which the form of the second histogram 208B is changed by the adjustment instruction will be described.

[0113] The second histogram 208B has a plurality of bins 210. The adjustment instruction is an example of an instruction to change the form of the second histogram 208B. To explain the adjustment instruction in more detail, the adjustment instruction is an instruction to select a signal value from the plurality of bins 210 and move the bin 210 corresponding to the selected signal value. In the example shown in FIG. 12, an instruction to move the bin 210 by swiping on the touch panel 30 is shown as an example of the adjustment instruction. In addition, in the example shown in FIG. 12, a mode in which the bin 210 is moved in a direction in which the signal value selected based on the adjustment instruction accepted by the acceptance device 76 becomes smaller compared to before the adjustment instruction is accepted by the acceptance device 76 is shown. Hereinafter, when it is necessary to distinguish the adjustment instruction of the mode shown in FIG. 12 from adjustment instructions of other modes, the adjustment instruction of the mode shown in FIG. 12 is referred to as a minus side swipe instruction.

[0114] The adjustment instruction is an example of a "first instruction" according to the technology of the present disclosure. The second histogram 208B is an example of a "first luminance information," a "first histogram," and a "second histogram" according to the technology of the present disclosure. The multiple bins 210 are an example of a "multiple bins" according to the technology of the present disclosure. The signal value selected based on the adjustment instruction is an example of a "third signal value" according to the technology of the present disclosure.

[0115] When an adjustment instruction is received by the reception device 76, the image adjustment processing unit 120 stores adjustment instruction data 86 indicating the adjustment instruction in the RAM 66. Specifically, data indicating a signal value selected based on the adjustment instruction and the amount of movement of the bin 210 are stored in the RAM 66 as the adjustment instruction data 86. The amount of movement of the bin 210 corresponds to the difference between the signal value corresponding to the bin 210 before the movement and the signal value corresponding to the bin 210 after the movement.

[0116] The adjustment instruction data acquisition unit 129 acquires the adjustment instruction data 86 stored in the RAM 66 when the adjustment instruction determination unit 128 determines that the adjustment instruction data 86 indicating an adjustment instruction is stored in the RAM 66 .

[0117] As an example, as shown in FIG. 13, processing intensity data 87 is stored in NVM 64. Processing intensity data 87 is data indicating the relationship between the subject distance corresponding to an image pixel and processing intensity. The processing intensity is the amount of change in the signal value changed for each image pixel based on an adjustment instruction. The reference intensity is the amount of change in the signal value selected based on the adjustment instruction (i.e., the difference between the signal value corresponding to the bin 210 before movement and the signal value corresponding to the bin 210 after movement). In the explanation of processing intensity using FIG. 13, processing refers to changing the signal value based on the adjustment instruction.

[0118] The NVM 64 stores processing intensity data 87 corresponding to each histogram 208. As an example, the processing intensity data 87 shown in Fig. 13 indicates data for changing the form of the second histogram 208B based on an adjustment instruction.

[0119] In the processing intensity data 87, the range of the subject distance is classified into a plurality of distance ranges 212. Hereinafter, when the plurality of distance ranges 212 are described while distinguishing them from one another, the plurality of distance ranges 212 are referred to as a first distance range 212A, a second distance range 212B, a third distance range 212C, and a fourth distance range 212D, respectively. The first distance range 212A is a range of the subject distance corresponding to the first region 206A. The second distance range 212B is a range of the subject distance corresponding to the second region 206B. The third distance range 212C is a range of the subject distance corresponding to the third region 206C. The fourth distance range 212D is a range of the subject distance corresponding to the fourth region 206D. The plurality of distance ranges 212 are ranges in which the subject distance increases in the order of the first distance range 212A, the second distance range 212B, the third distance range 212C, and the fourth distance range 212D.

[0120] A processing strength is set for each of the multiple distance ranges 212. Hereinafter, the processing strength corresponding to the first distance range 212A is referred to as the first processing strength, the processing strength corresponding to the second distance range 212B is referred to as the second processing strength, the processing strength corresponding to the third distance range 212C is referred to as the third processing strength, and the processing strength corresponding to the fourth distance range 212D is referred to as the fourth processing strength.

[0121] In the example shown in FIG. 13, the second processing intensity is set to a constant value corresponding to the reference intensity. That is, for each image pixel corresponding to the second distance range 212B, the amount of change in the signal value to be changed based on the adjustment instruction is constant. Also, in the example shown in FIG. 13, the first processing intensity, the third processing intensity, and the fourth processing intensity are set to 0. That is, for each image pixel corresponding to the first distance range 212A, the third distance range 212C, and the fourth distance range 212D, the amount of change in the signal value is 0.

[0122] The processing intensity setting unit 130 sets the processing intensity corresponding to each image pixel based on the processing intensity data 87. In the example shown in FIG. 13, the processing intensity is set to the reference intensity for the image pixels corresponding to the second distance range 212B, and the processing intensity is set to 0 for the image pixels corresponding to the first distance range 212A, the third distance range 212C, and the fourth distance range 212D.

[0123] The signal value processing unit 131 calculates the signal value after processing for each image pixel based on the processing intensity set by the processing intensity setting unit 130. As an example, when the adjustment instruction is an instruction to move the bin 210 in a direction in which the signal value becomes smaller compared to before the adjustment instruction is received by the receiving device 76, the signal value corresponding to each image pixel is calculated by formulas (1) and (2). However, formula (2) is applied when Value1 < Black.

Number

Number

[0124] Note that Value0 is the signal value before processing corresponding to each image pixel (hereinafter referred to as the signal value before processing). Value1 is the signal value after processing corresponding to each image pixel (hereinafter referred to as the signal value after processing). Sel0 is the pre-processing value of the signal value selected based on the adjustment instruction. Sel1 is the post-processing value of the signal value selected based on the adjustment instruction. Black is the minimum signal value (hereinafter referred to as the minimum signal value). White is the maximum signal value (hereinafter referred to as the maximum signal value). Note that the pre-processing signal value corresponds to the signal value before being adjusted according to the adjustment instruction. Also, the post-processing signal value corresponds to the signal value after being adjusted according to the adjustment instruction.

[0125] As an example, FIG. 14 shows a graph illustrating the relationship between the signal value before processing and the signal value after processing when a minus side swipe instruction is received by the receiving device 76. In the example shown in FIG. 14, the value Sel0 before processing of the signal value selected based on the minus side swipe instruction is 0.2, and the value Sel1 after processing of the signal value selected based on the minus side swipe instruction is 0.1. The minimum signal value Black is 0, and the maximum signal value White is 1.0. When the signal value before processing is 0.2, the signal value after processing becomes 0.1. The maximum signal value remains 1.0 before and after processing.

[0126] 15 as an example, the histogram adjustment unit 132 performs processing to reflect the contents of the adjustment instruction in at least one of the multiple histograms 208. Specifically, the histogram adjustment unit 132 generates adjusted histogram data 88 in which the contents of the adjustment instruction are reflected in at least one of the multiple histograms 208, based on the signal value calculated by the signal value processing unit 131 (see FIG. 13). The adjusted histogram data 88 is data indicating the multiple histograms 208 including the histogram 208 whose form has been changed in accordance with the adjustment instruction.

[0127] The histogram 208 whose shape has been changed in accordance with the adjustment instruction is a histogram showing the relationship between the signal value and the number of pixels after processing. As an example, Fig. 15 shows an example in which the shape of the second histogram 208B is changed in accordance with a minus side swipe instruction. The minus side swipe instruction changes the shape of the second histogram 208B to a lower signal value compared to before the minus side swipe instruction was accepted by the accepting device 76.

[0128] 12 to 15, the processing strength of second distance range 212B corresponding to second histogram 208B is set to a value greater than 0 (see FIG. 13). Therefore, the contents of the adjustment instruction for second histogram 208B are reflected in second histogram 208B. Then, adjusted histogram data 88 shown in FIG. 15 includes second histogram data 85B indicating second histogram 208B reflecting the contents of the adjustment instruction.

[0129] 12 to 15, the processing strength of the first distance range 212A corresponding to the first histogram 208A, the processing strength of the third distance range 212C corresponding to the third histogram 208C, and the processing strength of the fourth distance range 212D corresponding to the fourth histogram 208D are all set to 0 (see FIG. 13). Therefore, the contents of the adjustment instruction for the second histogram 208B are prohibited from being reflected in the first histogram 208A, the third histogram 208C, and the fourth histogram 208D. The adjusted histogram data 88 shown in FIG. 15 includes the first histogram data 85A, the third histogram data 85C, and the fourth histogram data 85D included in the histogram data 85 as they are.

[0130] 12 to 15 are examples in which the shape of the second histogram 208B is changed in accordance with the adjustment instruction, but if the adjustment instruction is an instruction to change the shape of the first histogram 208A, the shape of the first histogram 208A is changed in accordance with the adjustment instruction. Similarly, if the adjustment instruction is an instruction to change the shape of the third histogram 208C, the shape of the third histogram 208C is changed in accordance with the adjustment instruction, and if the adjustment instruction is an instruction to change the shape of the fourth histogram 208D, the shape of the fourth histogram 208D is changed in accordance with the adjustment instruction.

[0131] The process of generating adjusted histogram data 88 by histogram adjustment unit 132 is an example of a "first process" according to the technology of the present disclosure. Among the processes executed by histogram adjustment unit 132, the process of prohibiting the content of the adjustment instruction for second histogram 208B from being reflected in first histogram 208A, third histogram 208C, and fourth histogram 208D is an example of a "second process" according to the technology of the present disclosure.

[0132] The second region 206B is an example of a "first region" according to the technology of the present disclosure. The first region 206A, the third region 206C, and the fourth region 206D are an example of a "second region" according to the technology of the present disclosure. The signal corresponding to the second distance range 212B is an example of a "first signal" according to the technology of the present disclosure. The signals corresponding to the first distance range 212A, the third distance range 212C, and the fourth distance range 212D are an example of a "second signal" according to the technology of the present disclosure.

[0133] The second histogram 208B is an example of "first luminance information" according to the technology of the present disclosure. The second histogram data 85B is an example of "first luminance information data" according to the technology of the present disclosure. The first histogram 208A, the third histogram 208C, and the fourth histogram 208D are an example of "second luminance information" according to the technology of the present disclosure. The first histogram data 85A, the third histogram data 85C, and the fourth histogram data 85D are an example of "second luminance information data" according to the technology of the present disclosure.

[0134] 16, the image data 81 includes first image data 81A, second image data 81B, third image data 81C, and fourth image data 81D. The first image data 81A is data corresponding to the first region 206A. The second image data 81B is data corresponding to the second region 206B. The third image data 81C is data corresponding to the third region 206C. The fourth image data 81D is data corresponding to the fourth region 206D.

[0135] The image adjustment unit 133 performs processing to reflect the contents of the adjustment instructions in the image 200. Specifically, the image adjustment unit 133 generates adjusted image data 89 in which the contents of the adjustment instructions are reflected in at least one of the multiple regions 206 based on the signal values ​​calculated by the signal value processing unit 131 (see FIG. 13). The adjusted image data 89 is data in which the signal values ​​of the image data 81 have been changed in accordance with the adjustment instructions.

[0136] 16, the adjusted image data 89 includes second image data 81B whose signal values ​​have been changed in accordance with the adjustment instruction. That is, the signal values ​​included in the second image data 81B are the signal values ​​after processing.

[0137] 12 to 16, the processing strength of the second distance range 212B corresponding to the second region 206B is set to a value greater than 0 (see FIG. 13). Therefore, the contents of the adjustment instruction for the second histogram 208B are reflected in the second region 206B. The adjusted image data 89 shown in FIG. 16 includes second image data 81B indicating the second region 206B in which the contents of the adjustment instruction are reflected.

[0138] 12 to 16, the processing strength of the first distance range 212A corresponding to the first region 206A, the processing strength of the third distance range 212C corresponding to the third region 206C, and the processing strength of the fourth distance range 212D corresponding to the fourth region 206D are all set to 0 (see FIG. 13). Therefore, the content of the adjustment instruction for the second region 206B is prohibited from being reflected in the first region 206A, the third region 206C, and the fourth region 206D. The adjusted image data 89 shown in FIG. 16 includes the first image data 81A, the third image data 81C, and the fourth image data 81D contained in the image data 81 as they are.

[0139] 12 to 16, the signal value of the second image data 81B is changed by changing the shape of the second histogram 208B in accordance with the adjustment instruction, but if the adjustment instruction is an instruction to change the shape of the first histogram 208A, the signal value of the first image data 81A is changed in accordance with the adjustment instruction. Similarly, if the adjustment instruction is an instruction to change the shape of the third histogram 208C, the signal value of the third image data 81C is changed in accordance with the adjustment instruction, and if the adjustment instruction is an instruction to change the shape of the fourth histogram 208D, the signal value of the fourth image data 81D is changed in accordance with the adjustment instruction.

[0140] The process of generating adjusted image data 89 by image adjustment unit 133 is an example of a "first process" according to the technology of the present disclosure. Among the processes executed by image adjustment unit 133, the process of prohibiting the content of the adjustment instruction for second histogram 208B from being reflected in first region 206A, third region 206C, and fourth region 206D is an example of a "second process" according to the technology of the present disclosure.

[0141] As an example, FIG. 17 shows how the image 200 and histogram 208 change when moving from a state before the adjustment instruction is accepted by the reception device 76 to a state after the adjustment instruction is accepted by the reception device 76.

[0142] As an example, as shown in FIG. 17, when an adjustment instruction is not accepted by the reception device 76, the video data generation unit 134 generates video data 80 including image data 81 and histogram data 85, but when an adjustment instruction is accepted by the reception device 76, it generates video data 80 including adjusted image data 89 and adjusted histogram data 88.

[0143] The moving image data output unit 135 outputs the moving image data 80 generated by the moving image data generation unit 134 to the display 28. The display 28 displays an image based on the moving image data 80.

[0144] In the example shown in Fig. 17, the form of the second histogram 208B is changed based on the adjustment instruction. That is, the display mode of the second histogram 208B displayed on the display 28 is changed. Also, the signal value corresponding to the second region 206B is changed based on the adjustment instruction, thereby changing the display mode of the second region 206B displayed on the display 28. In the example shown in Fig. 17, the signal value corresponding to the second region 206B is changed in accordance with a minus side swipe instruction as an adjustment instruction, so that the brightness of the second region 206B becomes darker than before the minus side swipe instruction is accepted by the accepting device 76, and the haze 204 reflected in the second region 206B is suppressed.

[0145] As an example, as shown in FIG. 18, the adjustment instruction may be an instruction to move the bin 210 in a direction in which the signal value selected based on the adjustment instruction becomes larger compared to before the adjustment instruction is accepted by the accepting device 76. Hereinafter, when it is necessary to distinguish the adjustment instruction shown in FIG. 18 from other adjustment instructions, the adjustment instruction shown in FIG. 18 will be referred to as a plus side swipe instruction. When the adjustment instruction is a plus side swipe instruction, the signal value processing unit 131 calculates the signal value corresponding to each image pixel by formulas (3) and (4). However, formula (4) is applied when Value1>White.

number

number

[0146] As an example, FIG. 19 shows a graph showing the relationship between the signal value before processing and the signal value after processing when a positive side swipe instruction is received by the receiving device 76. In the example shown in FIG. 19, the value Sel0 before processing of the signal value selected based on the positive side swipe instruction is 0.7, and the value Sel1 after processing of the signal value selected based on the positive side swipe instruction is 0.8. The minimum signal value Black is 0, and the maximum signal value White is 1.0. When the signal value before processing is 0.7, the signal value after processing becomes 0.8. The minimum signal value Black remains 0 before and after processing. In the example shown in FIG. 19, the larger the signal value before processing, the larger the signal value after processing becomes, until the signal value after processing becomes 1.0.

[0147] As an example, FIG. 20 shows a state in which the adjusted histogram data 88 is generated by the histogram adjustment unit 132 in accordance with a positive swipe instruction. As an example, as shown in FIG. 20, the shape of the second histogram 208B is changed in accordance with the positive swipe instruction. Specifically, the positive swipe instruction changes the shape of the second histogram 208B to a higher signal value compared to before the positive swipe instruction is accepted by the acceptance device 76. The adjusted histogram data 88 generated by the histogram adjustment unit 132 includes second histogram data 85B indicating the second histogram 208B reflecting the content of the positive swipe instruction. According to the adjusted histogram data 88 shown in FIG. 20 as an example, the brightness of the second region 206B (see FIG. 17) becomes brighter compared to the histogram data 85.

[0148] Also, as shown in FIG. 21 as an example, the adjustment instruction may be an instruction to move the bin 210 in a direction in which the difference between the two signal values selected based on the adjustment instruction becomes larger than before the adjustment instruction is received by the reception device 76. Also, the adjustment instruction may be an instruction to move the bin 210 by performing a pinch-out on the touch panel 30 (see FIG. 12). Hereinafter, when it is necessary to distinguish the adjustment instruction in the mode shown in FIG. 21 from the adjustment instructions in other modes, the adjustment instruction in the mode shown in FIG. 21 is referred to as a pinch-out instruction. When the adjustment instruction is a pinch-out instruction, the signal value processing unit 131 calculates the signal value corresponding to each image pixel by Expressions (5), (6), and (7). However, Expression (6) is applied when Value1 < Black, and Expression (7) is applied when Value1 > White.

Number

Number

Number

[0149] Note that a is the slope. b is the intercept. The slope a and the intercept b are obtained by the following expressions. Sel0 is the value before processing of the smaller one of the two signal values selected based on the adjustment instruction (hereinafter referred to as the first signal value). Sel1 is the value after processing of the first signal value selected based on the adjustment instruction. Sel2 is the value before processing of the larger one of the two signal values selected based on the adjustment instruction (hereinafter referred to as the second signal value). Sel3 is the value after processing of the second signal value selected based on the adjustment instruction. However, Sel1 ≦ Sel0 ≦ Sel2 ≦ Sel3.

[0150] When Value0 ≦ Sel0, the slope a is calculated by Expression (8), and the intercept b is calculated by Expression (9).

Number

Mathematics

[0151] When Sel0 < Value0 ≤ Sel3, the slope a is calculated by Equation (10), and the intercept b is calculated by Equation (11).

Mathematics

Mathematics

[0152] When Sel3 < Value0, the slope a is calculated by Equation (12), and the intercept b is calculated by Equation (13).

Mathematics

Mathematics

[0153] As an example, FIG. 22 shows a graph illustrating the relationship between the signal value before processing and the signal value after processing when a pinch-out instruction is received by the reception device 76. In the example shown in FIG. 22, the value Sel0 before processing of the first signal value selected based on the pinch-out instruction is 0.4, and the value Sel1 after processing of the first signal value selected based on the pinch-out instruction is 0.2. The value Sel2 before processing of the second signal value selected based on the pinch-out instruction is 0.6, and the value Sel3 after processing of the second signal value selected based on the pinch-out instruction is 0.7. The minimum signal value Black is 0, and the maximum signal value White is 1.0. When the signal value before processing is 0.4, the signal value after processing becomes 0.2. When the signal value before processing is 0.6, the signal value after processing becomes 0.7. The minimum signal value Black remains 0 before and after processing. The range of signal values ​​before processing, 0.4 to 0.6, changes to 0.2 to 0.7 after processing.

[0154] As an example, FIG. 23 shows a state in which adjusted histogram data 88 is generated by histogram adjustment unit 132 in response to a pinch-out instruction. As an example, as shown in FIG. 23, the shape of second histogram 208B is changed in response to the pinch-out instruction. Specifically, the pinch-out instruction causes the shape of second histogram 208B to expand compared to before the pinch-out instruction is accepted by acceptance device 76. The adjusted histogram data 88 generated by histogram adjustment unit 132 includes second histogram data 85B indicating second histogram 208B reflecting the contents of the pinch-out instruction. According to the adjusted histogram data 88 shown in FIG. 23 as an example, the contrast of second region 206B (see FIG. 17) is enhanced compared to histogram data 85.

[0155] 21 to 23 show an example in which a pinch out instruction is received by the reception device 76, the adjustment instruction may be an instruction to move the bin 210 by pinching in on the touch panel (i.e., a pinch in instruction). In this case, the contrast of the region 206 corresponding to the adjustment instruction is weakened compared to before the pinch in instruction was received by the reception device 76.

[0156] 12 to 15 and the plus side swipe instructions shown in Fig. 18 to 20 may be sequentially accepted by the accepting device 76. The minus side swipe instructions shown in Fig. 12 to 15 and the plus side swipe instructions shown in Fig. 18 to 20 may be accepted as one adjustment instruction by the accepting device 76. In this case, a process of calculating a signal value after processing corresponding to the minus side swipe instruction and a process of calculating a signal value after processing corresponding to the plus side swipe instruction are sequentially performed.

[0157] The adjustment instruction may also be an instruction to slide the entire histogram 208 (i.e., a slide instruction). In this case, the overall brightness of the area 206 corresponding to the adjustment instruction is changed. The adjustment instruction is received by the touch panel 30, but may also be received by the hard key unit 78 or an external device (not shown) connected to the external I / F 50.

[0158] 24 as an example, reference distance change processing unit 140 performs reference distance change processing for changing reference distance data 83 based on a change instruction received by reception device 76. Reference distance change processing unit 140 has an imaging control unit 141, a distance information data acquisition unit 142, a reference distance data acquisition unit 143, an area classification data generation unit 144, a distance map image data generation unit 145, a reference distance image data generation unit 146, an area classification image data generation unit 147, a change instruction determination unit 148, a change instruction data acquisition unit 149, a reference distance data change unit 150, a reference distance image change unit 151, an area classification image change unit 152, a moving image data generation unit 153, and a moving image data output unit 154.

[0159] Imaging control unit 141 、 Distance information data acquisition section 142 and reference distance data acquisition section 143 are the same as second imaging control section 123, distance information data acquisition section 124, and reference distance data acquisition section 125 (all of the above, see FIG. 8) in image adjustment processing section 120. Also, area classification data generation section 144 is the same as area classification data generation section 126 (see FIG. 9) in image adjustment processing section 120.

[0160] 25, the distance map image data generating unit 145 generates distance map image data 90 indicating a distance map image 214 based on the distance information data 82. The distance map image 214 is an image that indicates the distribution of the subject distance with respect to the angle of view of the imaging device 10. In other words, the horizontal axis of the distance map indicated by the distance map image 214 indicates the horizontal axis of the angle of view of the imaging device 10, and the vertical axis of the distance map indicated by the distance map image 214 indicates the subject distance. The distance map image 214 is an example of a "distance map image" according to the technology of the present disclosure. The distance map image data 90 is an example of "third image data" according to the technology of the present disclosure.

[0161] The reference distance image data generation unit 146 generates reference distance image data 91 indicating a reference distance image 216 based on the reference distance data 83. The reference distance image 216 is an image representing a plurality of reference distances for classifying a plurality of regions 206. The reference distance image 216 is an example of a "reference distance image" according to the technology of the present disclosure. The reference distance image data 91 is an example of "fourth image data" according to the technology of the present disclosure.

[0162] As an example, the reference distance image 216 is an image showing a scale bar 218 and a plurality of sliders 220. The scale bar 218 shows a plurality of distance ranges 212 corresponding to the plurality of regions 206. Specifically, the scale bar 218 shows a first distance range 212A corresponding to the first region 206A, a second distance range 212B corresponding to the second region 206B, a third distance range 212C corresponding to the third region 206C, and a fourth distance range 212D corresponding to the fourth region 206D. The scale bar 218 is a single scale bar showing the plurality of distance ranges 212 collectively.

[0163] A plurality of sliders 220 are provided on the scale bar 218. The position of each slider 220 indicates a reference distance. Hereinafter, when it is necessary to distinguish between a plurality of reference distances, the reference distance for classifying the first region 206A and the second region 206B will be referred to as the first reference distance, the reference distance for classifying the second region 206B and the third region 206C will be referred to as the second reference distance, and the reference distance for classifying the third region 206C and the fourth region 206D will be referred to as the third reference distance.

[0164] In addition, in the following, when it is necessary to distinguish between the multiple sliders 220, the slider 220 corresponding to the first reference distance is referred to as the first slider 220A, the slider 220 corresponding to the second reference distance is referred to as the second slider 220B, and the slider 220 corresponding to the third reference distance is referred to as the third slider 220C. The first slider 220A indicates the first reference distance that defines the boundary between the first distance range 212A and the second distance range 212B. The second slider 220B indicates the second reference distance that defines the boundary between the second distance range 212B and the third distance range 212C. The third slider 220C indicates the third reference distance that defines the boundary between the third distance range 212C and the fourth distance range 212D.

[0165] As an example, as shown in FIG. 26, the region classification image data generating unit 147 generates region classification image data 92 indicating a region classification image 222 based on the region classification data 84. The region classification image 222 is an image in which a plurality of regions 206 are divided in different manners according to the subject distance. Examples of different manners include different colors, dots of different densities, and hatching of different forms. The region classification image 222 is an example of a "second image" and a "third image" according to the technology of the present disclosure. The region classification image data 92 is an example of "second image data" according to the technology of the present disclosure.

[0166] As an example, Fig. 27 shows a case where a change instruction has not been accepted by the accepting device 76. The change instruction is an instruction to change one of a plurality of reference distances. When the change instruction has been accepted by the accepting device 76, change instruction data 95 (see Fig. 29) is stored in the RAM 66, but when the change instruction has not been accepted by the accepting device 76, the change instruction data 95 is not stored in the RAM 66. The change instruction determination unit 148 determines whether or not the change instruction data 95 is stored in the RAM 66.

[0167] When the change instruction determination unit 148 determines that change instruction data 95 indicating a change instruction is not stored in RAM 66, the moving image data generation unit 153 generates moving image data 80 including distance map image data 90 acquired by the distance map image data generation unit 145 and reference distance image data 91 generated by the reference distance image data generation unit 146.

[0168] Moving image data output unit 154 outputs moving image data 80 generated by moving image data generation unit 153 to display 28. Display 28 displays an image based on moving image data 80. In the example shown in Fig. 27, a distance map image 214 indicated by distance map image data 90 and a reference distance image 216 indicated by reference distance image data 91 are displayed on display 28.

[0169] Also, as an example, as shown in FIG. 28, when the change instruction determination unit 148 determines that adjustment instruction data 86 indicating a change instruction is not stored in RAM 66, the video data generation unit 153 generates video data 80 including area classification image data 92 generated by the area classification image data generation unit 147.

[0170] The moving image data output unit 154 outputs the moving image data 80 generated by the moving image data generation unit 153 to the display 28. The display 28 displays an image based on the moving image data 80. In the example shown in Fig. 28, the area classification image 222 indicated by the area classification image data 92 is displayed on the display 28.

[0171] The region classification image 222 may be displayed on the display 28 together with the distance map image 214 and the reference distance image 216, or may be displayed on the display 28 in place of the distance map image 214 and the reference distance image 216.

[0172] As an example, FIG. 29 shows a case where a change instruction is received by the reception device 76. The change instruction is to change at least one of the sliders 220. too The change instruction is an instruction to slide any one of the sliders 220. As an example, the change instruction is an instruction to change the position of the slider 220 displayed on the display 28 through the touch panel 30. As an example, Fig. 29 shows an example in which the change instruction is an instruction to slide the first slider 220A. Note that the change instruction is accepted by the touch panel 30, but may also be accepted by the hard key unit 78 or an external device (not shown) connected to the external I / F 50.

[0173] When a change instruction is accepted by the accepting device 76, the reference distance change processing unit 140 stores change instruction data 95 indicating the change instruction in the RAM 66. Specifically, data indicating the slider 220 selected based on the change instruction and the slide amount of the slider 220 is stored in the RAM 66 as the change instruction data 95.

[0174] When the change instruction determination unit 148 determines that change instruction data 95 indicating a change instruction is stored in the RAM 66, the change instruction data acquisition unit 149 acquires the change instruction data 95 stored in the RAM 66. The reference distance data change unit 150 changes the reference distance data 83 in accordance with the change instruction data 95. This changes the reference distance in accordance with the change instruction. The reference distance data change unit 150 rewrites the reference distance data 83 stored in the NVM 64 with the changed reference distance data 83. This updates the reference distance data 83 stored in the NVM 64. The change instruction is an example of a "second instruction" and a "third instruction" according to the technology of the present disclosure.

[0175] As an example, as shown in FIG. 30, reference distance image change unit 151 performs a process of reflecting the contents of the change instruction in reference distance image 216. Specifically, changed reference distance image data 93 is generated in which the contents of the change instruction are reflected in reference distance image 216. The changed reference distance image data 93 is data indicating reference distance image 216 in which the reference distance has been changed in accordance with the change instruction. As an example, FIG. 30 shows an example in which the change instruction is an instruction to slide first slider 220A toward second distance range 212B. By sliding first slider 220A toward second distance range 212B, the first reference distance is changed. The process of reflecting the contents of the change instruction in reference distance image 216 is an example of a "fourth process" according to the technology of the present disclosure.

[0176] As an example, as shown in FIG. 31, the region classification image change unit 152 performs a process of reflecting the contents of the change instruction in the region classification image 222. Specifically, the changed region classification image data 94 is generated by reflecting the contents of the change instruction in the region classification image 222. The changed region classification image data 94 is data showing the region classification image 222 in which the widths of adjacent regions among the multiple regions 206 have been changed in accordance with the change instruction. As an example, FIG. 31 shows a state in which the first region 206A is enlarged compared to before the change instruction is accepted by the acceptance device 76, in response to the change instruction being an instruction to slide the first slider 220A toward the second distance range 212B (see FIG. 30). The changed region classification image data 94 is an example of the "fifth image data" according to the technology of the present disclosure. The process of reflecting the contents of the change instruction in the region classification image 222 is an example of the "fourth process" according to the technology of the present disclosure.

[0177] As an example, FIG. 32 shows how the reference distance image 216 changes when transitioning from a state before the acceptance device 76 accepts a change instruction to a state after the acceptance device 76 accepts the change instruction.

[0178] As an example, as shown in FIG. 32, when a change instruction is not accepted by the reception device 76, the video data generation unit 153 generates video data 80 including distance map image data 90 and reference distance image data 91, but when a change instruction is accepted by the reception device 76, it generates video data 80 including distance map image data 90 and changed reference distance image data 93.

[0179] The moving image data output unit 154 outputs the moving image data 80 generated by the moving image data generation unit 153 to the display 28. The display 28 displays an image based on the moving image data 80. In the example shown in Fig. 32, the position of the first slider 220A is changed based on a change instruction.

[0180] As an example, FIG. 33 shows how the area classification image 222 changes when transitioning from a state before the reception device 76 receives a change instruction to a state after the reception device 76 receives a change instruction.

[0181] If a change instruction is not accepted by the reception device 76, the video data generation unit 153 generates video data 80 including area classification image data 92, but if a change instruction is accepted by the reception device 76, it generates video data 80 including changed area classification image data 94.

[0182] The moving image data output unit 154 outputs the moving image data 80 generated by the moving image data generation unit 153 to the display 28. The display 28 displays an image based on the moving image data 80. In the example shown in Fig. 33, the first area 206A is enlarged compared to before the change instruction was accepted by the acceptance device 76.

[0183] Next, the operation of the imaging device 10 according to this embodiment will be described with reference to FIGS.

[0184] First, an example of the flow of an operation mode setting process performed by the CPU 62 will be described with reference to FIG.

[0185] 34, first, in step ST10, the imaging mode setting unit 101 sets an imaging mode as an initial setting of the operation mode of the imaging device 10. After the process of step ST10 is executed, the operation mode setting process proceeds to step ST11.

[0186] In step ST11, the first mode switching determination unit 102 determines whether or not a first mode switching condition for switching the operation mode of the imaging device 10 from the imaging mode to the image adjustment mode is satisfied. An example of the first mode switching condition is, for example, a condition in which a first mode switching instruction for switching the operation mode of the imaging device 10 from the imaging mode to the image adjustment mode is accepted by the acceptance device 76. In step ST11, if the first mode switching condition is satisfied, the determination is affirmative, and the operation mode setting process proceeds to step ST12. In step ST11, if the first mode switching condition is not satisfied, the determination is negative, and the operation mode setting process proceeds to step ST13.

[0187] In step ST12, the image adjustment mode setting section 103 sets the image adjustment mode as the operation mode of the imaging device 10. After the process of step ST12 is executed, the operation mode setting process proceeds to step ST13.

[0188] In step ST13, the second mode switching determination unit 104 determines whether or not a second mode switching condition for switching the operation mode of the imaging device 10 from the imaging mode or the image adjustment mode to the reference distance change mode is satisfied. An example of the second mode switching condition is, for example, a condition in which a second mode switching instruction for switching the operation mode of the imaging device 10 from the imaging mode or the image adjustment mode to the reference distance change mode is accepted by the acceptance device 76. In step ST13, if the second mode switching condition is satisfied, the determination is affirmative, and the operation mode setting process proceeds to step ST14. In step ST13, if the second mode switching condition is not satisfied, the determination is negative, and the operation mode setting process proceeds to step ST15.

[0189] In step ST14, reference distance change mode setting section 105 sets the reference distance change mode as the operation mode of imaging device 10. After the process of step ST14 is executed, the operation mode setting process proceeds to step ST15.

[0190] In step ST15, the third mode switching determination unit 106 determines whether the operation mode of the imaging device 10 is in the image adjustment mode or the reference distance change mode. In step ST15, if the operation mode of the imaging device 10 is in the image adjustment mode or the reference distance change mode, the determination is affirmative, and the operation mode setting process proceeds to step ST16. In step ST15, if the operation mode of the imaging device 10 is not in the image adjustment mode or the reference distance change mode, the determination is negative, and the operation mode setting process proceeds to step ST17.

[0191] In step ST16, the third mode switching determination unit 106 determines whether or not a third mode switching condition for switching the operation mode of the imaging device 10 from the image adjustment mode or the reference distance change mode to the imaging mode is satisfied. An example of the third mode switching condition is, for example, a condition in which a third mode switching instruction for switching the operation mode of the imaging device 10 from the image adjustment mode or the reference distance change mode to the imaging mode is accepted by the acceptance device 76. In step ST16, if the third mode switching condition is satisfied, the determination is affirmative, and the operation mode setting process proceeds to step ST10. In step ST16, if the third mode switching condition is not satisfied, the determination is negative, and the operation mode setting process proceeds to step ST17.

[0192] In step ST17, the CPU 62 determines whether or not a condition for terminating the operation mode setting process is satisfied. One example of a condition for terminating the operation mode setting process is that an end instruction (e.g., an instruction to turn off the power supply to the imaging device 10) is accepted by the acceptance device 76. If the condition for terminating the operation mode setting process is not satisfied in step ST17, the judgment is denied, and the operation mode setting process proceeds to step ST11. Step ST1 7 If the condition for terminating the operation mode setting process is met, the determination is affirmative and the operation mode setting process is terminated.

[0193] Next, an example of the flow of imaging processing performed by the CPU 62 will be described with reference to FIG.

[0194] 35, first, in step ST20, the imaging control unit 111 controls the photoelectric conversion element 72 to output the non-phase difference pixel data 73A. After the process of step ST20 is performed, the imaging process proceeds to step ST21.

[0195] In step ST21, the image data acquisition unit 112 acquires image data 81 generated by digitizing the non-phase difference pixel data 73A by the signal processing circuit 74. After the processing of step ST21 is executed, the imaging processing proceeds to step ST22.

[0196] In step ST22, the moving image data generating section 113 generates moving image data 80 based on the image data 81 acquired by the image data acquiring section 112. After the process of step ST22 is executed, the imaging process proceeds to step ST23.

[0197] In step ST23, the moving image data output unit 114 outputs the moving image data 80 generated by the moving image data generation unit 113 to the display 28. After the process of step ST23 is executed, the imaging process proceeds to step ST24.

[0198] In step ST24, the CPU 62 determines whether or not a condition for ending the imaging process is satisfied. An example of a condition for ending the imaging process is that a first mode switching instruction or a second mode switching instruction is accepted by the acceptance device 76. If the condition for ending the imaging process is not satisfied in step ST24, the determination is negative, and the imaging process proceeds to step ST20. If the condition for ending the imaging process is satisfied in step ST24, the determination is positive, and the imaging process ends.

[0199] Next, an example of the flow of image adjustment processing performed by the CPU 62 will be described with reference to FIG.

[0200] 36, first, in step ST30, the first imaging control unit 121 controls the photoelectric conversion element 72 to output the non-phase difference pixel data 73A. After the process of step ST30 is performed, the image adjustment process proceeds to step ST31.

[0201] In step ST31, the image data acquisition unit 122 acquires image data 81 generated by digitizing the non-phase difference pixel data 73A by the signal processing circuit 74. After the process of step ST31 is executed, the image adjustment process proceeds to step ST32.

[0202] In step ST32, the second imaging control unit 123 controls the photoelectric conversion element 72 to output the phase difference pixel data 73B. After the process of step ST32 is executed, the image adjustment process proceeds to step ST33.

[0203] In step ST33, the distance information data acquisition unit 124 acquires the distance information data 82 based on the phase difference pixel data 73B acquired from the signal processing circuit 74. After the process of step ST33 is executed, the image adjustment process proceeds to step ST34.

[0204] In step ST34, the reference distance data acquisition unit 125 acquires the reference distance data 83 pre-stored in the NVM 64. After the process of step ST34 is executed, the image adjustment process proceeds to step ST35.

[0205] In step ST35, the area classification data generating unit 126 generates area classification data 84 for classifying the image 200 into a plurality of areas 206 according to the subject distance, based on the distance information data 82 and the reference distance data 83. After the process of step ST35 is executed, the image adjustment process proceeds to step ST36.

[0206] In step ST36, the histogram data generating unit 127 generates histogram data 85 corresponding to each region 206 based on the image data 81 and the region classification data 84. After the process of step ST36 is executed, the image adjustment process proceeds to step ST37.

[0207] In step ST37, adjustment instruction determination unit 128 determines whether or not adjustment instruction data 86 is stored in RAM 66. If adjustment instruction data 86 is not stored in RAM 66 in step ST37, the determination is negative, and the image adjustment process proceeds to step ST43A. If adjustment instruction data 86 is stored in RAM 66 in step ST37, the determination is positive, and the image adjustment process proceeds to step ST38.

[0208] In step ST38, the adjustment instruction data acquisition section 129 acquires the adjustment instruction data 86 stored in the RAM 66. After the process of step ST38 is executed, the image adjustment process proceeds to step ST39.

[0209] In step ST39, the processing strength setting unit 130 sets a processing strength corresponding to each image pixel based on the processing strength data 87 stored in the NVM 64. After the processing of step ST39 is executed, the image adjustment processing proceeds to step ST40.

[0210] In step ST40, the signal value processing unit 131 calculates an adjusted signal value for each image pixel based on the processing intensity set by the processing intensity setting unit 130. After the processing of step ST40 is executed, the image adjustment processing proceeds to step ST41.

[0211] In step ST41, the histogram adjustment unit 132 generates adjusted histogram data 88 in which the content of the adjustment instruction is reflected in at least one of the multiple histograms 208, based on the signal value calculated by the signal value processing unit 131. After the process of step ST40 is executed, the image adjustment process proceeds to step ST42.

[0212] In step ST42, the image adjustment unit 133 generates adjusted image data 89 in which the content of the adjustment instruction is reflected in at least one of the multiple regions 206, based on the signal value calculated by the signal value processing unit 131. After the process of step ST42 is executed, the image adjustment process proceeds to step ST43B.

[0213] In step ST43A, the moving image data generation unit 134 generates moving image data 80 including image data 81 and histogram data 85. After the process of step ST43A is executed, the image adjustment process proceeds to step ST44.

[0214] In step ST43B, the moving image data generation unit 134 generates moving image data 80 including adjusted image data 89 and adjusted histogram data 88. After the process of step ST43B is executed, the image adjustment process proceeds to step ST44.

[0215] In step ST44, the moving image data output unit 135 outputs the moving image data 80 generated by the moving image data generation unit 134 to the display 28. After the process of step ST44 is executed, the image adjustment process proceeds to step ST45.

[0216] In step ST45, the CPU 62 determines whether or not a condition for terminating the image adjustment process is satisfied. One example of a condition for terminating the image adjustment process is that a second mode switching instruction or a third mode switching instruction is accepted by the acceptance device 76. If the condition for terminating the image adjustment process is not satisfied in step ST45, the judgment is negative, and the image adjustment process proceeds to step ST30. If the condition for terminating the image adjustment process is satisfied in step ST45, the judgment is positive, and the image adjustment process ends.

[0217] Next, an example of the flow of a reference distance change process performed by the CPU 62 of the imaging device 10 will be described with reference to FIG.

[0218] 37, first, in step ST50, the imaging control unit 141 controls the photoelectric conversion element 72 to output the phase difference pixel data 73B. After the process of step ST50 is executed, the reference distance change process proceeds to step ST51.

[0219] In step ST51, the distance information data acquisition unit 142 acquires the distance information data 82 based on the phase difference pixel data 73B acquired from the signal processing circuit 74. After the process of step ST51 is executed, the reference distance change process proceeds to step ST52.

[0220] In step ST52, the reference distance data acquisition unit 143 acquires the reference distance data 83 pre-stored in the NVM 64. After the process of step ST52 is executed, the reference distance change process proceeds to step ST53.

[0221] In step ST53, the area classification data generating unit 144 generates area classification data 84 for classifying the image 200 into a plurality of areas 206 according to the subject distance, based on the distance information data 82 and the reference distance data 83. After the process of step ST53 is executed, the reference distance changing process is 5 Move to 4.

[0222] In step ST54, the distance map image data generating unit 145 generates distance map image data 90 indicating the distance map image 214, based on the distance information data 82. After the process of step ST54 is executed, the reference distance changing process proceeds to step ST55.

[0223] In step ST55, the reference distance image data generating unit 146 generates reference distance image data 91 indicating the reference distance image 216 based on the reference distance data 83. After the process of step ST55 is executed, the reference distance changing process proceeds to step ST56.

[0224] In step ST56, the area classification image data generating unit 147 teeth Based on the area classification data 84, area classification image data 92 indicating the area classification image 222 is generated. After the process of step ST56 is executed, the reference distance change process proceeds to step ST57.

[0225] In step ST57, the change instruction determination unit 148 determines whether or not the change instruction data 95 is stored in the RAM 66. If the change instruction data 95 is not stored in the RAM 66 in step ST57, the determination is negative, and the reference distance change processing proceeds to step ST62A. If the change instruction data 95 is stored in the RAM 66 in step ST57, the determination is positive, and the reference distance change processing proceeds to step ST58.

[0226] In step ST58, the change instruction data acquisition unit 149 acquires the change instruction data 95 stored in the RAM 66. After the process of step ST58 is executed, the reference distance change process proceeds to step ST59.

[0227] In step ST59, the reference distance data change section 150 changes the reference distance data 83 in accordance with the change instruction data 95. After the process of step ST59 is executed, the reference distance change process proceeds to step ST60.

[0228] In step ST60, the reference distance image modification section 151 generates modified reference distance image data 93 by reflecting the contents of the modification instruction in the reference distance image 216. After the process of step ST60 is executed, the reference distance modification process proceeds to step ST61.

[0229] In step ST61, the area classification image change section 152 generates changed area classification image data 94 by reflecting the contents of the change instruction in the area classification image 222. After the process of step ST61 is executed, the reference distance change process proceeds to step ST62B.

[0230] In step ST62A, the video data generation unit 153 generates video data 80 including the reference distance image data 91 and area classification image data 92. After the process of step ST62A is executed, the reference distance change process proceeds to step ST63.

[0231] In step ST62B, the video data generation unit 153 generates video data 80 including the changed reference distance image data 93 and the changed area classification image data 94. After the process of step ST62B is executed, the reference distance change process proceeds to step ST63.

[0232] In step ST63, the moving image data output unit 154 outputs the moving image data 80 generated by the moving image data generation unit 153 to the display 28. After the process of step ST63 is executed, the reference distance change process proceeds to step ST64.

[0233] In step ST64, the CPU 62 determines whether or not a condition for terminating the reference distance change process is satisfied. One example of a condition for terminating the reference distance change process is that a first mode switching instruction or a third mode switching instruction is accepted by the acceptance device 76. If the condition for terminating the reference distance change process is not satisfied in step ST64, the determination is negative, and the reference distance change process proceeds to step ST50. If the condition for terminating the reference distance change process is satisfied in step ST64, the determination is positive, and the reference distance change process ends.

[0234] The control method described above as the operation of the imaging device 10 is an example of an "image processing method" according to the technique of the present disclosure.

[0235] As described above, in the imaging device 10 according to this embodiment, the CPU 62 acquires distance information data 82 relating to the subject distance corresponding to each photosensitive pixel 72B. The CPU 62 outputs image data 81 indicating the image 200 obtained by imaging by the image sensor 20. The CPU 62 also classifies the image 200 into a plurality of regions 206 according to the distance based on the distance information data 82, and outputs histogram data 85 indicating a histogram 208 created based on a signal of the image data 81 for at least one of the plurality of regions 206. Then, when an adjustment instruction relating to the histogram 208 is accepted by the accepting device 76, the CPU 62 performs a process of reflecting the content of the adjustment instruction in the image 200 and the histogram 208. Therefore, the aspects of the image 200 and the histogram 208 can be changed according to the adjustment instruction accepted by the accepting device 76. For example, the user can adjust the intensity of the haze 204 depicted as an image in the image 200 according to his intention.

[0236] Moreover, the CPU 62 outputs the histogram data 85 indicating the histogram 208. Therefore, the user can obtain the luminance information of the region 206 corresponding to the histogram 208 based on the histogram 208.

[0237] Moreover, the histogram 208 is a histogram showing the relationship between the signal value and the number of pixels. Therefore, the user can understand the relationship between the signal value and the number of pixels based on the histogram 208.

[0238] Furthermore, the CPU 62 outputs histogram data 85 indicating the histogram 208 created based on the signal value for each region 206. The process of reflecting the contents of the adjustment instruction in the histogram 208 includes a process of prohibiting the contents of the adjustment instruction from being reflected in other histograms 208 different from the histogram 208 corresponding to the adjustment instruction. Therefore, it is possible to avoid changing the form of other histograms 208 different from the histogram 208 corresponding to the adjustment instruction. Furthermore, the process of reflecting the contents of the adjustment instruction in the image 200 includes a process of prohibiting the contents of the adjustment instruction from being reflected in other regions 206 different from the region 206 corresponding to the adjustment instruction. Therefore, it is possible to avoid changing the form of other regions 206 different from the region 206 corresponding to the adjustment instruction.

[0239] The process of reflecting the contents of the adjustment instruction in the image 200 and the histogram 208 is This is a process for changing the signal value according to the content of the instruction. Therefore, the aspects of the image 200 and the histogram 208 can be changed according to the content of the adjustment instruction accepted by the acceptance device 76.

[0240] The adjustment instruction is an instruction to change the form of the histogram 208. Therefore, the user can change the appearance of the image 200 and the histogram 208 by giving an instruction to change the form of the histogram 208 as an adjustment instruction to the reception device 76.

[0241] Moreover, the histogram 208 has a plurality of bins 210, and the adjustment instruction is an instruction to move a bin 210 corresponding to a signal value selected based on the adjustment instruction from among the plurality of bins 210. Therefore, by moving the bin 210, the form of the histogram 208 can be changed.

[0242] Furthermore, the CPU 62 outputs region classification image data 92 showing a region classification image 222 in which the multiple regions 206 are divided in different ways according to the subject distance. Therefore, the user can grasp the multiple regions 206 based on the region classification image 222.

[0243] Furthermore, CPU 62 outputs distance map image data 90 indicating distance map image 214 expressing the distribution of subject distances relative to the angle of view of imaging device 10. Therefore, the user can grasp the distribution of subject distances relative to the angle of view of imaging device 10 based on distance map image 214.

[0244] Furthermore, the CPU 62 outputs reference distance image data 91 showing a reference distance image 216 that indicates a reference distance for classifying the multiple regions 206. Therefore, the user can grasp the reference distance based on the reference distance image 216.

[0245] Moreover, the reference distance image 216 is an image showing a scale bar 218 and a slider 220. The scale bar 218 shows a plurality of distance ranges 212 corresponding to a plurality of regions 206, and the slider 220 is provided on the scale bar 218. The position of the slider 220 indicates the reference distance. Therefore, the user can change the position of the slider 220 to change the reference distance. Moreover, the user can grasp the reference distance based on the position of the slider 220.

[0246] Moreover, the scale bar 218 is a single scale bar that collectively indicates the multiple distance ranges 212. Therefore, the user can adjust the multiple distance ranges 212 based on the single scale bar.

[0247] Furthermore, when a change instruction is accepted by the acceptance device 76, the CPU 62 outputs the area classification image data 92 indicating the area classification image 222. Therefore, the user can confirm the contents of the change instruction based on the area classification image 222.

[0248] Furthermore, when a change instruction is accepted by the acceptance device 76, the CPU 62 performs a process of reflecting the content of the change instruction in the reference distance image 216. Therefore, the user can confirm the content of the change instruction based on the reference distance image 216.

[0249] Furthermore, when a change instruction is received by the reception device 76, the CPU 62 changes the reference distance in accordance with the contents of the change instruction. Therefore, the multiple areas 206 classified based on the reference distance can be changed based on the change instruction.

[0250] Moreover, the image data 81 output by the CPU 62 is included in the moving image data 80. Therefore, the contents of the adjustment instructions can be reflected in the image 200 (that is, the moving image) displayed on the display 28 based on the moving image data 80.

[0251] The imaging device 10 also includes an image sensor 20 and a display 28. Therefore, the user can check, on the display 28, an image 200 obtained by imaging using the image sensor 20.

[0252] Furthermore, the CPU 62 outputs the image data 81 and the histogram data 85 to the display 28. Therefore, the image 200 and the histogram 208 can be displayed on the display 28.

[0253] Furthermore, the CPU 62 performs processing to change the display mode of the image 200 and the histogram 208 displayed on the display 28. Therefore, the user can give an adjustment instruction to the reception device 76 while checking the change in the display mode of the image 200 and the histogram 208.

[0254] Furthermore, the photoelectric conversion element 72 included in the image sensor 20 has a plurality of photosensitive pixels 72B, and the CPU 62 acquires distance information data 82 based on the phase difference pixel data 73B output from the photosensitive pixels 72B. Therefore, a distance sensor other than the image sensor 20 can be eliminated.

[0255] Further, the photosensitive pixel 72B is a pixel that selectively outputs non-phase difference pixel data 73A and phase difference pixel data 73B. The non-phase difference pixel data 73A is pixel data obtained by performing photoelectric conversion by the entire region of the photosensitive pixel 72B, and the phase difference pixel data 73B is pixel data obtained by performing photoelectric conversion by a partial region of the photosensitive pixel 72B. Therefore, image data 81 and distance information data 82 can be obtained from the imaging data 73.

[0256] In the above embodiment, the CPU 62 reflects the contents of the adjustment instruction in the image 200 and the histogram 208. However, the adjustment instruction may be reflected in only one of the image 200 and the histogram 208.

[0257] In addition, the CPU 62 may perform processing to prohibit the contents of the adjustment instruction from being reflected in areas 206 other than the area 206 corresponding to the adjustment instruction, and may perform processing to reflect the contents of the adjustment instruction in histograms 208 other than the histogram 208 corresponding to the adjustment instruction.

[0258] In addition, the CPU 62 may perform processing to prohibit the contents of the adjustment instruction from being reflected in histograms 208 other than the histogram 208 corresponding to the adjustment instruction, and may perform processing to reflect the contents of the adjustment instruction in areas 206 other than the area 206 corresponding to the adjustment instruction.

[0259] Furthermore, the CPU 62 may output only one of the image data 81 and the histogram data 85 to the display .

[0260] Furthermore, the CPU 62 may change the display mode of only one of the image 200 and the histogram 208 displayed on the display 28 based on an adjustment instruction.

[0261] In addition, in the image adjustment process, the CPU 62 However, still image data including adjusted image data 89 and adjusted histogram data 88 may also be output.

[0262] In addition, in the reference distance change process, the CPU 62 outputs moving image data 80 including distance map image data 90 and changed reference distance image data 93, but may also output still image data including distance map image data 90 and changed reference distance image data 93.

[0263] Furthermore, the imaging device 10 includes a display 28, and the CPU 62 outputs the moving image data 80 to the display 28. However, the CPU 62 may output the moving image data 80 to a display (not shown) provided outside the imaging device 10.

[0264] In addition, the CPU 62 performs a process of reflecting the contents of the adjustment instructions accepted by the acceptance device 76 in the image 200 and the histogram 208, but may also perform a process of deriving adjustment instructions based on the image data 81 and the distance information data 82, and a process of reflecting the contents of the derived adjustment instructions in the image 200 and the histogram 208.

[0265] Furthermore, the CPU 62 outputs the histogram data 85 indicating the histogram 208 corresponding to each region 206, but may output the histogram data 85 indicating only the histogram 208 corresponding to any one of the multiple regions 206.

[0266] In addition, when a change instruction is not accepted by the reception device 76, the CPU 62 outputs moving image data 80 including distance map image data 90 and reference distance image data 91, but the moving image data 80 may also be data including only one of the distance map image data 90 and the reference distance image data 91.

[0267] Furthermore, in the reference distance change process, CPU 62 outputs moving image data 80 including area classification image data 92, but moving image data 80 does not have to include area classification image data 92. In this case, a distance map image 214 and a reference distance image 216 may be displayed on display 28 based on distance map image data 90 and reference distance image data 91 included in moving image data 80.

[0268] Furthermore, in the reference distance change process, CPU 62 may output moving image data 80 including image data 81. In this case, image 200, distance map image 214, and reference distance image 216 may be displayed on display 28 based on image data 81, distance map image data 90, and reference distance image data 91 included in moving image data 80.

[0269] Furthermore, in the reference distance change process, CPU 62 may output moving image data 80 including image data 81 and area classification image data 92. In this case, image 200, area classification image 222, distance map image 214, and reference distance image 216 may be displayed on display 28 based on image data 81, area classification image data 92, distance map image data 90, and reference distance image data 91 included in moving image data 80.

[0270] Furthermore, the region classification image 222 may be incorporated into a part of the image 200 by a PinP function, or may be superimposed on the image 200. Furthermore, the region classification image 222 may be superimposed on the image 200 by alpha blending. Furthermore, the region classification image 222 may be switched with the image 200.

[0271] In the above embodiment, the distance information data is acquired by the photoelectric conversion element 72 of the phase difference type, but the method is not limited to the phase difference type, and the distance information data may be acquired by using a photoelectric conversion element of the TOF type, or may be acquired by using a stereo camera or a depth sensor. As a method of acquiring distance information data using a photoelectric conversion element of the TOF type, for example, a method using LiDAR can be mentioned. information The data may be acquired in accordance with the frame rate of the image sensor 20, or may be acquired at time intervals longer or shorter than the time interval defined by the frame rate of the image sensor 20.

[0272] Furthermore, the CPU 62 performs processing to reflect the contents of the adjustment instruction accepted by the acceptance device 76 in the region 206 and the histogram 208 corresponding to the adjustment instruction, and to prohibit the contents of the adjustment instruction from being reflected in the region 206 other than the region 206 corresponding to the adjustment instruction and the histogram 208 other than the histogram 208 corresponding to the adjustment instruction. However, the CPU 62 may also perform processing to reflect the contents of the adjustment instruction accepted by the acceptance device 76 in the region 206 other than the region 206 corresponding to the adjustment instruction and the histogram 208 other than the histogram 208 corresponding to the adjustment instruction. In this case, the state of the region 206 other than the region 206 corresponding to the adjustment instruction and the state of the histogram 208 other than the histogram 208 corresponding to the adjustment instruction can also be changed based on the adjustment instruction accepted by the acceptance device 76.

[0273] The region 206 other than the region 206 corresponding to the adjustment instruction is an example of a "third region" according to the technology of the present disclosure. The histogram 208 other than the histogram 208 corresponding to the adjustment instruction is an example of "third luminance information" according to the technology of the present disclosure. The histogram data 85 indicating the histogram 208 other than the histogram 208 corresponding to the adjustment instruction is an example of "third luminance information data" according to the technology of the present disclosure. The process of reflecting the content of the adjustment instruction accepted by the accepting device 76 in the region 206 other than the region 206 corresponding to the adjustment instruction and the histogram 208 other than the histogram 208 corresponding to the adjustment instruction is an example of a "third process" according to the technology of the present disclosure.

[0274] In addition, when performing processing to reflect the contents of the adjustment instructions accepted by the acceptance device 76 in areas 206 other than the area 206 corresponding to the adjustment instructions and in histograms 208 other than the histograms 208 corresponding to the adjustment instructions, the CPU 62 may set processing strengths corresponding to the multiple distance ranges 212 as follows.

[0275] As an example, Fig. 38 shows a first modified example of processing intensities corresponding to a plurality of distance ranges 212. As an example, processing intensity data 87 shown in Fig. 38 shows data for changing the form of second histogram 208B based on an adjustment instruction. As an example, in the processing intensity data 87 shown in Fig. 38, the processing intensities corresponding to a plurality of distance ranges 212 differ depending on the subject distance.

[0276] As an example, in the processing intensity data 87 shown in FIG. 38, the second processing intensity is set to a constant value corresponding to the reference intensity. That is, for each image pixel corresponding to the second distance range 212B, the change amount of the signal value changed based on the adjustment instruction is constant. Also, in the example shown in FIG. 38, the first processing intensity, the third processing intensity, and the fourth processing intensity are set to constant values ​​equal to or greater than 0. That is, for each image pixel corresponding to the first distance range 212A, the change amount of the signal value changed based on the adjustment instruction is constant. Similarly, for each image pixel corresponding to the third distance range 212C, the change amount of the signal value changed based on the adjustment instruction is constant, and for each image pixel corresponding to the fourth distance range 212D, the change amount of the signal value changed based on the adjustment instruction is constant.

[0277] Furthermore, the processing intensities are different for the multiple distance ranges 212. Specifically, the first processing intensity is set lower than the second processing intensity. The third processing intensity is set higher than the second processing intensity. The fourth processing intensity is set higher than the third processing intensity. That is, the processing intensities corresponding to the multiple distance ranges 212 are set to increase as the subject distance value of each distance range 212 increases.

[0278] Then, the processing intensity setting unit 130 may set processing intensities corresponding to the multiple distance ranges 212 based on the processing intensity data 87. Furthermore, the signal value processing unit 131 may calculate a post-processing signal value for each image pixel based on the processing intensity set by the processing intensity setting unit 130.

[0279] In the example shown in Fig. 38, the signal values ​​corresponding to each distance range 212 can be changed according to the contents of the adjustment instruction. As a result, as shown in Fig. 39 as an example, when an adjustment instruction to change the shape of the second histogram 208B is accepted by the accepting device 76, the shape of the first histogram 208A can be changed based on the first processing intensity. Similarly, the shape of the third histogram 208C can be changed based on the third processing intensity, and the shape of the fourth histogram 208D can be changed according to the fourth processing intensity.

[0280] As an example, Fig. 40 shows a second modified example of processing intensities corresponding to the plurality of distance ranges 212. As an example, processing intensity data 87 shown in Fig. 40 shows data in the case where the form of second histogram 208B is changed based on an adjustment instruction. As an example, in the processing intensity data 87 shown in Fig. 40, the processing intensities corresponding to the plurality of distance ranges 212 differ depending on the subject distance. Specifically, the processing intensities corresponding to the plurality of distance ranges 212 are set to be higher as the subject distance becomes longer.

[0281] The reference intensity is set based on the representative distance, which may be the average value of the distance range 212 corresponding to the adjustment instruction, the average value of the subject distance in the distance range 212 corresponding to the adjustment instruction, or the median value of the subject distance in the distance range 212 corresponding to the adjustment instruction.

[0282] Then, the processing intensity setting unit 130 may set processing intensities corresponding to the multiple distance ranges 212 based on the processing intensity data 87. Furthermore, the signal value processing unit 131 may calculate a post-processing signal value for each image pixel based on the processing intensity set by the processing intensity setting unit 130.

[0283] 40, the signal values ​​corresponding to each distance range 212 can be changed according to the contents of the adjustment instruction. As a result, as shown in FIG 39 as an example, when an adjustment instruction to change the shape of the second histogram 208B is received by the reception device 76, the shape of the first histogram 208A can be changed based on the first processing intensity. Similarly, the shape of the third histogram 208C can be changed based on the third processing intensity, and the shape of the fourth histogram 208D can be changed according to the fourth processing intensity.

[0284] As an example, Fig. 41 shows a third modified example of processing intensities corresponding to a plurality of distance ranges 212. As an example, processing intensity data 87 shown in Fig. 41 shows data for changing the form of second histogram 208B based on an adjustment instruction. As an example, in the processing intensity data 87 shown in Fig. 41, the processing intensities corresponding to a plurality of distance ranges 212 differ depending on the subject distance.

[0285] As an example, in the processing strength data 87 shown in FIG. 41, the second processing strength is set to a constant value corresponding to the reference strength. That is, for each image pixel corresponding to the second distance range 212B, the change amount of the signal value changed based on the adjustment instruction is constant. 41 In the example shown in FIG. 1, the first processing intensity, the third processing intensity, and the fourth processing intensity differ depending on the subject distance. Specifically, the first processing intensity, the third processing intensity, and the fourth processing intensity are set to increase as the subject distance increases.

[0286] Then, the processing intensity setting unit 130 may set processing intensities corresponding to the multiple distance ranges 212 based on the processing intensity data 87. Furthermore, the signal value processing unit 131 may calculate a post-processing signal value for each image pixel based on the processing intensity set by the processing intensity setting unit 130.

[0287] 41, the signal values ​​corresponding to each distance range 212 can also be changed according to the contents of the adjustment instruction. As a result, as shown in FIG 39 as an example, when an adjustment instruction to change the shape of the second histogram 208B is received by the reception device 76, the shape of the first histogram 208A can be changed based on the first processing intensity. Similarly, the shape of the third histogram 208C can be changed based on the third processing intensity, and the shape of the fourth histogram 208D can be changed in accordance with the fourth processing intensity.

[0288] In the examples shown in FIGS. 38 to 41, the signal corresponding to the second distance range 212B is an example of a "first signal" according to the technology of the present disclosure. The signals corresponding to the first distance range 212A, the third distance range 212C, and the fourth distance range 212D are an example of a "third signal" according to the technology of the present disclosure. The signal value corresponding to the second distance range 212B is an example of a "first signal value included in the first signal" according to the technology of the present disclosure. The signal values ​​corresponding to the first distance range 212A, the third distance range 212C, and the fourth distance range 212D are an example of a "second signal value included in the third signal" according to the technology of the present disclosure.

[0289] In the examples shown in FIGS. 38 to 41, the photosensitive pixels 72B corresponding to the second distance range 212B are an example of a "first pixel" according to the technology of the present disclosure. The photosensitive pixels 72B corresponding to the first distance range 212A, the third distance range 212C, and the fourth distance range 212D are an example of a "second pixel" according to the technology of the present disclosure. The second distance range 212B is an example of a "first distance range" according to the technology of the present disclosure. The first distance range 212A, the third distance range 212C, and the fourth distance range 212D are an example of a "second distance range" according to the technology of the present disclosure. The second region 206B corresponding to the second distance range 212B is an example of a "first region" according to the technology of the present disclosure. The first region 206A corresponding to the first distance range 212A, the third region 206C corresponding to the third distance range 212C, and the fourth region 206D corresponding to the fourth distance range 212D are examples of a "third region" according to the technology of the present disclosure.

[0290] The processing strength may be set to the same across all of the multiple distance ranges 212 .

[0291] 42 shows an example of a modified example of the reference distance image 216. The reference distance image 216 may show a plurality of scale bars 218 each indicating a plurality of distance ranges 212. Furthermore, each scale bar 218 is provided with two sliders 220.

[0292] Hereinafter, when it is necessary to distinguish between the multiple scale bars 218, the multiple scale bars 218 will be referred to as a first scale bar 218A, a second scale bar 218B, a third scale bar 218C, and a fourth scale bar 218D. Also, when it is necessary to distinguish between the multiple sliders 220, the multiple sliders 220 will be referred to as a first upper limit slider 220A1, a first lower limit slider 220A2, a second upper limit slider 220B1, a second lower limit slider 220B2, a third upper limit slider 220C1, a third lower limit slider 220C2, a fourth upper limit slider 220D1, and a fourth lower limit slider 220D2.

[0293] The first scale bar 218A indicates the first distance range 212A. A scale bar 218B indicates the second distance range 212B. A third scale bar 218C indicates the third distance range 212C. A fourth scale bar 218D indicates the fourth distance range 212D.

[0294] The first upper limit slider 220A1 is provided on the first scale bar 218A and indicates a reference distance that defines the upper limit of the first distance range 212A. The first lower limit slider 220A2 is provided on the first scale bar 218A and indicates a reference distance that defines the lower limit of the first distance range 212A. The second upper limit slider 220B1 is provided on the second scale bar 218B and indicates a reference distance that defines the upper limit of the second distance range 212B. The second lower limit slider 220B2 is provided on the second scale bar 218B and indicates a reference distance that defines the lower limit of the second distance range 212B. The third upper limit slider 220C1 is provided on the third scale bar 218C and indicates a reference distance that defines the upper limit of the third distance range 212C. The third lower limit slider 220C2 is provided on the third scale bar 218C and indicates a reference distance that defines the lower limit of the third distance range 212C. The fourth upper limit slider 220D1 is provided on the fourth scale bar 218D and indicates a reference distance that defines the upper limit of the fourth distance range 212D. The fourth lower limit slider 220D2 is provided on the fourth scale bar 218D and indicates a reference distance that defines the lower limit of the fourth distance range 212D.

[0295] 42, a plurality of distance ranges 212 can be set independently. When adjacent distance ranges 212 partially overlap, the regions 206 corresponding to the adjacent distance ranges 212 are displayed in a mixed manner in the region classification image 222 (see FIG. 33).

[0296] In the above embodiment, the histogram 208 is displayed on the display 28 (see FIG. 17) as brightness information indicating the brightness of each region 206, but as an example, as shown in FIG. 43, a bar 224 visually indicating the maximum, minimum and median brightness values ​​for each region 206 as statistical values ​​may be displayed on the display 28. The brightness information indicating the brightness of each region 206 may be displayed in a form other than the histogram 208 and the bar 224.

[0297] In addition, the above-described embodiments and modifications may be combined with each other as long as no contradiction occurs. In addition, when the above-described embodiments and modifications are combined, if there are multiple overlapping steps, the multiple steps may be prioritized according to various conditions, etc.

[0298] Further, in each of the above embodiments, the CPU 62 is exemplified, but instead of the CPU 62 or together with the CPU 62, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0299] In addition, in each of the above embodiments, the program 65 is stored in the NVM 64, but the technology of the present disclosure is not limited to this. For example, the program 65 may be stored in a portable non-transient computer-readable storage medium (hereinafter simply referred to as a "non-transient storage medium") such as an SSD or a USB memory. The program 65 stored in the non-transient storage medium is installed in the controller 12 of the imaging device 10. The CPU 62 executes processing according to the program 65.

[0300] In addition, the program 65 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the program 65 may be downloaded and installed in the controller 12 upon request from the imaging device 10.

[0301] It is not necessary to store the entire program 65 in a storage device such as another computer or server device connected to the imaging device 10, or in the NVM 64; only a part of the program 65 may be stored therein.

[0302] Furthermore, although the imaging device 10 has a built-in controller 12, the technology of the present disclosure is not limited to this. For example, the controller 12 may be provided outside the imaging device 10.

[0303] In addition, in each of the above embodiments, the controller 12 including the CPU 62, the NVM 64, and the RAM 66 is exemplified, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the controller 12. Furthermore, instead of the controller 12, a combination of a hardware configuration and a software configuration may be used.

[0304] In addition, the hardware resources for executing the moving image generation process described in each of the above embodiments can use various processors as shown below. An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the moving image generation process by executing software, i.e., a program. Another example of a processor is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes the moving image generation process by using the memory.

[0305] The hardware resource for executing the moving image generation process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the moving image generation process may be a single processor.

[0306] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the video generation process. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute the video generation process in a single IC chip, as typified by SoC. In this way, the video generation process is realized using one or more of the above various processors as hardware resources.

[0307] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above-mentioned moving image generation process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the invention.

[0308] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0309] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0310] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0311] The following supplementary notes are further disclosed regarding the above embodiment.

[0312] (Appendix 1) An image processing device including a processor, The processor, Obtaining distance information data relating to distance information between the image sensor and the subject; outputting first image data representing a first image captured by the image sensor; outputting first luminance information data indicating first luminance information created based on a signal of the first image data for at least a first region among a plurality of regions into which the first image is classified according to the distance information; performing a first process for reflecting the content of a first instruction derived based on the first image data and the distance information data in the first image and / or the first luminance information; Image processing device.

Claims

1. An image processing device including a processor, The processor, Obtaining distance information data relating to distance information between the image sensor and the subject; outputting, to a display, first image data representing a first image captured by the image sensor; outputting, to the display, first luminance information data indicating first luminance information created based on a first signal of the first image data for each of a plurality of regions of the first image classified according to the distance information; When a first instruction for adjusting at least one of the first luminance information pieces among a plurality of the first luminance information pieces is received by a reception device, a first process is performed to reflect the content of the first instruction in the first image and any of the first luminance information pieces. Image processing device.

2. The first luminance information is a first histogram. The image processing device according to claim 1 .

3. The first histogram shows the relationship between the signal value and the number of pixels. The image processing device according to claim 2 .

4. The processor outputs second luminance information data indicating second luminance information created based on a second signal of the first image data for a second region of the plurality of regions; The first process includes a second process of prohibiting the content of the first instruction from being reflected in the second area and / or the second luminance information. The image processing device according to claim 1 .

5. The processor outputs third luminance information data indicating third luminance information created based on a third signal of the first image data for a third region of the plurality of regions; The first process includes a third process of reflecting the content of the first instruction in the third area and / or the third luminance information. The image processing device according to claim 1 .

6. the first processing is processing for changing the first signal in accordance with the content of the first instruction, the third process is a process of changing the third signal in accordance with the content of the first instruction, A change amount of a first signal value included in the first signal is different from a change amount of a second signal value included in the third signal. The image processing device according to claim 5 .

7. When a range of distances between a plurality of first pixels corresponding to a first region among the plurality of regions and the subject is defined as a first distance range, and a range of distances between a plurality of second pixels corresponding to the third region and the subject is defined as a second distance range, a change amount of the first signal value is constant within the first distance range, In the second distance range, the change amount of the second signal value is constant. The image processing device according to claim 6.

8. the first processing is processing for changing the first signal in accordance with the content of the first instruction, the third process is a process of changing the third signal in accordance with the content of the first instruction, The amount of change in the second signal value included in the third signal varies depending on the distance between the subject and a plurality of second pixels corresponding to the third region. The image processing device according to claim 5 .

9. The first process is a process for changing the first signal in accordance with the content of the first instruction. The image processing device according to claim 1 .

10. The first instruction is an instruction to change a form of the first luminance information. The image processing device according to claim 1 .

11. the first luminance information is a second histogram having a plurality of bins; The first instruction is an instruction to move a bin corresponding to a third signal value selected based on the first instruction from among the plurality of bins. The image processing device according to claim 1 .

12. The processor outputs second image data showing a second image in which the plurality of regions are divided in different manners according to the distance information. The image processing device according to claim 1 .

13. The processor, outputting third image data indicating a distance map image representing a distribution of the distance information with respect to an angle of view of a first imaging device having the image sensor mounted thereon; outputting fourth image data showing a reference distance image representing a reference distance for classifying the plurality of regions; The image processing device according to claim 1 .

14. the reference distance image is an image showing a scale bar and a slider, the scale bar indicates a plurality of distance ranges corresponding to the plurality of regions; The slider is provided on the scale bar, The position of the slider indicates the reference distance. The image processing device according to claim 13.

15. The scale bar is a single scale bar that collectively indicates the plurality of distance ranges. The image processing device according to claim 14.

16. The scale bar is a plurality of scale bars each showing a respective one of the distance ranges. The image processing device according to claim 14.

17. When the reception device receives a second instruction to output the third image data and / or the fourth image data, the processor outputs fifth image data indicating a third image in which the plurality of regions are divided in different manners according to the distance information. The image processing device according to claim 13.

18. When the reception device receives a third instruction related to the reference distance, the processor performs a fourth process of reflecting the content of the third instruction in the reference distance image, and changes the reference distance in accordance with the content of the third instruction. The image processing device according to claim 13.

19. The first image data is video image data. The image processing device according to claim 1 .

20. The image processing device is an imaging device. The image processing device according to claim 1 .

21. The processor outputs the first image data and / or the first luminance information data to a display destination. The image processing device according to claim 1 .

22. The first processing is processing for changing a display mode of the first image and / or the first luminance information displayed on the display device. The image processing device according to claim 21.

23. The image sensor includes a plurality of phase difference pixels, The processor acquires the distance information data based on the phase difference pixel data output from the phase difference pixels. The image processing device according to claim 1 .

24. The phase difference pixel is a pixel that selectively outputs non-phase difference pixel data and the phase difference pixel data, The non-phase difference pixel data is pixel data obtained by performing photoelectric conversion by an entire region of the phase difference pixel, The phase difference pixel data is pixel data obtained by performing photoelectric conversion in a partial region of the phase difference pixel. The image processing device according to claim 23.

25. Obtaining distance information data relating to distance information between an image sensor and a subject; outputting, to a display, first image data representing a first image captured by the image sensor; outputting, to the display, first luminance information data indicating first luminance information created based on a first signal of the first image data for each of a plurality of regions of the first image classified according to the distance information; and When a first instruction for adjusting at least one of the first luminance information pieces among a plurality of the first luminance information pieces is received by a reception device, a first process is performed to reflect the content of the first instruction in the first image and any of the first luminance information pieces. An image processing method comprising:

26. Obtaining distance information data relating to distance information between an image sensor and a subject; outputting, to a display, first image data representing a first image captured by the image sensor; outputting, to the display, first luminance information data indicating first luminance information created based on a first signal of the first image data for each of a plurality of regions of the first image classified according to the distance information; and When a first instruction for adjusting at least one of the first luminance information pieces among a plurality of the first luminance information pieces is received by a reception device, a first process is performed to reflect the content of the first instruction in the first image and any of the first luminance information pieces. A program for causing a computer to execute a process including the steps of:

Images