Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program capable of contributing to solving various problems that occur in the case that a first range-finding result and a second range-finding result acquired by different range-finding systems are different.SOLUTION: An information processing device includes a processor, and a memory connected to or built in the processor, and the processor performs first range-finding for measuring distance to an imaging area on the basis of irradiation timing at which a light irradiator irradiates light to an imaging area and light receiving timing at which a light receiver receives reflection light from the light imaging area, performs second range-finding for measuring distance to the imaging area on the basis of a first image obtained by imaging the imaging area by an imaging device, and executes specific processing in the case that a first range-finding result obtained by performing the first range-finding and a second range-finding result obtained by performing the second range-finding are different.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The technology disclosed herein relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Patent document 1 discloses an optical sensor including a light-emitting element that emits light in one direction along an optical axis, a plurality of light-receiving elements which are avalanche photodiodes arranged in a direction perpendicular to the optical axis, a housing that separates the detected object and the light-emitting element, and a control unit that determines the distance using a TOF method and a triangulation method.

[0003] Patent document 2 discloses a vehicle radar device that, when there is a person near the vehicle, performs scan ranging with low light transmission power so as not to harm the person, and stores the direction in which the person is present, and when the light transmission power is next increased to perform scan ranging, stops transmitting light in the direction in which the previously stored person is present, and performs scan ranging with high light transmission power only in other directions, thereby enabling the presence of a distant vehicle to be reliably detected without harming nearby people. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-223902 A [Patent Document 2] Japanese Patent Application Publication No. 09-197045 Summary of the Invention

[0005] One embodiment of the technology disclosed herein provides an information processing device, an information processing method, and a program that can contribute to solving various problems that arise when a first ranging result and a second ranging result obtained using different ranging methods differ. [Means for solving the problem]

[0006] A first aspect of the technology disclosed herein is an information processing device that includes a processor and a memory connected to or built into the processor, where the processor performs a first distance measurement to measure the distance to the imaging area based on the irradiation timing when a light irradiator irradiates light toward an imaging area and the light reception timing when a light receiver receives the light reflected from the imaging area, performs a second distance measurement to measure the distance to the imaging area based on a first image obtained by imaging the imaging area by an imaging device, and executes a specific process when the first distance measurement result obtained by performing the first distance measurement and the second distance measurement result obtained by performing the second distance measurement differ.

[0007] A second aspect of the technology of the present disclosure is the information processing device according to the first aspect, in which the specific process includes a process of notifying that the first distance measurement result and the second distance measurement result are different.

[0008] A third aspect of the technology disclosed herein is an information processing device according to the first or second aspect, in which the specific processing includes processing for performing focus control on the imaging device based on each of the first ranging result and the second ranging result, and for imaging the imaging area.

[0009] A fourth aspect of the technique disclosed herein is a process in which a specific process displays, on a display, a first focused image obtained by capturing an image of an imaging area under focus control based on a first distance measurement result and a second focused image obtained by capturing an image of an imaging area under focus control based on a second distance measurement result, and prompts a user to select at least one of the first focused image and the second focused image while the first focused image and the second focused image are displayed on the display. The present invention relates to an information processing device according to a third aspect.

[0010] A fifth aspect of the technology disclosed herein is an information processing device according to the first or second aspect, in which the specific processing includes processing for performing focus control on the imaging device based on the first ranging result or the second ranging result, and for imaging the imaging area.

[0011] A sixth aspect of the technology disclosed herein is an information processing device related to the fifth aspect, in which focus control is performed on the imaging device based on either the first ranging result or the second ranging result, which is determined in accordance with given instructions.

[0012] A seventh aspect of the technology of the present disclosure is an information processing device related to the fifth aspect, in which the imaging device is capable of imaging on the wide-angle side and the telephoto side, and when the imaging device images on the wide-angle side, focus control is performed on the imaging device based on the first ranging result.

[0013] An eighth aspect of the technology disclosed herein is an information processing device according to the fifth aspect, in which, when the amount of reflected light received by the optical receiver is greater than or equal to a threshold value, focus control is performed based on the second ranging result.

[0014] A ninth aspect of the technology of the present disclosure is an information processing device according to the fifth aspect, in which focus control is performed on the imaging device based on either the first ranging result or the second ranging result, which is determined according to the proportion of high reflectivity areas indicating areas having a reflectivity equal to or higher than a predetermined reflectivity in a second image obtained by imaging an imaging area by the imaging device, or the difference between the area of ​​the high reflectivity areas in the second image and the area of ​​areas in the second image that are different from the high reflectivity areas.

[0015] A tenth aspect of the technology of the present disclosure is an information processing device related to the fifth aspect, in which the processor detects a specific subject image showing a specific subject from a second image obtained by imaging the imaging area by the imaging device, and when the specific subject image is detected by the processor, focus control is performed based on the second ranging result.

[0016] An eleventh aspect of the technology of the present disclosure is an information processing device according to the tenth aspect, in which the specific subject includes a high-reflectivity area having a reflectivity equal to or greater than a predetermined reflectivity, and the high-reflectivity area is an area in which an image of at least one of a person and an imaging device is reflected.

[0017] A twelfth aspect of the technology of the present disclosure is an information processing device that includes a processor and a memory connected to or built into the processor, where the processor measures a distance to the imaging area based on the irradiation timing when a light emitter irradiates light toward the imaging area and the light receiving timing when a light receiver receives the light reflected from the imaging area, derives a first focusing position for focusing on a subject included in the imaging area based on the distance, derives a second focusing position for focusing on the subject based on an image obtained by imaging the imaging area by the imaging device, and executes a specific process if the first focusing position and the second focusing position are different.

[0018] A thirteenth aspect of the technology disclosed herein is an information processing method that includes performing a first distance measurement to measure the distance to the imaging area based on the irradiation timing when a light irradiator irradiates light toward an imaging area and the reception timing when a light receiver receives the light reflected from the imaging area, performing a second distance measurement to measure the distance to the imaging area based on a first image obtained by imaging the imaging area by an imaging device, and executing a specific process when the first distance measurement result obtained by performing the first distance measurement and the second distance measurement result obtained by performing the second distance measurement differ.

[0019] A fourteenth aspect of the technology of the present disclosure is an information processing method that includes measuring a distance to an imaging area based on an irradiation timing when a light irradiator irradiates light toward an imaging area and a light receiving timing when a light receiver receives the light reflected from the imaging area, deriving a first focusing position for focusing on a subject included in the imaging area based on the distance, deriving a second focusing position for focusing on the subject based on an image obtained by imaging the imaging area by an imaging device, and executing a specific process when the first focusing position and the second focusing position are different.

[0020] A fifteenth aspect of the technology disclosed herein is a program for causing a computer to perform processing including performing a first distance measurement to measure the distance to the imaging area based on the irradiation timing when a light emitter irradiates light toward an imaging area and the light receiving timing when a light receiver receives the light reflected from the imaging area, performing a second distance measurement to measure the distance to the imaging area based on a first image obtained by imaging the imaging area by an imaging device, and performing a specific process when the first distance measurement result obtained by performing the first distance measurement and the second distance measurement result obtained by performing the second distance measurement differ.

[0021] A sixteenth aspect of the technology of the present disclosure is a program for causing a computer to execute processing including measuring a distance to an imaging area based on an irradiation timing when a light emitter irradiates light toward an imaging area and a light receiving timing when a light receiver receives the light reflected from the imaging area, deriving a first focusing position for focusing on a subject included in the imaging area based on the distance, deriving a second focusing position for focusing on the subject based on an image obtained by imaging the imaging area by an imaging device, and executing a specific processing when the first focusing position and the second focusing position are different. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic perspective view showing an example of a usage mode of a smart device according to a first embodiment. [Diagram 2] FIG. 2 is a rear perspective view showing an example of the appearance of the rear side of the smart device according to the first embodiment. [Diagram 3] 1 is a conceptual diagram showing an example of an aspect in which an imaging area is imaged by a distance measuring imaging device included in a smart device according to a first embodiment, and a visible light image is displayed on a display. FIG. [Figure 4] 1 is a conceptual diagram showing an example of a state in which an imaging region is irradiated with laser light by a distance measuring imaging device included in the smart device according to the first embodiment. FIG. [Diagram 5]1 is a schematic perspective view showing an example of an arrangement of pixels included in a photoelectric conversion element of a smart device according to a first embodiment. FIG. [Figure 6] 6 is a conceptual diagram showing an example of incident characteristics of subject light with respect to a first phase difference pixel and a second phase difference pixel included in the photoelectric conversion element shown in FIG. 5. [Figure 7] 6 is a schematic configuration diagram showing an example of the configuration of a non-phase difference pixel included in the photoelectric conversion element shown in FIG. 5. [Figure 8] FIG. 2 is a block diagram showing an example of the configuration of electrical hardware of the smart device according to the first embodiment. [Figure 9] 2 is a block diagram showing an example of functions of a CPU included in the smart device according to the first embodiment. FIG. [Figure 10] 10 is a conceptual diagram showing an example of the processing contents of a first distance measurement control unit and an acquisition unit shown in FIG. 9. [Figure 11] 10 is a conceptual diagram showing an example of the processing contents of a second distance measurement control unit and an acquisition unit shown in FIG. 9. [Figure 12] 1 is a conceptual diagram showing an example of the relationship between a distance measurement result (first distance) obtained by a first distance measurement and a distance measurement result (second distance) obtained by a second distance measurement when a mirror surface and an image reflected in the mirror are each set as distance measurement targets. [Figure 13] 10 is a block diagram showing an example of processing contents of a determination unit and an execution unit shown in FIG. 9. FIG. [Figure 14] FIG. 14 is a conceptual diagram showing an example of details of a specific process shown in FIG. 13. [Figure 15] FIG. 11 is a conceptual diagram showing an example of a process performed when a first visible light image on an image selection screen displayed on a display of the smart device according to the first embodiment is selected by a user via a touch panel. [Figure 16] FIG. 11 is a conceptual diagram showing an example of a process performed when a second visible light image on an image selection screen displayed on a display of the smart device in accordance with the first embodiment is selected by a user via a touch panel. [Figure 17A] 6 is a flowchart showing an example of the flow of distance measurement imaging processing according to the first embodiment. [Figure 17B]This is a continuation of the flowchart shown in FIG. 17A. [Figure 18A] 13 is a flowchart showing a first modified example of the flow of distance measurement imaging processing according to the first embodiment. [Figure 18B] This is a continuation of the flowchart shown in FIG. 18A. [Figure 18C] This is a continuation of the flowchart shown in FIG. 18B. [Figure 19] 13 is a flowchart showing a second modified example of the flow of distance measurement imaging processing according to the first embodiment. [Figure 20] 13 is a flowchart showing a third modified example of the flow of distance measurement imaging processing according to the first embodiment. [Figure 21] 13 is a flowchart showing a fourth modified example of the flow of distance measurement imaging processing according to the first embodiment. [Figure 22] FIG. 2 is a rear perspective view showing an example of an aspect in which the smart device according to the first embodiment further includes a wide-angle light receiver. [Figure 23] 13 is a flowchart showing a fifth modified example of the flow of distance measurement imaging processing according to the first embodiment. [Figure 24] FIG. 11 is a block diagram showing an example of functions of a CPU included in a smart device according to a second embodiment. [Diagram 25] FIG. 11 is a conceptual diagram showing an example of processing contents for calculating a first focus position used in focus control performed by the smart device according to the second embodiment. [Figure 26] FIG. 11 is a conceptual diagram showing an example of processing contents for calculating a second focus position used in focus control performed by the smart device according to the second embodiment. [Figure 27] 25 is a block diagram showing an example of the processing contents of a determination unit and an execution unit shown in FIG. 24. [Figure 28] FIG. 28 is a conceptual diagram showing an example of details of a specific process shown in FIG. 27. [Figure 29A] 10 is a flowchart showing an example of the flow of distance measurement imaging processing according to the second embodiment. [Figure 29B] This is a continuation of the flowchart shown in FIG. 29A. [Figure 29C] This is a continuation of the flowchart shown in Figure 29B. [Diagram 30] 1 is a conceptual diagram showing an example of a manner in which a distance measuring imaging processing program according to an embodiment is installed in a smart device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an example of an embodiment of an imaging device according to the technique of the present disclosure will be described with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] CPU is an abbreviation for "Central Processing Unit". RAM is an abbreviation for "Random Access Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". A / D is an abbreviation for "Analog / Digital". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". LTE is an abbreviation for "Long Term Evolution". 5G is an abbreviation for "5th Generation". LD is an abbreviation for "Laser Diode". IR is an abbreviation for "Infrared". APD is an abbreviation for "Avalanche Photodiode". TOF is an abbreviation for "Time of Flight". fps is an abbreviation for "frame per second". LED is an abbreviation for "Light Emitting Diode". ROI is an abbreviation for "Region of Interest". LAN is an abbreviation for "Local Area Network". AF is an abbreviation for "Auto Focus". IC is an abbreviation for "Integrated Circuit".

[0026] In the description of this specification, "horizontal" refers to horizontal in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to being completely horizontal. In the description of this specification, "parallel" refers to parallel in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to being completely parallel. In the description of this specification, "vertical" refers to vertical in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to being completely vertical. In the description of this specification, "matching" refers to matching in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to being completely matched.

[0027] [First embodiment] 1, as an example, the smart device 10 performs an imaging operation (hereinafter also simply referred to as an "imaging operation") for capturing an image of an imaging area defined by an angle of view θ1, and a distance measurement operation. In the first embodiment, "distance measurement" refers to a process for measuring the distance from the smart device 10 to the imaging area. Note that the smart device 10 is an example of an "information processing device" according to the technology of the present disclosure, and examples of the smart device 10 include a smartphone or a tablet terminal, which are electronic devices with an imaging function.

[0028] In the smart device 10, first and second distance measurement are performed using different distance measurement methods. The first distance measurement is a distance measurement using a method (hereinafter also referred to as an "active method") for measuring the distance to the imaging area based on the timing when the smart device 10 irradiates the imaging area with laser light and the timing when the smart device 10 receives the reflected light of the laser light from the imaging area. The second distance measurement is a distance measurement using a method (hereinafter also referred to as a "passive method") for measuring the distance to the imaging area based on an image obtained by imaging the imaging area with the smart device 10. Here, the image obtained by imaging the imaging area with the smart device 10 is an example of the "first image" according to the technology of the present disclosure. Also, the laser light is an example of the "light" according to the technology of the present disclosure. Also, here, the laser light is surface-irradiated toward the imaging area. Note that surface irradiation refers to irradiation in which the beam diameter of the laser light toward the imaging area is wider than that of spot irradiation. In surface irradiation, the beam diameter of the laser light gradually expands along the irradiation direction of the laser light, and the degree to which the beam diameter expands per unit time is greater in surface irradiation than in spot irradiation. That is, in spot irradiation, the laser light is irradiated in a dot shape onto a surface of the imaging area, whereas in surface irradiation, the laser light is irradiated in a planar shape onto a surface of the imaging area. The laser light may be irradiated in a single shot, or may be irradiated periodically (e.g., every 0.1 seconds), in which case the first distance measurement may be performed for each irradiation of the laser light, and processing based on the distance measurement result may be performed.

[0029] The smart device 10 performs hybrid distance measurement that combines active distance measurement and passive distance measurement. The smart device 10 performs imaging with active focus control and imaging with passive focus control. The active focus control refers to focus control based on the distance measurement result obtained by performing active distance measurement. The passive focus control refers to focus control based on the distance measurement result obtained by performing passive distance measurement.

[0030] As an example, as shown in Fig. 2, a smart device 10 includes a housing 12. A distance measuring imager 14 is housed in the housing 12. The distance measuring imager 14 includes a light illuminator 16 and a light receiver 18. The light illuminator 16 includes an LD 24, and in the smart device 10, an image capturing operation and a distance measuring operation are performed by the distance measuring imager 14.

[0031] The smart device 10 has command keys 13 arranged on a side thereof. The command keys 13 receive various commands. The "various commands" referred to here include, for example, a command to display a menu screen from which various menus can be selected, a command to select one or more menus, a command to confirm the selected content, and a command to erase the selected content.

[0032] When the smart device 10 is placed vertically, the housing 12 has light-transmitting windows 20 and 22 at the upper left of the rear surface 12A (upper left when viewed from the rear of the smart device 10 placed vertically). The light-transmitting windows 20 and 22 are optical elements (e.g., lenses) having light transmissibility, arranged at a predetermined interval (e.g., an interval of several millimeters) along the horizontal direction, and exposed from the rear surface 12A. The light irradiator 16 irradiates the laser light emitted from the LD 24 toward the imaging area through the light-transmitting window 20. In the first embodiment, the light irradiator 16 irradiates the imaging area with laser light in an infrared wavelength range. Note that the wavelength range of the laser light is not limited to this, and may be laser light in another wavelength range.

[0033] The light receiver 18 captures the reflected IR light through the light-transmitting window 22. The reflected IR light refers to light reflected from the target to be measured of the laser light irradiated to the target by the light irradiator 16. The light receiver 18 also captures the reflected visible light through the light-transmitting window 22. The reflected visible light refers to light reflected from the imaging region of visible light irradiated to the imaging region (for example, visible light contained in sunlight). In the following description, for ease of explanation, when there is no need to distinguish between the reflected IR light and the reflected visible light, they will simply be referred to as "reflected light."

[0034] The optical receiver 18 is an example of an "imaging device" according to the technology of the present disclosure, and includes a photoelectric conversion element 26. The photoelectric conversion element 26 receives the reflected light taken into the optical receiver 18 through the light-transmitting window 22, and outputs an electrical signal according to the amount of reflected light received.

[0035] 3, a touch panel display 59 is provided on the front surface 12B of the housing 12. The touch panel display 59 includes a display 46 and a touch panel 48. An example of the display 46 is an organic electroluminescence (EL) display. The display 46 need not be an organic electroluminescence (EL) display, and may be another type of display such as a liquid crystal display.

[0036] The display 46 displays images (for example, live view images and playback images) and text information. The touch panel 48 is a transmissive touch panel, and is overlaid on the surface of the display area of ​​the display 46. The touch panel 48 receives instructions from a user by detecting contact with a pointer such as a finger or a stylus pen. Note that, here, as an example of the touch panel display 59, an out-cell type touch panel display in which the touch panel 48 is overlaid on the surface of the display area of ​​the display 46 is given, but this is merely one example. For example, an on-cell type or in-cell type touch panel display can also be applied as the touch panel display 59.

[0037] In the smart device 10, when an instruction to start capturing an image is received by the touch panel 48, the image capture area is captured by the optical receiver 18. That is, the optical receiver 18 receives reflected visible light and generates a visible light image showing the captured area as an image corresponding to the reflected visible light received. The visible light image is an example of the "first image" and "second image" according to the technology of the present disclosure.

[0038] The visible light image is displayed on the display 46 as a live view image or a still image in accordance with an instruction received by the touch panel 48. In the example shown in Fig. 3, the imaging area is defined by an angle of view θ1. The angle of view θ1 is changed in accordance with an instruction received by the touch panel 48.

[0039] As an example, as shown in FIG. 4, in the smart device 10, when an instruction to start distance measurement and imaging (hereinafter also referred to as a "distance measurement imaging start instruction") is received by the touch panel 48, the light irradiator 16 irradiates laser light. The angle at which the laser light is irradiated (hereinafter also referred to as an "irradiation angle") is θ2, and the irradiation angle θ2 is changed according to the instruction received by the touch panel 48. Note that, in the example shown in FIG. 4, an example is given in which distance measurement is started in response to a distance measurement imaging start instruction received by the touch panel 48 in a state in which a visible light image is displayed on the display 46 as a live view image, but the technology of the present disclosure is not limited thereto. For example, distance measurement may be started when a distance measurement imaging start instruction is received by the touch panel 48 in a state in which a visible light image is not displayed on the display 46.

[0040] In the smart device 10, the distance from the smart device 10 to the target is measured based on the time required from when the laser light is emitted by the light irradiator 16 until the reflected IR light is received by the light receiver 18 and on the speed of light. For example, if the distance to the target is "L0", the speed of light is "c", and the time required from when the laser light is emitted by the light irradiator 16 until the reflected IR light is received by the light receiver 18 is "t", the distance L0 is calculated according to the formula "L0=c×t×0.5".

[0041] 5, the photoelectric conversion element 26 has a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896×3265" pixels.

[0042] A color filter is disposed on each photodiode included in the photoelectric conversion element 26. The color filters include a G filter corresponding to the G (green) wavelength region that contributes most to obtaining a luminance signal, an R filter corresponding to the R (red) wavelength region, a B filter corresponding to the B (blue) wavelength region, and an IR filter corresponding to the IR (infrared) wavelength region. Note that in the first embodiment, the G filter, R filter, and B filter also function as infrared light cut filters that cut infrared light.

[0043] The photoelectric conversion element 26 is formed of two types of photosensitive pixels, namely, phase difference pixels and non-phase difference pixels N, which are pixels different from the phase difference pixels. In general, the non-phase difference pixels N are also called normal pixels. The photoelectric conversion element 26 has four types of photosensitive pixels, namely, R pixels, G pixels, B pixels, and IR pixels, as the non-phase difference pixels. The R pixels, G pixels, B pixels, IR pixels, and phase difference pixels are regularly arranged with a predetermined periodicity in each of the row direction (horizontal direction) and column direction (vertical direction). The R pixels are pixels corresponding to photodiodes in which R filters are arranged, the G pixels and phase difference pixels are pixels corresponding to photodiodes in which G filters are arranged, the B pixels are pixels corresponding to photodiodes in which B filters are arranged, and the IR pixels are pixels corresponding to photodiodes in which IR filters are arranged. An example of an IR pixel is an InGaAs APD.

[0044] In the following, for convenience of explanation, when it is not necessary to distinguish between the G filter, the R filter, and the B filter, they will also be referred to as "visible light filters." In the following, for convenience of explanation, when it is not necessary to distinguish between the R pixel, the G pixel, and the B pixel, they will be referred to as "visible light pixels."

[0045] A plurality of phase difference pixel lines 26A and a plurality of non-phase difference pixel lines 26B are arranged on the light receiving surface of the photoelectric conversion element 26. The phase difference pixel line 26A is a horizontal line including phase difference pixels. Specifically, the phase difference pixel line 26A is a horizontal line including a mixture of phase difference pixels and non-phase difference pixels N. The non-phase difference pixel line 26B is a horizontal line including only a plurality of non-phase difference pixels N.

[0046] On the light receiving surface of the photoelectric conversion element 26, the phase difference pixel lines 26A and the non-phase difference pixel lines 26B corresponding to the predetermined number of lines are alternately arranged along the column direction. The "predetermined number of lines" referred to here refers to, for example, two lines. Note that, although two lines are exemplified as the predetermined number of lines here, the technology of the present disclosure is not limited thereto, and the predetermined number of lines may be a few lines, such as three or more lines, a dozen lines, several tens of lines, or several hundreds of lines.

[0047] The phase difference pixel line 26A is arranged every two rows from the first row to the last row in the column direction. Some pixels in the phase difference pixel line 26A are phase difference pixels. Specifically, the phase difference pixel line 26A is a horizontal line in which phase difference pixels and non-phase difference pixels N are periodically arranged. The phase difference pixels are broadly classified into first phase difference pixels L and second phase difference pixels R. In the phase difference pixel line 26A, the first phase difference pixels L and the second phase difference pixels R are alternately arranged as G pixels at intervals of several pixels in the line direction.

[0048] The first phase difference pixel L and the second phase difference pixel R are arranged so as to alternate in the column direction. In the example shown in FIG. 5, in the fourth column, the first phase difference pixel L, the second phase difference pixel R, the first phase difference pixel L, and the second phase difference pixel R are arranged in this order along the column direction from the first row. That is, the first phase difference pixel L and the second phase difference pixel R are arranged alternately along the column direction from the first row. Also, in the example shown in FIG. 5, in the tenth column, the second phase difference pixel R, the first phase difference pixel L, the second phase difference pixel R, and the first phase difference pixel L are arranged in this order along the column direction from the first row. That is, the second phase difference pixel R and the first phase difference pixel L are arranged alternately along the column direction from the first row.

[0049] The photoelectric conversion element 26 is divided into three regions. That is, the photoelectric conversion element 26 has a visible light image division region 26N1, a first distance measurement system division region 26N2, and a second distance measurement system division region 26N3. The visible light image division region 26N1 is a visible light pixel group consisting of a plurality of visible light pixels, and is used for generating a visible light image. The first distance measurement system division region 26N2 is an IR pixel group consisting of a plurality of IR pixels, and is used for first distance measurement. The second distance measurement system division region 26N3 is a phase difference pixel group consisting of a plurality of phase difference pixels, and is used for second distance measurement. The visible light image division region 26N1 and the second distance measurement system division region 26N3 receive visible reflected light and output an electrical signal according to the amount of received light. The first distance measurement system division region 26N2 receives IR reflected light and outputs an electrical signal according to the amount of received light. The sensor outputs an electrical signal.

[0050] 6, the first phase difference pixel L includes a microlens 19, a light shielding member 17A, and a photodiode PD. In the first phase difference pixel L, the light shielding member 17A is disposed between the microlens 19 and the light receiving surface of the photodiode PD. The left half of the light receiving surface of the photodiode PD in the row direction (the left side when the light receiving surface faces an object (in other words, the right side when the light receiving surface faces an object)) is shielded from light by the light shielding member 17A.

[0051] The second phase difference pixel R includes a microlens 19, a light shielding member 17B, and a photodiode PD. In the second phase difference pixel R, the light shielding member 17B is disposed between the microlens 19 and the light receiving surface of the photodiode PD. The right half of the light receiving surface of the photodiode PD in the row direction (the right side when the light receiving surface faces the subject (in other words, the left side when the light receiving surface faces the subject)) is shielded by the light shielding member 17B. In the following description, for convenience of explanation, when it is not necessary to distinguish between the light shielding members 17A and 17B, they will be referred to as "light shielding members" without reference numerals.

[0052] The light beam passing through the exit pupil of the imaging lens 41 is roughly divided into left region passing light 300L and right region passing light 300R. The left region passing light 300L refers to the left half of the light beam passing through the exit pupil of the imaging lens 41 when facing the subject side from the phase difference pixel side, and the right region passing light 300R refers to the right half of the light beam passing through the exit pupil of the imaging lens 41 when facing the subject side from the phase difference pixel side. The light beam passing through the exit pupil of the imaging lens 41 is divided into left and right by the microlens 19, the light shielding member 17A, and the light shielding member 17B that function as pupil division units, and the first phase difference pixel L receives the left region passing light 300L as subject light, and the second phase difference pixel R receives the right region passing light 300R as subject light. As a result, the photoelectric conversion element 26 generates a first phase-contrast image corresponding to the subject image corresponding to the left area passing light 300L and a second phase-contrast image corresponding to the subject image corresponding to the right area passing light 300R.

[0053] In the smart device 10, for example, in the same phase difference pixel line 26A, the distance to the imaging region is measured based on the shift amount α between one line of the first phase difference image and one line of the second phase difference image.

[0054] 7, the non-phase difference pixel N is different from the phase difference pixel in that it does not have a light blocking member. The photodiode PD of the non-phase difference pixel N receives the left area passing light 300L and the right area passing light 300R as the subject light.

[0055] In the smart device 10, the reflected IR light is received by each of the multiple IR pixels included in the second ranging system section area 26N3 (see FIG. 5), and thus distance measurement is performed for each IR pixel. In the smart device 10, the distance measurement results for each IR pixel are displayed as a distance image on the display 46 in accordance with an instruction received by the touch panel 48. Here, the distance image refers to an image in which the distance to the target measured for each IR pixel is expressed in different colors and / or shades.

[0056] In addition, in the smart device 10, the distance measurement result is displayed on the display 46 as a distance image or a distance superimposed image in accordance with an instruction received by the touch panel 48. The distance superimposed image displayed on the display 46 is, for example, an image in which a numerical value indicating the distance measurement result is superimposed on a visible light image (for example, a live view image). For example, the distance from the smart device 10 to each of a plurality of representative locations (for example, three locations) within the imaging area is displayed on the display 46 in a visible light image. One example of the representative locations is a plurality of locations among specific subjects (for example, subjects included in the central area of ​​the screen and / or humans, etc.) within the imaging area where the difference in contrast between each other is equal to or greater than a preset value.

[0057] As an example, as shown in FIG. 8, in addition to a light irradiator 16 and a light receiver 18, a smart device 10 includes a controller 15, an input / output interface 40, an image memory 42, a UI device 44, an external I / F 52, and a communication I / F 54.

[0058] The controller 15 includes a CPU 15A, a storage 15B, and a memory 15C. The CPU 15A, the storage 15B, and the memory 15C are connected via a bus 50, and the bus 50 is connected to an input / output interface 40. In the example shown in Fig. 8, for convenience of illustration, one bus is shown as the bus 50, but multiple buses may be used. The bus 50 may be a serial bus, or may be a parallel bus including a data bus, an address bus, and a control bus.

[0059] The storage 15B stores various parameters and various programs. The storage 15B is a non-volatile storage device. Here, a flash memory is adopted as an example of the storage 15B. The flash memory is merely an example, and the storage 15B may be, for example, various non-volatile memories such as a magnetoresistive memory and / or a ferroelectric memory instead of the flash memory or in addition to the flash memory. The non-volatile storage device may be an EEPROM, a HDD, and / or an SSD. The memory 15C temporarily stores various information and is used as a work memory. An example of the memory 15C is a RAM, but is not limited to this and may be another type of storage device.

[0060] Various programs are stored in the storage 15B. The CPU 15A reads out necessary programs from the storage 15B and executes the read out programs on the memory 15C. The CPU 15A controls the entire smart device 10 according to the programs executed on the memory 15C. The storage 15B and the memory 15C are examples of the "memory" according to the technology of the present disclosure.

[0061] A plurality of devices are connected to the input / output interface 40, and the input / output interface 40 controls the transmission and reception of various information between the plurality of devices. In the example shown in Fig. 8, the plurality of devices connected to the input / output interface 40 include the controller 15, the light irradiator 16, the light receiver 18, the image memory 42, the UI device 44, the external I / F 52, and the communication I / F 54.

[0062] The external I / F 52 is responsible for transmitting and receiving various information between the smart device 10 and a device (hereinafter, also referred to as an "external device") that exists outside the smart device 10. An example of the external I / F 52 is a USB interface. The USB interface can be directly or indirectly connected to an external device (not shown) such as a smart device, a personal computer, a server, a USB memory, a memory card, and / or a printer.

[0063] The communication I / F 54 has a communication function such as LTE, 5G, wireless LAN, and / or Bluetooth (registered trademark), and controls the exchange of various information between an external device and the CPU 15A. For example, the communication I / F 54 is communicatively connected to a network 56 (e.g., the Internet) via a base station (not shown), and controls the exchange of various information between an external device on the network 56 and the CPU 15A.

[0064] The UI device 44 includes a display 46, and the CPU 15A controls the display 46 to display various information. The UI device 44 also includes a reception device 47. The reception device 47 includes a touch panel 48 and a hard key unit 53. The hard key unit 53 is at least one hard key including the instruction key 13 (see FIG. 2). The CPU 15A operates according to various instructions received by the touch panel 48. Note that, although the hard key unit 53 is included in the UI device 44 here, the technology of the present disclosure is not limited to this, and for example, the hard key unit 53 may be connected to the external I / F 52.

[0065] The light irradiator 16 includes a light-transmitting window 20, a beam expander 21, a collimating lens 23, an LD 24, and an LD driver 25, and the light-transmitting window 20, the beam expander 21, and the collimating lens 23 are arranged in this order along the optical axis L1 from the imaging region side (object side) to the LD 24. The LD driver 25 is connected to the LD 24 and the input / output interface 40, and drives the LD 24 in accordance with instructions from the CPU 15A to cause the LD 24 to emit laser light.

[0066] The laser light emitted from the LD 24 is converted into parallel light by the collimator lens 23, the beam diameter of which is expanded by the beam expander 21, and the laser light is irradiated from the light transmitting window 20 toward the object to be measured.

[0067] The optical receiver 18 includes a light-transmitting window 22, an objective lens 30A, a focus lens 30B, an aperture 30C, a photoelectric conversion element 26, a photoelectric conversion element driver 32, and a signal processing circuit 34. The CPU 15A and the signal processing circuit 34 are an example of a "processor" according to the technology of the present disclosure.

[0068] In the light receiver 18, the light-transmitting window 22, the objective lens 30A, the focus lens 30B, and the aperture 30C are arranged in this order along the optical axis L2 from the imaging region side (object side) to the photoelectric conversion element 26. The photoelectric conversion element driver 32 is connected to the photoelectric conversion element 26 and the input / output interface 40, and drives the photoelectric conversion element 26 according to an instruction from the CPU 15A. For example, under the control of the CPU 15A, the photoelectric conversion element driver 32 supplies the photoelectric conversion element 26 with an imaging timing signal that specifies the timing of imaging performed by the photoelectric conversion element 26. The photoelectric conversion element 26 performs resetting, exposure, and output of an electrical signal according to the imaging timing signal supplied from the photoelectric conversion element driver 32. Examples of the imaging timing signal include a vertical synchronization signal and a horizontal synchronization signal.

[0069] The light receiver 18 includes a focusing control mechanism 31. The focusing control mechanism 31 includes a focus lens 30B, a moving mechanism 60, a motor 62, and a motor driver 64. The focus lens 30B is supported by the moving mechanism 60 so as to be slidable along the optical axis L2. The motor 62 is connected to the moving mechanism 60 and the motor driver 64. The motor driver 64 is connected to the input / output interface 40, and drives the motor 62 according to an instruction from the CPU 15A. The moving mechanism 60 is connected to a drive shaft (not shown) of the motor 62, and selectively moves the focus lens 30B along the optical axis L2 to the object side and the image side by receiving power from the motor 62. That is, the CPU 15A adjusts the focusing position by controlling the driving of the motor 62 via the motor driver 64. Here, the "focusing position" refers to the position of the focus lens 30B on the optical axis L2 in a focused state (for example, a state in which the contrast of the visible light image is maximized, or a state in which a predetermined depth of field is realized). In the first embodiment, the control for adjusting the focus lens 30B to the in-focus position is called "focus control."

[0070] The aperture 30C is a fixed aperture whose opening does not change. In the case of a fixed aperture, exposure adjustment is performed by an electronic shutter of the photoelectric conversion element 26. The aperture 30C may not be a fixed aperture, but may be a variable aperture. The lens configuration and / or the position of the aperture 30C are merely examples, and the technology of the present disclosure is valid even if the lens configuration and / or the position of the aperture 30C are changed.

[0071] The reflected light is incident on the light receiver 18 through the light-transmitting window 22. The reflected light incident on the light-transmitting window 22 is imaged on the photoelectric conversion element 26 via the objective lens 30A, the focus lens 30B, and the aperture 30C.

[0072] The photoelectric conversion element 26 is connected to a signal processing circuit 34, and outputs pixel data indicating a pixel value for each of the visible light pixels and the IR pixels to the signal processing circuit 34. The signal processing circuit 34 digitizes the pixel data input from the photoelectric conversion element 26 by performing A / D conversion, and performs various types of signal processing on the digitized pixel data.

[0073] The signal processing circuit 34 includes a visible light pixel data processing circuit 34A, a first ranging system processing circuit 34B, and a second ranging system processing circuit 34C. The visible light pixel data processing circuit 34A generates a visible light image by performing known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction on the visible light pixel data, which is pixel data for visible light pixels. The visible light pixel data processing circuit 34A then stores the visible light image in an image memory 42. Note that the visible light image in the image memory 42 is updated by overwriting and saving one frame of the visible light image in the image memory 42.

[0074] The distance measuring imaging device 14 includes a TOF camera 27. The TOF camera 27 includes a light irradiator 16, a first distance measuring system partition area 26N2 (see FIG. 5) of the photoelectric conversion element 26, and a first distance measuring system processing circuit 34B. The first distance measuring system processing circuit 34B acquires from the CPU 15A an irradiation timing signal indicating the irradiation timing (hereinafter also simply referred to as "irradiation timing") at which the light irradiator 16 irradiates the imaging area with laser light.

[0075] The first ranging system processing circuit 34B measures the distance from the smart device 10 to the imaging area for each IR pixel based on the irradiation timing indicated by the irradiation timing signal and the timing at which the IR reflected light is received by each IR pixel (hereinafter also referred to as the "light receiving timing"). Here, the timing at which the IR pixel data having an output value exceeding a threshold is received by the first ranging system processing circuit 34B is adopted as the light receiving timing. Note that the threshold used here is a value derived in advance as the output value of a noise component (e.g., a noise component generated unrelated to the IR reflected light) output from the IR pixel, for example, by testing using an actual device and / or computer simulation.

[0076] The first ranging system processing circuit 34B measures the distance from the smart device 10 to the imaging area for each IR pixel based on the irradiation timing and the light reception timing, generates a distance image based on the measurement result for each IR pixel, and stores the generated distance image in the image memory 42. Note that the distance image in the image memory 42 is updated by overwriting and saving one frame of the distance image in the image memory 42.

[0077] The second ranging system processing circuit 34C acquires phase difference pixel data indicating pixel values ​​of phase difference pixels from each of a plurality of phase difference pixels in the second ranging system partition area 26N3 (see FIG. 5) included in an area (so-called ROI) designated by a user or the like in the photoelectric conversion element 26. The second ranging system processing circuit 34C generates a first phase difference image and a second phase difference image (see FIG. 5) from the phase difference pixel data, and calculates a shift amount α (see FIG. 5) between the generated first phase difference image and the second phase difference image. Then, the second ranging system processing circuit 34C calculates the distance from the smart device 10 to the imaging area based on the calculated shift amount α. To be more specific, the second ranging system processing circuit 34C calculates the distance from the smart device 10 to the imaging area by using an arithmetic expression in which the shift amount α is an independent variable and the distance is a dependent variable.

[0078] Although an arithmetic formula is exemplified here, the technology of the present disclosure is not limited to this, and the second ranging system processing circuit 34C may derive the distance from the smart device 10 to the imaging area by using a table in which the shift amount α corresponds to the distance.

[0079] The CPU 15A acquires the distance measured by the first ranging system processing circuit 34B (hereinafter referred to as the "first distance") from the first ranging system processing circuit 34B, and acquires the distance measured by the second ranging system processing circuit 34C (hereinafter referred to as the "second distance") from the second ranging system processing circuit 34C.

[0080] 9, the storage 15B stores a distance measurement imaging processing program 70. The CPU 15A reads out the distance measurement imaging processing program 70 from the storage 15B and executes the read out distance measurement imaging processing program 70 to operate as a first distance measurement control unit 15A1, a second distance measurement control unit 15A2, an acquisition unit 15A3, a determination unit 15A4, and an execution unit 15A5.

[0081] Here, with reference to FIG. 10, a case where the smart device 10 performs the first distance measurement on an imaging area including the entire mirror 100 in which the user is reflected (hereinafter, also referred to as the "mirror-including imaging area") will be described. When the touch panel 48 receives a distance measurement imaging start instruction with the mirror-including imaging area being within the irradiation angle θ2, the first distance measurement control unit 15A1 outputs a first distance measurement start signal to the light irradiator 38B and the light receiver 18. When the first distance measurement start signal is input from the first distance measurement control unit 15A1, the light irradiator 38B irradiates a laser beam. When the first distance measurement start signal is input from the first distance measurement control unit 15A1, the first distance measurement system processing circuit 34B calculates the first distance based on the irradiation timing and light reception timing indicated by the irradiation timing signal acquired from the CPU 15A. The acquisition unit 15A3 acquires the first distance from the first distance measurement system processing circuit 34B. The first distance is an example of the "first distance measurement result" according to the technology of the present disclosure. Note that, here, the irradiation timing is the timing at which a predetermined time has elapsed from the timing at which the first distance measurement start signal is input from the first distance measurement control unit 15A1 to the light receiver 18. Here, the predetermined time may be, for example, a time required from the output of the first distance measurement start signal until the light irradiator 38B irradiates the laser light, which is derived in advance by a test using an actual device and / or a computer simulation.

[0082] Next, with reference to FIG. 11, a case where the smart device 10 performs the second distance measurement on the mirror-including imaging area will be described. When the touch panel 48 receives a distance measurement imaging start instruction with the mirror-including imaging area being within the angle of view θ1, the second distance measurement control unit 15A2 outputs a second distance measurement start signal to the light receiver 18. When the second distance measurement start signal is input from the second distance measurement control unit 15A2 to the light receiver 18, the second distance measurement system division area 26N3 images the mirror-including imaging area and outputs phase difference pixel data corresponding to the mirror-including imaging area to the second distance measurement system processing circuit 34C. The second distance measurement system processing circuit 34C generates a first phase difference image and a second phase difference image (see FIG. 6) based on the phase difference pixel data input from the second distance measurement system division area 26N3, and calculates a shift amount α (see FIG. 6) based on the generated first phase difference image and second phase difference image. The second distance measurement system processing circuit 34C calculates a second distance from the calculated shift amount α. The acquisition unit 15A3 acquires the second distance from the second ranging system processing circuit 34C. Note that the second distance is an example of a "second ranging result" according to the technique of the present disclosure.

[0083] 12, the result of distance measurement obtained by the smart device 10 performing the first distance measurement on the mirror-including imaging area, i.e., the first distance, is the distance from the smart device 10 to the mirror surface 100A of the mirror 100. Also, the result of distance measurement obtained by the smart device 10 performing the second distance measurement on the mirror-including imaging area, i.e., the second distance, is the distance from the smart device 10 to the mirror surface 100A of the mirror 100. The second distance is the distance from the smart device 10 to the subject image reflected in the mirror 100. In the example shown in Fig. 12, an image of the smart device 10 (in the example shown in Fig. 12, a "smart device image") and an image of the user (in the example shown in Fig. 12, a "user image") are shown as subject images reflected in the mirror 100, and the second distance is the distance from the smart device 10 to the user image.

[0084] In this way, when the first distance and the second distance are different, various problems occur. For example, when a user wants to capture an image of the entire mirror 100 as a subject using the smart device 10, if the smart device 10 performs focus control based on the second distance to capture the entire mirror 100, an image with the mirror 100 out of focus is obtained. On the other hand, when a user wants to capture an image of a user reflected in the mirror 100 as a subject using the smart device 10, if the smart device 10 performs focus control based on the first distance to capture the user image, an image with the user image out of focus is obtained.

[0085] In order to solve such a problem, the determination unit 15A4 and the execution unit 15A5 operate as shown in FIG. 13 as an example. The determination unit 15A4 determines whether or not the first distance (see FIG. 10) and the second distance (see FIG. 11) acquired by the acquisition unit 15A3 are different. If the first distance and the second distance match, that is, if the determination unit 15A4 determines that the first distance and the second distance are not different, the execution unit 15A5 performs a first imaging process. The first imaging process refers to imaging accompanied by focus control based on the first distance.

[0086] The storage 15B stores a focus position derivation table 72. In the focus position derivation table 72, the distance from the smart device 10 to the imaging area is associated with the focus position. In the first imaging process, the execution unit 15A5 derives a focus position corresponding to the first distance from the focus position derivation table 72, and controls the motor 62 of the light receiver 18 to move the focus lens 30B to the derived focus position. Then, the execution unit 15A5 controls the visible light image division area 26N1 of the light receiver 18, so that the imaging area is imaged by the visible light image division area 26N1, and visible light pixel data obtained by imaging is output from the visible light image division area 26N1 to the visible light pixel data processing circuit 34A. The visible light pixel data processing circuit 34A generates a first visible light image indicating the imaging area based on the visible light pixel data input from the visible light image division area 26N1, and outputs the generated first visible light image to the image memory 42. The image memory 42 stores a first visible light image.

[0087] On the other hand, when the determination unit 15A4 determines that the first distance and the second distance are different, the execution unit 15A5 performs a specific process. Here, the specific process will be described with reference to FIG. 14. As shown in FIG. 14 as an example, the specific process is a process including, for example, a first imaging process, a second imaging process, and an image selection screen display process. The second imaging process refers to imaging accompanied by focus control based on the second distance. In the second imaging process, similar to the first imaging process, the execution unit 15A5 derives a focus position corresponding to the second distance from the focus position derivation table 72, and controls the motor 62 of the light receiver 18 to move the focus lens 30B to the derived focus position. Then, similar to the first imaging process, the imaging area is imaged by the visible light image division region 26N1, and visible light pixel data obtained by imaging is output from the visible light image division region 26N1 to the visible light pixel data processing circuit 34A. The visible light pixel data processing circuit 34A generates a second visible light image showing the imaging area based on the visible light pixel data input from the visible light image partition area 26N1, and outputs the generated second visible light image to the image memory 42. The second visible light image is stored in the image memory 42.

[0088] In the image selection screen display process, the execution unit 15A5 acquires the first visible light image and the second visible light image from the image memory 42. Then, the execution unit 15A5 generates an image selection screen based on the first visible light image, the second visible light image, the first distance acquired by the acquisition unit 15A3 (see FIG. 10), the second distance acquired by the acquisition unit 15A3 (see FIG. 11), and various messages, and causes the generated image selection screen to be displayed on the display 46. The first visible light image and the second visible light image are displayed side by side on the image selection screen. In addition, a message saying "active method" is displayed below the first visible light image. The message saying "active method" is a message indicating that the first visible light image is an image obtained by imaging accompanied by focus control based on the distance measurement result by the active method (first distance measurement). In addition, a message saying "passive method" is displayed below the second visible light image. The message "passive method" is a message indicating that the second visible light image is an image obtained by imaging with focus control based on the distance measurement result by passive method distance measurement (second distance measurement).

[0089] In the first embodiment, the first imaging process, the second imaging process, and the image selection screen display process are exemplified as processes included in the specific process, but the technology of the present disclosure is not limited thereto, and one or two of the first imaging process, the second imaging process, and the image selection screen display process may be used. In the first embodiment, the display examples of "active method" and "passive method" are shown, but the display does not necessarily have to be "active method" and "passive method", and any display may be used as long as the user can understand the difference between the distance measurement methods. For example, when performing laser distance measurement and phase difference distance measurement, the display may be "laser" and the display may be "phase difference", or an icon representing the distance measurement method may be displayed. In addition, instead of displaying the distance measurement method, the focus position may be displayed. For example, "focus position: front" and "focus position: back" may be displayed, or "focus position: object" and "focus position: image reflected on the object" may be displayed. Alternatively, two or more characters and icons indicating the distance measurement method and the focus position may be displayed in combination.

[0090] Further, the first visible light image is displayed in association with a value of "1.8m" which is the distance measurement result of the first distance measurement, and the second visible light image is displayed in association with a value of "2.5m" which is the distance measurement result of the second distance measurement. Furthermore, a message "distance measurement results are different" is also displayed, indicating that the distance measurement result of the first distance measurement and the distance measurement result of the second distance measurement are different. Then, a message "Please select one of the images" is displayed as a message to prompt the user to select one of the first visible light image and the second visible light image. Here, an example of a form in which various messages are visually displayed is described, but this is not limiting, and various messages may be output by voice in parallel with the visual display. When the image selection screen is displayed on the display 46, the user selects the first visible light image or the second visible light image via the touch panel 48. In the example shown in FIG. 14, a form in which the first visible light image is selected via the touch panel 48 by the user's finger is shown. The first visible light image is an example of a "first focused image" according to the technology of the present disclosure, and the second visible light image is an example of a "second focused image" according to the technology of the present disclosure.

[0091] As an example, as shown in FIG. 15, when the first visible light image is selected by the user's finger via the touch panel 48 while the image selection screen is displayed on the display 46, the first distance measurement control unit 15A1 controls the distance measurement imaging device 14 to cause the first distance measurement system processing circuit 34B to perform the first distance measurement. The execution unit 15A5 executes the first imaging process and the first visible light image display process. Here, the execution unit 15A5 executes the first imaging process using a new first distance obtained by performing the first distance measurement. The first visible light image display process is a process for displaying the latest first visible light image obtained by performing the first imaging process on the display 48. Note that, in this embodiment, the first distance measurement and the first imaging process are performed again on the condition that the first visible light image is selected, and the latest first visible light image obtained by performing the first imaging process is displayed on the display 48. However, this is merely an example. For example, on condition that the first visible light image is selected, the selected first visible light image may be stored in a storage medium such as storage 15B and / or a memory card. Furthermore, imaging with focus control based on the distance measured by the selected distance measurement method and saving of the selected image (e.g., the first visible light image or the second visible light image) may be selected according to an instruction received by the receiving device 47 (an instruction given by the user).

[0092] When the first visible light image display process is executed by the execution unit 15A5, the second visible light image, the numerical value "2.5 m", the message "passive method", the message "distance measurement results are different", and the message "select one of the images" are hidden from the image selection screen. Then, the first distance (the numerical value "1.8 m" in the example shown in FIG. 15) which is the latest distance measurement result obtained by the latest first distance measurement is displayed on the display 46, the first visible light image obtained by the execution of the first imaging process is displayed on the display 46, and the message "active method" is also displayed below the first visible light image. In addition, the display area of ​​the first distance, the first visible light image, and the message "active method" is larger than the display area of ​​the first distance, the first visible light image, and the message "active method" in the image selection screen.

[0093] 16, when a second visible light image is selected by the user's finger via touch panel 48 while an image selection screen is displayed on display 46, second distance measurement control unit 15A2 controls distance measurement imaging device 14 to cause second distance measurement system processing circuit 34C to perform second distance measurement. Furthermore, execution unit 15A5 executes second imaging process and second visible light image display process. Here, execution unit 15A5 executes the second imaging process using a new second distance obtained by performing the second distance measurement. The second visible light image display process is a process for displaying on display 48 the latest second visible light image obtained by performing the second imaging process.

[0094] When the second visible light image display process is executed by the execution unit 15A5, the first visible light image, the numerical value "1.8m", the message "active method", the message "distance measurement results are different", and the message "select one of the images" are hidden from the image selection screen. Then, the second distance (the numerical value "2.5m" in the example shown in FIG. 16) which is the latest distance measurement result obtained by the latest second distance measurement is displayed on the display 46, the second visible light image obtained by the execution of the second imaging process is displayed on the display 46, and the message "passive method" is also displayed below the second visible light image. In addition, the display areas of the second distance, the second visible light image, and the message "passive method" are enlarged compared to the display areas of the second distance, the second visible light image, and the message "passive method" in the image selection screen.

[0095] Next, the operation of the smart device 10 according to the technology of the present disclosure will be described with reference to Fig. 17A and Fig. 17B. Fig. 17A and Fig. 17B are flowcharts showing an example of the flow of distance measurement imaging processing executed by the CPU 15A according to the distance measurement imaging processing program 70.

[0096] 17A, first, in step ST100, the first distance measurement control unit 15A1 controls the first distance measurement system processing circuit 34B to perform a first distance measurement by controlling the distance measurement image pickup device 14. The first distance measurement is realized by the first distance measurement control unit 15A1 causing the first distance measurement system processing circuit 34B to calculate a first distance.

[0097] In the next step ST102, the acquisition unit 15A3 acquires the first distance obtained by performing the first distance measurement in step ST100.

[0098] In the next step ST104, the second distance measurement control unit 15A2 causes the second distance measurement system processing circuit 34C to perform the second distance measurement by controlling the distance measurement imaging device 14. The second distance measurement is realized by the second distance measurement control unit 15A2 causing the second distance measurement system processing circuit 34C to calculate the second distance.

[0099] In the next step ST106, the acquisition unit 15A3 acquires the second distance obtained by performing the second distance measurement in step ST104.

[0100] In the next step ST108, the determination unit 15A4 determines whether or not the first distance acquired in step ST102 is different from the second distance acquired in step ST106. In step ST108, if the first distance acquired in step ST102 is the same as the second distance acquired in step ST106, the determination is negative, and the distance measurement imaging process proceeds to step ST120. In step ST108, if the first distance acquired in step ST102 is different from the second distance acquired in step ST106, the determination is positive, and the distance measurement imaging process proceeds to step ST110.

[0101] In step ST110, the execution unit 15A5 executes the first imaging process using the first distance acquired in step ST102.

[0102] In the next step ST112, the execution unit 15A5 executes the second imaging process using the second distance acquired in step ST106.

[0103] In the next step ST114, the execution section 15A5 executes an image selection screen display process, whereby an image selection screen is displayed on the display 46, as shown in FIG.

[0104] In the next step ST116, the execution unit 15A5 determines whether or not either the first visible light image or the second visible light image displayed in the image selection screen has been selected by the user via the touch panel 48. In step ST116, if either the first visible light image or the second visible light image displayed in the image selection screen has not been selected by the user via the touch panel 48, the determination is negative, and the determination of step ST116 is performed again. In step ST116, if either the first visible light image or the second visible light image displayed in the image selection screen has been selected by the user via the touch panel 48, the determination is positive, and the distance measurement imaging process proceeds to step ST118.

[0105] In step ST118, the execution unit 15A5 determines whether or not the image selected from within the image selection screen is the first visible light image. If the image selected from within the image selection screen is the second visible light image in step ST118, the determination is negative, and the distance measurement imaging process proceeds to step 130 shown in Fig. 17B. If the image selected from within the image selection screen is the first visible light image in step ST118, the determination is positive, and the distance measurement imaging process proceeds to step ST120.

[0106] In step ST120, the execution unit 15A5 determines whether or not the imaging start timing has arrived. The imaging start timing refers to, for example, the timing to start imaging one frame defined by the frame rate for live view images. For example, if the frame rate for live view images is 60 fps, the imaging start timing is every 1 / 60th of a second. Note that, although the timing to start imaging one frame defined by the frame rate for live view images is exemplified here as the imaging start timing, the technology of the present disclosure is not limited to this. For example, the imaging start timing may be the timing to start imaging one frame defined by the frame rate for recorded moving images. This may be the timing at which imaging for one frame is started, which is defined by the frame rate for the camera, or the timing at which an instruction to start imaging for a still image is accepted by the accepting device 47.

[0107] In step ST120, if the imaging start timing has not arrived, the determination is negative, and the distance measurement imaging process proceeds to step ST126. In step ST120, if the imaging start timing has arrived, the determination is positive, and the distance measurement imaging process proceeds to step ST122.

[0108] In step ST122, the first distance measurement control section 15A1 controls the distance measurement imaging device 14 to cause the first distance measurement system processing circuit 34B to perform the first distance measurement.

[0109] In the next step ST124, the execution unit 15A5 executes a first imaging process using the first distance obtained by performing the first distance measurement in step ST122. In addition, the execution unit 15A5 executes a first visible light image display process using the latest first visible light image obtained by performing the first imaging process.

[0110] In the next step ST126, the execution unit 15A5 determines whether or not a condition for terminating the distance measurement imaging process (hereinafter referred to as the "termination condition") has been satisfied. One example of the termination condition is that an instruction to terminate the distance measurement imaging process has been accepted by the acceptance device 47. In step ST126, if the termination condition has not been satisfied, the determination is negative, and the distance measurement imaging process proceeds to step ST120. In step ST126, if the termination condition has been satisfied, the determination is positive, and the distance measurement imaging process ends.

[0111] 17B, ​​the execution unit 15A5 determines whether or not the imaging start timing has arrived. If the imaging start timing has not arrived in step ST130, the determination is negative, and the distance measurement imaging process proceeds to step ST136. If the imaging start timing has arrived in step ST130, the determination is positive, and the distance measurement imaging process proceeds to step ST132.

[0112] In step ST132, the second distance measurement control section 15A2 controls the distance measurement imaging device 14 to cause the second distance measurement system processing circuit 34C to perform the second distance measurement.

[0113] In the next step ST134, the execution unit 15A5 executes a second imaging process using the second distance obtained by performing the second distance measurement in step ST132. In addition, the execution unit 15A5 executes a second visible light image display process using the latest second visible light image obtained by performing the second imaging process.

[0114] In the next step ST136, the execution unit 15A5 judges whether or not the end condition is satisfied. If the end condition is not satisfied in step ST136, the judgment is negative, and the distance measurement imaging process proceeds to step ST130. If the end condition is satisfied in step ST136, the judgment is positive, and the distance measurement imaging process ends.

[0115] As described above, in the smart device 10, when the first distance obtained by performing the first distance measurement differs from the second distance obtained by performing the second distance measurement, the execution unit 15A5 executes a specific process. Therefore, this configuration can contribute to solving various problems that occur when the first distance differs from the second distance.

[0116] Furthermore, in the smart device 10, distance measurement is performed using laser light as directional light, which is light having directionality. Therefore, according to this configuration, the distance to a distance measurement target located at a long distance can be measured with high accuracy compared to a case where distance measurement is performed without using directional light.

[0117] Furthermore, in the smart device 10, an image selection screen display process is executed by the execution unit 15A5 as a specific process. The image selection screen displayed on the display 46 by the execution unit 15A5 of the image selection screen display process includes information (e.g., a numerical value indicating the first distance, a numerical value indicating the second distance, and a message) notifying the user that the first distance and the second distance are different. Therefore, according to this configuration, it is possible to make the user recognize that the first distance and the second distance are different.

[0118] In addition, in the smart device 10, the first imaging process and the second imaging process are executed by the execution unit 15A5. That is, the execution unit 15A5 causes the light receiver 18 of the distance measuring imaging device 14 to perform imaging with focus control based on the first distance, and causes the light receiver 18 of the distance measuring imaging device 14 to perform imaging with focus control based on the second distance. Therefore, according to this configuration, it is easier to obtain an image in a focused state as intended by the user, compared to when imaging is performed under focus control based on the first distance or the second distance.

[0119] Furthermore, in the smart device 10, an image selection screen including a first visible light image obtained by capturing an image under focus control based on the first distance and a second visible light image obtained by capturing an image under focus control based on the second distance is displayed on the display 46. A message is displayed on the image selection screen to prompt the user to select the first visible light image or the second visible light image. Therefore, this configuration can contribute to improving usability compared to a case where there is no room to select the first visible light image and the second visible light image.

[0120] In the first embodiment, the execution unit 15A5 prompts the user to select the first visible light image or the second visible light image through the image selection screen, but the technology of the present disclosure is not limited to this, and both the first visible light image and the second visible light image may be selected. In this case, for example, the first imaging process and the second imaging process may be performed alternately, and each of the first visible light image and the second visible light image obtained by performing each imaging process may be displayed on the display 46 as a live view image or a still image, etc.

[0121] Furthermore, a plurality of images including a first visible light image and a second visible light image may be displayed in a selectable manner on the image selection screen. Furthermore, when a plurality of first distances are obtained by the first distance measurement, a plurality of first visible light images obtained by performing image capture under focus control based on each of the plurality of first distances may be displayed in a selectable manner on the image selection screen on the display 46, or a plurality of first visible light images and second visible light images may be displayed in a selectable manner. For example, when the imaging area includes a first mirror surface and a second mirror surface (two mirror surfaces) (there are two areas where the first distance measurement result and the second distance measurement result are different), at least two of the following may be displayed on the display 46 in a selectable state: an image obtained by capturing the first mirror surface with the first mirror surface in focus; an image obtained by capturing the first image with the image (first image) reflected on the first mirror surface in focus; an image obtained by capturing the second mirror surface with the second mirror surface in focus; and an image obtained by capturing the second image with the image (second image) reflected on the second mirror surface in focus. The number of mirror surfaces included in the imaging area is not limited to two, and may be three or more, but the same processing may be performed in this case as well. In addition, although a mirror surface is exemplified here, the image may be reflected on a shiny surface (for example, a surface with silver plating or a polished metal surface). In this case, too, an image obtained by capturing an image with the glossy surface in focus and an image obtained by capturing an image within the glossy surface with the image reflected on the glossy surface in focus may be selectively displayed on the display 46. The number of glossy surfaces included in the imaging area may be more than one. However, in this case, the same processing can be performed.

[0122] In the above first embodiment, an example has been described in which the first visible light image or the second visible light image is selected by the user from the image selection screen via the touch panel 48 when the first distance and the second distance are different, but the technology of the present disclosure is not limited to this. For example, when the first distance and the second distance are different, the user may be allowed to select in advance which of active ranging, i.e., imaging with focus control based on the ranging result by the first ranging, and passive ranging, i.e., imaging with focus control based on the ranging result by the second ranging, is to be prioritized.

[0123] In this case, instead of the distance measurement imaging process (see FIG. 17A and FIG. 17B) described in the first embodiment, the distance measurement imaging process shown in FIG. 18A to FIG. 18C is executed by the CPU 15A. The distance measurement imaging process shown in FIG. 18A to FIG. 18C is different from the distance measurement imaging process described in the first embodiment in that the processes of steps ST200 to ST222 are newly added. Also, the distance measurement imaging process shown in FIG. 18A to FIG. 18C is different from the distance measurement imaging process described in the first embodiment in that the processes of steps ST110 to ST118 are not included. Also, the distance measurement imaging process shown in FIG. 18A to FIG. 18C is different from the distance measurement imaging process described in the first embodiment in that the processes of steps ST224 and ST226 are included instead of the processes of steps ST132 and ST134. Hereinafter, the distance measurement imaging process shown in FIG. 18A to FIG. 18C will be described in terms of the differences from the distance measurement imaging process described in the first embodiment.

[0124] In the distance measurement imaging process shown in Fig. 18A, first, in step ST200, the execution unit 15A5 causes the display 46 to display a distance measurement method reception screen. The distance measurement method reception screen displays a message that prompts the user to select whether to give priority to active distance measurement, i.e., first distance measurement, or passive distance measurement, i.e., second distance measurement. In the example shown in Fig. 18A, the distance measurement method reception screen displays a message saying "Please select which distance measurement method you would like to give priority to."

[0125] Furthermore, two softkeys are displayed on the distance measurement method reception screen. One softkey is a softkey that is pressed by the user via the touch panel 48 when the user selects to give priority to active distance measurement, and is labeled "active method" in the example shown in Fig. 18A. The other softkey is a softkey that is pressed by the user via the touch panel 48 when the user selects to give priority to passive distance measurement, and is labeled "passive method" in the example shown in Fig. 18A.

[0126] In the next step ST202, the execution unit 15A5 judges whether or not an instruction from the user on the distance measurement method reception screen has been accepted. Here, "an instruction from the user on the distance measurement method reception screen" means that the soft key labeled "active method" or the soft key labeled "passive method" has been pressed by the user via the touch panel 48. In step ST202, if an instruction from the user on the distance measurement method reception screen has not been accepted, the judgment is negative, and the judgment of step ST202 is performed again. In step ST202, if an instruction from the user on the distance measurement method reception screen has been accepted, the judgment is positive, and the distance measurement imaging process proceeds to step ST204.

[0127] In step ST204, the execution unit 15A5 judges whether or not the softkey pressed by the user via the touch panel 48 is a softkey labeled "active method." In step ST204, if the softkey pressed by the user via the touch panel 48 is a softkey labeled "active method," the judgment is affirmative, and the distance measurement imaging process proceeds to step ST100. In step ST204, if the softkey pressed by the user via the touch panel 48 is a softkey labeled "passive method," the judgment is negative, and the distance measurement imaging process proceeds to step ST206 shown in FIG. 18B.

[0128] The process of step ST206 to step ST212 shown in FIG. 18B is the same as the process of step ST100 to step ST106 shown in FIG. 18A. In step ST214 shown in FIG. 18B, the determination unit 15A4 determines whether or not the first distance acquired in step ST208 is different from the second distance acquired in step ST212. In step ST214, if the first distance acquired in step ST208 is the same as the second distance acquired in step ST212, the determination is negative, and the distance measurement imaging process proceeds to step ST130 shown in FIG. 18C. In step ST214, if the first distance acquired in step ST208 is different from the second distance acquired in step ST212, the determination is positive, and the distance measurement imaging process proceeds to step ST216.

[0129] In step ST216, the execution unit 15A5 judges whether or not the imaging start timing has arrived. In step ST216, if the imaging start timing has not arrived, the judgment is negative, and the distance measurement imaging process proceeds to step ST222. In step ST216, if the imaging start timing has arrived, the judgment is positive, and the distance measurement imaging process proceeds to step ST218.

[0130] In step ST218, the execution unit 15A5 executes the same process as the process of step ST132 shown in Fig. 17B, and in the next step ST220, the execution unit 15A5 executes the same process as the process of step ST134 shown in Fig. 17B. In the next step ST222, the execution unit 15A5 judges whether or not the end condition is satisfied. In step ST222, if the end condition is not satisfied, the judgment is negative, and the distance measurement imaging process proceeds to step ST216. In step ST222, if the end condition is satisfied, the judgment is positive, and the distance measurement imaging process ends.

[0131] In step ST130 shown in Fig. 18C, if the imaging start timing has not arrived, the determination is negative, and the distance measurement imaging process proceeds to step ST136. In step ST130 shown in Fig. 18C, if the imaging start timing has arrived, the determination is positive, and the distance measurement imaging process proceeds to step ST224.

[0132] In step ST224, the first distance measurement control section 15A1 controls the distance measuring image pickup device 14 to cause the first distance measurement system processing circuit 34B to perform the first distance measurement.

[0133] In the next step ST226, the execution unit 15A5 executes a first imaging process using the first distance obtained by performing the first distance measurement in step ST224. In addition, the execution unit 15A5 executes a first visible light image display process using the latest first visible light image obtained by performing the first imaging process.

[0134] In step ST136 shown in Fig. 18C, if the end condition is not satisfied, the determination is negative, and the distance measurement imaging process proceeds to step ST130 shown in Fig. 18C. In step ST136 shown in Fig. 18C, if the end condition is satisfied, the determination is positive, and the distance measurement imaging process ends.

[0135] In this manner, when the determination is affirmative in step ST204 shown in Fig. 18A and the determination is affirmative in step ST108 shown in Fig. 18A, the execution unit 15A5 performs the first distance measurement in step ST122 shown in Fig. 18A. Then, in step ST124 shown in Fig. 18A, the first imaging process is performed using the first distance obtained by performing the first distance measurement in step ST122. That is, in step ST124 shown in Fig. 18A, imaging is performed with focus control based on the first distance obtained by performing the first distance measurement in step ST122.

[0136] On the other hand, when the determination is negative in step ST204 shown in Fig. 18A and positive in step ST214 shown in Fig. 18B, the execution unit 15A5 performs second distance measurement in step ST218 shown in Fig. 18B. Then, in step ST220 shown in Fig. 18B, a second imaging process is performed using the second distance obtained by performing the second distance measurement in step ST218. That is, in step ST220 shown in Fig. 18B, imaging is performed with focus control based on the second distance obtained by performing the second distance measurement in step ST218.

[0137] Therefore, according to the ranging imaging process shown in Figures 18A to 18C, when the first distance obtained by performing the first ranging and the second distance obtained by performing the second ranging are different, imaging is always performed under focus control based on the first distance, and the power consumption required for focus control and imaging can be reduced compared to when imaging is performed under focus control based on the second distance.

[0138] 18A to 18C, when the soft key labeled "active method" on the distance measurement method reception screen is pressed by the user via the touch panel 48, focus control is performed based on the first distance in the first imaging process of step ST124 shown in FIG. 18A. When the soft key labeled "passive method" on the distance measurement method reception screen is pressed by the user via the touch panel 48, focus control is performed based on the second distance in the second imaging process of step ST220 shown in FIG. 18B. That is, focus control is performed based on either the first distance or the second distance, which is determined according to an instruction given by the user.

[0139] Therefore, according to the distance measurement imaging process shown in FIGS. 18A to 18C, focus control intended by the user can be easily achieved compared to a case where focus control is not performed based on an instruction given by the user.

[0140] Here, the focus control based on the distance determined according to the instruction given by the user is exemplified, but the technology of the present disclosure is not limited thereto. For example, instead of the instruction given by the user, an instruction given from an external device (e.g., a personal computer and / or a server, etc.) capable of communicating with the smart device 10 may be applied. In this case, for example, the external device may selectively give the smart device 10 an instruction for active distance measurement and an instruction for passive distance measurement depending on the time of day and / or weather, etc.

[0141] In a relatively dark environment such as at night and / or on a cloudy day, it is generally considered that the accuracy of active distance measurement is higher than that of passive distance measurement. Therefore, when an external device selectively gives an instruction for active distance measurement and an instruction for passive distance measurement to the smart device 10 depending on the time of day, for example, the external device may give an instruction for active distance measurement to the smart device 10 at night and an instruction for passive distance measurement to the smart device 10 during the day. Also, when an external device selectively gives an instruction for active distance measurement and an instruction for passive distance measurement to the smart device 10 depending on the weather, for example, the external device may give an instruction for passive distance measurement to the smart device 10 when the weather is sunny and an instruction for active distance measurement to the smart device 10 when the weather is other than sunny. In addition, the external device may be configured to give the smart device 10 an instruction to perform active distance measurement when it is nighttime and the weather is other than sunny, and to give the smart device 10 an instruction to perform passive distance measurement when it is daytime and the weather is sunny.

[0142] In this way, in a case where an external device provides instructions to the smart device 10 without the user providing instructions, for example, in the distance measurement imaging process shown in FIG. 18A, the processing of step ST200 is not necessary, and in step ST202, it is sufficient that the execution unit 15A5 determines whether or not an instruction has been provided from the external device.

[0143] In addition, in the above-described first embodiment, an example is given assuming a situation in which the entire mirror 100 is captured as a subject by the smart device 10, but when the mirror 100 is included in the imaging area, it may not be in line with the user's intention to use the distance measurement result obtained by performing the first distance measurement based on the IR reflected light of the mirror 100 for focus control. In this case, the distance measurement imaging process shown in FIG. 19 may be executed by the CPU 15A.

[0144] The distance measurement imaging process shown in FIG. 19 is different from the distance measurement imaging process shown in FIG. 17A and FIG. 17B in that it has steps ST300 and ST302 between step ST100 and step ST102. Also, the distance measurement imaging process shown in FIG. 19 is different from the distance measurement imaging process shown in FIG. 17A and FIG. 17B in that it does not have steps ST110 to ST118. Also, the distance measurement imaging process shown in FIG. 19 is different from the distance measurement imaging process shown in FIG. 17A and FIG. 17B in that it has step ST108A instead of step ST108. Furthermore, the distance measurement imaging process shown in FIG. 19 is different from the distance measurement imaging process shown in FIG. 17A and FIG. 17B in that it has steps ST304 and ST306 instead of steps ST122 and ST124. Hereinafter, the distance measurement imaging process shown in FIG. 19 will be described in terms of the differences from the distance measurement imaging process described in the first embodiment.

[0145] 19, the acquisition unit 15A3 acquires IR pixel data from the first ranging system processing circuit 34B as data indicating the amount of reflected IR light received by the first ranging system partition area 26N2 (see FIG. 5). Here, for example, the IR pixel data for each IR pixel included in the ROI of the first ranging system partition area 26N2 is acquired by the acquisition unit 15A3.

[0146] In the next step ST302, the judgment unit 15A4 refers to the IR pixel data acquired in step ST300 and determines whether the amount of received light of the IR pixel whose IR pixel data shows the maximum amount of received light is greater than or equal to a threshold value, among the IR pixels included in the ROI of the first ranging system partition area 26N2.

[0147] The threshold value to be compared with the amount of light received by the IR pixel is a value derived in advance by testing an actual device and / or computer simulation, etc., as the amount of light received when the smart device 10 irradiates a laser beam on a mirror surface located at a predetermined distance (e.g., several tens of meters) away, and the IR reflected light obtained by the laser beam being reflected by the mirror surface is received by the IR pixel. Note that, although a mirror surface is exemplified here as the target of irradiation of the laser beam, the present invention is not limited to this, and may be a surface with a lower reflectance than the mirror surface. Examples of surfaces with a lower reflectance than the mirror surface include a glass surface, a surface of a plate-like member made of translucent plastic, and a glossy surface.

[0148] In step ST302, if the amount of received light is less than the threshold, the determination is negative, and the distance measurement imaging process proceeds to step ST130 shown in Fig. 18C. In step ST302, if the amount of received light is equal to or greater than the threshold, the determination is positive, and the distance measurement imaging process proceeds to step ST102.

[0149] In step ST108A, the determination unit 15A4 determines whether or not the first distance acquired in step ST102 is different from the second distance acquired in step ST106. In step ST108A, if the first distance acquired in step ST102 is the same as the second distance acquired in step ST106, the determination is negative, and the distance measurement imaging process proceeds to step ST130 shown in Fig. 18C. In step ST108A, if the first distance acquired in step ST102 is different from the second distance acquired in step ST106, the determination is positive, and the distance measurement imaging process proceeds to step ST120.

[0150] In step ST304, the second distance measurement control section 15A2 controls the distance measurement imaging device 14 to cause the second distance measurement system processing circuit 34C to perform the second distance measurement.

[0151] In the next step ST306, the execution unit 15A5 executes a second imaging process using the second distance obtained by performing the second distance measurement in step ST304. That is, the execution unit 15A5 performs focus control based on the second distance obtained by performing the second distance measurement in step ST304, and causes the photoelectric conversion element 26 to capture an image. In addition, the execution unit 15A5 executes a second visible light image display process using the latest second visible light image obtained by performing the second imaging process.

[0152] In this way, by executing the distance measurement imaging process shown in Figure 19, focus control is performed based on the second distance when the amount of reflected IR light received at the IR pixel is greater than or equal to a threshold value, making it easier to capture an image in the focused state intended by the user (for example, capturing an image with the focus focused on a location other than mirror 100) compared to a case where focus control is performed based on the first distance even though the amount of reflected IR light received at the IR pixel is greater than or equal to the threshold value (for example, a case where the focus is focused on mirror 100).

[0153] In the first embodiment, the determination unit 15A4 determines whether the first distance and the second distance are different regardless of whether the imaging area includes a specific subject, but the technology of the present disclosure is not limited to this. For example, the determination unit 15A4 may determine whether the first distance and the second distance are different on the condition that the imaging area includes a specific subject.

[0154] In this case, the distance measurement imaging process shown in FIG. 20 is executed by the CPU 15A. The distance measurement imaging process shown in FIG. 20 is different from the distance measurement imaging process described in the first embodiment in that it has step ST350 and step ST352 between step ST106 and step ST108. Also, the distance measurement imaging process shown in FIG. 20 is different from the distance measurement imaging process described in the first embodiment in that it has step ST108B instead of step ST108. Furthermore, the distance measurement imaging process shown in FIG. 20 is different from the distance measurement imaging process described in the first embodiment in that it has step ST354 and step ST356 instead of step ST122 and step ST124. Hereinafter, the distance measurement imaging process shown in FIG. 20 will be described in terms of the differences from the distance measurement imaging process described in the first embodiment.

[0155] In step ST350 shown in FIG. 20, the execution unit 15A5 executes a specific subject detection process. The specific subject detection process is a process in which the light receiver 18 performs focus control based on the first distance acquired in step ST102 to capture an image of the imaging area, and detects a specific subject image showing a specific subject from the imaging area image obtained by capturing the image of the imaging area. Note that the imaging area image is an example of a "second image" according to the technology of the present disclosure. Here, the specific subject includes a high reflectance area having a default reflectance or higher. For example, the default reflectance may be a reflectance that is predetermined as an average reflectance of a glossy surface. Here, the mirror surface 100A is adopted as the high reflectance area. In addition, a user image and a smart device image are reflected in the mirror surface 100A.

[0156] In this step ST350, the specific subject image is detected from the imaging area image obtained by performing focus control based on the first distance acquired in step ST102 and imaging the imaging area, but the technology of the present disclosure is not limited to this. For example, the specific subject image may be detected from the image obtained by performing focus control based on the second distance acquired in step ST106 and imaging the imaging area.

[0157] In this step ST350, the execution unit 15A5 detects a specific subject image from the imaging area image using a machine learning algorithm (for example, a neural network). The execution unit 15A5 may also detect a specific subject image from the imaging area image using a dictionary for pattern matching.

[0158] In the next step ST352, the execution unit 15A5 judges whether or not a specific subject image is detected from the imaging area image. In step ST352, if a specific subject image is not detected from the imaging area image, the judgment is negative, and the distance measurement imaging process proceeds to step ST130 shown in Fig. 18C. In step ST352, if a specific subject image is detected from the imaging area image, the judgment is positive, and the distance measurement imaging process proceeds to step ST108B.

[0159] In step ST108B, the determination unit 15A4 determines whether or not the first distance acquired in step ST102 is different from the second distance acquired in step ST106. In step ST108B, if the first distance acquired in step ST102 is the same as the second distance acquired in step ST106, the determination is negative, and the distance measurement imaging process proceeds to step ST130 shown in Fig. 18C. In step ST108B, if the first distance acquired in step ST102 is different from the second distance acquired in step ST106, the determination is positive, and the distance measurement imaging process proceeds to step ST120.

[0160] In step ST354, the second distance measurement control section 15A2 controls the distance measurement imaging device 14 to cause the second distance measurement system processing circuit 34C to perform the second distance measurement.

[0161] In the next step ST356, the execution unit 15A5 executes a second imaging process using the second distance obtained by performing the second distance measurement in step ST354. That is, the execution unit 15A5 performs focus control based on the second distance obtained by performing the second distance measurement in step ST354, and causes the photoelectric conversion element 26 to capture an image. In addition, the execution unit 15A5 executes a second visible light image display process using the latest second visible light image obtained by performing the second imaging process.

[0162] In this way, by executing the distance measurement imaging process shown in Figure 20, focus control is performed based on the second distance when a specific subject image is detected by the execution unit 15A5, making it easier to achieve imaging in the focused state intended by the user (for example, imaging with the image of the user reflected in the mirror 100 in focus) compared to a case where focus control is performed based on the first distance even though a specific subject image is detected (for example, a case where the focus is set on the mirror 100).

[0163] In addition, the specific subject includes a high reflectance area having a reflectance equal to or higher than a predetermined reflectance, and the mirror surface 100A is used as the high reflectance area. In addition, the user image and the smart device image are reflected in the mirror surface 100A. Therefore, according to this configuration, it is easier to realize imaging in the focused state intended by the user (e.g., imaging in a state where the user image reflected in the mirror 100 is in focus) compared to a case where focus control is performed based on the first distance even though the mirror surface 100A in which the user image and the smart device image are reflected is detected (e.g., a case where the focus is set on the mirror surface 100A).

[0164] The distance measurement imaging process shown in Fig. 21 differs from the distance measurement imaging process shown in Fig. 20 in that it has steps ST400 to ST406 instead of steps ST350 and ST352. Hereinafter, the distance measurement imaging process shown in Fig. 21 will be described in terms of differences from the distance measurement imaging process described in Fig. 20.

[0165] 21, the execution unit 15A5 executes a high reflectance region detection process. The high reflectance region detection process is a process in which the light receiver 18 performs focus control based on the first distance acquired in step ST102 to capture an image of the imaging region, and detects a high reflectance region that indicates the above-mentioned high reflectance region in the imaging region image obtained by capturing the image of the imaging region.

[0166] In this step ST400, the high reflectance region is detected from the imaging region image obtained by performing focus control based on the first distance acquired in step ST102 to image the imaging region, but the technology of the present disclosure is not limited thereto. For example, the high reflectance region may be detected from the image obtained by performing focus control based on the second distance acquired in step ST106 to image the imaging region.

[0167] In the next step ST402, the execution unit 15A5 judges whether or not a high reflectance region is detected from the imaging area image. In step ST402, if a high reflectance region is not detected from the imaging area image, the judgment is negative, and the distance measurement imaging process proceeds to step ST130 shown in Fig. 18C. In step ST402, if a high reflectance region is detected from the imaging area image, the judgment is positive, and the distance measurement imaging process proceeds to step ST404.

[0168] In step ST404, the execution unit 15A5 calculates the high reflectance region occupancy rate. The high reflectance region occupancy rate refers to the ratio of the high reflectance region detected by the high reflectance region detection process to the captured region image.

[0169] In the next step ST406, the determination unit 15A4 determines whether or not the high reflectance region occupancy rate calculated in step ST404 exceeds a default occupancy rate (for example, 80%). In step ST406, if the high reflectance region occupancy rate calculated in step ST404 is equal to or less than the default occupancy rate, the determination is denied and the distance measurement imaging process proceeds to step ST130 shown in Fig. 18C. As a result, the processes of steps ST130 to ST136 are executed. That is, the photoreceiver 18 captures an image of the imaging region under focus control based on the first distance obtained by performing the first distance measurement in step ST224.

[0170] Also, in step ST406, if the high reflectance region occupancy rate calculated in step ST404 exceeds the default occupancy rate, the determination is affirmative, and the distance measurement imaging process proceeds to step ST108B. Then, if the first distance acquired in step ST102 and the second distance acquired in step ST106 are different (if the determination is affirmative in step ST108B), the processes of steps ST120 to ST126 are executed. That is, the photoreceiver 18 captures an image of the imaging region under focus control based on the second distance acquired by performing the second distance measurement in step ST354 (step ST356).

[0171] 21, imaging is performed under focus control based on the first distance or the second distance determined according to the high reflectivity region occupancy rate, making it easier to realize imaging in the focused state intended by the user compared to a case where focus control is always performed on the optical receiver 18 based only on either the first distance or the second distance. In other words, it is possible to accurately meet, for example, a user's request to capture an image by focusing on the entire mirror 100 and a user's request to capture an image by focusing on the user's image reflected in the mirror 100.

[0172] In this embodiment, the high reflectance region occupancy rate is the ratio of the high reflectance region detected by the high reflectance region detection process to the image of the image capturing region, but the technology of the present disclosure is not limited to this. For example, the high reflectance region occupancy rate may be the difference between the area of ​​the high reflectance region in the image of the image capturing region and the area of ​​the region other than the high reflectance region in the captured image.

[0173] In the above first embodiment, an example in which the imaging area is imaged only by the photoreceiver 18 has been described, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 22, a smart device 300 may be equipped with a wide-angle photoreceiver 350 in addition to the distance measuring imaging device 14 described above. The smart device 300 differs from the smart device 10 in that it further includes the wide-angle photoreceiver 350.

[0174] The wide-angle receiver 350 is capable of imaging at a wider angle than the receiver 18. In other words, the receiver 18 is capable of imaging at a telephoto side than the wide-angle receiver 350. In other words, the wide-angle receiver 350 has a longer focal length than the receiver 18. Furthermore, the wide-angle receiver 350 includes a light-transmitting window 352 and a photoelectric conversion element 354. The light-transmitting window 352 has the same structure as the light-transmitting windows 20 and 20, and the photoelectric conversion element 354 has an area similar to the visible light image division region 26N1 of the photoelectric conversion element 26.

[0175] In the smart device 300 configured in this manner, the CPU 15A executes the distance measurement imaging process shown in Fig. 23. The distance measurement imaging process shown in Fig. 23 differs from the distance measurement imaging process described in the first embodiment in that it includes step ST450 and does not include steps ST110 to ST118. Below, the distance measurement imaging process shown in Fig. 23 will be described in terms of the differences from the distance measurement imaging process described in the first embodiment. In addition, below, the distance measurement imaging process shown in Fig. 23 will be described on the premise that imaging is performed by the photoreceiver 18 or the wide-angle photoreceiver 350.

[0176] In the distance measurement imaging process shown in FIG. 23, first, in step ST450, the determination unit 15A4 determines whether the wide-angle light receiver 350 is in use. That is, in this step ST450, the determination unit 15A4 determines whether imaging is being performed by the wide-angle light receiver 350. In step ST450, if the light receiver 18 is in use, the determination is negative, and the distance measurement imaging process proceeds to step ST206 shown in FIG. 18B. In step ST450, if the wide-angle light receiver 350 is in use, the determination is positive, and the distance measurement imaging process proceeds to step ST100. In this case, on the condition that the determination is positive in step ST108, the processes of steps ST120 to ST126 are executed. As a result, in step ST124, imaging is performed under focus control based on the first distance obtained by performing the first distance measurement in step ST122.

[0177] Here, for example, when the entire mirror 100 is imaged as the subject, it is assumed that the image will be captured at a wider angle than the telephoto side so that the entire mirror 100 is included in the imaging area. Therefore, in order to accurately measure the distance to the mirror surface 100A, it is desirable to perform the first distance measurement, which is more advantageous for focusing on the mirror surface 100A than the second distance measurement, which is more advantageous for focusing on the user image reflected in the mirror 100.

[0178] Therefore, by executing the distance measurement imaging process shown in Figure 23, while the wide-angle photoreceiver 350 is in use, imaging is performed under focus control based on the first distance, making it easier to achieve imaging in the focused state intended by the user compared to a case in which imaging is performed under focus control based on the second distance even though imaging is performed at a wider angle than the photoreceiver 18.

[0179] In the first embodiment, the light irradiator 16 uses laser light as light for distance measurement, but the technology of the present disclosure is not limited to this. For example, the light for distance measurement may be light having directionality such as superluminescent light, light emitted from a xenon flash light source, or light emitted from an LED.

[0180] In the first embodiment, the visible light image section 26N1, the first distance measurement system section 26N2, and the second distance measurement system section 26N3 are integrated into one chip by the photoelectric conversion element 26, but the technology of the present disclosure is not limited to this. A plurality of visible light pixels may be integrated into one chip, a plurality of phase difference pixels may be integrated into one chip, and a plurality of IR pixels may be integrated into one chip. A plurality of visible light pixels and a plurality of phase difference pixels may be integrated into one chip, and a plurality of IR pixels may be integrated into one chip. In this way, when different types of photosensitive pixels are integrated into chips for each type and mounted on the smart device 10, an optical system such as an objective lens, a focus lens, and an aperture may be provided on the subject side (object side) of each chip.

[0181] In the above first embodiment, the distance measuring imaging device 14 is built into the smart device 10, but the technology of the present disclosure is not limited to this. For example, the distance measuring imaging device 14 may be externally attached to a general smart device, that is, a smart device that does not have a built-in distance measuring imaging device 14.

[0182] In the above first embodiment, an example has been described in which the UI device 44 is incorporated in the smart device 10, but at least some of the components included in the UI device 44 may be externally attached to the smart device 10. In addition, at least some of the components included in the UI device 44 may be used by being connected to the external I / F 52 as separate entities.

[0183] 1 illustrates a smart device 10, but the technology of the present disclosure is not limited thereto. That is, the technology of the present disclosure can be applied to various electronic devices (e.g., interchangeable lens cameras, fixed lens cameras, personal computers, and / or wearable terminal devices) incorporating a distance measuring imaging device 14, and the same actions and effects as those of the smart device 10 can be obtained with these electronic devices.

[0184] In the first embodiment, the display 46 is exemplified, but the technology of the present disclosure is not limited to this. For example, a display externally attached to the smart device 10 may be used in combination with the display 46.

[0185] In the first embodiment, the photoelectric conversion element 26 and the signal processing circuit 34 are separate, but a stacked imaging element in which the photoelectric conversion element 26 and the signal processing circuit 34 are integrated into a single chip may be used. At least a part of the signal processing circuit 34 may be eliminated, and the function of the signal processing circuit 34 may be assigned to the CPU 15A.

[0186] In the above first embodiment, an example is described in which an imaging timing signal is supplied from the photoelectric conversion element driver 32 to the photoelectric conversion element 26, but the technology of the present disclosure is not limited to this. For example, the photoelectric conversion element driver 32 may be omitted, in which case the function of the photoelectric conversion element driver 32 may be assigned to the CPU 15A.

[0187] In the above first embodiment, an example in which the second distance measurement is performed using phase difference pixels has been described, but the technology of the present disclosure is not limited thereto. For example, instead of distance measurement using phase difference pixels, distance measurement using a stereo camera or distance measurement using face detection may be performed. In distance measurement using a stereo camera, the distance to the subject is measured by using the parallax between a pair of images obtained from the stereo camera. In distance measurement using face detection, the distance to the subject is measured by using the ratio of the size of the detected face image to the size of one frame of image, or the like.

[0188] In addition, in the above first embodiment, an example was described in which the G filter, R filter, and B filter also function as infrared light cut filters that cut infrared light, but the technology of the present disclosure is not limited to this. Each color filter corresponding to each of the R pixels, G pixels, and B pixels may be a color filter that also transmits infrared light, and a pair of photodiodes, one for a visible light pixel and one for an IR pixel (e.g., an InGaAs APD), may be arranged for each color filter.

[0189] In the above first embodiment, the visible light image division region 26N1 and the second distance measurement system division region 26N3 are used in combination, but the technology of the present disclosure is not limited thereto. For example, instead of the visible light image division region 26N1 and the second distance measurement system division region 26N3, an area sensor may be used in which visible light pixel data and phase difference pixel data are selectively generated and read out. In this case, a plurality of photosensitive pixels are two-dimensionally arranged in the area sensor. For example, a pair of independent photodiodes without a light shielding member is used as the photosensitive pixels included in the area sensor. When visible light pixel data is generated and read out, photoelectric conversion is performed by the entire region of the photosensitive pixels (a pair of photodiodes), and when phase difference pixel data is generated and read out (for example, when performing passive distance measurement), photoelectric conversion is performed by one of the pair of photodiodes. Here, one of the pair of photodiodes is a photodiode corresponding to the first phase difference pixel L described in the first embodiment, and the other of the pair of photodiodes is a photodiode corresponding to the second phase difference pixel R described in the first embodiment. Note that, although the visible light pixel data and the phase difference pixel data may be selectively generated and read out by all the photosensitive pixels included in the area sensor, the present invention is not limited thereto, and the visible light pixel data and the phase difference pixel data may be selectively generated and read out by a part of the photosensitive pixels included in the area sensor.

[0190] [Second embodiment] In the first embodiment, an example was given in which a specific process is performed by the execution unit 15A5 when the first distance obtained by performing the first distance measurement and the second distance obtained by performing the second distance measurement are different, but in the second embodiment, an example will be described in which a specific process is performed by the execution unit 15A5 when the first focus position and the second focus position obtained by different focusing methods are different. Note that in the second embodiment, the same components as those described in the first embodiment are denoted by the same reference numerals and their description will be omitted, and only the parts different from the first embodiment will be described.

[0191] 24, a distance measurement imaging processing program 570 is stored in the storage 15B. The CPU 15A reads out the distance measurement imaging processing program 570 from the storage 15B. The CPU 15A then executes the distance measurement imaging processing program 570 read out from the storage 15B to operate as a first distance measurement control unit 15A1, an acquisition unit 15A3, a determination unit 15A4, an execution unit 15A5, a first focus position calculation unit 15A6, a contrast AF method imaging control unit 15A7, and a second focus position calculation unit 15A8.

[0192] As an example, as shown in FIG. 25, the first distance measurement control unit 15A1 causes the first distance measurement system processing circuit 34B to perform the first distance measurement, and the acquisition unit 15A3 acquires the first distance from the first distance measurement system processing circuit 34B, as in the first embodiment. The first focus position calculation unit 15A6 calculates the first focus position at which the subject included in the imaging area is focused, based on the first distance acquired by the acquisition unit 15A3. Specifically, the first focus position calculation unit 15A6 calculates the first focus position using a predetermined arithmetic expression in which the distance from the smart device 10 to the subject is an independent variable and the focus position is a dependent variable. Note that the technology of the present disclosure is not limited to this, and the first focus position calculation unit 15A6 may derive the first focus position using a focus position derivation table 72 shown in FIG. 13.

[0193] As an example, as shown in Fig. 26, when the contrast AF imaging control unit 15A7 causes the optical receiver 18 to capture an image by the contrast AF method, it outputs an imaging start signal to the optical receiver 18 and the second focus position calculation unit 15A8. By outputting the imaging start signal to the optical receiver 18, the contrast AF imaging control unit 15A7 causes the optical receiver 18 to capture an image at a default frame rate (e.g., 60 fps) in the visible light image division region 26N1 (see Fig. 5) and causes the visible light pixel data processing circuit 34A to generate a third visible light image indicating the imaging region. The visible light pixel data processing circuit 34A outputs the third visible light image to the image memory 42. The third visible light image is stored in the image memory 42, and the third visible light image in the image memory 42 is updated every time the third visible light image is input from the visible light pixel data processing circuit 34A.

[0194] In addition, in parallel with the output of the imaging start signal, the contrast AF imaging control unit 15A7 outputs a motor control signal to the photoreceiver 18 and the second focus position calculation unit 15A8. The motor control signal is a signal for causing the focus lens 30B to reciprocate (wobble) along the optical axis L2, and is input to a motor driver 64 of the photoreceiver 18. The motor driver 64 drives the motor 62 according to the input motor control signal to cause the focus lens 30B to wobble (see FIG. 8). Meanwhile, the second focus position calculation unit 15A8 calculates the current position of the focus lens 30B on the optical axis L2 using the motor control signals input from the input of the imaging start signal from the contrast AF imaging control unit 15A7 to the present time.

[0195] The contrast AF imaging control unit 15A7 acquires the third visible light image from the image memory 42 each time the third visible light image in the image memory 42 is updated, and calculates a contrast value of the acquired third visible light image. The contrast AF imaging control unit 15A7 then searches for the maximum value of the contrast of the third visible light image, and outputs a maximum value reached signal to the second focus position calculation unit 15A8 when the maximum value is detected. The maximum value reached signal is a signal indicating that the contrast of the third visible light image has reached its maximum value.

[0196] When the maximum value arrival signal is input, the second focus position calculation unit 15A8 calculates the current position of the focus lens 30B on the optical axis L2 as the second focus position using the motor control signal input from the input of the imaging start signal to the current time. The second focus position refers to a focus position at which a subject included in the imaging area is focused. Here, the second focus position calculation unit 15A8 calculates the second focus position using an arithmetic expression in which the motor control signal is an independent variable and the focus position is a dependent variable. Note that the technology of the present disclosure is not limited to this, and the second focus position calculation unit 15A8 may derive the second focus position using a table in which time series data of the motor control signal is associated with the position of the focus lens 30B on the optical axis L2.

[0197] As an example, as shown in Fig. 27, the determination unit 15A4 determines whether the first focus position calculated by the first focus position calculation unit 15A6 and the second focus position calculated by the second focus position calculation unit 15A8 are different. If the determination unit 15A4 determines that the first focus position and the second focus position are the same, the execution unit 15A5 executes the first imaging process described in the first embodiment. If the determination unit 15A4 determines that the first focus position and the second focus position are different, the execution unit 15A5 executes a specific process (see Fig. 28).

[0198] As an example, as shown in FIG. 28, the specific process is different from the first embodiment in that it includes a third imaging process instead of the second imaging process. The third imaging process refers to a process of performing imaging with focus control for the second focus position (imaging with contrast AF). In addition, the image selection screen generated by the image selection screen display process included in the specific process and displayed on the display 46 has some display contents different from those of the first embodiment. That is, the image selection screen displays the first focus position (in the example shown in FIG. 28, a numerical value "X1 mm" indicating the distance from the reference position to the focus lens 30B) instead of the first distance, and displays the second focus position (in the example shown in FIG. 28, a numerical value "X2 mm" indicating the distance from the reference position to the focus lens 30B) instead of the second distance. The reference position refers to, for example, the position of the imaging plane of the photoelectric conversion element 26, or the position of the focus lens 30B in a state where the focus is adjusted to infinity. In addition, the image selection screen displays a third visible light image instead of the second visible light image. Also, instead of the message "The distance measurement results are different," the image selection screen now displays the message "The focus position is different." Also, instead of the message "Active method," the image selection screen now displays the message "Active AF method." Furthermore, instead of the message "Passive method," the image selection screen now displays the message "Contrast AF method."

[0199] In the second embodiment, the display examples of "active AF method" and "contrast AF method" are shown, but the display does not necessarily have to be "active AF method" and "passive AF method", and any display may be used as long as the user can understand the difference between the AF methods. For example, when performing AF using laser distance measurement and contrast AF, the display may be "AF using laser distance measurement" and "AF using contrast", or an icon representing the AF method may be displayed. In addition, instead of displaying the AF method, the focus position may be displayed. For example, "focus position: near" and "focus position: far" may be displayed, or "focus position: object" and "focus position: image reflected on object" may be displayed. Alternatively, two or more characters and icons indicating the AF method and the focus position may be displayed in combination.

[0200] 29A to 29C, the operation of the smart device 500 according to the technique of the present disclosure will be described. Note that Fig. 29A to Fig. 29C are flowcharts showing an example of the flow of distance measurement imaging processing executed by the CPU 15A according to the distance measurement imaging processing program 570.

[0201] 29A, first, in step ST500, the first distance measurement control unit 15A1 controls the first distance measurement system processing circuit 34B to perform the first distance measurement by controlling the distance measurement image pickup device 14. The first distance measurement is realized by the first distance measurement control unit 15A1 causing the first distance measurement system processing circuit 34B to calculate the first distance.

[0202] In the next step ST502, the acquisition unit 15A3 acquires the first distance obtained by performing the first distance measurement in step ST500.

[0203] In the next step ST504, the first focusing position calculation section 15A6 calculates the first focusing position based on the first distance acquired in step ST502.

[0204] In the next step ST506, the contrast AF imaging control unit 15A7 causes the photoreceiver 18 to start imaging by the contrast AF method.

[0205] In the next step ST508, the contrast AF imaging control unit 15A7 acquires the third visible light image generated by the visible light pixel data processing circuit 34A through imaging by the contrast AF method.

[0206] In the next step ST510, the second focus position calculation unit 15A8 judges whether or not a maximum value reaching signal has been input from the contrast AF imaging control unit 15A7. If a maximum value reaching signal has not been input from the contrast AF imaging control unit 15A7 in step ST510, the judgment is negative and the distance measurement imaging process proceeds to step ST508. If a maximum value reaching signal has been input from the contrast AF imaging control unit 15A7 in step ST510, the judgment is positive and the distance measurement imaging process proceeds to step ST512.

[0207] In step ST512, the second focus position calculation unit 15A8 calculates the current position of the focus lens 30B on the optical axis L2 as the second focus position, using the motor control signal input from when the processing of step ST506 is executed to the present time.

[0208] 29B, the determination unit 15A4 determines whether or not the first in-focus position calculated in step ST504 is different from the second in-focus position calculated in step ST512. In step ST514, if the first in-focus position calculated in step ST504 is the same as the second in-focus position calculated in step ST512, the determination is negative, and the distance measurement imaging process proceeds from step ST514 to step ST516.

[0209] In step ST516, the execution unit 15A5 executes the first imaging process using the first distance acquired in step ST502. Note that here, the first imaging process using the first distance refers to, for example, a process of performing imaging accompanied by focus control for the first focusing position calculated based on the first distance in step ST504.

[0210] In the next step ST518, the execution unit 15A5 executes a third imaging process using the second focusing position calculated in step ST512. Note that here, the first imaging process using the second focusing position refers to a process of performing imaging accompanied by focus control for the second focusing position.

[0211] In the next step ST520, the execution section 15A5 executes an image selection screen display process, whereby an image selection screen is displayed on the display 46, as shown in FIG.

[0212] In the next step ST522, the execution unit 15A5 determines whether or not either the first visible light image or the third visible light image displayed in the image selection screen has been selected by the user via the touch panel 48. If either the first visible light image or the third visible light image displayed in the image selection screen has not been selected by the user via the touch panel 48 in step ST522, the determination is negative, and the determination of step ST522 is made again. If either the first visible light image or the third visible light image displayed in the image selection screen has been selected by the user via the touch panel 48 in step ST522, the determination is positive, and the distance measurement imaging process proceeds to step ST524.

[0213] In step ST524, the execution unit 15A5 determines whether or not the image selected from within the image selection screen is the third visible light image. If the image selected from within the image selection screen is the first visible light image in step ST524, the determination is negative, and the distance measurement imaging process proceeds to step ST532 shown in Fig. 29C. If the image selected from within the image selection screen is the third visible light image in step ST524, the determination is positive, and the distance measurement imaging process proceeds to step ST526.

[0214] In step ST526, the execution unit 15A5 judges whether or not the imaging start timing has arrived. In step ST526, if the imaging start timing has not arrived, the judgment is negative and the distance measurement imaging process proceeds to step ST530. In step ST526, if the imaging start timing has arrived, the judgment is positive and the distance measurement imaging process proceeds to step ST528.

[0215] In step ST528, the execution unit 15A5 executes a third imaging process. Furthermore, the execution unit 15A5 executes a third visible light image display process using the latest third visible light image obtained by executing the third imaging process. Here, the third visible light image display process differs from the second visible light image display process shown in Fig. 16 in that the third visible light image is displayed on the display 46 instead of the second visible light image, and that a message "contrast AF method" is displayed on the display 46 instead of the message "passive method."

[0216] In step ST528, an example is given in which the third imaging process is performed on condition that the third visible light image is selected in steps ST522 and ST524, and the latest third visible light image obtained by performing the third imaging process is displayed on the display 48, but this is merely one example. For example, on condition that the third visible light image is selected, the selected third visible light image may be stored in a storage medium such as the storage 15B and / or a memory card. Furthermore, imaging using the selected AF method and saving the selected image (for example, the first visible light image or the third visible light image) may be selected according to an instruction received by the receiving device 47 (an instruction given by the user).

[0217] In the next step ST530, the execution unit 15A5 judges whether or not the end condition is satisfied. If the end condition is not satisfied in step ST530, the judgment is negative, and the distance measurement imaging process proceeds to step ST526. If the end condition is satisfied in step ST530, the judgment is positive, and the distance measurement imaging process ends.

[0218] 29C, the execution unit 15A5 judges whether or not the imaging start timing has arrived. If the imaging start timing has not arrived in step ST532, the judgment is negative, and the distance measurement imaging process proceeds to step ST536. If the imaging start timing has arrived in step ST532, the judgment is positive, and the distance measurement imaging process proceeds to step ST534.

[0219] In step ST534, the first distance measurement control section 15A1 controls the distance measurement imaging device 14 to cause the first distance measurement system processing circuit 34B to perform the first distance measurement.

[0220] In the next step ST536, the execution unit 15A5 executes a first imaging process using the first distance obtained by performing the first distance measurement in step ST534. In addition, the execution unit 15A5 executes a first visible light image display process using the latest first visible light image obtained by performing the first imaging process.

[0221] In the next step ST538, the execution unit 15A5 judges whether or not the end condition is satisfied. If the end condition is not satisfied in step ST538, the judgment is negative, and the distance measurement imaging process proceeds to step ST532. If the end condition is satisfied in step ST538, the judgment is positive, and the distance measurement imaging process ends.

[0222] As described above, in the smart device 500, when the first focusing position calculated based on the first distance obtained by performing the first distance measurement differs from the second focusing position obtained by performing imaging using the contrast AF method, the execution unit 15A5 executes a specific process. Therefore, this configuration can contribute to solving various problems that occur when the first distance and the second distance differ.

[0223] Note that the distance measurement imaging process described in the second embodiment is merely an example, and the technical idea included in the distance measurement imaging process described in the second embodiment may be applied to, for example, the distance measurement imaging process shown in Figures 18A to 18C, the distance measurement imaging process shown in Figure 19, the distance measurement imaging process shown in Figure 20, the distance measurement imaging process shown in Figure 21, or the distance measurement imaging process shown in Figure 23. In this case, within a range that does not deviate from the gist of the invention, the first distance may be read as the first focus position, the second distance may be read as the second focus position, unnecessary steps may be deleted, new steps may be added, or the processing order may be changed.

[0224] In the above first embodiment, the distance measurement imaging processing program 70 is stored in the storage 15B, and in the above second embodiment, the distance measurement imaging processing program 570 is stored in the storage 15B. However, the technology of the present disclosure is not limited to this. For example, as shown in Fig. 30, the distance measurement imaging processing program 70 or 570 (hereinafter, when there is no need to distinguish between the distance measurement imaging processing programs 70 and 570, they will be referred to as "distance measurement imaging processing programs" without any reference numerals) may be stored in a storage medium 900. An example of the storage medium 900 is any portable storage medium such as an SSD or a USB memory.

[0225] The distance measurement imaging processing program stored in the storage medium 900 is installed in the controller 15. The CPU 15A executes the distance measurement imaging processing in accordance with the distance measurement imaging processing program.

[0226] In addition, the distance measurement imaging processing program may be stored in a memory unit of another computer or server device connected to the controller 15 via a communication network (not shown), and the distance measurement imaging processing program may be downloaded and installed in the controller 15 in response to a request from the smart device 10 described above.

[0227] It is not necessary to store the entire distance measurement imaging processing program in a storage unit of another computer or server device connected to the controller 15, or in the storage 15B, and only a part of the distance measurement imaging processing program may be stored therein.

[0228] In the example shown in FIG. 30, an example is shown in which the controller 15 is built into the smart device 10, but the technology of the present disclosure is not limited to this, and for example, the controller 15 may be provided outside the smart device 10.

[0229] 30, the CPU 15A is a single CPU, but it may be a plurality of CPUs. Also, a GPU may be applied in place of the CPU 15A.

[0230] 30 illustrates a controller 15, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the controller 15. Also, instead of the controller 15, a combination of a hardware configuration and a software configuration may be used.

[0231] As the hardware resource for executing the distance measurement imaging process described in each of the above embodiments, the following various processors can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the distance measurement imaging process by executing software, i.e., a program. Another example of the processor is a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically for executing specific processing, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes the distance measurement imaging process by using the memory.

[0232] The hardware resource for executing the distance measurement imaging process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the distance measurement imaging process may be a single processor.

[0233] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the distance measurement imaging process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute the distance measurement imaging process in a single IC chip, as typified by SoC. In this way, the distance measurement imaging process is realized using one or more of the above various processors as hardware resources.

[0234] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above-mentioned distance measurement and imaging process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0235] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0236] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0237] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0238] The following supplementary notes are further disclosed regarding the above embodiment.

[0239] (Appendix 1) A processor; A memory connected to or embedded in the processor; The processor is measuring a distance to the imaging region based on an irradiation timing at which a light irradiator irradiates light toward the imaging region and a light receiving timing at which a light receiver receives reflected light of the light from the imaging region; deriving a first focusing position at which a subject included in the imaging area is focused based on the distance; deriving a second focusing position at which the object is focused on based on an image obtained by capturing an image of the imaging area by an imaging device; Executing a specific process when the first focus position and the second focus position are different Information processing device.

[0240] (Appendix 2) The information processing device according to claim 1, wherein the specific process includes a process of notifying that the first focusing position and the second focusing position are different.

[0241] (Appendix 3) The information processing device according to claim 1 or 2, wherein the specific process includes a process of causing the imaging device to capture an image of the imaging area at each of the first focusing position and the focusing position.

[0242] (Appendix 4) The information processing device described in Appendix 3, wherein the specific processing includes a process of displaying on a display a first focused image obtained by capturing an image of the imaging area at the first focusing position and a second focused image obtained by capturing an image of the imaging area at the second focusing position, and prompting a user to select at least one of the first focused image and the second composite image while the first focused image and the second composite image are displayed on the display.

[0243] (Appendix 5) The information processing device according to claim 2 or 3, wherein the specific processing includes a processing for causing the imaging device to capture an image of the imaging area at the first focusing position, or a processing for causing the imaging device to capture an image of the imaging area at the second focusing position.

[0244] (Appendix 6) The imaging device is capable of imaging on a wide-angle side and a telephoto side, 6. The information processing device according to claim 5, wherein when the imaging device captures an image on the wide-angle side, the specific process is a process of causing the imaging device to capture an image at the first focusing position.

[0245] (Appendix 7) 6. The information processing device according to claim 5, wherein the specific process is a process of causing the imaging device to capture an image at the second focusing position when an amount of the reflected light received by the optical receiver is equal to or greater than a threshold value. [Explanation of symbols]

[0246] 10,300,500 smart devices 12. Chassis 12A back 12B Front 13 Command keys 14 Distance measuring imaging device 15 Controller 15A CPU 15A1 First distance measurement control section 15A2 Second distance measurement control section 15A3 Acquisition Department 15A4 Judgment section 15A5 Executive Department 15A6 1st focus position calculation section 15A7 Contrast AF imaging control unit 15A8 Second focus position calculation section 15B Storage 15C Memory 16 Light irradiator 17A, 17B Light shielding member 18 Receiver 19 Microlenses 20,22,352 Translucent window 21 Beam Expander 23 Collimating Lens 24LD 25 LD driver 26,354 Photoelectric conversion element 26N1 Segmented area for visible light images 26N2 1st ranging system division area 26N3 2nd ranging system division area 27 TOF Camera 30A Objective Lens 30B Focus Lens 30C Aperture 31 Focusing control mechanism 32 Photoelectric conversion element driver 34 Signal Processing Circuit 34A Visible light pixel data processing circuit 34B First distance measurement processing circuit 34C Second distance measurement processing circuit 40 Input / Output Interface 41 Imaging lens 42 Image Memory 44 UI Devices 46 Display 47 Reception Device 48 Touch Panel 50 Bus 52 External I / F 53 Hard key section 54 Communication I / F 56 Network 59 Touch Panel Display 60 Moving mechanism 62 Motor 64 Motor Driver 70,570 Distance measurement and imaging processing program 72 Focus position derivation table 100 mirror 100A mirror surface 300L left area passing light 300R right area passing light 350 Wide-angle receiver 900 Storage medium L 1st phase detection pixel L1,L2 optical axis N Non-phase detection pixels PD Photodiode R 2nd phase detection pixel α deviation amount θ1 Angle of view θ2 Irradiation angle

Claims

1. A processor; A memory connected to or embedded in the processor; The processor, performing a first distance measurement to measure a distance to the imaging region based on an irradiation timing at which a light irradiator irradiates light toward an imaging region and a light receiving timing at which a light receiver receives reflected light of the light from the imaging region; performing a second distance measurement to measure a distance to the imaging region based on a first image obtained by imaging the imaging region with an imaging device; executing a specific process when a first distance measurement result obtained by performing the first distance measurement and a second distance measurement result obtained by performing the second distance measurement are different; the specific processing includes a processing of notifying that the first distance measurement result and the second distance measurement result are different, and a processing of performing focus control on the imaging device and imaging the imaging area based on the first distance measurement result or the second distance measurement result, When the amount of the reflected light received by the light receiver is equal to or greater than a threshold, the focus control is performed based on the second distance measurement result. Information processing device.

2. The specific process includes a process of notifying the user that the first distance measurement result and the second distance measurement result are different before performing focus adjustment. The information processing device according to claim 1 .

3. performing a first distance measurement to measure a distance to the imaging region based on an irradiation timing at which a light irradiator irradiates light toward the imaging region and a light receiving timing at which a light receiver receives reflected light of the light from the imaging region; performing a second distance measurement to measure a distance to the imaging region based on a first image obtained by imaging the imaging region with an imaging device; and executing a specific process when a first distance measurement result obtained by performing the first distance measurement and a second distance measurement result obtained by performing the second distance measurement are different; the specific processing includes a processing of notifying that the first distance measurement result and the second distance measurement result are different, and a processing of performing focus control on the imaging device and imaging the imaging area based on the first distance measurement result or the second distance measurement result, When the amount of the reflected light received by the light receiver is equal to or greater than a threshold, the focus control is performed based on the second distance measurement result. Information processing methods.

4. A program for causing a computer to execute information processing, The information processing includes: performing a first distance measurement to measure a distance to the imaging region based on an irradiation timing at which a light irradiator irradiates light toward the imaging region and a light receiving timing at which a light receiver receives reflected light of the light from the imaging region; performing a second distance measurement to measure a distance to the imaging region based on a first image obtained by imaging the imaging region with an imaging device; and executing specific processing including a process of notifying the imaging device of a difference between the first distance measurement result and the second distance measurement result when the first distance measurement result and the second distance measurement result differ, and a process of performing focus control on the imaging device based on the first distance measurement result or the second distance measurement result and imaging the imaging area, When the amount of the reflected light received by the light receiver is equal to or greater than a threshold value, The control is performed based on the second distance measurement result. program.

Images