Information processing device, information processing method, and program

Abstract

The present invention provides an information processing device, an information processing method, and a program that can contribute to solving various problems that arise when the first distance measurement result and the second distance measurement result obtained by different distance measurement methods differ. [Solution] The information processing device includes a processor and a memory connected to or built into the processor. The processor performs a first distance measurement to measure the distance to the imaging area based on the irradiation timing when a light irradiator irradiates light toward the imaging area and the light reception timing when a light receiver receives reflected light from the imaging area. It then performs a second distance measurement to measure the distance to the imaging area based on a first image obtained when the imaging area is imaged by an imaging device. If the first distance measurement result obtained by performing the first distance measurement is different from the second distance measurement result obtained by performing the second distance measurement, the processor performs a specific process.

Description

Technical Field

[0001] The technology of the present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Patent Document 1 discloses an optical sensor including a light emitting element that emits light in one direction along an optical axis, a plurality of light receiving elements that are avalanche photodiodes arranged in a direction perpendicular to the optical axis, a housing that separates a detection object and the light emitting element, and a control unit that specifies a distance by a TOF method and a triangulation method.

[0003] Patent Document 2 discloses a vehicle radar device that, when there is a human near the vehicle, performs scan ranging with a small light emission power so as not to harm the human, memorizes the direction in which the human exists, and then, when performing scan ranging with a large light emission power, stops light emission in the direction in which the human is memorized in advance and performs scan ranging with a large light emission power only in other directions, so that the presence of a vehicle existing far away can be surely detected without harming the human existing nearby.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

[0005] One embodiment of the technology according to the present disclosure provides an information processing apparatus, an information processing method, and a program that can contribute to solving various problems that occur when a first distance measurement result and a second distance measurement result obtained by different distance measurement methods are different.

Means for Solving the Problems

[0006] A first aspect of the technology of this disclosure is an information processing device that includes a processor and a memory connected to or built into the processor, wherein the processor performs a first distance measurement to measure the distance to the imaging area based on the irradiation timing when a light irradiator irradiates light toward the imaging area and the light reception timing when a light receiver receives reflected light from the imaging area, performs a second distance measurement to measure the distance to the imaging area based on a first image obtained when the imaging area is imaged by an imaging device, and performs a specific process when the first distance measurement result obtained by performing the first distance measurement and the second distance measurement result obtained by performing the second distance measurement are different.

[0007] A second aspect of the technology of this disclosure is an information processing device according to the first aspect, wherein a specific process includes a process for notifying that a first distance measurement result and a second distance measurement result are different.

[0008] A third aspect of the technology of this disclosure is an information processing device according to the first or second aspect, wherein a specific process includes a process that controls the focus of an imaging device based on a first distance measurement result and a second distance measurement result, respectively, and causes the imaging area to be imaged.

[0009] A fourth aspect of the technology of this disclosure is an information processing device according to the third aspect, wherein a specific process causes a first in-focus image obtained by imaging an imaging area under focus control based on a first distance measurement result and a second in-focus image obtained by imaging an imaging area under focus control based on a second distance measurement result to be displayed on a display, and prompts the user to select at least one of the first in-focus image and the second in-focus image while the first in-focus image and the second in-focus image are displayed on the display.

[0010] A fifth aspect of the technology of this disclosure is an information processing device according to the first or second aspect, wherein a specific process includes a process that controls the focus of an imaging device based on a first distance measurement result or a second distance measurement result, and causes the imaging area to be imaged.

[0011] A sixth aspect of the technology of this disclosure is an information processing device according to the fifth aspect, wherein focus control is performed on an imaging device based on a distance measurement result, which is determined in accordance with a given instruction, from among a first distance measurement result and a second distance measurement result.

[0012] A seventh aspect of the technology of this disclosure is an information processing device according to a fifth aspect, wherein the imaging device is capable of imaging at both the wide-angle and telephoto ends, and when the imaging device is imaging at the wide-angle end, focus control is performed on the imaging device based on a first distance measurement result.

[0013] An eighth aspect of the technology of this disclosure is an information processing device according to a fifth aspect, wherein focus control is performed based on a second distance measurement result when the amount of reflected light received by a photodetector is equal to or greater than a threshold.

[0014] A ninth aspect of the technology of this disclosure is an information processing device according to the fifth aspect, wherein focus control is performed on the imaging device based on a distance measurement result, which is determined according to either the ratio of high-reflectance regions having a reflectance greater than or equal to a predetermined reflectance to a second image obtained by imaging the imaging area by the imaging device, or the difference between the area of ​​the high-reflectance regions in the second image and the area of ​​regions in the second image that are different from the high-reflectance regions.

[0015] A tenth aspect of the technology of this disclosure is an information processing device according to a fifth aspect, wherein the processor detects a specific subject image representing a specific subject in a second image obtained when an imaging area is imaged by an imaging device, and when a specific subject image is detected by the processor, focus control is performed based on the second distance measurement result.

[0016] An eleventh aspect of the technology of this disclosure is an information processing device according to the tenth aspect, wherein a specific subject includes a high-reflectance region having a reflectance greater than or equal to a predetermined reflectance, and the high-reflectance region is a region in which the image of at least one of a person and an imaging device is reflected.

[0017] A twelfth aspect of the technology of this disclosure is an information processing device that includes a processor and a memory connected to or built into the processor, wherein the processor measures the distance to the imaging area based on the irradiation timing when a light irradiator irradiates light toward the imaging area and the reception timing when a light receiver receives reflected light from the imaging area, derives a first focus position in which the object included in the imaging area is in focus based on the distance, derives a second focus position in which the object is in focus based on an image obtained when the imaging area is imaged by an imaging device, and performs a specific process when the first focus position and the second focus position are different.

[0018] A thirteenth aspect of the technology of this disclosure is an information processing method that includes: performing a first distance measurement based on the irradiation timing at which a light irradiator irradiates light toward the imaging area and the reception timing at which a light receiver receives reflected light from the imaging area; performing a second distance measurement based on a first image obtained when the imaging area is imaged by an imaging device; and performing a specific process when the first distance measurement result obtained by performing the first distance measurement and the second distance measurement result obtained by performing the second distance measurement are different.

[0019] A fourteenth aspect of the technology of this disclosure is an information processing method that includes measuring the distance to an imaging area based on the irradiation timing at which a light irradiator irradiates light toward the imaging area and the reception timing at which a light receiver receives reflected light from the imaging area; deriving a first focus position that is in focus on a subject included in the imaging area based on the distance; deriving a second focus position that is in focus on the subject based on an image obtained when the imaging area is imaged by an imaging device; and performing a specific process when the first focus position and the second focus position are different.

[0020] A fifteenth aspect of the technology of the present disclosure is to cause a computer to perform a first distance measurement for measuring the distance to an imaging region based on the irradiation timing when a light irradiator irradiates light toward the imaging region and the reception timing when a light receiver receives the reflected light from the imaging region, perform a second distance measurement for measuring the distance to the imaging region based on a first image obtained by imaging the imaging region by an imaging device, and execute a specific process when the first distance measurement result obtained by performing the first distance measurement and the second distance measurement result obtained by performing the second distance measurement are different. It is a program for executing a process including this.

[0021] A sixteenth aspect of the technology of the present disclosure is to cause a computer to measure the distance to an imaging region based on the irradiation timing when a light irradiator irradiates light toward the imaging region and the reception timing when a light receiver receives the reflected light from the imaging region, derive a first focusing position that focuses on a subject included in the imaging region based on the distance, derive a second focusing position that focuses on the subject based on an image obtained by imaging the imaging region by an imaging device, and execute a specific process when the first focusing position and the second focusing position are different. It is a program for executing a process including this.

Brief Description of Drawings

[0022] [Figure 1] It is a schematic perspective view showing an example of the usage mode of the smart device according to the first embodiment. [Figure 2] It is a rear view perspective view showing an example of the appearance of the back side of the smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of a mode in which an imaging region is imaged by a distance measurement imaging device included in the smart device according to the first embodiment, and a visible light image is displayed on a display. [Figure 4] It is a conceptual diagram showing an example of a mode in which laser light is irradiated to an imaging region by a distance measurement imaging device included in the smart device according to the first embodiment. [Figure 5]It is a schematic perspective view showing an example of the arrangement of each pixel included in the photoelectric conversion element of the smart device according to the first embodiment. [Figure 6] It is a conceptual diagram showing an example of the incident characteristics of subject light with respect to the first retardation pixel and the second retardation pixel included in the photoelectric conversion element shown in FIG. 5. [Figure 7] It is a schematic configuration diagram showing an example of the configuration of the non-retardation pixel included in the photoelectric conversion element shown in FIG. 5. [Figure 8] It is a block diagram showing an example of the configuration of the electrical system hardware of the smart device according to the first embodiment. [Figure 9] It is a block diagram showing an example of the functions of the CPU included in the smart device according to the first embodiment. [Figure 10] It is a conceptual diagram showing an example of the processing contents of the first distance measurement control unit and the acquisition unit shown in FIG. 9. [Figure 11] It is a conceptual diagram showing an example of the processing contents of the second distance measurement control unit and the acquisition unit shown in FIG. 9. [Figure 12] It is a conceptual diagram showing an example of the relationship between the distance measurement result (first distance) by the first distance measurement and the distance measurement result (second distance) by the second distance measurement when each of the mirror surface and the image reflected in the mirror is the distance measurement target. [Figure 13] It is a block diagram showing an example of the processing contents of the determination unit and the execution unit shown in FIG. 9. [Figure 14] It is a conceptual diagram showing an example of the details of a specific process shown in FIG. 13. [Figure 15] It is a conceptual diagram showing an example of the processing contents when the first visible light image in the image selection screen displayed on the display by the smart device according to the first embodiment is selected by the user via the touch panel. [Figure 16] It is a conceptual diagram showing an example of the processing contents when the second visible light image in the image selection screen displayed on the display by the smart device according to the first embodiment is selected by the user via the touch panel. [Figure 17A] It is a flowchart showing an example of the flow of the distance measurement imaging process according to the first embodiment. [Figure 17B]This is a continuation of the flowchart shown in Figure 17A. [Figure 18A] This is a flowchart showing a first modified example of the distance measurement and imaging process according to the first embodiment. [Figure 18B] This is a continuation of the flowchart shown in Figure 18A. [Figure 18C] This is a continuation of the flowchart shown in Figure 18B. [Figure 19] This is a flowchart showing a second modified example of the distance measurement and imaging process according to the first embodiment. [Figure 20] This is a flowchart showing a third modified example of the distance measurement and imaging process according to the first embodiment. [Figure 21] This is a flowchart showing a fourth modified example of the distance measurement and imaging process according to the first embodiment. [Figure 22] This is a rear view perspective showing an example of a smart device according to the first embodiment that further includes a wide-angle light receiver. [Figure 23] This is a flowchart showing a fifth modified example of the distance measurement and imaging process according to the first embodiment. [Figure 24] This block diagram shows an example of the functions of a CPU included in a smart device according to the second embodiment. [Figure 25] This is a conceptual diagram showing an example of the processing content for calculating the first focus position used in focus control performed by a smart device according to the second embodiment. [Figure 26] This is a conceptual diagram showing an example of the processing content for calculating the second focus position used in focus control performed by a smart device according to the second embodiment. [Figure 27] Figure 24 is a block diagram showing an example of the processing content of the determination unit and the execution unit. [Figure 28] Figure 27 is a conceptual diagram showing an example of the details of a specific process. [Figure 29A] This flowchart shows an example of the distance measurement and imaging process according to the second embodiment. [Figure 29B] This is a continuation of the flowchart shown in Figure 29A. [Figure 29C] This is a continuation of the flowchart shown in Figure 29B. [Figure 30] This is a conceptual diagram showing an example of how the distance measurement and imaging processing program according to the embodiment is installed on a smart device. [Modes for carrying out the invention]

[0023] Hereinafter, an example of an embodiment of an imaging device relating to the technology of this disclosure will be described with reference to the attached drawings.

[0024] First, let's explain the terminology used in the following explanation.

[0025] CPU stands for "Central Processing Unit". RAM stands for "Random Access Memory". ASIC stands for "Application Specific Integrated Circuit". PLD stands for "Programmable Logic Device". FPGA stands for "Field-Programmable Gate Array". SoC stands for "System-on-a-chip". SSD stands for "Solid State Drive". USB stands for "Universal Serial Bus". HDD stands for "Hard Disk Drive". EEPROM stands for "Electrically Erasable and Programmable Read Only Memory". EL stands for "Electro-Luminescence". A / D stands for "Analog / Digital". I / F stands for "Interface". UI stands for "User Interface". LTE stands for "Long Term Evolution". 5G stands for "5th Generation". LD stands for "Laser Diode". IR stands for "Infrared". APD stands for "Avalanche Photodiode". TOF stands for "Time of Flight". fps stands for "frames per second". LED stands for "Light Emitting Diode". ROI stands for "Region of Interest". LAN stands for "Local Area Network". AF stands for "Auto Focus". IC stands for "Integrated Circuit".

[0026] In this specification, “horizontal” means horizontal in the sense of including errors that are generally accepted in the art to which the disclosed art belongs, in addition to perfect horizontality. In this specification, “parallel” means parallel in the sense of including errors that are generally accepted in the art to which the disclosed art belongs, in addition to perfect parallelism. In this specification, “perpendicular” means perpendicular in the sense of including errors that are generally accepted in the art to which the disclosed art belongs, in addition to perfect perpendicularity. In this specification, “identical” means identical in the sense of including errors that are generally accepted in the art to which the disclosed art belongs, in addition to perfect identicality.

[0027] [First Embodiment] As an example, as shown in Figure 1, the smart device 10 performs an imaging operation (hereinafter also simply referred to as "imaging operation") to image an imaging area defined by the field of view θ1, and a distance measurement operation. In this first embodiment, "distance measurement" refers to the process of measuring the distance from the smart device 10 to the imaging area. The smart device 10 is an example of an "information processing device" related to the technology of this disclosure, and examples of smart devices 10 include smartphones or tablet terminals, which are electronic devices with imaging functions.

[0028] The smart device 10 performs first and second distance measurements, each using a different distance measurement method. The first distance measurement is a method (hereinafter also referred to as the "active method") that measures the distance to the imaging area based on the timing at which the smart device 10 irradiates the imaging area with laser light and the timing at which the smart device 10 receives the reflected light from the imaging area. The second distance measurement is a method (hereinafter also referred to as the "passive method") that measures the distance to the imaging area based on an image obtained when the imaging area is imaged by the smart device 10. Here, the image obtained when the imaging area is imaged by the smart device 10 is an example of the "first image" related to the technology of this disclosure. The laser light is an example of the "light" related to the technology of this disclosure. Here, the laser light is irradiated onto the imaging area as a surface. Surface irradiation refers to irradiation where the beam diameter of the laser light is wider than that of spot irradiation towards the imaging area. In surface irradiation, the beam diameter of the laser light gradually expands along the direction of irradiation, and the degree to which the beam diameter expands per unit time is greater in surface irradiation than in spot irradiation. In other words, in spot irradiation, the laser light is irradiated in a point-like manner onto a surface of the imaging area, whereas in surface irradiation, the laser light is irradiated in a planar manner onto a surface of the imaging area. Furthermore, the laser light irradiation may be performed as a single pulse, or it may be performed intermittently at regular intervals (for example, every 0.1 seconds). In this case, the first distance measurement may be performed with each laser light irradiation, and processing may be performed based on the distance measurement result.

[0029] Smart device 10 performs hybrid distance measurement, combining active and passive distance measurement. Smart device 10 also performs imaging with active focus control and imaging with passive focus control. Active focus control refers to focus control based on distance measurement results obtained through active distance measurement. Passive focus control refers to focus control based on distance measurement results obtained through passive distance measurement.

[0030] As an example, as shown in Figure 2, the smart device 10 includes a housing 12. The housing 12 houses a distance measuring and imaging device 14. The distance measuring and imaging device 14 includes a light illuminator 16 and a light receiver 18. The light illuminator 16 is equipped with an LD24, and in the smart device 10, imaging and distance measuring operations are performed by the distance measuring and imaging device 14.

[0031] An instruction key 13 is located on the side of the smart device 10. The instruction key 13 accepts various instructions. These "various instructions" include, for example, instructions to display a menu screen where various menus can be selected, instructions to select one or more menus, instructions to confirm the selection, and instructions to delete the selection.

[0032] When the smart device 10 is placed vertically, light-transmitting windows 20 and 22 are provided on the upper left of the back surface 12A of the housing 12 (upper left of the rear view of the smart device 10 when placed vertically). The light-transmitting windows 20 and 22 are light-transmitting optical elements (e.g., lenses) and are arranged horizontally at predetermined intervals (e.g., several millimeters apart) and are exposed from the back surface 12A. The light irradiator 16 irradiates the imaging area with laser light emitted from the LD24 through the light-transmitting windows 20. In this first embodiment, laser light in the infrared wavelength range is irradiated towards the imaging area by the light irradiator 16. Note that the wavelength range of the laser light is not limited to this range, and laser light in other wavelength ranges may also be used.

[0033] The light receiver 18 captures IR reflected light through the light-transmitting window 22. IR reflected light refers to the reflected light from the object being measured by the laser light irradiated onto the object by the light irradiator 16. The light receiver 18 also captures visible reflected light through the light-transmitting window 22. Visible reflected light refers to the reflected light from the imaging area of ​​visible light (for example, visible light contained in sunlight) irradiated onto the imaging area. For the sake of explanation, in the following, when it is not necessary to distinguish between IR reflected light and visible reflected light, they will simply be referred to as "reflected light."

[0034] The photodetector 18 is an example of an "imaging device" related to the technology of this disclosure and includes a photoelectric conversion element 26. The photoelectric conversion element 26 receives reflected light taken into the photodetector 18 through the light-transmitting window 22 and outputs an electrical signal corresponding to the amount of reflected light received.

[0035] As an example, as shown in Figure 3, a touch panel display 59 is provided on the front 12B of the housing 12. The touch panel display 59 comprises a display 46 and a touch panel 48. An example of the display 46 is an organic EL display. The display 46 may be a different type of display, such as a liquid crystal display, instead of an organic EL display.

[0036] The display 46 displays images (e.g., live view images and playback images) and text information. The touch panel 48 is a transparent touch panel and is superimposed on the surface of the display area of ​​the display 46. The touch panel 48 receives user input by detecting contact from an object such as a finger or stylus pen. Here, an out-cell type touch panel display in which the touch panel 48 is superimposed on the surface of the display area of ​​the display 46 is given as an example of the touch panel display 59, but this is only one example. For example, an on-cell type or in-cell type touch panel display can also be used as the touch panel display 59.

[0037] In the smart device 10, when an instruction to start imaging is received by the touch panel 48, the imaging area is imaged by the light receiver 18. That is, the light receiver 18 receives visible light reflected and generates a visible light image indicating the imaging area as an image corresponding to the received visible light reflected. The visible light image is an example of the "first image" and "second image" related to the technology of this disclosure.

[0038] The visible light image is displayed on the display 46 as a live view image or a still image, according to instructions received by the touch panel 48. In the example shown in Figure 3, the imaging area is defined by the field of view θ1. The field of view θ1 is changed according to instructions received by the touch panel 48.

[0039] As an example, as shown in Figure 4, when the smart device 10 receives an instruction to start distance measurement and imaging (hereinafter also referred to as the "distance measurement and imaging start instruction") via the touch panel 48, the light irradiator 16 emits laser light. The angle at which the laser light is emitted (hereinafter also referred to as the "irradiation angle") is θ2, and the irradiation angle θ2 is changed according to the instruction received via the touch panel 48. In the example shown in Figure 4, a configuration is described in which distance measurement is started in response to the distance measurement and imaging start instruction received via the touch panel 48 while a visible light image is displayed as a live view image on the display 46, but the technology of this disclosure is not limited to this. For example, distance measurement may be started when the distance measurement and imaging start instruction is received via the touch panel 48 while a visible light image is not displayed on the display 46.

[0040] In the smart device 10, the distance from the smart device 10 to the object to be measured is measured based on the time required from when the laser light is emitted by the light irradiator 16 until the IR reflected light is received by the light receiver 18, and the speed of light. For example, if the distance to the object to be measured is "L0", the speed of light is "c", and the time required from when the laser light is emitted by the light irradiator 16 until the IR reflected light is received by the light receiver 18 is "t", then the distance L0 is calculated according to the formula "L0 = c × t × 0.5".

[0041] As an example, as shown in Figure 5, the photoelectric conversion element 26 has multiple photodiodes arranged in a matrix. An example of multiple photodiodes is a set of photodiodes for "4896 × 3265" pixels.

[0042] Each photodiode in the photoelectric conversion element 26 is fitted with a color filter. The color filter includes a G filter corresponding to the G (green) wavelength range which contributes most to obtaining the luminance signal, an R filter corresponding to the R (red) wavelength range, a B filter corresponding to the B (blue) wavelength range, and an IR filter corresponding to the IR (infrared) wavelength range. In this first embodiment, the G filter, R filter, and B filter also function as infrared light cut filters to cut out infrared light.

[0043] The photoelectric conversion element 26 is formed by two types of photosensitive pixels: phase difference pixels and non-phase difference pixels N, which are different from the phase difference pixels. Generally, non-phase difference pixels N are also called normal pixels. The photoelectric conversion element 26 has four types of photosensitive pixels as non-phase difference pixels: R pixels, G pixels, B pixels, and IR pixels. The R pixels, G pixels, B pixels, IR pixels, and phase difference pixels are regularly arranged in the row direction (horizontal direction) and column direction (vertical direction) with a predetermined periodicity. R pixels are pixels corresponding to photodiodes on which R filters are placed, G pixels and phase difference pixels are pixels corresponding to photodiodes on which G filters are placed, B pixels are pixels corresponding to photodiodes on which B filters are placed, and IR pixels are pixels corresponding to photodiodes on which IR filters are placed. An example of an IR pixel is an InGaAs APD.

[0044] For the sake of clarity, in the following, when it is not necessary to distinguish between G filters, R filters, and B filters, they will also be referred to as "visible light filters." Similarly, in the following, when it is not necessary to distinguish between R pixels, G pixels, and B pixels, they will be referred to as "visible light pixels."

[0045] Multiple phase-difference pixel lines 26A and multiple non-phase-difference pixel lines 26B are arranged on the light-receiving surface of the photoelectric conversion element 26. Phase-difference pixel lines 26A are horizontal lines containing phase-difference pixels. Specifically, phase-difference pixel lines 26A are horizontal lines in which phase-difference pixels and non-phase-difference pixels N are mixed. Non-phase-difference pixel lines 26B are horizontal lines containing only multiple non-phase-difference pixels N.

[0046] On the light-receiving surface of the photoelectric conversion element 26, phase-difference pixel lines 26A and a predetermined number of non-phase-difference pixel lines 26B are alternately arranged along the column direction. The "determined number of lines" here refers to, for example, 2 lines. Although 2 lines are used as an example of the predetermined number of lines here, the technology of this disclosure is not limited to this, and the predetermined number of lines may be 3 or more lines, or it may be tens of lines, tens of lines, or hundreds of lines, etc.

[0047] The phase difference pixel line 26A is arranged in a column direction with two rows skipped from the first row to the last row. Some of the pixels in the phase difference pixel line 26A are phase difference pixels. Specifically, the phase difference pixel line 26A is a horizontal line in which phase difference pixels and non-phase difference pixels N are arranged periodically. Phase difference pixels are broadly classified into first phase difference pixels L and second phase difference pixels R. In the phase difference pixel line 26A, first phase difference pixels L and second phase difference pixels R are arranged alternately as G pixels at intervals of several pixels in the line direction.

[0048] The first phase difference pixels L and the second phase difference pixels R are arranged to appear alternately in the column direction. In the example shown in Figure 5, in the fourth column, the first phase difference pixels L, second phase difference pixels R, first phase difference pixels L, and second phase difference pixels R are arranged in that order from the first row along the column direction. That is, the first phase difference pixels L and second phase difference pixels R are arranged alternately from the first row along the column direction. Also in the example shown in Figure 5, in the tenth column, the second phase difference pixels R, first phase difference pixels L, second phase difference pixels R, and first phase difference pixels L are arranged in that order from the first row along the column direction. That is, the second phase difference pixels R and first phase difference pixels L are arranged alternately from the first row along the column direction.

[0049] The photoelectric conversion element 26 is divided into three regions. Specifically, the photoelectric conversion element 26 has a visible light image region 26N1, a first distance measuring system region 26N2, and a second distance measuring system region 26N3. The visible light image region 26N1 is a group of visible light pixels consisting of multiple visible light pixels and is used for generating a visible light image. The first distance measuring system region 26N2 is a group of IR pixels consisting of multiple IR pixels and is used for the first distance measurement. The second distance measuring system region 26N3 is a group of phase difference pixels consisting of multiple phase difference pixels and is used for the second distance measurement. The visible light image region 26N1 and the second distance measuring system region 26N3 receive visible reflected light and output an electrical signal corresponding to the amount of light received. The first distance measuring system region 26N2 receives IR reflected light and outputs an electrical signal corresponding to the amount of light received.

[0050] As an example, as shown in Figure 6, the first phase difference pixel L comprises a microlens 19, a light-shielding member 17A, and a photodiode PD. In the first phase difference pixel L, the light-shielding member 17A is positioned between the microlens 19 and the light-receiving surface of the photodiode PD. The left half in the row direction of the light-receiving surface of the photodiode PD (the left side when viewing the subject from the light-receiving surface (in other words, the right side when viewing the light-receiving surface from the subject)) is shielded by the light-shielding member 17A.

[0051] The second phase-difference pixel R comprises a microlens 19, a light-shielding member 17B, and a photodiode PD. In the second phase-difference pixel R, the light-shielding member 17B is positioned between the microlens 19 and the light-receiving surface of the photodiode PD. The right half of the photodiode PD's light-receiving surface in the row direction (the right side when viewing the subject from the light-receiving surface (in other words, the left side when viewing the light-receiving surface from the subject)) is shielded by the light-shielding member 17B. For the sake of explanation, in the following, when it is not necessary to distinguish between the light-shielding members 17A and 17B, they will be referred to simply as "light-shielding member" without any reference numerals.

[0052] The light beam passing through the exit pupil of the imaging lens 41 is broadly divided into left-region passing light 300L and right-region passing light 300R. Left-region passing light 300L refers to the left half of the light beam passing through the exit pupil of the imaging lens 41 when viewed from the phase difference pixel side towards the subject side, and right-region passing light 300R refers to the right half of the light beam passing through the exit pupil of the imaging lens 41 when viewed from the phase difference pixel side towards the subject side. The light beam passing through the exit pupil of the imaging lens 41 is divided into left and right by the microlens 19, light-shielding member 17A, and light-shielding member 17B, which function as pupil division parts, and the first phase difference pixel L receives the left-region passing light 300L as subject light, and the second phase difference pixel R receives the right-region passing light 300R as subject light. As a result, the photoelectric conversion element 26 generates a first phase difference image corresponding to the subject image corresponding to the light passing through the left region 300L, and a second phase difference image corresponding to the subject image corresponding to the light passing through the right region 300R.

[0053] In the smart device 10, for example, the distance to the imaging area is measured based on the amount of difference α between the first phase difference image and the second phase difference image for one line in the same phase difference pixel line 26A.

[0054] As an example, as shown in Figure 7, non-phase difference pixels N differ from phase difference pixels in that they do not have a light-shielding member. The photodiode PD of the non-phase difference pixel N receives light passing through the left region 300L and light passing through the right region 300R as the subject light.

[0055] In the smart device 10, IR reflected light is received by each of the multiple IR pixels included in the second distance measurement system division area 26N3 (see Figure 5), and distance measurement is performed for each IR pixel. Then, in the smart device 10, the distance measurement result for each IR pixel is displayed as a distance image on the display 46 according to the instructions received by the touch panel 48. Here, a distance image refers to an image that represents the distance to the distance measurement target measured for each IR pixel using different colors and / or shades of gray.

[0056] Furthermore, in the smart device 10, the distance measurement results are displayed on the display 46 as a distance image or a distance superimposed image, according to the instructions received by the touch panel 48. The distance superimposed image displayed on the display 46 is, for example, an image in which numerical values ​​indicating the distance measurement results are superimposed on a visible light image (for example, a live view image). For example, the distance from the smart device 10 to each of several representative locations (for example, three locations) within the imaging area is displayed on the display 46 in a visible light image. An example of several representative locations is several locations within the imaging area where the difference in contrast between them is greater than or equal to a predetermined value, among specific subjects (for example, subjects included in the central area of ​​the screen, and / or people, etc.).

[0057] As an example, as shown in Figure 8, the smart device 10 includes a controller 15, an input / output interface 40, an image memory 42, a UI device 44, an external I / F 52, and a communication I / F 54, in addition to the light irradiator 16 and light receiver 18.

[0058] The controller 15 includes a CPU 15A, a storage device 15B, and a memory device 15C. The CPU 15A, storage device 15B, and memory device 15C are connected via a bus 50, which is connected to an input / output interface 40. In the example shown in Figure 8, for illustrative purposes, only one bus is shown as bus 50, but there may be multiple buses. Bus 50 may be a serial bus, or a parallel bus including a data bus, address bus, and control bus, etc.

[0059] Storage 15B stores various parameters and programs. Storage 15B is a non-volatile memory device. Flash memory is used here as an example of storage 15B. Flash memory is merely one example; for example, various non-volatile memories such as magnetoresistive memory and / or ferroelectric memory can be used as storage 15B instead of flash memory, or in combination with flash memory. The non-volatile memory device may also be EEPROM, HDD, and / or SSD. Memory 15C temporarily stores various information and is used as work memory. RAM is an example of memory 15C, but it is not limited to this, and other types of memory devices may be used.

[0060] Storage 15B stores various programs. CPU 15A reads the necessary programs from storage 15B and executes the read programs on memory 15C. CPU 15A controls the entire smart device 10 according to the programs executed on memory 15C. Note that storage 15B and memory 15C are examples of "memory" related to the technology of this disclosure.

[0061] Multiple devices are connected to the input / output interface 40, and the input / output interface 40 is responsible for the exchange of various information between these multiple devices. In the example shown in Figure 8, the multiple devices connected to the input / output interface 40 include a controller 15, a light illuminator 16, a light receiver 18, an image memory 42, a UI system device 44, an external I / F 52, and a communication I / F 54.

[0062] External I / F 52 is responsible for the exchange of various types of information between the smart device 10 and devices located outside of it (hereinafter also referred to as "external devices"). An example of external I / F 52 is a USB interface. External devices such as smart devices, personal computers, servers, USB memory, memory cards, and / or printers (not shown) can be connected directly or indirectly to the USB interface.

[0063] Communication I / F 54 has communication functions such as LTE, 5G, wireless LAN, and / or Bluetooth®, and is responsible for the exchange of various information between external devices and CPU 15A. For example, communication I / F 54 is communicably connected to network 56 (e.g., the Internet) via a base station (not shown), and is responsible for the exchange of various information between external devices on network 56 and CPU 15A.

[0064] The UI device 44 includes a display 46, and the CPU 15A displays various information on the display 46. The UI device 44 also includes a reception device 47. The reception device 47 includes a touch panel 48 and a hard key unit 53. The hard key unit 53 is at least one hard key, including an instruction key 13 (see Figure 2). The CPU 15A operates according to the various instructions received by the touch panel 48. Although the hard key unit 53 is included in the UI device 44 here, the technology of this disclosure is not limited to this, and for example, the hard key unit 53 may be connected to an external I / F 52.

[0065] The light irradiator 16 is equipped with a light-transmitting window 20, a beam expander 21, a collimating lens 23, an LD 24, and an LD driver 25. Along the optical axis L1, the light-transmitting window 20, the beam expander 21, and the collimating lens 23 are arranged in order from the imaging area side (object side) to the LD 24. The LD driver 25 is connected to the LD 24 and the input / output interface 40, and drives the LD 24 according to the instructions of the CPU 15A to emit laser light from the LD 24.

[0066] The laser light emitted from the LD24 is converted into parallel light by the collimating lens 23, then its beam diameter is expanded by the beam expander 21, and it is projected from the light-transmitting window 20 toward the distance-measuring target.

[0067] The light receiver 18 includes a light-transmitting window 22, an objective lens 30A, a focusing lens 30B, an aperture 30C, a photoelectric conversion element 26, a photoelectric conversion element driver 32, and a signal processing circuit 34. The CPU 15A and the signal processing circuit 34 are examples of a "processor" related to the technology of this disclosure.

[0068] In the photodetector 18, the light-transmitting window 22, objective lens 30A, focus lens 30B, and aperture 30C are arranged in order along the optical axis L2, from the imaging area side (object side) to the photoelectric conversion element 26. The photoelectric conversion element driver 32 is connected to the photoelectric conversion element 26 and the input / output interface 40, and drives the photoelectric conversion element 26 according to the instructions of the CPU 15A. For example, under the control of the CPU 15A, the photoelectric conversion element driver 32 supplies an imaging timing signal to the photoelectric conversion element 26 that defines the timing of imaging performed by the photoelectric conversion element 26. The photoelectric conversion element 26 performs reset, exposure, and output of electrical signals according to the imaging timing signal supplied from the photoelectric conversion element driver 32. Examples of imaging timing signals include a vertical synchronization signal and a horizontal synchronization signal.

[0069] The light receiver 18 is equipped with a focus control mechanism 31. The focus control mechanism 31 comprises a focus lens 30B, a moving mechanism 60, a motor 62, and a motor driver 64. The focus lens 30B is supported by the moving mechanism 60 so as to be slidable along the optical axis L2. The motor 62 is connected to the moving mechanism 60 and the motor driver 64. The motor driver 64 is connected to the input / output interface 40 and drives the motor 62 according to instructions from the CPU 15A. The moving mechanism 60 is connected to the drive shaft (not shown) of the motor 62 and receives power from the motor 62 to selectively move the focus lens 30B along the optical axis L2 towards the object side and the image side. That is, the CPU 15A adjusts the focus position by controlling the drive of the motor 62 via the motor driver 64. Here, "focus position" refers to the position of the focus lens 30B on the optical axis L2 when it is in focus (for example, when the contrast of the visible light image is maximized, or when a predetermined depth of field is achieved). In this first embodiment, the control that adjusts the focus lens 30B to the focus position is referred to as "focus control."

[0070] The aperture 30C is a fixed aperture whose opening does not change. In the case of a fixed aperture, exposure adjustment is performed by the electronic shutter of the photoelectric conversion element 26. The aperture 30C may be a variable aperture instead of a fixed aperture. Note that the objective lens 30A, focusing lens 30B, and aperture 30C included in the light receiver 18 are merely examples, and the technology of this disclosure will still be valid even if the lens configuration and / or the position of the aperture 30C are changed.

[0071] Reflected light enters the light receiver 18 through the light-transmitting window 22. The reflected light that enters the light-transmitting window 22 is imaged onto the photoelectric conversion element 26 via the objective lens 30A, the focusing lens 30B, and the aperture 30C.

[0072] The photoelectric conversion element 26 is connected to the signal processing circuit 34 and outputs pixel data indicating the pixel value for each visible light pixel and IR pixel to the signal processing circuit 34. The signal processing circuit 34 digitizes the pixel data by performing A / D conversion on the pixel data input from the photoelectric conversion element 26, and then applies various signal processing to the digitized pixel data.

[0073] The signal processing circuit 34 includes a visible light pixel data processing circuit 34A, a first distance measuring system processing circuit 34B, and a second distance measuring system processing circuit 34C. The visible light pixel data processing circuit 34A generates a visible light image by applying known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction to the visible light pixel data, which is the pixel data of the visible light pixels. The visible light pixel data processing circuit 34A then stores the visible light image in the image memory 42. The visible light image in the image memory 42 is updated when one frame of visible light image is overwritten and saved.

[0074] The distance measuring and imaging device 14 is equipped with a TOF camera 27. The TOF camera 27 includes a light irradiator 16, a first distance measuring system division area 26N2 (see Figure 5) of the photoelectric conversion element 26, and a first distance measuring system processing circuit 34B. The first distance measuring system processing circuit 34B acquires an irradiation timing signal from the CPU 15A that indicates the irradiation timing (hereinafter also simply referred to as "irradiation timing") when the light irradiator 16 irradiates the imaging area with laser light.

[0075] The first distance measuring system processing circuit 34B measures the distance from the smart device 10 to the imaging area for each IR pixel based on the irradiation timing indicated by the irradiation timing signal and the timing at which IR reflected light is received by each IR pixel (hereinafter also referred to as the "reception timing"). Here, the reception timing is the timing at which IR pixel data having an output value exceeding a threshold is received by the first distance measuring system processing circuit 34B. The threshold used here is a value that has been derived in advance as the output value of the noise component (for example, a noise component that occurs independently of IR reflected light) output from the IR pixel, for example, through actual device testing and / or computer simulation.

[0076] The first distance measurement system processing circuit 34B measures the distance from the smart device 10 to the imaging area for each IR pixel based on the illumination timing and light reception timing, generates a distance image based on the measurement result for each IR pixel, and stores the generated distance image in the image memory 42. Note that the distance image in the image memory 42 is updated by overwriting and saving the distance image for one frame.

[0077] The second distance measuring system processing circuit 34C acquires phase difference pixel data, which indicates the pixel value of a phase difference pixel, from each of the multiple phase difference pixels in the second distance measuring system division area 26N3 (see Figure 5) that is included in the area (so-called ROI) specified by the user or the like among the photoelectric conversion elements 26. The second distance measuring system processing circuit 34C generates a first phase difference image and a second phase difference image (see Figure 5) from the phase difference pixel data and calculates the amount of shift α (see Figure 5) between the generated first phase difference image and the second phase difference image. Then, the second distance measuring system processing circuit 34C calculates the distance from the smart device 10 to the imaging area based on the calculated amount of shift α. Specifically, the second distance measuring system processing circuit 34C calculates the distance from the smart device 10 to the imaging area by using a calculation formula in which the amount of shift α is the independent variable and the distance is the dependent variable.

[0078] Although the calculation formula is shown as an example here, the technology of this disclosure is not limited to this, and the second distance measuring system processing circuit 34B may derive the distance from the smart device 10 to the imaging area by using a table in which the displacement amount α and distance are associated.

[0079] The CPU 15A obtains the distance measured by the first distance measuring system processing circuit 34B (hereinafter referred to as the "first distance") from the first distance measuring system processing circuit 34B, and obtains the distance measured by the second distance measuring system processing circuit 34C (hereinafter referred to as the "second distance") from the second distance measuring system processing circuit 34C.

[0080] As an example, as shown in Figure 9, the storage 15B stores the distance measurement and imaging processing program 70. The CPU 15A reads the distance measurement and imaging processing program 70 from the storage 15B and executes the read distance measurement and imaging processing program 70, thereby operating as the first distance measurement control unit 15A1, the second distance measurement control unit 15A2, the acquisition unit 15A3, the determination unit 15A4, and the execution unit 15A5.

[0081] Here, referring to Figure 10, we will explain the case in which the smart device 10 performs the first distance measurement on an imaging area that includes the entire mirror 100 in which the user is reflected (hereinafter also referred to as the "mirror-inclusive imaging area"). When the touch panel 48 receives a command to start distance measurement imaging with the mirror-inclusive imaging area positioned at the irradiation angle θ2, the first distance measurement control unit 15A1 outputs a first distance measurement start signal to the light irradiator 38B and the light receiver 18. When the light irradiator 38B receives the first distance measurement start signal from the first distance measurement control unit 15A1, it irradiates laser light. When the light receiver 18 receives the first distance measurement start signal from the first distance measurement control unit 15A1, the first distance measurement system processing circuit 34B calculates the first distance based on the irradiation timing and light reception timing indicated by the irradiation timing signal acquired from the CPU 15A. The acquisition unit 15A3 acquires the first distance from the first distance measurement system processing circuit 34B. The first distance is an example of a "first distance measurement result" relating to the technology of this disclosure. Here, the irradiation timing is set to the time after a predetermined time has elapsed from the time when the first distance measurement start signal is input from the first distance measurement control unit 15A1 to the light receiver 18. The predetermined time is, for example, the time required from the time the first distance measurement start signal is output until the laser light is emitted from the light irradiator 38B, which is a time that has been determined in advance through actual equipment testing and / or computer simulation.

[0082] Next, with reference to Figure 11, the case in which the second distance measurement is performed by the smart device 10 on the mirror-inclusive imaging area will be described. When a distance measurement imaging start instruction is received by the touch panel 48 with the mirror-inclusive imaging area within the field of view θ1, the second distance measurement control unit 15A2 outputs a second distance measurement start signal to the photodetector 18. When the second distance measurement start signal is input to the photodetector 18 from the second distance measurement control unit 15A2, the second distance measurement system division area 26N3 images the mirror-inclusive imaging area and outputs phase difference pixel data corresponding to the mirror-inclusive imaging area to the second distance measurement system processing circuit 34C. The second distance measurement system processing circuit 34C generates a first phase difference image and a second phase difference image (see Figure 6) based on the phase difference pixel data input from the second distance measurement system division area 26N3, and calculates the displacement amount α (see Figure 6) based on the generated first phase difference image and second phase difference image. The second distance measurement system processing circuit 34C calculates the second distance from the calculated displacement amount α. The acquisition unit 15A3 acquires the second distance from the second distance measurement system processing circuit 34C. The second distance is an example of the "second distance measurement result" related to the technology of this disclosure.

[0083] As an example, as shown in Figure 12, the distance measurement result obtained by performing a first distance measurement by the smart device 10 on the mirror-inclusive imaging area, i.e., the first distance, is the distance from the smart device 10 to the mirror surface 100A of the mirror 100. The distance measurement result obtained by performing a second distance measurement by the smart device 10 on the mirror-inclusive imaging area, i.e., the second distance, is the distance from the smart device 10 to the subject image reflected in the mirror 100. In the example shown in Figure 12, the subject images reflected in the mirror 100 are the image of the smart device 10 (in the example shown in Figure 12, "smart device image") and the image of the user (in the example shown in Figure 12, "user image"), and the second distance is the distance from the smart device 10 to the user image.

[0084] Thus, when the first distance and the second distance are different, various problems arise. For example, if a user wants to capture an image of the entire mirror 100 using the smart device 10, and the smart device 10 performs focus control based on the second distance to capture the entire mirror 100, the resulting image will be out of focus on the mirror 100. On the other hand, if a user wants to capture an image of their reflection in the mirror 100 using the smart device 10, and the smart device 10 performs focus control based on the first distance to capture the user's image, the resulting image will be out of focus on the user's image.

[0085] To solve these problems, the determination unit 15A4 and the execution unit 15A5 operate as shown in Figure 13 as an example. The determination unit 15A4 determines whether the first distance (see Figure 10) and the second distance (see Figure 11) acquired by the acquisition unit 15A3 are different. If the first distance and the second distance are the same, that is, if the determination unit 15A4 determines that the first distance and the second distance are not different, the execution unit 15A5 performs the first imaging process. The first imaging process refers to imaging accompanied by focus control based on the first distance.

[0086] The storage 15B stores a focus position derivation table 72. The focus position derivation table 72 associates the distance from the smart device 10 to the imaging area with the focus position. In the first imaging process, the execution unit 15A5 derives the focus position corresponding to the first distance from the focus position derivation table 72, and controls the motor 62 of the photodetector 18 to move the focus lens 30B to the derived focus position. The execution unit 15A5 then controls the visible light image section 26N1 of the photodetector 18, so that the imaging area is imaged by the visible light image section 26N1, and the visible light pixel data obtained from the image is output from the visible light image section 26N1 to the visible light pixel data processing circuit 34A. The visible light pixel data processing circuit 34A generates a first visible light image showing the imaging area based on the visible light pixel data input from the visible light image section 26N1, and outputs the generated first visible light image to the image memory 42. The first visible light image is stored in the image memory 42.

[0087] On the other hand, if the determination unit 15A4 determines that the first distance and the second distance are different, the execution unit 15A5 performs a specific process. Here, the specific process will be explained with reference to Figure 14. As an example, as shown in Figure 14, the specific process includes, for example, a first imaging process, a second imaging process, and an image selection screen display process. The second imaging process refers to imaging accompanied by focus control based on the second distance. In the second imaging process, as with the first imaging process, the execution unit 15A5 derives the focus position corresponding to the second distance from the focus position derivation table 72, and the motor 62 of the light receiving unit 18 is controlled to move the focus lens 30B to the derived focus position. Then, as with the first imaging process, the imaging area is imaged by the visible light image section 26N1, and the visible light pixel data obtained from the imaging is output from the visible light image section 26N1 to the visible light pixel data processing circuit 34A. The visible light pixel data processing circuit 34A generates a second visible light image indicating the imaging area based on the visible light pixel data input from the visible light image section area 26N1, and outputs the generated second visible light image to the image memory 42. The image memory 42 stores the second visible light image.

[0088] In the image selection screen display process, the execution unit 15A5 acquires the first visible light image and the second visible light image from the image memory 42. Then, the execution unit 15A5 generates an image selection screen based on the first visible light image, the second visible light image, the first distance acquired by the acquisition unit 15A3 (see Figure 10), the second distance acquired by the acquisition unit 15A3 (see Figure 11), and various messages, and displays the generated image selection screen on the display 46. The image selection screen displays the first visible light image and the second visible light image side by side. Below the first visible light image, the message "Active Method" is displayed. The message "Active Method" indicates that the first visible light image is an image obtained by imaging accompanied by focus control based on the distance measurement result of the active method distance measurement (first distance measurement). Below the second visible light image, the message "Passive Method" is displayed. The message "Passive method" indicates that the second visible light image was obtained through imaging accompanied by focus control based on the distance measurement results from the passive method (second distance measurement).

[0089] In this first embodiment, the processes included in a specific process are exemplified as a first imaging process, a second imaging process, and an image selection screen display process. However, the technology of this disclosure is not limited to these, and may include one or two of the first imaging process, the second imaging process, and the image selection screen display process. Furthermore, in this first embodiment, the display examples "active method" and "passive method" are shown, but the display does not necessarily have to be "active method" and "passive method." Any display is acceptable as long as the user can understand the difference between the distance measurement methods. For example, when performing laser distance measurement and phase difference distance measurement, the display could be "laser" and "phase difference," or an icon representing the distance measurement method could be displayed. Alternatively, the focus position could be displayed instead of the distance measurement method. For example, it could be "focus position: foreground" and "focus position: background," or "focus position: object" and "focus position: image projected onto the object," etc. Alternatively, two or more combinations of characters and icons indicating the distance measurement method and the focus position could be displayed.

[0090] Furthermore, the first visible light image is associated with the numerical value "1.8m," which is the distance measurement result from the first distance measurement, and the second visible light image is associated with the numerical value "2.5m," which is the distance measurement result from the second distance measurement. In addition, a message "The distance measurement results are different" is displayed to indicate that the distance measurement results from the first distance measurement and the distance measurement results from the second distance measurement are different. Then, a message "Please select one image" is displayed to prompt the user to select either the first or second visible light image. Here, an example of how various messages are displayed visually is given, but the system is not limited to this, and various messages may be output audibly in parallel with the visual display. When the image selection screen is displayed on the display 46, the user selects either the first visible light image or the second visible light image via the touch panel 48. In the example shown in Figure 14, the first visible light image is shown being selected via the touch panel 48 by the user's finger. The first visible light image is an example of a "first focused image" relating to the technology of this disclosure, and the second visible light image is an example of a "second focused image" relating to the technology of this disclosure.

[0091] As an example, as shown in Figure 15, when the image selection screen is displayed on the display 46 and the first visible light image is selected by the user via the touch panel 48 using their finger, the first distance measurement control unit 15A1 controls the distance measurement imaging device 14 to cause the first distance measurement system processing circuit 34B to perform the first distance measurement. The execution unit 15A5 also performs the first imaging process and the first visible light image display process. Here, the execution unit 15A5 performs the first imaging process using the new first distance obtained by the first distance measurement. The first visible light image display process is the process of displaying the latest first visible light image obtained by the first imaging process on the display 48. Note that here, we have given an example of a configuration in which the first distance measurement and first imaging process are performed again on the condition that the first visible light image has been selected, and the latest first visible light image obtained by the first imaging process is displayed on the display 48, but this is only an example. For example, if a first visible light image is selected, the selected first visible light image may be stored in the storage 15B and / or a storage medium such as a memory card. Alternatively, imaging with focus control based on the distance measured by the selected ranging method and saving the selected image (e.g., the first visible light image or the second visible light image) may be selected according to instructions received by the receiving device 47 (instructions given by the user).

[0092] When the first visible light image display processing is executed by the execution unit 15A5, the second visible light image, the value "2.5m", the message "Passive method", the message "Measurement results are different", and the message "Please select one image" are hidden from the image selection screen. Then, the first distance (the value "1.8m" in the example shown in Figure 15), which is the latest measurement result obtained by the latest first distance measurement, is displayed on the display 46, the first visible light image obtained by the execution of the first imaging processing is displayed on the display 46, and the message "Active method" is also displayed below the first visible light image. Furthermore, the display area for the first distance, the first visible light image, and the message "Active method" is larger than the display area for the first distance, the first visible light image, and the message "Active method" on the image selection screen.

[0093] As an example, as shown in Figure 16, when an image selection screen is displayed on the display 46 and a second visible light image is selected by the user via the touch panel 48 using their finger, the second distance measurement control unit 15A2 controls the distance measurement imaging device 14 to cause the second distance measurement system processing circuit 34C to perform a second distance measurement. The execution unit 15A5 also performs a second imaging process and a second visible light image display process. Here, the execution unit 15A5 performs the second imaging process using the new second distance obtained by the second distance measurement. The second visible light image display process is the process of displaying the latest second visible light image obtained by the second imaging process on the display 48.

[0094] When the second visible light image display processing is executed by the execution unit 15A5, the first visible light image, the value "1.8m", the message "Active method", the message "Measurement results are different", and the message "Please select one image" are hidden from the image selection screen. Then, the second distance (the value "2.5m" in the example shown in Figure 16), which is the latest measurement result obtained by the latest second distance measurement, is displayed on the display 46, the second visible light image obtained by the execution of the second imaging processing is displayed on the display 46, and the message "Passive method" is also displayed below the second visible light image. Furthermore, the display areas for the second distance, the second visible light image, and the message "Passive method" are larger than the display areas for the second distance, the second visible light image, and the message "Passive method" on the image selection screen.

[0095] Next, the operation of the part of the smart device 10 relating to the technology of this disclosure will be explained with reference to Figures 17A and 17B. Figures 17A and 17B are flowcharts showing an example of the flow of the distance measurement and imaging process executed by the CPU 15A according to the distance measurement and imaging processing program 70.

[0096] In the distance measurement and imaging process shown in Figure 17A, first, in step ST100, the first distance measurement control unit 15A1 controls the distance measurement and imaging device 14 to cause the first distance measurement system processing circuit 34B to perform a first distance measurement. The first distance measurement is achieved by the first distance measurement control unit 15A1 causing the first distance measurement system processing circuit 34B to calculate a first distance.

[0097] In the next step, ST102, the acquisition unit 15A3 acquires the first distance obtained by the first distance measurement performed in step ST100.

[0098] In the next step, ST104, the second distance measurement control unit 15A2 controls the distance measurement imaging device 14 to cause the second distance measurement system processing circuit 34C to perform a second distance measurement. The second distance measurement is achieved by the second distance measurement control unit 15A2 causing the second distance measurement system processing circuit 34C to calculate a second distance.

[0099] In the next step, ST106, the acquisition unit 15A3 acquires the second distance obtained by the second distance measurement performed in step ST104.

[0100] In the next step, ST108, the determination unit 15A4 determines whether the first distance obtained in step ST102 and the second distance obtained in step ST106 are different. If the first distance obtained in step ST102 and the second distance obtained in step ST106 are the same in step ST108, the determination is denied and the distance measurement imaging process proceeds to step 120. If the first distance obtained in step ST102 and the second distance obtained in step ST106 are different in step ST108, the determination is affirmed and the distance measurement imaging process proceeds to step ST110.

[0101] In step ST110, the execution unit 15A5 performs a first imaging process using the first distance acquired in step ST102.

[0102] In the next step, ST112, the execution unit 15A5 performs a second imaging process using the second distance acquired in step ST106.

[0103] In the next step, ST114, the execution unit 15A5 performs the image selection screen display process. As a result, the image selection screen is displayed on the display 46, as shown in Figure 14 as an example.

[0104] In the next step, ST116, the execution unit 15A5 determines whether either the first visible light image or the second visible light image displayed on the image selection screen has been selected by the user via the touch panel 48. If, in step ST116, either the first visible light image or the second visible light image displayed on the image selection screen has not been selected by the user via the touch panel 48, the determination is denied, and the determination in step ST116 is repeated. If, in step ST116, either the first visible light image or the second visible light image displayed on the image selection screen has been selected by the user via the touch panel 48, the determination is affirmed, and the distance measurement imaging process proceeds to step ST118.

[0105] In step ST118, the execution unit 15A5 determines whether the image selected from the image selection screen is a first visible light image. If the image selected from the image selection screen in step ST118 is a second visible light image, the determination is denied, and the distance measurement imaging process proceeds to step 130 shown in Figure 17B. If the image selected from the image selection screen in step ST118 is a first visible light image, the determination is affirmed, and the distance measurement imaging process proceeds to step ST120.

[0106] In step ST120, the execution unit 15A5 determines whether or not the image capture start timing has arrived. The image capture start timing refers to the timing at which the imaging of one frame, defined by the frame rate for the live view image, begins. For example, if the frame rate for the live view image is 60fps, the image capture start timing is every 1 / 60th of a second. Here, the image capture start timing is exemplified as the timing at which the imaging of one frame, defined by the frame rate for the live view image, begins, but the technology of this disclosure is not limited to this. For example, the image capture start timing may be the timing at which the imaging of one frame, defined by the frame rate for the recorded video image, begins, or it may be the timing at which the instruction to start imaging for still images is received by the receiving device 47.

[0107] In step ST120, if the imaging start timing has not yet arrived, the determination is rejected, and the distance measurement imaging process proceeds to step ST126. In step ST120, if the imaging start timing has arrived, the determination is affirmed, and the distance measurement imaging process proceeds to step ST122.

[0108] In step ST122, the first distance measurement control unit 15A1 controls the distance measurement imaging device 14 to cause the first distance measurement system processing circuit 34B to perform the first distance measurement.

[0109] In the next step, ST124, the execution unit 15A5 performs a first imaging process using the first distance obtained by the first distance measurement performed in step ST122. The execution unit 15A5 also performs a first visible light image display process using the latest first visible light image obtained by performing the first imaging process.

[0110] In the next step, ST126, the execution unit 15A5 determines whether the conditions for terminating the distance measurement and imaging process (hereinafter referred to as "termination conditions") have been met. An example of a termination condition is that an instruction to terminate the distance measurement and imaging process has been received by the receiving device 47. If the termination conditions are not met in step ST126, the determination is denied, and the distance measurement and imaging process proceeds to step ST120. If the termination conditions are met in step ST126, the determination is affirmed, and the distance measurement and imaging process is terminated.

[0111] On the other hand, in step ST130 shown in Figure 17B, the execution unit 15A5 determines whether or not the imaging start timing has arrived. If the imaging start timing has not arrived in step ST130, the determination is denied, and the distance measurement imaging process proceeds to step ST136. If the imaging start timing has arrived in step ST130, the determination is affirmed, and the distance measurement imaging process proceeds to step ST132.

[0112] In step ST132, the second distance measurement control unit 15A2 controls the distance measurement imaging device 14 to cause the second distance measurement system processing circuit 34C to perform the second distance measurement.

[0113] In the next step, ST134, the execution unit 15A5 performs a second imaging process using the second distance obtained by the second distance measurement performed in step ST132. The execution unit 15A5 also performs a second visible light image display process using the latest second visible light image obtained by performing the second imaging process.

[0114] In the next step, ST136, the execution unit 15A5 determines whether the termination conditions have been met. If the termination conditions are not met in step ST136, the determination is denied, and the distance measurement and imaging process proceeds to step ST130. If the termination conditions are met in step ST136, the determination is affirmed, and the distance measurement and imaging process ends.

[0115] As explained above, in the smart device 10, if the first distance obtained by the first distance measurement and the second distance obtained by the second distance measurement are different, the execution unit 15A5 performs a specific process. Therefore, this configuration can help solve various problems that arise when the first distance and the second distance are different.

[0116] Furthermore, in the smart device 10, distance measurement is performed using laser light as directional light. Therefore, with this configuration, the distance to a distance-measuring target located at a distance can be measured with higher accuracy compared to when distance measurement is performed without using directional light.

[0117] Furthermore, in the smart device 10, the image selection screen display process is executed by the execution unit 15A5 as a specific process. The image selection screen displayed on the display 46 when the image selection screen display process is executed by the execution unit 15A5 includes information that notifies the user that the first distance and the second distance are different (for example, a numerical value indicating the first distance, a numerical value indicating the second distance, and a message). Therefore, with this configuration, the user can be made aware that the first distance and the second distance are different.

[0118] Furthermore, in the smart device 10, the first imaging process and the second imaging process are performed by the execution unit 15A5. That is, the execution unit 15A5 causes the photodetector 18 of the distance measuring imaging device 14 to perform imaging with focus control based on the first distance, and also causes the photodetector 18 of the distance measuring imaging device 14 to perform imaging with focus control based on the second distance. Therefore, with this configuration, it is possible to obtain an image in the in-focus state intended by the user more easily than when imaging is performed under focus control based on the first or second distance.

[0119] Furthermore, on the smart device 10, an image selection screen is displayed on the display 46, which includes a first visible light image obtained by imaging under focus control based on a first distance, and a second visible light image obtained by imaging under focus control based on a second distance. The image selection screen also displays a message prompting the user to select either the first visible light image or the second visible light image. Therefore, this configuration contributes to improved usability compared to a configuration where there is no option to select between the first and second visible light images.

[0120] In the first embodiment described above, the execution unit 15A5 prompts the user to select either a first visible light image or a second visible light image through an image selection screen. However, the technology of this disclosure is not limited thereto, and both the first visible light image and the second visible light image may be selected. In this case, for example, the first imaging process and the second imaging process may be performed alternately, and the first visible light image and the second visible light image obtained by each imaging process may be displayed on the display 46 as a live view image or a still image, etc.

[0121] Furthermore, the image selection screen may display multiple images, including a first visible light image and a second visible light image, for selection. Also, if multiple first distances are obtained by the first distance measurement, multiple first visible light images obtained by imaging under focus control based on each of the multiple first distances may be displayed on the display 46 in a selectable state on the image selection screen, or multiple first visible light images and second visible light images may be displayed in a selectable state. For example, if the imaging area includes a first mirror surface and a second mirror surface (two mirror surfaces) (i.e., there are two areas where the first and second distance measurement results differ), at least two of the following images may be displayed on the display 46 for selection: an image obtained by imaging the first mirror surface with the first mirror surface in focus, an image obtained by imaging the first image with the reflection of the first mirror surface (first image) in focus, an image obtained by imaging the second mirror surface with the second mirror surface in focus, and an image obtained by imaging the second image with the reflection of the second mirror surface (second image) in focus. Note that the number of mirror surfaces included in the imaging area is not limited to two; there may be three or more, but the same processing can be applied in this case as well. In addition, although mirror surfaces are used as an example here, it is also possible that an image may be reflected on a glossy surface (for example, a silver-plated surface or a polished metal surface). In this case as well, the image obtained by capturing an image with the glossy surface in focus, and the image obtained by capturing an image of the glossy surface with the image reflected on the glossy surface in focus, may be selectively displayed on the display 46. There may be multiple glossy surfaces included in the imaging area, but in this case as well, the same processing can be performed.

[0122] Furthermore, in the first embodiment described above, an example was given in which the user selects either the first visible light image or the second visible light image from the image selection screen via the touch panel 48 when the first distance and the second distance are different, but the technology of this disclosure is not limited thereto. For example, when the first distance and the second distance are different, the user may be allowed to pre-select whether to prioritize active distance measurement, i.e., imaging with focus control based on the distance measurement result of the first distance measurement, or passive distance measurement, i.e., imaging with focus control based on the distance measurement result of the second distance measurement.

[0123] In this case, instead of the distance measurement imaging process described in the first embodiment (see Figures 17A and 17B), the distance measurement imaging process shown in Figures 18A to 18C is executed by the CPU 15A. The distance measurement imaging process shown in Figures 18A to 18C differs from the distance measurement imaging process described in the first embodiment in that it newly includes the processes of steps ST200 to ST222. Furthermore, the distance measurement imaging process shown in Figures 18A to 18C differs from the distance measurement imaging process described in the first embodiment in that it does not have the processes of steps ST110 to ST118. Also, the distance measurement imaging process shown in Figures 18A to 18C differs from the distance measurement imaging process described in the first embodiment in that it includes the processes of steps ST224 and ST226 instead of steps ST132 and ST134. The differences between the distance measurement imaging process shown in Figures 18A to 18C and the distance measurement imaging process described in the first embodiment will be explained below.

[0124] In the distance measurement imaging process shown in Figure 18A, first, in step ST200, the execution unit 15A5 displays a distance measurement method request screen on the display 46. The distance measurement method request screen displays a message prompting the user to select whether to prioritize the active distance measurement method, i.e., the first distance measurement, or the passive distance measurement method, i.e., the second distance measurement. In the example shown in Figure 18A, the distance measurement method request screen displays the message, "Please select which distance measurement method to prioritize."

[0125] Furthermore, the distance measurement method selection screen displays two soft keys. One soft key is pressed by the user via the touch panel 48 when the user selects to prioritize using the active distance measurement method, and in the example shown in Figure 18A, it is labeled "Active Method". The other soft key is pressed by the user via the touch panel 48 when the user selects to prioritize using the passive distance measurement method, and in the example shown in Figure 18A, it is labeled "Passive Method".

[0126] In the next step, ST202, the execution unit 15A5 determines whether or not a user instruction has been received for the distance measurement method reception screen. Here, "user instruction for the distance measurement method reception screen" means that a soft key labeled "active method" or a soft key labeled "passive method" has been pressed by the user via the touch panel 48. If no user instruction has been received for the distance measurement method reception screen in step ST202, the determination is denied, and the determination in step ST202 is repeated. If a user instruction has been received for the distance measurement method reception screen in step ST202, the determination is affirmed, and the distance measurement imaging process proceeds to step ST204.

[0127] In step ST204, the execution unit 15A5 determines whether the soft key pressed by the user via the touch panel 48 is a soft key labeled "active mode". If, in step ST204, the soft key pressed by the user via the touch panel 48 is a soft key labeled "active mode", the determination is affirmed and the distance measurement and imaging process proceeds to step ST100. If, in step ST204, the soft key pressed by the user via the touch panel 48 is a soft key labeled "passive mode", the determination is denied and the distance measurement and imaging process proceeds to step ST206 shown in Figure 18B.

[0128] The processing in steps ST206 to ST212 shown in Figure 18B is the same as the processing in steps ST100 to ST106 shown in Figure 18A. In step ST214 shown in Figure 18B, the determination unit 15A4 determines whether the first distance obtained in step ST208 and the second distance obtained in step ST212 are different. In step ST214, if the first distance obtained in step ST208 and the second distance obtained in step ST212 are the same, the determination is denied, and the distance measurement imaging process proceeds to step ST130 shown in Figure 18C. In step ST214, if the first distance obtained in step ST208 and the second distance obtained in step ST212 are different, the determination is affirmed, and the distance measurement imaging process proceeds to step ST216.

[0129] In step ST216, the execution unit 15A5 determines whether or not the imaging start timing has arrived. If the imaging start timing has not arrived in step ST216, the determination is denied, and the distance measurement imaging process proceeds to step ST222. If the imaging start timing has arrived in step ST216, the determination is affirmed, and the distance measurement imaging process proceeds to step ST218.

[0130] In step ST218, the execution unit 15A5 performs the same process as in step ST132 shown in Figure 17B, and in the next step ST220, the execution unit 15A5 performs the same process as in step ST134 shown in Figure 17B. In the next step ST222, the execution unit 15A5 determines whether or not the termination conditions have been met. In step ST222, if the termination conditions have not been met, the determination is denied, and the distance measurement and imaging process proceeds to step ST216. In step ST222, if the termination conditions have been met, the determination is affirmed, and the distance measurement and imaging process ends.

[0131] In step ST130 shown in Figure 18C, if the imaging start timing has not yet arrived, the determination is rejected and the distance measurement imaging process proceeds to step ST136. In step ST130 shown in Figure 18C, if the imaging start timing has arrived, the determination is affirmed and the distance measurement imaging process proceeds to step ST224.

[0132] In step ST224, the first distance measurement control unit 15A1 controls the distance measurement imaging device 14 to cause the first distance measurement system processing circuit 34B to perform the first distance measurement.

[0133] In the next step, ST226, the execution unit 15A5 performs a first imaging process using the first distance obtained by the first distance measurement performed in step ST224. The execution unit 15A5 also performs a first visible light image display process using the latest first visible light image obtained by performing the first imaging process.

[0134] In step ST136 shown in Figure 18C, if the termination condition is not met, the determination is rejected, and the distance measurement and imaging process proceeds to step ST130 shown in Figure 18C. In step ST136 shown in Figure 18C, if the termination condition is met, the determination is affirmed, and the distance measurement and imaging process ends.

[0135] Thus, if the execution unit 15A5 confirms the determination in step ST204 shown in Figure 18A, and also confirms the determination in step ST108 shown in Figure 18A, then the first distance measurement is performed in step ST122 shown in Figure 18A. Then, in step ST124 shown in Figure 18A, the first imaging process is performed using the first distance obtained by the first distance measurement in step ST122. That is, in step ST124 shown in Figure 18A, imaging is performed with focus control based on the first distance obtained by the first distance measurement in step ST122.

[0136] On the other hand, if the execution unit 15A5 denies the determination in step ST204 shown in Figure 18A and affirms the determination in step ST214 shown in Figure 18B, then a second distance measurement is performed in step ST218 shown in Figure 18B. Then, in step ST220 shown in Figure 18B, a second imaging process is performed using the second distance obtained by the second distance measurement in step ST218. That is, in step ST220 shown in Figure 18B, imaging is performed with focus control based on the second distance obtained by the second distance measurement in step ST218.

[0137] Therefore, according to the distance measurement and imaging process shown in Figures 18A to 18C, when the first distance obtained by the first distance measurement and the second distance obtained by the second distance measurement are different, imaging is always performed under focus control based on the first distance, and the power consumption required for focus control and imaging can be reduced compared to when imaging is performed under focus control based on the second distance.

[0138] Furthermore, in the distance measurement imaging process shown in Figures 18A to 18C, if the user presses the soft key labeled "Active Method" on the distance measurement method reception screen via the touch panel 48, the first imaging process in step ST124 shown in Figure 18A will perform focus control based on the first distance. Also, if the user presses the soft key labeled "Passive Method" on the distance measurement method reception screen via the touch panel 48, the second imaging process in step ST220 shown in Figure 18B will perform focus control based on the second distance. In other words, focus control is performed based on either the first distance or the second distance, whichever is determined according to the instructions given by the user.

[0139] Therefore, the distance measurement and imaging process shown in Figures 18A to 18C makes it easier to achieve the focus control intended by the user compared to a case where focus control is not performed based on instructions given by the user.

[0140] Here, we have exemplified focus control based on a distance determined according to instructions given by the user, but the technology of this disclosure is not limited to this. For example, instead of instructions given by the user, instructions given by an external device (e.g., a personal computer and / or server, etc.) that can communicate with the smart device 10 may be applied. In this case, for example, the external device may selectively give the smart device 10 instructions for active distance measurement and instructions for passive distance measurement depending on the time of day and / or weather conditions.

[0141] In relatively dark environments such as at night and / or on cloudy days, the accuracy of active distance measurement is generally considered to be higher than that of passive distance measurement. Therefore, if the external device selectively provides instructions for active distance measurement and passive distance measurement to the smart device 10 depending on the time of day, for example, it could provide instructions for active distance measurement to the smart device 10 at night and instructions for passive distance measurement to the smart device 10 during the day. Alternatively, if the external device selectively provides instructions for active distance measurement and passive distance measurement to the smart device 10 depending on the weather, for example, it could provide instructions for passive distance measurement to the smart device 10 when the weather is sunny and instructions for active distance measurement to the smart device 10 when the weather is not sunny. Furthermore, the external device may be configured to provide instructions for active distance measurement to the smart device 10 when it is nighttime and the weather is not clear, or to provide instructions for passive distance measurement to the smart device 10 when it is daytime and the weather is clear.

[0142] Thus, in cases where an external device provides instructions to the smart device 10 without the user giving instructions, for example, in the distance measurement and imaging process shown in Figure 18A, the processing in step ST200 becomes unnecessary, and in step ST202, the execution unit 15A5 can determine whether or not instructions were given from the external device.

[0143] Furthermore, while the first embodiment described above provides an example where the entire mirror 100 is imaged as the subject by the smart device 10, it is conceivable that using the distance measurement result obtained by performing the first distance measurement based on the IR reflected light of the mirror 100 for focus control when the mirror 100 is included in the imaging area may not be in line with the user's intention. In this case, the distance measurement and imaging process shown in Figure 19 can be performed by the CPU 15A.

[0144] The distance measurement imaging process shown in Figure 19 differs from the distance measurement imaging processes shown in Figures 17A and 17B in that it includes steps ST300 and ST302 between steps ST100 and ST102. Furthermore, the distance measurement imaging process shown in Figure 19 differs from the distance measurement imaging processes shown in Figures 17A and 17B in that it does not include steps ST110 to ST118. Also, the distance measurement imaging process shown in Figure 19 differs from the distance measurement imaging processes shown in Figures 17A and 17B in that it includes step ST108A instead of step ST108. Moreover, the distance measurement imaging process shown in Figure 19 differs from the distance measurement imaging process described in the first embodiment above in that it includes steps ST304 and ST306 instead of steps ST122 and ST124. The differences between the distance measurement imaging process shown in Figure 19 and the distance measurement imaging process described in the first embodiment above will be explained below.

[0145] In step ST300 shown in Figure 19, the acquisition unit 15A3 acquires IR pixel data from the first distance measuring system processing circuit 34B as data indicating the amount of IR reflected light received by the first distance measuring system division area 26N2 (see Figure 5). Here, for example, IR pixel data for each IR pixel included in the ROI of the first distance measuring system division area 26N2 is acquired by the acquisition unit 15A3.

[0146] In the next step, ST302, the determination unit 15A4 refers to the IR pixel data acquired in step ST300 and determines whether the amount of light received by the IR pixel showing the maximum amount of light received in the ROI of the first distance measuring system division area 26N2 is equal to or greater than a threshold.

[0147] The threshold used for comparison with the amount of light received by the IR pixel is a value derived in advance through actual device testing and / or computer simulation, etc., when the smart device 10 irradiates a laser beam onto a mirror surface placed at a predetermined distance (e.g., several tens of meters) away, and the IR reflected light obtained from the reflection of the laser beam onto the mirror surface is received by the IR pixel. Here, a mirror surface is used as an example of the target of the laser beam irradiation, but it is not limited to this, and any surface with a lower reflectivity than a mirror surface may be used. Examples of surfaces with a lower reflectivity than a mirror surface include glass surfaces, the surface of a translucent plastic plate-like member, and glossy surfaces.

[0148] In step ST302, if the amount of light received is less than the threshold, the determination is rejected, and the distance measurement imaging process proceeds to step ST130 as shown in Figure 18C. In step ST302, if the amount of light received is equal to or greater than the threshold, the determination is affirmed, and the distance measurement imaging process proceeds to step ST102.

[0149] In step ST108A, the determination unit 15A4 determines whether the first distance obtained in step ST102 and the second distance obtained in step ST106 are different. If the first distance obtained in step ST102 and the second distance obtained in step ST106 are the same in step ST108A, the determination is denied and the distance measurement imaging process proceeds to step ST130 shown in Figure 18C. If the first distance obtained in step ST102 and the second distance obtained in step ST106 are different in step ST108A, the determination is affirmed and the distance measurement imaging process proceeds to step ST120.

[0150] In step ST304, the second distance measurement control unit 15A2 controls the distance measurement imaging device 14 to cause the second distance measurement system processing circuit 34C to perform the second distance measurement.

[0151] In the next step, ST306, the execution unit 15A5 performs a second imaging process using the second distance obtained by the second distance measurement in step ST304. That is, the execution unit 15A5 performs focus control based on the second distance obtained by the second distance measurement in step ST304 and causes the photoelectric conversion element 26 to take an image. The execution unit 15A5 also performs a second visible light image display process using the latest second visible light image obtained by performing the second imaging process.

[0152] Thus, by performing the distance measurement imaging process shown in Figure 19, focus control is performed based on the second distance when the amount of IR reflected light received by the IR pixel is above the threshold. This makes it easier to achieve imaging with the user's intended focus state (for example, imaging with focus on a location other than the mirror 100) compared to the case where focus control is performed based on the first distance even though the amount of IR reflected light received by the IR pixel is above the threshold (for example, when the image is focused on the mirror 100).

[0153] Furthermore, in the first embodiment described above, the determination unit 15A4 determines whether the first distance and the second distance are different regardless of whether the imaging area contains a specific subject, but the technology of this disclosure is not limited thereto. For example, the determination unit 15A4 may determine whether the first distance and the second distance are different only if the imaging area contains a specific subject.

[0154] In this case, the distance measurement and imaging process shown in Figure 20 is executed by the CPU 15A. The distance measurement and imaging process shown in Figure 20 differs from the distance measurement and imaging process described in the first embodiment above in that it has steps ST350 and ST352 between steps ST106 and ST108. Furthermore, the distance measurement and imaging process shown in Figure 20 differs from the distance measurement and imaging process described in the first embodiment above in that it has step ST108B instead of step ST108. Moreover, the distance measurement and imaging process shown in Figure 20 differs from the distance measurement and imaging process described in the first embodiment above in that it has steps ST354 and ST356 instead of steps ST122 and ST124. The differences between the distance measurement and imaging process shown in Figure 20 and the distance measurement and imaging process described in the first embodiment above will be explained below.

[0155] In step ST350 shown in Figure 20, the execution unit 15A5 performs a specific subject detection process. The specific subject detection process involves controlling the focus of the light receiver 18 based on the first distance acquired in step ST102 to image the imaging area, and then detecting a specific subject image representing the specific subject in the image of the imaging area obtained by imaging the imaging area. Here, the image of the imaging area is an example of the "second image" related to the technology of this disclosure. Here, the specific subject includes a high reflectance region with a predetermined reflectance or higher. The predetermined reflectance is, for example, a reflectance predetermined as the average reflectance of a glossy surface. Here, a mirror surface 100A is used as the high reflectance region. The mirror surface 100A reflects the user image and the smart device image.

[0156] In step ST350, a specific subject image is detected on an image of the imaging area obtained by controlling the focus based on the first distance acquired in step ST102, but the technology of this disclosure is not limited to this. For example, a specific subject image may be detected on an image obtained by controlling the focus based on the second distance acquired in step ST106 and capturing the imaging area.

[0157] In step ST350, the execution unit 15A5 uses a machine learning algorithm (e.g., a neural network) to detect a specific subject image in the captured area image. Alternatively, the execution unit 15A5 may use a dictionary for pattern matching to detect a specific subject image in the captured area image.

[0158] In the next step, ST352, the execution unit 15A5 determines whether or not a specific subject image has been detected from the image capture area image. If, in step ST352, the specific subject image has not been detected from the image capture area image, the determination is denied, and the distance measurement imaging process proceeds to step ST130 shown in Figure 18C. If, in step ST352, the specific subject image has been detected from the image capture area image, the determination is affirmed, and the distance measurement imaging process proceeds to step ST108B.

[0159] In step ST108B, the determination unit 15A4 determines whether the first distance obtained in step ST102 and the second distance obtained in step ST106 are different. If the first distance obtained in step ST102 and the second distance obtained in step ST106 are the same in step ST108B, the determination is denied, and the distance measurement imaging process proceeds to step ST130 shown in Figure 18C. If the first distance obtained in step ST102 and the second distance obtained in step ST106 are different in step ST108B, the determination is affirmed, and the distance measurement imaging process proceeds to step ST120.

[0160] In step ST354, the second distance measurement control unit 15A2 controls the distance measurement imaging device 14 to cause the second distance measurement system processing circuit 34C to perform the second distance measurement.

[0161] In the next step, ST356, the execution unit 15A5 performs a second imaging process using the second distance obtained by the second distance measurement in step ST354. That is, the execution unit 15A5 performs focus control based on the second distance obtained by the second distance measurement in step ST354 and causes the photoelectric conversion element 26 to take an image. The execution unit 15A5 also performs a second visible light image display process using the latest second visible light image obtained by performing the second imaging process.

[0162] As described above, when the distance measurement imaging process shown in Figure 20 is executed, focus control is performed based on the second distance when a specific subject image is detected by the execution unit 15A5. This makes it easier to achieve imaging in the user's intended focused state (for example, imaging with the user's image reflected in the mirror 100) compared to when focus control is performed based on the first distance even though a specific subject image has been detected (for example, when the focus is set on the mirror 100).

[0163] Furthermore, the specific subject includes a high-reflectivity region exceeding a predetermined reflectivity, and the mirror surface 100A is used as the high-reflectivity region. The user image and the smart device image are reflected in the mirror surface 100A. Therefore, with this configuration, it is easier to achieve imaging in the in-focus state intended by the user (for example, imaging with the user image reflected in the mirror 100) compared to the case where focus control is performed based on the first distance even though the mirror surface 100A, which reflects the user image and the smart device image, is detected (for example, when the focus is set on the mirror surface 100A).

[0164] The distance measurement imaging process shown in Figure 21 differs from the distance measurement imaging process shown in Figure 20 in that it includes steps ST400 to ST406 instead of steps ST350 and ST352. Below, the differences between the distance measurement imaging process shown in Figure 21 and the distance measurement imaging process described in Figure 20 will be explained.

[0165] In step ST400 shown in Figure 21, the execution unit 15A5 performs a high reflectivity region detection process. The high reflectivity region detection process involves controlling the focus of the photodetector 18 based on the first distance acquired in step ST102 to image the imaging region, and then detecting the high reflectivity region that indicates the high reflectivity region described above in the image of the imaging region obtained by the imaging region.

[0166] In step ST400, high reflectivity regions are detected in the image of the imaging region obtained by controlling the focus based on the first distance acquired in step ST102, but the technology of this disclosure is not limited to this. For example, high reflectivity regions may be detected in the image obtained by controlling the focus based on the second distance acquired in step ST106 and capturing the imaging region.

[0167] In the next step, ST402, the execution unit 15A5 determines whether or not a high-reflectivity region has been detected in the image of the captured area. If no high-reflectivity region is detected in the image of the captured area in step ST402, the determination is denied, and the distance measurement imaging process proceeds to step ST130 shown in Figure 18C. If a high-reflectivity region is detected in the image of the captured area in step ST402, the determination is affirmed, and the distance measurement imaging process proceeds to step ST404.

[0168] In step ST404, the execution unit 15A5 calculates the high reflectivity region occupancy rate. The high reflectivity region occupancy rate refers to the ratio of the high reflectivity region detected by the high reflectivity region detection process to the image of the captured region.

[0169] In the next step, ST406, the determination unit 15A4 determines whether the high reflectivity region occupancy rate calculated in step ST404 exceeds a predetermined occupancy rate (for example, 80%). If the high reflectivity region occupancy rate calculated in step ST404 is less than or equal to the predetermined occupancy rate in step ST406, the determination is rejected, and the distance measurement imaging process proceeds to step ST130 shown in Figure 18C. This executes the processes from steps ST130 to ST136. That is, imaging of the imaging area is performed by the photodetector 18 under focus control based on the first distance obtained by the first distance measurement in step ST224.

[0170] Furthermore, in step ST406, if the high reflectance region occupancy rate calculated in step ST404 exceeds the predetermined occupancy rate, the determination is affirmed, and the distance measurement imaging process proceeds to step ST108B. Then, if the first distance obtained in step ST102 and the second distance obtained in step ST106 are different (if the determination is affirmed in step ST108B), the processing in steps ST120 to ST126 is executed. That is, imaging of the imaging area is performed by the photodetector 18 under focus control based on the second distance obtained by the second distance measurement in step ST354 (step ST356).

[0171] As described above, by performing the distance measurement imaging process shown in Figure 21, imaging is performed under focus control based on a first or second distance determined according to the high reflectivity area occupancy rate. This makes it easier to achieve imaging in the in-focus state intended by the user compared to when focus control for the photodetector 18 is always performed based on only one of the first or second distance. In other words, it becomes possible to accurately respond to the user's request to focus on the entire mirror 100 and the user's request to focus on the user's image reflected in the mirror 100.

[0172] Here, the high reflectivity region occupancy rate is exemplified as the ratio of the high reflectivity region detected by the high reflectivity region detection process to the captured region image, but the technology of this disclosure is not limited to this. For example, the high reflectivity region occupancy rate may be the difference between the area of ​​the high reflectivity region in the captured region image and the area of ​​a region in the captured image that is not a high reflectivity region.

[0173] Furthermore, although the first embodiment described above illustrates an example in which the imaging area is imaged solely by the light receiver 18, the technology of this disclosure is not limited thereto. For example, as shown in Figure 22, the smart device 300 may be equipped with a wide-angle light receiver 350 in addition to the distance measuring and imaging device 14 described above. The smart device 300 differs from the smart device 10 in that it further includes a wide-angle light receiver 350.

[0174] The wide-angle receiver 350 is capable of imaging at a wider angle than the receiver 18. In other words, the receiver 18 is capable of imaging at a longer focal length than the wide-angle receiver 350. That is, the wide-angle receiver 350 has a longer focal length than the receiver 18. Furthermore, the wide-angle receiver 350 is equipped with a light-transmitting window 352 and a photoelectric conversion element 354. The light-transmitting window 352 has the same structure as the light-transmitting windows 20 and 20, and the photoelectric conversion element 354 has a region similar to the visible light image division region 26N1 of the photoelectric conversion element 26.

[0175] In the smart device 300 configured in this way, the CPU 15A executes the distance measurement and imaging process shown in Figure 23. The distance measurement and imaging process shown in Figure 23 differs from the distance measurement and imaging process described in the first embodiment above in that it has step ST450 and does not have steps ST110 to ST118. The following will explain the differences between the distance measurement and imaging process shown in Figure 23 and the distance measurement and imaging process described in the first embodiment above. Furthermore, the following explanation of the distance measurement and imaging process shown in Figure 23 will assume that imaging is performed by the light receiver 18 or the wide-angle light receiver 350.

[0176] In the distance measurement and imaging process shown in Figure 23, first, in step ST450, the determination unit 15A4 determines whether or not the wide-angle light receiver 350 is in use. That is, in step ST450, the determination unit 15A4 determines whether or not imaging is being performed by the wide-angle light receiver 350. In step ST450, if the light receiver 18 is in use, the determination is denied, and the distance measurement and imaging process proceeds to step ST206 shown in Figure 18B. In step ST450, if the wide-angle light receiver 350 is in use, the determination is affirmed, and the distance measurement and imaging process proceeds to step ST100. In this case, on the condition that the determination was affirmed in step ST108, the processes in steps ST120 to ST126 are executed. As a result, in step ST124, imaging is performed under focus control based on the first distance obtained by the first distance measurement performed in step ST122.

[0177] Here, for example, if the entire mirror 100 is to be imaged as the subject, it is assumed that the image will be taken at a wide-angle setting rather than a telephoto setting so that the entire mirror 100 is included in the imaging area. Therefore, in order to accurately measure the distance to the mirror surface 100A, it is desirable to perform the first distance measurement, which is advantageous for focusing on the mirror surface 100A, rather than the second distance measurement, which is advantageous for focusing on the user image reflected in the mirror 100.

[0178] Therefore, when the distance measurement imaging process shown in Figure 23 is performed, imaging is carried out under focus control based on the first distance while the wide-angle receiver 350 is in use. This makes it easier to achieve the user's intended in-focus state compared to the case where imaging is carried out under focus control based on the second distance, even though imaging is performed at a wider angle than the receiver 18.

[0179] Furthermore, in the first embodiment described above, the light irradiator 16 used laser light as an example of light for distance measurement, but the technology of this disclosure is not limited thereto. For example, the light for distance measurement may be directional light such as superluminescent light, light emitted from a xenon flash light source, or light emitted from an LED.

[0180] Furthermore, in the first embodiment described above, the visible light image division area 26N1, the first distance measuring system division area 26N2, and the second distance measuring system division area 26N3 are integrated into a single chip by the photoelectric conversion element 26. However, the technology of this disclosure is not limited thereto, and multiple visible light pixels may be integrated into a single chip, multiple phase difference pixels may be integrated into a single chip, and multiple IR pixels may be integrated into a single chip. Alternatively, multiple visible light pixels and multiple phase difference pixels may be integrated into a single chip, and multiple IR pixels may be integrated into a single chip. In this way, when different types of photosensitive pixels are integrated into chips for each type and mounted on the smart device 10, an optical system such as an objective lens, a focus lens, and an aperture should be provided on the subject side (object side) for each chip.

[0181] Furthermore, although the first embodiment described above included an example in which the distance measuring and imaging device 14 is built into the smart device 10, the technology of this disclosure is not limited to this. For example, the distance measuring and imaging device 14 may be externally attached to a general smart device, i.e., a smart device that does not have a built-in distance measuring and imaging device 14.

[0182] Furthermore, although the first embodiment described above included an example in which the UI system device 44 is incorporated into the smart device 10, at least some of the multiple components included in the UI system device 44 may be externally connected to the smart device 10. Alternatively, at least some of the multiple components included in the UI system device 44 may be used by being connected to an external I / F 52 as a separate unit.

[0183] Furthermore, although Figure 1 illustrates a smart device 10, the technology of this disclosure is not limited to this. That is, the technology of this disclosure is also applicable to various electronic devices (e.g., interchangeable lens cameras, fixed lens cameras, personal computers, and / or wearable terminal devices, etc.) that incorporate a distance measuring and imaging device 14, and the same functions and effects as those of the smart device 10 can be obtained with these electronic devices as well.

[0184] Furthermore, although the first embodiment described above illustrates a display 46, the technology of this disclosure is not limited thereto. For example, an external display attached to the smart device 10 may be used in conjunction with the display 46.

[0185] Furthermore, although the photoelectric conversion element 26 and the signal processing circuit 34 are separate components in the first embodiment described above, a stacked image sensor in which the photoelectric conversion element 26 and the signal processing circuit 34 are integrated into a single chip may be used. Alternatively, at least a portion of the signal processing circuit 34 may be eliminated, and the functions of the signal processing circuit 34 may be assigned to the CPU 15A.

[0186] Furthermore, although the first embodiment described above included an example in which an imaging timing signal is supplied from the photoelectric conversion element driver 32 to the photoelectric conversion element 26, the technology of this disclosure is not limited to this. For example, the photoelectric conversion element driver 32 may not be necessary, in which case the functions of the photoelectric conversion element driver 32 can be assigned to the CPU 15A.

[0187] Furthermore, although the first embodiment described above illustrates an example in which the second distance measurement is performed using phase difference pixels, the technology of this disclosure is not limited thereto. For example, instead of distance measurement using phase difference pixels, distance measurement may be performed using a stereo camera, or distance measurement may be performed using face detection. In distance measurement using a stereo camera, the distance to the subject is measured by using the parallax of a pair of images obtained from the stereo camera. In distance measurement using face detection, the distance to the subject is measured by using the ratio of the size of the detected face image to the size of the image for one frame, etc.

[0188] Furthermore, in the first embodiment described above, an example was given in which the G filter, R filter, and B filter also function as infrared light cut filters that cut infrared light. However, the technology of this disclosure is not limited to this, and each color filter corresponding to the R pixel, G pixel, and B pixel may be a color filter that also transmits infrared light, and a pair of photodiodes (e.g., InGaAs APD) consisting of a photodiode for the visible light pixel and a photodiode for the IR pixel may be arranged for each color filter.

[0189] Furthermore, although the first embodiment described above included an example of a configuration in which the visible light image section 26N1 and the second distance measuring system section 26N3 are used in combination, the technology of this disclosure is not limited thereto. For example, instead of the visible light image section 26N1 and the second distance measuring system section 26N3, an area sensor may be used in which visible light pixel data and phase difference pixel data are selectively generated and read out. In this case, the area sensor has a plurality of photosensitive pixels arranged in two dimensions. For example, a pair of independent photodiodes without light-shielding members are used as photosensitive pixels in the area sensor. When visible light pixel data is generated and read out, photoelectric conversion is performed by the entire area of ​​the photosensitive pixels (the pair of photodiodes), and when phase difference pixel data is generated and read out (for example, when performing passive distance measurement), photoelectric conversion is performed by one of the photodiodes in the pair. Here, one of the pair of photodiodes is a photodiode corresponding to the first phase difference pixel L described in the first embodiment, and the other of the pair of photodiodes is a photodiode corresponding to the second phase difference pixel R described in the first embodiment. While it is possible to selectively generate and read out visible light pixel data and phase difference pixel data using all the photosensitive pixels included in the area sensor, the system is not limited to this, and may also be configured so that visible light pixel data and phase difference pixel data are selectively generated and read out using some of the photosensitive pixels included in the area sensor.

[0190] [Second Embodiment] In the first embodiment described above, an example was given in which a specific process is performed by the execution unit 15A5 when the first distance obtained by the first distance measurement and the second distance obtained by the second distance measurement are different. In this second embodiment, an example will be described in which a specific process is performed by the execution unit 15A5 when the first focused position and the second focused position obtained by different focusing methods are different. In this second embodiment, components that are the same as those described in the first embodiment are denoted by the same reference numerals and their descriptions are omitted, and the parts that differ from the first embodiment will be described.

[0191] As an example, as shown in Figure 24, the storage 15B stores the distance measurement and imaging processing program 570. The CPU 15A reads the distance measurement and imaging processing program 570 from the storage 15B. Then, the CPU 15A executes the distance measurement and imaging processing program 570 read from the storage 15B, and operates as the first distance measurement control unit 15A1, acquisition unit 15A3, determination unit 15A4, execution unit 15A5, first focus position calculation unit 15A6, contrast AF method imaging control unit 15A7, and second focus position calculation unit 15A8.

[0192] As an example, as shown in Figure 25, the first distance measurement control unit 15A1 causes the first distance measurement system processing circuit 34B to perform a first distance measurement, similar to the first embodiment described above, and the acquisition unit 15A3 acquires a first distance from the first distance measurement system processing circuit 34B. The first focus position calculation unit 15A6 calculates a first focus position that is in focus on the subject included in the imaging area, based on the first distance acquired by the acquisition unit 15A3. Specifically, the first focus position calculation unit 15A6 calculates the first focus position using a predetermined calculation formula in which the distance from the smart device 10 to the subject is the independent variable and the focus position is the dependent variable. However, the technology of this disclosure is not limited to this, and the first focus position calculation unit 15A6 may derive the first focus position using the focus position derivation table 72 shown in Figure 13.

[0193] As an example, as shown in Figure 26, when the contrast AF imaging control unit 15A7 instructs the light receiving unit 18 to perform imaging using the contrast AF method, it outputs an imaging start signal to the light receiving unit 18 and the second focus position calculation unit 15A8. By outputting the imaging start signal to the light receiving unit 18, the contrast AF imaging control unit 15A7 instructs the visible light image division area 26N1 (see Figure 5) to perform imaging at a predetermined frame rate (for example, 60 fps), and causes the visible light pixel data processing circuit 34A to generate a third visible light image indicating the imaging area. The visible light pixel data processing circuit 34A outputs the third visible light image to the image memory 42. The image memory 42 stores the third visible light image, and the third visible light image in the image memory 42 is updated each time a third visible light image is input from the visible light pixel data processing circuit 34A.

[0194] Furthermore, the contrast AF imaging control unit 15A7 outputs a motor control signal to the photodetector 18 and the second focus position calculation unit 15A8 in parallel with the output of the imaging start signal. The motor control signal is a signal to cause the focus lens 30B to move back and forth (wobble) along the optical axis L2, and is input to the motor driver 64 of the photodetector 18. The motor driver 64 drives the motor 62 according to the input motor control signal to cause the focus lens 30B to wobble (see Figure 8). Meanwhile, the second focus position calculation unit 15A8 calculates the current position of the focus lens 30B on the optical axis L2 using the motor control signals that have been input from the time the imaging start signal was input from the contrast AF imaging control unit 15A7 until the present.

[0195] The contrast AF imaging control unit 15A7 acquires the third visible light image from the image memory 42 each time the third visible light image in the image memory 42 is updated, and calculates the contrast value of the acquired third visible light image. The contrast AF imaging control unit 15A7 then searches for the maximum contrast value of the third visible light image, and when the maximum value is found, outputs a maximum value reached signal to the second focus position calculation unit 15A8. The maximum value reached signal is a signal indicating that the contrast of the third visible light image has reached its maximum value.

[0196] When the second focus position calculation unit 15A8 receives a maximum value signal, it uses the motor control signals received from the time the imaging start signal was received until the present time to calculate the current position of the focus lens 30B on the optical axis L2 as the second focus position. The second focus position refers to the focus position where the subject included in the imaging area is in focus. Here, the second focus position calculation unit 15A8 calculates the second focus position using an arithmetic formula in which the motor control signal is the independent variable and the focus position is the dependent variable. However, the technology of this disclosure is not limited to this, and the second focus position calculation unit 15A8 may derive the second focus position using a table that associates time-series data of the motor control signals with the position of the focus lens 30B on the optical axis L2.

[0197] As an example, as shown in Figure 27, the determination unit 15A4 determines whether the first focus position calculated by the first focus position calculation unit 15A6 and the second focus position calculated by the second focus position calculation unit 15A8 are different. If the determination unit 15A4 determines that the first focus position and the second focus position are the same, the execution unit 15A5 executes the first imaging process described in the first embodiment above. If the determination unit 15A4 determines that the first focus position and the second focus position are different, the execution unit 15A5 executes a specific process (see Figure 28).

[0198] As an example, as shown in Figure 28, a specific process differs from the first embodiment in that it includes a third imaging process instead of the second imaging process. The third imaging process refers to the process of performing imaging with focus control for the second focus position (imaging with contrast AF). In addition, the image selection screen generated by the image selection screen display process included in the specific process and displayed on the display 46 differs in some of its display content from the first embodiment. Specifically, the image selection screen displays the first focus position (in the example shown in Figure 28, a value of "X1mm" indicating the distance from the reference position to the focus lens 30B) instead of the first distance, and the second focus position (in the example shown in Figure 28, a value of "X2mm" indicating the distance from the reference position to the focus lens 30B) instead of the second distance. The reference position refers to, for example, the position of the imaging plane of the photoelectric conversion element 26, or the position of the focus lens 30B when focused at infinity. Also, the image selection screen displays the third visible light image instead of the second visible light image. Furthermore, the image selection screen displays the message "Focus position is different" instead of the message "Distance measurement result is different." Also, the image selection screen displays the message "Active AF method" instead of the message "Active method." In addition, the image selection screen displays the message "Contrast AF method" instead of the message "Passive method."

[0199] In this second embodiment, the display examples "Active AF method" and "Contrast AF method" are shown, but the display does not necessarily have to be "Active AF method" and "Passive AF method." Any display is acceptable as long as the user can understand the difference between the AF methods. For example, when performing AF using laser ranging and contrast AF, the display could be "AF using laser ranging" and "AF using contrast," or an icon representing the AF method could be displayed. Alternatively, the focus position could be displayed instead of the AF method. For example, it could be "Focus position: Foreground" and "Focus position: Background," or "Focus position: Object" and "Focus position: Image reflected in the object," etc. Alternatively, two or more elements such as text and icons indicating the AF method and the focus position could be combined and displayed.

[0200] Next, the operation of the part of the smart device 500 relating to the technology of this disclosure will be explained with reference to Figures 29A to 29C. Figures 29A to 29C are flowcharts showing an example of the flow of the distance measurement and imaging process executed by the CPU 15A according to the distance measurement and imaging processing program 570.

[0201] In the distance measurement and imaging process shown in Figure 29A, first, in step ST500, the first distance measurement control unit 15A1 controls the distance measurement and imaging device 14 to cause the first distance measurement system processing circuit 34B to perform a first distance measurement. The first distance measurement is achieved by the first distance measurement control unit 15A1 causing the first distance measurement system processing circuit 34B to calculate a first distance.

[0202] In the next step, ST502, the acquisition unit 15A3 acquires the first distance obtained by the first distance measurement performed in step ST500.

[0203] In the next step, ST504, the first focus position calculation unit 15A6 calculates the first focus position based on the first distance obtained in step ST502.

[0204] In the next step, ST506, the contrast AF imaging control unit 15A7 instructs the light receiver 18 to start imaging using the contrast AF method.

[0205] In the next step, ST508, the contrast AF imaging control unit 15A7 acquires the third visible light image generated by the visible light pixel data processing circuit 34A when contrast AF imaging is performed.

[0206] In the next step, ST510, the second focus position calculation unit 15A8 determines whether or not a maximum value reached signal has been input from the contrast AF system imaging control unit 15A7. If no maximum value reached signal has been input from the contrast AF system imaging control unit 15A7 in step ST510, the determination is denied, and the distance measurement imaging process proceeds to step ST508. If a maximum value reached signal has been input from the contrast AF system imaging control unit 15A7 in step ST510, the determination is affirmed, and the distance measurement imaging process proceeds to step ST512.

[0207] In step ST512, the second focus position calculation unit 15A8 uses the motor control signals input since the processing in step ST506 was executed to calculate the current position of the focus lens 30B on the optical axis L2 as the second focus position.

[0208] In step ST514, shown in Figure 29B, the determination unit 15A4 determines whether the first focus position calculated in step ST504 and the second focus position calculated in step ST512 are different. If the first focus position calculated in step ST504 and the second focus position calculated in step ST512 are the same in step ST514, the determination is rejected, and the distance measurement imaging process moves from step ST514 to step ST516.

[0209] In step ST516, the execution unit 15A5 performs a first imaging process using the first distance acquired in step ST502. Here, the first imaging process using the first distance refers to, for example, the process of performing imaging with focus control for the first focus position calculated based on the first distance in step ST504.

[0210] In the next step, ST518, the execution unit 15A5 performs a third imaging process using the second focus position calculated in step ST512. Here, the first imaging process using the second focus position refers to the process of imaging with focus control relative to the second focus position.

[0211] In the next step, ST520, the execution unit 15A5 performs the image selection screen display process. As a result, the image selection screen is displayed on the display 46, as shown in Figure 28 as an example.

[0212] In the next step, ST522, the execution unit 15A5 determines whether either the first visible light image or the third visible light image displayed on the image selection screen has been selected by the user via the touch panel 48. If, in step ST522, either the first visible light image or the third visible light image displayed on the image selection screen has not been selected by the user via the touch panel 48, the determination is denied, and the determination in step ST522 is repeated. If, in step ST522, either the first visible light image or the third visible light image displayed on the image selection screen has been selected by the user via the touch panel 48, the determination is affirmed, and the distance measurement imaging process proceeds to step ST524.

[0213] In step ST524, the execution unit 15A5 determines whether the image selected from the image selection screen is a third visible light image. If the image selected from the image selection screen in step ST524 is a first visible light image, the determination is denied, and the distance measurement imaging process proceeds to step ST532 shown in Figure 29C. If the image selected from the image selection screen in step ST524 is a third visible light image, the determination is affirmed, and the distance measurement imaging process proceeds to step ST526.

[0214] In step ST526, the execution unit 15A5 determines whether or not the imaging start timing has arrived. If the imaging start timing has not arrived in step ST526, the determination is denied, and the distance measurement imaging process proceeds to step ST530. If the imaging start timing has arrived in step ST526, the determination is affirmed, and the distance measurement imaging process proceeds to step ST528.

[0215] In step ST528, the execution unit 15A5 performs a third imaging process. The execution unit 15A5 also performs a third visible light image display process using the latest third visible light image obtained by performing the third imaging process. Here, the third visible light image display process differs from the second visible light image display process shown in Figure 16 in that it displays the third visible light image on the display 46 instead of the second visible light image, and displays the message "Contrast AF method" on the display 46 instead of the message "Passive method".

[0216] In step ST528, an example is given in which a third imaging process is performed on the condition that a third visible light image is selected in steps ST522 and ST524, and the latest third visible light image obtained by the third imaging process is displayed on the display 48. However, this is merely one example. For example, on the condition that a third visible light image is selected, the selected third visible light image may be stored in a storage medium such as storage 15B and / or a memory card. Furthermore, imaging with the selected AF method and saving the selected image (for example, the first visible light image or the third visible light image) may be selected according to instructions received by the receiving device 47 (instructions given by the user).

[0217] In the next step, ST530, the execution unit 15A5 determines whether the termination conditions have been met. If the termination conditions are not met in step ST530, the determination is denied, and the distance measurement and imaging process proceeds to step ST526. If the termination conditions are met in step ST530, the determination is affirmed, and the distance measurement and imaging process ends.

[0218] On the other hand, in step ST532 shown in Figure 29C, the execution unit 15A5 determines whether or not the imaging start timing has arrived. If the imaging start timing has not arrived in step ST532, the determination is denied, and the distance measurement imaging process proceeds to step ST536. If the imaging start timing has arrived in step ST532, the determination is affirmed, and the distance measurement imaging process proceeds to step ST534.

[0219] In step ST534, the first distance measurement control unit 15A1 controls the distance measurement imaging device 14 to cause the first distance measurement system processing circuit 34B to perform the first distance measurement.

[0220] In the next step, ST536, the execution unit 15A5 performs a first imaging process using the first distance obtained by the first distance measurement performed in step ST534. The execution unit 15A5 also performs a first visible light image display process using the latest first visible light image obtained by performing the first imaging process.

[0221] In the next step, ST538, the execution unit 15A5 determines whether the termination conditions have been met. If the termination conditions are not met in step ST538, the determination is denied, and the distance measurement and imaging process proceeds to step ST532. If the termination conditions are met in step ST538, the determination is affirmed, and the distance measurement and imaging process ends.

[0222] As explained above, in the smart device 500, if the first focus position calculated based on the first distance obtained by the first distance measurement differs from the second focus position obtained by the contrast AF method imaging, the execution unit 15A5 performs a specific process. Therefore, this configuration can help solve various problems that arise when the first distance and the second distance differ.

[0223] It should be noted that the distance measurement imaging process described in the second embodiment above is merely an example, and for example, the technical ideas included in the distance measurement imaging process described in the second embodiment above may be applied to the distance measurement imaging process shown in Figures 18A to 18C, the distance measurement imaging process shown in Figure 19, the distance measurement imaging process shown in Figure 20, the distance measurement imaging process shown in Figure 21, or the distance measurement imaging process shown in Figure 23. In this case, within the scope that does not deviate from the main point, the first distance may be reinterpreted as the first focus position, the second distance as the second focus position, unnecessary steps may be deleted, new steps may be added, or the processing order may be changed.

[0224] Furthermore, while the first embodiment described above included an example in which the distance measurement and imaging processing program 70 is stored in storage 15B, and the second embodiment described above included an example in which the distance measurement and imaging processing program 570 is stored in storage 15B, the technology of this disclosure is not limited thereto. For example, as shown in Figure 30, the distance measurement and imaging processing program 70 or 570 (hereinafter, when it is not necessary to distinguish between the distance measurement and imaging processing programs 70 and 570, they will be referred to as "distance measurement and imaging processing program" without reference numerals) may be stored in a storage medium 900. An example of the storage medium 900 is any portable storage medium such as an SSD or a USB memory.

[0225] The distance measurement and imaging processing program stored in the storage medium 900 is installed in the controller 15. The CPU 15A executes the distance measurement and imaging processing according to the distance measurement and imaging processing program.

[0226] Alternatively, the ranging image processing program may be stored in the memory of another computer or server device connected to the controller 15 via a communication network (not shown), and the ranging image processing program may be downloaded and installed on the controller 15 in response to a request from the smart device 10.

[0227] Furthermore, it is not necessary to store the entire distance measurement and imaging processing program in the memory unit of another computer or server device connected to the controller 15, or in storage 15B; it is acceptable to store only a portion of the distance measurement and imaging processing program.

[0228] In the example shown in Figure 30, the controller 15 is built into the smart device 10, but the technology of this disclosure is not limited to this, and for example, the controller 15 may be provided outside the smart device 10.

[0229] In the example shown in Figure 30, CPU15A is a single CPU, but it may be multiple CPUs. Alternatively, a GPU may be used instead of CPU15A.

[0230] In the example shown in Figure 30, a controller 15 is illustrated, but the technology of this disclosure is not limited thereto, and devices including ASICs, FPGAs, and / or PLDs may be used instead of the controller 15. Alternatively, a combination of hardware and software configurations may be used instead of the controller 15.

[0231] The hardware resources used to perform the distance measurement and imaging processing described in each of the above embodiments include the following types of processors. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for performing distance measurement and imaging processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, which are processors with circuit configurations specifically designed to perform particular processing, such as FPGAs, PLDs, or ASICs. Each processor has built-in or connected memory, and each processor uses this memory to perform distance measurement and imaging processing.

[0232] The hardware resources that perform the distance measurement and imaging process may consist of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources that perform the distance measurement and imaging process may consist of a single processor.

[0233] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs distance measurement and imaging processing. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform distance measurement and imaging processing, on a single IC chip, as exemplified by SoCs. Thus, distance measurement and imaging processing is realized using one or more of the above types of processors as hardware resources.

[0234] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits combining circuit elements such as semiconductor devices. Also, the distance measurement and imaging process described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0235] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0236] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0237] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0238] The following additional information is disclosed regarding the embodiments described above.

[0239] (Note 1) Processor and Includes memory connected to or built into the above processor, The above processor is The distance to the imaging area is measured based on the irradiation timing when the light irradiator directs light toward the imaging area and the reception timing when the light receiver receives the reflected light from the imaging area. A first focus position that focuses on the subject included in the above imaging area is derived based on the above distance. The second focus position that focuses on the above subject is derived based on the image obtained when the above imaging area is imaged by the imaging device. If the first focus position and the second focus position are different, a specific process is executed. Information processing device.

[0240] (Note 2) The information processing apparatus described in Appendix 1 includes a process for notifying that the first focus position and the second focus position are different.

[0241] (Note 3) The information processing apparatus according to Appendix 1 or Appendix 2, wherein the specific processing described above includes a process of causing the imaging device to image the imaging area at the first focus position and the focus position, respectively.

[0242] (Note 4) The information processing apparatus according to Appendix 3, wherein the specific processing described above includes displaying a first focused image obtained by imaging the imaging area at the first focus position and a second focused image obtained by imaging the imaging area at the second focus position on a display, and prompting the user to select at least one of the first focused image and the second combined image while the first focused image and the second combined image are displayed on the display.

[0243] (Note 5) The information processing apparatus described in Appendix 2 or Appendix 3 includes, the above-mentioned specific processing, a process of causing the imaging device to image the imaging area at the first focus position, or a process of causing the imaging device to image the imaging area at the second focus position.

[0244] (Note 6) The above imaging device is capable of capturing images at both the wide-angle and telephoto ends. When the above-mentioned imaging device takes an image at the wide-angle side, the above-mentioned specific processing is the process of causing the imaging device to take an image at the first focus position, as described in Appendix 5 of the information processing device.

[0245] (Note 7) The information processing apparatus described in Appendix 5, wherein, if the amount of reflected light received by the above-mentioned light receiver is greater than or equal to a threshold, the above-mentioned specific process is a process of causing the imaging device to take an image at the second focus position. [Explanation of symbols]

[0246] 10,300,500 smart devices 12 cabinets 12A back 12B Front 13 Instruction Keys 14. Rangefinder and imaging device 15 Controllers 15A CPU 15A1 First Distance Measurement Control Unit 15A2 Second Distance Measurement Control Unit 15A3 Acquisition Department 15A4 Judgment section 15A5 Execution Department 15A6 1st focus position calculation section 15A7 Contrast AF System Imaging Control Unit 15A8 Second focus position calculation section 15B Storage 15C memory 16 Light irradiator 17A, 17B Light-shielding members 18 Receiver 19 Microlenses 20,22,352 Translucent windows 21 Beam Expander 23 Collimating Lens 24 LD 25 LD Driver 26,354 Photoelectric Conversion Element 26N1 Visible Light Image Division Region 26N2 First Distance Measurement System Division Region 26N3 Second Distance Measurement System Division Region 27 TOF Camera 30A Objective Lens 30B Focusing Lens 30C Aperture 31 Focus Control Mechanism 32 Photoelectric Conversion Element Driver 34 Signal Processing Circuit 34A Visible Light Pixel Data Processing Circuit 34B First Distance Measurement System Processing Circuit 34C Second Distance Measurement System Processing Circuit 40 Input / Output Interface 41 Imaging Lens 42 Image Memory 44 UI System Device 46 Display 47 Reception Device 48 Touch Panel 50 Bus 52 External I / F 53 Hard Key Section 54 Communication I / F 56 Network 59 Touch Panel - Display 60 Moving Mechanism 62 Motor 64 Motor Driver 70,570 Distance Measurement Imaging Processing Program 72 Focus Position Derivation Table 100 Mirror 100A Mirror Surface 300L Left Region Passing Light 300R Right Region Passing Light 350 Wide - Angle Light Receiver 900 Storage Medium L First Phase Difference Pixel L1,L2 optical axis N Non-phase-detection pixels PD photodiode R is the second phase difference pixel. α shift amount θ1 field of view θ2 Illumination field of view

Claims

1. Processor and Includes memory connected to or built into the processor, The aforementioned processor, A first distance measurement is performed to measure the distance to the imaging area based on the irradiation timing when the light irradiator irradiates light toward the imaging area and the reception timing when the light receiver receives the reflected light from the imaging area. Based on the first image obtained by imaging the imaging area using the imaging device, a second distance measurement is performed to measure the distance to the imaging area. If the first distance measurement result obtained by performing the first distance measurement is different from the second distance measurement result obtained by performing the second distance measurement, a specific process is executed. The aforementioned specific process includes a process that notifies the user that the first distance measurement result and the second distance measurement result are different before adjusting the focus. Information processing device.

2. Processor and Includes memory connected to or built into the processor, The aforementioned processor, A first distance measurement is performed to measure the distance to the imaging area based on the irradiation timing when the light irradiator irradiates light toward the imaging area and the reception timing when the light receiver receives the reflected light from the imaging area. Based on the first image obtained by imaging the imaging area using the imaging device, a second distance measurement is performed to measure the distance to the imaging area. If the first distance measurement result obtained by performing the first distance measurement is different from the second distance measurement result obtained by performing the second distance measurement, a specific process is executed. The aforementioned specific process includes a process of controlling the focus of the imaging device based on the first distance measurement result and the second distance measurement result, and causing the imaging area to be imaged. Information processing device.

3. The aforementioned specific processing includes displaying a first focused image obtained by imaging the imaging area under focus control based on the first distance measurement result, and a second focused image obtained by imaging the imaging area under focus control based on the second distance measurement result, on a display, and prompting the user to select at least one of the first focused image and the second focused image while both are displayed on the display. The information processing apparatus according to claim 2.

4. The aforementioned specific processing includes a process of controlling the focus of the imaging device based on the first distance measurement result or the second distance measurement result, and causing the imaging area to be imaged. The information processing apparatus according to claim 1.

5. The focus control is performed on the imaging device based on either of the first and second distance measurement results, as determined according to the given instructions. The information processing apparatus according to claim 4.

6. The aforementioned imaging device is capable of capturing images at both the wide-angle and telephoto ends. When the imaging device takes images at the wide-angle end, the focus control is performed on the imaging device based on the first distance measurement result. The information processing apparatus according to claim 4.

7. If the amount of reflected light received by the light receiver is equal to or greater than a threshold, the focus control is performed based on the second distance measurement result. The information processing apparatus according to claim 4.

8. The focus control is performed on the imaging device based on either the first distance measurement result or the second distance measurement result, which is determined according to the ratio of high-reflectance regions, which have a reflectance equal to or greater than a predetermined reflectance, to the second image obtained when the imaging area is imaged by the imaging device, or the difference between the area of ​​the high-reflectance regions in the second image and the area of ​​regions in the second image that are not high-reflectance regions. The information processing apparatus according to claim 4.

9. The processor detects a specific subject image representing a specific subject in the second image obtained by the imaging device capturing the imaging area. When the specific subject image is detected by the processor, the focus control is performed based on the second distance measurement result. The information processing apparatus according to claim 4.

10. The aforementioned specific subject includes a high-reflectance region having a reflectance greater than or equal to a predetermined reflectance, The aforementioned high-reflectivity region is the region in which the image of a person and at least one of the imaging device are reflected. The information processing apparatus according to claim 9.

11. Processor and Includes memory connected to or built into the processor, The aforementioned processor, The distance to the imaging area is measured based on the irradiation timing when the light irradiator directs light toward the imaging area and the light reception timing when the light receiver receives the reflected light from the imaging area. A first focusing position that focuses on the subject included in the imaging area is derived based on the distance. A second focus position that focuses on the subject is derived based on the image obtained when the imaging area is imaged by the imaging device. If the first focus position and the second focus position are different, a specific process is executed. The aforementioned specific process includes a process that notifies that the first focus position and the second focus position are different before adjusting the focus. Information processing device.

12. A first distance measurement is performed to measure the distance to the imaging area based on the irradiation timing when the light irradiator irradiates light toward the imaging area and the light reception timing when the light receiver receives the reflected light from the imaging area. Based on the first image obtained by imaging the imaging area using the imaging device, a second distance measurement is performed to measure the distance to the imaging area, and The system includes performing a specific process when the first distance measurement result obtained by performing the first distance measurement and the second distance measurement result obtained by performing the second distance measurement are different. The aforementioned specific process includes a process that notifies the user that the first distance measurement result and the second distance measurement result are different before adjusting the focus. Information processing methods.

13. The distance to the imaging area is measured based on the irradiation timing when the light irradiator directs light toward the imaging area and the reception timing when the light receiver receives the reflected light from the imaging area. A first focus position that focuses on the subject included in the imaging area is derived based on the distance. The second focus position that focuses on the subject is derived based on the image obtained when the imaging area is imaged by the imaging device, and This includes performing a specific process when the first focus position and the second focus position are different, The aforementioned specific process includes a process that notifies that the first focus position and the second focus position are different before adjusting the focus. Information processing methods.

14. On the computer, A first distance measurement is performed to measure the distance to the imaging area based on the irradiation timing when the light irradiator irradiates light toward the imaging area and the light reception timing when the light receiver receives the reflected light from the imaging area. Based on the first image obtained by imaging the imaging area using the imaging device, a second distance measurement is performed to measure the distance to the imaging area, and A program for causing a system to execute a specific process that includes, if the first distance measurement result obtained by performing the first distance measurement and the second distance measurement result obtained by performing the second distance measurement are different, notifying the system that the first distance measurement result and the second distance measurement result are different before adjusting the focus.

15. On the computer, The distance to the imaging area is measured based on the irradiation timing when the light irradiator directs light toward the imaging area and the reception timing when the light receiver receives the reflected light from the imaging area. A first focus position that focuses on the subject included in the imaging area is derived based on the distance. The second focus position that focuses on the subject is derived based on the image obtained when the imaging area is imaged by the imaging device, and A program for causing a system to execute a specific process that includes, when the first focus position and the second focus position are different, notifying the system that the first focus position and the second focus position are different before adjusting the focus.

Images