Information processing apparatus, imaging apparatus, information processing method, and storage medium

Abstract

The invention provides an information processing apparatus, an imaging apparatus, an information processing method, and a storage medium capable of avoiding erroneous distance measurement caused by reflected light from a glossy object that is not desired to be a distance measurement object. The information processing apparatus includes: a processor; and a memory connected to or built in the processor, the processor performing a process for measuring a first distance to an object in the distance measurement target region on the basis of an irradiation timing at which the light irradiation is performed by the light irradiator toward the distance measurement target region and a light reception timing at which reflected light from the distance measurement target region receives the light by the light receiver; when specific pixels included in the light receiver generate a plurality of signals at different timings during a light receiving period corresponding to surface irradiation by the light irradiator, the processor measures the first distance on the basis of the relationship between the plurality of signals.

Description

[0001] This invention is a divisional application of the following application, and the information of the original application is as follows:

[0002] Filing date: November 10, 2020

[0003] Application number: 202080087482.2

[0004] Title of the invention: Information processing device, imaging device, information processing method, and program Technical field

[0005] The technology of the present invention relates to an information processing device, an imaging device, an information processing method, and a storage medium. Background art

[0006] A laser distance measuring device is disclosed in Patent Document 1, which includes a laser emitting device and an optical system that emits a laser beam emitted from the laser emitting device to a measurement object and condenses the reflected light from the measurement object. The laser distance measuring device described in Patent Document 1 includes a field-of-view limiting mechanism capable of arbitrarily changing the transmission position and size of the field of view received by the optical system within the field of view, and measures the distance to a measurement target within the measurement object passing through the position of the field-of-view limiting mechanism within the received field of view.

[0007] An object detection device is disclosed in Patent Document 2, which emits electromagnetic waves in the traveling direction of a vehicle and detects an object based on the reflected wave of the electromagnetic wave. The object detection device described in Patent Document 2 includes: an emission mechanism that emits electromagnetic waves while scanning within a scanning range along a horizontal direction and a vertical direction; a reception mechanism that receives the reflected wave of the electromagnetic wave; an acquisition mechanism that acquires the level of the reflected wave received by the reception mechanism; an arithmetic mechanism that calculates a first deviation correction amount based on the position in the vertical direction of the region where the level of the reflected wave becomes maximum and the position of the center in the vertical direction of the scanning range; and a correction mechanism that corrects the center position in the vertical direction of the scanning range based on the first deviation correction amount calculated by the arithmetic mechanism, and measures the distance to the object through the reflected wave. Further, the emission mechanism of the object detection device described in Patent Document 2 performs a first scanning process and a second scanning process. In the first scanning process, in a central region centered on the center position in the vertical direction of the scanning range, scanning is performed along a first direction that is a specified direction in the horizontal direction, and in a region above the central region, scanning is performed while emitting along a second direction that is a direction opposite to the first direction. In the second scanning process, scanning is performed along the first direction in the central region, and in a region below the central region, scanning is performed while emitting along the second direction.

[0008] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2003-057343

[0009] Patent Document 2: Japanese Patent Application Publication No. 2006-349694 Summary of the invention

[0010] One embodiment of the technology according to the present invention provides an information processing device, an imaging device, an information processing method, and a storage medium capable of avoiding erroneous distance measurement caused by reflected light from a glossy object that is not intended to be a distance measurement target.

[0011] Means for solving technical problems

[0012] The first method involved in the technology of the present invention is an information processing device, which includes: a processor; and a memory, which is connected to or built in the processor, and the processor performs the following processing: based on the irradiation timing of a light irradiator performing surface irradiation of light toward the distance measurement target area and the light reception timing of reflected light from the distance measurement target area received by the light receiver, a first distance is measured to an object in the distance measurement target area, and the processor measures the first distance based on the light reception timing and irradiation timing corresponding to a signal selected according to the relationship of the multiple signals and the intensities of the multiple signals from a plurality of signals generated by the light receiver during a light reception period corresponding to the surface irradiation based on the light irradiator.

[0013] A second aspect of the present invention is the information processing device according to the first aspect, wherein the light reception timing used in measuring the first distance is a timing at which the light receiver receives reflected light associated with a signal having the second highest intensity among the plurality of signals.

[0014] The third mode involved in the technology of the present invention is an information processing device involved in the first mode or the second mode, wherein, when the intensities of multiple signals include a first intensity and a second intensity, the processor performs specific processing, the first intensity is greater than a first threshold, and the second intensity is less than the first threshold and greater than a second threshold that is smaller than the first threshold.

[0015] A fourth aspect according to the technology of the present invention is the information processing device according to the third aspect, wherein the first threshold value and the second threshold value are defined based on a noise component based on ambient light.

[0016] A fifth aspect according to the technology of the present invention is the information processing device according to the third aspect or the fourth aspect, wherein the first threshold value is a value that decreases as the light reception timing is delayed.

[0017] A sixth aspect according to the technology of the present invention is the information processing device according to any one of the third aspect to the fifth aspect, wherein the specific processing includes processing of notifying that the strengths of the plurality of signals include the first strength.

[0018] The seventh mode related to the technology of the present invention is the information processing apparatus according to any one of the third to sixth modes, wherein the specific processing includes image-usage ranging for measuring the second distance to an object based on an image obtained by photographing a ranging target area.

[0019] The eighth mode related to the technology of the present invention is the information processing apparatus according to any one of the third to sixth modes, wherein the processor performs image-usage ranging for measuring the second distance to an object based on an image obtained by photographing a ranging target area in parallel with the operation of measuring the first distance according to the irradiation timing and the light reception timing, and the specific processing includes processing based on the ranging result of the image-usage ranging.

[0020] The ninth mode related to the technology of the present invention is the information processing apparatus according to the seventh or eighth mode, wherein the image-usage ranging is at least one of phase-difference image-usage ranging for measuring the second distance based on a phase-difference image obtained from phase-difference pixels as an image, stereo-image-usage ranging for measuring the second distance based on a stereo image obtained by a stereo imaging method as an image, and object-image-usage ranging for measuring the second distance based on an object image detected from the image and representing an object with a known size.

[0021] The tenth mode related to the technology of the present invention is the information processing apparatus according to any one of the third to ninth modes, wherein the specific processing includes focusing control of the imager based on the contrast of a subject image obtained by photographing a subject included in the ranging target area by the imager.

[0022] The eleventh mode related to the technology of the present invention is the information processing apparatus according to any one of the first to tenth modes, wherein the ranging target area is a specific real-space area defined according to a given instruction.

[0023] The twelfth mode related to the technology of the present invention is the information processing apparatus according to any one of the first to eleventh modes, wherein the ranging target area is a specific real-space area corresponding to an object subject image detected by the processor from a captured image obtained by photographing.

[0024] The thirteenth mode related to the technology of the present invention is the information processing apparatus according to any one of the first to twelfth modes, wherein the processor performs focusing control of the imager by using the first distance measured according to the irradiation timing and the light reception timing.

[0025] The fourteenth mode related to the technology of the present invention is the information processing apparatus according to any one of the first to thirteenth modes, wherein the intensity of a plurality of signals is adjusted according to the light reception timing.

[0026] The 15th aspect related to the technology of the present invention is the information processing apparatus according to any one of the 1st to 14th aspects, wherein the light receiver has a plurality of photoelectric conversion elements arranged in a two-dimensional shape, and the processor measures a first distance to a plurality of photoelectric conversion elements based on the irradiation timing and the light reception timing.

[0027] The 16th aspect related to the technology of the present invention is an imaging device, which includes the information processing apparatus according to any one of the 1st to 15th aspects and a focusing lens, and the processor performs focusing control to move the focusing lens to a focusing position determined based on the first distance.

[0028] The 17th aspect related to the technology of the present invention is an information processing method, which includes the following steps: performing surface irradiation of light on a ranging target area by a light irradiator; receiving reflected light from the ranging target area by a light receiver; and measuring a first distance to an object in the ranging target area based on the irradiation timing of the surface irradiation of light on the ranging target area by the light irradiator and the light reception timing of receiving the reflected light by the light receiver, and measuring the first distance by the processor based on the light reception timing and the irradiation timing corresponding to a signal selected according to the relationship and intensity of a plurality of signals generated by the light receiver at a plurality of light reception timings during the light reception period corresponding to the surface irradiation by the light irradiator.

[0029] The 18th aspect related to the technology of the present invention is a program for causing a computer to execute a process including the following steps: performing surface irradiation of light on a ranging target area by a light irradiator; receiving reflected light from the ranging target area by a light receiver; and measuring a first distance to an object in the ranging target area based on the irradiation timing of the surface irradiation of light on the ranging target area by the light irradiator and the light reception timing of receiving the reflected light by the light receiver, and measuring the first distance by the processor based on the light reception timing and the irradiation timing corresponding to a signal selected according to the relationship and intensity of a plurality of signals generated by the light receiver at a plurality of light reception timings during the light reception period corresponding to the surface irradiation by the light irradiator. Description of the Drawings

[0030] Figure 1 It is a schematic perspective view showing an example of the usage mode of the smart device according to the first embodiment.

[0031] Figure 2 It is a rear perspective view showing an example of the appearance of the back side of the smart device according to the first embodiment.

[0032] Figure 3It is a conceptual diagram showing an example of a method of capturing an imaging area by a distance measurement imaging device included in the smart device according to the first embodiment and displaying a visible light image on a display.

[0033] Figure 4 It is a conceptual diagram showing an example of a method of irradiating a laser beam to an imaging area by a distance measurement imaging device included in the smart device according to the embodiment.

[0034] Figure 5 It is a schematic perspective view showing an example of the arrangement of each pixel included in the photoelectric conversion element of the smart device according to the first embodiment.

[0035] Figure 6 It shows the subject light with respect to Figure 5 It is a conceptual diagram showing an example of the incident characteristics of the first phase difference pixel and the second phase difference pixel included in the photoelectric conversion element shown.

[0036] Figure 7 It shows Figure 5 It is a schematic structural diagram showing an example of the structure of the non-phase difference pixel included in the photoelectric conversion element shown.

[0037] Figure 8 It is a conceptual diagram showing an example of a method of displaying a distance image on a display based on the distance measurement result obtained by performing the first distance measurement on the imaging area by the distance measurement imaging device included in the smart device according to the first embodiment.

[0038] Figure 9 It is a block diagram showing an example of the structure of the electrical system hardware of the smart device according to the first embodiment.

[0039] Figure 10 It is a block diagram showing an example of the function of the CPU included in the smart device according to the first embodiment.

[0040] Figure 11 It shows Figure 10 It is a conceptual diagram showing an example of the processing content of the first distance measurement control unit shown.

[0041] Figure 12 It is a conceptual diagram showing an example of a scene of performing the first distance measurement by the smart device according to the first embodiment.

[0042] Figure 13 It is a conceptual diagram showing an example of the processing content of the first distance measurement system processing circuit included in the smart device according to the first embodiment.

[0043] Figure 14It is a conceptual diagram showing an example of the time series distribution generated by the time series distribution generation unit of the first distance measurement system processing circuit included in the smart device according to the first embodiment.

[0044] Figure 15 It shows Figure 10 A block diagram showing an example of the processing contents of the first distance acquisition unit, time series distribution acquisition unit, determination unit, and execution unit shown.

[0045] Figure 16 It is used to explain based on Figure 15 An explanatory diagram showing the determination method of the determination unit shown.

[0046] Figure 17 It shows Figure 15 A block diagram showing an example of the content of the first imaging process executed by the execution unit shown.

[0047] Figure 18 It shows Figure 15 A block diagram showing an example of the content of the specific process executed by the execution unit shown.

[0048] Figure 19 It shows Figure 15 A block diagram showing an example of the content of the second distance measurement start process and the second distance acquisition process executed by the execution unit shown.

[0049] Figure 20 It shows Figure 15 A conceptual diagram showing an example of the detailed content of the specific process executed by the execution unit shown.

[0050] Figure 21 It is a conceptual diagram showing an example of the processing content when the user selects the first visible light image in the image selection screen displayed on the display of the smart device according to the first embodiment via the touch panel.

[0051] Figure 22 It is a conceptual diagram showing an example of the processing content when the user selects the second visible light image in the image selection screen displayed on the display of the smart device according to the first embodiment via the touch panel.

[0052] Figure 23 It is a flowchart showing an example of the process of the first distance measurement system processing according to the first embodiment.

[0053] Figure 24A It is a flowchart showing an example of the process of the distance measurement imaging process according to the first embodiment.

[0054] Figure 24B It is Figure 24A A continuation of the flowchart shown.

[0055] Figure 24C is Figure 24A a continuation of the flowchart shown below.

[0056] Figure 25 is a block diagram showing an example of the functions of the CPU included in the intelligent device according to the second embodiment.

[0057] Figure 26 is a conceptual diagram showing an example of the processing content related to the imaging by the contrast AF method performed by the intelligent device according to the second embodiment.

[0058] Figure 27 is a conceptual diagram showing an example of the details of the specific processing executed by the execution unit according to the second embodiment.

[0059] Figure 28 is related to Figure 14 is a conceptual diagram showing an example of the time series distribution of a mode different from the mode shown below.

[0060] Figure 29 is a conceptual diagram showing an example of the method of applying a filter to the time series distribution.

[0061] Figure 30 is a conceptual diagram showing an example of the time series distribution adjusted by applying a filter.

[0062] Figure 31 is a conceptual diagram showing an example of the method of installing the ranging imaging processing program according to the first or second embodiment in the intelligent device. Specific Embodiments

[0063] Hereinafter, an example of an embodiment of the imaging device related to the technology of the present invention will be described with reference to the accompanying drawings.

[0064] First, the terms used in the following description will be described.

[0065] CPU stands for "Central Processing Unit". RAM stands for "Random Access Memory". ASIC stands for "Application Specific Integrated Circuit". PLD stands for "Programmable Logic Device". FPGA stands for "Field-Programmable Gate Array". SoC stands for "System-on-a-chip". SSD stands for "Solid State Drive". USB stands for "Universal Serial Bus". HDD stands for "Hard Disk Drive". EEPROM stands for "Electrically Erasable and Programmable Read Only Memory". EL stands for "Electro-Luminescence". A / D stands for "Analog / Digital". I / F stands for "Interface". UI stands for "User Interface". LTE stands for "Long Term Evolution". 5G stands for "5th Generation". LD stands for "Laser Diode". IR stands for "Infrared". APD stands for "Avalanche Photodiode". TOF stands for "Time of Flight".

[0066] fps stands for "frame per second". LED stands for "Light Emitting Diode". ROI stands for "Region of Interest". LAN stands for "Local Area Network". AF stands for "Auto Focus". IC stands for "Integrated Circuit".

[0067] In the description of this specification, "horizontal" in addition to being completely horizontal also means horizontal in the sense of including the errors that are generally tolerated in the technical field to which the technology of the present invention belongs. In the description of this specification, "parallel" in addition to being completely parallel also means parallel in the sense of including the errors that are generally tolerated in the technical field to which the technology of the present invention belongs. In the description of this specification, "vertical" in addition to being completely vertical also means vertical in the sense of including the errors that are generally tolerated in the technical field to which the technology of the present invention belongs. In the description of this specification, "identical" in addition to being completely identical also means identical in the sense of including the errors that are generally tolerated in the technical field to which the technology of the present invention belongs.

[0068] [First Embodiment]

[0069] As an example, as Figure 1 shown, the intelligent device 10 performs a shooting operation (hereinafter, also simply referred to as "shooting operation") and a distance measurement operation of a shooting area defined by a field of view angle θ1. In this first embodiment, "distance measurement" refers to the process of measuring the distance from the intelligent device 10 to the shooting area. Here, the shooting area is an example of the "distance measurement target area" related to the technology of the present invention. And the intelligent device 10 is an example of the "information processing device" and "shooting device" related to the technology of the present invention. As the intelligent device 10, for example, a smart phone or a tablet terminal, etc., which are electronic devices with a shooting function, can be cited.

[0070] In the intelligent device 10, a first distance measurement and a second distance measurement with different distance measurement methods are performed. The first distance measurement is a method of measuring the distance to an object in the shooting area based on the timing when the intelligent device 10 irradiates a laser beam toward the shooting area and the timing when the intelligent device 10 receives the reflected light of the laser beam from the shooting area (hereinafter, also referred to as the "active method"). Here, the laser beam is an example of the "light" related to the technology of the present invention. And here, surface irradiation of the laser beam is performed toward the shooting area. In addition, surface irradiation means that the beam diameter of the laser beam toward the shooting area is more expanded than point irradiation. In surface irradiation, the beam diameter of the laser beam gradually expands along the irradiation direction of the laser beam, and the degree of expansion of the beam diameter of surface irradiation per unit time is larger than that of point irradiation. That is, in point irradiation, the laser beam is irradiated onto the surface where the shooting area is located in a point shape, while in surface irradiation, the laser beam is irradiated onto the surface where the shooting area is located in a surface shape. And the irradiation of the laser beam can be a single shot or can be intermittently performed periodically (for example, every 0.1 second). At this time, the first distance measurement can be performed in each irradiation of the laser beam, and processing based on the distance measurement result can be performed.

[0071] The second distance measurement is a distance measurement method (hereinafter, also simply referred to as the "passive method") for measuring the distance to an object within the imaging area based on an image obtained by the smart device 10 capturing the imaging area. In addition, the second distance measurement is an example of the "image-using distance measurement" related to the technology of the present invention.

[0072] In the smart device 10, a hybrid distance measurement method that combines the active distance measurement method and the passive distance measurement method is performed. And in the smart device 10, imaging accompanied by active focusing control and imaging accompanied by passive focusing control are performed. Active focusing control refers to focusing control based on the distance measurement result obtained by performing the active distance measurement method. Passive focusing control refers to focusing control based on the distance measurement result obtained by performing the passive distance measurement method.

[0073] As an example, as Figure 2 shown, the smart device 10 includes a housing 12. A distance measurement imaging device 14 is accommodated in the housing 12. The distance measurement imaging device 14 includes a light irradiator 16 and a light receiver 18. The light irradiator 16 includes an LD 24. In the smart device 10, the imaging operation and the distance measurement operation are performed by the distance measurement imaging device 14.

[0074] An instruction key 13 is arranged on the side surface of the smart device 10. The instruction key 13 receives various instructions. The "various instructions" mentioned here, for example, refer to an instruction to display a menu screen capable of selecting various menus, an instruction to select one or more menus, an instruction to confirm the selection content, and an instruction to delete the selection content, etc.

[0075] When the smart device 10 is in the vertical state, a light transmission window 20 and 22 are provided at the upper left part of the back surface 12A of the housing 12 (the upper left part when observing the rear view of the vertically placed smart device 10). The light transmission windows 20 and 22 are optical elements with light transmissibility (for example, lenses), arranged at a specified interval (for example, an interval of several millimeters) along the horizontal direction and exposed from the back surface 12A. The light irradiator 16 irradiates the imaging area with the laser beam emitted from the LD 24 through the light transmission window 20. In this first embodiment, the laser beam in the infrared wavelength region is irradiated onto the imaging area by the light irradiator 16. However, the wavelength region of the laser beam is not limited to this, and it can also be a laser beam in other wavelength regions.

[0076] The light receiver 18 receives the IR reflected light through the light transmission window 22. The IR reflected light refers to the reflected light of the laser beam irradiated on the distance measurement object by the light irradiator 16 from the distance measurement object. And the light receiver 18 takes in the visible light reflected light through the light transmission window 22. The visible light reflected light refers to the reflected light of the visible light (for example, the visible light contained in sunlight) irradiated on the imaging area from the imaging area. In addition, hereinafter, for the convenience of description, when it is not necessary to distinguish between the IR reflected light and the visible light reflected light for explanation, it is simply referred to as "reflected light".

[0077] The light receiver 18 is an example of the "imaging device" related to the technology of the present invention, and includes a photoelectric conversion element 26. The photoelectric conversion element 26 receives the reflected light taken into the light receiver 18 through the light transmission window 22, and outputs an electric signal corresponding to the amount of the received reflected light.

[0078] As an example, as Figure 3 shown, a touch panel display 59 is provided on the front surface 12B of the housing 12. The touch panel display 59 includes a display 46 and a touch panel 48. As an example of the display 46, an organic EL display can be cited. The display 46 may not be an organic EL display, but other types of displays such as a liquid crystal display.

[0079] The display 46 displays images (for example, live view images and playback images) and character information, etc. The touch panel 48 is a transmissive touch panel and overlaps the surface of the display area of the display 46. The touch panel 48 receives an instruction from the user by detecting contact based on an indicator such as a finger or a stylus. In addition, here, as an example of the touch panel display 59, an external touch panel display in which the touch panel 48 overlaps the surface of the display area of the display 46 is cited, but this is only an example. For example, as the touch panel display 59, an inlaid type or an embedded type touch panel display can also be applied.

[0080] In the intelligent device 10, if an instruction to start imaging is received through the touch panel 48, the imaging area is imaged by the light receiver 18. That is, the light receiver 18 receives visible light reflected light, and generates a visible light image representing the imaging area as an image corresponding to the received visible light reflected light. The visible light image is an example of the "image", "subject image" and "imaging image" related to the technology of the present invention.

[0081] The visible light image is displayed on the display 46 as a live view image or a still image according to the instruction received through the touch panel 48. In the Figure 3 example shown, the imaging area is defined by the field of view angle θ1. The field of view angle θ1 is changed according to the instruction received through the touch panel 48.

[0082] As an example, as Figure 4 shown, in the intelligent device 10, if an instruction to start distance measurement and imaging (hereinafter, also referred to as "distance measurement imaging start instruction") is received through the touch panel 48, a laser beam is irradiated by the light irradiator 16. The angle of the irradiated laser beam (hereinafter, also referred to as "irradiation angle") is θ2, and the irradiation angle θ2 is changed according to the instruction received through the touch panel 48. In addition, in the Figure 4In the example shown, a method example of starting distance measurement is described based on a distance measurement imaging start instruction received via the touch panel 48 in a state where a visible light image is displayed as a live view image on the display 46. However, the technology of the present invention is not limited thereto. For example, distance measurement may also be started when a distance measurement imaging start instruction is received via the touch panel 48 in a state where the visible light image is not displayed on the display 46.

[0083] In the intelligent device 10, the distance from the intelligent device 10 to the distance measurement object is measured based on the time required from when the laser beam is irradiated by the light irradiator 16 until the IR reflected light is received by the light receiver 18 and the speed of light. For example, if the distance to the distance measurement object is set as "L0", the speed of light is set as "c", and the flight time of the laser beam, that is, the time required from when the laser beam is irradiated by the light irradiator 16 until the IR reflected light is received by the light receiver 18 (hereinafter, also simply referred to as "flight time") is set as "t", then the distance L0 is calculated according to the formula "L0 = c × t × 0.5".

[0084] As an example, as Figure 5 shown, the photoelectric conversion element 26 has a plurality of photodiodes arranged in a matrix. As an example of the plurality of photodiodes, photodiodes with a pixel amount of "4896 × 3265" can be cited.

[0085] Color filters are arranged for each photodiode included in the photoelectric conversion element 26. The color filter includes a G filter corresponding to the G (green) wavelength region that is most helpful for obtaining a luminance signal, an R filter corresponding to the R (red) wavelength region, a B filter corresponding to the B (blue) wavelength region, and an IR filter corresponding to the IR (infrared) wavelength region. In addition, in the present embodiment, the G filter, the R filter, and the B filter also have the function of an infrared light cut-off filter for cutting off infrared light.

[0086] The photoelectric conversion element 26 is formed of two types of photosensitive pixels, namely, phase difference pixels and non-phase difference pixels N that are different from the phase difference pixels. Usually, the non-phase difference pixels N are also referred to as normal pixels. The photoelectric conversion element 26 has four types of photosensitive pixels, namely, R pixels, G pixels, B pixels, and IR pixels, as non-phase difference pixels. The R pixels, G pixels, B pixels, IR pixels, and phase difference pixels are regularly arranged in a prescribed periodicity in the row direction (horizontal direction) and the column direction (vertical direction). The R pixels are pixels corresponding to the photodiodes arranged with R filters, the G pixels and the phase difference pixels are pixels corresponding to the photodiodes arranged with G filters, the B pixels are pixels corresponding to the photodiodes arranged with B filters, and the IR pixels are pixels corresponding to the photodiodes arranged with IR filters. As an example of the IR pixels, InGaAs APD can be cited.

[0087] In addition, hereinafter, for the sake of convenience of explanation, when it is not necessary to distinguish between the G filter, the R filter, and the B filter, they are also referred to as "visible light filters". Further, hereinafter, for the sake of convenience of explanation, when it is not necessary to distinguish between R pixels, G pixels, and B pixels, they are referred to as "visible light pixels".

[0088] A plurality of phase difference pixel lines 26A and a plurality of non-phase difference pixel lines 26B are arranged on the light-receiving surface of the photoelectric conversion element 26. The phase difference pixel line 26A is a horizontal line including phase difference pixels. Specifically, the phase difference pixel line 26A is a horizontal line in which phase difference pixels and non-phase difference pixels N are mixed. The non-phase difference pixel line 26B is a horizontal line including only a plurality of non-phase difference pixels N.

[0089] On the light-receiving surface of the photoelectric conversion element 26, the phase difference pixel line 26A and a specified number of non-phase difference pixel lines 26B are alternately arranged in the column direction. The "specified number of lines" mentioned here is, for example, 2 rows. In addition, here, 2 rows are illustrated as the specified number of lines, but the technology of the present invention is not limited to this, and the specified number of lines may be several rows of 3 rows or more, or a dozen rows, dozens of rows, or hundreds of rows, etc.

[0090] The phase difference pixel lines 26A are arranged at 2-row intervals in the column direction from the first row to the last row. A part of the pixels of the phase difference pixel line 26A are phase difference pixels. Specifically, the phase difference pixel line 26A is a horizontal line in which phase difference pixels and non-phase difference pixels N are periodically arranged. The phase difference pixels are roughly divided into a first phase difference pixel L and a second phase difference pixel R. In the phase difference pixel line 26A, the first phase difference pixel L and the second phase difference pixel R are alternately arranged as G pixels at a certain pixel interval in the row direction.

[0091] The first phase difference pixel L and the second phase difference pixel R are arranged to appear alternately in the column direction. In Figure 5 In the example shown, in the 4th column, the first phase difference pixel L, the second phase difference pixel R, the first phase difference pixel L, and the second phase difference pixel R are arranged in sequence from the first row in the column direction. That is, the first phase difference pixel L and the second phase difference pixel R are alternately arranged from the first row in the column direction. And, in Figure 5 In the example shown, in the 10th column, the second phase difference pixel R, the first phase difference pixel L, the second phase difference pixel R, and the first phase difference pixel L are arranged in sequence from the first row in the column direction. That is, the second phase difference pixel R and the first phase difference pixel L are alternately arranged from the first row in the column direction.

[0092] The photoelectric conversion element 26 is divided into three regions. That is, the photoelectric conversion element 26 has a visible light image division region 26N1, a first distance measurement system division region 26N2, and a second distance measurement system division region 26N3. The visible light image division region 26N1 is a visible light pixel group based on a plurality of visible light pixels and is used to generate a visible light image. The first distance measurement system division region 26N2 is an IR pixel group based on a plurality of IR pixels arranged in a two-dimensional shape and is used for the first distance measurement. Here, the IR pixel is an example of the "specific pixel" related to the technology of the present invention. The second distance measurement system division region 26N3 is a phase difference pixel group based on a plurality of phase difference pixels and is used for the second distance measurement. The visible light image division region 26N1 and the second distance measurement system division region 26N3 receive visible reflected light and output an electric signal corresponding to the amount of received light. The first distance measurement system division region 26N2 receives IR reflected light and outputs an electric signal corresponding to the amount of received light.

[0093] As an example, as Figure 6 shown, the first phase difference pixel L includes a microlens 19, a light shielding member 17A, and a photodiode PD. In the first phase difference pixel L, the light shielding member 17A is disposed between the microlens 19 and the light receiving surface of the photodiode PD. The left half of the light receiving surface of the photodiode PD in the row direction (the left side when facing the subject from the light receiving surface (in other words, the right side when facing the light receiving surface from the subject)) is shielded by the light shielding member 17A.

[0094] The second phase difference pixel R includes a microlens 19, a light shielding member 17B, and a photodiode PD. In the second phase difference pixel R, the light shielding member 17B is disposed between the microlens 19 and the light receiving surface of the photodiode PD. The right half of the light receiving surface of the photodiode PD in the row direction (the right side when facing the subject from the light receiving surface (in other words, the left side when facing the light receiving surface from the subject)) is shielded by the light shielding member 17B. In addition, hereinafter, for the sake of convenience of explanation, when there is no need to distinguish between the light shielding members 17A and 17B, they are referred to as "light shielding member" without symbols.

[0095] The light beam passing through the exit pupil of the imaging lens 41 is roughly divided into a left-region passing light 300L and a right-region passing light 300R. The left-region passing light 300L refers to the left half of the light beam among the light beams passing through the exit pupil of the imaging lens 41 when viewed from the phase difference pixel side towards the subject side, and the right-region passing light 300R refers to the right half of the light beam among the light beams passing through the exit pupil of the imaging lens 41 when viewed from the phase difference pixel side towards the subject side. The light beam passing through the exit pupil of the imaging lens 41 is divided left and right by the microlens 19, the light-shielding member 17A, and the light-shielding member 17B that function as a pupil splitting unit. The first phase difference pixel L receives the left-region passing light 300L as subject light, and the second phase difference pixel R receives the right-region passing light 300R as subject light. As a result, a first phase difference image corresponding to the subject image corresponding to the left-region passing light 300L and a second phase difference image corresponding to the subject image corresponding to the right-region passing light 300R are generated by the photoelectric conversion element 26.

[0096] In the intelligent device 10, for example, on the same phase difference pixel line 26A, the distance to the imaging region is measured based on the offset amount α between the first phase difference image of one row amount and the second phase difference image of one row amount.

[0097] As an example, as Figure 7 shown, the non-phase difference pixel N differs from the phase difference pixel in that it does not have a light-shielding member. The photodiode PD of the non-phase difference pixel N receives the left-region passing light 300L and the right-region passing light 300R as subject light.

[0098] In addition, in the intelligent device 10, the reflected IR light is received by a plurality of IR pixels included in the region 26N3 divided by the second distance measurement system (refer to Figure 5 ), and thus the distance measurement is performed for each IR pixel. And, in the intelligent device 10, according to the instruction received by the touch panel 48, as an example, as Figure 8 shown, the distance measurement result of each IR pixel is displayed as a distance image on the display 46. Here, the distance image refers to an image that represents the distance to the distance measurement object measured for each IR pixel in terms of color and / or brightness.

[0099] Also, in the smart device 10, according to the instruction received by the touch panel 48, the ranging result is displayed on the display 46 as a distance image or a distance-overlapped image (not shown). The distance-overlapped image displayed on the display 46 is, for example, an image obtained by overlapping numerical values representing the ranging result on a visible light image (e.g., an instant preview image). For example, the distances from the smart device 10 to each of a plurality of representative positions (e.g., three positions) within the imaging area are displayed on the visible light image and shown on the display 46. As an example of the plurality of representative positions, a plurality of positions within the imaging area where the contrast difference between each other in a specific subject (e.g., a subject and / or a person included in the central area of the screen) is equal to or greater than a specified value can be cited.

[0100] As an example, as Figure 9 shown, in addition to the light irradiator 16 and the light receiver 18, the smart device 10 further includes a controller 15, an input / output interface 40, an image memory 42, a UI system device 44, an external I / F 52, and a communication I / F 54.

[0101] The controller 15 includes a CPU 15A, a storage device 15B, and a memory 15C. The CPU 15A, the storage device 15B, and the memory 15C are connected via a bus 50, and the bus 50 is connected to the input / output interface 40. Additionally, in Figure 9 the example shown, for the sake of easy illustration, one bus is illustrated as the bus 50, but it may also be multiple buses. The bus 50 can be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0102] The storage device 15B stores various parameters and various programs. The storage device 15B is a non-volatile storage device. Here, as an example of the storage device 15B, a flash memory is adopted. The flash memory is merely an example, and as the storage device 15B, various non-volatile memories such as a magnetoresistive memory and / or a ferroelectric memory can be cited, for example, to replace the flash memory or be used together with the flash memory. Also, the non-volatile storage device can be an EEPROM, an HDD, and / or an SSD, etc. And the memory 15C temporarily stores various information and serves as a working memory. As an example of the memory 15C, a RAM can be cited, but it is not limited thereto, and other types of storage devices can also be used.

[0103] Various programs are stored in the storage device 15B. The CPU 15A reads out the required program from the storage device 15B and executes the read-out program on the memory 15C. The CPU 15A controls the entire smart device 10 according to the program executed on the memory 15C. Additionally, the storage device 15B and the memory 15C are an example of the "memory" related to the technology of the present invention.

[0104] Multiple devices are connected to the input / output interface 40, and the input / output interface 40 is responsible for the transmission and reception of various information between the multiple devices. Figure 9 In the example shown, as the multiple devices connected to the input / output interface 40, a controller 15, a light irradiator 16, a light receiver 18, an image memory 42, a UI system device 44, an external I / F 52, and a communication I / F 54 are shown.

[0105] The external I / F 52 is responsible for the transmission and reception of various information between devices existing outside the intelligent device 10 (hereinafter, also referred to as "external devices"). As an example of the external I / F 52, a USB interface can be cited. External devices such as intelligent devices, personal computers, servers, USB memories, memory cards, and / or printers (not shown) can be directly or indirectly connected to the USB interface.

[0106] The communication I / F 54 has communication functions such as LTE, 5G, wireless LAN, and / or Bluetooth (registered trademark), and is responsible for the transmission and reception of various information between external devices and the CPU 15A. For example, the communication I / F 54 is communicably connected to the network 56 (e.g., the Internet) via a base station (not shown), and is responsible for the transmission and reception of various information between external devices on the network 56 and the CPU 15A.

[0107] The UI system device 44 includes a display 46, and the CPU 15A causes the display 46 to display various information. Also, the UI system device 44 includes a receiving device 47. The receiving device 47 includes a touch panel 48 and a hard key unit 53. The hard key unit 53 is at least one hard key including the instruction key 13 (refer to Figure 2 ). The CPU 15A operates according to various instructions received through the touch panel 48. Additionally, here, the hard key unit 53 is included in the UI system device 44, but the technology of the present invention is not limited thereto. For example, the hard key unit 53 can also be connected to the external I / F 52.

[0108] The light irradiator 16 includes a light transmission window 20, a beam expander 21, a collimating lens 23, an LD 24, and an LD driver 25. Along the optical axis L1, the light transmission window 20, the beam expander 21, and the collimating lens 23 are sequentially arranged from the imaging area side (object side) to the LD 24. The LD driver 25 is connected to the LD 24 and the input / output interface 40, and drives the LD 24 according to the instruction of the CPU 15A to emit a laser beam from the LD 24.

[0109] After the laser beam emitted from the LD 24 is converted into parallel light by the collimating lens 23, the beam diameter is expanded by the beam expander 21, and is irradiated toward the distance measurement object from the light transmission window 20.

[0110] The light receiver 18 includes a light-transmitting window 22, an objective lens 30A, a focusing lens 30B, an aperture 30C, a photoelectric conversion element 26, a photoelectric conversion element driver 32, and a signal processing circuit 34.

[0111] In addition, the CPU 15A and the signal processing circuit 34 are examples of the "processor" involved in the technology of the present invention.

[0112] In the light receiver 18, along the optical axis L2, from the imaging region side (object side) to the photoelectric conversion element 26, a light-transmitting window 22, an objective lens 30A, a focusing lens 30B, and an aperture 30C are arranged in sequence. The photoelectric conversion element driver 32 is connected to the photoelectric conversion element 26 and the input / output interface 40, and drives the photoelectric conversion element 26 according to the instruction of the CPU 15A. For example, under the control of the CPU 15A, the photoelectric conversion element driver 32 supplies a imaging timing signal for specifying the timing of imaging performed by the photoelectric conversion element 26 to the photoelectric conversion element 26. The photoelectric conversion element 26 performs reset, exposure, and output of an electrical signal according to the imaging timing signal supplied from the photoelectric conversion element driver 32. As the imaging timing signal, for example, a vertical synchronization signal and a horizontal synchronization signal can be cited.

[0113] The light receiver 18 includes a focus control mechanism 31. The focus control mechanism 31 includes a focusing lens 30B, a moving mechanism 60, a motor 62, and a motor driver 64. The focusing lens 30B is supported by the moving mechanism 60 so as to be slidable along the optical axis L2. The motor 62 is connected to the moving mechanism 60 and the motor driver 64. The motor driver 64 is connected to the input / output interface 40, and drives the motor 62 according to the instruction from the CPU 15A. The moving mechanism 60 is connected to the drive shaft (not shown) of the motor 62, and selectively moves the focusing lens 30B along the optical axis L2 toward the object side and the image side by receiving power from the motor 62. That is, the CPU 15A adjusts the focus position by controlling the drive of the motor 62 via the motor driver 64. Here, the "focus position" refers to the position of the focusing lens 30B on the optical axis L2 in a state where the focus is aligned (for example, a state where the contrast of the visible light image is set to the maximum value or a state where a predetermined subject depth is achieved). In addition, in the first embodiment, the control of aligning the focusing lens 30B to the focus position is referred to as "focus control".

[0114] The aperture 30C is a fixed aperture with a constant opening. In the case of a fixed aperture, the exposure adjustment is performed by the electronic shutter of the photoelectric conversion element 26. The aperture 30C may not be a fixed aperture but a variable aperture. In addition, the objective lens 30A, the focusing lens 30B, and the aperture 30C included in the light receiver 18 are merely examples, and the technology of the present invention is also valid even if the structure of the lens and / or the position of the aperture 30C are changed.

[0115] In the light receiver 18, the reflected light is incident from the light-transmitting window 22. The reflected light incident on the light-transmitting window 22 is imaged on the photoelectric conversion element 26 via the objective lens 30A, the focusing lens 30B, and the aperture 30C.

[0116] The photoelectric conversion element 26 is connected to the signal processing circuit 34, and outputs pixel data representing the pixel values of the respective pixels of the visible light pixels and the IR pixels to the signal processing circuit 34. The signal processing circuit 34 digitizes the pixel data by performing A / D conversion on the pixel data input from the photoelectric conversion element 26, and performs various signal processes on the digitized pixel data.

[0117] The signal processing circuit 34 includes a visible light pixel data processing circuit 34A, a first distance measurement system processing circuit 34B, and a second distance measurement system processing circuit 34C. The visible light pixel data processing circuit 34A performs known signal processes such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction on the pixel data regarding the visible light pixels, that is, the visible light pixel data, thereby generating a visible light image. Then, the visible light pixel data processing circuit 34A stores the visible light image in the image memory 42. In addition, by covering and storing one frame amount of the visible light image in the image memory 42, the visible light image in the image memory 42 is updated.

[0118] The distance measurement imaging device 14 includes a TOF camera 27. The TOF camera 27 includes a light irradiator 16, a first distance measurement system division area 26N2 of the photoelectric conversion element 26 (reference Figure 5 ), and a first distance measurement system processing circuit 34B. The first distance measurement system processing circuit 34B acquires an irradiation timing signal indicating the irradiation timing (hereinafter, also simply referred to as "irradiation timing") at which the laser beam is irradiated toward the imaging area by the light irradiator 16 from the CPU 15A.

[0119] The first distance measurement system processing circuit 34B measures the distance from the smart device 10 to an object in the imaging area for each IR pixel based on the irradiation timing indicated by the irradiation timing signal and the timing at which each IR pixel receives the IR reflected light (hereinafter, also referred to as "light reception timing"). Here, as the light reception timing, the timing at which the first distance measurement system processing circuit 34B receives IR pixel data having an output value exceeding a reference threshold value (reference Figure 14 ) described later is adopted. In addition, as an example of the noise component, a noise component generated independently of the IR reflected light (for example, IR light included in the ambient light) can be cited.

[0120] The first distance measurement system processing circuit 34B measures the distance from the smart device 10 to an object in the imaging area for each IR pixel based on the irradiation timing and the light reception timing. Further, the first distance measurement system processing circuit 34B generates a distance image based on the distance measurement results for each IR pixel, and stores the generated distance image in the image memory 42. In addition, by covering and saving a distance image of one frame amount in the image memory 42, the distance image in the image memory 42 is updated.

[0121] The second distance measurement system processing circuit 34C acquires phase difference pixel data representing pixel values of phase difference pixels from a plurality of phase difference pixels included in a region (so-called ROI) specified by a user or the like in the photoelectric conversion element 26 from the second distance measurement system divided region 26N3 (refer to Figure 5 ). The second distance measurement system processing circuit 34C generates a first phase difference image and a second phase difference image based on the phase difference pixel data (refer to Figure 5 ), and calculates an offset α between the generated first phase difference image and the second phase difference image (refer to Figure 5 ). Further, the second distance measurement system processing circuit 34C calculates the distance from the smart device 10 to the imaging area based on the calculated offset α. More specifically, the second distance measurement system processing circuit 34C calculates the distance from the smart device 10 to an object in the imaging area by using an arithmetic expression in which the offset α is an independent variable and the distance is a dependent variable.

[0122] In addition, an arithmetic expression is exemplified herein, but the technology of the present invention is not limited thereto, and the second distance measurement system processing circuit 34C may also derive the distance from the smart device 10 to the imaging area by using a table in which the offset α and the distance are associated with each other.

[0123] The CPU 15A acquires the distance measured by the first distance measurement system processing circuit 34B (hereinafter, referred to as "first distance") from the first distance measurement system processing circuit 34B, and acquires the distance measured by the second distance measurement system processing circuit 34C (hereinafter, referred to as "second distance") from the second distance measurement system processing circuit 34C.

[0124] As an example, as Figure 10 shown, a distance measurement imaging processing program 70 is stored in the storage device 15B. The CPU 15A reads out the distance measurement imaging processing program 70 from the storage device 15B, and executes the read-out distance measurement imaging processing program 70, thereby operating as a first distance measurement control unit 15A1, a first distance acquisition unit 15A2, a time series distribution acquisition unit 15A3, a determination unit 15A4, and an execution unit 15A5.

[0125] Here, refer to Figure 11, the case of performing first distance measurement on the imaging area including the object subject ( Figure 11 In the example shown, the person) 98 and the entire dressing mirror 100 (hereinafter, also referred to as the "mirror-including imaging area") will be described. When a distance measurement imaging start instruction is received through the touch panel 48 in a state where the mirror-including imaging area is included in the irradiation angle θ2, the first distance measurement control unit 15A1 outputs a first distance measurement start signal to the light irradiator 16 and the light receiver 18. When the first distance measurement start signal is input from the first distance measurement control unit 15A1, the light irradiator 16 irradiates a laser beam. In the light receiver 18, when the first distance measurement start signal is input from the first distance measurement control unit 15A1, the first distance measurement system processing circuit 34B calculates the first distance based on the irradiation timing and the light reception timing indicated by the irradiation timing signal obtained from the CPU15A.

[0126] In addition, here, as the irradiation timing, the timing of a point in time after a predetermined time has elapsed from the timing when the first distance measurement start signal is input from the first distance measurement control unit 15A1 to the light receiver 18 is adopted. Here, regarding the predetermined time, for example, the time required from the time of outputting the first distance measurement start signal to the time when the laser beam is irradiated from the light irradiator 16 is adopted, and the time is derived in advance through experiments based on the actual machine and / or computer simulations, etc.

[0127] When the laser beam is irradiated from the light irradiator 16 to the mirror-including imaging area, as an example, as Figure 12 shown, the first distance measurement system division area 26N2 of the light receiver 18 receives the IR reflected light from the object subject 98 (hereinafter, also referred to as the "object subject IR reflected light") and the IR reflected light from the dressing mirror 100 (hereinafter, also referred to as the "mirror surface IR reflected light"). A part of the IR pixels (hereinafter, also referred to as "specific IR pixels") within the first distance measurement system division area 26N2 receive the object subject IR reflected light. If only the object subject IR reflected light is received by the specific IR pixels, the first distance measurement system processing circuit 34B uses the timing when the object subject IR reflected light is received by the specific IR pixels as the above-mentioned light reception timing, and thus the distance from the smart device 10 to the object subject 98 can be calculated as the first distance.

[0128] However, depending on the installation conditions of the dressing mirror 100, that is, depending on the position where the dressing mirror 100 is installed, the shape of the mirror surface 100A of the dressing mirror 100, and the angle of the mirror surface 100A, etc., the total reflection light of the laser beam with respect to the mirror surface 100A is also received by the specific IR pixels as the mirror surface IR reflected light. When the intensity of the laser beam irradiated from the light irradiator 16 is set to "100", the intensity of the object subject IR reflected light is about 10 to 20, while the intensity of the total reflection light of the laser beam with respect to the mirror surface 100A is about 40 to 50.

[0129] Here, it is assumed that during a light reception period (hereinafter, also simply referred to as "light reception period") preset during the period of receiving IR reflected light corresponding to the irradiation of the laser beam from the light irradiator 16, among the multiple IR reflected lights received by a specific IR pixel at different timings, when ranging is performed according to the timing of the IR reflected light with the maximum intensity received by the specific IR pixel, it can be considered that ranging is performed according to the timing of the specular IR reflected light received by the specific IR pixel. This means that when the object the user wishes to be the ranging object is the target subject, false ranging that the user does not wish for is performed. That is, the distance from the smart device 10 to an object that the user does not wish to be the ranging object (in the Figure 12 example shown, the mirror 100A) is measured instead of the distance from the smart device 10 to the target subject 98.

[0130] Therefore, as an example, in order to avoid such false ranging, as Figure 13 shown, the first ranging system processing circuit 34B includes an IR pixel data acquisition unit 34B1, a time series distribution generation unit 34B2, a light reception timing determination unit 34B3, and a first distance measurement unit 34B4. In addition, an IR pixel data acquisition unit 34B1, a time series distribution generation unit 34B2, a light reception timing determination unit 34B3, and a first distance measurement unit 34B4 are provided for each IR pixel. Hereinafter, for the sake of convenience in explanation, regarding the first ranging, a specific IR pixel (refer to Figures 12 - 14 ) is focused on for explanation.

[0131] The first ranging control unit 15A1 outputs an irradiation timing signal to the IR pixel data acquisition unit 34B1, the time series distribution generation unit 34B2, and the first distance measurement unit 34B4. And the first ranging control unit 15A1 outputs an acquisition timing signal that specifies the timing for the IR pixel data acquisition unit 34B1 to acquire IR pixel data from the photoelectric conversion element 26 to the IR pixel data acquisition unit 34B1. The acquisition timing signal is output to the IR pixel data acquisition unit 34B1 at a prescribed time interval (for example, a time interval of about one tenth or one hundredth of the light reception period).

[0132] If an irradiation timing signal and an acquisition timing signal are input from the first distance measurement control unit 15A1, the IR pixel data acquisition unit 34B1 acquires IR pixel data from the photoelectric conversion element 26. If an irradiation timing is input from the first distance measurement control unit 15A1, the time series distribution generation unit 34B2 generates a time series distribution in which the intensity of the IR reflected light represented by a plurality of IR pixel data acquired by the IR pixel data acquisition unit 34B1 during the light reception period starting from the point in time when the irradiation timing is input from the first distance measurement control unit 15A1 is specified in time series. In the time series distribution, the horizontal axis represents time and the vertical axis represents intensity. Here, the plurality of IR pixel data is an example of the "plurality of signals" related to the technology of the present invention. The details of the time series distribution will be described later.

[0133] The intensity of the IR reflected light has a one-to-one relationship with the signal level (intensity) of the IR pixel data, and the intensity of the IR reflected light is determined according to the signal level of the IR pixel data. Therefore, the time series distribution that specifies the intensity of the IR reflected light in time series is equivalent to the time series distribution of the signal level of the IR pixel data. And in the first embodiment, the intensity of the IR reflected light is determined by the absolute value amount of the IR reflected light, but the technology of the present invention is not limited to this, and the intensity of the IR reflected light may also be offset by the amount of the intensity of the noise component based on the ambient light (for example, the IR light included in the ambient light). At this time, for example, the entire time series distribution may be offset by the amount of the intensity of the noise component based on the ambient light.

[0134] In the time series distribution generated by the time series distribution generation unit 34B2, the intensity of the IR reflected light represented by the IR pixel data obtained from a specific IR pixel is represented in time series. The light reception timing determination unit 34B3 determines the light reception timing for the first distance measurement (hereinafter, also referred to as "the light reception timing for the first distance measurement") according to the time series distribution generated by the time series distribution generation unit 34B2. A reference threshold is applied to the time series distribution in the light reception timing determination unit 34B3. The reference threshold is a value derived in advance by an actual machine and / or computer simulation or the like as the lower limit value of the intensity of the IR reflected light from a standard subject (for example, a specific subject other than a mirror surface and a glossy surface). The reference threshold is a value that decreases as the light reception timing is delayed. In Figure 13 the example shown, the reference threshold gradually (for example, exponentially) decreases with the passage of time. If there is only one IR reflected light with an intensity exceeding the reference threshold in the time series distribution, the light reception timing determination unit 34B3 determines the light reception timing of the IR reflected light with an intensity exceeding the reference threshold as the light reception timing for the first distance measurement.

[0135] Further, when there are multiple IR reflected lights with intensities exceeding the reference threshold in the time series distribution, the light reception timing determination unit 34B3 determines the light reception timing corresponding to the IR pixel data selected based on the relationship of the multiple IR pixel data and the intensities of the multiple IR reflected lights represented by the multiple IR pixel data among the multiple IR pixel data acquired by the IR pixel data acquisition unit 34B1 during the light reception period as the first light reception timing for distance measurement. The relationship of the multiple IR pixel data and the intensities of the multiple IR reflected lights are determined by the light reception timing determination unit 34B3 based on the time series distribution generated by the time series distribution generation unit 34B2. In addition, the detailed content of the method for determining the first light reception timing for distance measurement when there are multiple IR reflected lights with intensities exceeding the reference threshold in the time series distribution will be described later.

[0136] The first distance measurement unit 34B4 measures the first distance based on the irradiation timing indicated by the irradiation timing signal input from the first distance measurement control unit 15A1 and the first light reception timing for distance measurement determined by the light reception timing determination unit 34B3. The first distance is half of the product of the speed of light and the flight time of the laser beam. The flight time of the laser beam is the time from the irradiation timing to the first light reception timing for distance measurement.

[0137] In Figure 14 In the example shown, the time series distribution includes the intensities of specular IR reflected light, object subject IR reflected light, and noise light (e.g., noise components such as IR light contained in ambient light). On the time axis of the light reception period, the specular IR reflected light is first received by a specific IR pixel, then the noise light is received, and then the object subject IR reflected light is received. Moreover, the intensities of the specular IR reflected light and the object subject IR reflected light exceed the reference threshold.

[0138] At this time, the light reception timing determination unit 34B3 determines the timing when the second largest intensity IR reflected light is received by the specific IR pixel in the time series distribution as the first light reception timing for distance measurement. That is, in Figure 14 In the example shown, the second largest intensity IR reflected light in the time series distribution is the object subject IR reflected light. Therefore, the light reception timing of the object subject IR reflected light is determined as the first light reception timing for distance measurement. Thereby, it is possible to avoid performing the first distance measurement based on the light reception timing of the specular IR reflected light and performing the first distance measurement based on the noise light.

[0139] As an example, as Figure 15As shown, the first distance acquisition unit 15A2 acquires the first distance measured by the first distance measurement unit 34B4. Further, the time-series distribution acquisition unit 15A3 acquires a time-series distribution from the time-series distribution generation unit 34B2. The time-series distribution acquired by the time-series distribution acquisition unit 15A3 is a time-series distribution for determining the first light-receiving timing for distance measurement used in the measurement of the first distance acquired by the first distance acquisition unit 15A2.

[0140] The determination unit 15A4 determines whether the time-series distribution acquired by the time-series distribution acquisition unit 15A3 is a specific time-series distribution. The details of this determination method will be described later.

[0141] When the determination unit 15A4 determines that the time-series distribution acquired by the time-series distribution acquisition unit 15A3 is not a specific time-series distribution, the execution unit 15A5 executes imaging (hereinafter, also referred to as "first imaging process") accompanied by focusing control based on the first distance acquired by the first distance acquisition unit 15A2. Further, when it is determined that the time-series distribution acquired by the time-series distribution acquisition unit 15A3 is a specific time-series distribution, the execution unit 15A5 executes a specific process. In addition, the details of the first imaging process and the specific process will be described later.

[0142] Here, the determination method based on the determination unit 15A4 will be described. As an example, as Figure 16 shown, the determination unit 15A4 applies a first threshold value and a second threshold value to the time-series distribution acquired by the time-series distribution acquisition unit 15A3. The first threshold value is a lower limit value of the intensity of the IR reflected light when the laser beam irradiated on a glossy surface (for example, a mirror surface preset as an average mirror surface) is totally reflected by the glossy surface, and is a value derived in advance through actual machines and computer simulations, etc. The first threshold value gradually (for example, exponentially) decreases over time in the same manner as the above-mentioned reference threshold value. Further, the first threshold value is a value that decreases as the light-receiving timing is delayed in the same manner as the above-mentioned reference threshold value. The second threshold value is the same value as the above-mentioned reference threshold value (refer to Figure 13 and Figure 14 ). Further, at the same moment on the time axis during the light-receiving period, the second threshold value is a value smaller than the first threshold value.

[0143] The determination unit 15A4 determines whether the time-series distribution acquired by the time-series distribution acquisition unit 15A3 includes an intensity equal to or higher than the first threshold value and an intensity lower than the first threshold value and equal to or higher than the second threshold value, and thereby determines whether the time-series distribution acquired by the time-series distribution acquisition unit 15A3 is a specific time-series distribution. In Figure 16In the example shown, the intensity of the specular IR reflected light is above the first threshold, and the intensity of the IR reflected light of the object subject is less than the first threshold and above the second threshold. Therefore, the determination unit 15A4 determines that the time series distribution obtained by the time series distribution acquisition unit 15A3 is a specific time series distribution. In addition, in Figure 16 In the example shown, the intensity of the specular IR reflected light corresponds to the intensity of the IR pixel data (signal) generated by receiving the specular IR reflected light. The intensity of the IR pixel data (signal) generated by receiving the specular IR reflected light is an example of the "first intensity" related to the technology of the present invention. Similarly, the intensity of the IR reflected light of the object subject corresponds to the intensity of the IR pixel data (signal) generated by receiving the IR reflected light of the object subject. The intensity of the IR pixel data (signal) generated by receiving the IR reflected light of the object subject is an example of the "second intensity" related to the technology of the present invention.

[0144] As an example, as Figure 17 shown, a focus position derivation table 72 is stored in the storage device 15B. In the focus position derivation table 72, a correspondence is established between the distance from the smart device 10 to the imaging area and the focus position. In the first imaging process, first, the execution unit 15A5 performs focus control on the light receiver 18 by using the first distance (active mode focus control). That is, the execution unit 15A5 derives the focus position corresponding to the first distance from the focus position derivation table 72, and controls the motor 62 of the light receiver 18 so that the focusing lens 30B moves to the derived focus position. Then, the execution unit 15A5 controls the visible light image division area 26N1 of the light receiver 18, thereby capturing the imaging area through the visible light image division area 26N1, and the visible light image data obtained by the capture is output from the visible light image division area 26N1 to the visible light pixel data processing circuit 34A. The visible light pixel data processing circuit 34A generates a first visible light image representing the imaging area based on the visible light image data input from the visible light image division area 26N1, and outputs the generated first visible light image to the image memory 42. The first visible light image is stored in the image memory 42.

[0145] Here, a specific process executed by the execution unit 15A5 will be described with reference to Figure 18 As an example, as Figure 18 shown, the specific process is, for example, a process including a second distance measurement start process, a second distance acquisition process, a second imaging process, and an image selection screen display process. In addition, the specific process is not limited to these, and may also be a process of measuring the first distance based on the light reception timing of the reflected light (IR reflected light of the object subject) whose intensity received by the IR pixel is less than the first threshold and above the second threshold.

[0146] As an example, as Figure 19 shown, in the second distance measurement start process, if a distance measurement camera start instruction is received by the touch panel 48, the execution unit 15A5 outputs a second distance measurement start signal to the light receiver 18. Additionally, Figure 19 in the example shown, a state where the mirror includes an imaging area within the field of view angle θ1 is shown, but the technology of the present invention is not limited thereto, and it may also be an imaging area other than the imaging area where the mirror is included.

[0147] If a second distance measurement start signal is input from the execution unit 15A5 to the light receiver 18, the second distance measurement system divided area 26N3 captures the mirror-included imaging area and outputs phase difference pixel data corresponding to the mirror-included imaging area to the second distance measurement system processing circuit 34C. The second distance measurement system processing circuit 34C generates a first phase difference image and a second phase difference image based on the phase difference pixel data input from the second distance measurement system divided area 26N3 (refer to Figure 6 ), and calculates an offset α based on the generated first phase difference image and second phase difference image (refer to Figure 6 ). The second distance measurement system processing circuit 34C calculates the second distance based on the calculated offset α. In the second distance acquisition process, the execution unit 15A5 acquires the second distance from the second distance measurement system processing circuit 34C.

[0148] As an example, as Figure 20 shown, the second imaging process refers to imaging accompanied by focus control based on the second distance. In the second imaging process, similarly to the first imaging process, the execution unit 15A5 derives a focus position corresponding to the second distance from the focus position derivation table 72 and controls the motor 62 of the light receiver 18 so that the focus lens 30B moves to the derived focus position. Then, similarly to the first imaging process, the visible light image divided area 26N1 captures the imaging area, and the visible light pixel data obtained by the capture is output from the visible light image divided area 26N1 to the visible light pixel data processing circuit 34A. The visible light pixel data processing circuit 34A generates a second visible light image representing the imaging area based on the visible light pixel data input from the visible light image divided area 26N1 and outputs the generated second visible light image to the image memory 42. The second visible light image is stored in the image memory 42.

[0149] In the image selection screen display process, the execution unit 15A5 acquires the first visible light image and the second visible light image from the image memory 42. Then, the execution unit 15A5 based on the first visible light image, the second visible light image, the first distance acquired by the first distance acquisition unit 15A2 (refer to Figure 15 ), the second distance acquired by the execution unit 15A5 in the second distance acquisition process (refer to Figure 19) and various messages to generate an image selection screen, and display the generated image selection screen on the display 46. The first visible light image and the second visible light image are arranged and displayed on the image selection screen. And, a message "active mode" is displayed below the first visible light image. The message "active mode" is a message indicating that the first visible light image is an image obtained by imaging with focus control accompanying the ranging result based on ranging (first ranging) in the active mode. And, a message "passive mode" is displayed below the second visible light image. The message "passive mode" is a message indicating that the second visible light image is an image obtained by imaging with focus control accompanying the ranging result based on ranging (second ranging) in the passive mode.

[0150] In addition, in this first embodiment, as the processes included in the specific process, the first imaging process, the second imaging process, and the image selection screen display process are exemplified, but the technology of the present invention is not limited thereto, and it may also be one process or two processes among the first imaging process, the second imaging process, and the image selection screen display process. And, in this first embodiment, a display example of "active mode" and "passive mode" is shown, but it is not necessary to display "active mode" and "passive mode", and any display may be used as long as the user can understand the difference in the ranging methods. For example, when performing laser ranging and phase difference ranging, it may be the display of "laser" and "phase difference", or an icon indicating the ranging method may be displayed. And, the focus position may be displayed instead of the display of the ranging method. For example, it may be "focus position: near" and "focus position: far", or it may be displayed as "focus position: object" and "focus position: image of the object reflected" etc. Or, two or more of the characters and icons indicating the ranging method and the focus position may be combined for display.

[0151] And, in the first visible light image, a value of "1.8 m", which is the ranging result based on the first ranging, is displayed in a corresponding association, and in the second visible light image, a value of "1.9 m", which is the ranging result based on the second ranging, is displayed in a corresponding association. And, on the image selection screen, as a message prompting the user to select either the first visible light image or the second visible light image, a message "please select any one image" is displayed to the user.

[0152] Moreover, a strong reflected light notification message is displayed on the image selection screen. The strong reflected light notification message means that when it is determined by the determination unit 15A4 (refer to Figure 15 ) that the time series distribution obtained by the time series distribution acquisition unit 15A3 (refer to Figure 15 ) is a specific time series distribution (for example, Figure 16As shown, when the time series distribution includes an intensity above the first threshold, a message indicating that the time series distribution obtained by the time series distribution acquisition unit 15A3 includes an intensity above the first threshold is notified to the user. In Figure 20 In the example shown, as a strong reflected light notification message, a message "Strong reflected light detected, but ranging was performed using reflected light other than the strong reflected light." is shown. Here, the strong reflected light means IR reflected light with an intensity above the first threshold. As an example of the strong reflected light, specular IR reflected light can be cited (refer to Figure 12 , Figure 14 and Figure 16 ). In addition, even for IR reflected light from a glossy surface other than the mirror surface 100A, if it is IR reflected light with an intensity above the first threshold, a strong reflected light notification message is displayed on the image selection screen. In addition, the image selection screen display process is an example of the "process of notifying that the intensity of multiple signals includes the first intensity" related to the technology of the present invention.

[0153] The various messages displayed on the image selection screen are not limited to the above messages. For example, when the ranging result (first distance) based on the first ranging is different from the ranging result (second distance) based on the second ranging, a message notifying the user of this content (e.g., "The ranging results are different") can also be displayed.

[0154] And here, an example of a method of visually displaying various messages has been described, but it is not limited to this. It can also be set to output various messages by sound in parallel with the visual display. When the image selection screen is displayed on the display 46, the user selects the first visible light image or the second visible light image via the touch panel 48. In Figure 20 In the example shown, a method is shown in which the user's finger selects the first visible light image via the touch panel 48.

[0155] As an example, as Figure 21 shown, when the user's finger selects the first visible light image via the touch panel 48 in a state where the image selection screen is displayed on the display 46, the first ranging control unit 15A1 controls the ranging imaging device 14 to cause the first ranging system processing circuit 34B to perform the first ranging. And the execution unit 15A5 executes the first imaging process and the first visible light image display process. Here, the execution unit 15A5 performs the first imaging process using the new first distance obtained by performing the first ranging. The first visible light image display process is a process of displaying the latest first visible light image obtained by performing the first imaging process on the display 46.

[0156] In addition, here, an example is given in which, under the condition that the first visible light image is selected, the first distance measurement and the first imaging process are performed again, and the latest first visible light image obtained by performing the first imaging process is displayed on the display 46. However, this is merely an example. For example, under the condition that the first visible light image is selected, the selected first visible light image may be stored in a storage device 15B and / or a storage medium such as a memory card. And it may be set as follows: according to an instruction received by the receiving device 47 (an instruction issued by the user), imaging with focus control based on the distance measured by the selected distance measurement method and the saving of the selected image (for example, the first visible light image or the second visible light image) are selected.

[0157] If the first visible light image display process is executed by the execution unit 15A5, then within the image selection screen, the second visible light image, the numerical value "1.9 m", the message "passive mode", the message "Please select any one image", and the strong reflected light notification message are not displayed on the screen. Then, as the first distance ( Figure 21 In the example shown, the numerical value "1.8 m") of the latest distance measurement result obtained by performing the latest first distance measurement is displayed on the display 46, the first visible light image obtained by executing the first imaging process is displayed on the display 46, and the message "active mode" is also displayed below the first visible light image. Also, the display areas of the first distance, the first visible light image, and the message "active mode" are enlarged compared to the display areas of the first distance, the first visible light image, and the message "active mode" within the image selection screen. Additionally, at this time, it may also be set as follows: when it is determined by the determination unit 15A4 (refer to Figure 15 ) that the time series distribution obtained by the time series distribution acquisition unit 15A3 (refer to Figure 15 ) is a specific time series distribution through the first distance measurement, the strong reflected light notification message is displayed on the display 46.

[0158] As an example, as Figure 22 shown, when the user's finger selects the second visible light image via the touch panel 48 in a state where the image selection screen is displayed on the display 46, the execution unit 15A5 starts the second distance measurement process, the second distance acquisition process, and the second imaging process by executing the second distance measurement, and causes the second distance measurement system processing circuit 34C to perform the second distance measurement. And the execution unit 15A5 executes the second imaging process and the second visible light image display process. Here, the execution unit 15A5 performs the second imaging process using the new second distance obtained by performing the second distance measurement. The second visible light image display process is a process of displaying the latest second visible light image obtained by executing the second imaging process on the display 46.

[0159] If the second visible light image display process is executed by the execution unit 15A5, in the image selection screen, the first visible light image, the numerical value "1.8 m", the message "Active mode", the message "Distance measurement results are different", and the message "Please select any one image" are not displayed on the screen. Then, as the second distance ( Figure 22 In the example shown, the numerical value "1.9 m") obtained by performing the latest second distance measurement is displayed on the display 46, the second visible light image obtained by executing the second imaging process is displayed on the display 46, and the message "Passive mode" is also displayed below the second visible light image. In addition, the display areas of the second distance, the second visible light image, and the message "Passive mode" are enlarged compared to the display areas of the second distance, the second visible light image, and the message "Passive mode" in the image selection screen.

[0160] Next, the operation of the part related to the technology of the present invention of the smart device 10 will be described.

[0161] First, refer to Figure 23 The first distance measurement system process executed by the first distance measurement system processing circuit 34B will be described. In addition, Figure 23 is a flowchart showing an example of the process of the first distance measurement system process executed by the first distance measurement system processing circuit 34B when an irradiation timing signal is input from the first distance measurement control unit 15A1.

[0162] In Figure 23 In the first distance measurement system process shown, first, in step ST100, the IR pixel data acquisition unit 34B1 determines whether a acquisition timing signal is input from the first distance measurement control unit 15A1. In step ST100, when the acquisition timing signal is not input from the first distance measurement control unit 15A1, the determination is negative, and the determination of step ST100 is performed again. In step ST100, when the acquisition timing signal is input from the first distance measurement control unit 15A1, the determination is affirmative, and the first distance measurement system process proceeds to step ST102.

[0163] In step ST102, the IR pixel data acquisition unit 34B1 acquires IR pixel data from the photoelectric conversion element 26, and then the first distance measurement system process proceeds to step ST104.

[0164] When measuring the distance based on the light reception timing and irradiation timing of the following IR reflected light received by the IR pixel, it is possible to measure the distance based on the light reception timing and irradiation timing of the IR reflected light from a glossy object ( Figure 12 In the example shown, the mirror 100A) that is not desired to be the object of distance measurement, and the IR reflected light is received by a plurality of light reception timings (for example,Figure 14 The highest signal level IR pixel data among the multiple IR pixel data generated by dividing the area 26N2 by the first distance measurement system (the "time interval" shown) is represented.

[0165] Therefore, in the first distance measurement system processing, the processing of steps ST104 to ST110 is executed. First, in step ST104, the time series distribution generation unit 34B2 determines whether the light reception period has elapsed after the start of the first distance measurement system processing. In step ST104, when the light reception period has not elapsed after the start of the first distance measurement system processing, the determination is negative, and the first distance measurement system processing transitions to step ST100. In step ST104, when the light reception period has elapsed after the start of the first distance measurement system processing, the determination is affirmative, and the first distance measurement system processing transitions to step ST106.

[0166] In step ST106, the time series distribution generation unit 34B2 generates a time series distribution based on the IR pixel data acquired by the IR pixel data acquisition unit by executing the processing of steps ST100 to ST104 (refer to Figure 14 ). After executing step ST106, the first distance measurement system processing transitions to step ST108.

[0167] In step ST108, the light reception timing determination unit 34B3 determines the first distance measurement light reception timing based on the time series distribution generated in step ST106. In this step ST108, if there is only one IR reflected light with an intensity exceeding the reference threshold in the time series distribution, the light reception timing determination unit 34B3 determines the light reception timing of the IR reflected light with an intensity exceeding the reference threshold as the first distance measurement light reception timing. And when there are multiple IR reflected lights with intensities exceeding the reference threshold in the time series distribution, the light reception timing determination unit 34B3 determines the timing when the second largest intensity IR reflected light in the time series distribution is received by a specific IR pixel as the first distance measurement light reception timing. After executing the processing of step ST108, the first distance measurement system processing transitions to step ST110.

[0168] In step ST110, the first distance measurement unit 34B4 measures the first distance based on the irradiation timing represented by the irradiation timing signal input from the first distance measurement control unit 15A1 and the first distance measurement light reception timing determined in step ST108. After executing the processing of step ST110, the first distance measurement system processing ends.

[0169] Next, refer to Figures 24A - 24C The distance measurement imaging process executed by the CPU15A will be described. Additionally, Figures 24A - 24C is a flowchart showing an example of the flow of the distance measurement imaging process executed by the CPU15A according to the distance measurement imaging process program 70.

[0170] In Figure 24A the distance measurement imaging process shown, first, in step ST200, the time-series distribution acquisition unit 15A3 determines whether a time-series distribution has been generated by the time-series distribution generation unit 34B2. In step ST200, when the time-series distribution generation unit 34B2 has not generated a time-series distribution, the determination is negative, and the determination in step ST200 is performed again. In step ST200, when the time-series distribution generation unit 34B2 has generated a time-series distribution, the determination is affirmative, and the distance measurement imaging process proceeds to step ST202.

[0171] In step ST202, the time-series distribution acquisition unit 15A3 acquires the time-series distribution from the time-series distribution generation unit 34B2, and then the distance measurement imaging process proceeds to step ST204.

[0172] In step ST204, the determination unit 15A4 determines whether the time-series distribution acquired in step ST202 is a specific time-series distribution. In step ST204, when the time-series distribution acquired in step ST202 is not a specific time-series distribution, the determination is negative, and the distance measurement imaging process proceeds to Figure 24B the step ST218 shown. In step ST204, when the time-series distribution acquired in step ST202 is a specific time-series distribution, the determination is affirmative, and the distance measurement imaging process proceeds to step ST205.

[0173] In step ST205, the first distance acquisition unit 15A2 acquires the first distance measured through Figure 23 the step ST110 shown. After performing the process of step ST205, the distance measurement imaging process proceeds to step ST206.

[0174] In step ST206, the execution unit 15A5 performs imaging (first imaging process) with focus control (active mode focus control) based on the first distance acquired in step ST205, and then the distance measurement imaging process proceeds to step ST208.

[0175] The processes of steps ST208 to ST212 are the above-mentioned specific processes. In step ST208, the execution unit 15A5 causes the second distance measurement system processing circuit 34C to perform second distance measurement by executing the second distance measurement start process and the second distance acquisition process, and acquires the distance measurement result based on the second distance measurement, that is, the second distance. After performing the process of step ST208, the distance measurement imaging process proceeds to step ST210.

[0176] In step ST210, the execution unit 15A5 performs imaging (second imaging process) accompanied by focus control (passive mode focus control) based on the second distance obtained in step ST208. After the process of step ST210 is executed, the distance measurement imaging process transitions to step ST212.

[0177] In step ST212, the execution unit 15A5 performs an image selection screen display process. Thereby, an image selection screen is generated based on the first distance obtained in step ST205, the first visible light image obtained by performing the first imaging process of step ST206, the second distance obtained in step ST208, the second visible light image obtained by performing the second imaging process of step ST210, and the above various messages (refer to Figure 20 ), and the image selection screen is displayed on the display 46. After the process of step ST212 is executed, the distance measurement imaging process transitions to step ST214.

[0178] The first visible light image obtained by performing the first imaging process of step ST206 and the second visible light image obtained by performing the second imaging process of step ST210 are displayed on the image selection screen. Therefore, in step ST214, the execution unit 15A5 determines whether the user has selected either the first visible light image or the second visible light image displayed in the image selection screen via the touch panel 48. In step ST214, when the user has not selected either the first visible light image or the second visible light image displayed in the image selection screen via the touch panel 48, the determination is negative, and the determination of step ST214 is performed again. In step ST214, when the user has selected either the first visible light image or the second visible light image displayed in the image selection screen via the touch panel 48, the determination is positive, and the distance measurement imaging process transitions to step ST216.

[0179] In step ST216, the execution unit 15A5 determines whether the image selected from within the image selection screen is the first visible light image. In step ST216, when the image selected from within the image selection screen is the second visible light image, the determination is negative, and the distance measurement imaging process transitions to Figure 24C step ST226 shown. In step ST216, when the image selected from within the image selection screen is the first visible light image, the determination is positive, and the distance measurement imaging process transitions to Figure 24B step ST218 shown.

[0180] In Figure 24BIn step ST218 shown above, the execution unit 15A5 determines whether the imaging start timing has been reached. The imaging start timing refers to, for example, the timing to start imaging for one frame amount defined by the frame rate for the live preview image. For example, if the frame rate for the live preview image is 60 fps, the imaging start timing is the timing every 1 / 60 second. Additionally, here, the timing to start imaging for one frame amount defined by the frame rate for the live preview image is exemplified as the imaging start timing, but the technology of the present invention is not limited thereto. For example, the imaging start timing may be the timing to start imaging for one frame amount defined by the frame rate for recording a moving image, or may be the timing when the receiving device 47 receives an instruction to start imaging a still image.

[0181] In step ST218, when the imaging start timing has not been reached, the determination is negative, and the distance measurement imaging process transitions to step ST222. In step ST218, when the imaging start timing has been reached, the determination is positive, and the distance measurement imaging process transitions to step ST220.

[0182] In step ST220, the execution unit 15A5 performs the first imaging process using the latest first distance. And the execution unit 15A5 performs the first visible light image display process using the latest first visible light image obtained by performing the first imaging process. Additionally, here, the latest first distance refers to the latest first distance among the first distance measured by performing the process of step ST110 and the first distance measured by performing the process of step ST224 described later.

[0183] In step ST222, the execution unit 15A5 determines whether the condition to end the distance measurement imaging process (hereinafter referred to as the "end condition") is satisfied. As an example of the end condition, there may be mentioned the condition that the receiving device 47 receives an instruction to end the distance measurement imaging process. In step ST222, when the end condition is not satisfied, the determination is negative, and the imaging process transitions to step ST224. In step ST222, when the end condition is satisfied, the determination is positive, and the distance measurement imaging process ends.

[0184] In step ST224, the execution unit 15A5 causes the first distance measurement system processing circuit 34B to perform the first distance measurement. Here, the first distance measurement is, for example, the same process as the processes of steps ST100 to ST110. After performing the process of step ST224, the distance measurement imaging process transitions to step ST218.

[0185] In Figure 24CIn step ST226 shown above, the execution unit 15A5 determines whether the imaging start timing has been reached. In step ST226, when the imaging start timing has not been reached, the determination is negative, and the distance measurement imaging process transitions to step ST232. In step ST226, when the imaging start timing has been reached, the determination is positive, and the distance measurement imaging process transitions to step ST228.

[0186] In step ST228, the execution unit 15A5 causes the second distance measurement system processing circuit 34C to perform second distance measurement by executing the second distance measurement start process and the second distance acquisition process, and acquires the second distance, which is the distance measurement result based on the second distance measurement. After the process of step ST228 is executed, the distance measurement imaging process transitions to step ST230.

[0187] In step ST230, the execution unit 15A5 executes the second imaging process using the second distance obtained by performing the second distance measurement in step ST228. Further, the execution unit 15A5 executes the second visible light image display process using the latest second visible light image obtained by executing the second imaging process. After the process of step ST230 is executed, the distance measurement imaging process transitions to step ST232.

[0188] In step ST232, the execution unit 15A5 determines whether the end condition is satisfied. In step ST232, when the end condition is not satisfied, the determination is negative, and the imaging process transitions to step ST226. In step ST232, when the end condition is satisfied, the determination is positive, and the distance measurement imaging process ends.

[0189] As described above, the first distance measurement system process (refer to Figure 23 ) and the distance measurement imaging process (refer to Figures 24A - 24C ) are executed in the intelligent device 10. That is, the first distance is measured based on the timing (the first light reception timing for distance measurement) at which the IR pixel receives the following IR reflected light and the irradiation timing, where the IR reflected light is represented by the IR pixel data selected according to the relationship between the multiple IR pixel data and the intensities of the multiple IR reflected lights represented by the multiple IR pixel data among the multiple IR pixel data generated by dividing the area 26N2 by the first distance measurement system at multiple light reception timings (for example, the "time interval" shown in Figure 14 ) within the light reception period corresponding to the irradiation of the laser beam from the light irradiator 16. Therefore, according to this configuration, it is possible to avoid erroneous distance measurement caused by the reflected light from a shiny object that is not desired to be the distance measurement target (in the example shown in Figure 12 , the mirror surface 100A).

[0190] Also, in the smart device 10, as the light reception timing for the first distance measurement, the light reception timing of the IR reflected light related to the IR pixel data with the second largest signal level among the multiple IR pixel data is adopted. Therefore, according to this configuration, compared with the case where the light reception timing of the IR reflected light related to the IR pixel data with the largest signal level among the multiple IR pixel data is adopted as the light reception timing for the first distance measurement, it is possible to accurately avoid false distance measurement caused by the reflected light from a glossy object that is not desired to be the object of distance measurement.

[0191] Also, in the smart device 10, when the intensity above the first threshold and the intensity less than the first threshold and above the second threshold are included in the time series distribution, the specific process is executed by the execution unit 15A5. Therefore, according to this configuration, it helps to solve many problems when obtaining the distance measurement result based on the IR reflected light from a glossy object that the user does not desire to be the object of distance measurement and the distance measurement result based on the IR reflected light from an object that the user desires to be the object of distance measurement ( Figure 12 In the example shown, the object subject 98).

[0192] Also, in the smart device 10, as the first threshold, a value that decreases as the light reception timing is delayed is adopted. Therefore, according to this configuration, compared with the case where the first threshold is fixed regardless of the light reception timing, it is possible to accurately determine the distance measurement result based on the reflected light from a glossy object.

[0193] Also, in the smart device 10, a strong reflected light notification message is displayed on the image selection screen presented to the user. Therefore, according to this configuration, it is possible to make the user aware of the situation where the IR reflected light from a glossy object that the user does not desire to be the object of distance measurement is received.

[0194] Also, in the smart device 10, the second distance measurement is performed as the specific process (refer to Figure 24A Step ST208). Therefore, according to this configuration, compared with the case where the first distance measurement is still performed although the time series distribution generated by the time series distribution generation unit 34B2 is a specific time series distribution, it is possible to accurately measure the distance to the object of distance measurement desired by the user ( Figure 12 In the example shown, the object subject 98).

[0195] Also, in the smart device 10, active mode focusing control is performed based on the first distance obtained through the first distance measurement. Therefore, according to this configuration, it is possible to meet the needs of users who prefer active mode focusing control more than passive mode focusing control.

[0196] Also, in the intelligent device 10, for a plurality of IR pixels included in the divided area 26N2 of the first ranging system, the first distance is measured according to the irradiation timing and the first light receiving timing for ranging. Therefore, according to this configuration, compared with the case where the first distance is measured only for a single IR pixel according to the irradiation timing and the first light receiving timing for ranging, the first distance can be measured over a large range. Also, a distance image can be generated.

[0197] Also, in the intelligent device 10, an image selection screen including a first visible light image obtained by imaging under focus control based on the first distance and a second visible light image obtained by imaging under focus control based on the second distance is displayed on the display 46. Then, a message prompting the user to select the first visible light image or the second visible light image is displayed on the image selection screen. Therefore, according to this configuration, compared with the case where there is no room to select the first visible light image and the second visible light image, it is possible to help improve usability.

[0198] In addition, in the above first embodiment, the execution unit 15A5 prompts the user to select the first visible light image or the second visible light image through the image selection screen, but the technology of the present invention is not limited thereto, and both the first visible light image and the second visible light image can also be selected. At this time, for example, the first imaging process and the second imaging process can be alternately performed, and the first visible light image and the second visible light image obtained by performing each imaging process are respectively displayed on the display 46 as an instant preview image or a still image, etc.

[0199] Also, in the above first embodiment, an example of the method in which the user selects the first visible light image or the second visible light image from the image selection screen via the touch panel 48 has been described, but the technology of the present invention is not limited thereto. For example, when the first distance and the second distance are different, the user can also be made to select in advance which one of the ranging using the active method and the ranging using the passive method to be preferentially used. The ranging using the active method is imaging accompanied by focus control based on the ranging result obtained by the first ranging, and the ranging using the passive method is imaging accompanied by focus control based on the ranging result obtained by the second ranging.

[0200] Also, in the above first embodiment, a laser beam is exemplified as the ranging light irradiated by the light irradiator 16, but the technology of the present invention is not limited thereto. For example, the ranging light can be a directional light such as super-radiant light, light emitted from a xenon flash light source, or light emitted from an LED.

[0201] Also, in the above-described first embodiment, the visible light image division region 26N1, the first distance measurement system division region 26N2, and the second distance measurement system division region 26N3 are integrated into a single chip by the photoelectric conversion element 26. However, the technology of the present invention is not limited thereto. Multiple visible light pixels may be integrated into a single chip, multiple phase difference pixels may be integrated into a single chip, and multiple IR pixels may be integrated into a single chip. Also, multiple visible light pixels and multiple phase difference pixels may be integrated into a single chip, and multiple IR pixels may be integrated into a single chip. Thus, when various photosensitive pixels are integrated into chips by type and mounted on the smart device 10, optical systems such as an objective lens, a focusing lens, and an aperture may be provided for each chip on the subject side (object side).

[0202] Also, in the above-described first embodiment, an example in which the distance measurement imaging device 14 is built into the smart device 10 has been described. However, the technology of the present invention is not limited thereto. For example, the distance measurement imaging device 14 may be externally mounted on a normal smart device, i.e., a smart device that does not have the distance measurement imaging device 14 built therein.

[0203] Also, in the above-described first embodiment, an example in which the UI system device 44 is assembled into the smart device 10 has been described. However, at least a part of the multiple components included in the UI device 44 may be mounted outside the smart device 10. Also, at least a part of the multiple components included in the UI system device 44 may be used as independent individuals by connecting to the external I / F 52.

[0204] Also, in Figure 1 the example shown, the smart device 10 has been illustrated. However, the technology of the present invention is not limited thereto. That is, the technology of the present invention can also be applied to various electronic devices (e.g., interchangeable lens cameras, fixed lens cameras, personal computers, and / or wearable terminal devices, etc.) having the distance measurement imaging device 14 built therein. Even with these electronic devices, the same functions and effects as those of the smart device 10 can be obtained.

[0205] Also, in the above-described first embodiment, the display 46 has been illustrated. However, the technology of the present invention is not limited thereto. For example, a display provided outside the smart device 10 may be used in combination with the display 46.

[0206] Also, in the above-described first embodiment, the photoelectric conversion element 26 and the signal processing circuit 34 are provided separately. However, a stacked imaging element in which the photoelectric conversion element 26 and the signal processing circuit 34 are integrated into a single chip may be used. Also, at least a part of the signal processing circuit 34 may be removed, and the CPU 15A may perform the functions of the signal processing circuit 34.

[0207] Also, in the above-described first embodiment, an example of a method of supplying a shooting timing signal from the photoelectric conversion element driver 32 to the photoelectric conversion element 26 has been described, but the technology of the present invention is not limited thereto. For example, the photoelectric conversion element driver 32 can be removed, and in this case, the CPU 15A can be made to perform the function of the photoelectric conversion element driver 32.

[0208] Also, in the above-described first embodiment, as an example of "image-using ranging" related to the technology of the present invention, the second ranging, that is, phase difference image-using ranging for measuring a distance based on a phase difference image obtained from phase difference pixels, has been exemplified, but the technology of the present invention is not limited thereto. For example, instead of ranging using phase difference pixels, ranging using a stereo camera, that is, stereo image-using ranging for measuring a distance based on a stereo image, can be performed, or ranging using face detection, that is, object image-using ranging for measuring a distance based on an object image showing an object of a known size, can be performed. In ranging using a stereo camera, the distance to the subject is measured by using the parallax of a pair of images obtained from the stereo camera (stereo image obtained by stereo shooting). Also, in ranging using face detection, the distance to the subject is measured by using, for example, the ratio of the size of the detected face image to the size of the image for one frame. Here, the size of the face image has been exemplified, but the technology of the present invention is not limited thereto, and any object image showing an object of a known size (for example, a specific car) may be used. Even in the case of using stereo image-using ranging and / or object image-using ranging in this way, the same effects as those obtained in the case of using phase difference image-using ranging can be obtained.

[0209] Also, in the above-described first embodiment, an example of a method in which the G filter, R filter, and B filter also have a function as an infrared cut-off filter for cutting off infrared light has been described, but the technology of the present invention is not limited thereto, and each color filter corresponding to the R pixel, G pixel, and B pixel can be made a color filter that also transmits infrared light, and a pair of photodiodes based on a visible light pixel photodiode and an IR pixel photodiode (for example, InGaAs APD) can be arranged for one color filter.

[0210] Further, in the above-described first embodiment, an example in which the visible light image is divided by the divided area 26N1 and the second distance measurement system by the divided area 26N3 has been described, but the technology of the present invention is not limited thereto. For example, instead of the visible light image divided by the divided area 26N1 and the second distance measurement system divided by the divided area 26N3, an area sensor that selectively generates and reads visible light pixel data and phase difference pixel data may be provided. At this time, a plurality of photosensitive pixels are arranged two-dimensionally in the area sensor. As the photosensitive pixels included in the area sensor, for example, a pair of independent photodiodes without a light shielding member are used. When generating and reading visible light pixel data, photoelectric conversion is performed through the entire area of the photosensitive pixel (a pair of photodiodes), and when generating and reading phase difference pixel data (for example, when performing passive distance measurement), photoelectric conversion is performed through one of the pair of photodiodes. Here, one of the pair of photodiodes is a photodiode corresponding to the first phase difference pixel L described in the above-described first embodiment, and the other photodiode of the pair of photodiodes is a photodiode corresponding to the second phase difference pixel R described in the above-described first embodiment. Further, it may be configured to selectively generate and read visible light pixel data and phase difference pixel data through all the photosensitive pixels included in the area sensor, but it is not limited thereto, and it may be configured to selectively generate and read visible light pixel data and phase difference pixel data through a part of the photosensitive pixels included in the area sensor.

[0211] Further, in the above-described first embodiment, an example in which the timing at which the second largest intensity IR reflected light in the time series distribution is received by a specific IR pixel is used as the first distance measurement light receiving timing has been described, but the technology of the present invention is not limited thereto. For example, the light receiving timing of the IR reflected light whose intensity exceeds the first threshold ( Figure 16 in the example shown, specular IR reflected light) and the noise light whose intensity is less than the second threshold may not be used as the first distance measurement light receiving timing, but the IR reflected light whose intensity is less than the first threshold and is equal to or more than the second threshold ( Figure 16 in the example shown, object subject IR reflected light) may be used as the first distance measurement light receiving timing. At this time, similar to the above-described first embodiment, the first distance measurement based on the light receiving timing of the specular IR reflected light and the first distance measurement based on the noise light are avoided.

[0212] Further, in the above-described first embodiment, an example in which Figure 23 is performed as shown after the first distance measurement Figure 24AThe method example of the second distance measurement in step ST208 has been described, but the technology of the present invention is not limited thereto. For example, the CPU 15A may perform the first distance measurement and the second distance measurement in parallel, and perform a process including a process based on the distance measurement result of the second distance measurement as a specific process. Here, as a process based on the distance measurement result of the second distance measurement, for example, it is possible to cite performing imaging with focus control accompanying the second distance obtained by performing the second distance measurement and / or a process of causing the display 46 to display the second distance. In this way, by performing the first distance measurement and the second distance measurement in parallel and performing a process including a process based on the distance measurement result of the second distance measurement as a specific process, compared with the case where the second distance measurement is performed after the first distance measurement, it is possible to perform the process of the second distance, which is the distance measurement result based on the second distance measurement, earlier.

[0213] Moreover, in the above first embodiment, the first threshold value and the second threshold value are set without considering the noise component based on ambient light, but the technology of the present invention is not limited thereto, and the first threshold value and the second threshold value may also be set according to the noise component based on ambient light. At this time, for example, as the upper limit value of the intensity of the IR light included in the ambient light, the first threshold value and the second threshold value may be set so as to be higher than the value previously derived through experiments based on the actual machine and / or computer simulations, etc. Therefore, according to this configuration, compared with the case where the first threshold value and the second threshold value are set without considering the noise component based on ambient light, it is possible to avoid the situation where the noise component based on ambient light is erroneously detected as the IR reflected light of the object to be photographed.

[0214] [Second Embodiment]

[0215] In the above first embodiment, a method example of performing the second imaging process based on the distance measurement result of the second distance measurement, which is image-using distance measurement, has been described, but in this second embodiment, a case of performing imaging accompanying contrast AF instead of the second imaging process will be described. In addition, in this second embodiment, the same reference numerals are assigned to the constituent elements that are the same as those described in the above first embodiment, and the description thereof is omitted, and the parts different from the above first embodiment will be described.

[0216] As an example, as Figure 25 shown, in the smart device 500 according to this second embodiment, a distance measurement imaging process program 570 is stored in the storage device 15B. The CPU 15A reads out the distance measurement imaging process program 570 from the storage device 15B. Then, the CPU 15A executes the distance measurement imaging process program 570 read out from the storage device 15B, and thus operates as the first distance measurement control unit 15A1, the first distance acquisition unit 15A2, the time series distribution acquisition unit 15A3, the determination unit 15A4, the execution unit 15A5, the focus position calculation unit 15A6, and the contrast AF method imaging control unit 15A7.

[0217] As an example, as Figure 26 shown, when the contrast AF mode imaging control unit 15A7 performs imaging in the contrast AF mode using the light receiver 18, it outputs an imaging start signal to the light receiver 18 and the focus position calculation unit 15A6. By outputting the imaging start signal to the light receiver 18, the contrast AF mode imaging control unit 15A7 causes the visible light image divided area 26N1 (refer to Figure 5 ) to perform imaging at a prescribed frame rate (e.g., 60 fps), and causes the visible light pixel data processing circuit 34A to generate a third visible light image representing the imaging area. The visible light pixel data processing circuit 34A outputs the third visible light image to the image memory 42. In the image memory 42, the third visible light image is stored, and each time the third visible light image is input from the visible light pixel data processing circuit 34A, the third visible light image in the image memory 42 is updated. In addition, the third visible light image is an example of the "subject image" related to the technology of the present invention.

[0218] Moreover, in parallel with the output of the imaging start signal, the contrast AF mode imaging control unit 15A7 outputs a motor control signal to the light receiver 18 and the focus position calculation unit 15A6. The motor control signal is a signal for causing the focusing lens 30B to reciprocate (oscillate) along the optical axis L2, and is input to the motor driver 64 of the light receiver 18. The motor driver 64 causes the focusing lens 30B to oscillate by driving the motor 62 according to the input motor control signal (refer to Figure 8 ). On the other hand, the focus position calculation unit 15A6 calculates the current position of the focusing lens 30B on the optical axis L2 using the motor control signal input from the time when the imaging start signal is input from the contrast AF mode imaging control unit 15A7 until the current time point.

[0219] Each time the third visible light image in the image memory 42 is updated, the contrast AF mode imaging control unit 15A7 acquires the third visible light image from the image memory 42 and calculates the contrast value of the acquired third visible light image. Then, the contrast AF mode imaging control unit 15A7 searches for the maximum value of the contrast of the third visible light image, and at the time when the maximum value is found, outputs a maximum value reached signal to the focus position calculation unit 15A6. The maximum value reached signal is a signal indicating that the contrast of the third visible light image has reached the maximum value.

[0220] When the maximum value reached signal is input, the focus position calculation unit 15A6 calculates the current position of the focusing lens 30B on the optical axis L2 as the focus position using the motor control signal input from the time when the imaging start signal is input until the current time point. The focus position refers to the position where the subject included in the imaging area is in focus ( Figure 26In the example shown, the focus position of the object subject 98). Here, the focus position calculation unit 15A6 calculates the focus position using an arithmetic expression with the motor control signal as the independent variable and the focus position as the dependent variable. Additionally, the technology of the present invention is not limited to this, and the focus position calculation unit 15A6 can also derive the focus position using a table that establishes a correspondence between the time series data of the motor control signal and the position of the focusing lens 30B on the optical axis L2.

[0221] As an example, as Figure 27 shown, compared with the above-described first embodiment, the difference in the specific process is that it includes a third imaging process instead of the second imaging process. The third imaging process refers to a process of performing imaging (imaging accompanied by contrast AF) with focus control for the focus position. Specifically, in the third imaging process, the execution unit 15A5 instructs the contrast AF mode imaging control unit 15A7 to perform imaging accompanied by contrast AF, and the contrast AF mode imaging control unit 15A7 causes the light receiver 18 to perform imaging accompanied by contrast AF according to the instruction from the execution unit 15A5.

[0222] Moreover, the image selection screen generated and displayed on the display 46 by executing the image selection screen display process included in the specific process by the execution unit 15A5 is different in part of the display content compared with the above-described first embodiment. That is, in the image selection screen, instead of the second distance, the focus position calculated by the focus position calculation unit 15A6 is displayed ( Figure 27 in the example shown, the numerical value "Xmm" representing the distance from the reference position to the focusing lens 30B). The reference position is, for example, the position of the imaging surface of the photoelectric conversion element 26 or the position of the focusing lens 30B in the state of infinite focus.

[0223] And, in Figure 27 the example shown, the third visible light image is displayed instead of the second visible light image in the image selection screen. And, in the image selection screen, the message "active AF mode" is displayed instead of the message "active mode". Moreover, the message "contrast AF mode" is displayed instead of the message "passive mode" in the image selection screen. Additionally, in Figure 27 the example shown, the first distance "1.8m" is displayed in correspondence with the first visible light image, but the technology of the present invention is not limited to this, and instead of the first distance or together with the first distance, the focus position derived from the first distance, that is, the distance from the reference position to the focusing lens 30B, can also be displayed.

[0224] In addition, in the second embodiment, display examples of the "active AF mode" and the "contrast AF mode" are shown. However, it is not necessary to display the "active AF mode" and the "passive AF mode". As long as the user can understand the difference in AF modes, any display is acceptable. For example, when performing AF using laser rangefinding and contrast AF, it can be a display of "AF using laser rangefinding" and "AF using contrast", or an icon indicating the AF mode can be displayed. Also, the focus position can be displayed instead of the display of the AF mode. For example, it can be "Focus position: near" and "Focus position: far", or it can be displayed as "Focus position: object" and "Focus position: image of the object", etc. Or, two or more of the characters and icons indicating the AF mode and the focus position can be combined for display.

[0225] As described above, in the intelligent device 500, as a specific process, the execution unit 15A5 performs imaging (the third imaging process) accompanied by contrast AF. Therefore, according to this structure, when the time-series distribution generated by the time-series distribution generation unit 34B2 is a specific time-series distribution, imaging accompanied by contrast AF is performed, so that the needs of users who prefer the contrast AF mode to the active AF mode can be satisfied.

[0226] In addition, in each of the above embodiments, the imaging area included within the field of view angle θ1 is used as the ranging object. However, the technology of the present invention is not limited thereto. For example, the imaging area as the ranging object can also be a specific real space area defined according to an instruction issued by the user via the receiving device 47. Thereby, ranging can be performed on the real space area desired by the user. Here, an example of the method of defining the imaging area according to an instruction issued by the user via the receiving device 47 is given, but the technology of the present invention is not limited thereto. For example, the imaging area can also be defined according to an instruction issued by an external device (not shown) capable of communicating with the intelligent device 10.

[0227] Also, according to an instruction issued by the user or an external device, the CPU 15A can selectively define the ranging object area for the first ranging and the ranging object area for the second ranging. As a method of defining the ranging object area for the first ranging, a method of adjusting the irradiation angle θ2 and / or a method of defining the range of IR pixels used within the first ranging system division area 26N2 can be cited. And, as a method of defining the ranging object area for the second ranging, a method of adjusting the field of view angle θ1 and / or a method of defining the range of phase difference pixels used within the second ranging system division area 26N3 can be cited.

[0228] Moreover, the imaging area to be distance-measured can be a specific real-space area corresponding to the object subject image detected from the visible light image by the CPU 15A (for example, an image representing the object subject 98). At this time, for example, the first distance measurement and / or the second distance measurement can be performed on the real-space area corresponding to the object subject image detected by performing image recognition processing using machine learning (for example, deep learning). Moreover, the object subject image can be an image representing the face of a specific person, or an image representing a vehicle such as a car or an aircraft, as long as it is an image representing an object.

[0229] In this way, by performing distance measurement on the specific real-space area corresponding to the object subject image detected from the visible light image by the CPU 15A, it is possible to easily perform distance measurement on the object subject as compared with the case where the user visually locates the object subject and performs distance measurement on the object subject.

[0230] Moreover, in each of the above-described embodiments, as an example, in the Figure 14 shown time series distribution, the light reception timing of the specular IR reflected light is earlier than the light reception timing of the object subject IR reflected light, but it is not limited thereto. For example, as shown in Figure 28 , even when the light reception timing of the object subject IR reflected light is earlier than the light reception timing of the specular IR reflected light, the technology of the present invention is applicable. Even in this case, the light reception timing of the IR reflected light having the second largest intensity in the time series distribution is used as the light reception timing for the first distance measurement.

[0231] Moreover, in each of the above-described embodiments, the determination of whether the time series distribution is a specific time series distribution is performed by using the first threshold value and the second threshold value (refer to Figure 16 ), but the technology of the present invention is not limited thereto. For example, the intensity of the time series distribution can be adjusted according to the light reception timing. At this time, as an example, as shown in Figure 29 , by applying a filter to the time series distribution (in the example shown in Figure 29 , a filter configured to gradually increase the intensity as the light reception timing is delayed), as an example, as shown in Figure 30 , the time series distribution is adjusted. Thereby, as compared with the case where the time series distribution is affected by the light reception timing, it is possible to accurately determine the distance measurement result caused by the material of the object to be distance-measured from the time series distribution.

[0232] In addition, when filtering the time-series distribution in this way, the first threshold value and the second threshold value applicable to the time-series distribution after filtering change according to the filter. Along with this, the method for determining whether the time-series distribution conforms to the specific time-series distribution also changes. For example, in the first embodiment described above, the magnitude relationship between the first threshold value and the second threshold value is "first threshold value < second threshold value". If these first threshold value and second threshold value are applied to Figure 30 the time-series distribution shown, and if the time-series distribution includes an intensity greater than the second threshold value ( Figure 30 in the example shown, the IR reflected light of the object subject), and includes an intensity less than the second threshold value and equal to or greater than the first threshold value ( Figure 30 in the example shown, the specular IR reflected light), the determination unit 15A4 determines that the time-series distribution after filtering is a specific time-series distribution.

[0233] In addition, the degree of eliminating the influence of the light reception timing can also be dispersed by the filter (refer to Figure 29 ). For example, half of the influence of the light reception timing can be eliminated by the filter, and half of the influence of the light reception timing can be eliminated by the first threshold value.

[0234] In the above first embodiment, an example in which the distance measurement imaging processing program 70 is stored in the storage device 15B has been described. In the above second embodiment, an example in which the distance measurement imaging processing program 570 is stored in the storage device 15B has been described. However, the technology of the present invention is not limited thereto. For example, as Figure 31 shown, the distance measurement imaging processing program 70 or 570 (hereinafter, when it is not necessary to distinguish between the distance measurement imaging processing program 70 and 570 for description, it is referred to as the "distance measurement imaging processing program" without symbols) may be stored in the storage medium 900. As an example of the storage medium 900, any portable storage medium such as an SSD or a USB memory can be cited.

[0235] The distance measurement imaging processing program stored in the storage medium 900 is installed in the controller 15. The CPU 15A executes the distance measurement imaging processing according to the distance measurement imaging processing program.

[0236] In addition, the distance measurement imaging processing program may be stored in the storage unit of another computer or server device connected to the controller 15 via a communication network (not shown), and the distance measurement imaging processing program may be downloaded and installed in the controller 15 according to the request of the intelligent device 10.

[0237] In addition, it is not necessary to store all of the distance measurement imaging processing program in the storage unit or storage device 15B of another computer or server device connected to the controller 15. A part of the distance measurement imaging processing program may be stored.

[0238] In Figure 31 the example shown, an example of the manner in which the controller 15 is built into the smart device 10 is shown, but the technology of the present invention is not limited thereto. For example, the controller 15 may also be provided outside the smart device 10.

[0239] In Figure 31 the example shown, the CPU 15A is a single CPU, but it may also be multiple CPUs. Further, a GPU may be used instead of the CPU 15A.

[0240] In Figure 31 the example shown, the controller 15 is illustrated, but the technology of the present invention is not limited thereto. A device including an ASIC, an FPGA, and / or a PLD may be used instead of the controller 15. Further, a combination of a hardware structure and a software structure may be used instead of the controller 15.

[0241] As the hardware resources for executing the distance measurement imaging process described in the above embodiments, various processors shown below can be used. As the processor, for example, a general-purpose processor, i.e., a CPU, which functions as a hardware resource for executing the distance measurement imaging process by executing software, i.e., a program, can be cited. Further, as the processor, for example, a dedicated circuit such as an FPGA, a PLD, or an ASIC, which is a processor having a circuit structure specifically designed for executing a specific process, can be cited. A memory is built in or connected to any processor, and any processor executes the distance measurement imaging process by using the memory.

[0242] The hardware resources for executing the distance measurement imaging process may be constituted by one of these various processors, or may be constituted by a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Further, the hardware resources for executing the distance measurement imaging process may also be one processor.

[0243] As an example of being constituted by one processor, first, there is a manner in which one or more CPUs and software are combined to constitute one processor, and this processor functions as a hardware resource for executing the distance measurement imaging process. Second, there is a manner such as represented by an SoC, in which a processor that uses one IC chip to implement the entire system including multiple hardware resources for executing the distance measurement imaging process is used. Thus, the distance measurement imaging process is implemented by using one or more of the above various processors as hardware resources.

[0244] Moreover, as the hardware structure of these various processors, more specifically, a circuit combining circuit elements such as semiconductor elements can be used. Further, the above distance measurement imaging process is merely an example. Therefore, of course, unnecessary steps can be deleted, new steps can be added, or the processing order can be replaced without departing from the gist.

[0245] The description and illustrations shown above are detailed descriptions of parts related to the technology of the present invention and are merely examples of the technology of the present invention. For example, the descriptions related to the above structures, functions, actions, and effects are descriptions related to examples of the structures, functions, actions, and effects of parts related to the technology of the present invention. Therefore, it is undoubtedly possible to delete unnecessary parts, add new elements, or make replacements to the description and illustrations shown above without departing from the technical spirit of the present invention. Also, in order to avoid complication and facilitate understanding of the parts related to the technology of the present invention, descriptions related to common technical knowledge that do not require special explanation in the implementation of the technology of the present invention are omitted from the description and illustrations shown above.

[0246] In this specification, the meaning of "A and / or B" is the same as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters combined with "and / or", the same concept as "A and / or B" is applicable.

[0247] Regarding all the documents, patent applications, and technical specifications described in this specification, they are incorporated into this specification by reference in the same manner as when each document, patent application, and technical specification incorporated by reference is specifically and separately described.

[0248] Regarding the above embodiments, the following appendices are also disclosed.

[0249] (Appendix 1)

[0250] An information processing device, comprising:

[0251] a processor; and

[0252] a memory connected to or built into the above processor,

[0253] The processor performs the following processing:

[0254] When the number of times the reflected light is received by the light receiver is 1 in the time series distribution, the imaging device is focused by using the first distance measured based on the irradiation timing and the light reception timing. When the number of times the reflected light is received by the light receiver is multiple in the time series distribution, the imaging device is focused by using the second distance measured from the image obtained by photographing the ranging object area. In the time series distribution, the intensity of the reflected light received by the light receiver at regular time intervals within the light reception period corresponding to the surface irradiation of the light based on the light irradiator is specified in time series.

[0255] (Appendix 2)

[0256] A ranging device, comprising:

[0257] A processor;

[0258] A light irradiator that performs surface irradiation of light toward a ranging target area; and

[0259] A light receiver that receives reflected light from the ranging target area,

[0260] The processor performs the following processing:

[0261] Measure a first distance to an object in the ranging target area based on the irradiation timing of the surface irradiation of light by the light irradiator toward the ranging target area and the light reception timing of the reflected light from the ranging target area received by the light receiver,

[0262] Based on the light reception timing and irradiation timing corresponding to a signal selected according to the relationship and intensity of a plurality of signals generated by the light receiver at a plurality of light reception timings during the light reception period corresponding to the surface irradiation by the light irradiator, the processor measures the first distance.

[0263] (Appendix 3)

[0264] An information processing device, comprising:

[0265] A processor; and

[0266] A memory connected to or built into the above-mentioned processor,

[0267] The processor performs the following processing:

[0268] Measure a first distance to an object in the ranging target area based on the irradiation timing of the surface irradiation of light by the light irradiator toward the ranging target area and the light reception timing of the reflected light from the ranging target area received by the light receiver,

[0269] Based on the light reception timing and irradiation timing corresponding to a signal selected according to the relationship and intensity of a plurality of signals generated by the light receiver at a plurality of light reception timings during the light reception period corresponding to the surface irradiation by the light irradiator, the processor measures the first distance.

[0270] Symbol Explanation

[0271] 10. 500 - Intelligent device, 12 - housing, 12A - back surface, 12B - front surface, 13 - indicating key, 14 - distance measuring camera device, 15 - controller, 15A - CPU, 15A1 - First distance measuring control unit, 15A2 - First distance acquisition unit, 15A3 - Time - series distribution acquisition unit, 15A4 - Determination unit, 15A5 - Execution unit, 15A6 - Focus position calculation unit, 15A7 - Contrast AF mode camera control unit, 15A8 - Focus position calculation unit, 15B - Storage device, 15C - Memory, 16 - Light irradiator, 17A, 17B - Light - shielding members, 18 - Light receiver, 19 - Microlens, 20, 22, 352 - Translucent window, 21 - Beam expander, 23 - Collimating lens, 24 - LD, 25 - LD driver, 26, 354 - Photoelectric conversion element, 26N1 - Division area for visible - light image, 26N2 - Division area for first distance - measuring system, 26N3 - Division area for second distance - measuring system, 27 - TOF camera, 30A - Objective lens, 30B - Focusing lens, 30C - Aperture, 31 - Focus control mechanism, 32 - Photoelectric conversion element driver, 34 - Signal processing circuit, 34A - Visible - light pixel data processing circuit, 34B - First distance - measuring system processing circuit, 34B1 - IR pixel data acquisition unit, 34B2 - Time - series distribution generation unit, 34B3 - Light - receiving timing determination unit, 34B4 - First distance measurement unit, 34C - Second distance - measuring system processing circuit, 40 - Input / output interface, 41 - Imaging lens, 42 - Image memory, 44 - UI system device, 46 - Display, 47 - Receiving device, 48 - Touch panel, 50 - Bus, 52 - External I / F, 53 - Hard key section, 54 - Communication I / F, 56 - Network, 59 - Touch panel display, 60 - Moving mechanism, 62 - Motor, 64 - Motor driver, 70, 570 - Distance - measuring camera processing program, 72 - Focus position derivation table, 100 - Mirror, 100A - Mirror surface, 300L - Left - area passing light, 300R - Right - area passing light, 900 - Storage medium, L - First phase - difference pixel, L1, L2 - Optical axes, N - Non - phase - difference pixel, PD - Photoelectric diode, R - Second phase - difference pixel, α - Offset, θ1 - Field of view angle, θ2 - Irradiation angle.

Claims

1. An information processing apparatus, comprising: a processor; and a memory connected to or built in the processor, wherein the processor performs the following processing: measuring a first distance to an object within the distance measurement target area based on an irradiation timing at which light is surface-irradiated onto the distance measurement target area by a light irradiator and a light reception timing at which the reflected light from the distance measurement target area is received by a light receiver; when, during a light reception period corresponding to the surface irradiation, a plurality of signals are generated by a specific pixel included in the light receiver at different timings, the processor measures the first distance based on a relationship of the plurality of signals generated by the specific pixel and including a first signal having a first intensity and a second signal having a second intensity, the first intensity being equal to or greater than a first threshold value, the second intensity being less than the first threshold value and equal to or greater than a second threshold value smaller than the first threshold value; the first threshold value is a value defined as a lower limit value of the intensity of the reflected light from a glossy object; the second threshold value is a value defined as a lower limit value of the intensity of the reflected light from a standard subject assumed to be the object and being a standard subject other than a specular surface and a glossy surface.

2. The information processing apparatus according to claim 1, wherein the relationship of the plurality of signals is a time series distribution of the intensities of the plurality of signals.

3. The information processing apparatus according to claim 1, wherein when a plurality of signals having intensities exceeding a reference threshold value are included in the plurality of signals, the processor measures the first distance based on the light reception timing at which the light receiver receives the reflected light corresponding to the signal having the second largest intensity among the plurality of signals.

4. The information processing apparatus according to any one of claims 1 to 3, wherein when the first signal and the second signal are included in the plurality of signals, the processor performs a specific process.

5. The information processing apparatus according to claim 4, wherein the first threshold value and the second threshold value are defined based on a noise component of ambient light.

6. The information processing apparatus according to claim 4, wherein the first threshold value is a value that decreases as the light reception timing is delayed.

7. The information processing apparatus according to claim 4, wherein the specific process is a process of measuring the first distance based on the light reception timing at which the specific pixel receives the reflected light corresponding to the second signal.

8. The information processing apparatus according to claim 4, wherein the specific process includes a process of notifying that the first intensity is included in the intensities of the plurality of signals.

9. The information processing apparatus according to claim 4, wherein the specific process includes image-based distance measurement of measuring a second distance to the object based on an image obtained by photographing the distance measurement target area with a first image pickup device.

10. The information processing apparatus according to claim 4, wherein The processor performs image-usage ranging for measuring a second distance to the object based on an image obtained by photographing the distance measurement target area with a second image pick-up device in parallel with the operation of measuring the first distance based on the irradiation timing and the light reception timing. The specific processing includes processing based on the ranging result of the image-usage ranging.

11. The information processing apparatus according to claim 9, wherein the image-usage ranging is at least one of phase-difference image-usage ranging for measuring the second distance based on a phase-difference image obtained from a phase-difference pixel as the image, stereo-image-usage ranging for measuring the second distance based on a stereo image obtained by a stereo photographing method as the image, and object-image-usage ranging for measuring the second distance based on an object image detected from the image and representing an object with a known size.

12. The information processing apparatus according to claim 4, wherein the specific processing includes focus control of the third image pick-up device based on the contrast of a subject image obtained by photographing a subject included in the distance measurement target area with the third image pick-up device.

13. The information processing apparatus according to any one of claims 1 to 3, wherein the distance measurement target area is a specific real space area defined according to a given instruction.

14. The information processing apparatus according to any one of claims 1 to 3, wherein the distance measurement target area is a specific real space area corresponding to an object subject image detected by the processor from a photographed image obtained by photographing with a fourth image pick-up device.

15. The information processing apparatus according to any one of claims 1 to 3, wherein the processor performs focus control of a fifth image pick-up device by using the first distance measured based on the irradiation timing and the light reception timing.

16. The information processing apparatus according to any one of claims 1 to 3, wherein the intensity of the plurality of signals is adjusted according to the light reception timing.

17. The information processing apparatus according to any one of claims 1 to 3, wherein the light receiver has a plurality of the specific pixels arranged in a two-dimensional shape, and the processor measures the first distance for the plurality of specific pixels based on the irradiation timing and the light reception timing.

18. An image pick-up apparatus, comprising: the information processing apparatus according to any one of claims 1 to 17; and a focusing lens, wherein the processor performs focus control to move the focusing lens to a focusing position determined based on the first distance.

19. An information processing method, comprising the following steps: A light irradiator performs surface irradiation of light toward a distance measurement target area; A light receiver receives reflected light of the light from the distance measurement target area; A first distance to an object in the distance measurement target area is measured based on an irradiation timing at which the light irradiator has performed the surface irradiation of the light toward the distance measurement target area and a light reception timing at which the light receiver has received the reflected light; and During a light receiving period corresponding to the surface illumination, when a plurality of signals are generated at different timings by specific pixels included in the light receiver, a first distance to an object within the distance measurement target area is measured based on a relationship between the plurality of signals generated by the specific pixels and including a first signal having a first intensity and a second signal having a second intensity, the first intensity being equal to or greater than a first threshold value, the second intensity being less than the first threshold value and equal to or greater than a second threshold value smaller than the first threshold value. The first threshold value is a value defined as a lower limit value of the intensity of the reflected light from a glossy object. The second threshold value is a value defined as a lower limit value of the intensity of the reflected light from a standard subject assumed to be the object and being a standard subject other than a specular surface and a glossy surface.

20. A computer-readable storage medium storing a program for causing a computer to execute a process including the following steps: A light irradiator performs surface illumination of light toward a distance measurement target area. A light receiver receives reflected light from the distance measurement target area of the light. A first distance to an object within the distance measurement target area is measured based on an irradiation timing at which the light irradiator performs the surface illumination of light toward the distance measurement target area and a light receiving timing at which the light receiver receives the reflected light. And During a light receiving period corresponding to the surface illumination, when a plurality of signals are generated at different timings by specific pixels included in the light receiver, a first distance to an object within the distance measurement target area is measured based on a relationship between the plurality of signals generated by the specific pixels and including a first signal having a first intensity and a second signal having a second intensity, the first intensity being equal to or greater than a first threshold value, the second intensity being less than the first threshold value and equal to or greater than a second threshold value smaller than the first threshold value. The first threshold value is a value defined as a lower limit value of the intensity of the reflected light from a glossy object. The second threshold value is a value defined as a lower limit value of the intensity of the reflected light from a standard subject assumed to be the object and being a standard subject other than a specular surface and a glossy surface.

Images