Information Processing Apparatus, Imaging Apparatus, Information Processing Method, and Storage Medium

Abstract

The present invention provides an information processing device, an imaging device, an information processing method, and a storage medium capable of avoiding erroneous distance measurement caused by reflected light from a glossy object that is not intended to be a distance measurement object. The information processing device includes: a processor; and a memory connected to or built in the processor, and the processor performs the following processing: a first distance to an object in a distance measurement object area is measured based on the timing of surface irradiation of light by a light irradiator toward the distance measurement object area and the timing of light reception of reflected light from the distance measurement object area by a light receiver, and when a specific pixel included in the light receiver generates a plurality of signals at different timings during the light reception period corresponding to the surface irradiation by the light irradiator, the processor measures the first distance based on the relationship between the plurality of signals.

Description

Technical Field

[0001] The technology of the present invention relates to an information processing device, a camera device, an information processing method and a storage medium. Background Art

[0002] Patent document 1 discloses a laser distance measuring device, which includes a laser emitting device and an optical system that transmits a laser beam emitted from the laser emitting device to a measurement object and focuses reflected light from the measurement object. The laser distance measuring device described in Patent document 1 includes a field of view limiting mechanism that can arbitrarily change the transmission position and size of the optical system receiving field of view within the field of view, and measures the distance to a measurement target within the measurement object passing through the position of the field of view limiting mechanism within the receiving field of view.

[0003] Patent document 2 discloses an object detection device that emits electromagnetic waves in the direction of travel of a vehicle and detects an object based on the reflected waves of the electromagnetic waves. The object detection device described in Patent document 2 includes: an emitting mechanism that emits electromagnetic waves while scanning in a scanning range in the horizontal and vertical directions; a receiving mechanism that receives the reflected waves of the electromagnetic waves; an acquiring mechanism that acquires the level of the reflected waves received by the receiving mechanism; a computing mechanism that computes a first deviation correction amount based on the vertical position of the area where the level of the reflected waves becomes maximum and the position of the center of the scanning range in the vertical direction; and a correction mechanism that corrects the center position of the scanning range in the vertical direction based on the first deviation correction amount calculated by the computing mechanism, and measures the distance to the object based on the reflected waves. Furthermore, the emission mechanism of the object detection device described in Patent Document 2 performs a first scanning process and a second scanning process. In the first scanning process, a central area centered on the center position in the vertical direction of the scanning range is scanned along a first direction which is a prescribed direction in the horizontal direction, and an area above the central area is scanned and emitted along a second direction which is a direction opposite to the first direction. In the second scanning process, the central area is scanned along the first direction, and an area below the central area is scanned and emitted along the second direction.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2003-057343

[0005] Patent Document 2: Japanese Patent Application Publication No. 2006-349694 Summary of the invention

[0006] One embodiment of the technology according to the present invention provides an information processing device, an imaging device, an information processing method, and a program that can avoid erroneous distance measurement caused by reflected light from a glossy object that is not intended to be a distance measurement target.

[0007] Means for solving technical problems

[0008] The first mode related to the technology of the present invention is an information processing apparatus, which includes: a processor; and a memory connected to or built in the processor, and the processor performs the following processing: measuring a first distance to an object in a ranging target area based on an irradiation timing at which light is surface-irradiated onto the ranging target area by a light irradiator and a light reception timing at which reflected light from the ranging target area is received by a light receiver, and measuring the first distance by the processor based on the light reception timing and the irradiation timing corresponding to a signal selected according to the relationship and intensity of a plurality of signals generated by the light receiver at a plurality of light reception timings during a light reception period corresponding to the surface irradiation by the light irradiator.

[0009] The second mode related to the technology of the present invention is the information processing apparatus according to the first mode, wherein the light reception timing used in the measurement of the first distance is the timing at which the light receiver receives reflected light related to the signal having the second largest intensity among the plurality of signals.

[0010] The third mode related to the technology of the present invention is the information processing apparatus according to the first mode or the second mode, wherein when the first intensity and the second intensity are included in the intensities of the plurality of signals, the processor executes a specific process, the first intensity being equal to or greater than a first threshold, and the second intensity being less than the first threshold and equal to or greater than a second threshold smaller than the first threshold.

[0011] The fourth mode related to the technology of the present invention is the information processing apparatus according to the third mode, wherein the first threshold and the second threshold are defined based on a noise component of ambient light.

[0012] The fifth mode related to the technology of the present invention is the information processing apparatus according to the third mode or the fourth mode, wherein the first threshold is a value that decreases as the light reception timing is delayed.

[0013] The sixth mode related to the technology of the present invention is the information processing apparatus according to any one of the third mode to the fifth mode, wherein the specific process includes a process of notifying that the first intensity is included in the intensities of the plurality of signals.

[0014] The seventh mode related to the technology of the present invention is the information processing apparatus according to any one of the third mode to the sixth mode, wherein the specific process includes image-usage ranging for measuring a second distance to an object based on an image obtained by photographing the ranging target area.

[0015] The eighth mode related to the technology of the present invention is the information processing apparatus according to any one of the third mode to the sixth mode. Among them, the processor performs image-using ranging for measuring the second distance to an object based on an image obtained by photographing a ranging object area in parallel with the operation of measuring the first distance according to the irradiation timing and the light reception timing. The specific processing includes processing based on the ranging result of the image-using ranging.

[0016] The ninth mode related to the technology of the present invention is the information processing apparatus according to the seventh mode or the eighth mode. Among them, the image-using ranging is at least one of phase-difference image-using ranging for measuring the second distance based on a phase-difference image obtained as an image from phase-difference pixels, stereo-image-using ranging for measuring the second distance based on a stereo image obtained as an image by a stereo imaging method, and object-image-using ranging for measuring the second distance based on an object image detected from an image and representing an object with a known size.

[0017] The tenth mode related to the technology of the present invention is the information processing apparatus according to any one of the third mode to the ninth mode. Among them, the specific processing includes focusing control of the imager according to the contrast of the subject image obtained by photographing a subject included in the ranging object area by the imager.

[0018] The eleventh mode related to the technology of the present invention is the information processing apparatus according to any one of the first mode to the tenth mode. Among them, the ranging object area is a specific real space area defined according to the given instruction.

[0019] The twelfth mode related to the technology of the present invention is the information processing apparatus according to any one of the first mode to the eleventh mode. Among them, the ranging object area is a specific real space area corresponding to the object subject image detected by the processor from the captured image obtained by photographing.

[0020] The thirteenth mode related to the technology of the present invention is the information processing apparatus according to any one of the first mode to the twelfth mode. Among them, the processor performs focusing control of the imager by using the first distance measured according to the irradiation timing and the light reception timing.

[0021] The fourteenth mode related to the technology of the present invention is the information processing apparatus according to any one of the first mode to the thirteenth mode. Among them, the intensity of a plurality of signals is adjusted according to the light reception timing.

[0022] The fifteenth mode related to the technology of the present invention is the information processing apparatus according to any one of the first mode to the fourteenth mode. Among them, the light receiver has a plurality of photoelectric conversion elements arranged in a two-dimensional shape, and the processor measures the first distance for the plurality of photoelectric conversion elements according to the irradiation timing and the light reception timing.

[0023] The 16th mode related to the technology of the present invention is an imaging device, which includes an information processing device and a focusing lens involved in any one of the 1st to 15th modes, and the processor performs focusing control to move the focusing lens to a focusing position determined according to the 1st distance.

[0024] The 17th mode related to the technology of the present invention is an information processing method, which includes the following steps: performing surface illumination of light by a light irradiator toward a ranging object area; receiving reflected light of the light from the ranging object area by a light receiver; and measuring the 1st distance to an object in the ranging object area according to the irradiation timing of the surface illumination of light by the light irradiator toward the ranging object area and the light reception timing of the reflected light received by the light receiver, and measuring the 1st distance by the processor according to the light reception timing and the irradiation timing corresponding to a signal selected according to the relationship and intensity of a plurality of signals generated by the light receiver at a plurality of light reception timings during the light reception period corresponding to the surface illumination based on the light irradiator.

[0025] The 18th mode related to the technology of the present invention is a program for causing a computer to execute a process including the following steps: performing surface illumination of light by a light irradiator toward a ranging object area; receiving reflected light of the light from the ranging object area by a light receiver; and measuring the 1st distance to an object in the ranging object area according to the irradiation timing of the surface illumination of light by the light irradiator toward the ranging object area and the light reception timing of the reflected light received by the light receiver, and measuring the 1st distance by the processor according to the light reception timing and the irradiation timing corresponding to a signal selected according to the relationship and intensity of a plurality of signals generated by the light receiver at a plurality of light reception timings during the light reception period corresponding to the surface illumination based on the light irradiator. Description of the Drawings

[0026] Figure 1 It is a schematic perspective view showing an example of the usage mode of the smart device related to the 1st embodiment.

[0027] Figure 2 It is a rear perspective view showing an example of the appearance of the back side of the smart device related to the 1st embodiment.

[0028] Figure 3 It is a conceptual diagram showing an example of a mode in which an imaging area is photographed by a ranging imaging device included in the smart device related to the 1st embodiment, and a visible light image is displayed on a display.

[0029] Figure 4 It is a conceptual diagram showing an example of a mode in which a laser beam is irradiated to an imaging area by a ranging imaging device included in the smart device related to the embodiment.

[0030] Figure 5 This is a schematic perspective view showing an example of the configuration of each pixel included in the photoelectric conversion element of the intelligent device according to the first embodiment.

[0031] Figure 6 This is a conceptual diagram showing an example of the incident characteristics of subject light with respect to Figure 5 the first phase difference pixel and the second phase difference pixel included in the photoelectric conversion element shown.

[0032] Figure 7 This is a schematic structural diagram showing an example of the structure of the non-phase difference pixel included in the photoelectric conversion element shown in Figure 5 the photoelectric conversion element shown.

[0033] Figure 8 This is a conceptual diagram showing an example of the manner in which a distance image based on a ranging result obtained by performing first ranging on a imaging area by a ranging imaging device included in the intelligent device according to the first embodiment is displayed on a display.

[0034] Figure 9 This is a block diagram showing an example of the structure of the electrical system hardware of the intelligent device according to the first embodiment.

[0035] Figure 10 This is a block diagram showing an example of the functions of the CPU included in the intelligent device according to the first embodiment.

[0036] Figure 11 This is a conceptual diagram showing an example of the processing content of Figure 10 the first ranging control unit shown.

[0037] Figure 12 This is a conceptual diagram showing an example of a scene of performing first ranging by the intelligent device according to the first embodiment.

[0038] Figure 13 This is a conceptual diagram showing an example of the processing content of the first ranging system processing circuit included in the intelligent device according to the first embodiment.

[0039] Figure 14 This is a conceptual diagram showing an example of the time series distribution generated by the time series distribution generation unit of the first ranging system processing circuit included in the intelligent device according to the first embodiment.

[0040] Figure 15 This is a block diagram showing an example of the processing content of Figure 10 the first distance acquisition unit, the time series distribution acquisition unit, the determination unit, and the execution unit shown.

[0041] Figure 16 This is for explaining based onFigure 15 Explanation diagram of the determination method of the determination unit shown

[0042] Figure 17 It shows the content of the first imaging process executed by Figure 15 The block diagram showing an example of the content of the first imaging process executed by the execution unit shown

[0043] Figure 18 It shows the content of the specific process executed by Figure 15 The block diagram showing an example of the content of the specific process executed by the execution unit shown

[0044] Figure 19 It shows the content of the second ranging start process and the second distance acquisition process executed by Figure 15 The block diagram showing an example of the content of the second ranging start process and the second distance acquisition process executed by the execution unit shown

[0045] Figure 20 It shows the content of the specific process executed by Figure 15 The conceptual diagram showing an example of the detailed content of the specific process executed by the execution unit shown

[0046] Figure 21 The conceptual diagram showing an example of the processing content when the user selects the first visible light image in the image selection screen displayed on the display by the intelligent device according to the first embodiment via the touch panel

[0047] Figure 22 The conceptual diagram showing an example of the processing content when the user selects the second visible light image in the image selection screen displayed on the display by the intelligent device according to the first embodiment via the touch panel

[0048] Figure 23 The flowchart showing an example of the process of the first ranging system process according to the first embodiment

[0049] Figure 24A The flowchart showing an example of the process of the ranging imaging process according to the first embodiment

[0050] Figure 24B It is Figure 24A The continuation of the flowchart shown

[0051] Figure 24C It is Figure 24A The continuation of the flowchart shown

[0052] Figure 25 The block diagram showing an example of the functions of the CPU included in the intelligent device according to the second embodiment

[0053] Figure 26 The conceptual diagram showing an example of the processing content related to the imaging in the contrast AF method performed by the intelligent device according to the second embodiment

[0054] Figure 27 It is a conceptual diagram showing an example of the details of a specific process executed by the execution unit according to the second embodiment.

[0055] Figure 28 It is a conceptual diagram showing an example of the time series distribution of a pattern different from the pattern shown in Figure 14 It is a conceptual diagram showing an example of the time series distribution of a pattern different from the pattern shown in

[0056] Figure 29 It is a conceptual diagram showing an example of a method of applying a filter to a time series distribution.

[0057] Figure 30 It is a conceptual diagram showing an example of a time series distribution adjusted by applying a filter.

[0058] Figure 31 It is a conceptual diagram showing an example of a method of installing the distance measurement imaging processing program according to the first or second embodiment in a smart device. Detailed Embodiment

[0059] Hereinafter, an example of an embodiment of a camera device related to the technology of the present invention will be described with reference to the drawings.

[0060] First, the terms used in the following description will be described.

[0061] CPU stands for "Central Processing Unit". RAM stands for "Random Access Memory". ASIC stands for "Application Specific Integrated Circuit". PLD stands for "Programmable Logic Device". FPGA stands for "Field-Programmable Gate Array". SoC stands for "System-on-a-chip". SSD stands for "Solid State Drive". USB stands for "Universal Serial Bus". HDD stands for "Hard Disk Drive". EEPROM stands for "Electrically Erasable and Programmable Read Only Memory". EL stands for "Electro-Luminescence". A / D stands for "Analog / Digital". I / F stands for "Interface". UI stands for "User Interface". LTE stands for "Long Term Evolution". 5G stands for "5th Generation". LD stands for "Laser Diode". IR stands for "Infrared". APD stands for "Avalanche Photodiode". TOF stands for "Time of Flight".

[0062] fps stands for "frame per second". LED stands for "Light Emitting Diode". ROI stands for "Region of Interest". LAN stands for "Local Area Network". AF stands for "Auto Focus". IC stands for "Integrated Circuit".

[0063] In the description of this specification, "horizontal" means horizontal in the sense of including the errors that are generally allowed in the technical field to which the technology of the present invention belongs, in addition to being completely horizontal. In the description of this specification, "parallel" means parallel in the sense of including the errors that are generally allowed in the technical field to which the technology of the present invention belongs, in addition to being completely parallel. In the description of this specification, "vertical" means vertical in the sense of including the errors that are generally allowed in the technical field to which the technology of the present invention belongs, in addition to being completely vertical. In the description of this specification, "identical" means identical in the sense of including the errors that are generally allowed in the technical field to which the technology of the present invention belongs, in addition to being completely identical.

[0064] [First Embodiment]

[0065] As an example, as Figure 1 shown, the intelligent device 10 performs a shooting operation (hereinafter, also simply referred to as "shooting operation") and a distance measurement operation on a shooting area defined by a field of view angle θ1. In this first embodiment, "distance measurement" refers to the process of measuring the distance from the intelligent device 10 to the shooting area. Here, the shooting area is an example of the "distance measurement target area" related to the technology of the present invention. And the intelligent device 10 is an example of the "information processing device" and "shooting device" related to the technology of the present invention. As the intelligent device 10, for example, a smart phone or a tablet terminal, etc., which are electronic devices with a shooting function, can be cited.

[0066] In the intelligent device 10, a first distance measurement and a second distance measurement with different distance measurement methods are performed. The first distance measurement is a method of measuring the distance to an object in the shooting area based on the timing when the intelligent device 10 irradiates a laser beam toward the shooting area and the timing when the intelligent device 10 receives the reflected light of the laser beam from the shooting area (hereinafter, also referred to as the "active method"). Here, the laser beam is an example of the "light" related to the technology of the present invention. And here, surface irradiation of the laser beam is performed toward the shooting area. In addition, surface irradiation means that the beam diameter of the laser beam toward the shooting area is more expanded than point irradiation. In surface irradiation, the beam diameter of the laser beam gradually expands along the irradiation direction of the laser beam, and the degree of expansion of the beam diameter of surface irradiation per unit time is larger than that of point irradiation. That is, in point irradiation, the laser beam is irradiated onto the surface where the shooting area is located in a dot shape, while in surface irradiation, the laser beam is irradiated onto the surface where the shooting area is located in a surface shape. And the irradiation of the laser beam can be a single shot or can be intermittently performed periodically (for example, every 0.1 second). At this time, the first distance measurement can be performed in each irradiation of the laser beam, and processing based on the distance measurement result can be performed.

[0067] The second distance measurement is a distance measurement method (hereinafter, also simply referred to as the "passive method") for measuring the distance to an object in the imaging area based on an image obtained by the smart device 10 photographing the imaging area. In addition, the second distance measurement is an example of the "image-using distance measurement" related to the technology of the present invention.

[0068] In the smart device 10, a hybrid distance measurement method that combines the active distance measurement method and the passive distance measurement method is performed. And in the smart device 10, imaging accompanied by active focusing control and imaging accompanied by passive focusing control are performed. Active focusing control refers to focusing control based on the distance measurement result obtained by performing active distance measurement. Passive focusing control refers to focusing control based on the distance measurement result obtained by performing passive distance measurement.

[0069] As an example, as Figure 2 shown, the smart device 10 includes a housing 12. A distance measurement imaging device 14 is housed in the housing 12. The distance measurement imaging device 14 includes a light irradiator 16 and a light receiver 18. The light irradiator 16 includes an LD24. In the smart device 10, the imaging operation and the distance measurement operation are performed by the distance measurement imaging device 14.

[0070] An instruction key 13 is arranged on the side surface of the smart device 10. The instruction key 13 receives various instructions. The "various instructions" mentioned here are, for example, instructions to display a menu screen that can select various menus, instructions to select one or more menus, instructions to confirm the selection content, and instructions to delete the selection content, etc.

[0071] When the smart device 10 is in the vertical state, a light transmission window 20 and 22 are provided at the upper left part of the back surface 12A of the housing 12 (the upper left part when observing the rear view of the vertically placed smart device 10). The light transmission windows 20 and 22 are optical elements with light transmissibility (for example, lenses), and are arranged at a predetermined interval (for example, an interval of several millimeters) along the horizontal direction and exposed from the back surface 12A. The light irradiator 16 irradiates a laser beam emitted from the LD24 toward the imaging area through the light transmission window 20. In this first embodiment, a laser beam in the infrared wavelength region is irradiated toward the imaging area by the light irradiator 16. However, the wavelength region of the laser beam is not limited to this, and it can also be a laser beam in other wavelength regions.

[0072] The light receiver 18 receives IR reflected light through the light transmission window 22. The IR reflected light refers to the reflected light of the laser beam irradiated on the distance measurement object by the light irradiator 16 from the distance measurement object. And the light receiver 18 takes in visible light reflected light through the light transmission window 22. The visible light reflected light refers to the reflected light of the visible light (for example, the visible light contained in sunlight) irradiated on the imaging area from the imaging area. In addition, hereinafter, for the convenience of description, when it is not necessary to distinguish between IR reflected light and visible light reflected light for description, it is simply referred to as "reflected light".

[0073] The light receiver 18 is an example of the "imaging device" related to the technology of the present invention, and includes a photoelectric conversion element 26. The photoelectric conversion element 26 receives the reflected light taken into the light receiver 18 through the light transmission window 22, and outputs an electric signal corresponding to the amount of the received reflected light.

[0074] As an example, as Figure 3 shown, a touch panel display 59 is provided on the front surface 12B of the housing 12. The touch panel display 59 includes a display 46 and a touch panel 48. As an example of the display 46, an organic EL display can be cited. The display 46 may not be an organic EL display, but may be other types of displays such as a liquid crystal display.

[0075] The display 46 displays images (for example, live view images and reproduced images) and character information, etc. The touch panel 48 is a transmissive touch panel and overlaps the surface of the display area of the display 46. The touch panel 48 receives an instruction from the user by detecting contact based on an indicator such as a finger or a stylus. In addition, here, as an example of the touch panel display 59, an external touch panel display in which the touch panel 48 overlaps the surface of the display area of the display 46 is cited, but this is only an example. For example, as the touch panel display 59, an external embedded type or an internal embedded type touch panel display can also be applied.

[0076] In the smart device 10, if an instruction to start imaging is received through the touch panel 48, the imaging area is imaged by the light receiver 18. That is, the light receiver 18 receives visible light reflected light, and generates a visible light image representing the imaging area as an image corresponding to the received visible light reflected light. The visible light image is an example of the "image", "subject image" and "captured image" related to the technology of the present invention.

[0077] The visible light image is displayed on the display 46 as a live view image or a still image according to the instruction received through the touch panel 48. In the Figure 3 shown example, the imaging area is defined by the field of view angle θ1. The field of view angle θ1 is changed according to the instruction received through the touch panel 48.

[0078] As an example, as Figure 4 shown, in the smart device 10, if an instruction to start distance measurement and imaging (hereinafter, also referred to as "distance measurement imaging start instruction") is received through the touch panel 48, a laser beam is irradiated by the light irradiator 16. The angle of the irradiated laser beam (hereinafter, also referred to as "irradiation angle") is θ2, and the irradiation angle θ2 is changed according to the instruction received through the touch panel 48. In addition, in the Figure 4In the example shown, a method example of starting distance measurement is described based on a distance measurement start instruction received via the touch panel 48 in a state where a visible light image is displayed as a live view image on the display 46. However, the technology of the present invention is not limited thereto. For example, distance measurement may also be started when a distance measurement start instruction is received via the touch panel 48 in a state where the visible light image is not displayed on the display 46.

[0079] In the smart device 10, the distance from the smart device 10 to the distance measurement object is measured based on the time required from when the laser beam is irradiated by the light irradiator 16 until the IR reflected light is received by the light receiver 18 and the speed of light. For example, if the distance to the distance measurement object is set to "L0", the speed of light is set to "c", and the flight time of the laser beam, that is, the time required from when the laser beam is irradiated by the light irradiator 16 until the IR reflected light is received by the light receiver 18 (hereinafter, also simply referred to as "flight time") is set to "t", then the distance L0 is calculated according to the formula "L0 = c × t × 0.5".

[0080] As an example, as Figure 5 shown, the photoelectric conversion element 26 has a plurality of photodiodes arranged in a matrix. As an example of the plurality of photodiodes, photodiodes with a pixel amount of "4896 × 3265" can be cited.

[0081] Color filters are arranged for each photodiode included in the photoelectric conversion element 26. The color filter includes a G filter corresponding to the G (green) wavelength region that is most helpful for obtaining a luminance signal, an R filter corresponding to the R (red) wavelength region, a B filter corresponding to the B (blue) wavelength region, and an IR filter corresponding to the IR (infrared) wavelength region. In addition, in the present embodiment, the G filter, the R filter, and the B filter also have the function of an infrared light cut-off filter for cutting off infrared light.

[0082] The photoelectric conversion element 26 is formed of two types of photosensitive pixels, namely, phase difference pixels and non-phase difference pixels N that are different from the phase difference pixels. Usually, the non-phase difference pixels N are also referred to as normal pixels. The photoelectric conversion element 26 has four types of photosensitive pixels, namely, R pixels, G pixels, B pixels, and IR pixels, as non-phase difference pixels. The R pixels, G pixels, B pixels, IR pixels, and phase difference pixels are regularly arranged in a prescribed period in the row direction (horizontal direction) and the column direction (vertical direction). The R pixels are pixels corresponding to the photodiodes arranged with the R filter, the G pixels and the phase difference pixels are pixels corresponding to the photodiodes arranged with the G filter, the B pixels are pixels corresponding to the photodiodes arranged with the B filter, and the IR pixels are pixels corresponding to the photodiodes arranged with the IR filter. As an example of the IR pixels, InGaAs APD can be cited.

[0083] In addition, hereinafter, for the sake of convenience of explanation, when it is not necessary to distinguish between the G filter, the R filter, and the B filter, they are also referred to as "visible light filters". Further, hereinafter, for the sake of convenience of explanation, when it is not necessary to distinguish between R pixels, G pixels, and B pixels, they are referred to as "visible light pixels".

[0084] A plurality of phase difference pixel lines 26A and a plurality of non-phase difference pixel lines 26B are arranged on the light-receiving surface of the photoelectric conversion element 26. The phase difference pixel line 26A is a horizontal line including phase difference pixels. Specifically, the phase difference pixel line 26A is a horizontal line in which phase difference pixels and non-phase difference pixels N are mixed. The non-phase difference pixel line 26B is a horizontal line including only a plurality of non-phase difference pixels N.

[0085] On the light-receiving surface of the photoelectric conversion element 26, the phase difference pixel line 26A and a specified number of non-phase difference pixel lines 26B are alternately arranged in the column direction. The "specified number of lines" mentioned here is, for example, 2 rows. In addition, here, 2 rows are exemplified as the specified number of lines, but the technology of the present invention is not limited thereto, and the specified number of lines may be several lines of 3 rows or more, or a dozen lines, dozens of lines, or hundreds of lines, etc.

[0086] The phase difference pixel lines 26A are arranged at 2-row intervals in the column direction from the first row to the last row. A part of the pixels of the phase difference pixel line 26A are phase difference pixels. Specifically, the phase difference pixel line 26A is a horizontal line in which phase difference pixels and non-phase difference pixels N are periodically arranged. The phase difference pixels are roughly divided into a first phase difference pixel L and a second phase difference pixel R. In the phase difference pixel line 26A, the first phase difference pixel L and the second phase difference pixel R are alternately arranged at a certain pixel interval in the row direction as G pixels.

[0087] The first phase difference pixel L and the second phase difference pixel R are arranged to appear alternately in the column direction. In Figure 5 the example shown, in the 4th column, the first phase difference pixel L, the second phase difference pixel R, the first phase difference pixel L, and the second phase difference pixel R are arranged in sequence from the first row along the column direction. That is, the first phase difference pixel L and the second phase difference pixel R are alternately arranged from the first row along the column direction. And, in Figure 5 the example shown, in the 10th column, the second phase difference pixel R, the first phase difference pixel L, the second phase difference pixel R, and the first phase difference pixel L are arranged in sequence from the first row along the column direction. That is, the second phase difference pixel R and the first phase difference pixel L are alternately arranged from the first row along the column direction.

[0088] The photoelectric conversion element 26 is divided into three regions. That is, the photoelectric conversion element 26 has a division region 26N1 for visible light images, a first distance measurement system division region 26N2, and a second distance measurement system division region 26N3. The division region 26N1 for visible light images is a group of visible light pixels based on a plurality of visible light pixels and is used to generate a visible light image. The first distance measurement system division region 26N2 is a group of IR pixels based on a plurality of IR pixels arranged in a two-dimensional shape and is used for the first distance measurement. Here, the IR pixel is an example of the "specific pixel" related to the technology of the present invention. The second distance measurement system division region 26N3 is a group of phase difference pixels based on a plurality of phase difference pixels and is used for the second distance measurement. The division region 26N1 for visible light images and the second distance measurement system division region 26N3 receive visible reflected light and output an electric signal corresponding to the amount of received light. The first distance measurement system division region 26N2 receives IR reflected light and outputs an electric signal corresponding to the amount of received light.

[0089] As an example, as Figure 6 shown, the first phase difference pixel L includes a microlens 19, a light shielding member 17A, and a photodiode PD. In the first phase difference pixel L, the light shielding member 17A is disposed between the microlens 19 and the light receiving surface of the photodiode PD. The left half of the light receiving surface of the photodiode PD in the row direction (the left side when facing the subject from the light receiving surface (in other words, the right side when facing the light receiving surface from the subject)) is shielded by the light shielding member 17A.

[0090] The second phase difference pixel R includes a microlens 19, a light shielding member 17B, and a photodiode PD. In the second phase difference pixel R, the light shielding member 17B is disposed between the microlens 19 and the light receiving surface of the photodiode PD. The right half of the light receiving surface of the photodiode PD in the row direction (the right side when facing the subject from the light receiving surface (in other words, the left side when facing the light receiving surface from the subject)) is shielded by the light shielding member 17B. In addition, hereinafter, for the sake of convenience of explanation, when there is no need to distinguish between the light shielding members 17A and 17B, they are referred to as "light shielding members" without symbols.

[0091] The light beam passing through the exit pupil of the imaging lens 41 is roughly divided into a left-region passing light 300L and a right-region passing light 300R. The left-region passing light 300L refers to the left half of the light beam passing through the exit pupil of the imaging lens 41 when looking from the phase difference pixel side towards the subject side, and the right-region passing light 300R refers to the right half of the light beam passing through the exit pupil of the imaging lens 41 when looking from the phase difference pixel side towards the subject side. The light beam passing through the exit pupil of the imaging lens 41 is divided into left and right by the microlens 19, the light shielding member 17A, and the light shielding member 17B that function as a pupil splitting section. The first phase difference pixel L receives the left-region passing light 300L as subject light, and the second phase difference pixel R receives the right-region passing light 300R as subject light. As a result, a first phase difference image corresponding to the subject image corresponding to the left-region passing light 300L and a second phase difference image corresponding to the subject image corresponding to the right-region passing light 300R are generated by the photoelectric conversion element 26.

[0092] In the smart device 10, for example, on the same phase difference pixel line 26A, the distance to the imaging region is measured based on the offset amount α between the first phase difference image of one line amount and the second phase difference image of one line amount.

[0093] As an example, as Figure 7 shown, the non-phase difference pixel N is different from the phase difference pixel in that it does not have a light shielding member. The photodiode PD of the non-phase difference pixel N receives the left-region passing light 300L and the right-region passing light 300R as subject light.

[0094] In addition, in the smart device 10, the IR reflected light is received by a plurality of IR pixels included in the region 26N3 divided by the second distance measurement system (refer to Figure 5 ), and thus distance measurement is performed for each IR pixel. And, in the smart device 10, according to the instruction received by the touch panel 48, as an example, as Figure 8 shown, the distance measurement result of each IR pixel is displayed as a distance image on the display 46. Here, the distance image refers to an image representing the distance to the distance measurement object measured for each IR pixel in terms of color and / or brightness.

[0095] Also, in the smart device 10, according to the instruction received by the touch panel 48, the ranging result is displayed on the display 46 as a distance image or a distance-overlaid image (not shown). The distance-overlaid image displayed on the display 46 is, for example, an image obtained by overlapping numerical values representing the ranging result on a visible light image (e.g., an instant preview image). For example, the distances from the smart device 10 to each of a plurality of representative positions (e.g., three positions) within the imaging area are displayed on the visible light image and displayed on the display 46. As an example of the plurality of representative positions, a plurality of positions within the imaging area where the contrast difference between each other in a specific subject (e.g., a subject and / or a person included in the central area of the screen, etc.) is equal to or greater than a specified value can be cited.

[0096] As an example, as Figure 9 shown, in addition to the light irradiator 16 and the light receiver 18, the smart device 10 further includes a controller 15, an input / output interface 40, an image memory 42, a UI system device 44, an external I / F 52, and a communication I / F 54.

[0097] The controller 15 includes a CPU 15A, a storage device 15B, and a memory 15C. The CPU 15A, the storage device 15B, and the memory 15C are connected via a bus 50, and the bus 50 is connected to the input / output interface 40. In addition, in Figure 9 the example shown, for the convenience of illustration, one bus is illustrated as the bus 50, but it may also be multiple buses. The bus 50 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0098] The storage device 15B stores various parameters and various programs. The storage device 15B is a non-volatile storage device. Here, as an example of the storage device 15B, a flash memory is adopted. The flash memory is merely an example, and as the storage device 15B, for example, various non-volatile memories such as a magnetoresistive memory and / or a ferroelectric memory can be cited instead of the flash memory or used together with the flash memory. Also, the non-volatile storage device may be an EEPROM, an HDD, and / or an SSD, etc. And the memory 15C temporarily stores various information and is used as a working memory. As an example of the memory 15C, a RAM can be cited, but it is not limited thereto, and other types of storage devices may also be used.

[0099] Various programs are stored in the storage device 15B. The CPU 15A reads out the required program from the storage device 15B and executes the read-out program on the memory 15C. The CPU 15A controls the entire smart device 10 according to the program executed on the memory 15C. In addition, the storage device 15B and the memory 15C are an example of the "memory" related to the technology of the present invention.

[0100] Multiple devices are connected to the input / output interface 40, and the input / output interface 40 is responsible for the transmission and reception of various information between the multiple devices. Figure 9 In the example shown, as the multiple devices connected to the input / output interface 40, a controller 15, a light irradiator 16, a light receiver 18, an image memory 42, a UI system device 44, an external I / F 52, and a communication I / F 54 are shown.

[0101] The external I / F 52 is responsible for the transmission and reception of various information between devices existing outside the intelligent device 10 (hereinafter, also referred to as "external devices"). As an example of the external I / F 52, a USB interface can be cited. External devices such as intelligent devices, personal computers, servers, USB memories, memory cards, and / or printers (not shown) can be directly or indirectly connected to the USB interface.

[0102] The communication I / F 54 has communication functions such as LTE, 5G, wireless LAN, and / or Bluetooth (registered trademark), and is responsible for the transmission and reception of various information between external devices and the CPU 15A. For example, the communication I / F 54 is communicably connected to the network 56 (e.g., the Internet) via a base station (not shown), and is responsible for the transmission and reception of various information between external devices on the network 56 and the CPU 15A.

[0103] The UI system device 44 includes a display 46, and the CPU 15A causes the display 46 to display various information. In addition, the UI system device 44 includes a receiving device 47. The receiving device 47 includes a touch panel 48 and a hard key unit 53. The hard key unit 53 is at least one hard key including an instruction key 13 (refer to Figure 2 ). The CPU 15A operates according to various instructions received through the touch panel 48. In addition, here, the hard key unit 53 is included in the UI system device 44, but the technology of the present invention is not limited thereto. For example, the hard key unit 53 may also be connected to the external I / F 52.

[0104] The light irradiator 16 includes a light transmissive window 20, a beam expander 21, a collimating lens 23, an LD 24, and an LD driver 25. Along the optical axis L1, the light transmissive window 20, the beam expander 21, and the collimating lens 23 are sequentially arranged from the imaging area side (object side) to the LD 24. The LD driver 25 is connected to the LD 24 and the input / output interface 40, and drives the LD 24 according to the instruction of the CPU 15A to emit a laser beam from the LD 24.

[0105] After the laser beam emitted from the LD 24 is converted into parallel light by the collimating lens 23, the beam diameter is expanded by the beam expander 21, and is irradiated toward the distance measurement object from the light transmissive window 20.

[0106] The light receiver 18 includes a light-transmitting window 22, an objective lens 30A, a focusing lens 30B, an aperture 30C, a photoelectric conversion element 26, a photoelectric conversion element driver 32, and a signal processing circuit 34.

[0107] In addition, the CPU 15A and the signal processing circuit 34 are examples of the "processor" involved in the technology of the present invention.

[0108] In the light receiver 18, along the optical axis L2, from the imaging region side (object side) to the photoelectric conversion element 26, a light-transmitting window 22, an objective lens 30A, a focusing lens 30B, and an aperture 30C are arranged in sequence. The photoelectric conversion element driver 32 is connected to the photoelectric conversion element 26 and the input / output interface 40, and drives the photoelectric conversion element 26 according to the instruction of the CPU 15A. For example, under the control of the CPU 15A, the photoelectric conversion element driver 32 supplies a imaging timing signal for specifying the timing of imaging performed by the photoelectric conversion element 26 to the photoelectric conversion element 26. The photoelectric conversion element 26 performs reset, exposure, and output of an electrical signal according to the imaging timing signal supplied from the photoelectric conversion element driver 32. As the imaging timing signal, for example, a vertical synchronization signal and a horizontal synchronization signal can be cited.

[0109] The light receiver 18 includes a focus control mechanism 31. The focus control mechanism 31 includes a focusing lens 30B, a moving mechanism 60, a motor 62, and a motor driver 64. The focusing lens 30B is supported by the moving mechanism 60 so as to be slidable along the optical axis L2. The motor 62 is connected to the moving mechanism 60 and the motor driver 64. The motor driver 64 is connected to the input / output interface 40 and drives the motor 62 according to the instruction from the CPU 15A. The moving mechanism 60 is connected to the drive shaft (not shown) of the motor 62, and by receiving power from the motor 62, the focusing lens 30B is selectively moved toward the object side and the image side along the optical axis L2. That is, the CPU 15A adjusts the focus position by controlling the drive of the motor 62 via the motor driver 64. Here, the "focus position" refers to the position of the focusing lens 30B on the optical axis L2 in a state where the focus is aligned (for example, a state where the contrast of the visible light image is set to the maximum value or a state where a predetermined subject depth is achieved). In addition, in the first embodiment, the control of aligning the focusing lens 30B to the focus position is referred to as "focus control".

[0110] The aperture 30C is a fixed aperture with a constant opening. In the case of a fixed aperture, the exposure adjustment is performed by the electronic shutter of the photoelectric conversion element 26. The aperture 30C may not be a fixed aperture but a variable aperture. In addition, the objective lens 30A, the focusing lens 30B, and the aperture 30C included in the light receiver 18 are merely examples, and the technology of the present invention is still valid even if the structure of the lens and / or the position of the aperture 30C are changed.

[0111] In the light receiver 18, the reflected light is incident from the light-transmitting window 22. The reflected light incident on the light-transmitting window 22 is imaged on the photoelectric conversion element 26 via the objective lens 30A, the focusing lens 30B, and the aperture 30C.

[0112] The photoelectric conversion element 26 is connected to the signal processing circuit 34, and outputs pixel data representing the pixel values of the respective pixels of the visible light pixels and the IR pixels to the signal processing circuit 34. The signal processing circuit 34 digitizes the pixel data by performing A / D conversion on the pixel data input from the photoelectric conversion element 26, and performs various signal processes on the digitized pixel data.

[0113] The signal processing circuit 34 includes a visible light pixel data processing circuit 34A, a first distance measurement system processing circuit 34B, and a second distance measurement system processing circuit 34C. The visible light pixel data processing circuit 34A performs known signal processes such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction on the pixel data regarding the visible light pixels, that is, the visible light pixel data, thereby generating a visible light image. Then, the visible light pixel data processing circuit 34A stores the visible light image in the image memory 42. In addition, by covering and storing one frame amount of the visible light image in the image memory 42, the visible light image in the image memory 42 is updated.

[0114] The distance measurement imaging device 14 includes a TOF camera 27. The TOF camera 27 includes a light irradiator 16, a first distance measurement system division area 26N2 (refer to Figure 5 ) of the photoelectric conversion element 26, and a first distance measurement system processing circuit 34B. The first distance measurement system processing circuit 34B acquires an irradiation timing signal indicating the irradiation timing (hereinafter, also simply referred to as "irradiation timing") at which the light irradiator 16 irradiates the laser beam toward the imaging area from the CPU 15A.

[0115] The first distance measurement system processing circuit 34B measures the distance from the smart device 10 to an object in the imaging area for each IR pixel based on the irradiation timing indicated by the irradiation timing signal and the light reception timing (hereinafter, also referred to as "light reception timing") at which each IR pixel receives the IR reflected light. Here, as the light reception timing, the timing at which the first distance measurement system processing circuit 34B receives IR pixel data having an output value exceeding a reference threshold value (refer to Figure 14 ) described later is adopted. In addition, as an example of the noise component, a noise component generated independently of the IR reflected light (for example, IR light included in the ambient light) can be cited.

[0116] The first distance measurement system processing circuit 34B measures the distance from the smart device 10 to an object in the imaging area for each IR pixel based on the irradiation timing and the light reception timing. Further, the first distance measurement system processing circuit 34B generates a distance image based on the distance measurement results for each IR pixel, and stores the generated distance image in the image memory 42. In addition, by covering and saving a distance image of one frame amount in the image memory 42, the distance image in the image memory 42 is updated.

[0117] The second distance measurement system processing circuit 34C obtains phase difference pixel data representing pixel values of phase difference pixels from a plurality of phase difference pixels included in a region (so-called ROI) specified by a user or the like from the photoelectric conversion element 26 in the second distance measurement system divided region 26N3 (refer to Figure 5 ). The second distance measurement system processing circuit 34C generates a first phase difference image and a second phase difference image based on the phase difference pixel data (refer to Figure 5 ), and calculates an offset α between the generated first phase difference image and the second phase difference image (refer to Figure 5 ). Further, the second distance measurement system processing circuit 34C calculates the distance from the smart device 10 to the imaging area based on the calculated offset α. More specifically, the second distance measurement system processing circuit 34C calculates the distance from the smart device 10 to an object in the imaging area by using an arithmetic expression in which the offset α is an independent variable and the distance is a dependent variable.

[0118] In addition, although an arithmetic expression is exemplified herein, the technique of the present invention is not limited thereto, and the second distance measurement system processing circuit 34C may also derive the distance from the smart device 10 to the imaging area by using a table in which the offset α and the distance are associated with each other.

[0119] The CPU 15A acquires the distance measured by the first distance measurement system processing circuit 34B (hereinafter referred to as "first distance") from the first distance measurement system processing circuit 34B, and acquires the distance measured by the second distance measurement system processing circuit 34C (hereinafter referred to as "second distance") from the second distance measurement system processing circuit 34C.

[0120] As an example, as shown in Figure 10 , a distance measurement imaging processing program 70 is stored in the storage device 15B. The CPU 15A reads out the distance measurement imaging processing program 70 from the storage device 15B and executes the read-out distance measurement imaging processing program 70, thereby operating as a first distance measurement control unit 15A1, a first distance acquisition unit 15A2, a time series distribution acquisition unit 15A3, a determination unit 15A4, and an execution unit 15A5.

[0121] Here, refer to Figure 11, the case of performing the first distance measurement on the imaging area including the object subject ( Figure 11 In the example shown, the person) 98 and the entire dressing mirror 100 (hereinafter, also referred to as the "mirror-including imaging area") will be described. If a distance measurement imaging start instruction is received through the touch panel 48 in a state where the mirror-including imaging area is included in the irradiation angle θ2, the first distance measurement control unit 15A1 outputs a first distance measurement start signal to the light irradiator 16 and the light receiver 18. If the first distance measurement start signal is input from the first distance measurement control unit 15A1, the light irradiator 16 irradiates a laser beam. In the light receiver 18, if the first distance measurement start signal is input from the first distance measurement control unit 15A1, the first distance measurement system processing circuit 34B calculates the first distance based on the irradiation timing and the light reception timing indicated by the irradiation timing signal acquired from the CPU15A.

[0122] In addition, here, as the irradiation timing, the timing of a point in time after a predetermined time has elapsed from the timing when the first distance measurement start signal is input from the first distance measurement control unit 15A1 to the light receiver 18 is adopted. Here, regarding the predetermined time, for example, the time required from the point in time when the first distance measurement start signal is output to the time when the laser beam is irradiated from the light irradiator 16 is adopted, and the time is derived in advance through actual machine tests and / or computer simulations, etc.

[0123] If a laser beam is irradiated from the light irradiator 16 to the mirror-including imaging area, then as an example, as Figure 12 shown, the first distance measurement system division area 26N2 of the light receiver 18 receives the IR reflected light from the object subject 98 (hereinafter, also referred to as the "object subject IR reflected light") and the IR reflected light from the dressing mirror 100 (hereinafter, also referred to as the "mirror surface IR reflected light"). A part of the IR pixels (hereinafter, also referred to as "specific IR pixels") within the first distance measurement system division area 26N2 receive the object subject IR reflected light. If only the object subject IR reflected light is received by the specific IR pixels, the first distance measurement system processing circuit 34B uses the timing when the object subject IR reflected light is received by the specific IR pixels as the above-mentioned light reception timing, and thus the distance from the smart device 10 to the object subject 98 can be calculated as the first distance.

[0124] However, depending on the installation conditions of the dressing mirror 100, that is, depending on the position where the dressing mirror 100 is installed, the shape of the mirror surface 100A of the dressing mirror 100, and the angle of the mirror surface 100A, etc., the total reflection light of the laser beam with respect to the mirror surface 100A is also received by the specific IR pixels as the mirror surface IR reflected light. When the intensity of the laser beam irradiated from the light irradiator 16 is set to "100", the intensity of the object subject IR reflected light is about 10 to 20, while the intensity of the total reflection light of the laser beam with respect to the mirror surface 100A is about 40 to 50.

[0125] Here, it is assumed that during a light reception period (hereinafter, also simply referred to as the "light reception period") preset during the period of receiving IR reflected light corresponding to the irradiation of the laser beam from the light irradiator 16, among the multiple IR reflected lights received by a specific IR pixel at different timings, when ranging is performed based on the timing of the IR reflected light with the maximum intensity received by the specific IR pixel, it can be considered that ranging is performed based on the timing of the specular IR reflected light received by the specific IR pixel. This means that when the object that the user wishes to be the ranging object is the target subject, incorrect ranging that the user does not wish for is performed. That is, the distance from the smart device 10 to an object that the user does not wish to be the ranging object (in the Figure 12 example shown, the mirror 100A) is measured instead of the distance from the smart device 10 to the target subject 98.

[0126] Therefore, in order to avoid such incorrect ranging, as an example, as Figure 13 shown, the first ranging system processing circuit 34B includes an IR pixel data acquisition unit 34B1, a time series distribution generation unit 34B2, a light reception timing determination unit 34B3, and a first distance measurement unit 34B4. In addition, an IR pixel data acquisition unit 34B1, a time series distribution generation unit 34B2, a light reception timing determination unit 34B3, and a first distance measurement unit 34B4 are provided for each IR pixel. Hereinafter, for the sake of convenience in explanation, regarding the first ranging, a specific IR pixel (reference Figures 12 - 14 ) is focused on for explanation.

[0127] The first ranging control unit 15A1 outputs an irradiation timing signal to the IR pixel data acquisition unit 34B1, the time series distribution generation unit 34B2, and the first distance measurement unit 34B4. And the first ranging control unit 15A1 outputs an acquisition timing signal that specifies the timing for the IR pixel data acquisition unit 34B1 to acquire IR pixel data from the photoelectric conversion element 26 to the IR pixel data acquisition unit 34B1. The acquisition timing signal is output to the IR pixel data acquisition unit 34B1 at a predetermined time interval (for example, a time interval of about one tenth or one hundredth of the light reception period).

[0128] If an irradiation timing signal and an acquisition timing signal are input from the first distance measurement control unit 15A1, the IR pixel data acquisition unit 34B1 acquires IR pixel data from the photoelectric conversion element 26. If an irradiation timing is input from the first distance measurement control unit 15A1, the time series distribution generation unit 34B2 generates a time series distribution in which the intensity of the IR reflected light represented by a plurality of IR pixel data acquired by the IR pixel data acquisition unit 34B1 during the light reception period starting from the point in time when the irradiation timing is input from the first distance measurement control unit 15A1 is specified in time series. In the time series distribution, the horizontal axis represents time and the vertical axis represents intensity. Here, the plurality of IR pixel data is an example of the "plurality of signals" related to the technology of the present invention. The detailed content of the time series distribution will be described later.

[0129] The intensity of the IR reflected light has a one-to-one relationship with the signal level (intensity) of the IR pixel data, and the intensity of the IR reflected light is determined according to the signal level of the IR pixel data. Therefore, the time series distribution that specifies the intensity of the IR reflected light in time series is equivalent to the time series distribution of the signal level of the IR pixel data. And in the present first embodiment, the intensity of the IR reflected light is determined by the absolute value amount of the IR reflected light, but the technology of the present invention is not limited to this, and the intensity of the IR reflected light may also be offset by the amount of the intensity of the noise component based on ambient light (for example, IR light included in the ambient light). At this time, for example, the entire time series distribution may be offset by the amount of the intensity of the noise component based on ambient light.

[0130] In the time series distribution generated by the time series distribution generation unit 34B2, the intensity of the IR reflected light represented by the IR pixel data obtained from a specific IR pixel is represented in time series. The light reception timing determination unit 34B3 determines the light reception timing for the first distance measurement (hereinafter, also referred to as "the light reception timing for the first distance measurement") according to the time series distribution generated by the time series distribution generation unit 34B2. A reference threshold is applied to the time series distribution in the light reception timing determination unit 34B3. The reference threshold is a value derived in advance by actual machines and / or computer simulations, etc., as the lower limit value of the intensity of the IR reflected light from a standard subject (for example, a specific subject other than a mirror surface and a glossy surface). The reference threshold is a value that decreases as the light reception timing is delayed. In Figure 13 the example shown, the reference threshold gradually (for example, exponentially) decreases with the passage of time. If there is only one IR reflected light with an intensity exceeding the reference threshold in the time series distribution, the light reception timing determination unit 34B3 determines the light reception timing of the IR reflected light with an intensity exceeding the reference threshold as the light reception timing for the first distance measurement.

[0131] Further, when there are multiple IR reflected lights with intensities exceeding a reference threshold in the time series distribution, the light reception timing determination unit 34B3 determines the light reception timing corresponding to the IR pixel data selected based on the relationship between the multiple IR pixel data and the intensities of the multiple IR reflected lights represented by the multiple IR pixel data among the multiple IR pixel data acquired by the IR pixel data acquisition unit 34B1 during the light reception period as the first light reception timing for distance measurement. The relationship between the multiple IR pixel data and the intensities of the multiple IR reflected lights is determined by the light reception timing determination unit 34B3 based on the time series distribution generated by the time series distribution generation unit 34B2. In addition, the detailed content of the method for determining the first light reception timing for distance measurement when there are multiple IR reflected lights with intensities exceeding a reference threshold in the time series distribution will be described later.

[0132] The first distance measurement unit 34B4 measures the first distance based on the irradiation timing indicated by the irradiation timing signal input from the first distance measurement control unit 15A1 and the first light reception timing for distance measurement determined by the light reception timing determination unit 34B3. The first distance is one-half of the product of the speed of light and the flight time of the laser beam. The flight time of the laser beam is the time from the irradiation timing to the first light reception timing for distance measurement.

[0133] In Figure 14 the example shown, the time series distribution includes the intensities of specular IR reflected light, object subject IR reflected light, and noise light (e.g., noise components such as IR light contained in ambient light). On the time axis of the light reception period, the specular IR reflected light is first received by a specific IR pixel, then the noise light is received, and then the object subject IR reflected light is received. Moreover, the intensities of the specular IR reflected light and the object subject IR reflected light exceed the reference threshold.

[0134] At this time, the light reception timing determination unit 34B3 determines the timing when the second-largest intensity IR reflected light is received by the specific IR pixel in the time series distribution as the first light reception timing for distance measurement. That is, in Figure 14 the example shown, the second-largest intensity IR reflected light in the time series distribution is the object subject IR reflected light, so the light reception timing of the object subject IR reflected light is determined as the first light reception timing for distance measurement. Thereby, distance measurement based on the light reception timing of the specular IR reflected light and distance measurement based on the noise light are avoided.

[0135] As an example, as Figure 15As shown, the first distance acquisition unit 15A2 acquires the first distance measured by the first distance measurement unit 34B4. Further, the time series distribution acquisition unit 15A3 acquires a time series distribution from the time series distribution generation unit 34B2. The time series distribution acquired by the time series distribution acquisition unit 15A3 is a time series distribution for determining the first light reception timing for distance measurement used in the measurement of the first distance acquired by the first distance acquisition unit 15A2.

[0136] The determination unit 15A4 determines whether the time series distribution acquired by the time series distribution acquisition unit 15A3 is a specific time series distribution. Details of this determination method will be described later.

[0137] When the determination unit 15A4 determines that the time series distribution acquired by the time series distribution acquisition unit 15A3 is not a specific time series distribution, the execution unit 15A5 executes imaging (hereinafter, also referred to as "first imaging process") accompanied by focusing control based on the first distance acquired by the first distance acquisition unit 15A2. Further, when it is determined that the time series distribution acquired by the time series distribution acquisition unit 15A3 is a specific time series distribution, the execution unit 15A5 executes a specific process. In addition, details of the first imaging process and the specific process will be described later.

[0138] Here, the determination method based on the determination unit 15A4 will be described. As an example, as Figure 16 shown, the determination unit 15A4 applies a first threshold value and a second threshold value to the time series distribution acquired by the time series distribution acquisition unit 15A3. The first threshold value is a lower limit value of the intensity of the IR reflected light when the laser beam irradiated on a glossy surface (for example, a mirror surface preset as an average mirror surface) is totally reflected by the glossy surface, and is a value derived in advance through actual machines and computer simulations, etc. The first threshold value gradually (for example, exponentially) decreases over time in the same manner as the above-mentioned reference threshold value. Further, the first threshold value is a value that decreases as the light reception timing is delayed in the same manner as the above-mentioned reference threshold value. The second threshold value is the same value as the above-mentioned reference threshold value (refer to Figure 13 and Figure 14 ). Further, at the same moment on the time axis during the light reception period, the second threshold value is a value smaller than the first threshold value.

[0139] The determination unit 15A4 determines whether the time series distribution acquired by the time series distribution acquisition unit 15A3 includes an intensity equal to or higher than the first threshold value and an intensity lower than the first threshold value and equal to or higher than the second threshold value, thereby determining whether the time series distribution acquired by the time series distribution acquisition unit 15A3 is a specific time series distribution. In Figure 16In the example shown, the intensity of the specular IR reflected light is above the first threshold, and the intensity of the IR reflected light of the object subject is less than the first threshold and above the second threshold. Therefore, the determination unit 15A4 determines that the time series distribution obtained by the time series distribution acquisition unit 15A3 is a specific time series distribution. Additionally, in Figure 16 In the example shown, the intensity of the specular IR reflected light corresponds to the intensity of the IR pixel data (signal) generated by receiving the specular IR reflected light. The intensity of the IR pixel data (signal) generated by receiving the specular IR reflected light is an example of the "first intensity" related to the technology of the present invention. Similarly, the intensity of the IR reflected light of the object subject corresponds to the intensity of the IR pixel data (signal) generated by receiving the IR reflected light of the object subject. The intensity of the IR pixel data (signal) generated by receiving the IR reflected light of the object subject is an example of the "second intensity" related to the technology of the present invention.

[0140] As an example, as Figure 17 shown, a focus position derivation table 72 is stored in the storage device 15B. In the focus position derivation table 72, a correspondence is established between the distance from the smart device 10 to the imaging area and the focus position. In the first imaging process, first, the execution unit 15A5 performs focus control (active mode focus control) on the light receiver 18 by using the first distance. That is, the execution unit 15A5 derives the focus position corresponding to the first distance from the focus position derivation table 72, and controls the motor 62 of the light receiver 18 so that the focusing lens 30B moves to the derived focus position. Then, the execution unit 15A5 controls the visible light image division area 26N1 of the light receiver 18, thereby capturing the imaging area through the visible light image division area 26N1, and the visible light image data obtained by the capture is output from the visible light image division area 26N1 to the visible light pixel data processing circuit 34A. The visible light pixel data processing circuit 34A generates a first visible light image representing the imaging area based on the visible light image data input from the visible light image division area 26N1, and outputs the generated first visible light image to the image memory 42. The first visible light image is stored in the image memory 42.

[0141] Here, a specific process executed by the execution unit 15A5 will be described with reference to Figure 18 As an example, as Figure 18 shown, the specific process is, for example, a process including a second distance measurement start process, a second distance acquisition process, a second imaging process, and an image selection screen display process. Additionally, the specific process is not limited to these, and it can also be a process of measuring the first distance based on the light reception timing of the reflected light (IR reflected light of the object subject) whose intensity received by the IR pixel is less than the first threshold and above the second threshold.

[0142] As an example, as Figure 19 shown, in the second distance measurement start process, if a distance measurement camera start instruction is received by the touch panel 48, the execution unit 15A5 outputs a second distance measurement start signal to the light receiver 18. Additionally, Figure 19 in the example shown, a state where the mirror includes a imaging area within the field of view angle θ1 is shown, but the technology of the present invention is not limited thereto, and it may also be an imaging area other than the mirror-included imaging area.

[0143] If a second distance measurement start signal is input from the execution unit 15A5 to the light receiver 18, the second distance measurement system divided area 26N3 captures the mirror-included imaging area and outputs phase difference pixel data corresponding to the mirror-included imaging area to the second distance measurement system processing circuit 34C. The second distance measurement system processing circuit 34C generates a first phase difference image and a second phase difference image based on the phase difference pixel data input from the second distance measurement system divided area 26N3 (refer to Figure 6 ), and calculates an offset α based on the generated first phase difference image and second phase difference image (refer to Figure 6 ). The second distance measurement system processing circuit 34C calculates a second distance based on the calculated offset α. In the second distance acquisition process, the execution unit 15A5 acquires the second distance from the second distance measurement system processing circuit 34C.

[0144] As an example, as Figure 20 shown, the second imaging process refers to imaging accompanied by focus control based on the second distance. In the second imaging process, similarly to the first imaging process, the execution unit 15A5 derives a focus position corresponding to the second distance from the focus position derivation table 72 and controls the motor 62 of the light receiver 18 so that the focusing lens 30B moves to the derived focus position. Then, similarly to the first imaging process, the visible light image divided area 26N1 captures the imaging area, and the visible light pixel data obtained by the capture is output from the visible light image divided area 26N1 to the visible light pixel data processing circuit 34A. The visible light pixel data processing circuit 34A generates a second visible light image representing the imaging area based on the visible light pixel data input from the visible light image divided area 26N1 and outputs the generated second visible light image to the image memory 42. The second visible light image is stored in the image memory 42.

[0145] In the image selection screen display process, the execution unit 15A5 acquires the first visible light image and the second visible light image from the image memory 42. Then, the execution unit 15A5 based on the first visible light image, the second visible light image, the first distance acquired by the first distance acquisition unit 15A2 (refer to Figure 15 ), the second distance acquired by the execution unit 15A5 in the second distance acquisition process (refer to Figure 19) and various messages to generate an image selection screen, and display the generated image selection screen on the display 46. The first visible light image and the second visible light image are arranged and displayed on the image selection screen. And, a message "active mode" is displayed below the first visible light image. The message "active mode" is a message indicating that the first visible light image is an image obtained by imaging with focus control accompanying the ranging result based on ranging (first ranging) in the active mode. And, a message "passive mode" is displayed below the second visible light image. The message "passive mode" is a message indicating that the second visible light image is an image obtained by imaging with focus control accompanying the ranging result based on ranging (second ranging) in the passive mode.

[0146] In addition, in the present first embodiment, as the processes included in the specific process, the first imaging process, the second imaging process, and the image selection screen display process are exemplified, but the technology of the present invention is not limited thereto, and it may also be one process or two processes among the first imaging process, the second imaging process, and the image selection screen display process. And, in the present first embodiment, a display example of "active mode" and "passive mode" is shown, but it is not necessary to display "active mode" and "passive mode", and any display may be used as long as the user can understand the difference in the ranging methods. For example, when performing laser ranging and phase difference ranging, it may be the display of "laser" and "phase difference", or an icon indicating the ranging method may be displayed. And, the focus position may be displayed instead of the display of the ranging method. For example, it may be "focus position: near" and "focus position: far", or it may be displayed as "focus position: object" and "focus position: image of the object reflected" etc. Or, two or more of the characters and icons indicating the ranging method and the focus position may be combined for display.

[0147] And, in the first visible light image, a value of "1.8 m", which is the ranging result based on the first ranging, is displayed in a corresponding association, and in the second visible light image, a value of "1.9 m", which is the ranging result based on the second ranging, is displayed in a corresponding association. And, on the image selection screen, as a message prompting the user to select either the first visible light image or the second visible light image, a message "please select any one image" is displayed to the user.

[0148] Moreover, a strong reflected light notification message is displayed on the image selection screen. The strong reflected light notification message means that when it is determined by the determination unit 15A4 (reference Figure 15 ) that the time series distribution obtained by the time series distribution acquisition unit 15A3 (reference Figure 15 ) is a specific time series distribution (for example, Figure 16As shown, when the intensity above the first threshold is included in the time series distribution), the user is notified of the message that the intensity above the first threshold is included in the time series distribution obtained by the time series distribution acquisition unit 15A3. In Figure 20 In the example shown, as the strong reflected light notification message, there is shown a message "Strong reflected light detected, but ranging was performed using reflected light other than the strong reflected light." Here, the strong reflected light means IR reflected light with an intensity above the first threshold. As an example of the strong reflected light, specular IR reflected light can be cited (refer to Figure 12 , Figure 14 and Figure 16 ). In addition, even for IR reflected light from a glossy surface other than the mirror surface 100A, if it is IR reflected light with an intensity above the first threshold, the strong reflected light notification message is displayed on the image selection screen. In addition, the image selection screen display process is an example of the "process of notifying that the intensity of multiple signals includes the first intensity" related to the technology of the present invention.

[0149] The various messages displayed on the image selection screen are not limited to the above messages. For example, it can be set that when the ranging result (first distance) based on the first ranging is different from the ranging result (second distance) based on the second ranging, a message notifying the user of this content (for example, "The ranging results are different") is also displayed.

[0150] And here, an example of a method of visually displaying various messages has been described, but it is not limited to this. It can also be set to output various messages by sound in parallel with the visual display. If the image selection screen is displayed on the display 46, the user selects the first visible light image or the second visible light image via the touch panel 48. In Figure 20 In the example shown, a method is shown in which the user's finger selects the first visible light image via the touch panel 48.

[0151] As an example, as Figure 21 shown, when the user's finger selects the first visible light image via the touch panel 48 in a state where the image selection screen is displayed on the display 46, the first ranging control unit 15A1 controls the ranging imaging device 14 to cause the first ranging system processing circuit 34B to perform the first ranging. And the execution unit 15A5 executes the first imaging process and the first visible light image display process. Here, the execution unit 15A5 executes the first imaging process using the new first distance obtained by performing the first ranging. The first visible light image display process is a process of displaying the latest first visible light image obtained by executing the first imaging process on the display 46.

[0152] In addition, here, an example is given in which, under the condition that the first visible light image is selected, the first distance measurement and the first imaging process are performed again, and the latest first visible light image obtained by performing the first imaging process is displayed on the display 46. However, this is merely an example. For example, under the condition that the first visible light image is selected, the selected first visible light image may be stored in a storage medium such as the storage device 15B and / or the memory card. And it can be set as follows: according to the instruction received by the receiving device 47 (the instruction issued by the user), select the imaging with focus control based on the distance measured by the selected distance measurement method and the saving of the selected image (for example, the first visible light image or the second visible light image).

[0153] If the first visible light image display process is executed by the execution unit 15A5, then in the image selection screen, the second visible light image, the numerical value "1.9m", the message "passive mode", the message "please select any one image", and the strong reflected light notification message are not displayed on the screen. Then, as the first distance ( Figure 21 In the example shown, the numerical value "1.8m") of the latest distance measurement result obtained by performing the latest first distance measurement is displayed on the display 46, the first visible light image obtained by executing the first imaging process is displayed on the display 46, and the message "active mode" is also displayed below the first visible light image. And the display areas of the first distance, the first visible light image, and the message "active mode" are enlarged compared to the display areas of the first distance, the first visible light image, and the message "active mode" in the image selection screen. In addition, at this time, it can also be set as follows: by performing the first distance measurement, when the determination unit 15A4 (refer to Figure 15 ) determines that the time series distribution obtained by the time series distribution acquisition unit 15A3 (refer to Figure 15 ) is a specific time series distribution, the strong reflected light notification message is displayed on the display 46.

[0154] As an example, as Figure 22 shown, when the user's finger selects the second visible light image via the touch panel 48 in the state where the image selection screen is displayed on the display 46, the execution unit 15A5 starts the second distance measurement process, the second distance acquisition process, and the second imaging process by executing the second distance measurement, and causes the second distance measurement system processing circuit 34C to perform the second distance measurement. And the execution unit 15A5 executes the second imaging process and the second visible light image display process. Here, the execution unit 15A5 uses the new second distance obtained by performing the second distance measurement to execute the second imaging process. The second visible light image display process is a process of displaying the latest second visible light image obtained by executing the second imaging process on the display 46.

[0155] If the second visible light image display process is executed by the execution unit 15A5, in the image selection screen, the first visible light image, the numerical value "1.8 m", the message "active mode", the message "ranging results are different", and the message "please select any one image" are not displayed on the screen. Then, as the second distance ( Figure 22 In the example shown, the numerical value "1.9 m") obtained by performing the latest second ranging is displayed on the display 46, the second visible light image obtained by executing the second imaging process is displayed on the display 46, and the message "passive mode" is also displayed below the second visible light image. Also, the display areas of the second distance, the second visible light image, and the message "passive mode" are enlarged compared to the display areas of the second distance, the second visible light image, and the message "passive mode" in the image selection screen.

[0156] Next, the operation of the part related to the technology of the present invention of the smart device 10 will be described.

[0157] First, refer to Figure 23 The first ranging system process executed by the first ranging system processing circuit 34B will be described. In addition, Figure 23 is a flowchart showing an example of the process of the first ranging system process executed by the first ranging system processing circuit 34B when an irradiation timing signal is input from the first ranging control unit 15A1.

[0158] In Figure 23 In the first ranging system process shown, first, in step ST100, the IR pixel data acquisition unit 34B1 determines whether an acquisition timing signal is input from the first ranging control unit 15A1. In step ST100, when the acquisition timing signal is not input from the first ranging control unit 15A1, the determination is negative, and the determination in step ST100 is performed again. In step ST100, when the acquisition timing signal is input from the first ranging control unit 15A1, the determination is affirmative, and the first ranging system process proceeds to step ST102.

[0159] In step ST102, the IR pixel data acquisition unit 34B1 acquires IR pixel data from the photoelectric conversion element 26, and then the first ranging system process proceeds to step ST104.

[0160] Assuming that the distance is measured based on the light reception timing and irradiation timing of the following IR reflected light received by the IR pixel, it is possible to measure the distance based on the light reception timing and irradiation timing of the IR reflected light from a glossy object ( Figure 12 In the example shown, the mirror 100A) that is not desired to be the ranging object, and the IR reflected light is composed of a plurality of light reception timings (for example,Figure 14 The "time interval" shown is represented by the IR pixel data with the highest signal level among the multiple IR pixel data generated by dividing the area 26N2 by the first distance measurement system.

[0161] Therefore, in the first distance measurement system processing, the processes of step ST104 to step ST110 are executed. First, in step ST104, the time series distribution generation unit 34B2 determines whether the light receiving period has elapsed after the start of the first distance measurement system processing. In step ST104, when the light receiving period has not elapsed after the start of the first distance measurement system processing, the determination is negative, and the first distance measurement system processing transitions to step ST100. In step ST104, when the light receiving period has elapsed after the start of the first distance measurement system processing, the determination is positive, and the first distance measurement system processing transitions to step ST106.

[0162] In step ST106, the time series distribution generation unit 34B2 generates a time series distribution based on the IR pixel data acquired by the IR pixel data acquisition unit through the processes of executing step ST100 to step ST104 (reference Figure 14 ). After executing step ST106, the first distance measurement system processing transitions to step ST108.

[0163] In step ST108, the light receiving timing determination unit 34B3 determines the first distance measurement light receiving timing based on the time series distribution generated in step ST106. In this step ST108, if there is only one IR reflected light with an intensity exceeding the reference threshold in the time series distribution, the light receiving timing determination unit 34B3 determines the light receiving timing of the IR reflected light with an intensity exceeding the reference threshold as the first distance measurement light receiving timing. And when there are multiple IR reflected lights with intensities exceeding the reference threshold in the time series distribution, the light receiving timing determination unit 34B3 determines the timing when the second largest intensity IR reflected light in the time series distribution is received by a specific IR pixel as the first distance measurement light receiving timing. After executing the process of step ST108, the first distance measurement system processing transitions to step ST110.

[0164] In step ST110, the first distance measurement unit 34B4 measures the first distance based on the irradiation timing indicated by the irradiation timing signal input from the first distance measurement control unit 15A1 and the first distance measurement light receiving timing determined in step ST108. After executing the process of step ST110, the first distance measurement system processing ends.

[0165] Next, refer to Figures 24A - 24C The distance measurement imaging process executed by the CPU15A will be described. In addition, Figures 24A - 24C is a flowchart showing an example of the flow of the distance measurement imaging process executed by the CPU15A according to the distance measurement imaging process program 70.

[0166] In Figure 24A the distance measurement imaging process shown, first, in step ST200, the time-series distribution acquisition unit 15A3 determines whether a time-series distribution has been generated by the time-series distribution generation unit 34B2. When the time-series distribution generation unit 34B2 has not generated a time-series distribution in step ST200, the determination is negative, and the determination in step ST200 is performed again. When the time-series distribution generation unit 34B2 has generated a time-series distribution in step ST200, the determination is positive, and the distance measurement imaging process proceeds to step ST202.

[0167] In step ST202, the time-series distribution acquisition unit 15A3 acquires the time-series distribution from the time-series distribution generation unit 34B2, and then the distance measurement imaging process proceeds to step ST204.

[0168] In step ST204, the determination unit 15A4 determines whether the time-series distribution acquired in step ST202 is a specific time-series distribution. When the time-series distribution acquired in step ST202 is not a specific time-series distribution in step ST204, the determination is negative, and the distance measurement imaging process proceeds to Figure 24B the step ST218 shown. When the time-series distribution acquired in step ST202 is a specific time-series distribution in step ST204, the determination is positive, and the distance measurement imaging process proceeds to step ST205.

[0169] In step ST205, the first distance acquisition unit 15A2 acquires the first distance measured through Figure 23 the step ST110 shown. After performing the process of step ST205, the distance measurement imaging process proceeds to step ST206.

[0170] In step ST206, the execution unit 15A5 performs imaging (first imaging process) with focus control (active mode focus control) based on the first distance acquired in step ST205, and then the distance measurement imaging process proceeds to step ST208.

[0171] The processes of steps ST208 to ST212 are the above-mentioned specific processes. In step ST208, the execution unit 15A5 causes the second distance measurement system processing circuit 34C to perform second distance measurement by executing the second distance measurement start process and the second distance acquisition process, and acquires the distance measurement result based on the second distance measurement, that is, the second distance. After performing the process of step ST208, the distance measurement imaging process proceeds to step ST210.

[0172] In step ST210, the execution unit 15A5 performs imaging (second imaging process) with focus control (passive focus control) accompanying the second distance acquired in step ST208. After the process of step ST210 is executed, the distance measurement imaging process transitions to step ST212.

[0173] In step ST212, the execution unit 15A5 performs an image selection screen display process. Thereby, an image selection screen is generated based on the first distance acquired in step ST205, the first visible light image obtained by performing the first imaging process of step ST206, the second distance acquired in step ST208, the second visible light image obtained by performing the second imaging process of step ST210, and the various messages described above (see Figure 20 ), and the image selection screen is displayed on the display 46. After the process of step ST212 is executed, the distance measurement imaging process transitions to step ST214.

[0174] The first visible light image obtained by performing the first imaging process of step ST206 and the second visible light image obtained by performing the second imaging process of step ST210 are displayed on the image selection screen. Therefore, in step ST214, the execution unit 15A5 determines whether the user has selected either the first visible light image or the second visible light image displayed within the image selection screen via the touch panel 48. In step ST214, when the user has not selected either the first visible light image or the second visible light image displayed within the image selection screen via the touch panel 48, the determination is negative, and the determination of step ST214 is performed again. In step ST214, when the user has selected either the first visible light image or the second visible light image displayed within the image selection screen via the touch panel 48, the determination is affirmative, and the distance measurement imaging process transitions to step ST216.

[0175] In step ST216, the execution unit 15A5 determines whether the image selected from within the image selection screen is the first visible light image. In step ST216, when the image selected from within the image selection screen is the second visible light image, the determination is negative, and the distance measurement imaging process transitions to Figure 24C step ST226 shown. In step ST216, when the image selected from within the image selection screen is the first visible light image, the determination is affirmative, and the distance measurement imaging process transitions to Figure 24B step ST218 shown.

[0176] In Figure 24BIn step ST218 shown above, the execution unit 15A5 determines whether the imaging start timing has been reached. The imaging start timing refers to, for example, the timing to start imaging for one frame amount defined by the frame rate for the live preview image. For example, if the frame rate for the live preview image is 60 fps, the imaging start timing is the timing for every 1 / 60 second. Additionally, here, the timing to start imaging for one frame amount defined by the frame rate for the live preview image is exemplified as the imaging start timing, but the technology of the present invention is not limited thereto. For example, the imaging start timing may be the timing to start imaging for one frame amount defined by the frame rate for recording a moving image, or may be the timing when the receiving device 47 receives an instruction to start imaging a still image.

[0177] In step ST218, when the imaging start timing has not been reached, the determination is negative, and the distance measurement imaging process transitions to step ST222. In step ST218, when the imaging start timing has been reached, the determination is positive, and the distance measurement imaging process transitions to step ST220.

[0178] In step ST220, the execution unit 15A5 performs the first imaging process using the latest first distance. And the execution unit 15A5 performs the first visible light image display process using the latest first visible light image obtained by performing the first imaging process. Additionally, here, the latest first distance refers to the latest first distance among the first distance measured by performing the process of step ST110 and the first distance measured by performing the process of step ST224 described later.

[0179] In step ST222, the execution unit 15A5 determines whether the condition to end the distance measurement imaging process (hereinafter referred to as the "end condition") is satisfied. As an example of the end condition, the condition that the receiving device 47 receives an instruction to end the distance measurement imaging process can be cited. In step ST222, when the end condition is not satisfied, the determination is negative, and the imaging process transitions to step ST224. In step ST222, when the end condition is satisfied, the determination is positive, and the distance measurement imaging process ends.

[0180] In step ST224, the execution unit 15A5 causes the first distance measurement system processing circuit 34B to perform the first distance measurement. Here, the first distance measurement is, for example, the same process as the processes of steps ST100 to ST110. After performing the process of step ST224, the distance measurement imaging process transitions to step ST218.

[0181] In Figure 24CIn step ST226 shown, the execution unit 15A5 determines whether the imaging start timing has been reached. In step ST226, when the imaging start timing has not been reached, the determination is negative, and the distance measurement imaging process transitions to step ST232. In step ST226, when the imaging start timing has been reached, the determination is affirmative, and the distance measurement imaging process transitions to step ST228.

[0182] In step ST228, the execution unit 15A5 causes the second distance measurement system processing circuit 34C to perform second distance measurement by executing the second distance measurement start process and the second distance acquisition process, and acquires the second distance, which is the distance measurement result based on the second distance measurement. After the process of step ST228 is executed, the distance measurement imaging process transitions to step ST230.

[0183] In step ST230, the execution unit 15A5 performs the second imaging process using the second distance obtained by performing the second distance measurement in step ST228. Further, the execution unit 15A5 performs the second visible light image display process using the latest second visible light image obtained by executing the second imaging process. After the process of step ST230 is executed, the distance measurement imaging process transitions to step ST232.

[0184] In step ST232, the execution unit 15A5 determines whether the end condition is satisfied. In step ST232, when the end condition is not satisfied, the determination is negative, and the imaging process transitions to step ST226. In step ST232, when the end condition is satisfied, the determination is affirmative, and the distance measurement imaging process ends.

[0185] As described above, the first distance measurement system process (refer to Figure 23 ) and the distance measurement imaging process (refer to Figures 24A - 24C ) are executed in the intelligent device 10. That is, the first distance is measured based on the timing (the first light reception timing for distance measurement) at which the IR pixel receives the following IR reflected light and the irradiation timing, where the IR reflected light is represented by the IR pixel data selected according to the relationship between the multiple IR pixel data and the intensities of the multiple IR reflected lights represented by the multiple IR pixel data among the multiple IR pixel data generated by dividing the area 26N2 by the first distance measurement system at multiple light reception timings (for example, the "time interval" shown in Figure 14 ) within the light reception period corresponding to the irradiation of the laser beam from the light irradiator 16. Therefore, according to this configuration, it is possible to avoid erroneous distance measurement caused by the reflected light from a shiny object that is not desired to be the distance measurement target (in the example shown in Figure 12 , the mirror surface 100A).

[0186] Further, in the smart device 10, as the light reception timing for the first distance measurement, the light reception timing of the IR reflected light related to the IR pixel data with the second largest signal level among the multiple IR pixel data is adopted. Therefore, according to this configuration, compared with the case where the light reception timing of the IR reflected light related to the IR pixel data with the largest signal level among the multiple IR pixel data is adopted as the light reception timing for the first distance measurement, it is possible to accurately avoid false distance measurement caused by the reflected light from a shiny object that is not desired to be the object of distance measurement.

[0187] Further, in the smart device 10, when the intensity above the first threshold and the intensity less than the first threshold and above the second threshold are included in the time series distribution, the specific process is executed by the execution unit 15A5. Therefore, according to this configuration, it helps to solve many problems when obtaining the distance measurement result based on the IR reflected light from a shiny object that the user does not desire to be the object of distance measurement and the distance measurement result based on the IR reflected light from an object that the user desires to be the object of distance measurement ( Figure 12 In the example shown, the object subject 98).

[0188] Further, in the smart device 10, as the first threshold, a value that decreases as the light reception timing is delayed is adopted. Therefore, according to this configuration, compared with the case where the first threshold is fixed regardless of the light reception timing, it is possible to accurately determine the distance measurement result based on the reflected light from a shiny object.

[0189] Further, in the smart device 10, a strong reflected light notification message is displayed on the image selection screen presented to the user. Therefore, according to this configuration, it is possible to make the user aware of the situation where the IR reflected light from a shiny object that the user does not desire to be the object of distance measurement is received.

[0190] Further, in the smart device 10, the second distance measurement is performed as the specific process (refer to Figure 24A Step ST208). Therefore, according to this configuration, compared with the case where the first distance measurement is still performed although the time series distribution generated by the time series distribution generation unit 34B2 is a specific time series distribution, it is possible to accurately measure the distance to the object of distance measurement desired by the user ( Figure 12 In the example shown, the object subject 98).

[0191] Further, in the smart device 10, active mode focusing control is performed based on the first distance obtained by the first distance measurement. Therefore, according to this configuration, it is possible to meet the needs of users who prefer active mode focusing control more than passive mode focusing control.

[0192] Also, in the intelligent device 10, for a plurality of IR pixels included in the divided area 26N2 of the first distance measurement system, the first distance is measured according to the irradiation timing and the first light receiving timing for distance measurement. Therefore, according to this configuration, compared with the case where the first distance is measured for only a single IR pixel according to the irradiation timing and the first light receiving timing for distance measurement, the first distance can be measured over a large range. Also, a distance image can be generated.

[0193] Also, in the intelligent device 10, an image selection screen including a first visible light image obtained by imaging under focus control based on the first distance and a second visible light image obtained by imaging under focus control based on the second distance is displayed on the display 46. Then, a message prompting the user to select the first visible light image or the second visible light image is displayed on the image selection screen. Therefore, according to this configuration, compared with the case where there is no room to select the first visible light image and the second visible light image, it helps to improve usability.

[0194] In addition, in the above first embodiment, the execution unit 15A5 prompts the user to select the first visible light image or the second visible light image through the image selection screen, but the technology of the present invention is not limited thereto, and both the first visible light image and the second visible light image can also be selected. At this time, for example, the first imaging process and the second imaging process can be alternately performed, and the first visible light image and the second visible light image obtained by performing each imaging process are respectively displayed on the display 46 as an instant preview image or a still image.

[0195] Also, in the above first embodiment, an example of the method in which the user selects the first visible light image or the second visible light image from the image selection screen via the touch panel 48 has been described, but the technology of the present invention is not limited thereto. For example, when the first distance and the second distance are different, the user can also be made to select in advance which one of the active distance measurement and the passive distance measurement to preferentially use. The active distance measurement is imaging accompanied by focus control based on the distance measurement result obtained by the first distance measurement, and the passive distance measurement is imaging accompanied by focus control based on the distance measurement result obtained by the second distance measurement.

[0196] Also, in the above first embodiment, a laser beam is exemplified as the distance measurement light irradiated by the light irradiator 16, but the technology of the present invention is not limited thereto. For example, the distance measurement light can be a directed light such as super-radiant light, light emitted from a xenon flash light source, or light emitted from an LED.

[0197] Also, in the above-described first embodiment, the visible light image division area 26N1, the first distance measurement system division area 26N2, and the second distance measurement system division area 26N3 are integrated into a single chip by the photoelectric conversion element 26. However, the technology of the present invention is not limited thereto. It is also possible to integrate multiple visible light pixels into a single chip, multiple phase difference pixels into a single chip, and multiple IR pixels into a single chip. Moreover, it is also possible to integrate multiple visible light pixels and multiple phase difference pixels into a single chip, and multiple IR pixels into a single chip. In this way, when various photosensitive pixels are integrated into a chip according to each type and mounted on the smart device 10, an optical system such as an objective lens, a focusing lens, and an aperture can be provided for each chip on the subject side (object side).

[0198] Also, in the above-described first embodiment, an example in which the distance measurement imaging device 14 is built into the smart device 10 has been described. However, the technology of the present invention is not limited thereto. For example, it is also possible to externally install the distance measurement imaging device 14 on a normal smart device, that is, a smart device without the built-in distance measurement imaging device 14.

[0199] Also, in the above-described first embodiment, an example in which the UI system device 44 is assembled into the smart device 10 has been described. However, at least a part of the multiple components included in the UI device 44 can also be installed outside the smart device 10. Moreover, at least a part of the multiple components included in the UI system device 44 can also be used as an independent unit by connecting to the external I / F 52.

[0200] Also, in Figure 1 the example shown, the smart device 10 has been illustrated. However, the technology of the present invention is not limited thereto. That is, the technology of the present invention can also be applied to various electronic devices (for example, interchangeable lens cameras, fixed lens cameras, personal computers, and / or wearable terminal devices, etc.) with the built-in distance measurement imaging device 14. Even for these electronic devices, the same functions and effects as those of the smart device 10 can be obtained.

[0201] Also, in the above-described first embodiment, the display 46 has been illustrated. However, the technology of the present invention is not limited thereto. For example, it is also possible to use a display provided outside the smart device 10 in combination with the display 46.

[0202] Also, in the above-described first embodiment, the photoelectric conversion element 26 and the signal processing circuit 34 are provided separately. However, it is also possible to use a stacked imaging element in which the photoelectric conversion element 26 and the signal processing circuit 34 are integrated into a single chip. Moreover, at least a part of the signal processing circuit 34 can be removed, and the CPU 15A can perform the functions of the signal processing circuit 34.

[0203] Further, in the above-described first embodiment, an example of a method of supplying a shooting timing signal from the photoelectric conversion element driver 32 to the photoelectric conversion element 26 has been described, but the technology of the present invention is not limited thereto. For example, the photoelectric conversion element driver 32 may be removed, and in this case, the CPU 15A may perform the function of the photoelectric conversion element driver 32.

[0204] Further, in the above-described first embodiment, as an example of "image-using ranging" related to the technology of the present invention, the second ranging, that is, phase difference image-using ranging for measuring a distance based on a phase difference image obtained from phase difference pixels, has been exemplified, but the technology of the present invention is not limited thereto. For example, instead of ranging using phase difference pixels, ranging using a stereo camera, that is, stereo image-using ranging for measuring a distance based on a stereo image, may be performed, or ranging using face detection, that is, object image-using ranging for measuring a distance based on an object image showing an object of a known size, may be performed. In the ranging using a stereo camera, the distance to the subject is measured by using the parallax of a pair of images (stereo images obtained by stereo shooting) obtained from the stereo camera. Further, in the ranging using face detection, the distance to the subject is measured by using, for example, the ratio of the size of the detected face image to the size of the image for one frame. Here, the size of the face image has been exemplified, but the technology of the present invention is not limited thereto, and any object image showing an object of a known size (for example, a specific car) may be used. Even in the case of using such stereo image-using ranging and / or object image-using ranging, the same effects as those obtained in the case of using phase difference image-using ranging can be obtained.

[0205] Further, in the above-described first embodiment, an example in which the G filter, the R filter, and the B filter also function as infrared cut-off filters for cutting off infrared light has been described, but the technology of the present invention is not limited thereto, and each color filter corresponding to the R pixel, the G pixel, and the B pixel may be a color filter that also transmits infrared light, and a pair of photodiodes based on a visible light pixel photodiode and an IR pixel photodiode (for example, InGaAs APD) may be arranged for one color filter.

[0206] Further, in the above-described first embodiment, an example in which the visible light image is divided into the divided area 26N1 and the second distance measurement system divided area 26N3 has been described, but the technology of the present invention is not limited thereto. For example, instead of the visible light image divided area 26N1 and the second distance measurement system divided area 26N3, an area sensor that selectively generates and reads visible light pixel data and phase difference pixel data may be provided. At this time, a plurality of photosensitive pixels are arranged two-dimensionally in the area sensor. As the photosensitive pixels included in the area sensor, for example, a pair of independent photodiodes without a light shielding member are used. When generating and reading visible light pixel data, photoelectric conversion is performed through the entire area of the photosensitive pixel (a pair of photodiodes), and when generating and reading phase difference pixel data (for example, when performing passive distance measurement), photoelectric conversion is performed through one of the pair of photodiodes. Here, one of the pair of photodiodes is a photodiode corresponding to the first phase difference pixel L described in the above-described first embodiment, and the other photodiode of the pair of photodiodes is a photodiode corresponding to the second phase difference pixel R described in the above-described first embodiment. In addition, it may be configured to selectively generate and read visible light pixel data and phase difference pixel data through all the photosensitive pixels included in the area sensor, but it is not limited thereto, and it may also be configured to selectively generate and read visible light pixel data and phase difference pixel data through a part of the photosensitive pixels included in the area sensor.

[0207] Further, in the above-described first embodiment, an example in which the timing of receiving the IR reflected light having the second largest intensity in the time series distribution by a specific IR pixel is used as the first distance measurement light receiving timing has been described, but the technology of the present invention is not limited thereto. For example, instead of using the light receiving timing of the IR reflected light having an intensity exceeding the first threshold ( Figure 16 in the example shown, the specular IR reflected light) and the noise light having an intensity less than the second threshold as the first distance measurement light receiving timing, the IR reflected light having an intensity less than the first threshold and equal to or greater than the second threshold ( Figure 16 in the example shown, the object subject IR reflected light) may be used as the first distance measurement light receiving timing. At this time, similar to the above-described first embodiment, the first distance measurement based on the light receiving timing of the specular IR reflected light and the first distance measurement based on the noise light are avoided.

[0208] Further, in the above-described first embodiment, an example is given as Figure 23 shown, after performing the first distance measurement, Figure 24AThe method example of the second distance measurement in step ST208 has been described, but the technology of the present invention is not limited thereto. For example, the CPU 15A may perform the first distance measurement and the second distance measurement in parallel, and perform a process including a process based on the distance measurement result of the second distance measurement as a specific process. Here, as a process based on the distance measurement result of the second distance measurement, for example, it is possible to cite imaging with focus control accompanying the second distance obtained by performing the second distance measurement and / or a process of causing the display 46 to display the second distance. In this way, by performing the first distance measurement and the second distance measurement in parallel and performing a process including a process based on the distance measurement result of the second distance measurement as a specific process, compared with the case where the second distance measurement is performed after the first distance measurement, it is possible to perform the process of the second distance, which is the distance measurement result based on the second distance measurement, earlier.

[0209] Moreover, in the above first embodiment, the first threshold and the second threshold are set without considering the noise component based on ambient light, but the technology of the present invention is not limited thereto, and the first threshold and the second threshold may also be set according to the noise component based on ambient light. At this time, for example, as the upper limit value of the intensity of the IR light included in the ambient light, the first threshold and the second threshold may be set so as to be higher than the value obtained in advance through experiments based on the actual machine and / or computer simulations. Therefore, according to this configuration, compared with the case where the first threshold and the second threshold are set without considering the noise component based on ambient light, it is possible to avoid the situation where the noise component based on ambient light is erroneously detected as the IR reflected light of the object to be photographed.

[0210] [Second Embodiment]

[0211] In the above first embodiment, an example of a method of performing the second imaging process based on the distance measurement result of the second distance measurement, which is image-using distance measurement, has been described. However, in this second embodiment, a case where imaging accompanying contrast AF is performed instead of the second imaging process will be described. In addition, in this second embodiment, the same reference numerals are assigned to the constituent elements that are the same as those described in the above first embodiment, and the description thereof is omitted, and only the parts different from the above first embodiment will be described.

[0212] As an example, as Figure 25 shown, in the intelligent device 500 according to this second embodiment, a distance measurement imaging process program 570 is stored in the storage device 15B. The CPU 15A reads out the distance measurement imaging process program 570 from the storage device 15B. Then, the CPU 15A executes the distance measurement imaging process program 570 read out from the storage device 15B, and thus operates as the first distance measurement control unit 15A1, the first distance acquisition unit 15A2, the time series distribution acquisition unit 15A3, the determination unit 15A4, the execution unit 15A5, the focus position calculation unit 15A6, and the contrast AF method imaging control unit 15A7.

[0213] As an example, as Figure 26 shown, when the contrast AF mode imaging control unit 15A7 performs imaging in the contrast AF mode using the light receiver 18, it outputs an imaging start signal to the light receiver 18 and the focus position calculation unit 15A6. By outputting the imaging start signal to the light receiver 18, the contrast AF mode imaging control unit 15A7 causes the visible light image divided area 26N1 (refer to Figure 5 ) to perform imaging at a specified frame rate (e.g., 60 fps), and causes the visible light pixel data processing circuit 34A to generate a third visible light image representing the imaging area. The visible light pixel data processing circuit 34A outputs the third visible light image to the image memory 42. In the image memory 42, the third visible light image is stored, and each time the third visible light image is input from the visible light pixel data processing circuit 34A, the third visible light image in the image memory 42 is updated. In addition, the third visible light image is an example of the "subject image" related to the technology of the present invention.

[0214] Moreover, in parallel with the output of the imaging start signal, the contrast AF mode imaging control unit 15A7 outputs a motor control signal to the light receiver 18 and the focus position calculation unit 15A6. The motor control signal is a signal for causing the focusing lens 30B to reciprocate (oscillate) along the optical axis L2, and is input to the motor driver 64 of the light receiver 18. The motor driver 64 causes the focusing lens 30B to oscillate by driving the motor 62 according to the input motor control signal (refer to Figure 8 ). On the other hand, the focus position calculation unit 15A6 calculates the current position of the focusing lens 30B on the optical axis L2 using the motor control signal input from the time when the imaging start signal is input from the contrast AF mode imaging control unit 15A7 to the current time point.

[0215] Each time the third visible light image in the image memory 42 is updated, the contrast AF mode imaging control unit 15A7 acquires the third visible light image from the image memory 42 and calculates the contrast value of the acquired third visible light image. Then, the contrast AF mode imaging control unit 15A7 searches for the maximum value of the contrast of the third visible light image, and at the time when the maximum value is found, outputs a maximum value reached signal to the focus position calculation unit 15A6. The maximum value reached signal is a signal indicating that the contrast of the third visible light image has reached the maximum value.

[0216] When the maximum value reached signal is input, the focus position calculation unit 15A6 calculates the current position of the focusing lens 30B on the optical axis L2 as the focus position using the motor control signal input from the time when the imaging start signal is input to the current time point. The focus position refers to the position where the subject included in the imaging area is in focus ( Figure 26In the example shown, the focus position of the object subject 98). Here, the focus position calculation unit 15A6 calculates the focus position using an arithmetic expression with the motor control signal as the independent variable and the focus position as the dependent variable. Additionally, the technology of the present invention is not limited to this, and the focus position calculation unit 15A6 can also derive the focus position using a table that establishes a correspondence between the time series data of the motor control signal and the position of the focusing lens 30B on the optical axis L2.

[0217] As an example, as Figure 27 shown, compared with the first embodiment described above, the difference in the specific process is that it includes a third imaging process instead of the second imaging process. The third imaging process refers to a process of performing imaging (imaging accompanied by contrast AF) with focus control for the focus position. Specifically, in the third imaging process, the execution unit 15A5 instructs the contrast AF mode imaging control unit 15A7 to perform imaging accompanied by contrast AF, and the contrast AF mode imaging control unit 15A7 causes the light receiver 18 to perform imaging accompanied by contrast AF according to the instruction from the execution unit 15A5.

[0218] Moreover, the image selection screen generated and displayed on the display 46 by executing the image selection screen display process included in the specific process by the execution unit 15A5 has some differences in the display content compared with the first embodiment described above. That is, in the image selection screen, instead of the second distance, the focus position calculated by the focus position calculation unit 15A6 is displayed ( Figure 27 in the example shown, the numerical value "Xmm" representing the distance from the reference position to the focusing lens 30B). The reference position is, for example, the position of the imaging surface of the photoelectric conversion element 26 or the position of the focusing lens 30B in the state of infinite focus.

[0219] And, in Figure 27 the example shown, the third visible light image is displayed instead of the second visible light image in the image selection screen. And, in the image selection screen, the message "active AF mode" is displayed instead of the message "active mode". Moreover, the message "contrast AF mode" is displayed instead of the message "passive mode" in the image selection screen. Additionally, in Figure 27 the example shown, the first distance "1.8m" is displayed in association with the first visible light image, but the technology of the present invention is not limited to this, and instead of the first distance or together with the first distance, the focus position derived from the first distance, that is, the distance from the reference position to the focusing lens 30B, can also be displayed.

[0220] In addition, in the second embodiment, display examples of the "active AF mode" and the "contrast AF mode" are shown. However, it is not necessary to display the "active AF mode" and the "passive AF mode". As long as the user can understand the difference in the AF modes, any display is acceptable. For example, when performing AF using laser ranging and contrast AF, it can be the display of "AF using laser ranging" and "AF using contrast", or an icon indicating the AF mode can be displayed. Also, the focus position can be displayed instead of the display of the AF mode. For example, it can be "Focus position: near" and "Focus position: far", or it can be displayed as "Focus position: object" and "Focus position: image of the object", etc. Or, two or more of the characters and icons indicating the AF mode and the focus position can be combined for display.

[0221] As described above, in the intelligent device 500, as a specific process, the execution unit 15A5 performs imaging (the third imaging process) accompanied by contrast AF. Therefore, according to this structure, when the time-series distribution generated by the time-series distribution generation unit 34B2 is a specific time-series distribution, imaging accompanied by contrast AF is performed, so that the needs of users who prefer the contrast AF mode to the active AF mode can be satisfied.

[0222] In addition, in the above embodiments, the imaging area included within the field of view angle θ1 is used as the ranging object. However, the technology of the present invention is not limited thereto. For example, the imaging area used as the ranging object can also be a specific real space area defined according to an instruction issued by the user via the receiving device 47. Thereby, ranging can be performed on the real space area desired by the user. Here, an example of the method of defining the imaging area according to an instruction issued by the user via the receiving device 47 is given. However, the technology of the present invention is not limited thereto. For example, the imaging area can also be defined according to an instruction issued by an external device (not shown) capable of communicating with the intelligent device 10.

[0223] Also, according to an instruction issued by the user or an external device, the CPU 15A can selectively define the ranging object area for the first ranging and the ranging object area for the second ranging. As a method of defining the ranging object area for the first ranging, a method of adjusting the irradiation angle θ2 and / or a method of defining the range of IR pixels used within the divided area 26N2 of the first ranging system can be cited. And, as a method of defining the ranging object area for the second ranging, a method of adjusting the field of view angle θ1 and / or a method of defining the range of phase difference pixels used within the divided area 26N3 of the second ranging system can be cited.

[0224] Further, the imaging area to be distance-measured can be a specific real space area corresponding to the object subject image detected from the visible light image by the CPU 15A (for example, an image representing the object subject 98). At this time, for example, the first distance measurement and / or the second distance measurement can be performed on the real space area corresponding to the object subject image detected by performing image recognition processing using machine learning (for example, deep learning). Further, the object subject image can be an image representing the face of a specific person, or an image representing a vehicle such as a car or an aircraft, as long as it is an image representing an object.

[0225] In this way, by performing distance measurement on the specific real space area corresponding to the object subject image detected from the visible light image by the CPU 15A, it is possible to easily perform distance measurement on the object subject as compared with the case where the user visually locates the object subject and performs distance measurement on the object subject.

[0226] Further, in each of the above embodiments, as an example, in Figure 14 the time series distribution shown, the light reception timing of the specular IR reflected light is earlier than the light reception timing of the object subject IR reflected light, but it is not limited thereto. For example, as in Figure 28 shown, even when the light reception timing of the object subject IR reflected light is earlier than the light reception timing of the specular IR reflected light, the technology of the present invention is also applicable. Even in this case, the light reception timing of the IR reflected light having the second largest intensity in the time series distribution is used as the light reception timing for the first distance measurement.

[0227] Further, in each of the above embodiments, the determination of whether the time series distribution is a specific time series distribution is performed by using the first threshold value and the second threshold value (refer to Figure 16 ), but the technology of the present invention is not limited thereto. For example, the intensity of the time series distribution can be adjusted according to the light reception timing. At this time, as an example, as in Figure 29 shown, by applying a filter to the time series distribution (in the example shown in Figure 29 configured to gradually increase the intensity as the light reception timing is delayed), as an example, as in Figure 30 shown, the time series distribution is adjusted. Thereby, as compared with the case where the time series distribution is affected by the light reception timing, it is possible to accurately determine the distance measurement result caused by the material of the object to be distance-measured from the time series distribution.

[0228] In addition, when filtering the time series distribution in this way, the first threshold value and the second threshold value applicable to the time series distribution after filtering change according to the filter. Along with this, the determination method as to whether the time series distribution conforms to the specific time series distribution also changes. For example, in the first embodiment described above, the magnitude relationship between the first threshold value and the second threshold value is "first threshold value < second threshold value". If these first threshold value and second threshold value are applied to Figure 30 the time series distribution shown, and if the time series distribution includes an intensity greater than the second threshold value ( Figure 30 in the example shown, the IR reflected light of the object subject), and includes an intensity that is less than the second threshold value and equal to or greater than the first threshold value ( Figure 30 in the example shown, the specular IR reflected light), the determination unit 15A4 determines that the time series distribution after filtering is the specific time series distribution.

[0229] In addition, the degree of eliminating the influence of the light reception timing can also be dispersed by the filter (refer to Figure 29 ). For example, half of the influence of the light reception timing can be eliminated by the filter, and half of the influence of the light reception timing can be eliminated by the first threshold value.

[0230] In the first embodiment described above, an example in which the distance measurement imaging processing program 70 is stored in the storage device 15B has been described. In the second embodiment described above, an example in which the distance measurement imaging processing program 570 is stored in the storage device 15B has been described. However, the technology of the present invention is not limited thereto. For example, as Figure 31 shown, the distance measurement imaging processing program 70 or 570 (hereinafter, when it is not necessary to distinguish between the distance measurement imaging processing program 70 and 570 for description, it is referred to as the "distance measurement imaging processing program" without reference numerals) may be stored in the storage medium 900. As an example of the storage medium 900, any portable storage medium such as an SSD or a USB memory can be cited.

[0231] The distance measurement imaging processing program stored in the storage medium 900 is installed in the controller 15. The CPU 15A executes the distance measurement imaging processing according to the distance measurement imaging processing program.

[0232] In addition, the distance measurement imaging processing program may be stored in the storage unit of another computer or server device etc. connected to the controller 15 via a communication network (not shown), and the distance measurement imaging processing program may be downloaded and installed in the controller 15 according to the request of the intelligent device 10.

[0233] In addition, it is not necessary to store all of the distance measurement imaging processing program in the storage unit or the storage device 15B of another computer or server device etc. connected to the controller 15, and a part of the distance measurement imaging processing program may be stored.

[0234] In Figure 31 the example shown, a way of having the controller 15 built in the intelligent device 10 is shown, but the technology of the present invention is not limited thereto. For example, the controller 15 may also be provided outside the intelligent device 10.

[0235] In Figure 31 the example shown, the CPU 15A is a single CPU, but it may also be multiple CPUs. Also, a GPU may be applied instead of the CPU 15A.

[0236] In Figure 31 the example shown, the controller 15 is illustrated, but the technology of the present invention is not limited thereto. Devices including ASIC, FPGA, and / or PLD may be applied instead of the controller 15. Also, a combination of a hardware structure and a software structure may be used instead of the controller 15.

[0237] As the hardware resources for executing the distance measurement imaging process described in each of the above embodiments, various processors shown below can be used. As the processor, for example, a general-purpose processor, i.e., a CPU, which functions as the hardware resources for executing the distance measurement imaging process by executing software, i.e., a program, can be cited. Also, as the processor, for example, a dedicated circuit such as an FPGA, a PLD, or an ASIC, which is a processor having a circuit structure specifically designed for executing a specific process, can be cited. A memory is built in or connected to any processor, and any processor executes the distance measurement imaging process by using the memory.

[0238] The hardware resources for executing the distance measurement imaging process may be constituted by one of these various processors, or may be constituted by a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Also, the hardware resources for executing the distance measurement imaging process may be one processor.

[0239] As an example of being constituted by one processor, first, there is a way in which a combination of one or more CPUs and software constitutes one processor, and this processor functions as the hardware resources for executing the distance measurement imaging process. Second, there is a way such as represented by an SoC, in which a processor that uses one IC chip to implement the entire system including multiple hardware resources for executing the distance measurement imaging process is used. Thus, the distance measurement imaging process is implemented by using one or more of the above various processors as the hardware resources.

[0240] Moreover, as the hardware structure of these various processors, more specifically, a circuit combining circuit elements such as semiconductor elements can be used. And the above distance measurement imaging process is merely an example. Therefore, of course, unnecessary steps can be deleted, new steps can be added, or the processing order can be replaced without departing from the gist.

[0241] The description and illustration shown above are a detailed description of the part related to the technology of the present invention and are merely an example of the technology of the present invention. For example, the description related to the above structure, function, action, and effect is a description related to an example of the structure, function, action, and effect of the part related to the technology of the present invention. Therefore, it is undoubted that within the scope not departing from the technical purpose of the present invention, the description and illustration shown above can be deleted of unnecessary parts, or new elements can be added, or replacements can be made. And, in order to avoid complication and facilitate understanding of the part related to the technology of the present invention, in the description and illustration shown above, the description related to common technical knowledge that does not require special explanation in the implementation of the technology of the present invention is omitted.

[0242] In this specification, the meaning of "A and / or B" is the same as "at least one of A and B". That is, "A and / or B" means that it can be only A, or only B, or a combination of A and B. And, in this specification, when expressing three or more matters combined with "and / or", the same concept as "A and / or B" is also applicable.

[0243] Regarding all the documents, patent applications, and technical specifications described in this specification, they are incorporated into this specification by reference in the same manner as the case where each document, patent application, and technical specification incorporated by reference is specifically and separately described.

[0244] Regarding the above embodiments, the following appendices are also disclosed.

[0245] (Appendix 1)

[0246] An information processing apparatus, comprising:

[0247] a processor; and

[0248] a memory connected to or built in the above processor,

[0249] The processor performs the following processing:

[0250] In a time series distribution, when the number of times the reflected light is received by the light receiver is 1 time, the imaging device is focused by using the first distance measured according to the irradiation timing and the light reception timing. When the number of times the reflected light is received by the light receiver is multiple times in the time series distribution, the imaging device is focused by using the second distance measured from the image obtained by photographing the ranging object area. In the time series distribution, the intensity of the reflected light received by the light receiver at a prescribed time interval within the light reception period corresponding to the surface irradiation of the light based on the light irradiator is specified in time series.

[0251] (Appendix 2)

[0252] A ranging device, comprising:

[0253] A processor;

[0254] A light irradiator that performs surface irradiation of light toward a ranging target area; and

[0255] A light receiver that receives reflected light from the ranging target area,

[0256] The processor performs the following processing:

[0257] Measure a first distance to an object in the ranging target area based on the irradiation timing of the surface irradiation of light by the light irradiator toward the ranging target area and the light reception timing of the reflected light from the ranging target area received by the light receiver,

[0258] Based on the light reception timing and irradiation timing corresponding to a signal selected according to the relationship and intensity of a plurality of signals generated by the light receiver at a plurality of light reception timings during the light reception period corresponding to the surface irradiation based on the light irradiator, the processor measures the first distance.

[0259] (Appendix 3)

[0260] An information processing device, comprising:

[0261] A processor; and

[0262] A memory connected to or built into the above-mentioned processor,

[0263] The processor performs the following processing:

[0264] Measure a first distance to an object in the ranging target area based on the irradiation timing of the surface irradiation of light by the light irradiator toward the ranging target area and the light reception timing of the reflected light from the ranging target area received by the light receiver,

[0265] Based on the light reception timing and irradiation timing corresponding to a signal selected according to the relationship and intensity of a plurality of signals generated by the light receiver at a plurality of light reception timings during the light reception period corresponding to the surface irradiation based on the light irradiator, the processor measures the first distance.

[0266] Symbol Explanation

[0267] 10. 500 - Intelligent device, 12 - housing, 12A - back surface, 12B - front surface, 13 - indicator key, 14 - ranging camera device, 15 - controller, 15A - CPU, 15A1 - First ranging control unit, 15A2 - First distance acquisition unit, 15A3 - Time - series distribution acquisition unit, 15A4 - Determination unit, 15A5 - Execution unit, 15A6 - Focus position calculation unit, 15A7 - Contrast AF mode camera control unit, 15A8 - Focus position calculation unit, 15B - Storage device, 15C - Memory, 16 - Light irradiator, 17A, 17B - Light - shielding members, 18 - Light receiver, 19 - Microlens, 20, 22, 352 - Translucent window, 21 - Beam expander, 23 - Collimating lens, 24 - LD, 25 - LD driver, 26, 354 - Photoelectric conversion element, 26N1 - Division area for visible - light image, 26N2 - First ranging system division area, 26N3 - Second ranging system division area, 27 - TOF camera, 30A - Objective lens, 30B - Focusing lens, 30C - Aperture, 31 - Focus control mechanism, 32 - Photoelectric conversion element driver, 34 - Signal processing circuit, 34A - Visible - light pixel data processing circuit, 34B - First ranging system processing circuit, 34B1 - IR pixel data acquisition unit, 34B2 - Time - series distribution generation unit, 34B3 - Light - receiving timing determination unit, 34B4 - First distance measurement unit, 34C - Second ranging system processing circuit, 40 - Input / output interface, 41 - Imaging lens, 42 - Image memory, 44 - UI system device, 46 - Display, 47 - Receiving device, 48 - Touch panel, 50 - Bus, 52 - External I / F, 53 - Hard key section, 54 - Communication I / F, 56 - Network, 59 - Touch panel display, 60 - Moving mechanism, 62 - Motor, 64 - Motor driver, 70, 570 - Ranging camera processing program, 72 - Focus position derivation table, 100 - Mirror, 100A - Mirror surface, 300L - Left - region passing light, 300R - Right - region passing light, 900 - Storage medium, L - First phase - difference pixel, L1, L2 - Optical axis, N - Non - phase - difference pixel, PD - Photodiode, R - Second phase - difference pixel, α - Offset, θ1 - Field of view angle, θ2 - Irradiation angle.

Claims

1. An information processing apparatus, comprising: a processor; and a memory connected to or built in the processor, wherein the processor performs the following processing: measuring a first distance to an object within the distance measurement target area based on an irradiation timing at which light is surface-irradiated toward the distance measurement target area by a light irradiator and a light reception timing at which the light reflected from the distance measurement target area is received by a light receiver, when, during a light reception period corresponding to the surface irradiation, a plurality of signals are generated by a specific pixel included in the light receiver at different timings, the processor measures the first distance based on a relationship of the plurality of signals including a first signal having a first intensity equal to or greater than a first threshold and a second signal having a second intensity less than the first threshold and equal to or greater than a second threshold smaller than the first threshold generated by the specific pixel, wherein the first threshold is a value defined as a lower limit value of the intensity of the reflected light from a glossy object, wherein the second threshold is a value defined as a lower limit value of the intensity of the reflected light from a standard subject assuming the object, when the plurality of signals include the first signal and the second signal, the processor performs a specific process, wherein the specific process includes a process of measuring the first distance based on the light reception timing at which the reflected light corresponding to the second signal is received by the specific pixel.

2. The information processing apparatus according to claim 1, wherein the relationship of the plurality of signals is a time series distribution of the intensities of the plurality of signals.

3. The information processing apparatus according to claim 1, wherein when the plurality of signals include a plurality of signals having intensities exceeding a reference threshold, the processor measures the first distance based on the light reception timing at which the reflected light corresponding to the signal having the second largest intensity among the plurality of signals is received by the light receiver.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the first threshold and the second threshold are defined based on a noise component of ambient light.

5. The information processing apparatus according to any one of claims 1 to 3, wherein the first threshold is a value that decreases as the light reception timing is delayed.

6. The information processing apparatus according to any one of claims 1 to 3, wherein the specific process includes a process of notifying that the first intensity is included in the intensities of the plurality of signals.

7. The information processing apparatus according to any one of claims 1 to 3, wherein the specific process includes image-based distance measurement of measuring a second distance to the object based on an image obtained by capturing the distance measurement target area with a first camera.

8. The information processing apparatus according to any one of claims 1 to 3, wherein the processor performs image-based distance measurement of measuring a second distance to the object based on an image obtained by capturing the distance measurement target area with a second camera in parallel with an operation of measuring the first distance based on the irradiation timing and the light reception timing, and the specific process includes a process based on a distance measurement result of the image-based distance measurement.

9. The information processing apparatus according to claim 7, wherein the image-using distance measurement is at least one of phase-difference image-using distance measurement for measuring the second distance based on a phase-difference image obtained from phase-difference pixels as the image, stereo-image-using distance measurement for measuring the second distance based on a stereo image obtained by a stereo imaging method as the image, and object-image-using distance measurement for measuring the second distance based on an object image detected from the image and representing an object with a known size.

10. The information processing apparatus according to any one of claims 1 to 3, wherein the specific processing includes focus control of the third imager based on the contrast of a subject image obtained by photographing a subject included in the distance measurement target area by the third imager.

11. The information processing apparatus according to any one of claims 1 to 3, wherein the distance measurement target area is a specific real-space area defined according to a given instruction.

12. The information processing apparatus according to any one of claims 1 to 3, wherein the distance measurement target area is a specific real-space area corresponding to an object subject image detected by the processor from a captured image obtained by photographing by a fourth imager.

13. The information processing apparatus according to any one of claims 1 to 3, wherein the processor performs focus control of a fifth imager by using the first distance measured based on the irradiation timing and the light reception timing.

14. The information processing apparatus according to any one of claims 1 to 3, wherein the intensity of the plurality of signals is adjusted according to the light reception timing.

15. The information processing apparatus according to any one of claims 1 to 3, wherein the light receiver has a plurality of the specific pixels arranged in a two-dimensional shape, and the processor measures the first distance for the plurality of specific pixels based on the irradiation timing and the light reception timing.

16. The information processing apparatus according to any one of claims 1 to 3, wherein the distance measurement target area includes the object and the glossy object.

17. An imaging apparatus, comprising: the information processing apparatus according to any one of claims 1 to 16; and a focusing lens, wherein the processor performs focus control to move the focusing lens to a focus position determined based on the first distance.

18. An information processing method, comprising the following steps: A light irradiator performs surface irradiation of light toward a distance measurement target area; A light receiver receives reflected light of the light from the distance measurement target area; and The first distance to an object in the distance measurement target area is measured based on the irradiation timing when the light irradiator performs the surface irradiation of the light toward the distance measurement target area and the light reception timing when the light receiver receives the reflected light. During a light receiving period corresponding to the surface illumination, when a plurality of signals are generated at different timings by specific pixels included in the light receiver, the first distance is measured based on the relationship of the plurality of signals including a first signal having a first intensity equal to or greater than a first threshold value and a second signal having a second intensity less than the first threshold value and equal to or greater than a second threshold value smaller than the first threshold value, which are generated by the specific pixels. The first threshold value is a value defined as a lower limit value of the intensity of the reflected light from a glossy object. The second threshold value is a value defined as a lower limit value of the intensity of the reflected light from a standard subject assumed to be the object. The method further includes a step of performing a specific process when the first signal and the second signal are included in the plurality of signals. The specific process includes a process of measuring the first distance based on the light receiving timing at which the specific pixel receives the reflected light corresponding to the second signal.

19. A computer-readable storage medium storing a program for causing a computer to execute a process including the following steps: A light irradiator performs surface illumination of light toward a distance measurement target area; A light receiver receives the reflected light from the distance measurement target area of the light; and The first distance to an object in the distance measurement target area is measured based on the irradiation timing at which the light irradiator performs the surface illumination of the light toward the distance measurement target area and the light receiving timing at which the light receiver receives the reflected light. During a light receiving period corresponding to the surface illumination, when a plurality of signals are generated at different timings by specific pixels included in the light receiver, the first distance is measured based on the relationship of the plurality of signals including a first signal having a first intensity equal to or greater than a first threshold value and a second signal having a second intensity less than the first threshold value and equal to or greater than a second threshold value smaller than the first threshold value, which are generated by the specific pixels. The first threshold value is a value defined as a lower limit value of the intensity of the reflected light from a glossy object. The second threshold value is a value defined as a lower limit value of the intensity of the reflected light from a standard subject assumed to be the object. The process further includes a step of performing a specific process when the first signal and the second signal are included in the plurality of signals. The specific process includes a process of measuring the first distance based on the light receiving timing at which the specific pixel receives the reflected light corresponding to the second signal.

Images