Processing device, electronic device, processing method, and program

Abstract

To provide a processing apparatus, electronic equipment, a processing method, and a program that can reduce an influence of reflection light for distance measurement on an imaging region image as compared with a case in which irradiation energy of light emitted for distance measurement is determined independently of a reflection factor in an imaging region.SOLUTION: A processing apparatus includes: a control part which performs control to perform an imaging operation to image an imaging region by an imaging part and a distance measuring operation to measure a distance by irradiating the imaging region with light by a distance measuring part and then receiving reflected light of the light to the imaging region; an acquisition part which acquires reflection factor information enabling the reflection factor in the imaging region to be specified; and a change part which changes the irradiation energy of the light according to the reflection factor information acquired by the acquisition part while the imaging operation and distance measuring operation are carried out.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The technology disclosed herein relates to a processing device, an electronic device, a processing method, and a program. [Background technology]

[0002] Japanese Patent No. 6321145 discloses a distance measuring device. The distance measuring device described in Japanese Patent No. 6321145 includes an image capturing unit that captures an image of a subject formed by an imaging optical system that captures an image of a subject, an emission unit that emits directional light that is light with directionality along the optical axis direction of the imaging optical system, the emission unit being adjustable in emission intensity of the directional light and emitting the directional light by adjusting the emission intensity based on at least one of focus state specification information and subject brightness or exposure state specification information, a light receiving unit that receives reflected light of the directional light from the subject, a derivation unit that derives the distance to the subject based on the timing at which the directional light is emitted by the emission unit and the timing at which the reflected light is received by the light receiving unit, an execution unit that performs at least one of focus adjustment of the imaging optical system on the subject and exposure adjustment prior to shooting by the image capturing unit, and a control unit that controls the focus adjustment and exposure adjustment for the execution unit. and a control unit that, when the first pressing operation is accepted by the acceptance unit, controls the execution unit to execute at least one of focus adjustment and exposure adjustment, controls the emission unit, the light receiving unit, and the derivation unit to start distance measurement on the condition that at least one of focus adjustment and exposure adjustment has been executed, and after the distance measurement is completed, causes a presentation unit that presents information to present information regarding the result of the distance measurement while the first pressing operation is maintained, and then controls the photographing unit to perform the actual exposure if the acceptance unit accepts a second pressing operation successively from the first pressing operation without releasing the pressing operation on the acceptance unit.

[0003] Japanese Patent Laid-Open Publication No. 2006-171120 discloses a photographing device. The photographing device described in Japanese Patent Laid-Open Publication No. 2006-171120 detects subject contrast and adjusts focus prior to photographing, and is characterized by including a first fill light emitter that irradiates the subject with light having a relatively wide wavelength band for subject contrast detection, a second fill light emitter that irradiates the subject with light having a relatively narrow wavelength band for subject contrast detection, and a switching means that switches between the first fill light emitter and the second fill light emitter.

[0004] WO 2018 / 142993 discloses a light emission control device. The light emission control device described in WO 2018 / 142993 includes a light amount setting unit that sets the amount of light of an AF (Auto Focus) assist light, and a light emission control unit that controls the emission of the AF assist light according to the setting by the light amount setting unit. Summary of the Invention

[0005] One embodiment of the technology disclosed herein provides a processing device, electronic device, processing method, and program that can reduce the effect of reflected light for distance measurement on an imaging area image, compared to a case in which the irradiation energy of light irradiated for distance measurement is determined independently of the reflectance within the imaging area. [Means for solving the problem]

[0006] A first aspect of the technology of the present disclosure is a processing device that includes a control unit that controls an imaging operation by an imaging unit that images an imaging area and a distance measurement operation in which a distance measurement unit irradiates light onto the imaging area and measures distance by receiving reflected light from the light onto the imaging area, an acquisition unit that acquires reflectance information that can identify the reflectance within the imaging area, and a change unit that changes the light irradiation energy in accordance with the reflectance information acquired by the acquisition unit while the imaging operation and distance measurement operation are being performed.

[0007] A second aspect of the technique of the present disclosure is the processing device according to the first aspect, in which the reflectance information is information based on an imaging area image obtained by imaging the imaging area by an imaging section.

[0008] A third aspect of the technology of the present disclosure is the processing device according to the second aspect, in which the information based on the imaging area image is an image recognition result obtained by performing image recognition on the imaging area image by a recognition unit.

[0009] A fourth aspect of the technology disclosed herein is a processing device according to the third aspect, in which an acquisition unit acquires an image recognition result based on a learning result obtained by machine learning of a correspondence between an image and a subject recognition result.

[0010] A fifth aspect of the technology disclosed herein is a processing device according to the third or fourth aspect, in which, when the image recognition result acquired by the acquisition unit indicates that the imaging area image contains an image showing an object with a reflectance equal to or higher than the reference reflectance, the change unit weakens the irradiation energy below the first reference irradiation energy.

[0011] A sixth aspect of the technique of the present disclosure is the processing device according to the fifth aspect, in which the object is an object that is predetermined as an object having glossiness.

[0012] A seventh aspect of the technique of the present disclosure is the processing device according to the sixth aspect, in which the predetermined object is a mirror-like object.

[0013] An eighth aspect of the technology of the present disclosure is a processing device relating to any one of the first to seventh aspects, in which a change unit changes the light irradiation energy according to imaging scene instruction information received by a reception unit capable of receiving imaging scene instruction information that instructs an imaging scene and reflectance information acquired by an acquisition unit.

[0014] A ninth aspect of the technology of the present disclosure is a processing device according to the eighth aspect, in which the change unit weakens the irradiation energy below the second standard irradiation energy when the imaging scene indicated by the imaging scene indication information accepted by the acceptance unit is a specific imaging scene having an imaging scene reflectance equal to or greater than the standard reflectance.

[0015] A tenth aspect of the technology of the present disclosure is a processing device relating to any one of the first to seventh aspects, in which a change unit changes the light irradiation energy according to an imaging scene identified by an identification unit capable of identifying the imaging scene based on an imaging area image and reflectance information acquired by an acquisition unit.

[0016] An eleventh aspect of the technology disclosed herein is a processing device according to the tenth aspect, in which the change unit weakens the irradiation energy below the third standard irradiation energy when the imaging scene identified by the identification unit is a specific imaging scene having an imaging scene reflectance equal to or greater than the standard reflectance.

[0017] A twelfth aspect of the technology of the present disclosure is a processing device relating to any one of the first to eleventh aspects, in which a control unit generates auxiliary light as light from a light source and controls a light receiving element to receive auxiliary reflected light due to the auxiliary light on the imaging area, and the reflectance information is information based on a light receiving result in which the auxiliary reflected light is received by the light receiving element.

[0018] A thirteenth aspect of the technology of the present disclosure is a processing device according to the twelfth aspect, in which the information based on the light receiving result is information based on the amount of auxiliary reflected light received by the light receiving element per unit time.

[0019] A fourteenth aspect of the technology disclosed herein is a processing device according to the twelfth or thirteenth aspect, wherein the change unit weakens the irradiation energy below the fourth reference irradiation energy when the reflectance determined by the reflectance information acquired by the acquisition unit is equal to or greater than a threshold value.

[0020] A fifteenth aspect of the technology of the present disclosure is the processing device according to any one of the first to fourteenth aspects, in which the distance measuring unit has a TOF camera.

[0021] A sixteenth aspect of the technique of the present disclosure is the processing device according to the fifteenth aspect, in which the control unit causes the display unit to display the distance image generated by the TOF camera.

[0022] A 17th aspect of the technology disclosed herein is a processing device according to the 15th or 16th aspect, in which the TOF camera has a plurality of photoelectric conversion pixels, and distance measurement is performed using reflected light received only by at least one photoelectric conversion pixel that is included within a specified area of ​​the plurality of photoelectric conversion pixels.

[0023] An 18th aspect of the technology disclosed herein is a processing device related to the 17th aspect, in which at least one photoelectric conversion pixel is a photoelectric conversion pixel at a position corresponding to the position of an image area specified from within a screen in a state in which an imaging area image is displayed.

[0024] A 19th aspect of the technology disclosed herein is a processing device relating to any one of the 1st to 18th aspects, in which the change unit changes the irradiation energy in accordance with the timing of the main exposure performed during the imaging operation.

[0025] A twentieth aspect of the technology of the present disclosure is a processing device relating to any one of the first to nineteenth aspects, in which the imaging operation includes an imaging operation for a live view image in which the imaging unit captures an imaging area for a live view image.

[0026] A twenty-first aspect according to the technique of the present disclosure is the processing device according to any one of the first to twentieth aspects, in which the distance measurement operation is a distance measurement operation for focusing by an imaging section.

[0027] A 22nd aspect of the technology of the present disclosure is a processing device related to the 21st aspect, in which an imaging unit focuses on an imaging area based on a distance measurement result obtained by performing distance measurement through a focusing distance measurement operation.

[0028] A twenty-third aspect of the technique of the present disclosure is the processing device according to any one of the first to twenty-second aspects, in which the light is directional light.

[0029] A twenty-fourth aspect of the technique of the present disclosure is an electronic device including a processing device according to any one of the first to twenty-third aspects, an imaging unit, and a distance measuring unit.

[0030] A 25th aspect of the technology of the present disclosure is a processing method that includes controlling an imaging operation by an imaging unit that images an imaging area and a ranging operation by a ranging unit that irradiates light onto the imaging area and measures distance by receiving reflected light from the light onto the imaging area, acquiring reflectance information that can identify the reflectance within the imaging area, and changing the light irradiation energy in accordance with the acquired reflectance information while the imaging operation and ranging operation are being performed.

[0031] A 26th aspect of the technology of the present disclosure is a program for causing a computer to execute processing including controlling an imaging operation by an imaging unit that images an imaging area and a distance measurement operation by a distance measurement unit that irradiates light onto the imaging area and measures distance by receiving light reflected from the light onto the imaging area, acquiring reflectance information that can identify the reflectance within the imaging area, and changing the light irradiation energy in accordance with the acquired reflectance information while the imaging operation and distance measurement operation are being performed. [Brief description of the drawings]

[0032] [Figure 1] 1 is a schematic perspective view showing an example of a usage mode of a smart device according to an embodiment. FIG. [Diagram 2] FIG. 2 is a rear perspective view showing an example of the appearance of the rear side of the smart device according to the embodiment. [Diagram 3] 1 is a schematic perspective view showing an example of the structure of a photoelectric conversion element included in a smart device according to an embodiment. FIG. [Figure 4] 3 is a front perspective view showing an example of the appearance of the front side of the smart device shown in FIG. 2. [Diagram 5] 1 is a conceptual diagram showing an example of an aspect in which an imaging area is captured by a distance measuring imaging device included in a smart device according to an embodiment, and a visible light image is displayed on a display. FIG. [Figure 6] 1 is a conceptual diagram showing an example of a manner in which laser light is irradiated onto an imaging area by a distance measuring imaging device included in a smart device according to an embodiment. FIG. [Figure 7] 1 is a conceptual diagram showing an example of an aspect in which a distance measuring imaging device included in a smart device according to an embodiment measures distance to an imaging area as a distance measurement target, and the distance measurement result is displayed on a display as a distance image. [Figure 8] A conceptual diagram showing an example of an aspect in which a distance measuring imaging device included in a smart device according to an embodiment measures distance to an imaging area as a distance measurement target, and a distance superimposed image reflecting the distance measurement result is displayed on a display. [Figure 9] FIG. 2 is a block diagram showing an example of the configuration of electrical hardware of the smart device according to the embodiment. [Figure 10] FIG. 2 is a conceptual diagram showing an example of the contents of a storage included in a smart device according to an embodiment. [Figure 11] FIG. 11 is a conceptual diagram showing an example of details of distance measuring and imaging processing executed by a CPU included in a smart device according to an embodiment. [Figure 12] FIG. 11 is a functional block diagram showing an example of functions of a CPU when distance measurement and imaging processing is performed by the CPU included in the smart device according to the embodiment. [Figure 13] FIG. 11 is a schematic screen diagram showing an example of a display mode selection screen displayed on a display included in a smart device according to an embodiment. [Figure 14] FIG. 11 is a schematic screen diagram showing an example of a ranging range designation guide screen displayed on a display included in the smart device according to the embodiment. [Figure 15] 1A is a schematic screen diagram showing an example of a visible light image and a designated image region displayed as a live view image on a display included in a smart device according to an embodiment. FIG. [Figure 16] 11 is a conceptual diagram illustrating an example of a method of narrowing down a distance measurement segment area of ​​a photoelectric conversion element included in a smart device to a distance measurement designated segment area. FIG. [Figure 17] 11 is a flowchart illustrating an example of a flow of parallel processing according to the embodiment. [Figure 18] 10 is a flowchart showing an example of a flow of an imaging process according to the embodiment. [Figure 19] 10 is a flowchart showing an example of the flow of distance measuring processing according to the embodiment. [Figure 20] 10 is a flowchart illustrating an example of the flow of a change process according to the embodiment. [Figure 21A] 10 is a flowchart showing an example of the flow of a display control process according to the embodiment. [Figure 21B] This is a continuation of the flowchart shown in FIG. 21A. [Figure 22] FIG. 11 is a functional block diagram showing a first modified example of the functions of a CPU when distance measurement and imaging processing is performed by the CPU included in the smart device according to the embodiment. [Diagram 23] FIG. 11 is a schematic screen diagram showing an example of an imaging scene selection screen displayed on a display included in the smart device according to the embodiment. [Figure 24] 13 is a flowchart showing a first modified example of the flow of the change process according to the embodiment. [Diagram 25] FIG. 11 is a functional block diagram showing a second modified example of the functions of a CPU when distance measurement and imaging processing is performed by the CPU included in the smart device according to the embodiment. [Figure 26] 13 is a flowchart showing a second modified example of the flow of the change process according to the embodiment. [Figure 27] FIG. 11 is a block diagram showing a first modified example of the configuration of electrical hardware of the smart device according to the embodiment. [Figure 28] FIG. 11 is a functional block diagram showing a third modified example of the functions of a CPU when distance measurement and imaging processing is performed by the CPU included in the smart device according to the embodiment. [Figure 29]13 is a flowchart showing a third modified example of the flow of the change process according to the embodiment. [Diagram 30] FIG. 11 is a functional block diagram showing a fourth modified example of the functions of a CPU when distance measurement and imaging processing is performed by the CPU included in the smart device according to the embodiment. [Diagram 31] 13 is a flowchart showing a fourth modified example of the flow of the change process according to the embodiment. [Diagram 32] 13 is a flowchart showing a fifth modified example of the flow of the change process according to the embodiment. [Diagram 33] 10A and 10B are conceptual diagrams showing an example of timing for changing the intensity of distance measuring light emitted by a distance measuring imaging device included in a smart device according to an embodiment. [Diagram 34] FIG. 11 is a schematic perspective view showing a first modified example of the external configuration of the smart device according to the embodiment. [Diagram 35] FIG. 11 is a schematic perspective view showing a second modified example of the external configuration of the smart device according to the embodiment. [Diagram 36] FIG. 1 is a schematic perspective view showing an example of an external configuration of a range-measuring imaging device according to an embodiment in combination with a smart device. [Figure 37] 1 is a conceptual diagram showing an example of a manner in which a distance measuring imaging processing program according to an embodiment is installed in a smart device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Hereinafter, an example of an embodiment of an imaging device according to the technique of the present disclosure will be described with reference to the accompanying drawings.

[0034] First, the terms used in the following description will be explained.

[0035] CPU is an abbreviation for "Central Processing Unit". RAM is an abbreviation for "Random Access Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". A / D is an abbreviation for "Analog / Digital". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". LTE is an abbreviation for "Long Term Evolution." 5G is an abbreviation for "5th Generation." LD is an abbreviation for "Laser Diode." IR is an abbreviation for "Infrared." APD is an abbreviation for "Avalanche Photodiode." TOF is an abbreviation for "Time of Flight." fps is an abbreviation for "frame per second." LED is an abbreviation for "Light Emitting Diode." ROI is an abbreviation for "Region of Interest." LAN is an abbreviation for "Local Area Network."

[0036] In the description of this specification, "horizontal" refers to horizontal in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to being completely horizontal. In the description of this specification, "parallel" refers to parallel in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to being completely parallel. In the description of this specification, "vertical" refers to vertical in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to being completely vertical. In the description of this specification, "same" refers to same in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to being completely same.

[0037] 1, the smart device 10 performs an imaging operation (hereinafter also simply referred to as an "imaging operation") for imaging an imaging area defined by an angle of view θ1, and a distance measurement operation (hereinafter also simply referred to as a "distance measurement operation") for measuring distance by irradiating a laser beam onto the imaging area and receiving reflected light of the laser beam onto the imaging area. The laser beam is an example of the "directional light" according to the technology of the present disclosure.

[0038] In this embodiment, "distance measurement" refers to a process of measuring the distance from the smart device 10 to a distance measurement target in the imaging area. Here, the "distance measurement target" refers to an object that reflects laser light, and in the example shown in Fig. 1, a person and a tree are shown as the distance measurement target in the imaging area. Note that examples of the smart device 10 include a smartphone or a tablet terminal, which is an electronic device with an imaging function.

[0039] As an example, as shown in Fig. 2, a smart device 10 includes a housing 12. The housing 12 houses a distance measuring imager 14. The distance measuring imager 14 includes a light illuminator 16 and a light receiver 18. The light illuminator 16 includes an LD 24, and the light receiver 18 includes a photoelectric conversion element 26. In the smart device 10, the imaging operation and distance measuring operation are performed by the distance measuring imager 14. Note that the distance measuring imager 14 is an example of an "imaging section (imaging device)" and a "distance measuring section (distance measuring device)" according to the technology of the present disclosure.

[0040] The smart device 10 has command keys 13 arranged on a side thereof. The command keys 13 receive various commands. The "various commands" referred to here include, for example, a command to display a menu screen from which various menus can be selected, a command to select one or more menus, a command to confirm the selected content, and a command to erase the selected content.

[0041] When the smart device 10 is placed vertically, the housing 12 has light-transmitting windows 20 and 22 at the upper left of the rear surface 12A (upper left when viewed from the rear of the smart device 10 placed vertically). The light-transmitting windows 20 and 22 are optical elements (e.g., lenses) having light-transmitting properties, and are arranged at a predetermined interval (e.g., an interval of several millimeters) along the horizontal direction and exposed from the rear surface 12A. The light irradiator 16 irradiates the laser light emitted from the LD 24 to the distance measurement target through the light-transmitting window 20. In this embodiment, a laser light in the infrared wavelength range is used. However, the wavelength range of the laser light is not limited to this, and may be a laser light in another wavelength range.

[0042] The light receiver 18 captures the reflected IR light through the light-transmitting window 22. The reflected IR light refers to reflected light caused by the laser light irradiated onto the target to be measured by the light irradiator 16. The light receiver 18 also captures reflected visible light through the light-transmitting window 22. The reflected visible light refers to reflected light caused by visible light (for example, visible light contained in sunlight) irradiated onto the imaging area. For ease of explanation, hereinafter, when there is no need to distinguish between reflected IR light and reflected visible light, they will simply be referred to as "reflected light."

[0043] The light receiver 18 is equipped with a photoelectric conversion element 26, which receives the reflected light that is taken into the light receiver 18 through the light-transmitting window 22, and outputs an electrical signal according to the amount of reflected light received.

[0044] 3, the photoelectric conversion element 26 has a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896×3265" pixels.

[0045] A color filter is disposed on each photodiode included in the photoelectric conversion element 26. The color filters include a G filter corresponding to the G (green) wavelength range that contributes most to obtaining a luminance signal, an R filter corresponding to the R (red) wavelength range, a B filter corresponding to the B (blue) wavelength range, and an IR filter corresponding to the IR (infrared) wavelength range. In this embodiment, the G filter, the R filter, and the B filter also function as infrared light cut filters that cut infrared light. In the following description, for convenience of explanation, when it is not necessary to distinguish between the G filter, the R filter, and the B filter, they are also referred to as "visible light filters."

[0046] The photoelectric conversion element 26 has an R pixel, a G pixel, a B pixel, and an IR pixel. The R pixel is a pixel corresponding to a photodiode in which an R filter is arranged, the G pixel is a pixel corresponding to a photodiode in which a G filter is arranged, the B pixel is a pixel corresponding to a photodiode in which a B filter is arranged, and the IR pixel is a pixel corresponding to a photodiode in which an IR filter is arranged. The R pixel, the G pixel, the B pixel, and the IR pixel are arranged with a predetermined periodicity in each of the row direction (horizontal direction) and the column direction (vertical direction). In this embodiment, the arrangement of the R pixel, the G pixel, the B pixel, and the IR pixel is an arrangement obtained by replacing some of the G pixels in the X-Trans (registered trademark) arrangement with IR pixels. The IR pixel is arranged with a specific periodicity along the row direction and the column direction.

[0047] Note that, here, an arrangement based on an X-Trans arrangement is exemplified as the arrangement of the R pixels, G pixels, B pixels, and IR pixels, but the technology of the present disclosure is not limited to this, and the arrangement of the R pixels, G pixels, B pixels, and IR pixels may be an arrangement based on other arrangements, such as a Bayer arrangement or a Honeycomb (registered trademark) arrangement.

[0048] In addition, an arrangement obtained by replacing some G pixels with IR pixels among an arrangement generally known as an arrangement of R pixels, G pixels, and B pixels is exemplified here as an arrangement of R pixels, G pixels, B pixels, and IR pixels, but the technology of the present disclosure is not limited to this. For example, each color filter corresponding to each of the R pixels, G pixels, and B pixels (hereinafter, these are also referred to as "visible light pixels") may be a color filter that transmits infrared light as well, and a pair of photodiodes, one for visible light pixels and the other for IR pixels (e.g., InGaAs APD), may be arranged for each color filter.

[0049] In this embodiment, the photoelectric conversion element 26 is divided into two regions. That is, the photoelectric conversion element 26 has a visible light image division region 26N1 and a distance measurement division region 26N2. The visible light image division region 26N1 is a visible light pixel group consisting of a plurality of visible light pixels, and is used to generate a visible light image. The distance measurement division region 26N2 is an IR pixel group consisting of a plurality of IR pixels, and is used for distance measurement. The visible light image division region 26N1 receives visible reflected light and outputs an electrical signal according to the amount of received light. The distance measurement division region 26N2 receives IR reflected light and outputs an electrical signal according to the amount of received light.

[0050] As an example, as shown in FIG. 4, a touch panel display 59 is provided on the front surface 12B of the housing 12. The touch panel display 59 includes a display 46 and a touch panel 48. An example of the display 46 is an organic EL display. The display 46 may be other types of displays such as a liquid crystal display or an inorganic EL display instead of an organic EL display. The display 46 is an example of a "display unit (display)" according to the technology of the present disclosure. The touch panel 48 is an example of a "reception unit (reception device (acceptor))" according to the technology of the present disclosure.

[0051] The display 46 displays images (for example, live view images and playback images) and text information. The touch panel 48 is a transmissive touch panel, and is overlaid on the surface of the display area of ​​the display 46. The touch panel 48 receives instructions from a user by detecting contact with a pointer such as a finger or a stylus pen. Note that, here, as an example of the touch panel display 59, an out-cell type touch panel display in which the touch panel 48 is overlaid on the surface of the display area of ​​the display 46 is given, but this is merely one example. For example, an on-cell type or in-cell type touch panel display can also be applied as the touch panel display 59.

[0052] 5, in the smart device 10, when an instruction to start capturing an image is received by the touch panel 48, the optical receiver 18 captures an image of the imaging area. That is, the optical receiver 18 receives visible reflected light and generates a visible light image showing the imaging area as an image corresponding to the received visible reflected light. The visible light image is an example of an "imaging area image" according to the technology of the present disclosure.

[0053] The visible light image is displayed on the display 46 as a live view image or a still image in response to an instruction received by the touch panel 48. In the example shown in Fig. 5, the imaging area is defined by an angle of view θ1. The angle of view θ1 is changed in response to an instruction received by the touch panel 48.

[0054] As an example, as shown in FIG. 6, in the smart device 10, when an instruction to start distance measurement (hereinafter also referred to as a "distance measurement start instruction") is received by the touch panel 48, the light irradiator 16 irradiates laser light. The angle at which the laser light is irradiated (hereinafter also referred to as an "irradiation angle") is θ2, and the irradiation angle θ2 is changed according to the instruction received by the touch panel 48. Note that, in the example shown in FIG. 6, an example is given in which distance measurement is started in response to a distance measurement start instruction received by the touch panel 48 in a state in which a visible light image is displayed on the display 46 as a live view image, but the technology of the present disclosure is not limited to this. For example, distance measurement may be started when a distance measurement start instruction is received by the touch panel 48 in a state in which a visible light image is not displayed on the display 46.

[0055] In the smart device 10, the distance from the smart device 10 to the target is measured based on the time required from when the laser light is emitted by the light irradiator 16 until the reflected IR light is received by the distance measurement segment area 26N2 of the light receiver 18 (see FIG. 3) and the speed of light. For example, if the distance to the target is "L", the speed of light is "c", and the time required from when the laser light is emitted by the light irradiator 16 until the reflected IR light is received by the distance measurement segment area 26N2 is "t", the distance L is calculated according to the formula "L=c×t×0.5".

[0056] 7, in the smart device 10, the IR reflected light is received by each of the multiple IR pixels included in the distance measurement division area 26N2, and distance measurement is performed for each IR pixel. Then, the distance measurement result for each IR pixel is displayed as a distance image on the display 46. Here, the distance image refers to an image in which the distance to the distance measurement target measured for each IR pixel is expressed in different colors and / or shades.

[0057] In the smart device 10, the distance measurement result is displayed on the display 46 as a distance image or a distance superimposed image in accordance with an instruction received by the touch panel 48. In the example shown in FIG. 8, the distance superimposed image displayed on the display 46 is an image obtained by superimposing numerical values ​​indicating the distance measurement result (1.6 m, 1.8 m, and 5.3 m in the example shown in FIG. 8) on a visible light image (for example, a live view image). For example, the distance from the smart device 10 to each of a plurality of representative locations (three locations in the example shown in FIG. 8) in the imaging area is displayed on the display 46 in a visible light image. One example of the representative locations is a plurality of locations among specific subjects (for example, subjects included in the central area of ​​the screen and / or humans, etc.) in the imaging area where the difference in contrast between them is equal to or greater than a preset value.

[0058] 9, the smart device 10 includes a controller 15, an input / output interface 40, an image memory 42, a UI device 44, an external I / F 52, and a communication I / F 54 in addition to a light irradiator 16 and a light receiver 18. The controller 15 is an example of a "processing device" and a "computer" according to the technology of the present disclosure.

[0059] The controller 15 includes a CPU 15A, a storage 15B, and a memory 15C. The CPU 15A is an example of a "processor," a "recognition processor," and a "specific processor" according to the technology of the present disclosure, and the memory 15C is an example of a "memory" according to the technology of the present disclosure. The CPU 15A, the storage 15B, and the memory 15C are connected via a bus 50, and the bus 50 is connected to an input / output interface 40. In the example shown in FIG. 9, one bus is illustrated as the bus 50 for convenience of illustration, but multiple buses may be used. The bus 50 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, and the like.

[0060] The storage 15B stores various parameters and various programs. The storage 15B is a non-volatile storage device. Here, a flash memory is adopted as an example of the storage 15B. The flash memory is merely an example, and the storage 15B may be, for example, various non-volatile memories such as a magnetoresistive memory and / or a ferroelectric memory instead of the flash memory or in addition to the flash memory. The non-volatile storage device may be an EEPROM, a HDD, and / or an SSD. The memory 15C temporarily stores various information and is used as a work memory. An example of the memory 15C is a RAM, but is not limited to this and may be another type of storage device.

[0061] Various programs are stored in the storage 15B. The CPU 15A reads out a necessary program from the storage 15B and executes the read program on the memory 15C. The CPU 15A controls the entire smart device 10 according to the program executed on the memory 15C.

[0062] A plurality of devices are connected to the input / output interface 40, and the input / output interface 40 controls the transmission and reception of various information between the plurality of devices. In the example shown in Fig. 9, the plurality of devices connected to the input / output interface 40 include the controller 15, the light irradiator 16, the light receiver 18, the image memory 42, the UI device 44, the external I / F 52, and the communication I / F 54.

[0063] The external I / F 52 is responsible for transmitting and receiving various information between the smart device 10 and a device (hereinafter, also referred to as an "external device") that exists outside the smart device 10. An example of the external I / F 52 is a USB interface. The USB interface can be directly or indirectly connected to an external device (not shown) such as a smart device, a personal computer, a server, a USB memory, a memory card, and / or a printer.

[0064] The communication I / F 54 has a communication function such as LTE, 5G, wireless LAN, and / or Bluetooth (registered trademark), and controls the exchange of various information between an external device and the CPU 15A. For example, the communication I / F 54 is communicatively connected to a network 56 (e.g., the Internet) via a base station (not shown), and controls the exchange of various information between an external device on the network 56 and the CPU 15A.

[0065] The UI device 44 includes a display 46, and the CPU 15A causes the display 46 to display various pieces of information. The UI device 44 also includes a reception device 47. The reception device 47 includes a touch panel 48 and a hard key unit 53. The hard key unit 53 is at least one hard key including the instruction key 13 (see FIG. 2). The CPU 15A operates according to various instructions received by the touch panel 48. Note that, although the hard key unit 53 is included in the UI device 44 here, the technology of the present disclosure is not limited to this, and for example, the hard key unit 53 may be connected to an external I / F 52.

[0066] The light irradiator 16 includes a light-transmitting window 20, a beam expander 21, a collimating lens 23, an LD 24, and an LD driver 25, and the light-transmitting window 20, the beam expander 21, and the collimating lens 23 are arranged in this order along the optical axis L1 from the imaging region side (object side) to the LD 24. The LD driver 25 is connected to the LD 24 and the input / output interface 40, and drives the LD 24 in accordance with instructions from the CPU 15A to cause the LD 24 to emit laser light.

[0067] The laser light emitted from the LD 24 is converted into parallel light by the collimator lens 23, and the light diameter is expanded by the beam expander 21, and the light is irradiated from the light transmitting window 20 toward the object to be measured.

[0068] The light receiver 18 includes a light-transmitting window 22, an objective lens 30A, a focus lens 30B, an aperture 30C, a photoelectric conversion element 26, a photoelectric conversion element driver 32, and a signal processing circuit 34. In the light receiver 18, the light-transmitting window 22, the objective lens 30A, the focus lens 30B, and the aperture 30C are arranged in this order along the optical axis L2 from the imaging region side (object side) to the photoelectric conversion element 26. The photoelectric conversion element driver 32 is connected to the photoelectric conversion element 26 and the input / output interface 40, and drives the photoelectric conversion element 26 according to an instruction from the CPU 15A. For example, the photoelectric conversion element driver 32 supplies the photoelectric conversion element 26 with an imaging timing signal that specifies the timing of imaging performed by the photoelectric conversion element 26 under the control of the CPU 15A. The photoelectric conversion element 26 performs resetting, exposure, and output of an electrical signal according to the imaging timing signal supplied from the photoelectric conversion element driver 32. The imaging timing signal includes, for example, a vertical synchronization signal and a horizontal synchronization signal.

[0069] The light receiver 18 includes a focusing control mechanism 31. The focusing control mechanism 31 includes a focus lens 30B, a moving mechanism 60, a motor 62, and a motor driver 64. The focus lens 30B is supported by the moving mechanism 60 so as to be slidable along the optical axis L2. The motor 62 is connected to the moving mechanism 60 and the motor driver 64. The motor driver 64 is connected to the input / output interface 40, and drives the motor 62 according to an instruction from the CPU 15A. The moving mechanism 60 is connected to a drive shaft (not shown) of the motor 62, and selectively moves the focus lens 30B along the optical axis L2 to the object side and the image side by receiving power from the motor 62. That is, the CPU 15A adjusts the focusing position by controlling the driving of the motor 62 via the motor driver 64. Here, the "focusing position" refers to the position of the focus lens 30B on the optical axis L2 in a focused state (for example, a state in which the contrast of the visible light image is maximized, or a state in which a predetermined depth of field is realized). For ease of explanation, the control for adjusting the focus lens 30B to the in-focus position will also be referred to as "focus control" below.

[0070] The aperture 30C is a fixed aperture whose opening does not change. In the case of a fixed aperture, exposure adjustment is performed by an electronic shutter of the photoelectric conversion element 26. The aperture 30C may not be a fixed aperture, but may be a variable aperture. Note that the objective lens 30A, the focus lens 30B, and the aperture 30C included in the optical receiver 18 are merely examples, and the technology of the present disclosure is valid even if the lens configuration and / or the position of the aperture 30C are changed.

[0071] The reflected light is incident on the light receiver 18 through the light-transmitting window 22. The reflected light incident on the light-transmitting window 22 is imaged on the photoelectric conversion element 26 via the objective lens 30A, the focus lens 30B, and the aperture 30C.

[0072] The photoelectric conversion element 26 is connected to a signal processing circuit 34, and outputs pixel data indicating a pixel value for each of the visible light pixels and the IR pixels to the signal processing circuit 34. The signal processing circuit 34 digitizes the pixel data input from the photoelectric conversion element 26 by performing A / D conversion, and performs various types of signal processing on the digitized pixel data.

[0073] The signal processing circuit 34 includes a visible light pixel data processing circuit 34A and a distance image generation circuit 34B. The visible light pixel data processing circuit 34A generates a visible light image by performing known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction on the pixel data for the visible light pixels. The visible light pixel data processing circuit 34A then stores the visible light image in an image memory 42. Note that the visible light image in the image memory 42 is updated by overwriting and saving one frame of the visible light image in the image memory 42.

[0074] The distance measuring imaging device 14 includes a TOF camera 19. The TOF camera 19 includes a light irradiator 16, a distance measuring section area 26N2, and a distance image generating circuit 34B. The distance image generating circuit 34B acquires an emission timing signal indicating the timing at which the laser light is emitted from the LD 24 (hereinafter also referred to as "emission timing") from the CPU 15A. The distance image generating circuit 34B measures the distance from the smart device 10 to the target for distance measurement for each IR pixel based on the emission timing indicated by the emission timing signal and the timing at which the IR reflected light is received by each IR pixel (hereinafter also referred to as "light receiving timing"). Then, the distance image generating circuit 34B generates a distance image based on the measurement result for each IR pixel, and stores the generated distance image in the image memory 42. Note that the distance image in the image memory 42 is updated by overwriting and saving one frame of distance image in the image memory 42.

[0075] 10 as an example, the storage 15B stores an imaging processing program 70, a distance measurement processing program 72, a parallel processing program 74, a change processing program 76, a display control processing program 78, an image recognition dictionary 80, and an irradiation energy table 82. In the following, for convenience of explanation, when there is no need to distinguish between the imaging processing program 70, the distance measurement processing program 72, the parallel processing program 74, the change processing program 76, and the display control processing program 78, they will be referred to as the "distance measurement imaging processing program."

[0076] According to the conventional technology, the reflected IR light is received by the IR pixel, and the distance from the smart device 10 to the target is measured based on the emission timing and the reception timing. Ideally, the reflected IR light is completely blocked by the visible light filter, but depending on the intensity of the reflected IR light, it is difficult to completely block it with the visible light filter. In this case, the reflected IR light may reach the visible light pixel and appear as noise in the visible light image, which may affect the image quality of the visible light image.

[0077] One method for preventing reflected IR light from appearing as noise in a visible light image is to stop the distance measurement operation during the period when the image capture operation is performed (for example, the period when exposure is performed by photoelectric conversion element 26) when the distance to the subject is less than a predetermined distance (for example, 0.5 m).

[0078] However, when the distance measurement results are used for focus control, if the distance measurement operation is always stopped during the imaging operation, the distance measurement results cannot be reflected in the focus control while the distance measurement operation is stopped, and there is a concern that the accuracy of the focus control will decrease compared to when the distance measurement operation is always performed. In particular, when a moving subject is the subject of distance measurement, there is a risk that the focus control will be performed according to the distance measurement results that do not match the current situation. In addition, depending on the imaging scene, even if the distance to the subject is a predetermined distance, it is possible that the IR reflected light does not appear as noise in the visible light image depending on the reflectance of the subject. However, even in such a case, if the distance to the subject is less than the predetermined distance, the distance measurement operation will be stopped.

[0079] 11, the CPU 15A of the smart device 10 reads a distance measurement imaging processing program from the storage 15B and executes the distance measurement imaging processing according to the read distance measurement imaging processing program. The distance measurement imaging processing includes an imaging processing (see FIG. 18), a distance measurement processing (see FIG. 19), a parallel processing (see FIG. 17), a change processing (see FIG. 20), and a display control processing (see FIG. 21A and FIG. 21B).

[0080] The CPU 15A reads out the imaging processing program 70 from the storage 15B, and executes imaging processing (see FIG. 18) in accordance with the read imaging processing program 70. The CPU 15A also reads out the distance measurement processing program 72 from the storage 15B, and executes distance measurement processing (see FIG. 19) in accordance with the read distance measurement processing program 72. The CPU 15A also reads out the parallel processing program 74 from the storage 15B, and executes parallel processing (see FIG. 17) in accordance with the read parallel processing program 74. The CPU 15A also reads out the change processing program 76 from the storage 15B, and executes change processing (see FIG. 20) in accordance with the read change processing program 76. The CPU 15A also reads out the display control processing program 78 from the storage 15B, and executes display control processing (see FIG. 21A and FIG. 21B) in accordance with the read display control processing program 78.

[0081] As an example, as shown in FIG. 12, the distance measurement imaging process is realized by the CPU 15A operating as a control unit 90, a recognition unit 92, an acquisition unit 94, a determination unit 96, and a change unit 98.

[0082] The control unit 90 executes parallel processing. That is, the control unit 90 controls the distance measuring imaging device 14 to perform an imaging operation and a distance measuring operation in parallel. Note that, although an example of performing an imaging operation and a distance measuring operation in parallel is given here, this is merely an example, and the technology of the present disclosure can be realized even if the imaging operation and the distance measuring operation are not performed in parallel. For example, the technology of the present disclosure can be realized even if the period during which the imaging operation is performed and the period during which the distance measuring operation is performed overlap in part.

[0083] Furthermore, the control unit 90 acquires a visible light image and a distance image from the image memory 42, and causes an image based on the acquired visible light image and distance image to be displayed on the display 46. Specifically, under the control of the control unit 90, the visible light image, the distance image, and the distance superimposed image are switchably displayed on the display 46.

[0084] The recognition unit 92 acquires a visible light image from the image memory 42, and performs image recognition on the acquired visible light image. The recognition unit 92 acquires an image recognition dictionary 80 from the storage 15B, and recognizes an image included in the visible light image by referring to the acquired image recognition dictionary 80. The image recognition dictionary 80 is information in which a plurality of types of high reflectance object images are associated with identifiers (hereinafter also simply referred to as "identifiers") capable of identifying each of the plurality of types of high reflectance object images. A high reflectance object image refers to an image showing an object having a reflectance equal to or higher than a reference reflectance (hereinafter also referred to as a "high reflectance object"). The reference reflectance may be, for example, a reflectance derived in advance by a sensory test and / or a computer simulation, etc., as a reflectance that appears as noise in a visible light image when IR reflected light from a laser beam irradiated on an object passes through a visible light filter and reaches a visible light pixel. Examples of objects having a reflectance equal to or higher than the reference reflectance include objects predetermined as objects having glossiness, objects predetermined as white objects, etc. The object predetermined as a glossy object is, for example, a mirror plate and a glossy sphere. The object predetermined as a white object is, for example, a white coat, a white balloon, white hair, etc. The mirror plate is an example of a "mirror-like object" according to the technology of the present disclosure.

[0085] Here, a plurality of types of high reflectance object images are exemplified as information included in the image recognition dictionary 80, but the technology of the present disclosure is not limited thereto, and each feature amount (e.g., spatial frequency and / or brightness, etc.) of the plurality of types of high reflectance object images may be included. In addition, the image recognition dictionary 80 is information based on a learning result obtained by machine learning of the correspondence between the image and the subject recognition result. Note that, here, an example is given in which information based on the learning result obtained by machine learning is used as the image recognition dictionary 80, but this example is merely one example, and information derived in advance by a sensory test and / or a computer simulation, etc. may be used as the image recognition dictionary 80.

[0086] The acquisition unit 94 acquires reflectance information capable of identifying the reflectance in the imaging region. Here, the acquisition unit 94 acquires an image recognition result as the reflectance information. The image recognition result refers to a result obtained by performing image recognition on a visible light image by the recognition unit 92. Specifically, when a high reflectance object image is recognized by the recognition unit 92, the image recognition result includes an identifier capable of identifying the recognized high reflectance object image, and when a high reflectance object image is not recognized by the recognition unit 92, the image recognition result includes information indicating that the high reflectance object image was not recognized. Note that the "image recognition result" here is an example of the "reflectance information" and "information based on the imaging region image" according to the technology of the present disclosure.

[0087] The determination unit 96 determines whether or not a high reflectance object image is included in the visible light image by referring to the image recognition result acquired by the acquisition unit 94. When the determination unit 96 determines that a high reflectance object image is included in the visible light image, the change unit 98 changes the irradiation energy of the laser light (hereinafter also simply referred to as "irradiation energy") in accordance with the image recognition result acquired by the acquisition unit 94 while the distance measuring imaging device 14 is performing an imaging operation and a distance measuring operation.

[0088] Specifically, the change unit 98 acquires the irradiation energy table 82 from the storage 15B, and changes the intensity of the laser light emitted from the LD 24 via the LD driver 25 in accordance with the acquired irradiation energy table 82, thereby changing the irradiation energy. In other words, in a state in which the imaging operation and the distance measurement operation are performed in parallel by the parallel processing being executed by the control unit 90, the LD driver 25 causes the LD 24 to emit the laser light having the intensity changed by the change unit 98.

[0089] In the irradiation energy table 82, identifiers and intensity information are associated with each other. The intensity information refers to information indicating the intensity of the laser light to be emitted from the LD 24. The intensity of the laser light indicated by the intensity information is weaker than a reference intensity. Here, the reference intensity is an example of the "first to fourth reference irradiation energies" according to the technology of the present disclosure.

[0090] The reference intensity refers to, for example, an intensity set as a default value as the intensity of the laser light to be emitted from the LD 24. The intensity set as a default value refers to, for example, an intensity derived in advance by a sensory test and / or a computer simulation as the intensity of the laser light that is irradiated onto a reference subject (for example, a face of a person predetermined as a general human face) and is received by the IR pixel with an amount of light that can be measured.

[0091] The intensity information is determined such that the higher the reflectance of the high reflectance object shown by the high reflectance object image specified by the identifier, the weaker the intensity of the laser light. In the example shown in Fig. 12, a mirror plate, a glossy sphere, and a white coat are shown as high reflectance object images specified by identifiers in the irradiation energy table 82. For the mirror plate, the intensity X 11 For a glossy sphere, the intensity X 12 For the white coat, intensity information indicating X 13 The intensity information indicating the intensity X is associated with the 11 , intensity X 12 , and intensity X 13 Between them, "Strength X 11 <Strength X 12 <Strength X 13 " A size relationship is established.

[0092] When switching between the visible light image, distance image, and distance superimposed image displayed on the display 46 under the control of the control unit 90, as shown in FIG. 13 as an example, the control unit 90 causes the display 46 to display a display mode selection screen 100 in accordance with an instruction received by the touch panel 48. The display mode selection screen 100 is a screen used by the user, etc., when causing the user, etc., to select any one of the visible light image display mode, the distance image display mode, and the distance superimposed image display mode. The visible light image display mode is an operation mode in which the control unit 90 causes the display 46 to display a visible light image. The distance image display mode is an operation mode in which the control unit 90 causes the display 46 to display a distance image. The distance superimposed image display mode is an operation mode in which the control unit 90 causes the display 46 to display a distance superimposed image.

[0093] Soft keys 100A, 100B, and 100C are displayed on display mode selection screen 100. Soft key 100A is turned on by a user or the like via touch panel 48 when the user or the like selects the visible light image display mode. Soft key 100B is turned on by a user or the like via touch panel 48 when the user or the like selects the distance image display mode. Soft key 100C is turned on by a user or the like via touch panel 48 when the user or the like selects the distance superimposed image display mode. Note that, for convenience of explanation, the visible light image display mode, distance image display mode, and distance superimposed image display mode will be referred to as "display modes" below when they are not differentiated from one another.

[0094] When a display mode is selected by an operation on the display mode selection screen 100, the control unit 90 displays a distance measurement range designation guide screen 102 on the display 46, as shown in FIG. 14 as an example. The distance measurement range designation guide screen 102 is a screen that guides the user, etc., to designate a distance measurement range. The distance measurement range designation guide screen 102 displays a message (hereinafter also referred to as a "guide message") that asks the user whether or not to designate a distance measurement range. In the example shown in FIG. 14, a message "Do you want to designate a distance measurement range?" is displayed as an example of the guide message. In addition, soft keys 102A and 102B are displayed on the distance measurement range designation guide screen 102. The soft key 102A is turned on by the user, etc., via the touch panel 48 when the user, etc., designates a distance measurement range. The soft key 102B is turned on by the user, etc., via the touch panel 48 when the user, etc., does not designate a distance measurement range, that is, when the entire imaging area is set as the distance measurement range.

[0095] When the user or the like turns on the soft key 102A via the touch panel 48 while the ranging range designation guide screen 102 is displayed on the display 46, the control unit 90 causes the display 46 to display a visible light image as a live view image, as shown in Fig. 15 as an example. With the live view image displayed on the display 46, an image area (a rectangular area surrounded by a dashed line on the visible light image in the example shown in Fig. 15) is designated by the user or the like via the touch panel 48. A real space area corresponding to the image area designated by the user or the like (hereinafter also referred to as a "designated image area") is designated as a target for ranging by the ranging imaging device 14.

[0096] 16 as an example, area position specifying information (e.g., coordinates) capable of specifying the position of the designated image area on the visible light image is output from the touch panel 48 to the control unit 90. The control unit 90 outputs partition area position information (e.g., pixel address) capable of specifying the position of the designated partition area for distance measurement 26N2a in the partition area for distance measurement 26N2 to the photoelectric conversion element driver 32. Here, the designated partition area for distance measurement 26N2a refers to a partition area in the partition area for distance measurement 26N2 at a position corresponding to the position of the designated image area specified by the area position specifying information input from the touch panel 48.

[0097] In the distance measurement operation, the photoelectric conversion element driver 32 drives only the designated distance measurement segment area 26N2a of the distance measurement segment area 26N2, thereby causing the distance measurement imaging device 14 to perform distance measurement using IR reflected light received only by the designated distance measurement segment area 26N2a. In other words, distance measurement is performed using IR reflected light received only by at least one IR pixel included in a designated area among multiple IR pixels included in the TOF camera 19 (see FIG. 9).

[0098] Next, the operation of the portion of smart device 10 relating to the technique of the present disclosure will be described with reference to FIGS. 17 to 21B.

[0099] First, the parallel processing executed by the CPU 15A will be described with reference to FIG.

[0100] In the parallel processing shown in FIG. 17, first, in step ST10, the control unit 90 judges whether or not a condition for starting parallel processing (hereinafter also referred to as a "parallel processing start condition") is satisfied. One example of the parallel processing start condition is a condition in which an instruction to start parallel processing is received via the touch panel 48. In step ST10, if the parallel processing start condition is not satisfied, the judgment is negative, and the judgment of step ST10 is performed again. In step ST10, if the parallel processing start condition is satisfied, the judgment is positive, and the parallel processing proceeds to step ST12.

[0101] In step ST12, the control unit 90 starts the imaging process and the distance measurement process, thereby causing the distance measuring imaging device 14 to start the imaging operation and the distance measurement operation, and then the parallel processing proceeds to step ST14.

[0102] In step ST14, the control unit 90 determines whether or not a condition for terminating the parallel processing (hereinafter also referred to as a "parallel processing termination condition") has been satisfied. One example of the parallel processing termination condition is that an instruction to terminate the parallel processing has been received via the touch panel 48. If the parallel processing termination condition has not been satisfied in step ST14, the determination is negative, and the determination of step ST14 is made again. If the parallel processing termination condition has been satisfied in step ST14, the determination is positive, and the parallel processing proceeds to step ST16.

[0103] In step ST16, the control unit 90 causes the distance measuring image pickup device 14 to end the imaging operation and distance measurement operation by ending the imaging process and distance measurement process, and then the parallel process ends.

[0104] Next, the imaging process executed by the CPU 15A will be described with reference to FIG.

[0105] In the imaging process shown in FIG. 18, first, in step ST20, the control unit 90 judges whether or not the timing for causing the distance measuring imaging device 14 to capture an image (hereinafter also referred to as "imaging timing") has arrived. An example of the imaging timing is a timing determined by a period that defines a frame rate. For example, if the frame rate is 120 fps, the period that defines the frame rate is 1 / 120 seconds. If the imaging timing has not arrived in step ST20, the judgment is negative, and the imaging process proceeds to step ST28. If the imaging timing has arrived in step ST20, the judgment is positive, and the imaging process proceeds to step ST22.

[0106] In step ST22, the control unit 90 resets the photoelectric conversion element 26 by outputting an imaging timing signal to the photoelectric conversion element driver 32, and causes the photoelectric conversion element 26 to perform exposure, and then the imaging process proceeds to step ST24.

[0107] In step ST24, the control unit 90 causes the visible light pixel data processing circuit 34A to perform various types of signal processing. That is, the visible light pixel data processing circuit 34A performs various types of signal processing on pixel data for visible light pixels included in the photoelectric conversion element 26 to generate a visible light image.

[0108] In step ST26, the control unit 90 causes the visible light pixel data processing circuit 34A to store the visible light image in the image memory 42, and then the imaging process proceeds to step ST28.

[0109] In step ST28, the control unit 90 determines whether or not a condition for ending the imaging process (hereinafter also referred to as an "imaging process ending condition") has been satisfied. One example of the imaging process ending condition is that the process of step ST16 shown in FIG. 17 has been executed. In step ST28, if the imaging process ending condition has not been satisfied, the determination is negative, and the imaging process proceeds to step ST20. In step ST28, if the imaging process ending condition has been satisfied, the determination is positive, and the imaging process ends.

[0110] Next, the distance measurement process executed by the CPU 15A will be described with reference to FIG.

[0111] In the distance measurement process shown in FIG. 19, first, in step ST40, the control unit 90 determines whether or not the timing for causing the distance measurement imaging device 14 to perform distance measurement (hereinafter also referred to as "distance measurement timing") has arrived. The distance measurement timing is, for example, a timing that arrives at a shorter period than the imaging timing. In step ST40, if the distance measurement timing has not arrived, the determination is negative, and the distance measurement process proceeds to step ST56. In step ST40, if the distance measurement timing has arrived, the determination is positive, and the distance measurement process proceeds to step ST42.

[0112] In step ST42, the control unit 90 causes the light irradiator 16 to irradiate laser light. That is, the control unit 90 controls the LD driver 25 to cause the LD 24 to emit laser light, and then the distance measurement process proceeds to step ST44.

[0113] In step ST44, the control unit 90 determines whether or not reflected IR light has been received by the distance measurement designated segmented area 26N2a. If reflected IR light has not been received by the distance measurement designated segmented area 26N2a in step ST44, the determination is negative, and the determination of step ST44 is performed again. If reflected IR light has been received by the distance measurement designated segmented area 26N2a in step ST44, the determination is positive, and the distance measurement process proceeds to step ST46.

[0114] In step ST46, the control unit 90 causes the distance image generating circuit 34B to generate a distance image based on the distance measurement result, and then the distance measurement process proceeds to step ST48.

[0115] In step ST48, control unit 90 causes distance image generating circuit 34B to store the distance image generated in step ST46 in image memory 42, and then the distance measuring process proceeds to step ST50.

[0116] The distance measuring operation performed by the distance measuring imaging device 14 is a distance measuring operation for focusing, and the distance measuring imaging device 14 performs focusing on the imaging area based on the distance measuring result obtained by performing distance measuring by the distance measuring operation for focusing. Therefore, in the distance measuring process, the control unit 90 executes the processes of steps ST50 to ST54.

[0117] In step ST50, the control unit 90 acquires a distance image from the image memory 42, and derives the distance from the smart device 10 to a specific area in the imaging area based on the acquired distance image, after which the distance measurement process proceeds to step ST52. Here, the specific area refers to, for example, an area designated by a user or the like via the touch panel 48 as an area on the object side to be focused on. Note that the specific area is not limited to this, and may be, for example, a person's face identified by a so-called face detection function, or one of multiple representative areas (see FIG. 8).

[0118] In step ST52, the control unit 90 derives the in-focus position based on the distance derived in step ST50, and then the distance measurement process proceeds to step ST54. The in-focus position is derived by the control unit 90 from, for example, a in-focus position derivation table (not shown) in which distance and in-focus position are associated with each other, or a in-focus position derivation arithmetic expression (not shown) in which the distance is an independent variable and the in-focus position is a dependent variable.

[0119] In step ST54, the control unit 90 operates the focus control mechanism 31 to move the focus lens 30B to the focus position derived in step ST52, and then the distance measurement process proceeds to step ST56.

[0120] In step ST56, the control unit 90 determines whether or not a condition for terminating the distance measurement process (hereinafter also referred to as a "distance measurement process termination condition") has been satisfied. One example of the distance measurement process termination condition is that the process of step ST16 shown in FIG. 17 has been executed. In step ST56, if the distance measurement process termination condition has not been satisfied, the determination is negative, and the distance measurement process proceeds to step ST40. In step ST56, if the distance measurement process termination condition has been satisfied, the determination is positive, and the distance measurement process ends.

[0121] Next, a change process executed by the CPU 15A while parallel processes are being executed by the control unit 90 will be described with reference to FIG.

[0122] 20, first, in step ST70, the recognition unit 92 determines whether or not the visible light image in the image memory 42 has been updated. If the visible light image in the image memory 42 has not been updated in step ST70, the determination is negative, and the change process proceeds to step ST82. If the visible light image in the image memory 42 has been updated in step ST70, the determination is positive, and the change process proceeds to step ST71.

[0123] In step ST71, the recognition section 92 acquires a visible light image from the image memory 42, and then the change processing proceeds to step ST72.

[0124] In step ST72, the recognition unit 92 executes image recognition on the visible light image acquired in step ST70 by referring to the image recognition dictionary 80, and then the change processing proceeds to step ST74.

[0125] In step ST74, the acquisition section 94 acquires the image recognition result obtained by the image recognition performed by the recognition section 92 in step ST72, and thereafter, the change processing proceeds to step ST76.

[0126] In step ST76, the determination unit 96 determines whether or not a high-reflectance object image is included in the visible light image acquired in step ST70, based on the image recognition result acquired in step ST74. In step ST76, if a high-reflectance object image is not included in the visible light image acquired in step ST70, the determination is negative, and the change process proceeds to step ST82. In step ST76, if a high-reflectance object image is included in the visible light image acquired in step ST70, the determination is positive, and the change process proceeds to step ST78.

[0127] In step ST78, the change section 98 acquires, from the irradiation energy table 82, intensity information corresponding to the identifier included in the image recognition result acquired in step ST74, and then the change processing proceeds to step ST80.

[0128] In step ST80, the change unit 98 changes the intensity of the laser light emitted from the LD 24 via the LD driver 25 to the intensity indicated by the intensity information acquired in step ST78, and then the change process proceeds to step ST82.

[0129] In step ST82, the control unit 90 judges whether or not a condition for terminating the change processing (hereinafter also referred to as a "change processing termination condition") has been satisfied. One example of the change processing termination condition is that the processing of step ST16 shown in FIG. 17 has been executed. In step ST82, if the change processing termination condition has not been satisfied, the judgment is negative, and the change processing proceeds to step ST70. In step ST82, if the change processing termination condition has been satisfied, the judgment is positive, and the change processing ends.

[0130] Next, the display control process executed by the CPU 15A will be described with reference to FIGS. 21A and 21B.

[0131] 21A, first, in step ST100, the control unit 90 determines whether or not the visible light image display mode is set. If the visible light image display mode is set in step ST100, the determination is affirmative, and the display control process proceeds to step ST102. If the visible light image display mode is not set in step ST100, the determination is negative, and the display control process proceeds to step ST106.

[0132] In step ST102, the control unit 90 acquires the latest visible light image from the image memory 42, and then the display control process proceeds to step ST104.

[0133] In step ST104, the control unit 90 causes the display 46 to display the latest visible light image acquired in step ST102, and then the display control process proceeds to step ST122.

[0134] In addition, the live view image obtained by performing a live view image imaging operation for imaging an imaging area for a live view image by the distance measuring imaging device 14 as an imaging operation is displayed on the display 46 by repeatedly executing the processing of steps ST100 to ST104.

[0135] In step ST106, the control unit 90 determines whether or not the distance image display mode is set. If the distance image display mode is set in step ST106, the determination is affirmative, and the display control process proceeds to step ST108. If the distance image display mode is not set in step ST106, the determination is negative, and the display control process proceeds to step ST112 shown in FIG. 21B.

[0136] In step ST108, the control unit 90 acquires the latest distance image from the image memory 42, and then the display control process proceeds to step ST110.

[0137] In step ST110, the control unit 90 causes the display 46 to display the latest distance image acquired in step ST108, and then the display control process proceeds to step ST122.

[0138] In step ST112 shown in FIG. 21B, the control unit 90 acquires the latest distance image from the image memory 42, and then the display control process proceeds to step ST114.

[0139] In step ST114, the control unit 90 derives distances from the smart device 10 to a plurality of representative locations (see FIG. 8) based on the latest distance image acquired in step ST112, and then the display control process proceeds to step ST116.

[0140] In step ST116, the control unit 90 acquires the latest visible light image from the image memory 42, and then the display control process proceeds to step ST118.

[0141] In step ST118, the control unit 90 generates a distance superimposed image by superimposing the distance derived in step ST114 on the latest visible light image acquired in step ST116, and then the display control process proceeds to step ST120. Note that the positions at which the distance derived in step ST114 is superimposed on the latest visible light image acquired in step ST116 correspond to a plurality of representative locations.

[0142] In step ST120, the control unit 90 causes the display 46 to display the distance superimposed image generated in step ST118, and thereafter, the display control process proceeds to step ST122 shown in FIG. 21A.

[0143] In step ST122, the control unit 90 determines whether or not a condition for terminating the display control process (hereinafter also referred to as a "display control process termination condition") has been satisfied. One example of the display control process termination condition is that an instruction to terminate the display control process has been accepted by the touch panel 48. In step ST122, if the display control process termination condition is not satisfied, the determination is negative, and the display control process proceeds to step ST100. In step ST122, if the display control process termination condition is satisfied, the determination is positive, and the display control process ends.

[0144] As described above, in the smart device 10, the control unit 90 controls the distance measuring imaging device 14 to perform an imaging operation and a distance measuring operation. Also, the acquisition unit 94 acquires reflectance information capable of identifying the reflectance in the imaging area ("image recognition result" in the example shown in FIG. 12). Then, while the imaging operation and the distance measuring operation are being performed, the change unit 98 changes the intensity of the laser light according to the reflectance information acquired by the acquisition unit 94. Therefore, according to this configuration, the influence of the laser light on the visible light image can be reduced compared to a case where the intensity of the laser light is determined independently of the reflectance in the imaging area.

[0145] Furthermore, in the smart device 10, the intensity of the laser light is changed by the change unit 98 in accordance with information based on the visible light image (the "image recognition result" in the example shown in FIG. 12). Therefore, according to this configuration, the effect of reflected IR light on the visible light image can be reduced with high accuracy compared to a case where the intensity of the laser light is changed based only on the distance to the imaging area or only on the shutter speed.

[0146] Furthermore, in the smart device 10, the intensity of the laser light is changed by the change unit 98 in accordance with the image recognition result. Therefore, according to this configuration, the effect of the reflected IR light on the visible light image can be reduced with high accuracy compared to the case where the intensity of the laser light is changed based only on the distance to the imaging area or only on the shutter speed.

[0147] In addition, in the smart device 10, the recognition unit 92 refers to the image recognition dictionary 80 and performs image recognition on the visible light image, and the acquisition unit 94 acquires the image recognition result obtained by the recognition unit 92. The image recognition dictionary 80 is information based on a learning result obtained by machine learning the correspondence between the image and the subject recognition result. Therefore, according to this configuration, it is possible to reduce the influence of the reflected IR light on the visible light image with high accuracy compared to a case where it is determined whether or not the reflected IR light affects the visible light image only from the result of the user or the like visually recognizing the imaging area.

[0148] Furthermore, in the smart device 10, when the image recognition result acquired by the acquisition unit 94 indicates that a highly reflective object image is included, the intensity of the laser light is weakened to be lower than the reference intensity by the change unit 98. Therefore, according to this configuration, the influence of the IR reflected light from the highly reflective object on the visible light image can be reduced compared to the case where the intensity of the laser light is always equal to or higher than the reference intensity regardless of whether or not a highly reflective object is included in the imaging area.

[0149] Furthermore, in the smart device 10, when the image recognition result acquired by the acquisition unit 94 indicates that a glossy object is included, the intensity of the laser light is weakened to be lower than the reference intensity by the change unit 98. Therefore, according to this configuration, the effect of IR reflected light from a highly reflective object on a visible light image can be reduced compared to a case where the intensity of the laser light is always equal to or higher than the reference intensity regardless of whether a glossy object is included in the imaging area.

[0150] Furthermore, in the smart device 10, when the image recognition result acquired by the acquisition unit 94 indicates that a mirror-like object (e.g., a mirror plate) is included, the intensity of the laser light is weakened to be lower than the reference intensity by the change unit 98. Therefore, according to this configuration, the effect of IR reflected light from a highly reflective object on a visible light image can be reduced compared to a case where the intensity of the laser light is always equal to or higher than the reference intensity regardless of whether a mirror-like object is included in the imaging area.

[0151] Furthermore, in the smart device 10, the distance measuring image capturing device 14 has a TOF camera 19. Therefore, according to this configuration, the distance measuring result can be output as a distance image.

[0152] Furthermore, in the smart device 10, a distance image is displayed on the display 46. Therefore, according to this configuration, the user or the like can visually recognize the distance image.

[0153] Furthermore, in the smart device 10, distance measurement is performed using the IR reflected light received only by the designated distance measurement segment area 26N2a (see FIG. 16). Therefore, according to this configuration, the processing load required for distance measurement can be reduced compared to when distance measurement is performed using the IR reflected light received by all IR pixels included in the distance measurement segment area 26N2.

[0154] Furthermore, in the smart device 10, distance measurement is performed using IR reflected light received only by IR pixels at a position corresponding to the position of an image area designated within the screen while a live view image is being displayed (the "designated image area" in the example shown in FIG. 15). Therefore, according to this configuration, distance measurement can be performed using IR reflected light received only by the IR pixels intended by the user among all IR pixels included in the distance measurement division area 26N2.

[0155] Furthermore, in the smart device 10, an imaging operation for a live view image in which an imaging area for a live view image is captured is performed as an imaging operation by the distance measuring imaging device 14. Therefore, according to this configuration, the influence of the laser light on the live view image can be reduced compared to a case in which the intensity of the laser light is determined independently of the reflectance in the imaging area.

[0156] In addition, in the smart device 10, the distance measuring imaging device 14 performs a distance measuring operation for focusing as a distance measuring operation. That is, the distance measuring result is used for focus control. Therefore, according to this configuration, it is possible to focus on the distance measuring target with high accuracy compared to a case where the distance measuring result is not used for focus control.

[0157] Furthermore, in the smart device 10, distance measurement is performed using laser light as directional light, which is light having directionality. Therefore, according to this configuration, the distance to a distance measurement target located at a long distance can be measured with high accuracy compared to a case where distance measurement is performed without using directional light.

[0158] In the above embodiment, an example has been described in which the change unit 98 derives intensity information corresponding to an identifier included in the image recognition result from the irradiation energy table 82 and changes the intensity of the laser light according to the derived intensity information, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 22, the change unit 98 may derive intensity information corresponding to an identifier included in the image recognition result and an imaging scene from the irradiation energy table 182 and change the intensity of the laser light according to the derived intensity information.

[0159] In the example shown in FIG. 22, the irradiation energy table 182 is information in which imaging scenes and intensity information are associated with each other for each identifier. In the example shown in FIG. 22, a night view and a sunset are given as examples of imaging scenes. Among the multiple intensity information included in the irradiation energy table 182, the intensity indicated by the intensity information corresponding to a specific imaging scene having an imaging scene reflectance equal to or greater than the reference reflectance described in the above embodiment is an intensity lower than the reference intensity described in the above embodiment. The imaging scene reflectance refers to a reflectance determined in advance as the reflectance of the laser light irradiated to the imaging scene. The imaging scene reflectance is a reflectance derived in advance by a sensory test and / or a computer simulation, etc.

[0160] The touch panel 48 receives imaging scene instruction information that instructs an imaging scene. The change unit 98 derives intensity information corresponding to the imaging scene instructed by the imaging scene instruction information received by the touch panel 48 and the identifier included in the image recognition result from the irradiation energy table 182. Then, the change unit 98 changes the intensity of the laser light according to the intensity information derived from the irradiation energy table 182.

[0161] When the imaging scene instruction information is received by the touch panel 48, as shown in FIG. 23 as an example, an imaging scene selection screen 104 is displayed on the display 46 under the control of the control unit 90. The imaging scene selection screen 104 is a screen used by the user or the like when the user or the like selects one of a plurality of imaging scenes. A message (a message saying "Please select an imaging scene" in the example shown in FIG. 23) that prompts the user or the like to select an imaging scene is displayed on the imaging scene selection screen 104. In addition, a plurality of soft keys such as soft keys 104A, 104B, 104C, and 104D (hereinafter also referred to as "soft keys in the imaging scene selection screen 104") are displayed on the imaging scene selection screen 104. The soft key 104A is turned on by the user or the like via the touch panel 48 when the user or the like selects a night scene as an imaging scene. The soft key 104B is turned on by the user or the like via the touch panel 48 when the user or the like selects a sunset scene as an imaging scene. When the user or the like selects a landscape scene as the imaging scene, the soft key 104C is turned on by the user or the like via the touch panel 48. When the user or the like selects a portrait scene as the imaging scene, the soft key 104D is turned on by the user or the like via the touch panel 48. In this manner, imaging scene instruction information is received by the touch panel 48 when the user or the like turns on a soft key in the imaging scene selection screen 104.

[0162] In the smart device 10 configured in this manner, as an example, the change process shown in Fig. 24 is executed by the CPU 15A. The change process shown in Fig. 24 differs from the change process shown in Fig. 20 in that it has a process of step ST178 instead of the process of step ST78, and in that it has a process of step ST77 between the processes of steps ST76 and ST178.

[0163] In step ST77, the change section 98 acquires the imaging scene instruction information received by the touch panel 48, and then the change processing proceeds to step ST178.

[0164] In step ST178, the change unit 98 obtains from the irradiation energy table 182 intensity information corresponding to the identifier included in the image recognition result obtained in step ST74 and the imaging scene instruction information obtained in step ST77, and then the change processing proceeds to step ST80.

[0165] 22 to 24, the intensity of the laser light is changed by the change unit 98 in accordance with the imaging scene instruction information received by the touch panel 48 and the identifier included in the image recognition result. Therefore, according to this configuration, the effect of reflected IR light on the visible light image can be reduced compared to the case where the intensity of the laser light is determined independently of the imaging scene.

[0166] 22 to 24, when the imaging scene designated by the imaging scene designation information received by the touch panel 48 is a specific imaging scene having an imaging scene reflectance equal to or greater than the reference reflectance, the intensity of the laser light is weakened to be lower than the reference intensity by the change unit 98. Therefore, according to this configuration, it is possible to reduce the influence of the reflected IR light in the specific imaging scene on the visible light image, compared to a case where the intensity of the laser light is always equal to or greater than the reference intensity even though the designated imaging scene is a specific imaging scene having an imaging scene reflectance equal to or greater than the reference reflectance.

[0167] 22 to 24, an example is described in which the change unit 98 acquires intensity information corresponding to the imaging scene instruction information received by the touch panel 48, but the technology of the present disclosure is not limited to this. For example, the imaging scene may be specified by image recognition of the visible light image by the recognition unit 92.

[0168] In this case, as shown in FIG. 25 as an example, the recognition unit 92 acquires a visible light image from the image memory 42, and performs image recognition on the acquired visible light image to identify an imaging scene of an imaging area indicated by the visible light image. Specifically, the recognition unit 92 identifies an imaging scene by referring to the image recognition dictionary 80. The image recognition dictionary 80 includes information in which an imaging scene image indicating an imaging scene is associated with scene specification information capable of specifying an imaging scene indicated by the imaging scene image. The recognition unit 92 performs image recognition on the visible light image acquired from the image memory 42 by referring to the image recognition dictionary 80. That is, the recognition unit 92 identifies an imaging scene image corresponding to the visible light image from the image recognition dictionary 80, and acquires scene specification information corresponding to the identified imaging scene image from the image recognition dictionary 80.

[0169] The change unit 98 derives intensity information corresponding to the captured scene identified by the scene identification information acquired by the recognition unit 92 and the identifier included in the image recognition result from the irradiation energy table 182. Then, the change unit 98 changes the intensity of the laser light to the intensity indicated by the intensity information derived from the irradiation energy table 182. Note that in the example shown in Fig. 25, the recognition unit 92 is an example of a "identification unit" according to the technology of the present disclosure.

[0170] In the smart device 10 configured in this manner, as an example, the change process shown in Fig. 26 is executed by the CPU 15A. The change process shown in Fig. 26 differs from the change process shown in Fig. 24 in that it does not include the process of step ST77, it includes the process of step ST275 between the processes of step ST74 and step ST76, and it includes the process of step ST278 instead of the process of step ST178.

[0171] In step ST275, the recognition section 92 identifies an image capture scene image corresponding to the visible light image from the image recognition dictionary 80, and obtains from the image recognition dictionary 80 scene specification information corresponding to the identified image capture scene image.

[0172] In step ST278, the change unit 98 obtains from the irradiation energy table 182 intensity information corresponding to the identifier included in the image recognition result obtained in step ST74 and the imaging scene identified by the scene identification information obtained in step ST275, and then the change processing proceeds to step ST80.

[0173] 25 and 26, the intensity of the laser light is changed by the change unit 98 in accordance with the image capture scene identified by the recognition unit 92 and the identifier included in the image recognition result. Therefore, according to this configuration, the effect of reflected IR light on the visible light image can be reduced compared to the case where the intensity of the laser light is determined independently of the image capture scene.

[0174] 22 to 24, in the examples shown in Fig. 25 and 26, when the imaging scene identified by the scene identification information acquired by the recognition unit 92 is a specific imaging scene having an imaging scene reflectance equal to or greater than the reference reflectance, the intensity of the laser light is weakened to be lower than the reference intensity by the change unit 98. Therefore, according to this configuration, it is possible to reduce the influence of the reflected IR light in the specific imaging scene on the visible light image, compared to the case where the intensity of the laser light is always equal to or greater than the reference intensity even though the imaging scene identified by the recognition unit 92 is a specific imaging scene having an imaging scene reflectance equal to or greater than the reference reflectance.

[0175] In the above embodiment, the laser light is irradiated onto the object to be measured by the light irradiator 16, but the technology of the present disclosure is not limited to this. For example, the object to be measured may be irradiated with auxiliary light for imaging (for example, visible light used as a so-called red-eye prevention measure), and the reflected light of the auxiliary light reflected by the object to be measured (hereinafter also referred to as "auxiliary reflected light") may be received by the light receiver 18, thereby measuring the distance.

[0176] In this case, as an example, a smart device 200 is used as shown in Fig. 27. The smart device 200 differs from the smart device 10 described in the above embodiment in that it has a distance measuring imaging device 214 instead of the distance measuring imaging device 14, and a light receiver 218 instead of the light receiver 18. The distance measuring imaging device 214 differs from the distance measuring imaging device 14 in that it has an auxiliary light illuminator 213.

[0177] The auxiliary light illuminator 213 includes an LED 202 and an LED driver 204. The LED driver 204 is connected to the LED 202 and the input / output interface 40. The LED driver 204 controls the LED 202 according to an instruction from the CPU 15A (e.g., the control unit 90). Under the control of the CPU 15A, the LED driver 204 causes the LED 202 to generate auxiliary light, thereby irradiating the auxiliary light toward the imaging area. The LED 202 is an example of a "light source" according to the technology of the present disclosure.

[0178] The receiver 218 differs from the receiver 18 in that it has a photoelectric conversion element 226 instead of the photoelectric conversion element 26. The photoelectric conversion element 226 differs from the receiver 18 in that it has a designated segmented area 26N2b for distance measurement instead of the designated segmented area 26N2a for distance measurement (see FIG. 16). The designated segmented area 26N2b for distance measurement differs from the designated segmented area 26N2a for distance measurement in that G pixels are used instead of IR pixels. The designated segmented area 26N2b for distance measurement receives auxiliary reflected light under the control of the control unit 90. The designated segmented area 26N2b for distance measurement is an example of a "light receiving element" according to the technology of the present disclosure.

[0179] 28, the CPU 15A operates as a control unit 90, an acquisition unit 94, and a change unit 98 in accordance with a distance measurement imaging process program to realize distance measurement imaging process. The signal processing circuit 34 includes an A / D converter 34C. The A / D converter 34C is connected to the distance measurement designated segmented area 26N2b, and digitizes the amount of auxiliary reflected light received by the distance measurement designated segmented area 26N2b.

[0180] The acquisition unit 94 acquires the amount of received light digitized by the A / D converter 34C, and calculates the amount of auxiliary reflected light received per unit time by the distance measurement designated division area 26N2b (hereinafter also referred to as "unit time received light amount").

[0181] The storage 15B stores an irradiation energy table 282. The irradiation energy table 282 differs from the irradiation energy table 82 in that the amount of light received per unit time is used instead of the identifier.

[0182] The change unit 98 acquires, from the irradiation energy table 282, intensity information corresponding to the amount of light received per unit time acquired by the acquisition unit 94. Then, the change unit 98 changes the intensity of the auxiliary light irradiated from the LED 202 via the LED driver 204 to the intensity indicated by the intensity information acquired from the irradiation energy table 282.

[0183] In the smart device 200 configured in this manner, as an example, a change process shown in FIG. 29 is executed by the CPU 15A.

[0184] In the change process shown in FIG. 29, first, in step ST200, the control unit 90 judges whether or not the condition for causing the LED 202 to emit auxiliary light (hereinafter also referred to as the "auxiliary light emission condition") is satisfied. One example of the auxiliary light emission condition is that the timing for emitting auxiliary light as a red-eye prevention measure has arrived. In step ST200, if the auxiliary light emission condition is not satisfied, the judgment is negative, and the change process proceeds to step ST210. In step ST200, if the auxiliary light emission condition is satisfied, the judgment is positive, and the change process proceeds to step ST202.

[0185] In step ST202, the control unit 90 causes the LED 202 to emit auxiliary light, and then the change processing proceeds to step ST204.

[0186] In step ST204, the acquisition unit 94 acquires the amount of auxiliary reflected light received per unit time by the distance measurement designated division area 26N2b (hereinafter also referred to as "unit time received light amount") by calculating it, and then the change processing proceeds to step ST206.

[0187] Note that, although the amount of light received per unit time is exemplified here, the technology of the present disclosure is not limited to this, and for example, instead of the amount of light received per unit time, fine-adjusted information obtained by finely adjusting the amount of light received per unit time (for example, fine-adjusted information obtained by multiplying the amount of light received per unit time by a coefficient for fine adjustment) may be applied. Here, the "fine-adjusted information" is an example of "information based on the amount of light received per unit time" according to the technology of the present disclosure.

[0188] In step ST206, the change unit 98 acquires intensity information from the irradiation energy table 282, and then the change process proceeds to step ST208. In this step ST206, the change unit 98 acquires, from the irradiation energy table 282, intensity information corresponding to the amount of light received per unit time calculated in step ST204.

[0189] In step ST208, the change unit 98 changes the intensity of the auxiliary light emitted from the LED 202 via the LED driver 204 to the intensity indicated by the intensity information acquired in step ST206, and then the change processing proceeds to step ST210.

[0190] In step ST210, the control unit 90 judges whether or not the above-mentioned change processing end condition is satisfied. In step ST210, if the change processing end condition is not satisfied, the judgment is negative and the change processing proceeds to step ST200. In step ST210, if the change processing end condition is satisfied, the judgment is positive and the change processing ends.

[0191] 27 to 29, auxiliary light is irradiated from the LED 202 onto the distance measurement target, and the auxiliary reflected light is received by the distance measurement designated section 26N2b. Information based on the light reception result in which the auxiliary reflected light is received is acquired by the acquisition unit 94. The intensity of the auxiliary light is then changed by the change unit 98 in accordance with the information based on the light reception result acquired by the acquisition unit 94. Therefore, with this configuration, the effect of the auxiliary reflected light on the visible light image can be reduced compared to when the intensity of the auxiliary light is determined independently of the information based on the light reception result in which the auxiliary reflected light is received.

[0192] 27 to 29, the amount of received light per unit time is acquired by the acquisition unit 94. Then, the intensity of the auxiliary light is changed by the change unit 98 in accordance with the amount of received light per unit time acquired by the acquisition unit 94. Therefore, with this configuration, the effect of the auxiliary reflected light on the visible light image can be reduced compared to when the intensity of the auxiliary light is determined regardless of the amount of received light per unit time.

[0193] 27 to 29, the intensity of the auxiliary light is changed by the change unit 98 according to the intensity information derived from the irradiation energy table 282. However, the technology of the present disclosure is not limited to this. For example, as shown in FIG. 30, the acquisition unit 94 may acquire intensity information by converting the amount of light received per unit time into the reflectance in the imaging area (hereinafter, simply referred to as "reflectance"). A conversion formula used when converting the amount of light received per unit time into the reflectance may be, for example, a conversion formula in which the amount of light received per unit time is an independent variable and the reflectance in the imaging area is a dependent variable. In the example shown in FIG. 30, the storage 15B stores an irradiation energy table 382. The irradiation energy table 382 is different from the irradiation energy table 282 shown in FIG. 28 in that the reflectance is applied instead of the amount of light received per unit time. The change unit 98 acquires intensity information corresponding to the reflectance acquired by the acquisition unit 94 from the irradiation energy table 382. Then, the changing unit 98 changes the intensity to the intensity indicated by the intensity information obtained from the irradiation energy table 382.

[0194] In the smart device 200 configured in this manner, as an example, the change process shown in Fig. 31 is executed by the CPU 15A. The change process shown in Fig. 31 differs from the change process shown in Fig. 29 in that it includes processes of steps ST300 and ST302 instead of the process of step ST206.

[0195] 31, in step ST300, the acquisition unit 94 calculates the reflectance based on the amount of light received per unit time acquired in step ST204. Specifically, the acquisition unit 94 converts the amount of light received per unit time into the reflectance using a conversion formula.

[0196] In the next step ST302, the changing unit 98 acquires, from the irradiation energy table 382, ​​intensity information corresponding to the reflectance acquired in step ST300.

[0197] 30 and 31, the reflectance in the imaging region is acquired by the acquisition unit 94, and the intensity of the auxiliary light is changed to an intensity corresponding to the reflectance in the imaging region by the change unit 98. Therefore, according to this configuration, the effect of the auxiliary reflected light on the visible light image can be reduced compared to the case where the intensity of the auxiliary light is determined regardless of the reflectance in the imaging region.

[0198] 30 and 31, the intensity of the auxiliary light is changed by the change unit 98 according to the intensity information acquired by the change unit 98 from the irradiation energy table 382. However, the technology of the present disclosure is not limited to this. For example, the change unit 98 may weaken the intensity of the auxiliary light below the reference intensity when the reflectance acquired by the acquisition unit 94 is equal to or greater than a threshold value without using the irradiation energy table 382. Here, the threshold value is, for example, a value equivalent to the amount of received auxiliary reflected light derived in advance by a sensory test and / or a computer simulation, etc., as the amount of received auxiliary reflected light when auxiliary reflected light of an intensity at which the auxiliary reflected light appears as noise in a visible light image is irradiated into the imaging area.

[0199] In the smart device 200 configured in this manner, as an example, the change process shown in Fig. 32 is executed by the CPU 15A. The change process shown in Fig. 32 differs from the change process shown in Fig. 31 in that it includes processes of steps ST400 and ST402 instead of the processes of steps ST302 and ST208.

[0200] 32, in step ST400, the change unit 98 determines whether or not the reflectance calculated in step ST300 is equal to or greater than a threshold value. If the reflectance calculated in step ST300 is less than the threshold value, the determination is negative, and the change process proceeds to step ST210. If the reflectance calculated in step ST300 is equal to or greater than the threshold value, the determination is positive, and the change process proceeds to step ST402.

[0201] In step ST402, the change section 98 reduces the intensity of the auxiliary light below the reference intensity, and then the change process proceeds to step ST210.

[0202] In this way, when the reflectance converted from the amount of light received per unit time is equal to or greater than the threshold value, the intensity of the auxiliary light is made weaker than the reference intensity by the changing unit 98. Therefore, according to this configuration, the effect of the auxiliary reflected light on the visible light image can be reduced compared to when the intensity of the auxiliary light is always equal to or greater than the reference intensity regardless of whether the reflectance converted from the amount of light received per unit time is equal to or greater than the threshold value.

[0203] 27 to 32 illustrate auxiliary light emitted by LED 202, but the technology of the present disclosure is not limited thereto, and the technology of the present disclosure is valid even if the auxiliary light is replaced with laser light in the examples shown in Fig. 27 to 32. Note that, hereinafter, for convenience of explanation, when there is no need to distinguish between auxiliary light and laser light, they will be referred to as "distance measuring light", and when there is no need to distinguish between auxiliary reflected light and IR reflected light, they will be referred to as "reflected light".

[0204] In the above embodiment, the intensity of the distance measurement light is changed by the change unit 98 regardless of whether or not it is the timing for the main exposure to be performed, but the technology of the present disclosure is not limited to this. For example, the intensity of the distance measurement light may be changed by the change unit 98 in accordance with the timing of the main exposure to be performed during the imaging operation. The timing of the main exposure refers to, for example, the timing at which exposure is performed in imaging a still image.

[0205] In this case, as shown in Fig. 33 as an example, a live view image and a soft key 106A are displayed on the display 46 under the control of the control unit 90. The soft key 106A is turned on by a user or the like when the user or the like issues an instruction to start capturing an image for a still image. When the user or the like turns on the soft key 106A while the distance measuring imaging device 14 is capturing an image for a live view image, the intensity of the distance measuring light is changed by the change unit 98 in accordance with the timing of the main exposure. This makes it possible to irradiate the imaging area with distance measuring light at an appropriate intensity during the main exposure, compared to a case in which the intensity of the distance measuring light is changed at a timing unrelated to the main exposure.

[0206] In the above embodiment, the smart device 10 equipped with the light receiver 18 is exemplified, but the technology of the present disclosure is not limited thereto. For example, as shown in FIG. 34, the technology of the present disclosure can be applied to a smart device 300 equipped with light receivers 18 and 350. As an example, as shown in FIG. 34, a light-transmitting window 352 is provided adjacent to the light-transmitting window 22 in the upper left part of the back surface 12A of the housing 12 when the smart device 10 is placed vertically (upper left part when viewed from the back surface of the smart device 10 placed vertically). The light-transmitting window 352 is an optical element (e.g., a lens) having translucency, similar to the light-transmitting windows 20 and 22, and the light-transmitting windows 20, 22, and 352 are arranged at a predetermined interval along the horizontal direction. The light-transmitting window 352 is also exposed from the back surface 12A, similar to the light-transmitting windows 20 and 22.

[0207] The receiver 350 includes a photoelectric conversion element 354. The photoelectric conversion element 354 is a photoelectric conversion element specialized for receiving reflected IR light, and includes a plurality of IR pixels arranged in a matrix. An example of the plurality of IR pixels is a photodiode for IR pixels (e.g., InGaAs APD) of 4896×3265 pixels. The photoelectric conversion element 354 receives the reflected IR light taken into the receiver 350 through the light-transmitting window 352, and outputs an electrical signal according to the amount of light of the received reflected IR light to the signal processing circuit 34 (see FIG. 9).

[0208] In addition, in the example shown in FIG. 34, the smart device 300 equipped with the light receivers 18 and 350 is shown, but the technology of the present disclosure is not limited thereto. For example, as shown in FIG. 35, the technology of the present disclosure is also applicable to a smart device 400 equipped with the light receivers 18 and 450. As an example, as shown in FIG. 35, a light-transmitting window 452 is provided adjacent to the light-transmitting window 20 in the upper left part of the back surface 12A of the housing 12 when the smart device 10 is placed vertically (upper left part when viewed from the back surface of the smart device 10 placed vertically). The light-transmitting window 452 is an optical element (for example, a lens) having translucency, similar to the light-transmitting windows 20 and 22, and the light-transmitting windows 452, 20, and 22 are arranged at a predetermined interval along the horizontal direction. The light-transmitting window 452 is also exposed from the back surface 12A, similar to the light-transmitting windows 20 and 22.

[0209] The optical receiver 450 includes a single photodiode 454. The photodiode 454 is, for example, a photodiode capable of receiving reflected IR light. An example of the photodiode 454 is an InGaAs APD. The photodiode 454 receives the reflected IR light taken into the optical receiver 450 through the light-transmitting window 452, and outputs an electrical signal according to the amount of the received reflected IR light to the signal processing circuit 34 (see FIG. 9).

[0210] In addition, in the above embodiment, an example is given in which a distance image is generated by the distance image generating circuit 34B, but the technology of the present disclosure is not limited to this, and the distance measuring imaging device 14 may perform an operation of measuring the distance from the smart device 10 to the distance measurement target without generating a distance image as a distance measurement operation.

[0211] In addition, in the above embodiment, an example is described in which the intensity of the ranging light is changed, but the technology disclosed herein is not limited to this, and instead of or together with the intensity of the ranging light, the irradiation energy of the ranging light may be changed by changing the emission time of the ranging light and / or the number of times the ranging light is emitted per unit time.

[0212] In the above embodiment, the distance measurement section 26N2 is changed to the distance measurement designated section 26N2a, so that the distance measurement is performed by focusing on the distance measurement target (so-called ROI) designated by the user, etc., but the method of focusing on the distance measurement target is not limited to this. For example, the control unit 90 may control to change the beam diameter and / or direction of the laser light according to an instruction accepted by the accepting device 47 so that the laser light is irradiated on the distance measurement target designated by the user, etc.

[0213] In addition, in the above embodiment, laser light has been described as an example of "directional light" according to the technology of the present disclosure, but the technology of the present disclosure is not limited to this, and superluminescent light may be used instead of laser light, and distance measurement may be performed using light having a directionality that allows distance measurement.

[0214] In the above embodiment, the distance measuring imaging device 14 is built into the smart device 10, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 36, the distance measuring imaging device 14 may be externally attached to a general smart device 500, that is, a smart device 500 that does not have a built-in distance measuring imaging device 14.

[0215] In the above embodiment, the UI device 44 is incorporated in the smart device 10, but at least some of the components included in the UI device 44 may be externally attached to the smart device 10. In addition, at least some of the components included in the UI device 44 may be used by being connected to the external I / F 52 as separate entities.

[0216] 1 illustrates a smart device 10, but the technology of the present disclosure is not limited thereto. That is, the technology of the present disclosure can be applied to various electronic devices (e.g., interchangeable lens cameras, fixed lens cameras, personal computers, and / or wearable terminal devices) incorporating a distance measuring imaging device 14, and the same actions and effects as those of the smart device 10 can be obtained with these electronic devices.

[0217] In addition, in the above embodiment, the display 46 is exemplified, but the technology of the present disclosure is not limited to this. For example, a separate display attached to the smart device 10 may be used as the "display unit" according to the technology of the present disclosure.

[0218] In the above embodiment, the recognition unit 92 is mounted on the smart device 10, but the technology of the present disclosure is not limited thereto. For example, an external device (e.g., another smart device, a personal computer, and / or a server) having the recognition unit 92 may be connected to the smart device 10. In this case, the image recognition result provided from the external device to the smart device 10 may be acquired by the acquisition unit 94 of the smart device 10. In addition, the function of the recognition unit 92 may be performed by cloud computing (not shown), and the image recognition result may be provided from the cloud computing to the smart device 10. In this case, the image recognition result provided from the cloud computing to the smart device 10 may be acquired by the acquisition unit 94 of the smart device 10.

[0219] In the above embodiment, the distance measurement imaging processing program is stored in the storage 15B, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 37, the distance measurement imaging processing program may be stored in a storage medium 900. An example of the storage medium 900 is any portable storage medium such as an SSD or a USB memory, which is a non-transitory storage medium.

[0220] The distance measurement imaging processing program stored in the storage medium 900 is installed in the controller 15. The CPU 15A executes the distance measurement imaging processing in accordance with the distance measurement imaging processing program.

[0221] In addition, the distance measurement imaging processing program may be stored in a memory unit of another computer or server device connected to the controller 15 via a communication network (not shown), and the distance measurement imaging processing program may be downloaded and installed in the controller 15 in response to a request from the smart device 10 described above.

[0222] It is not necessary to store the entire distance measurement imaging processing program in a storage unit of another computer or server device connected to the controller 15, or in the storage 15B, and only a part of the distance measurement imaging processing program may be stored therein.

[0223] In the example shown in FIG. 37, an example is shown in which the controller 15 is built into the smart device 10, but the technology of the present disclosure is not limited to this, and for example, the controller 15 may be provided outside the smart device 10.

[0224] 37, the CPU 15A is a single CPU, but it may be a plurality of CPUs. Also, a GPU may be applied in place of the CPU 15A.

[0225] 37 illustrates a controller 15, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the controller 15. Also, instead of the controller 15, a combination of a hardware configuration and a software configuration may be used.

[0226] As the hardware resource for executing the distance measurement imaging process described in the above embodiment, the following various processors can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the distance measurement imaging process by executing software, i.e., a program. Another example of the processor is a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, PLD, or ASIC. Each processor has a built-in or connected memory, and each processor executes the distance measurement imaging process by using the memory.

[0227] The hardware resource for executing the distance measurement imaging process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the distance measurement imaging process may be a single processor.

[0228] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the distance measurement imaging process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute the distance measurement imaging process in a single IC chip, as typified by SoC. In this way, the distance measurement imaging process is realized using one or more of the above various processors as hardware resources.

[0229] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above-mentioned distance measurement and imaging process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0230] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0231] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0232] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0233] The following supplementary notes are further disclosed regarding the above embodiment.

[0234] (Additional Note) A processor; A memory connected to or embedded in the processor; The processor is performing control to cause an imaging operation by an imaging unit that images an imaging area, and a distance measurement operation by a distance measurement unit that irradiates light onto the imaging area and receives reflected light of the light onto the imaging area, thereby measuring distance; Obtaining reflectance information capable of identifying a reflectance in the imaging region; and and executing a process including changing the irradiation energy of the light in accordance with the acquired reflectance information while the imaging operation and the distance measuring operation are being performed. Processing unit.

Claims

1. A first processor; A memory connected to or embedded in the first processor, The first processor, performing control to cause an imaging operation by an imaging device that captures an imaging region and a distance measurement operation by a distance measuring device that irradiates light onto the imaging region and receives light reflected by the light onto the imaging region to measure distance; acquiring reflectance information capable of identifying a reflectance in the imaging region; changing the irradiation energy of the light in accordance with imaging scene instruction information that is received by a reception device capable of receiving imaging scene instruction information instructing an imaging scene and the acquired reflectance information while the imaging operation and the distance measurement operation are being performed; the reflectance information is information based on a visible light image obtained by capturing an image of the imaging area by the imaging device, the information based on the visible light image is an image recognition result obtained by performing image recognition on the visible light image by a second processor, the image recognition result includes an identifier capable of identifying each of a plurality of types of subjects having different reflectances; The irradiation energy is determined for each combination of the identifier and the imaging scene instruction information. Processing unit.

2. The imaging device and the distance measuring device each have a photoelectric conversion element, The photoelectric conversion element is The imaging device and the distance measuring device are used together, a visible light image segmented region having a visible light pixel group that receives visible reflected light included in the reflected light, and a distance measurement segmented region having a distance measurement pixel group that receives infrared reflected light included in the reflected light, In the imaging operation, the visible light image division area receives the visible reflected light to image the imaging area, thereby generating the visible light image; In the distance measurement operation, the distance measurement is performed by the distance measurement divided area receiving the infrared reflected light. The processing device of claim 1 .

3. A period during which the imaging operation using the visible light image division area is performed and a period during which the distance measurement operation using the distance measurement division area is performed overlap. The processing device according to claim 2 .

4. While one frame is captured by the imaging operation using the visible light image division area, the distance measurement is performed multiple times by the distance measurement operation using the distance measurement division area. The processing device according to claim 2 or 3.

5. The first processor acquires the image recognition result based on a learning result obtained by machine learning of a correspondence relationship between an image and an object recognition result. The processing apparatus according to any one of claims 1 to 4.

6. The second processor performs the image recognition based on dictionary data in which an image of an object having a reflectance equal to or greater than a reference reflectance is associated with an identifier capable of identifying the image of the object. The processing apparatus according to any one of claims 1 to 5.

7. When the image recognition result obtained by the first processor indicates that the visible light image contains an image showing an object having a reflectance equal to or higher than a reference reflectance, the first processor weakens the irradiation energy below a first reference irradiation energy. The processing apparatus according to any one of claims 1 to 6.

8. The object is a predetermined object having gloss. The processing device of claim 7.

9. The object is a predetermined object that is a white object. The processing device of claim 7.

10. The predetermined object is a mirror-like object. The processing device of claim 8.

11. The first processor performs control to generate auxiliary light as the light from a light source and to cause a light receiving element to receive auxiliary reflected light due to the auxiliary light with respect to the imaging area; The reflectance information is information based on a light receiving result in which the auxiliary reflected light is received by the light receiving element. The processing device according to any one of claims 1 to 10.

12. The information based on the light receiving result is information based on the amount of the auxiliary reflected light received by the light receiving element per unit time. The processing device of claim 11.

13. The first processor reduces the irradiation energy to be lower than a second reference irradiation energy when the reflectance specified by the acquired reflectance information is equal to or higher than a threshold value. The processing device according to any one of claims 1 to 12.

14. The distance measuring device has a TOF camera. The processing device according to any one of claims 1 to 13.

15. The first processor causes a display to display a range image generated by the TOF camera. The processing device of claim 14.

16. The TOF camera has a plurality of photoelectric conversion pixels, The distance measurement is performed using the reflected light received only by at least one photoelectric conversion pixel included in a specified region among the plurality of photoelectric conversion pixels. The processing device according to claim 14 or 15.

17. The at least one photoelectric conversion pixel is a photoelectric conversion pixel at a position corresponding to a position of an image area designated within a screen in a state in which an imaging area image obtained by imaging the imaging area by the imaging device is displayed.

17. The processing device of claim 16.

18. The first processor changes the irradiation energy in accordance with a timing of a main exposure performed during the imaging operation.

18. The processing device according to any one of claims 1 to 17.

19. The imaging operation includes an imaging operation for a live view image in which the imaging device captures an image of the imaging area for a live view image.

19. The processing device according to any one of claims 1 to 18.

20. The distance measurement operation is a distance measurement operation for focusing by the imaging device.

20. The processing device according to any one of claims 1 to 19.

21. The imaging device performs focusing on the imaging area based on a distance measurement result obtained by performing the distance measurement in the focusing distance measurement operation.

21. The processing device of claim 20.

22. The light is directional light.

22. The processing device according to any one of claims 1 to 21.

23. A processing device according to any one of claims 1 to 22; The imaging device; The distance measuring device; Electronic devices, including

24. performing control to cause an imaging operation by an imaging device that captures an imaging region and a distance measurement operation by a distance measurement device that irradiates light onto the imaging region and receives reflected light of the light onto the imaging region to measure distance; acquiring reflectance information capable of identifying a reflectance in the imaging region; and changing the irradiation energy of the light in accordance with image capture scene instruction information that is accepted by a reception device capable of accepting image capture scene instruction information instructing an image capture scene and the acquired reflectance information while the image capture operation and the distance measurement operation are being performed; the reflectance information is information based on a visible light image obtained by capturing an image of the imaging area by the imaging device, the information based on the visible light image is an image recognition result obtained by performing image recognition on the visible light image by a second processor, the image recognition result includes an identifier capable of identifying each of a plurality of types of subjects having different reflectances; The irradiation energy is determined for each combination of the identifier and the imaging scene instruction information. Processing methods.

25. A program for causing a computer to execute a process, The process comprises: performing control to cause an imaging operation by an imaging device that images an imaging region, and a distance measuring operation by a distance measuring device that irradiates light onto the imaging region and receives light reflected by the light onto the imaging region, thereby measuring distance; Obtaining reflectance information capable of identifying a reflectance in the imaging region; and changing the irradiation energy of the light in accordance with image capture scene instruction information that is accepted by a reception device capable of accepting image capture scene instruction information instructing an image capture scene and the acquired reflectance information while the image capture operation and the distance measurement operation are being performed; the reflectance information is information based on a visible light image obtained by capturing an image of the imaging area by the imaging device, the information based on the visible light image is an image recognition result obtained by performing image recognition on the visible light image by a second processor, the image recognition result includes an identifier capable of identifying each of a plurality of types of subjects having different reflectances; The irradiation energy is determined for each combination of the identifier and the imaging scene instruction information. program.

Images