Processing device, electronic device, processing method and storage medium

Abstract

The processing device of the present invention includes: a control unit that executes control of performing a camera action and a distance measurement action, wherein the camera action is performed by a camera unit that captures a camera area, and in the distance measurement action, the distance measurement is performed by irradiating light to the camera area by the distance measurement unit and receiving reflected light based on the light to the camera area; an acquisition unit that acquires reflectivity information that can determine the reflectivity in the camera area; and a change unit that changes the irradiation energy of the light according to the reflectivity information acquired by the acquisition unit while performing the camera action and the distance measurement action.

Description

Technical Field

[0001] The technology of the present invention relates to a processing device, an electronic device, a processing method and a storage medium. Background Art

[0002] Japanese Patent No. 6321145 discloses a distance measuring device. The distance measuring device described in Japanese Patent No. 6321145 includes: a photographing unit that photographs a subject image formed by an imaging optical system that images the subject image; an emitting unit that emits directional light as directional light along the optical axis direction of the imaging optical system, and can adjust the emission intensity of the directional light, and emits the directional light by adjusting the emission intensity according to at least one of focus state determination information, subject brightness, or exposure state determination information; a light receiving unit that receives reflected light of the directional light from the subject; a deriving unit that derives the distance to the subject according to the timing of emitting the directional light by the emitting unit and the timing of receiving the reflected light by the light receiving unit; an executing unit that performs at least one of focusing and exposure adjustment of the imaging optical system on the subject before photography based on the photographing unit; and a receiving unit that receives a first pressing operation and a photography preparation instruction indicating a photography preparation. The control unit comprises: a first pressing operation indicating the photographing preparation so that the execution unit starts to execute at least one of the focusing and exposure adjustments, and a second pressing operation indicating the photographing preparation starts the formal exposure based on the photographing unit; and a control unit, when the first pressing operation is received by the receiving unit, controls the execution unit to execute at least one of the focusing and exposure adjustments, and starts the distance measurement based on the emitting unit, the light receiving unit and the derivation unit on the condition that at least one of the focusing and exposure adjustments has been executed, and after the distance measurement is completed, the prompting unit for prompting information prompts information related to the result of the distance measurement while maintaining the first pressing operation, and when the second pressing operation performed subsequent to the first pressing operation is received by the receiving unit without releasing the pressing operation on the receiving unit, controls the formal exposure based on the photographing unit.

[0003] Japanese Patent Application Laid-Open No. 2006-171120 discloses a photographing device. The photographing device described in Japanese Patent Application Laid-Open No. 2006-171120 is a photographing device that detects the contrast of a subject to adjust the focus before a photographing operation, and is characterized by comprising: a first auxiliary light emitter that irradiates light with a relatively wide wavelength range toward the subject for subject contrast detection; a second auxiliary light emitter that irradiates light with a relatively narrow wavelength range toward the subject for subject contrast detection; and a switching mechanism that switches between the first auxiliary light emitter and the second auxiliary light emitter.

[0004] International Publication No. 2018 / 142993 discloses a light emission control device. The light emission control device described in International Publication No. 2018 / 142993 includes a light intensity setting unit for setting the light intensity of AF (auto focus) auxiliary light and a light emission control unit for controlling the emission of the AF auxiliary light according to the setting of the light intensity setting unit. Summary of the invention

[0005] One embodiment of the technology involved in the present invention provides a processing device, electronic device, processing method and program that can reduce the influence of reflected light for ranging on the image of the camera area, compared with the case where the irradiation energy of light irradiated for ranging is determined regardless of the reflectivity in the camera area.

[0006] Means for solving technical problems

[0007] The first method involved in the technology of the present invention is a processing device, which includes: a control unit that executes control of performing a camera action and a distance measurement action, the camera action is performed by a camera unit that captures a camera area, and in the distance measurement action, the distance measurement is performed by irradiating light to the camera area by the distance measurement unit and receiving reflected light based on the light with respect to the camera area; an acquisition unit that acquires reflectivity information that can determine the reflectivity in the camera area; and a change unit that changes the irradiation energy of the light according to the reflectivity information acquired by the acquisition unit while the camera action and the distance measurement action are performed.

[0008] A second aspect according to the technology of the present invention is the processing device according to the first aspect, wherein the reflectance information is information based on an imaging region image obtained by imaging the imaging region with the imaging unit.

[0009] A third aspect according to the technology of the present invention is the processing device according to the second aspect, wherein the information based on the imaging region image is an image recognition result obtained by performing image recognition on the imaging region image by the recognition unit.

[0010] A fourth aspect of the technology of the present invention is the processing device according to the third aspect, wherein the processor acquires the image recognition result based on a learning result obtained by machine learning a correspondence relationship between the image and the subject recognition result.

[0011] The fifth mode involved in the technology of the present invention is a processing device involved in the third mode or the fourth mode, wherein, when the image recognition result acquired by the acquisition unit is an image recognition result indicating that an image of an object having a reflectivity greater than a baseline is included in the image of the camera area, the changing unit makes the irradiation energy weaker than the first baseline irradiation energy.

[0012] A sixth aspect according to the technology of the present invention is the processing device according to the fifth aspect, wherein the object is an object predetermined as an object having glossiness.

[0013] A seventh aspect according to the technology of the present invention is the processing device according to the sixth aspect, wherein the predetermined object is a mirror-shaped object.

[0014] The 8th mode involved in the technology of the present invention is a processing device involved in any one of the 1st to 7th modes, wherein the changing unit changes the irradiation energy of light according to the shooting scene indication information received by the receiving unit and the reflectivity information acquired by the acquiring unit, and the receiving unit is capable of receiving the shooting scene indication information indicating the shooting scene.

[0015] The 9th method involved in the technology of the present invention is the processing device involved in the 8th method, wherein, when the shooting scene indicated by the shooting scene indication information received by the receiving unit is a specific shooting scene having a shooting scene reflectivity greater than a baseline reflectivity, the changing unit makes the irradiation energy weaker than the second baseline irradiation energy.

[0016] The 10th method involved in the technology of the present invention is a processing device involved in any one of the 1st to 7th methods, wherein the changing unit changes the irradiation energy of light according to the shooting scene determined by the determining unit and the reflectivity information acquired by the acquiring unit, and the determining unit can determine the shooting scene based on the camera area image.

[0017] The 11th method involved in the technology of the present invention is the processing device involved in the 10th method, wherein, when the shooting scene determined by the determination unit is a specific shooting scene having a shooting scene reflectivity greater than a baseline reflectivity, the changing unit makes the irradiation energy weaker than the third baseline irradiation energy.

[0018] The 12th method involved in the technology of the present invention is a processing device involved in any one of the 1st to 11th methods, wherein the control unit performs the following control, namely, auxiliary light is generated from a light source as light, and a light receiving element receives auxiliary reflected light based on the auxiliary light for a camera area, and the reflectance information is information based on the light receiving result of receiving the auxiliary reflected light by the light receiving element.

[0019] A thirteenth aspect according to the technology of the present invention is the processing device according to the twelfth aspect, wherein the information based on the light reception result is information based on the amount of light received per unit time of the auxiliary reflected light received by the light receiving element.

[0020] The 14th method involved in the technology of the present invention is a processing device involved in the 12th method or the 13th method, wherein when the reflectance determined based on the reflectance information acquired by the acquisition unit is above a threshold value, the change unit makes the irradiation energy weaker than the fourth reference irradiation energy.

[0021] A fifteenth aspect according to the technology of the present invention is the processing device according to any one of the first to fourteenth aspects, wherein the distance measuring unit includes a TOF camera.

[0022] A sixteenth aspect according to the technology of the present invention is the processing device according to the fifteenth aspect, wherein the control unit displays the distance image generated by the TOF camera on the display unit.

[0023] The 17th method involved in the technology of the present invention is a processing device involved in the 15th method or the 16th method, wherein the TOF camera has multiple photoelectric conversion pixels, and the ranging is performed using reflected light received only by at least one photoelectric conversion pixel included in a specified area among the multiple photoelectric conversion pixels.

[0024] The 18th mode involved in the technology of the present invention is the processing device involved in the 17th mode, wherein at least one photoelectric conversion pixel is a photoelectric conversion pixel at a position corresponding to the position of the image area specified in the screen in a state where the camera area image is displayed.

[0025] A nineteenth aspect according to the technology of the present invention is the processing device according to any one of the first to eighteenth aspects, wherein the changing unit changes the irradiation energy according to the timing of the main exposure performed in the imaging operation.

[0026] A 20th aspect according to the technology of the present invention is the processing device according to any one of the 1st to 19th aspects, wherein the imaging operation includes an imaging operation for a live view image in which the imaging unit images an imaging area for the live view image.

[0027] A 21st aspect according to the technology of the present invention is the processing device according to any one of the 1st to 20th aspects, wherein the distance measurement operation is a focusing distance measurement operation by an imaging unit.

[0028] A 22nd aspect according to the technology of the present invention is the processing device according to the 21st aspect, wherein the imaging unit focuses on the imaging area based on a distance measurement result obtained by distance measurement in a focusing distance measurement operation.

[0029] A 23rd aspect according to the technology of the present invention is the processing device according to any one of the first to 22nd aspects, wherein the light is directional light.

[0030] A 24th aspect according to the technology of the present invention is an electronic device including: the processing device according to any one of the 1st to 23rd aspects; an imaging unit; and a distance measuring unit.

[0031] The 25th method involved in the technology of the present invention is a processing method, which includes the following steps: executing control of performing a camera action and a distance measuring action, the camera action is performed by a camera unit that shoots a camera area, and in the distance measuring action, the distance measuring unit irradiates light to the camera area and receives reflected light based on the light for the camera area to perform distance measuring; obtaining reflectivity information that can determine the reflectivity in the camera area; and changing the irradiation energy of the light according to the obtained reflectivity information while performing the camera action and the distance measuring action.

[0032] The 26th method involved in the technology of the present invention is a program for causing a computer to execute processing including the following steps: executing control of performing a camera action and a distance measuring action, wherein the camera action is performed by a camera unit that captures a camera area, and in the distance measuring action, the distance measuring unit irradiates the camera area with light and receives reflected light based on the light for the camera area; obtains reflectivity information that can determine the reflectivity within the camera area; and changes the irradiation energy of the light according to the obtained reflectivity information while performing the camera action and the distance measuring action. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic three-dimensional diagram showing an example of how the smart device according to the embodiment is used.

[0034] Figure 2 This is a rear perspective view showing an example of the appearance of the rear side of the smart device involved in the embodiment.

[0035] Figure 3 It is a schematic perspective view showing an example of the structure of a photoelectric conversion element included in a smart device according to an embodiment.

[0036] Figure 4 Yes means Figure 2 A front perspective view showing an example of the appearance of the front surface side of the smart device shown.

[0037] Figure 5 This is a conceptual diagram showing an example of a method in which a ranging camera included in a smart device according to an embodiment captures an image of a shooting area and a visible light image is displayed on a display.

[0038] Figure 6 This is a conceptual diagram showing an example of a method in which a laser beam is irradiated onto an imaging area by a distance measuring imaging device included in a smart device according to an embodiment.

[0039] Figure 7 This is a conceptual diagram showing an example of a method in which a distance measurement imaging device included in a smart device according to an embodiment measures distance using an imaging area as a distance measurement target, and displays the distance measurement result on a display as a distance image.

[0040] Figure 8 This is a conceptual diagram showing an example of a method in which a distance-measuring imaging device included in a smart device according to an embodiment measures distance using an imaging area as a distance-measuring target, and a distance superimposed image reflecting the distance-measuring result is displayed on a display.

[0041] Fig. 9 This is a block diagram showing an example of the configuration of electrical system hardware of a smart device according to the embodiment.

[0042] Fig.10 This is a conceptual diagram showing an example of storage content of a storage device included in a smart device according to an embodiment.

[0043] Fig.11 This is a conceptual diagram showing an example of detailed content of the distance measurement imaging process executed by the CPU included in the smart device according to the embodiment.

[0044] Fig.12 This is a functional block diagram showing an example of the functions of the CPU when the CPU included in the smart device according to the embodiment performs distance measurement imaging processing.

[0045] Fig.13 This is a schematic screen diagram showing an example of a display mode selection screen displayed on a display included in a smart device according to an embodiment.

[0046] Fig.14 This is a schematic screen diagram showing an example of a distance measurement range designation guidance screen displayed on a display included in a smart device according to an embodiment.

[0047] Fig.15 This is a schematic screen diagram showing an example of a visible light image and a designated image area displayed as a live view image on a display included in a smart device according to an embodiment.

[0048] Fig.16 This is a conceptual diagram for explaining an example of a method of reducing the distance measurement divided area to the distance measurement designated divided area of ​​the photoelectric conversion element included in the smart device.

[0049] Fig.17 This is a flowchart showing an example of the flow of parallel processing according to the embodiment.

[0050] Fig.18 This is a flowchart showing an example of the flow of imaging processing involved in the embodiment.

[0051] Fig.19 This is a flowchart showing an example of the flow of the distance measurement process according to the embodiment.

[0052] Fig. 20 This is a flowchart showing an example of the flow of change processing involved in the embodiment.

[0053] Fig.21A This is a flowchart showing an example of the flow of the display control process according to the embodiment.

[0054] Fig.21B yes Fig.21A Continuation of the flowchart shown.

[0055] Fig. 22 This is a functional block diagram showing a first modified example of the functions of the CPU when the CPU included in the smart device according to the embodiment performs ranging imaging processing.

[0056] Fig.23 This is a schematic screen diagram showing an example of a shooting scene selection screen displayed on a display included in a smart device according to an embodiment.

[0057] Fig.24 This is a flowchart showing a first modified example of the flow of the change process according to the embodiment.

[0058] Fig.25 This is a functional block diagram showing a second modified example of the functions of the CPU when the CPU included in the smart device according to the embodiment performs ranging imaging processing.

[0059] Fig.26 This is a flowchart showing a second variation of the flow of the change process according to the embodiment.

[0060] Fig. 27 This is a block diagram showing a first variation of the configuration of the electrical system hardware of the smart device according to the embodiment.

[0061] Fig.28 This is a functional block diagram showing a third modified example of the functions of the CPU when the CPU included in the smart device according to the embodiment performs ranging imaging processing.

[0062] Fig.29 This is a flowchart showing a third variation of the flow of the change process according to the embodiment.

[0063] Fig.30 This is a functional block diagram showing a fourth modified example of the functions of the CPU when the CPU included in the smart device according to the embodiment performs ranging imaging processing.

[0064] Fig.31 This is a flowchart showing a fourth variation of the flow of the change process according to the embodiment.

[0065] Fig.32 This is a flowchart showing a fifth variation of the flow of the change process according to the embodiment.

[0066] Fig.33 This is a conceptual diagram showing an example of a timing for changing the intensity of distance measuring light emitted by a distance measuring imaging device included in a smart device according to an embodiment.

[0067] Fig.34 It is a schematic perspective view showing a first modified example of the external appearance structure of the smart device involved in the embodiment.

[0068] Fig.35 It is a schematic three-dimensional diagram showing a second modified example of the external appearance structure of the smart device involved in the embodiment.

[0069] Fig.36 It is a schematic perspective view showing an example of an external structure of a combination of a distance measuring camera apparatus and a smart device according to an embodiment.

[0070] Fig.37 This is a conceptual diagram showing an example of how the ranging imaging processing program according to the embodiment is installed in a smart device. DETAILED DESCRIPTION

[0071] Hereinafter, an example of an embodiment of an imaging device according to the technology of the present invention will be described with reference to the drawings.

[0072] First, the words and phrases used in the following description are explained.

[0073] CPU is the abbreviation of "Central Processing Unit". RAM is the abbreviation of "Random Access Memory". ASIC is the abbreviation of "Application Specific Integrated Circuit". PLD is the abbreviation of "Programmable Logic Device". FPGA is the abbreviation of "Field-Programmable Gate Array". SoC is the abbreviation of "System-on-a-chip". SSD is the abbreviation of "Solid State Drive". USB is the abbreviation of "Universal Serial Bus". HDD is the abbreviation of "Hard Disk Drive". EEPROM is the abbreviation of "Electrically Erasable and Programmable Read Only Memory". EL is the abbreviation of "Electro-Luminescence". A / D is the abbreviation of "Analog / Digital". I / F is the abbreviation of "Interface". UI is the abbreviation of "User Interface". LTE is the abbreviation of "Long Term Evolution". 5G is the abbreviation of "5th Generation". LD is the abbreviation of "Laser Diode". IR is the abbreviation of "Infrared". APD is the abbreviation of "Avalanche Photodiode". TOF is the abbreviation of "Time of Flight". fps is the abbreviation of "frame per second". LED is the abbreviation of "Light Emitting Diode". ROI is the abbreviation of "Region of Interest". LAN is the abbreviation of "Local Area Network".

[0074] In the description of this specification, "horizontal" means horizontal in the sense of including the error generally allowed in the technical field to which the technology of the present invention belongs, in addition to being completely horizontal. In the description of this specification, "parallel" means parallel in the sense of including the error generally allowed in the technical field to which the technology of the present invention belongs, in addition to being completely parallel. In the description of this specification, "vertical" means vertical in the sense of including the error generally allowed in the technical field to which the technology of the present invention belongs, in addition to being completely vertical. In the description of this specification, "identical" means identical in the sense of including the error generally allowed in the technical field to which the technology of the present invention belongs, in addition to being completely identical.

[0075] As an example, Figure 1 As shown, the smart device 10 performs a camera operation (hereinafter, also referred to as "camera operation") and a distance measurement operation (hereinafter, also referred to as "distance measurement operation"). In the camera operation, a camera area specified by a field angle θ1 is photographed. In the distance measurement operation, a laser beam is irradiated to the camera area and reflected light based on the laser beam to the camera area is received to perform distance measurement. The laser beam is an example of "directional light" involved in the technology of the present invention.

[0076] In this embodiment, “distance measurement” refers to the process of measuring the distance from the smart device 10 to the distance measurement object within the imaging area. Here, the “distance measurement object” refers to an object that reflects the laser beam. Figure 1 In the example shown, a person and a tree are shown as the distance measurement object in the imaging area. In addition, as the smart device 10, for example, a smartphone or a tablet terminal, which is an electronic device with an imaging function, can be cited.

[0077] As an example, Figure 2 As shown, the smart device 10 includes a housing 12. A distance measuring camera 14 is accommodated in the housing 12. The distance measuring camera 14 includes a light irradiator 16 and a light receiver 18. The light irradiator 16 includes an LD 24, and the light receiver 18 includes a photoelectric conversion element 26. In the smart device 10, the imaging operation and the distance measuring operation are performed by the distance measuring camera 14. In addition, the distance measuring camera 14 is an example of the "imaging unit (imaging device)" and the "distance measuring unit (distance measuring device)" involved in the technology of the present invention.

[0078] An indication key 13 is disposed on the side of the smart device 10. The indication key 13 receives various indications. The "various indications" mentioned here refer to, for example, an indication to display a menu screen from which various menus can be selected, an indication to select one or more menus, an indication to confirm the selected content, and an indication to delete the selected content.

[0079] Light-transmitting windows 20 and 22 are provided at the upper left portion of the back side 12A of the frame 12 when the smart device 10 is set to a vertical position (the upper left portion when the smart device 10 in the vertical position is viewed from the rear). The light-transmitting windows 20 and 22 are optical elements (e.g., lenses) having light transmittance, which are arranged at a predetermined interval (e.g., an interval of several millimeters) along the horizontal direction and exposed from the back side 12A. The light irradiator 16 irradiates the laser beam emitted from the LD 24 to the distance measurement object through the light-transmitting window 20. In the present embodiment, a laser beam in the infrared wavelength region is used. However, the wavelength region of the laser beam is not limited thereto, and it may also be a laser beam in other wavelength regions.

[0080] The light receiver 18 receives IR reflected light via the light-transmitting window 22. IR reflected light refers to reflected light based on the laser beam irradiated on the distance measurement object by the light irradiator 16. In addition, the light receiver 18 receives visible reflected light via the light-transmitting window 22. Visible reflected light refers to reflected light based on visible light (for example, visible light contained in sunlight) irradiated on the imaging area. In addition, for the sake of convenience, when there is no need to distinguish between IR reflected light and visible reflected light for explanation, it is simply referred to as "reflected light".

[0081] The photoreceiver 18 includes a photoelectric conversion element 26 , and the photoelectric conversion element 26 receives the reflected light received by the photoreceiver 18 via the light-transmitting window 22 , and outputs an electrical signal according to the light amount of the received reflected light.

[0082] As an example, Figure 3 As shown, the photoelectric conversion element 26 has a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896×3265" pixels.

[0083] Each photodiode included in the photoelectric conversion element 26 is provided with a color filter. The color filter includes a G filter corresponding to the G (green) wavelength region that is most helpful in obtaining a brightness signal, an R filter corresponding to the R (red) wavelength region, a B filter corresponding to the B (blue) wavelength region, and an IR filter corresponding to the IR (infrared) wavelength region. In addition, in the present embodiment, the G filter, the R filter, and the B filter also have the function of being an infrared light cutoff filter that cuts off infrared light. In addition, for the sake of convenience, when the G filter, the R filter, and the B filter are described without distinguishing between them, they are also referred to as "visible light filters".

[0084] The photoelectric conversion element 26 has R pixels, G pixels, B pixels, and IR pixels. R pixels are pixels corresponding to photodiodes equipped with R filters, G pixels are pixels corresponding to photodiodes equipped with G filters, B pixels are pixels corresponding to photodiodes equipped with B filters, and IR pixels are pixels corresponding to photodiodes equipped with IR filters. R pixels, G pixels, B pixels, and IR pixels are arranged in a prescribed periodicity in the row direction (horizontal direction) and the column direction (vertical direction), respectively. In the present embodiment, the arrangement of R pixels, G pixels, B pixels, and IR pixels is an arrangement obtained by replacing a part of the G pixels in the X-Trans (registered trademark) arrangement with IR pixels. The IR pixels are arranged in a specific periodicity along the row direction and the column direction.

[0085] In addition, here, as an arrangement of R pixels, G pixels, B pixels and IR pixels, an arrangement based on the X-Trans arrangement is illustrated, but the technology of the present invention is not limited to this, and the arrangement of R pixels, G pixels, B pixels and IR pixels may also be an arrangement based on other arrangements such as the Bayer arrangement or the Honeycomb (registered trademark) arrangement.

[0086] Here, as an arrangement of R pixels, G pixels, and B pixels, an arrangement obtained by replacing a part of the G pixels in a commonly known arrangement with IR pixels is exemplified as an arrangement of R pixels, G pixels, B pixels, and IR pixels, but the technology of the present invention is not limited to this. For example, each color filter corresponding to the R pixel, G pixel, and B pixel (hereinafter, these are also referred to as "visible light pixels") can be used as a color filter that transmits infrared light, and a pair of photodiodes based on a photodiode for a visible light pixel and a photodiode for an IR pixel (for example, InGaAs APD) can be configured in one color filter.

[0087] In the present embodiment, the photoelectric conversion element 26 is divided into two areas. That is, the photoelectric conversion element 26 has a division area 26N1 for visible light images and a division area 26N2 for distance measurement. The division area 26N1 for visible light images is a visible light pixel group based on a plurality of visible light pixels, and is used to generate visible light images. The division area 26N2 for distance measurement is an IR pixel group based on a plurality of IR pixels, and is used for distance measurement. The division area 26N1 for visible light images receives visible reflected light and outputs an electrical signal corresponding to the amount of light received. The division area 26N2 for distance measurement receives IR reflected light and outputs an electrical signal corresponding to the amount of light received.

[0088] As an example, Figure 4As shown, a touch panel display 59 is provided on the front surface 12B of the frame 12. The touch panel display 59 includes a display 46 and a touch panel 48. As an example of the display 46, an organic EL display can be cited. The display 46 may not be an organic EL display, but may be other types of displays such as a liquid crystal display or an inorganic EL display. In addition, the display 46 is an example of a "display unit (display)" involved in the technology of the present invention. And, the touch panel 48 is an example of a "receiving unit (receiving device (receiver))" involved in the technology of the present invention.

[0089] The display 46 displays images (e.g., live view images and reproduced images) and character information, etc. The touch panel 48 is a transmissive touch panel, and is superimposed on the surface of the display area of ​​the display 46. The touch panel 48 receives instructions from the user by detecting contact based on an indicator such as a finger or a stylus. In addition, here, as an example of the touch panel display 59, an external touch panel display in which the touch panel 48 is superimposed on the surface of the display area of ​​the display 46 is cited, but this is only an example. For example, as the touch panel display 59, an external or internal touch panel display can also be applied.

[0090] As an example, Figure 5 As shown, in the smart device 10, when an instruction to start photographing is received through the touch panel 48, the photographing area is photographed through the light receiver 18. That is, the light receiver 18 receives the visible reflected light, and generates a visible light image representing the photographing area as an image corresponding to the received visible reflected light. The visible light image is an example of the "photographing area image" involved in the technology of the present invention.

[0091] The visible light image is displayed on the display 46 as a live view image or a still image according to the instruction received through the touch panel 48. Figure 5 In the example shown, the imaging area is defined by a viewing angle θ1 . The viewing angle θ1 is changed in accordance with an instruction received via the touch panel 48 .

[0092] As an example, Figure 6 As shown, in the smart device 10, when an instruction to start distance measurement (hereinafter, also referred to as "distance measurement start instruction") is received through the touch panel 48, a laser beam is irradiated through the light irradiator 16. The angle of the irradiated laser beam (hereinafter, also referred to as "irradiation angle") is θ2, and the irradiation angle θ2 changes according to the instruction received through the touch panel 48. Figure 6In the example shown, the distance measurement is started according to the distance measurement start instruction received through the touch panel 48 in a state where the visible light image is displayed as a live view image on the display 46, but the technology of the present invention is not limited to this. For example, the distance measurement may be started when the distance measurement start instruction is received through the touch panel 48 in a state where the visible light image is not displayed on the display 46.

[0093] In the smart device 10, the distance measurement divided area 26N2 (reference Figure 3 ) to receive the IR reflected light and the speed of light, and measure the distance from the smart device 10 to the distance measurement object. For example, if the distance to the distance measurement object is set to "L", the speed of light is set to "c", and the time required from irradiation of the laser beam by the light irradiator 16 to reception of the IR reflected light by the distance measurement divided area 26N2 is set to "t", the distance L is calculated according to the formula "L = c × t × 0.5".

[0094] As an example, Figure 7 As shown, in the smart device 10, the IR reflected light is received by a plurality of IR pixels included in the distance measurement divided area 26N2, thereby performing distance measurement for each IR pixel. Then, the distance measurement result of each IR pixel is displayed as a distance image on the display 46. Here, the distance image refers to an image that represents the distance to the distance measurement object measured by each IR pixel in color and / or light and dark.

[0095] In the smart device 10, the distance measurement result is displayed on the display 46 in the form of a distance image or a distance superimposed image according to the instruction received through the touch panel 48. Figure 8 In the example shown, the distance overlay image displayed on the display 46 is obtained by overlaying a numerical value (in the example of FIG. Figure 8 In the example shown, the images are obtained by moving from the smart device 10 to a plurality of representative positions within the camera area (in Figure 8 In the example shown, the distance to each of three positions (three positions) is displayed in the visible light image and is displayed on the display 46. As an example of a representative plurality of positions, there can be cited a plurality of positions where the contrast difference between a specific subject (e.g., a subject and / or a person included in the central area of ​​the screen) in the imaging area is greater than a specified value.

[0096] As an example, Fig. 9As shown, the smart device 10 includes a controller 15, an input / output interface 40, an image memory 42, a UI device 44, an external I / F 52, and a communication I / F 54 in addition to a light irradiator 16 and a light receiver 18. The controller 15 is an example of a "processing device" and a "computer" involved in the technology of the present invention.

[0097] The controller 15 includes a CPU 15A, a storage device 15B, and a memory 15C. The CPU 15A is an example of a "processor", "recognition processor", and "determination processor" involved in the technology of the present invention, and the memory 15C is an example of a "memory" involved in the technology of the present invention. The CPU 15A, the storage device 15B, and the memory 15C are connected via a bus 50, and the bus 50 is connected to the input / output interface 40. Fig. 9 In the example shown, for the sake of convenience, one bus is shown as the bus 50, but a plurality of buses may be provided. The bus 50 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, and the like.

[0098] The storage device 15B stores various parameters and various programs. The storage device 15B is a non-volatile storage device. Here, as an example of the storage device 15B, a flash memory is used. The flash memory is just an example. As the storage device 15B, various non-volatile memories such as magnetoresistive memory and / or ferroelectric memory can be cited to replace the flash memory or be used together with the flash memory. In addition, the non-volatile storage device can also be an EEPROM, HDD and / or SSD, etc. In addition, the memory 15C temporarily stores various information and is used as a working memory. As an example of the memory 15C, a RAM can be cited, but it is not limited to this, and other types of storage devices can also be used.

[0099] The storage device 15B stores various programs. The CPU 15A reads out a desired program from the storage device 15B and executes the read program on the memory 15C. The CPU 15A controls the entire smart device 10 according to the program executed on the memory 15C.

[0100] Multiple devices are connected to the input / output interface 40, which is responsible for sending and receiving various information between the multiple devices. Fig. 9 In the illustrated example, as a plurality of devices connected to the input / output interface 40 , there are shown a controller 15 , a light irradiator 16 , a light receiver 18 , an image memory 42 , a UI device 44 , an external I / F 52 , and a communication I / F 54 .

[0101] The external I / F 52 is responsible for sending and receiving various information between a device (hereinafter, also referred to as an "external device") existing outside the smart device 10. As an example of the external I / F 52, a USB interface can be cited. The USB interface can directly or indirectly connect an external device such as a smart device, a personal computer, a server, a USB memory, a memory card, and / or a printer (not shown).

[0102] The communication I / F 54 has a communication function such as LTE, 5G, wireless LAN and / or Bluetooth (registered trademark), and is responsible for sending and receiving various information between the external device and the CPU 15A. For example, the communication I / F 54 is connected to the network 56 (for example, the Internet) via a base station (not shown) so as to be communicable, and is responsible for sending and receiving various information between the external device on the network 56 and the CPU 15A.

[0103] The UI device 44 includes a display 46, and the CPU 15A causes the display 46 to display various information. In addition, the UI device 44 includes a receiving device 47. The receiving device 47 includes a touch panel 48 and a hard key unit 53. The hard key unit 53 includes the indication key 13 (see Figure 2 ). The CPU 15A operates according to various instructions received through the touch panel 48. In addition, here, the hard key unit 53 is included in the UI device 44, but the technology of the present invention is not limited to this. For example, the hard key unit 53 can also be connected to the external I / F 52.

[0104] The light irradiator 16 includes a light transmission window 20, a beam expander 21, a collimating lens 23, an LD 24, and an LD driver 25. The light transmission window 20, the beam expander 21, and the collimating lens 23 are sequentially arranged along the optical axis L1 from the imaging area side (object side) to the LD 24. The LD driver 25 is connected to the LD 24 and the input / output interface 40, and drives the LD 24 according to the instruction of the CPU 15A to emit a laser beam from the LD 24.

[0105] The laser beam emitted from the LD 24 is converted into parallel light by the collimator lens 23 , and then the optical diameter is expanded by the beam expander 21 , and then the laser beam is irradiated toward the distance measurement object from the light transmission window 20 .

[0106] The photoreceiver 18 includes a light-transmitting window 22, an objective lens 30A, a focusing lens 30B, an aperture 30C, a photoelectric conversion element 26, a photoelectric conversion element driver 32, and a signal processing circuit 34. In the photoreceiver 18, along the optical axis L2, from the imaging area side (object side) to the photoelectric conversion element 26, the light-transmitting window 22, the objective lens 30A, the focusing lens 30B, and the aperture 30C are sequentially arranged. The photoelectric conversion element driver 32 is connected to the photoelectric conversion element 26 and the input / output interface 40, and drives the photoelectric conversion element 26 according to the instruction of the CPU 15A. For example, the photoelectric conversion element driver 32 supplies an imaging timing signal that specifies the timing of imaging performed by the photoelectric conversion element 26 to the photoelectric conversion element 26 under the control of the CPU 15A. The photoelectric conversion element 26 performs reset, exposure, and output of an electrical signal according to the imaging timing signal supplied from the photoelectric conversion element driver 32. As the imaging timing signal, for example, a vertical synchronization signal and a horizontal synchronization signal can be cited.

[0107] The photoreceiver 18 includes a focus control mechanism 31. The focus control mechanism 31 includes a focus lens 30B, a moving mechanism 60, a motor 62, and a motor driver 64. The focus lens 30B is supported by the moving mechanism 60 so as to be able to slide along the optical axis L2. The motor 62 is connected to the moving mechanism 60 and the motor driver 64. The motor driver 64 is connected to the input / output interface 40 and drives the motor 62 according to an instruction from the CPU 15A. The moving mechanism 60 is connected to a drive shaft (not shown) of the motor 62, and by receiving power from the motor 62, the focus lens 30B is selectively moved toward the object side and the image side along the optical axis L2. That is, the CPU 15A adjusts the focus position by controlling the drive of the motor 62 via the motor driver 64. Here, the "focus position" refers to the position of the focus lens 30B on the optical axis L2 in a focused state (for example, a state in which the contrast of the visible light image is set to a maximum value or a state in which a predetermined subject depth is achieved). Hereinafter, for convenience of explanation, the control for bringing the focus lens 30B to the in-focus position is also referred to as “focus control”.

[0108] The aperture 30C is a fixed aperture whose opening does not change. In the case of a fixed aperture, exposure adjustment is performed by an electronic shutter of the photoelectric conversion element 26. The aperture 30C may be a variable aperture instead of a fixed aperture. In addition, the objective lens 30A, the focusing lens 30B, and the aperture 30C included in the light receiver 18 are only examples, and the technology of the present invention is also applicable even if the structure of the lens and / or the position of the aperture 30C are changed.

[0109] The reflected light enters the light receiver 18 from the light transmission window 22. The reflected light entering the light transmission window 22 is focused on the photoelectric conversion element 26 via the objective lens 30A, the focusing lens 30B, and the aperture 30C.

[0110] The photoelectric conversion element 26 is connected to the signal processing circuit 34, and outputs pixel data indicating the pixel value of each of the visible light pixels and the IR pixels to the signal processing circuit 34. The signal processing circuit 34 digitizes the pixel data by performing A / D conversion on the pixel data input from the photoelectric conversion element 26, and performs various signal processing on the digitized pixel data.

[0111] The signal processing circuit 34 includes a visible light pixel data processing circuit 34A and a distance image generation circuit 34B. The visible light pixel data processing circuit 34A performs known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction on pixel data of visible light pixels, thereby generating a visible light image. Then, the visible light pixel data processing circuit 34A stores the visible light image in the image memory 42. In addition, by overwriting and storing the visible light image of one frame in the image memory 42, the visible light image in the image memory 42 is updated.

[0112] The ranging camera device 14 includes a TOF camera 19. The TOF camera 19 includes a light irradiator 16, a divided area 26N2 for ranging, and a distance image generation circuit 34B. The distance image generation circuit 34B obtains an emission timing signal indicating the timing of emitting a laser beam from the LD 24 (hereinafter, also referred to as "emission timing") from the CPU 15A. The distance image generation circuit 34B measures the distance from the smart device 10 to the ranging object for each IR pixel based on the emission timing indicated by the emission timing signal and the timing of receiving the IR reflected light by each IR pixel (hereinafter, also referred to as "light receiving timing"). Then, the distance image generation circuit 34B generates a distance image based on the measurement result of each IR pixel, and stores the generated distance image in the image memory 42. In addition, by overwriting and saving the distance image of one frame in the image memory 42, the distance image in the image memory 42 is updated.

[0113] As an example, Fig.10 As shown, the storage device 15B stores an imaging processing program 70, a distance measurement processing program 72, a parallel processing program 74, a change processing program 76, a display control processing program 78, an image recognition dictionary 80, and an irradiation energy table 82. In the following, for the sake of convenience, the imaging processing program 70, the distance measurement processing program 72, the parallel processing program 74, the change processing program 76, and the display control processing program 78 are referred to as "distance measurement imaging processing program" when they are described without distinguishing them.

[0114] According to the prior art, the IR reflected light is received by the IR pixel, and the distance from the smart device 10 to the distance measurement object is measured based on the emission timing and the light reception timing. It is ideal that the IR reflected light is completely cut off by the visible light filter, but it is difficult to completely cut off by the visible light filter depending on the intensity of the IR reflected light. At this time, the IR reflected light reaches the visible light pixel, and thus appears as noise in the visible light image, which may affect the image quality of the visible light image.

[0115] As a method for preventing IR reflected light from appearing as noise in the visible light image, the following method can be considered, namely, when the distance to the subject is less than a specified distance (for example, 0.5m), the ranging action is stopped during the video recording action (for example, during exposure by the photoelectric conversion element 26).

[0116] However, when the distance measurement result is used for focus control, if the distance measurement operation is always stopped during the video recording operation, the distance measurement result cannot be reflected in the focus control during the period when the distance measurement operation is stopped, so the accuracy of the focus control may be reduced compared with the case where the distance measurement operation is always performed. In particular, when a continuously moving subject is used as a distance measurement object, focus control may be performed based on a distance measurement result that does not conform to the current situation. In addition, the following situation can also be considered, that is, depending on the shooting scene, even if the distance to the subject is a specified distance, the IR reflected light will not appear as noise in the visible light image due to the reflectivity of the subject, but even in this case, if the distance to the subject is less than the specified distance, the distance measurement operation will be stopped.

[0117] Therefore, in the smart device 10, as an example, Fig.11 As shown in FIG. 1 , the CPU 15A reads the distance measurement imaging processing program from the storage device 15B, and executes the distance measurement imaging processing according to the read distance measurement imaging processing program. The distance measurement imaging processing includes the imaging processing described later (see FIG. 1 ). Fig.18 ), the distance measurement process described later (reference Fig.19 ), the parallel processing described later (refer to Fig.17 ), the following change processing (reference Fig. 20 ) and the display control processing described later (reference Fig.21A and Fig.21B ) processing.

[0118] The CPU 15A reads out the image pickup processing program 70 from the storage device 15B, and executes the image pickup processing according to the read-out image pickup processing program 70 (see FIG. 14 ). Fig.18 ). Then, the CPU 15A reads out the distance measurement processing program 72 from the storage device 15B, and executes the distance measurement processing according to the read out distance measurement processing program 72 (refer to Fig.19). Then, the CPU 15A reads the parallel processing program 74 from the storage device 15B, and executes parallel processing according to the read parallel processing program 74 (refer to Fig.17 ). Then, the CPU 15A reads the change processing program 76 from the storage device 15B, and executes the change processing according to the read change processing program 76 (refer to Fig. 20 ). Then, the CPU 15A reads out the display control processing program 78 from the storage device 15B, and executes the display control processing according to the read display control processing program 78 (refer to Fig.21A and Fig.21B ).

[0119] As an example, Fig.12 As shown, the distance measurement imaging process is realized by the CPU 15A operating as the control unit 90 , the recognition unit 92 , the acquisition unit 94 , the determination unit 96 , and the change unit 98 .

[0120] The control unit 90 performs parallel processing. That is, the control unit 90 controls the ranging camera device 14 to perform the imaging operation and the ranging operation in parallel. In addition, here, an example of a method of performing the imaging operation and the ranging operation in parallel is given, but this is only an example, and the technology of the present invention is also established even if the imaging operation and the ranging operation are not performed in parallel. For example, even if a portion of the period of performing the imaging operation and the period of performing the ranging operation overlap, the technology of the present invention is also established.

[0121] Furthermore, the control unit 90 acquires the visible light image and the distance image from the image memory 42, and displays an image based on the acquired visible light image and the distance image on the display 46. Specifically, the display 46 can switchably display the visible light image, the distance image, and the distance superimposed image under the control of the control unit 90.

[0122] The recognition unit 92 acquires a visible light image from the image memory 42 and performs image recognition on the acquired visible light image. The recognition unit 92 acquires an image recognition dictionary 80 from the storage device 15B and recognizes an image included in the visible light image with reference to the acquired image recognition dictionary 80. The image recognition dictionary 80 is information that establishes a correspondence between a plurality of high reflectivity object images and identifiers (hereinafter, also referred to as "identifiers") that can identify each of the plurality of high reflectivity object images. A high reflectivity object image refers to an image representing an object having a reflectivity greater than a reference reflectivity (hereinafter, also referred to as "high reflectivity object"). As the reference reflectivity, for example, the following reflectivity can be cited, that is, the reflectivity that appears as noise in the visible light image as IR reflected light based on the laser beam irradiated on the object passes through a visible light filter and reaches a visible light pixel, and the reflectivity is derived in advance through sensory testing and / or computer simulation. As an object having a reflectivity greater than the reference reflectivity, an object predetermined as a glossy object and an object predetermined as a white object can be cited. The objects predetermined as the glossy objects are, for example, mirror plates and glossy spheres, etc. Also, the objects predetermined as the white objects are, for example, white clothes, white balloons, white hair, etc. In addition, the mirror plate is an example of the "mirror-like object" involved in the technology of the present invention.

[0123] Here, a plurality of high reflectivity object images are exemplified as the information contained in the image recognition dictionary 80, but the technology of the present invention is not limited thereto, and the characteristic quantities (e.g., spatial frequency and / or brightness, etc.) of each of the plurality of high reflectivity object images may also be used. Furthermore, the image recognition dictionary 80 is information based on the learning result obtained by machine learning the correspondence between the image and the subject recognition result. In addition, here, an example of using the information based on the learning result obtained by machine learning as the image recognition dictionary 80 is given, but this example is only an example, and information derived in advance by sensory testing and / or computer simulation, etc. may be used as the image recognition dictionary 80.

[0124] The acquisition unit 94 acquires reflectivity information capable of determining the reflectivity within the imaging area. Here, the image recognition result is acquired by the acquisition unit 94 as the reflectivity information. The image recognition result refers to the result obtained by the recognition unit 92 performing image recognition on the visible light image. Specifically, when a high-reflectivity object image is identified by the recognition unit 92, the image recognition result includes an identifier capable of determining the identified high-reflectivity object image, and when a high-reflectivity object image is not identified by the recognition unit 92, the image recognition result includes information indicating that a high-reflectivity object image is not identified. In addition, here, the "image recognition result" is an example of the "reflectivity information" and "information based on the imaging area image" involved in the technology of the present invention.

[0125] The determination unit 96 determines whether the visible light image contains a high reflectivity object image by referring to the image recognition result obtained by the acquisition unit 94. When the determination unit 96 determines that the visible light image contains a high reflectivity object image, the change unit 98 changes the irradiation energy of the laser beam (hereinafter also referred to as "irradiation energy") according to the image recognition result obtained by the acquisition unit 94 while the ranging camera 14 performs the imaging operation and the ranging operation.

[0126] Specifically, the changing unit 98 acquires the irradiation energy table 82 from the storage device 15B, and changes the intensity of the laser beam emitted from the LD 24 via the LD driver 25 according to the acquired irradiation energy table 82, thereby changing the irradiation energy. That is, in a state where the control unit 90 performs parallel processing and the imaging operation and the distance measurement operation are performed in parallel, the LD driver 25 emits a laser beam of the intensity changed by the changing unit 98 from the LD 24.

[0127] In the irradiation energy table 82, the identifier is associated with the intensity information. The intensity information refers to information indicating the intensity of the laser beam emitted by the LD 24. The intensity of the laser beam indicated by the intensity information is an intensity weaker than the reference intensity. Here, the reference intensity is an example of the "first to fourth reference irradiation energies" involved in the technology of the present invention.

[0128] The reference intensity is, for example, an intensity set as a default value as the intensity of the laser beam emitted by the LD 24. The intensity set as a default value is, for example, an intensity derived in advance through a sensory test and / or a computer simulation, etc., as the intensity of the laser beam received by the IR pixel with an amount of light capable of measuring distance based on IR reflected light of the laser beam irradiated on the subject (for example, a person's face which is usually predetermined as a person's face).

[0129] The intensity information is set such that the higher the reflectivity of the high-reflectivity object represented by the high-reflectivity object image determined based on the identifier, the weaker the intensity of the laser beam. Fig.12 In the example shown, in the irradiation energy table 82, a mirror plate, a glossy sphere, and a white coat are shown as high reflectivity object images determined by the identifier. 11 The intensity information of the glossy sphere is represented by the intensity X 12 The strength information of the white clothes is represented by the strength X 13 The intensity information of 11 , Strength X 12 and strength X 13 Between, establish "intensity X 11 <Intensity X 12 <Intensity X 13 "The size relationship.

[0130] Under the control of the control unit 90, when the visible light image, the distance image, and the distance overlap image displayed on the display 46 are switched, as an example, Fig.13 As shown, the control unit 90 causes the display 46 to display a display mode selection screen 100 according to the instruction received through the touch panel 48. The display mode selection screen 100 is a screen used by the user or the like when the user or the like selects any one of the visible light image display mode, the distance image display mode, and the distance overlap image display mode. The visible light image display mode is an operation mode in which the control unit 90 causes the display 46 to display a visible light image. The distance image display mode is an operation mode in which the control unit 90 causes the display 46 to display a distance image. The distance overlap image display mode is an operation mode in which the control unit 90 causes the display 46 to display a distance overlap image.

[0131] Soft keys 100A, 100B, and 100C are displayed on the display mode selection screen 100. When the user or the like selects the visible light image display mode, the user or the like turns on the soft key 100A via the touch panel 48. When the user or the like selects the distance image display mode, the user or the like turns on the soft key 100B via the touch panel 48. When the user or the like selects the distance overlap image display mode, the user or the like turns on the soft key 100C via the touch panel 48. In addition, for the sake of convenience, the following description will be referred to as "display mode" without distinguishing between the visible light image display mode, the distance image display mode, and the distance overlap image display mode.

[0132] If the display mode is selected by operating the display mode selection screen 100, as an example, Fig.14 As shown, the control unit 90 causes the display 46 to display a distance measurement range designation guidance screen 102. The distance measurement range designation guidance screen 102 is a screen for guiding the user to designate a distance measurement range. The distance measurement range designation guidance screen 102 displays a message (hereinafter also referred to as a "guidance message") asking the user whether to designate a distance measurement range. Fig.14 In the example shown, as an example of the guidance message, a message "Do you want to specify the distance measurement range?" is shown. In addition, soft keys 102A and 102B are displayed on the distance measurement range specification guidance screen 102. When the user specifies the distance measurement range, the user turns on the soft key 102A via the touch panel 48. When the user does not specify the distance measurement range, that is, when the entire imaging area is used as the distance measurement range, the user turns on the soft key 102B via the touch panel 48.

[0133] When the distance measurement range designation guidance screen 102 is displayed on the display 46, the user or the like turns on the soft key 102A via the touch panel 48. Fig.15As shown, the control unit 90 causes the display 46 to display the visible light image as a live view image. When the live view image is displayed on the display 46, the user or the like specifies the image area (in Fig.15 In the example shown, a rectangular area surrounded by a dotted line on the visible light image. A real space area corresponding to an image area designated by a user or the like (hereinafter also referred to as a “designated image area”) is designated as a distance measurement target by the distance measurement camera 14.

[0134] At this time, as an example, Fig.16 As shown, the area position determination information (e.g., coordinates) capable of determining the position of the designated image area on the visible light image is output from the touch panel 48 to the control unit 90. The control unit 90 outputs the division area position information (e.g., pixel address) capable of determining the position of the designated division area 26N2a for distance measurement in the division area 26N2 for distance measurement to the photoelectric conversion element driver 32. Here, the designated division area 26N2a for distance measurement refers to a division area at a position corresponding to the position of the designated image area determined by the area position determination information input from the touch panel 48 in the division area 26N2 for distance measurement.

[0135] The photoelectric conversion element driver 32 drives only the designated divided area 26N2a for distance measurement in the divided area 26N2 for distance measurement during the distance measurement operation, thereby causing the distance measurement camera device 14 to perform distance measurement using the IR reflected light received only by the designated divided area 26N2a for distance measurement. Fig. 9 ) is used to measure the distance using IR reflected light received by at least one IR pixel included in a specified area among a plurality of IR pixels included in the array.

[0136] Next, refer to Figures 17 to 21B , the functions of the parts of the smart device 10 involved in the technology of the present invention are explained.

[0137] First, refer to Fig.17 The parallel processing executed by the CPU 15A will be described.

[0138] exist Fig.17 In the parallel processing shown, first, in step ST10, the control unit 90 determines whether the condition for starting the parallel processing (hereinafter, also referred to as the "parallel processing start condition") is satisfied. As an example of the parallel processing start condition, the condition that the touch panel 48 receives an instruction to start the parallel processing can be cited. In step ST10, if the parallel processing start condition is not satisfied, the determination is negative, and the determination of step ST10 is performed again. In step ST10, if the parallel processing start condition is satisfied, the determination is positive, and the parallel processing transitions to step ST12.

[0139] In step ST12 , the control unit 90 starts the imaging process and the distance measurement process, thereby causing the distance measurement imaging device 14 to start the imaging operation and the distance measurement operation, and then the parallel processing proceeds to step ST14 .

[0140] In step ST14, the control unit 90 determines whether the condition for terminating the parallel processing (hereinafter, also referred to as the "parallel processing termination condition") is satisfied. As an example of the parallel processing termination condition, the condition that the touch panel 48 receives an instruction to terminate the parallel processing can be cited. In step ST14, if the parallel processing termination condition is not satisfied, the determination is negative, and the determination of step ST14 is performed again. In step ST14, if the parallel processing termination condition is satisfied, the determination is positive, and the parallel processing transitions to step ST16.

[0141] In step ST16 , the control unit 90 ends the imaging process and the distance measurement process, thereby causing the distance measurement imaging device 14 to end the imaging operation and the distance measurement operation, and then the parallel processing ends.

[0142] Next, refer to Fig.18 The imaging process executed by the CPU 15A will be described.

[0143] exist Fig.18 In the video processing shown, first, in step ST20, the control unit 90 determines whether the timing for causing the ranging camera device 14 to perform video recording (hereinafter, also referred to as "video recording timing") has been reached. As an example of video recording timing, a timing set according to a period of a prescribed frame rate can be cited. For example, if the frame rate is 120fps, the period of the prescribed frame rate is 1 / 120 seconds. In step ST20, when the video recording timing has not been reached, the determination is negative, and the video recording processing transitions to step ST28. In step ST20, when the video recording timing has been reached, the determination is positive, and the video recording processing transitions to step ST22.

[0144] In step ST22 , the control unit 90 causes the photoelectric conversion element driver 32 to output an imaging timing signal to reset the photoelectric conversion element 26 and expose the photoelectric conversion element 26 , and then the imaging process proceeds to step ST24 .

[0145] In step ST24 , the control unit 90 causes the visible light pixel data processing circuit 34A to perform various signal processing. That is, the visible light pixel data processing circuit 34A generates a visible light image by performing various signal processing on pixel data about visible light pixels included in the photoelectric conversion element 26 .

[0146] In step ST26 , the control unit 90 causes the visible light pixel data processing circuit 34A to store the visible light image in the image memory 42 , and then the imaging process transitions to step ST28 .

[0147] In step ST28, the control unit 90 determines whether a condition for ending the image capture process (hereinafter also referred to as "image capture process ending condition") is satisfied. An example of an image capture process ending condition is: Fig.17 In step ST28, if the imaging process end condition is not satisfied, the determination is negative, and the imaging process transitions to step ST20. In step ST28, if the imaging process end condition is satisfied, the determination is positive, and the imaging process ends.

[0148] Next, refer to Fig.19 The distance measurement process executed by the CPU 15A will be described.

[0149] exist Fig.19 In the distance measurement process shown, first, in step ST40, the control unit 90 determines whether the timing for causing the distance measurement camera device 14 to perform distance measurement (hereinafter, also referred to as "distance measurement timing") has arrived. The distance measurement timing is, for example, a timing that arrives in a cycle shorter than the imaging timing. In step ST40, if the distance measurement timing has not been reached, the determination is negative, and the distance measurement process transitions to step ST56. In step ST40, if the distance measurement timing has been reached, the determination is positive, and the distance measurement process transitions to step ST42.

[0150] In step ST42, the control unit 90 causes the light irradiator 16 to irradiate the laser beam. That is, the control unit 90 controls the LD driver 25 to cause the LD 24 to emit the laser beam, and then the distance measurement process proceeds to step ST44.

[0151] In step ST44, the control unit 90 determines whether the IR reflected light is received through the designated divided area 26N2a for distance measurement. In step ST44, if the IR reflected light is not received through the designated divided area 26N2a for distance measurement, the determination is negative, and the determination of step ST44 is performed again. In step ST44, if the IR reflected light is received through the designated divided area 26N2a for distance measurement, the determination is positive, and the distance measurement process transitions to step ST46.

[0152] In step ST46 , the control unit 90 causes the distance image generation circuit 34B to generate a distance image based on the distance measurement result, and then the distance measurement process proceeds to step ST48 .

[0153] In step ST48 , the control unit 90 causes the distance image generation circuit 34B to store the distance image generated in step ST46 in the image memory 42 , and then the distance measurement process proceeds to step ST50 .

[0154] The distance measurement operation performed by the distance measurement camera 14 is a distance measurement operation for focusing, and the distance measurement camera 14 focuses on the imaging area based on the distance measurement result obtained by the distance measurement operation for focusing. Therefore, in the distance measurement process, the control unit 90 executes the processes of step ST50 to step ST54.

[0155] In step ST50, the control unit 90 acquires the distance image from the image memory 42, and derives the distance from the smart device 10 to the specific area in the camera area based on the acquired distance image, and then the distance measurement process transitions to step ST52. Here, the specific area refers to, for example, an area on the side of the object to be focused, and is an area specified by the user or the like via the touch panel 48. In addition, the specific area is not limited to this, and for example, it can be a person's face determined by running a so-called face detection function, or it can be a plurality of representative positions (see Figure 8 ) in a location.

[0156] In step ST52, the control unit 90 derives the focus position based on the distance derived in step ST50, and then the distance measurement process transitions to step ST54. The focus position is derived by the control unit 90, for example, from a focus position derivation table (not shown) that associates the distance with the focus position, or from a focus position derivation equation (not shown) that uses the distance as an independent variable and the focus position as a dependent variable.

[0157] In step ST54 , the control unit 90 operates the focus control mechanism 31 to move the focus lens 30B to the focus position derived in step ST52 , and then the distance measurement process proceeds to step ST56 .

[0158] In step ST56, the control unit 90 determines whether a condition for terminating the distance measurement process (hereinafter also referred to as a "distance measurement process termination condition") is satisfied. An example of a distance measurement process termination condition is: Fig.17 In step ST56, when the distance measurement process end condition is not satisfied, the determination is negative, and the distance measurement process transitions to step ST40. In step ST56, when the distance measurement process end condition is satisfied, the determination is positive, and the distance measurement process ends.

[0159] Next, refer to Fig. 20 The change process executed by the CPU 15A while the parallel processing is being executed by the control unit 90 will be described.

[0160] exist Fig. 20In the change process shown, first, in step ST70, the recognition unit 92 determines whether the visible light image in the image memory 42 is updated. In step ST70, if the visible light image in the image memory 42 is not updated, the determination is negative, and the change process transitions to step ST82. In step ST70, if the visible light image in the image memory 42 is updated, the determination is positive, and the change process transitions to step ST71.

[0161] In step ST71 , the recognition unit 92 acquires the visible light image from the image memory 42 , and then the change process transitions to step ST72 .

[0162] In step ST72 , the recognition unit 92 performs image recognition on the visible light image acquired in step ST70 with reference to the image recognition dictionary 80 , and then the change process proceeds to step ST74 .

[0163] In step ST74 , the acquisition unit 94 acquires the image recognition result obtained by the recognition unit 92 performing the image recognition in step ST72 , and then the change process proceeds to step ST76 .

[0164] In step ST76, the determination unit 96 determines whether the visible light image acquired in step ST70 includes a high reflectivity object image based on the image recognition result acquired in step ST74. In step ST76, if the visible light image acquired in step ST70 does not include a high reflectivity object image, the determination is negative, and the change process transitions to step ST82. In step ST76, if the visible light image acquired in step ST70 includes a high reflectivity object image, the determination is positive, and the change process transitions to step ST78.

[0165] In step ST78 , the changing unit 98 acquires the intensity information corresponding to the identifier included in the image recognition result acquired in step ST74 from the irradiation energy table 82 , and then the changing process proceeds to step ST80 .

[0166] In step ST80 , the changing unit 98 changes the intensity of the laser beam emitted from the LD 24 via the LD driver 25 to the intensity indicated by the intensity information acquired in step ST78 , and then the changing process proceeds to step ST82 .

[0167] In step ST82, the control unit 90 determines whether a condition for ending the change process (hereinafter also referred to as a "change process ending condition") is satisfied. An example of a change process ending condition is: Fig.17In step ST82, if the change processing end condition is not satisfied, the determination is negative, and the change processing transitions to step ST70. In step ST82, if the change processing end condition is satisfied, the determination is positive, and the change processing ends.

[0168] Next, refer to Fig.21A and Fig.21B The display control process executed by the CPU 15A will be described.

[0169] exist Fig.21A In the display control process shown, first, in step ST100, the control unit 90 determines whether the visible light image display mode is set. In step ST100, if the visible light image display mode is set, the determination is affirmative, and the display control process transitions to step ST102. In step ST100, if the visible light image display mode is not set, the determination is negative, and the display control process transitions to step ST106.

[0170] In step ST102 , the control unit 90 acquires the latest visible light image from the image memory 42 , and then the display control process transitions to step ST104 .

[0171] In step ST104 , the control unit 90 causes the display 46 to display the latest visible light image acquired in step ST102 , and then the display control process transitions to step ST122 .

[0172] Furthermore, by repeatedly executing the processing of step ST100 to step ST104 , the live view image obtained by the ranging imaging device 14 performing the imaging operation for a live view image of imaging the imaging area for the live view image as the imaging operation is displayed on the display 46 .

[0173] In step ST106, the control unit 90 determines whether the distance image display mode is set. In step ST106, if the distance image display mode is set, the determination is affirmative, and the display control process transitions to step ST108. In step ST106, if the distance image display mode is not set, the determination is negative, and the display control process transitions to step ST108. Fig.21B The transition is shown in step ST112.

[0174] In step ST108 , the control unit 90 acquires the latest range image from the image memory 42 , and then the display control process proceeds to step ST110 .

[0175] In step ST110 , the control unit 90 causes the display 46 to display the latest distance image acquired in step ST108 , and then the display control process proceeds to step ST122 .

[0176] exist Fig.21B In step ST112 shown in FIG. 1 , the control unit 90 acquires the latest range image from the image memory 42 , and then the display control process proceeds to step ST114 .

[0177] In step ST114, the control unit 90 derives the distance from the smart device 10 to a plurality of representative positions (reference Figure 8 ), after which the display control process transitions to step ST116.

[0178] In step ST116 , the control unit 90 acquires the latest visible light image from the image memory 42 , and then the display control process transitions to step ST118 .

[0179] In step ST118, the control unit 90 generates a distance superimposed image by superimposing the distance derived in step ST114 on the latest visible light image acquired in step ST116, and then the display control process transitions to step ST120. In addition, the position where the distance derived in step ST114 is superimposed on the latest visible light image acquired in step ST116 is a position corresponding to the representative plurality of positions.

[0180] In step ST120, the control unit 90 causes the display 46 to display the distance superimposed image generated in step ST118, and then the display control processing is turned to Fig.21A The transition to step ST122 is shown.

[0181] In step ST122, the control unit 90 determines whether a condition for ending the display control process (hereinafter, also referred to as a "display control process ending condition") is satisfied. As an example of a display control process ending condition, a condition in which an instruction to end the display control process is received through the touch panel 48 can be cited. In step ST122, if the display control process ending condition is not satisfied, the determination is negative, and the display control process transitions to step ST100. In step ST122, if the display control process ending condition is satisfied, the determination is positive, and the display control process ends.

[0182] As described above, in the smart device 10, the control unit 90 controls the ranging camera 14 to perform the imaging operation and the ranging operation. In addition, the acquisition unit 94 acquires the reflectivity information (in Fig.12 In the example shown, it is "image recognition result"). Then, while the camera action and the distance measurement action are being performed, the intensity of the laser beam is changed by the changing unit 98 according to the reflectivity information acquired by the acquiring unit 94. Therefore, according to this structure, compared with the case where the intensity of the laser beam is determined regardless of the reflectivity in the camera area, the influence of the laser beam on the visible light image can be reduced.

[0183] Furthermore, in the smart device 10, the changing unit 98 changes the image based on the information of the visible light image (in Fig.12 In the example shown, the intensity of the laser beam is changed for the "image recognition result". Therefore, according to this structure, compared with the case where the intensity of the laser beam is changed only according to the distance from the imaging area or only according to the shutter speed, the influence of the IR reflected light on the visible light image can be reduced with high precision.

[0184] Furthermore, in the smart device 10, the intensity of the laser beam is changed according to the image recognition result by the changing unit 98. Therefore, according to this configuration, compared with the case where the intensity of the laser beam is changed only according to the distance from the imaging area or only according to the shutter speed, the influence of the IR reflected light on the visible light image can be reduced with high accuracy.

[0185] Furthermore, in the smart device 10, the image recognition result is obtained by the acquisition unit 94, and the image recognition result is obtained by the recognition unit 92 performing image recognition on the visible light image with reference to the image recognition dictionary 80. The image recognition dictionary 80 is information based on the learning result obtained by machine learning the correspondence between the image and the subject recognition result. Therefore, according to this structure, compared with the case where it is determined whether the IR reflected light has an impact on the visible light image only based on the result of visual recognition of the camera area by the user, etc., the impact of the IR reflected light on the visible light image can be reduced with high accuracy.

[0186] Furthermore, in the smart device 10, when the image recognition result acquired by the acquisition unit 94 is an image recognition result indicating that the image includes an object with high reflectivity, the intensity of the laser beam is made weaker than the reference intensity by the change unit 98. Therefore, according to this configuration, the influence of IR reflected light from a high reflectivity object on the visible light image can be reduced compared to a case where the intensity of the laser beam is always greater than the reference intensity regardless of whether the imaging area includes an object with high reflectivity.

[0187] Furthermore, in the smart device 10, when the image recognition result acquired by the acquisition unit 94 is an image recognition result indicating that a glossy object is included, the intensity of the laser beam is made weaker than the reference intensity by the change unit 98. Therefore, according to this configuration, the influence of IR reflected light from a high reflectivity object on the visible light image can be reduced compared to a case where the intensity of the laser beam is always greater than the reference intensity regardless of whether a glossy object is included in the imaging area.

[0188] Furthermore, in the smart device 10, when the image recognition result acquired by the acquisition unit 94 is an image recognition result indicating that a mirror-like object (e.g., a mirror plate) is included, the intensity of the laser beam is made weaker than the reference intensity by the change unit 98. Therefore, according to this configuration, the influence of IR reflected light from a high reflectivity object on a visible light image can be reduced compared to a case where the intensity of the laser beam is always equal to or higher than the reference intensity regardless of whether a mirror-like object is included in the imaging area.

[0189] Furthermore, in the smart device 10, the distance measurement imaging device 14 includes the TOF camera 19. Therefore, according to this configuration, it is possible to output the distance measurement result as a distance image.

[0190] Furthermore, in the smart device 10, the distance image is displayed on the display 46. Therefore, according to this configuration, it is possible to allow the user or the like to visually recognize the distance image.

[0191] In addition, in the smart device 10, the distance measurement-only designated divided area 26N2a (reference Fig.16 Therefore, according to this configuration, the load of processing required for distance measurement can be reduced compared to the case where distance measurement is performed using IR reflected light received by all IR pixels included in the distance measurement divided area 26N2.

[0192] Then, in the smart device 10, the image area (in the state where the live view image is displayed) specified by only the image area specified by Fig.15 In the example shown, the distance measurement is performed using the IR reflected light received by the IR pixel at the position corresponding to the position of the "designated image area"). Therefore, according to this structure, the distance measurement can be performed using the IR reflected light received by only the IR pixel desired by the user among all the IR pixels included in the divided area 26N2 for distance measurement.

[0193] Furthermore, in the smart device 10, a live view image capturing operation for capturing the image capturing area for live view image is performed as an image capturing operation by the ranging image capturing device 14. Therefore, according to this configuration, the influence of the laser beam on the live view image can be reduced compared to the case where the intensity of the laser beam is determined regardless of the reflectivity in the image capturing area.

[0194] In addition, in the smart device 10, the ranging camera 14 performs a ranging operation for focusing as a ranging operation. That is, the ranging result is used for focus control. Therefore, according to this configuration, the ranging object can be focused with high accuracy compared to a case where the ranging result is not used for focus control.

[0195] In the smart device 10, the laser beam is used as directional light to measure the distance. Therefore, according to this configuration, the distance to a distant distance measuring object can be measured with high accuracy compared to the case of measuring the distance without using directional light.

[0196] In addition, in the above-mentioned embodiment, the changing unit 98 derives the intensity information corresponding to the identifier included in the image recognition result from the irradiation energy table 82, and changes the intensity of the laser beam according to the derived intensity information. However, the technology of the present invention is not limited to this. Fig. 22 As shown, the changing unit 98 may derive intensity information corresponding to the identifier and the imaging scene included in the image recognition result from the irradiation energy table 182 and change the intensity of the laser beam based on the derived intensity information.

[0197] exist Fig. 22 In the example shown, the irradiation energy table 182 is information in which the imaging scene and the intensity information are associated with each identifier. Fig. 22 In the example shown, night scenes and sunsets are cited as examples of shooting scenes. Among the multiple intensity information included in the irradiation energy table 182, the intensity represented by the intensity information corresponding to the specific shooting scene having a shooting scene reflectivity greater than the reference reflectivity described in the above embodiment is an intensity weaker than the reference intensity described in the above embodiment. The shooting scene reflectivity refers to a reflectivity predetermined as the reflectivity of the laser beam irradiated to the shooting scene. The shooting scene reflectivity is a reflectivity derived in advance by sensory testing and / or computer simulation.

[0198] The touch panel 48 receives the shooting scene indication information indicating the shooting scene. The changing unit 98 derives the intensity information corresponding to the shooting scene indicated by the shooting scene indication information received by the touch panel 48 and the identifier included in the image recognition result from the irradiation energy table 182. Then, the changing unit 98 changes the intensity of the laser beam according to the intensity information derived from the irradiation energy table 182.

[0199] When receiving shooting scene instruction information via the touch panel 48, as an example, Fig.23 As shown, the display 46 displays a shooting scene selection screen 104 under the control of the control unit 90. The shooting scene selection screen 104 is a screen used by the user or the like to select any one of a plurality of shooting scenes. The shooting scene selection screen 104 displays a message (in the Fig.23In the example shown, a message “Please select a shooting scene.” is displayed. Furthermore, a plurality of soft keys such as soft keys 104A, 104B, 104C, and 104D (hereinafter also referred to as “soft keys in the shooting scene selection screen 104”) are displayed on the shooting scene selection screen 104. When the user or the like selects a night scene as the shooting scene, the user or the like turns on the soft key 104A via the touch panel 48. When the user or the like selects a sunset scene as the shooting scene, the user or the like turns on the soft key 104B via the touch panel 48. When the user or the like selects a landscape scene as the shooting scene, the user or the like turns on the soft key 104C via the touch panel 48. When the user or the like selects a portrait scene as the shooting scene, the user or the like turns on the soft key 104D via the touch panel 48. In this way, by turning on the soft keys in the shooting scene selection screen 104 by the user or the like, the shooting scene indication information is received via the touch panel 48.

[0200] In the smart device 10 configured as above, as an example, the CPU 15A executes the following Fig.24 Change processing shown. Fig.24 The changes shown deal with Fig. 20 Compared with the changed processing shown in the figure, the difference lies in that the processing of step ST78 is replaced by the processing of step ST178 and the processing of step ST77 is provided between the processing of step ST76 and the processing of step ST178.

[0201] In step ST77 , the changing unit 98 acquires the shooting scene instruction information received via the touch panel 48 , and then the changing process proceeds to step ST178 .

[0202] In step ST178 , the changing unit 98 obtains intensity information corresponding to the identifier included in the image recognition result obtained in step ST74 and the shooting scene instruction information obtained in step ST77 from the irradiation energy table 182 , and then the changing process proceeds to step ST80 .

[0203] So, in Figure 22 to Figure 24 In the example shown, the intensity of the laser beam is changed by the changing unit 98 according to the shooting scene indication information received through the touch panel 48 and the identifier included in the image recognition result. Therefore, according to this configuration, the influence of the IR reflected light on the visible light image can be reduced compared to the case where the intensity of the laser beam is determined regardless of the shooting scene.

[0204] And, in Figure 22 to Figure 24In the example shown, when the shooting scene indicated by the shooting scene indication information received through the touch panel 48 is a specific shooting scene having a shooting scene reflectivity greater than the reference reflectivity, the intensity of the laser beam is made weaker than the reference intensity by the changing unit 98. Therefore, according to this configuration, the influence of IR reflected light on the visible light image in the specific shooting scene can be reduced, compared with a case where the intensity of the laser beam is always greater than the reference intensity even when the designated shooting scene is a specific shooting scene having a shooting scene reflectivity greater than the reference reflectivity.

[0205] exist Figure 22 to Figure 24 In the example shown, the changing unit 98 is used to obtain the intensity information corresponding to the shooting scene indication information received by the touch panel 48, but the technology of the present invention is not limited to this. For example, the shooting scene can also be determined by image recognition of the visible light image by the recognition unit 92.

[0206] At this time, as an example, Fig.25 As shown, the recognition unit 92 acquires a visible light image from the image memory 42, and performs image recognition on the acquired visible light image, thereby determining the shooting scene of the camera area represented by the visible light image. Specifically, the recognition unit 92 determines the shooting scene with reference to the image recognition dictionary 80. The image recognition dictionary 80 contains information that establishes a corresponding association between a shooting scene image representing a shooting scene and scene determination information that can determine the shooting scene represented by the shooting scene image. The recognition unit 92 performs image recognition on the visible light image acquired from the image memory 42 with reference to the image recognition dictionary 80. That is, the recognition unit 92 determines the shooting scene image corresponding to the visible light image from the image recognition dictionary 80, and acquires the scene determination information corresponding to the determined shooting scene image from the image recognition dictionary 80.

[0207] The changing unit 98 derives from the irradiation energy table 182 the intensity information corresponding to the shooting scene determined based on the scene determination information acquired by the recognition unit 92 and the identifier included in the image recognition result. Then, the changing unit 98 changes the intensity of the laser beam to the intensity indicated by the intensity information derived from the irradiation energy table 182. Fig.25 In the example shown, the recognition unit 92 is an example of a “determining unit” according to the technique of the present invention.

[0208] In the smart device 10 configured as above, as an example, the CPU 15A executes the following Fig.26 Change processing shown. Fig.26 The changes shown deal with Fig.24Compared with the changed processing shown, the differences are that the processing of step ST77 is not included, the processing of step ST275 is included between the processing of step ST74 and the processing of step ST76, and the processing of step ST278 is included instead of the processing of step ST178.

[0209] In step ST275 , the recognition unit 92 identifies the captured scene image corresponding to the visible light image from the image recognition dictionary 80 , and acquires scene identification information corresponding to the identified captured scene image from the image recognition dictionary 80 .

[0210] In step ST278 , the changing unit 98 obtains from the irradiation energy table 182 the intensity information corresponding to the identifier included in the image recognition result obtained in step ST74 and the shooting scene determined by the scene determination information obtained in step ST275 , and then the change processing transitions to step ST80 .

[0211] So, in Fig.25 and Fig.26 In the example shown, the intensity of the laser beam is changed by the changing unit 98 according to the photographing scene determined by the recognition unit 92 and the identifier included in the image recognition result. Therefore, according to this configuration, the influence of the IR reflected light on the visible light image can be reduced compared to the case where the intensity of the laser beam is determined regardless of the photographing scene.

[0212] And, with Figure 22 to Figure 24 The same example shown, Fig.25 and Fig.26 In the example shown, even when the shooting scene determined by the scene determination information acquired by the recognition unit 92 is a specific shooting scene having a shooting scene reflectivity greater than the reference reflectivity, the intensity of the laser beam is made weaker than the reference intensity by the change unit 98. Therefore, according to this configuration, compared with a case where the intensity of the laser beam is always greater than the reference intensity even when the shooting scene determined by the recognition unit 92 is a specific shooting scene having a shooting scene reflectivity greater than the reference reflectivity, the influence of IR reflected light on the visible light image in the specific shooting scene can be reduced.

[0213] Furthermore, in the above-mentioned embodiment, an example of irradiating a laser beam to a distance measurement object by the light irradiator 16 is given for explanation, but the technology of the present invention is not limited to this. For example, the distance measurement object may be irradiated with auxiliary light for imaging (for example, visible light used as a so-called red-eye prevention measure), and the reflected light of the auxiliary light reflected by the distance measurement object may be received by the light receiver 18 (hereinafter also referred to as "auxiliary reflected light"), thereby performing distance measurement.

[0214] At this time, as an example, Fig. 27As shown, a smart device 200 is used. The smart device 200 is different from the smart device 10 described in the above embodiment in that it has a distance-finding camera 214 instead of the distance-finding camera 14 and a light receiver 218 instead of the light receiver 18. The distance-finding camera 214 is different from the distance-finding camera 14 in that it has an auxiliary light irradiator 213.

[0215] The auxiliary light irradiator 213 includes an LED 202 and an LED driver 204. The LED driver 204 is connected to the LED 202 and the input / output interface 40. The LED driver 204 controls the LED 202 according to an instruction from the CPU 15A (for example, the control unit 90). The LED driver 204 causes the LED 202 to generate auxiliary light under the control of the CPU 15A, thereby irradiating the auxiliary light toward the imaging area. In addition, the LED 202 is an example of a "light source" involved in the technology of the present invention.

[0216] The light receiver 218 is different from the light receiver 18 in that it has a photoelectric conversion element 226 instead of the photoelectric conversion element 26. The photoelectric conversion element 226 is different in that it replaces the designated divided area 26N2a for distance measurement (see Fig.16 ) has a designated divided area 26N2b for distance measurement. The designated divided area 26N2b for distance measurement is different from the designated divided area 26N2a for distance measurement in that G pixels are used instead of IR pixels. The designated divided area 26N2b for distance measurement receives auxiliary reflected light under the control of the control unit 90. In addition, the designated divided area 26N2b for distance measurement is an example of a "light receiving element" involved in the technology of the present invention.

[0217] As an example, Fig.28 As shown, the CPU 15A realizes the ranging imaging process by operating as the control unit 90, the acquisition unit 94, and the change unit 98 according to the ranging imaging process program. The signal processing circuit 34 includes an A / D converter 34C. The A / D converter 34C is connected to the designated divided area 26N2b for ranging, and digitizes the light receiving amount of the auxiliary reflected light received by the designated divided area 26N2b for ranging.

[0218] The acquisition unit 94 acquires the light reception amount digitized by the A / D converter 34C, and acquires the light reception amount of the auxiliary reflected light received by the designated divided area 26N2b for distance measurement per unit time (hereinafter also referred to as “light reception amount per unit time”) by calculation.

[0219] The storage device 15B stores an irradiation energy table 282. The irradiation energy table 282 is different from the irradiation energy table 82 in that the amount of light received per unit time is used instead of the identifier.

[0220] The changing unit 98 acquires the intensity information corresponding to the received light amount per unit time acquired by the acquiring unit 94 from the irradiation energy table 282. Then, the changing unit 98 changes the intensity of the auxiliary light irradiated from the LED 202 via the LED driver 204 to the intensity indicated by the intensity information acquired from the irradiation energy table 282.

[0221] In the smart device 200 configured as above, as an example, the CPU 15A executes the following Fig.29 Change processing shown.

[0222] exist Fig.29 In the change process shown, first, in step ST200, the control unit 90 determines whether the condition for causing LED 202 to irradiate auxiliary light (hereinafter, also referred to as "auxiliary light irradiation condition") is satisfied. As an example of the auxiliary light irradiation condition, for example, a condition for reaching the timing of irradiating auxiliary light as a red-eye prevention measure can be cited. In step ST200, when the auxiliary light irradiation condition is not satisfied, the determination is negative, and the change process transitions to step ST210. In step ST200, when the auxiliary light irradiation condition is satisfied, the determination is positive, and the change process transitions to step ST202.

[0223] In step ST202 , the control unit 90 causes the LED 202 to emit auxiliary light, and then the change process proceeds to step ST204 .

[0224] In step ST204, the acquisition unit 94 obtains the amount of light received by the designated divided area 26N2b for distance measurement per unit time (hereinafter also referred to as "light received per unit time") through calculation, and then the change processing transitions to step ST206.

[0225] In addition, although the amount of light received per unit time is shown here, the technology of the present invention is not limited thereto, and for example, instead of the amount of light received per unit time, fine-tuning completion information obtained by fine-tuning the amount of light received per unit time (for example, fine-tuning completion information obtained by multiplying the amount of light received per unit time by a fine adjustment coefficient) may be applied. Here, the "fine-tuning completion information" is an example of "information based on the amount of light received per unit time" involved in the technology of the present invention.

[0226] In step ST206, the changing unit 98 obtains the intensity information from the irradiation energy table 282, and the changing process proceeds to step ST208. In step ST206, the changing unit 98 obtains the intensity information corresponding to the received light amount per unit time calculated in step ST204 from the irradiation energy table 282.

[0227] In step ST208 , the changing unit 98 changes the intensity of the auxiliary light emitted from the LED 202 via the LED driver 204 to the intensity indicated by the intensity information acquired in step ST206 , and then the changing process proceeds to step ST210 .

[0228] In step ST210, the control unit 90 determines whether the above-mentioned change processing end condition is satisfied. In step ST210, if the change processing end condition is not satisfied, the determination is negative, and the change processing transitions to step ST200. In step ST210, if the change processing end condition is satisfied, the determination is positive, and the change processing ends.

[0229] So, in Figure 27 to Figure 29 In the example shown, auxiliary light is irradiated from LED 202 to the distance measurement object, and auxiliary reflected light is received by the designated divided area 26N2b for distance measurement. Information based on the light reception result of the auxiliary reflected light is acquired by the acquisition unit 94. Then, the intensity of the auxiliary light is changed by the change unit 98 according to the information based on the light reception result acquired by the acquisition unit 94. Therefore, according to this structure, compared with the case where the intensity of the auxiliary light is determined regardless of the information based on the light reception result of the auxiliary reflected light, the influence of the auxiliary reflected light on the visible light image can be reduced.

[0230] And, in Figure 27 to Figure 29 In the example shown, the amount of light received per unit time is acquired by the acquisition unit 94. Then, the intensity of the auxiliary light is changed by the change unit 98 according to the amount of light received per unit time acquired by the acquisition unit 94. Therefore, according to this configuration, compared with the case where the intensity of the auxiliary light is determined regardless of the amount of light received per unit time, the influence of the auxiliary reflected light on the visible light image can be reduced.

[0231] And, in Figure 27 to Figure 29 In the example shown, the example of changing the intensity of the auxiliary light by the changing unit 98 according to the intensity information derived from the irradiation energy table 282 is described, but the technology of the present invention is not limited to this. Fig.30 As shown in FIG. 1 , the acquisition unit 94 may also acquire the intensity information by converting the reflectivity (hereinafter also referred to as “reflectivity”) in the imaging area from the amount of light received per unit time. As a conversion formula used when converting the reflectivity from the amount of light received per unit time, for example, a conversion formula that uses the amount of light received per unit time as an independent variable and the reflectivity in the imaging area as a dependent variable can be cited. Fig.30 In the example shown, the storage device 15B stores an irradiation energy table 382. The irradiation energy table 382 is Fig.28The difference from the irradiation energy table 282 shown in FIG. 3 is that the reflectivity is used instead of the amount of light received per unit time. The changing unit 98 obtains the intensity information corresponding to the reflectivity obtained by the obtaining unit 94 from the irradiation energy table 382. Then, the changing unit 98 changes to the intensity represented by the intensity information obtained from the irradiation energy table 382.

[0232] In the smart device 200 configured as above, as an example, the CPU 15A executes the following Fig.31 Change processing shown. Fig.31 The changes shown deal with Fig.29 Compared with the modified processing shown in the figure, the difference is that the processing of step ST300 and the processing of step ST302 are included instead of the processing of step ST206.

[0233] exist Fig.31 In the change process shown, in step ST300, the acquisition unit 94 calculates the reflectance based on the amount of light received per unit time acquired in step ST204. Specifically, the acquisition unit 94 converts the amount of light received per unit time into the reflectance using a conversion formula.

[0234] In the next step ST302 , the changing unit 98 acquires the intensity information corresponding to the reflectance acquired in step ST300 from the irradiation energy table 382 .

[0235] So, in Fig.30 and Fig.31 In the example shown, the reflectivity in the imaging area is acquired by the acquisition unit 94, and the intensity of the auxiliary light is changed to an intensity corresponding to the reflectivity in the imaging area by the change unit 98. Therefore, according to this configuration, compared with a case where the intensity of the auxiliary light is determined regardless of the reflectivity in the imaging area, the influence of the auxiliary reflected light on the visible light image can be reduced.

[0236] And, in Fig.30 and Fig.31 In the example shown, an example of changing the intensity of the auxiliary light by the changing unit 98 according to the intensity information obtained from the irradiation energy table 382 by the changing unit 98 is given, but the technology of the present invention is not limited to this. For example, the changing unit 98 may also make the intensity of the auxiliary light weaker than the reference intensity when the reflectivity obtained by the obtaining unit 94 is above the threshold value without using the irradiation energy table 382. Here, the threshold value is, for example, a value corresponding to the amount of light received when the auxiliary reflected light of the intensity that appears as noise in the visible light image as the auxiliary reflected light is irradiated into the imaging area, and the amount of light received is derived in advance by sensory testing and / or computer simulation.

[0237] In the smart device 200 configured as above, as an example, the CPU 15A executes the following Fig.32Change processing shown. Fig.32 The changes shown deal with Fig.31 Compared with the changed processing shown in the figure, the difference is that the processing of step ST302 and the processing of step ST208 are replaced by the processing of step ST400 and the processing of step ST402.

[0238] exist Fig.32 In the change process shown, in step ST400, the change unit 98 determines whether the reflectivity calculated in step ST300 is greater than the threshold value. In step ST400, when the reflectivity calculated in step ST300 is less than the threshold value, the determination is negative, and the change process transitions to step ST210. In step ST400, when the reflectivity calculated in step ST300 is greater than the threshold value, the determination is positive, and the change process transitions to step ST402.

[0239] In step ST402 , the changing unit 98 makes the intensity of the auxiliary light weaker than the reference intensity, and then the changing process proceeds to step ST210 .

[0240] In this way, when the reflectivity converted from the amount of light received per unit time is above the threshold, the intensity of the auxiliary light is reduced to be weaker than the reference intensity by the changing unit 98. Therefore, according to this configuration, compared with a case where the intensity of the auxiliary light is always above the reference intensity regardless of whether the reflectivity converted from the amount of light received per unit time is above the threshold, the influence of the auxiliary reflected light on the visible light image can be reduced.

[0241] And, in Figure 27 to Figure 32 In the example shown, the auxiliary light irradiated by LED 202 is illustrated, but the technology of the present invention is not limited to this. Figure 27 to Figure 32 In the example shown, the auxiliary light is replaced by a laser beam, and the technology of the present invention is also valid. In addition, for the sake of convenience, when there is no need to distinguish between auxiliary light and laser beam for explanation, it is called "distance measuring light", and when there is no need to distinguish between auxiliary reflected light and IR reflected light for explanation, it is called "reflected light".

[0242] Furthermore, in the above-mentioned embodiment, an example is given in which the intensity of the distance measuring light is changed by the changing unit 98 regardless of whether it is the timing of the main exposure, but the technology of the present invention is not limited to this. For example, the intensity of the distance measuring light may be changed by the changing unit 98 according to the timing of the main exposure performed in the imaging operation. The timing of the main exposure refers to the timing of exposure in the still image imaging, for example.

[0243] At this time, as an example, Fig.33As shown, the display 46 displays a live view image and a soft key 106A under the control of the control unit 90. The soft key 106A is turned on by the user or the like when the user or the like instructs to start still image shooting. When the soft key 106A is turned on by the user or the like in a state where live view image shooting is being performed by the ranging camera device 14, the intensity of the ranging light is changed by the changing unit 98 according to the timing of the main exposure. Thus, compared with the case where the intensity of the ranging light is changed at a timing unrelated to the main exposure, the ranging light can be irradiated to the imaging area at an appropriate intensity at the time of the main exposure.

[0244] Furthermore, in the above-mentioned embodiment, the smart device 10 equipped with the light receiver 18 is exemplified, but the technology of the present invention is not limited to this. Fig.34 As shown, even if the smart device 300 is equipped with the light receivers 18 and 350, the technology of the present invention is also applicable. Fig.34 As shown, a light-transmitting window 352 is provided adjacent to the light-transmitting window 22 at the upper left portion of the back side 12A of the frame body 12 when the smart device 10 is placed in a vertical position (the upper left portion when the smart device 10 in a vertical position is viewed from behind). The light-transmitting window 352 is the same as the light-transmitting windows 20 and 22, and is an optical element (e.g., a lens) having light transmittance. The light-transmitting windows 20, 22, and 352 are arranged at a predetermined interval in the horizontal direction. The light-transmitting window 352 is also the same as the light-transmitting windows 20 and 22, and is exposed from the back side 12A.

[0245] The photoreceiver 350 includes a photoelectric conversion element 354. The photoelectric conversion element 354 is a special photoelectric conversion element for receiving IR reflected light, and has a plurality of IR pixels arranged in a matrix. An example of a plurality of IR pixels is a photodiode for IR pixels (for example, InGaAs APD) with a pixel number of "4896×3265". The photoelectric conversion element 354 receives the IR reflected light received from the photoreceiver 350 via the light-transmitting window 352, and outputs an electrical signal corresponding to the light amount of the received IR reflected light to the signal processing circuit 34 (see Fig. 9 ).

[0246] And, in Fig.34 In the example shown, a smart device 300 equipped with light receivers 18 and 350 is shown, but the technology of the present invention is not limited to this. Fig.35 As shown, even if the smart device 400 is equipped with the light receiver 18 and 450, the technology of the present invention is also applicable. Fig.35As shown, a light-transmitting window 452 is provided adjacent to the light-transmitting window 20 at the upper left portion of the back side 12A of the frame body 12 when the smart device 10 is placed in a vertical position (the upper left portion when the smart device 10 in a vertical position is viewed from behind). The light-transmitting window 452 is a light-transmitting optical element (e.g., a lens) similar to the light-transmitting windows 20 and 22, and the light-transmitting windows 452, 20, and 22 are arranged at a predetermined interval in the horizontal direction. The light-transmitting window 452 is also similar to the light-transmitting windows 20 and 22, and is exposed from the back side 12A.

[0247] The light receiver 450 includes a single photodiode 454. The photodiode 454 is, for example, a photodiode capable of receiving IR reflected light. An example of the photodiode 454 is an InGaAs APD. The photodiode 454 receives the IR reflected light received from the light receiver 450 via the light-transmitting window 452, and outputs an electrical signal corresponding to the light amount of the received IR reflected light to the signal processing circuit 34 (see Fig. 9 ).

[0248] Furthermore, in the above-mentioned embodiment, an example of generating a distance image by the distance image generating circuit 34B is given for explanation, but the technology of the present invention is not limited to this. The distance measuring camera device 14 may also be used to perform an action of measuring the distance from the smart device 10 to the distance measuring object as a distance measuring action instead of generating a distance image.

[0249] Furthermore, in the above-mentioned embodiment, an example of a method of changing the intensity of the ranging light is given for explanation, but the technology of the present invention is not limited to this. The irradiation energy of the ranging light can also be changed by changing the emission time of the ranging light and / or the number of times the ranging light emits per unit time, instead of or together with the intensity of the ranging light.

[0250] Furthermore, in the above-mentioned embodiment, the distance measurement is performed by changing the distance measurement divided area 26N2 to the distance measurement designated divided area 26N2a, and narrowing down the distance measurement object (so-called ROI) designated by the user, etc., but the method of narrowing down the distance measurement object is not limited to this. For example, the control unit 90 may also control the change of the beam diameter and / or direction of the laser beam according to the instruction received by the receiving device 47 in such a way that the distance measurement object designated by the user is irradiated with a laser beam.

[0251] Furthermore, in the above-mentioned embodiment, a laser beam is cited as an example of "directional light" involved in the technology of the present invention, but the technology of the present invention is not limited to this. Superradiant light can also be used instead of the laser beam, as long as the distance measurement is performed using light with directionality that can measure the distance.

[0252] Furthermore, in the above-mentioned embodiment, the distance measuring camera device 14 is described as being built into the smart device 10, but the technology of the present invention is not limited thereto. Fig.36 As shown, the distance measuring camera 14 may be installed outside a common smart device 500 , that is, a smart device 500 that does not have the distance measuring camera 14 built therein.

[0253] Furthermore, in the above-mentioned embodiment, the UI device 44 is described as being assembled in the smart device 10, but at least a part of the multiple components included in the UI device 44 may be installed outside the smart device 10. Furthermore, at least a part of the multiple components included in the UI device 44 may be used as an independent individual by connecting to the external I / F 52.

[0254] And, in Figure 1 In the example shown, the smart device 10 is illustrated, but the technology of the present invention is not limited to this. That is, the technology of the present invention can also be applied to various electronic devices (for example, interchangeable lens cameras, fixed lens cameras, personal computers and / or wearable terminal devices, etc.) with built-in range-finding camera devices 14, and even these electronic devices can obtain the same functions and effects as the above-mentioned smart device 10.

[0255] Furthermore, in the above embodiment, the display 46 is exemplified, but the technology of the present invention is not limited thereto. For example, an independent display installed in the smart device 10 later may be used as the "display unit" involved in the technology of the present invention.

[0256] Furthermore, in the above-mentioned embodiment, an example in which the recognition unit 92 is mounted on the smart device 10 is given for explanation, but the technology of the present invention is not limited to this. For example, an external device (for example, another smart device, a personal computer and / or a server, etc.) having the recognition unit 92 may be connected to the smart device 10. In this case, the image recognition result provided from the external device to the smart device 10 may be obtained by the acquisition unit 94 of the smart device 10. Furthermore, cloud computing (not shown) may be made to assume the function of the recognition unit 92, and the image recognition result may be provided from the cloud computing to the smart device 10. In this case, the image recognition result provided from the cloud computing to the smart device 10 may be obtained by the acquisition unit 94 of the smart device 10.

[0257] Furthermore, in the above-mentioned embodiment, an example in which the ranging imaging processing program is stored in the storage device 15B is given for explanation, but the technology of the present invention is not limited to this. Fig.37 As shown, the ranging imaging processing program may be stored in the storage medium 900. As an example of the storage medium 900, any portable storage medium such as an SSD or a USB memory which is a non-temporary storage medium may be cited.

[0258] The distance measurement imaging processing program stored in the storage medium 900 is installed in the controller 15. The CPU 15A executes the distance measurement imaging processing according to the distance measurement imaging processing program.

[0259] Furthermore, the ranging camera processing program may be stored in a storage unit of another computer or server device connected to the controller 15 via a communication network (not shown), downloaded in response to a request from the smart device 10, and installed in the controller 15.

[0260] Furthermore, it is not necessary to store all of the distance measurement imaging processing program in the storage unit of another computer or server device connected to the controller 15 or the storage device 15B, and a part of the distance measurement imaging processing program may be stored.

[0261] exist Fig.37 In the example shown, the controller 15 is built into the smart device 10 , but the technology of the present invention is not limited to this. For example, the controller 15 may be provided outside the smart device 10 .

[0262] exist Fig.37 In the example shown, the CPU 15A is a single CPU, but it may be a plurality of CPUs. Furthermore, a GPU may be used instead of the CPU 15A.

[0263] exist Fig.37 In the example shown, the controller 15 is illustrated, but the technology of the present invention is not limited thereto, and a device including ASIC, FPGA and / or PLD may be applied instead of the controller 15. Furthermore, a combination of hardware structure and software structure may be used instead of the controller 15.

[0264] As hardware resources for executing the ranging camera processing described in the above embodiment, various processors shown below can be used. As a processor, for example, a general-purpose processor, i.e., a CPU, which functions as a hardware resource for executing the ranging camera processing by executing software, i.e., a program, can be cited. Also, as a processor, for example, a processor, i.e., a dedicated circuit, such as an FPGA, a PLD, or an ASIC, which has a circuit structure specially designed for executing a specific process can be cited. A memory is built into or connected to any processor, and any processor executes the ranging camera processing by using the memory.

[0265] The hardware resource for performing ranging camera processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resource for performing ranging camera processing may also be a single processor.

[0266] As an example of a processor, first, there is a method in which a processor is composed of a combination of one or more CPUs and software, and the processor functions as a hardware resource for executing ranging camera processing. Second, there is a method in which a processor that implements the entire system including multiple hardware resources for executing ranging camera processing is used by one IC chip, such as SoC. In this way, ranging camera processing is implemented as a hardware resource using one or more of the above-mentioned processors.

[0267] Moreover, as the hardware structure of these various processors, more specifically, a circuit combining circuit elements such as semiconductor elements can be used. In addition, the above-mentioned ranging camera processing is only an example. Therefore, it is of course possible to delete unnecessary steps, add new steps or replace the processing order within the scope of the main purpose.

[0268] The above-described contents and illustrated contents are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-described structures, functions, actions, and effects are descriptions related to an example of the structures, functions, actions, and effects of the parts involved in the technology of the present invention. Therefore, it is beyond doubt that the above-described contents and illustrated contents may delete unnecessary parts, add new elements, or replace them within the scope of the purpose of the technology of the present invention. Furthermore, in order to avoid complication and facilitate understanding of the parts involved in the technology of the present invention, the above-described contents and illustrated contents have been omitted. Descriptions related to technical common sense that does not need to be specifically explained in the implementation of the technology of the present invention are omitted.

[0269] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. In addition, in this specification, when "and / or" is used to express three or more items, the same concept as "A and / or B" also applies.

[0270] All documents, patent applications, and technical specifications described in this specification are incorporated into this specification by reference in the same manner as if each document, patent application, and technical specification was specifically and individually described as being incorporated by reference.

[0271] Regarding the above-mentioned embodiments, the following appendix is ​​also disclosed.

[0272] (appendix)

[0273] A processing device, comprising:

[0274] processor; and

[0275] Memory, connected to or built into the above processor,

[0276] The processor performs a process comprising the following steps:

[0277] Controlling an imaging operation and a distance measurement operation, wherein the imaging operation is performed by an imaging unit that captures an imaging area, and the distance measurement operation is performed by irradiating the imaging area with light by the distance measurement unit and receiving reflected light based on the light with respect to the imaging area;

[0278] Acquiring reflectivity information capable of determining the reflectivity within the imaging area; and

[0279] While the imaging operation and the distance measurement operation are being performed, the irradiation energy of the light is changed based on the acquired reflectivity information.

Claims

1. A processing device comprising: a first processor; and a memory connected to or built into the first processor, The first processor performs the following processing: executing control to perform an imaging operation and a distance measurement operation, wherein the imaging operation is performed by an imaging device that captures an imaging area, and in the distance measurement operation, the distance measurement is performed by irradiating the imaging area with light by the distance measurement device and receiving reflected light based on the light with respect to the imaging area, acquiring reflectivity information capable of determining the reflectivity within the imaging area, In the state of performing the imaging action and the ranging action, the irradiation energy of the light is changed according to the shooting scene indication information received by a receiving device capable of receiving shooting scene indication information indicating the shooting scene and the acquired reflectivity information, In a case where the photographing scene indicated by the photographing scene indication information received by the receiving device is a specific photographing scene having a photographing scene reflectivity greater than a reference reflectivity, the first processor makes the irradiation energy weaker than a first reference irradiation energy.

2. The processing device according to claim 1, wherein: The reflectivity information is information based on an imaging region image obtained by imaging the imaging region by the imaging device.

3. The processing device according to claim 2, wherein: The information based on the imaging area image is an image recognition result obtained by performing image recognition on the imaging area image by the second processor.

4. The processing device according to claim 3, wherein: The first processor obtains the image recognition result according to a learning result obtained by machine learning the correspondence between the image and the subject recognition result.

5. The processing device according to claim 3, wherein: The second processor performs the image recognition based on an image of an object having a reflectivity equal to or higher than a reference reflectivity and dictionary data corresponding to an identifier capable of identifying the image of the object.

6. The processing device according to claim 3 or 4, wherein: When the image recognition result acquired by the first processor indicates that an image representing an object having a reflectance greater than or equal to a reference reflectance is included in the imaging region image, the first processor makes the irradiation energy weaker than a second reference irradiation energy.

7. The processing device according to claim 6, wherein: The object is an object predetermined as an object having glossiness.

8. The processing device according to claim 6, wherein: The object is an object predetermined as a white object.

9. The processing device according to claim 7, wherein: The predetermined object is a mirror-like object.

10. The processing device according to any one of claims 1, 2, 3, 4, 5, 7, 8, and 9, wherein: The first processor performs control such that auxiliary light is generated from a light source as the light and a light receiving element receives auxiliary reflected light based on the auxiliary light with respect to the imaging area. The reflectivity information is information based on a light reception result of the auxiliary reflected light being received by the light receiving element.

11. The processing device according to claim 10, wherein: The information based on the light reception result is information based on the light reception amount per unit time of the auxiliary reflected light received by the light receiving element.

12. The processing device according to claim 10, wherein: The first processor makes the irradiation energy weaker than a third reference irradiation energy when the reflectance determined based on the acquired reflectance information is equal to or greater than a threshold value.

13. The processing device according to any one of claims 1, 2, 3, 4, 5, 7, 8, 9, 11, and 12, wherein: The distance measuring device has a TOF camera.

14. The processing device according to claim 13, wherein: The first processor displays the distance image generated by the TOF camera on a display.

15. The processing device according to claim 13, wherein: The TOF camera has a plurality of photoelectric conversion pixels. The distance measurement is performed using only the reflected light received by at least one photoelectric conversion pixel included in the designated area among the plurality of photoelectric conversion pixels.

16. The processing device according to claim 15, wherein: The at least one photoelectric conversion pixel is a photoelectric conversion pixel at a position corresponding to a position of an image region specified within a screen displaying an image of the imaging region obtained by imaging the imaging region by the imaging device.

17. The processing device according to any one of claims 1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 14, 15, and 16, wherein: The first processor changes the irradiation energy in accordance with the timing of the main exposure performed in the imaging operation.

18. The processing device according to any one of claims 1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 14, 15, and 16, wherein: The imaging operation includes an imaging operation for a live view image in which the imaging device captures the imaging area for a live view image.

19. The processing device according to any one of claims 1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 14, 15, and 16, wherein: The distance measurement operation is a distance measurement operation for focusing performed by the imaging device.

20. The processing device according to claim 19, wherein: The imaging device focuses on the imaging area based on a distance measurement result obtained by performing the distance measurement in the focusing distance measurement operation.

21. The processing device according to any one of claims 1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 14, 15, 16, and 20, wherein: The light is directional light.

22. An electronic device comprising: The processing device according to any one of claims 1 to 21; the camera device; and The distance measuring device.

23. A processing method comprising the following steps: executing control to perform an imaging operation and a distance measurement operation, wherein the imaging operation is performed by an imaging device that captures an imaging area, and in the distance measurement operation, the distance measurement is performed by irradiating the imaging area with light by the distance measurement device and receiving reflected light based on the light with respect to the imaging area; Acquiring reflectivity information capable of determining the reflectivity within the imaging area; and In the state of performing the imaging action and the ranging action, the irradiation energy of the light is changed according to the shooting scene indication information received by a receiving device capable of receiving shooting scene indication information indicating the shooting scene and the acquired reflectivity information, When the shooting scene indicated by the shooting scene instruction information received by the receiving device is a specific shooting scene having a shooting scene reflectivity greater than a reference reflectivity, the irradiation energy is made weaker than a first reference irradiation energy.

24. A computer-readable non-volatile storage medium storing a program for causing a computer to execute a process comprising the following steps: executing control to perform an imaging operation and a distance measurement operation, wherein the imaging operation is performed by an imaging device that captures an imaging area, and in the distance measurement operation, the distance measurement is performed by irradiating the imaging area with light by the distance measurement device and receiving reflected light based on the light with respect to the imaging area; Acquiring reflectivity information capable of determining the reflectivity within the imaging area; and In the state of performing the imaging action and the ranging action, the irradiation energy of the light is changed according to the shooting scene indication information received by a receiving device capable of receiving shooting scene indication information indicating the shooting scene and the acquired reflectivity information, When the shooting scene indicated by the shooting scene instruction information received by the receiving device is a specific shooting scene having a shooting scene reflectivity greater than a reference reflectivity, the irradiation energy is made weaker than a first reference irradiation energy.

Images