Processing device, electronic apparatus, processing method, and program

Abstract

To solve such a problem that when the recognition of a specific subject included in an imaging area is simultaneously performed, for example, when image recognition is performed to a visible light image obtained by imaging a paper sheet having a portrait of a person as a subject, reduction in accuracy of the image recognition such as erroneous determination of the subject as a "person" occurs.SOLUTION: A processing device includes: a control part for performing control for performing a recognition operation in which a recognition part recognizes a specific subject included in an imaging area on the basis of a captured image obtained by imaging the imaging area by an imaging part and a range-finding operation in which a range-finding part irradiates the imaging area with light and receives reflected light of the light to the imaging area, to perform range-finding a plurality of times in parallel; and a change part for changing irradiation energy of the light to the imaging area for each range-finding operation, and uses both of a recognition result obtained by the recognition operation and a range-finding result obtained by the range-finding operation for focusing control or the like.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The technology disclosed herein relates to a processing device, an electronic device, a processing method, and a program. [Background technology]

[0002] Japanese Patent No. 6321145 discloses a distance measuring device. The distance measuring device described in Japanese Patent No. 6321145 includes an image capturing unit that captures an image of a subject formed by an imaging optical system that captures an image of a subject, an emission unit that emits directional light that is light with directionality along the optical axis direction of the imaging optical system, the emission unit being adjustable in emission intensity of the directional light and emitting the directional light by adjusting the emission intensity based on at least one of focus state specification information and subject brightness or exposure state specification information, a light receiving unit that receives reflected light of the directional light from the subject, a derivation unit that derives the distance to the subject based on the timing at which the directional light is emitted by the emission unit and the timing at which the reflected light is received by the light receiving unit, an execution unit that performs at least one of focus adjustment of the imaging optical system on the subject and exposure adjustment prior to shooting by the image capturing unit, and a control unit that controls the focus adjustment and exposure adjustment for the execution unit. and a control unit that, when the first pressing operation is accepted by the acceptance unit, controls the execution unit to execute at least one of focus adjustment and exposure adjustment, controls the emission unit, the light receiving unit, and the derivation unit to start distance measurement on the condition that at least one of focus adjustment and exposure adjustment has been executed, and after the distance measurement is completed, causes a presentation unit that presents information to present information regarding the result of the distance measurement while the first pressing operation is maintained, and then controls the photographing unit to perform the actual exposure if the acceptance unit accepts a second pressing operation successively from the first pressing operation without releasing the pressing operation on the acceptance unit.

[0003] Japanese Patent Laid-Open Publication No. 2006-171120 discloses a photographing device. The photographing device described in Japanese Patent Laid-Open Publication No. 2006-171120 detects subject contrast and adjusts focus prior to photographing, and is characterized by including a first fill light emitter that irradiates the subject with light having a relatively wide wavelength band for subject contrast detection, a second fill light emitter that irradiates the subject with light having a relatively narrow wavelength band for subject contrast detection, and a switching means that switches between the first fill light emitter and the second fill light emitter.

[0004] WO 2018 / 142993 discloses a light emission control device. The light emission control device described in WO 2018 / 142993 includes a light amount setting unit that sets the amount of light of an AF (Auto Focus) assist light, and a light emission control unit that controls the emission of the AF assist light according to the setting by the light amount setting unit. Summary of the Invention

[0005] One embodiment of the technology disclosed herein provides a processing device, electronic device, processing method, and program that can achieve both distance measurement and recognition of a specific subject contained in the imaging area with high accuracy, compared to a case in which distance measurement based on light that is always irradiated to the imaging area with the same irradiation energy and recognition of a specific subject contained in the imaging area are performed simultaneously. [Means for solving the problem]

[0006] A first aspect of the technology of the present disclosure includes a control unit that controls a recognition operation in which a recognition unit recognizes a specific subject included in an imaging area based on an image obtained by imaging the imaging area with an imaging unit, and a distance measurement operation in which a distance measurement unit irradiates the imaging area with light and receives reflected light from the imaging area to measure distance, the recognition operation being performed multiple times in parallel; and a modification unit for modifying the illumination energy of the light onto the image area.

[0007] A second aspect of the technology disclosed herein is a processing device according to the first aspect, further including an output unit that outputs the recognition result obtained by the recognition operation and the ranging result obtained by the ranging operation to a specific output destination at a specific time out of multiple times.

[0008] A third aspect of the technology of the present disclosure is the processing device according to the second aspect, in which the specific output destination is a display unit capable of displaying at least one of the recognition result and the distance measurement result.

[0009] A fourth aspect of the technology of the present disclosure is the processing device according to the third aspect, in which the display unit displays a specific subject image showing the specific subject and an image surrounding the specific subject image as the recognition result.

[0010] A fifth aspect of the technology disclosed herein is a processing device according to any one of the first to fourth aspects, in which the imaging unit has a lens that can move along the optical axis, and the control unit moves the lens along the optical axis to a position on the optical axis that is determined according to the distance measurement result obtained by the distance measurement operation.

[0011] A sixth aspect of the technique of the present disclosure is the processing device according to the fifth aspect, in which the position is a focus position.

[0012] A seventh aspect of the technology disclosed herein is a processing device relating to any one of the first to fourth aspects, in which the imaging unit has a lens that can move along the optical axis, the ranging unit performs focusing control ranging by irradiating focusing control light onto the imaging area prior to the recognition operation and receiving focusing control reflected light from the focusing control light onto the imaging area, and the control unit moves the lens along the optical axis to a focusing position determined in accordance with the ranging result obtained by the focusing control ranging.

[0013] An eighth aspect of the technology disclosed herein is a processing device according to any one of the first to seventh aspects, further including a processing unit that performs specific processing using multiple recognition results obtained by multiple recognition operations.

[0014] A ninth aspect of the technology of the present disclosure is a processing device according to any one of the first to eighth aspects, in which the recognition result obtained by the recognition operation is weighted according to at least one of the specific subject and the irradiation energy used in a distance measurement operation parallel to the recognition operation.

[0015] A tenth aspect of the technique of the present disclosure is the processing device according to the ninth aspect, in which the control unit causes the imaging unit to perform imaging by main exposure based on the recognition result selected according to the weighting.

[0016] An eleventh aspect of the technology of the present disclosure is the processing device according to any one of the first to tenth aspects, in which the specific subject is a face.

[0017] A twelfth aspect of the technology of the present disclosure is the processing device according to the eleventh aspect, in which the face is a face having a specific expression.

[0018] A thirteenth aspect of the technique of the present disclosure is the processing device according to any one of the first to tenth aspects, in which the specific subject is an object having a reflectance less than a threshold value.

[0019] A fourteenth aspect of the technology disclosed herein is an electronic device including a processing device according to any one of the first to thirteenth aspects, and at least one of a recognition unit and a distance measurement unit.

[0020] A fifteenth aspect of the technology of the present disclosure is a processing method including controlling a recognition operation in which a recognition unit recognizes a specific subject included in an imaging area based on an image obtained by imaging the imaging area by an imaging unit, and a ranging operation in which a ranging unit irradiates light onto the imaging area and measures distance by receiving reflected light from the light onto the imaging area, to be performed multiple times in parallel, and changing the irradiation energy of light onto the imaging area for each ranging operation.

[0021] A sixteenth aspect of the technology of the present disclosure is a program for causing a computer to perform a process including controlling a recognition operation in which a recognition unit recognizes a specific subject included in an imaging area based on an image obtained by imaging the imaging area by an imaging unit, and a distance measurement operation in which a distance measurement unit irradiates light onto the imaging area and measures distance by receiving reflected light from the light onto the imaging area, in parallel multiple times, and changing the irradiation energy of light onto the imaging area for each distance measurement operation. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic perspective view showing an example of a usage mode of a smart device according to a first embodiment. [Diagram 2] FIG. 2 is a rear perspective view showing an example of the appearance of the rear side of the smart device according to the first embodiment. [Diagram 3] 1 is a schematic perspective view showing an example of the structure of a photoelectric conversion element included in the smart device according to the first embodiment. FIG. [Figure 4] 3 is a front perspective view showing an example of the appearance of the front side of the smart device shown in FIG. 2. [Diagram 5] 1 is a conceptual diagram showing an example of an aspect in which an imaging area is imaged by a distance measuring imaging device included in a smart device according to a first embodiment, and a visible light image is displayed on a display. FIG. [Figure 6] FIG. 11 is a conceptual diagram showing an example of an aspect in which a distance measuring operation and a recognition operation are performed by a distance measuring imaging device included in a smart device according to the first embodiment, and a live view image is displayed on a display. [Figure 7] FIG. 2 is a conceptual diagram showing an example of a still image captured by the smart device according to the first embodiment. [Figure 8] FIG. 2 is a block diagram showing an example of the configuration of electrical hardware of the smart device according to the first embodiment. [Figure 9] FIG. 4 is a conceptual diagram showing an example of reading an imaging processing program when imaging processing is executed by a CPU included in the smart device according to the first embodiment. [Figure 10] FIG. 2 is a functional block diagram showing an example of functions of a CPU and a distance measuring imaging device when imaging processing is performed by the CPU included in the smart device according to the first embodiment. [Figure 11] 5 is a flowchart showing an example of the flow of imaging processing according to the first embodiment. [Figure 12] FIG. 11 is a block diagram showing an example of the configuration of electrical hardware of a smart device according to a second embodiment. [Figure 13] 10 is a flowchart showing an example of the flow of imaging processing according to the second embodiment. [Figure 14] 13 is a flowchart showing a modified example of the flow of imaging processing according to the second embodiment. [Figure 15] 13 is a flowchart showing an example of an imaging process flow according to the third embodiment. [Figure 16] FIG. 13 is a schematic screen diagram showing an example of a ranging range designation guide screen displayed on a display included in a smart device according to another embodiment. [Figure 17] 13 is a schematic screen diagram showing an example of a visible light image and a designated image region displayed as a live view image on a display included in a smart device according to another embodiment. FIG. [Figure 18] 13 is a conceptual diagram for explaining an example of a method of narrowing down a distance measurement segment area of ​​a photoelectric conversion element included in a smart device according to another embodiment to a distance measurement designated segment area. FIG. [Figure 19] FIG. 11 is a schematic perspective view showing a first modified example of the external configuration of a smart device according to another embodiment. [Figure 20] FIG. 11 is a schematic perspective view showing a second modified example of the external configuration of a smart device according to another embodiment. [Figure 21] FIG. 11 is a schematic perspective view showing an example of an external configuration of a range-measuring imaging device according to another embodiment in combination with a smart device. [Figure 22] FIG. 11 is a conceptual diagram showing an example of an aspect in which an imaging processing program is installed in a smart device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an example of an embodiment of a distance measuring imaging device according to the technique of the present disclosure will be described with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] CPU is an abbreviation of "Central Processing Unit". RAM is an abbreviation of "Random Access Memory". ASIC is an abbreviation of "Application Specific Integrated Circuit". PLD is an abbreviation of "Programmable Logic Device". FPGA is an abbreviation of "Field-Programmable Gate Array”. SoC is an abbreviation for “System-on-a-chip”. SSD is an abbreviation for “Solid State Drive”. USB is an abbreviation for “Universal Serial Bus”. HDD is an abbreviation for “Hard Disk Drive”. EEPROM is an abbreviation for “Electrically Erasable and Programmable Read Only LTE is an abbreviation for "Electro-Luminescence". A / D is an abbreviation for "Analog / Digital". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". LTE is an abbreviation for "Long Term Evolution". 5G is an abbreviation for "5th Generation". LD is an abbreviation for "Laser Diode". IR is an abbreviation for "Infrared". APD is an abbreviation for "Avalanche Photodiode". fps is an abbreviation for "frame per second". LED is an abbreviation for "Light Emitting Diode". ROI is an abbreviation for "Region of Interest". LAN is an abbreviation for "Local Area Network". Exif is an abbreviation for "Exchangeable Image File Format".

[0026] In the description of this specification, "horizontal" refers to horizontal in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to completely horizontal. In the description of this specification, "parallel" refers to parallel in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to completely parallel. In the description of this specification, "vertical" refers to vertical in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to completely vertical. In the description of this specification, "same" refers to same in the sense of including a generally acceptable error in the technical field to which the technology of the present disclosure belongs, in addition to completely same. In addition, in the description of this specification, a numerical range expressed using "~" means a range including the numerical values ​​written before and after "~" as the lower limit and upper limit.

[0027] [First embodiment] 1, the smart device 10 according to the first embodiment performs a recognition operation for recognizing a specific subject included in an imaging area based on a captured image obtained by capturing an image of the imaging area defined by an angle of view θ1, and a distance measurement operation for measuring a distance by irradiating the imaging area with a laser beam and receiving the reflected light of the laser beam from the imaging area. The laser beam is an example of the "light" according to the technology of the present disclosure.

[0028] In this embodiment, "distance measurement" refers to a process of measuring the distance from the smart device 10 to a distance measurement target in the imaging area. Here, the "distance measurement target" refers to an object that reflects light, and in the example shown in Fig. 1, a person and a tree are shown as the distance measurement target in the imaging area. Note that examples of the smart device 10 include a smartphone or a tablet terminal, which is an electronic device with an imaging function.

[0029] As an example, as shown in Fig. 2, a smart device 10 includes a housing 12. The housing 12 houses a distance measuring imager 14. The distance measuring imager 14 includes a light illuminator 16 and a light receiver 18. The light illuminator 16 includes an LD 24, and the light receiver 18 includes a photoelectric conversion element 26. In the smart device 10, the imaging operation and distance measuring operation are performed by the distance measuring imager 14. Note that the distance measuring imager 14 is an example of an "imaging section (imaging device)" and a "distance measuring section (distance measuring device)" according to the technology of the present disclosure.

[0030] The smart device 10 has command keys 13 arranged on a side thereof. The command keys 13 receive various commands. The "various commands" referred to here include, for example, a command to display a menu screen from which various menus can be selected, a command to select one or more menus, a command to confirm the selected content, and a command to erase the selected content.

[0031] When the smart device 10 is placed vertically, the housing 12 has an upper part of the rear surface 12A (upper part when viewed from the rear of the smart device 10 placed vertically) provided with light-transmitting windows 20 and 22. The light-transmitting windows 20 and 22 are optical elements (e.g., lenses) having light-transmitting properties, and are arranged at a predetermined interval (e.g., an interval of several millimeters) along the horizontal direction, and are exposed from the rear surface 12A. The light irradiator 16 irradiates the laser light emitted from the LD 24 to the distance measurement target through the light-transmitting window 20. In this embodiment, a laser light in the infrared wavelength range is used. However, the wavelength range of the laser light is not limited to this, and may be a laser light in another wavelength range.

[0032] The light receiver 18 captures the reflected IR light through the light-transmitting window 22. The reflected IR light refers to reflected light caused by the laser light irradiated onto the target to be measured by the light irradiator 16. The light receiver 18 also captures reflected visible light through the light-transmitting window 22. The reflected visible light refers to reflected light caused by visible light (for example, visible light contained in sunlight) irradiated onto the imaging area. For ease of explanation, hereinafter, when there is no need to distinguish between reflected IR light and reflected visible light, they will simply be referred to as "reflected light."

[0033] The light receiver 18 is equipped with a photoelectric conversion element 26, which receives the reflected light that is taken into the light receiver 18 through the light-transmitting window 22, and outputs an electrical signal according to the amount of reflected light received.

[0034] 3, the photoelectric conversion element 26 has a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896×3265" pixels.

[0035] A color filter is disposed on each photodiode included in the photoelectric conversion element 26. The color filters include a G filter corresponding to the G (green) wavelength range that contributes most to obtaining a luminance signal, an R filter corresponding to the R (red) wavelength range, a B filter corresponding to the B (blue) wavelength range, and an IR filter corresponding to the IR (infrared) wavelength range. In this embodiment, the G filter, R filter, and B filter also function as infrared light cut filters that cut infrared light. In the following description, for convenience of explanation, when there is no need to distinguish between the G filter, R filter, and B filter, they will also be referred to as "visible light filters."

[0036] The photoelectric conversion element 26 has an R pixel, a G pixel, a B pixel, and an IR pixel. The R pixel is a pixel corresponding to a photodiode in which an R filter is arranged, the G pixel is a pixel corresponding to a photodiode in which a G filter is arranged, the B pixel is a pixel corresponding to a photodiode in which a B filter is arranged, and the IR pixel is a pixel corresponding to a photodiode in which an IR filter is arranged. The R pixel, the G pixel, the B pixel, and the IR pixel are arranged with a predetermined periodicity in each of the row direction (horizontal direction) and the column direction (vertical direction). In this embodiment, the arrangement of the R pixel, the G pixel, the B pixel, and the IR pixel is an arrangement obtained by replacing some of the G pixels in the X-Trans (registered trademark) arrangement with IR pixels. The IR pixel is arranged with a specific periodicity along the row direction and the column direction.

[0037] Note that, here, an arrangement based on an X-Trans arrangement is exemplified as the arrangement of the R pixels, G pixels, B pixels, and IR pixels, but the technology of the present disclosure is not limited to this, and the arrangement of the R pixels, G pixels, B pixels, and IR pixels may be an arrangement based on other arrangements, such as a Bayer arrangement or a Honeycomb (registered trademark) arrangement.

[0038] In addition, an arrangement obtained by replacing some G pixels with IR pixels among an arrangement generally known as an arrangement of R pixels, G pixels, and B pixels is exemplified here as an arrangement of R pixels, G pixels, B pixels, and IR pixels, but the technology of the present disclosure is not limited to this. For example, each color filter corresponding to each of the R pixels, G pixels, and B pixels (hereinafter, these are also referred to as "visible light pixels") may be a color filter that transmits infrared light as well, and a pair of photodiodes, one for visible light pixels and the other for IR pixels (e.g., InGaAs APD), may be arranged for each color filter.

[0039] In this embodiment, the photoelectric conversion element 26 is divided into two regions. That is, the photoelectric conversion element 26 has a visible light image division region 26N1 and a distance measurement division region 26N2. The visible light image division region 26N1 is a visible light pixel group consisting of a plurality of visible light pixels, and is used to generate a visible light image. The distance measurement division region 26N2 is an IR pixel group consisting of a plurality of IR pixels, and is used for distance measurement. The visible light image division region 26N1 receives visible reflected light and outputs an electrical signal according to the amount of received light. The distance measurement division region 26N2 receives IR reflected light and outputs an electrical signal according to the amount of received light.

[0040] As an example, as shown in Fig. 4, a touch panel display 59 is provided on the front surface 12B of the housing 12. The touch panel display 59 includes a display 46 and a touch panel 48. An example of the display 46 is an organic electroluminescence (EL) display. The display 46 may be another type of display, such as a liquid crystal display, instead of an organic electroluminescence (EL) display. The display 46 is an example of a "display unit" according to the technology of the present disclosure.

[0041] The display 46 displays images, text information, and the like. The touch panel 48 is a transmissive touch panel, and is overlaid on the surface of the display area of ​​the display 46. The touch panel 48 receives instructions from a user by detecting contact with a pointing object such as a finger or a stylus pen. Although an out-cell type touch panel display in which touch panel 48 is superimposed on the surface of the display area of ​​display 46 is given, this is merely an example. For example, an on-cell type or in-cell type touch panel display can also be applied as touch panel display 59.

[0042] 5, in the smart device 10, when an instruction to start capturing an image is received by the touch panel 48, the optical receiver 18 captures an image of the captured area. That is, the optical receiver 18 receives visible reflected light and generates a visible light image showing the captured area as an image corresponding to the received visible reflected light. The visible light image is an example of a "captured image" according to the technology of the present disclosure.

[0043] The visible light image is displayed on the display 46 in response to an instruction received by the touch panel 48. In the example shown in Fig. 5, the imaging area is defined by a field angle θ1. The field angle θ1 is changed in response to an instruction received by the touch panel 48.

[0044] The smart device 10 has an image recognition function. The smart device 10 recognizes a face image showing a person's face from a visible light image by activating the image recognition function. The smart device 10 performs a recognition operation (hereinafter, also simply referred to as a "recognition operation") for recognizing a person's face included in an imaging area. The recognition operation is realized by activating the image recognition function, that is, by recognizing a face image from a visible light image. In the example shown in FIG. 5, a face image is recognized from a visible light image, and a frame line 29 is displayed surrounding the face image as a recognition result. The recognition result includes, for example, the recognized face image, the number of recognized face images (hereinafter, also referred to as the "number of face images"), coordinates that can specify the position of the face image, and frame line information indicating the frame line 29. Note that a person's face is an example of a "specific subject" according to the technology of the present disclosure. Also, the frame line 29 is an example of a "distance measurement result" and an "image surrounding a specific subject image" according to the technology of the present disclosure.

[0045] The smart device 10 also has a distance measurement function. By using the distance measurement function, the smart device 10 irradiates the light irradiator 16 with a laser light at an irradiation angle θ2, and receives the reflected IR light at the distance measurement section area 26N2 of the light receiver 18, as shown in FIG. 6 as an example. The irradiation angle θ2 may be the same as the angle of view θ1, or may be different. The distance from the smart device 10 to the object to be measured is measured based on the time required from the irradiation of the laser light to the reception of the reflected IR light and the speed of light. For example, if the distance to the object to be measured is "L", the speed of light is "c", and the time required from the irradiation of the laser light by the light irradiator 16 to the reception of the reflected IR light by the distance measurement section area 26N2 is "t", the distance L is calculated according to the formula "L=c×t×0.5".

[0046] The recognition result and the distance measurement result are superimposed on the visible light image. The superimposed image obtained by superimposing the recognition result and the distance measurement result on the visible light image is displayed on the display 46 as a live view image. In the example shown in FIG. 6, a frame 29 surrounding a face image is superimposed on the visible light image as the recognition result, and numerical values ​​indicating the distance from the smart device 10 to the distance measurement target (1.6 m, 1.8 m, and 5.3 m in the example shown in FIG. 6) are superimposed on the visible light image as a superimposed image, and the superimposed image is displayed on the display 46 as a live view image. In the example shown in FIG. 6, the distance from the smart device 10 to each of a plurality of representative points (three points in the example shown in FIG. 6) in the imaging area is superimposed on the visible light image. One example of the representative plurality of points includes a plurality of points among specific subjects (e.g., subjects included in the screen center area and / or humans, etc.) in the imaging area whose contrast difference is equal to or greater than a preset value.

[0047] The display 46 displays the soft keys 28 together with the live view image. The key 28 is operated by a user or the like when instructing to start capturing an image for recording. Examples of the image for recording include a still image and / or a moving image. As shown in FIG. 7, the user determines the capturing area while viewing the live view image, and operates the soft key 28. In the smart device 10, when the user operates the soft key 28, the focus is adjusted according to the recognition result and the distance measurement result, and the image for recording is captured. In the following, the exposure performed in capturing the image for recording is also referred to as the "main exposure."

[0048] The configuration of the smart device 10 will be described with reference to Fig. 8. In addition to the light irradiator 16 and the light receiver 18, the smart device 10 includes a controller 15, an input / output interface 40, an image memory 42, a UI device 44, an external I / F 52, and a communication I / F 54. The controller 15 is an example of a "processing device" and a "computer" according to the technology of the present disclosure.

[0049] The controller 15 includes a CPU 15A, a storage 15B, and a memory 15C. The CPU 15A is an example of a "processor" and a "recognition processor" according to the technology of the present disclosure, and the memory 15C is an example of a "memory" according to the technology of the present disclosure. The CPU 15A, the storage 15B, and the memory 15C are connected via a bus 50, and the bus 50 is connected to an input / output interface 40. In the example shown in FIG. 8, one bus is illustrated as the bus 50 for convenience of illustration, but multiple buses may be used. The bus 50 may be a serial bus or a parallel bus including an information bus, an address bus, a control bus, and the like.

[0050] The storage 15B stores various parameters and various programs. The storage 15B is a non-volatile storage device. Here, a flash memory is adopted as an example of the storage 15B. The flash memory is merely an example, and the storage 15B may be, for example, various non-volatile memories such as a magnetoresistive memory and / or a ferroelectric memory instead of the flash memory or in addition to the flash memory. The non-volatile storage device may be an EEPROM, a HDD, and / or an SSD. The memory 15C temporarily stores various information and is used as a work memory. An example of the memory 15C is a RAM, but is not limited to this and may be another type of storage device.

[0051] The storage 15B stores various programs including an imaging processing program 70. The imaging processing program 70 is an example of a "program" according to the technology of the present disclosure. The CPU 15A reads out a necessary program from the storage 15B and executes the read program on the memory 15C. The CPU 15A controls the entire smart device 10 according to the program executed on the memory 15C.

[0052] A plurality of devices are connected to the input / output interface 40, and the input / output interface 40 controls the transmission and reception of various information between the plurality of devices. In the example shown in Fig. 8, the plurality of devices connected to the input / output interface 40 include the controller 15, the light irradiator 16, the light receiver 18, the image memory 42, the UI device 44, the external I / F 52, and the communication I / F 54.

[0053] The external I / F 52 is responsible for transmitting and receiving various information between the smart device 10 and a device (hereinafter, also referred to as an "external device") that exists outside the smart device 10. An example of the external I / F 52 is a USB interface. The USB interface can be directly or indirectly connected to an external device (not shown) such as a smart device, a personal computer, a server, a USB memory, a memory card, and / or a printer.

[0054] The communication I / F 54 has a communication function such as LTE, 5G, wireless LAN, and / or Bluetooth (registered trademark), and controls the exchange of various information between an external device and the CPU 15A. For example, the communication I / F 54 is communicatively connected to a network 56 (e.g., the Internet) via a base station (not shown), and controls the exchange of various information between an external device on the network 56 and the CPU 15A.

[0055] The UI device 44 includes a display 46, and the CPU 15A causes the display 46 to display various pieces of information. The UI device 44 also includes a reception device 47. The reception device 47 includes a touch panel 48 and a hard key unit 53. The hard key unit 53 is at least one hard key including the instruction key 13 (see FIG. 2). The CPU 15A operates according to various instructions received by the touch panel 48. Note that, although the hard key unit 53 is included in the UI device 44 here, the technology of the present disclosure is not limited to this, and for example, the hard key unit 53 may be connected to an external I / F 52.

[0056] The light irradiator 16 includes a light-transmitting window 20, a beam expander 21, a collimating lens 23, an LD 24, and an LD driver 25, and the light-transmitting window 20, the beam expander 21, and the collimating lens 23 are arranged in this order along the optical axis L1 from the imaging region side (object side) to the LD 24. The LD driver 25 is connected to the LD 24 and the input / output interface 40, and drives the LD 24 in accordance with instructions from the CPU 15A to cause the LD 24 to emit laser light.

[0057] The laser light emitted from the LD 24 is converted into parallel light by the collimator lens 23, and the light diameter is expanded by the beam expander 21, and the light is irradiated from the light transmitting window 20 toward the object to be measured.

[0058] The light receiver 18 includes a light-transmitting window 22, an objective lens 30A, a focus lens 30B, an aperture 30C, a photoelectric conversion element 26, a photoelectric conversion element driver 32, and a signal processing circuit 34. In the light receiver 18, the light-transmitting window 22, the objective lens 30A, the focus lens 30B, and the aperture 30C are arranged in this order along the optical axis L2 from the imaging region side (object side) to the photoelectric conversion element 26. The photoelectric conversion element driver 32 is connected to the photoelectric conversion element 26 and the input / output interface 40, and drives the photoelectric conversion element 26 according to an instruction from the CPU 15A. For example, the photoelectric conversion element driver 32 supplies the photoelectric conversion element 26 with an imaging timing signal that specifies the timing of imaging performed by the photoelectric conversion element 26 under the control of the CPU 15A. The photoelectric conversion element 26 performs resetting, exposure, and output of an electrical signal according to the imaging timing signal supplied from the photoelectric conversion element driver 32. The imaging timing signal includes, for example, a vertical synchronization signal and a horizontal synchronization signal.

[0059] The light receiver 18 includes a focusing control mechanism 31. The focusing control mechanism 31 includes a focus lens 30B, a moving mechanism 60, a motor 62, and a motor driver 64. The focus lens 30B is supported by the moving mechanism 60 so as to be slidable along the optical axis L2. The motor 62 is connected to the moving mechanism 60 and the motor driver 64. The motor driver 64 is connected to the input / output interface 40, and drives the motor 62 according to an instruction from the CPU 15A. The moving mechanism 60 is connected to a drive shaft (not shown) of the motor 62, and selectively moves the focus lens 30B along the optical axis L2 to the object side and the image side by receiving power from the motor 62. That is, the CPU 15A adjusts the focusing position by controlling the driving of the motor 62 via the motor driver 64. Here, the "focusing position" refers to the position of the focus lens 30B on the optical axis L2 in a focused state (for example, a state in which the contrast of the visible light image is maximized, or a state in which a predetermined depth of field is realized). In the following, for convenience of explanation, the focus lens 30B is set to the in-focus position. The control for adjusting the focus is also called "focus control."

[0060] The aperture 30C is a fixed aperture whose opening does not change. In the case of a fixed aperture, exposure adjustment is performed by an electronic shutter of the photoelectric conversion element 26. The aperture 30C may not be a fixed aperture, but may be a variable aperture. Note that the objective lens 30A, the focus lens 30B, and the aperture 30C included in the optical receiver 18 are merely examples, and the technology of the present disclosure is valid even if the lens configuration and / or the position of the aperture 30C are changed.

[0061] The reflected light is incident on the light receiver 18 through the light-transmitting window 22. The reflected light incident on the light-transmitting window 22 is imaged on the photoelectric conversion element 26 via the objective lens 30A, the focus lens 30B, and the aperture 30C.

[0062] The photoelectric conversion element 26 is connected to a signal processing circuit 34, and outputs pixel data indicating a pixel value for each of the visible light pixels and the IR pixels to the signal processing circuit 34. The signal processing circuit 34 digitizes the pixel data input from the photoelectric conversion element 26 by performing A / D conversion, and performs various types of signal processing on the digitized pixel data.

[0063] The signal processing circuit 34 includes a visible light pixel data processing circuit 34A and an IR pixel data processing circuit 34B. The visible light pixel data processing circuit 34A generates a visible light image by performing known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction on the pixel data of the visible light pixels. The visible light pixel data processing circuit 34A then stores the visible light image in an image memory 42. The image memory 42 stores 10 frames of visible light images captured sequentially while changing the intensity of the distance measurement laser light in stages, as described below.

[0064] In the distance measurement section area 26N2, the IR reflected light is received by the IR pixel and is used to measure the distance from the smart device 10 to the distance measurement target based on the emission timing and the light reception timing. The IR pixel data processing circuit 34B acquires an emission timing signal indicating the timing at which the laser light is emitted from the LD 24 (hereinafter also referred to as the "emission timing") from the CPU 15A. The IR pixel data processing circuit 34B measures the distance from the smart device 10 to the distance measurement target for each IR pixel based on the emission timing indicated by the emission timing signal and the timing at which the IR reflected light is received by each IR pixel (hereinafter also referred to as the "light reception timing"). Then, the IR pixel data processing circuit 34B associates the distance measurement result including the measured distance with the visible light image and stores it in the image memory 42.

[0065] Incidentally, when the smart device 10 performs image recognition on a visible light image obtained by capturing an image of a paper on which a person is drawn as a subject, the smart device 10 may erroneously determine that the subject is a "person". However, the smart device 10 can determine whether the subject is a "paper on which a person is drawn" or a "person" by performing image recognition using a distance measurement result obtained by measuring the distance in the depth direction of the subject. In other words, the smart device 10 can improve the accuracy of image recognition by performing image recognition using a distance measurement result. Image recognition is performed on a visible light image generated by the visible light pixel data processing circuit 34A based on the light reception result in the visible light image division region 26N1. In the visible light image division region 26N1, it is ideal that the IR reflected light is completely blocked by the visible light filter, but depending on the intensity of the IR reflected light, it is difficult to completely block it with the visible light filter. When the IR reflected light reaches a visible light pixel, the IR reflected light may appear in the visible light image as noise. When image recognition is performed on a visible light image, if the image quality of the visible light image deteriorates, the accuracy of the image recognition decreases. For example, if the intensity of the laser light from the LD 24 is set to zero, the reflected IR light will also be zero, so the reflected IR light will not be captured as noise in the visible light image, but distance measurement will not be possible. Therefore, the accuracy of image recognition is reduced compared to when the distance measurement results are used for image recognition.

[0066] Therefore, in the smart device 10, as shown in FIG. 9 as an example, the CPU 15A reads out an imaging processing program 70 from the storage 15B, and executes imaging processing in accordance with the read out imaging processing program.

[0067] 10 as an example, the imaging process is realized by the distance measuring imaging device 14 operating as an imaging section 91 and a distance measuring section 92, and the CPU 15A operating as a change section 96, a control section 93, a recognition section 94, a processing section 95, an output section 97, and a display control section 99. The imaging section 91 includes a light receiver 18, and the distance measuring section 92 includes a light irradiator 16, a distance measuring partition area 26N2 (see FIGS. 3 and 8), an IR pixel data processing circuit 34B, etc.

[0068] The change unit 96 changes the irradiation energy of the laser light (hereinafter, also simply referred to as "irradiation energy") in multiple stages. In the present embodiment, 10 stages are adopted as an example of "multiple stages". The change unit 96 changes the irradiation energy by changing the intensity of the laser light emitted from the LD 24 via the LD driver 25. In the state where the irradiation energy is the lowest, the irradiation energy may be zero. The laser light is irradiated in synchronization with the imaging of the visible light image. Here, an example in which the intensity of the laser light is changed by the change unit 96 is given, but the technology of the present disclosure is not limited thereto. For example, the irradiation energy of the laser light may be changed by changing the emission time of the laser light and / or the number of times the laser light is emitted per unit time together with or instead of the intensity of the laser light.

[0069] The control unit 93 causes the imaging unit 91 to capture an image of the imaging area at a default frame rate (e.g., 120 fps). The visible light image captured by the imaging unit 91 is stored in the image memory 42. The control unit 93 also performs continuous parallel processing. Continuous parallel processing refers to multiple consecutive parallel processing. Here, multiple times refers to 10 times. 10 times means, for example, 10 frames of imaging performed by the imaging unit 91. Parallel processing refers to processing in which the recognition unit 94 and the distance measurement unit 92 perform a recognition operation and a distance measurement operation in parallel. The recognition operation is performed by the recognition unit 94, and the distance measurement operation is performed by the distance measurement unit 92. In other words, the control unit 93 controls the recognition unit 94 and the distance measurement unit 92 to perform the recognition operation and the distance measurement operation in parallel for 10 frames.

[0070] In addition, although an example in which the recognition operation and the distance measurement operation are performed in parallel for 10 frames has been described here, the technology of the present disclosure is not limited thereto, and the recognition operation and the distance measurement operation may be performed in parallel for less than 10 frames or 11 frames or more, or the recognition operation and the distance measurement operation may be performed in parallel multiple times. The number of times the recognition operation and the distance measurement operation are performed in parallel may be a fixed value, may be a value that is changed according to an instruction accepted by the accepting device 47, or may be a value that is changed according to the operation mode of the smart device 10 and / or the imaging scene, etc.

[0071] In addition, in this embodiment, the continuous parallel processing is performed repeatedly. That is, the process of performing the recognition operation and the distance measurement operation in parallel for 10 frames is repeated multiple times. In this embodiment, the continuous parallel processing is repeated several tens of times (for example, 30 times) or more. The number of times the continuous parallel processing is repeated may be a fixed value, may be a value that is changed according to an instruction accepted by the accepting device 47, or may be a value that is changed according to the operation mode of the smart device 10 and / or the imaging scene, etc.

[0072] The IR pixel data processing circuit 34B performs the following operations after the laser light is emitted by the light emitter 16: Based on the time required for the reflected IR light to be received by the distance measurement division area 26N2 (see FIG. 3 and FIG. 8) and the speed of light, the distance from the smart device 10 to the measurement target is measured for each IR pixel. At this time, depending on the irradiation energy of the laser light and the distance to the measurement target, there may be IR pixels that cannot receive the reflected IR light sufficiently. Here, "IR pixels that cannot receive the reflected IR light sufficiently" refers to IR pixels that do not receive the amount of reflected IR light received from a measurement target several meters away, which is determined in advance by a test using an actual device and / or a computer simulation, as the amount of reflected IR light that can be measured. In general, the higher the irradiation energy of the laser light, the longer the measurable distance, and the weaker the energy, the shorter the measurable distance. The IR pixel data processing circuit 34B has a threshold value used for comparison with the amount of reflected IR light received by each IR pixel (for example, the amount of light received per unit time), and outputs blank information for IR pixels that receive an amount of reflected IR light that is less than the threshold value, as they could not be measured correctly. The distance measurement results for one frame obtained by the distance measurement unit 92 are stored in the image memory 42 in association with the visible light image.

[0073] The distance measurement result includes the distance measured by the distance measurement unit 92, i.e., the distance from the smart device 10 to the distance measurement target, the number of IR pixels (hereinafter also referred to as "correct distance measurement IR pixels") for which distance measurement has been performed correctly (hereinafter also referred to as "correct distance measurement IR pixel number"), and blank information. The correct distance measurement IR pixel number is calculated by the IR pixel data processing circuit 34B. The correct distance measurement IR pixel number is, for example, an IR pixel that receives IR reflected light with an amount of received light equal to or greater than a threshold, and generally increases as the irradiation energy of the laser light increases. Conversely, when the irradiation energy of the laser light is high, the amount of laser light reflected in the visible light image increases, deteriorating the image quality, so the higher the irradiation energy of the laser light, the fewer the number of face images.

[0074] The recognition unit 94 reads out the visible light image from the image memory 42, and performs image recognition of the face image on the read visible light image by referring to an image recognition dictionary (not shown). The image recognition dictionary has, for example, a plurality of types of face images registered. The recognition result for one frame obtained by performing the image recognition is stored in the image memory 42 in association with the visible light image and the distance measurement result.

[0075] The processing unit 95 reads out the distance measurement results and the recognition results for one frame from the image memory 42. Then, the processing unit 95 performs a rating process. Here, the rating process refers to a process of rating the corresponding frame for each frame (here, for each of 10 frames, as an example) according to the number of IR pixels and the number of face images included in the read distance measurement results. Rating each frame means that the recognition result corresponding to each frame is weighted. Note that the number of face images is an example of "multiple recognition results obtained by multiple recognition operations" according to the technology of the present disclosure. Also, the rating process is an example of "specific processing" according to the technology of the present disclosure. Also, hereinafter, for convenience of explanation, one frame to be the target of rating is also referred to as a "rating target frame".

[0076] The rating process is executed by the processing unit 95 according to the following formula (1). Specifically, as shown in the following formula (1), the processing unit 95 calculates the sum of a value obtained by multiplying the number of correct ranging IR pixels by a coefficient A and a value obtained by multiplying the number of face images by a coefficient B different from the coefficient A, as a rating value. Here, positive values ​​are used for the coefficients A and B.

[0077] Rating value = (number of correct ranging IR pixels) × A + (number of face images) × B (1)

[0078] In the above formula (1), the coefficients A and B do not have to be positive values. For example, to obtain a rating value ignoring the distance measurement result, the coefficient A can be set to "0", and to obtain a rating value ignoring the recognition result, the coefficient B can be set to "0". The coefficient A varies depending on the irradiation energy. For example, when the irradiation energy is "0", the coefficient A may be set to "0", and the higher the irradiation energy, the larger the value of the coefficient A may be. In this way, the rating value is calculated as a value according to the irradiation energy used in the distance measurement operation parallel to the recognition operation. In addition, the coefficient A and / or the coefficient B may be a fixed value, a value that is changed according to an instruction accepted by the accepting device 47, or a value that is changed according to the operation mode and / or the imaging scene of the smart device 10.

[0079] According to the above formula (1), the more the number of correct ranging IR pixels and the more the number of face images, the higher the rating value. The rating values ​​are stored in the image memory 42 in association with the visible light image, the ranging result, and the recognition result.

[0080] The image memory 42 stores visible light images for 10 frames captured sequentially while changing the irradiation energy level R of the laser light in 10 steps from 1 to 10, distance measurement results, recognition results, and rating values.

[0081] The output unit 97 outputs the recognition result and the distance measurement result for a specific frame of the multiple frames obtained by imaging the imaging area multiple times by the imaging unit 91 according to a default frame rate. Specifically, the output unit 97 refers to the rating values ​​for 10 frames, and outputs the visible light image, the distance measurement result, and the recognition result of the frame with the highest rating value among the 10 frames to the live view image storage area 42A of the image memory 42. Note that the frame with the highest rating value among the 10 frames is an example of a "specific number of times" according to the technology of the present disclosure, and the live view image storage area 42A is an example of a "specific output destination" according to the technology of the present disclosure.

[0082] The display control unit 99 acquires the visible light image, the distance measurement result, and the recognition result from the live view image storage area 42A, and generates a superimposed image based on the acquired visible light image, the distance measurement result, and the recognition result. The display control unit 99 outputs the generated superimposed image to the display 46 as a live view image, thereby causing the display 46 to display the live view image (see FIG. 6). The output unit 97 and the display control unit 99 are an example of an "output unit" according to the technology of the present disclosure. The display 46 is an example of a "specific output destination" and a "display unit" according to the technology of the present disclosure.

[0083] Note that, although an image obtained by superimposing both the distance measurement result and the frame line 29 on the visible light image is exemplified here as the superimposed image, the technology of the present disclosure is not limited to this, and an image may be obtained by superimposing only either the distance measurement result or the frame line 29 on the visible light image. Furthermore, the superimposed image and the visible light image may be selectively displayed on the display 46 as a live view image.

[0084] Next, the operation of the portion of the smart device 10 related to the technology of the present disclosure will be described with reference to Fig. 11. Note that Fig. 11 shows an example of the flow of imaging processing executed by the CPU 15A.

[0085] In the imaging process shown in FIG. 11, first, in step ST10, the control unit 93 determines whether or not a condition for starting the imaging process (hereinafter also referred to as an "imaging process start condition") is satisfied. One example of the imaging process start condition is a condition that an instruction to start the imaging process is received via the touch panel 48. If the imaging process start condition is not satisfied in step ST10, the determination in step ST10 is made again. If the imaging process start condition is satisfied in step ST10, the imaging process proceeds to step ST12.

[0086] In step ST12, the change unit 96 sets the irradiation energy level R of the laser light to the maximum value 10. After that, the imaging process proceeds to step ST14.

[0087] In step ST14, the control unit 93 determines whether or not the imaging timing has arrived. The imaging timing arrives, for example, at a period (for example, 1 / 120 seconds) determined by a default frame rate. If the imaging timing has not arrived in step ST14, the determination is negative, and the determination in step ST14 is performed again. If the imaging timing has arrived in step ST14, the determination is positive, and the imaging process proceeds to step ST16.

[0088] In step ST16, the control unit 93 controls the imaging unit 91 to expose the photoelectric conversion element 26. That is, the control unit 93 causes the photoelectric conversion element driver 32 to output an imaging timing signal to the photoelectric conversion element 26, thereby resetting the photoelectric conversion element 26 and causing the photoelectric conversion element 26 to accumulate new charges. Then, the control unit 93 causes the visible light pixel data processing circuit 34A to generate a visible light image according to the amount of charge accumulated in the photoelectric conversion element 26. The visible light image obtained by exposing the photoelectric conversion element 26 is stored in the image memory 42. Furthermore, the control unit 93 controls the distance measuring unit 92 to irradiate the laser light. After that, the imaging process proceeds to step ST18.

[0089] In step ST18, the control unit 93 performs parallel processing. That is, in step ST18, the distance measurement unit 92 performs a distance measurement operation, and the recognition unit 94 performs a recognition operation. The distance measurement unit 92 measures the distance from the smart device 10 to the distance measurement target for each IR pixel. The recognition unit 94 reads out a visible light image from the image memory 42, and performs image recognition of a face image on the read visible light image. The distance measurement results and recognition results for one frame obtained here are stored in the image memory 42 by the control unit 93 in association with the visible light image. After that, the imaging process proceeds to step ST20.

[0090] In step ST20, the processing unit 95 reads out the ranging results and the recognition results of the rating target frame from the image memory 42, and calculates a rating value for the ranging results and the recognition results of the rating target frame according to the number of correct ranging IR pixels and the number of face images. The rating value is stored in the image memory 42 in association with the visible light image, the ranging results, and the recognition results for the rating target frame. After that, the imaging process proceeds to step ST22.

[0091] In step ST22, the processing unit 95 judges whether or not the irradiation energy level R of the laser light is equal to the minimum value 1. In step ST22, if R=1 is not satisfied, the judgment is negative and the imaging process proceeds to step ST23. In step ST22, if R=1 is satisfied, the judgment is positive and the imaging process proceeds to step ST24.

[0092] In step ST23, the processing unit 95 subtracts "1" from the irradiation energy level R. After that, the imaging process proceeds to step ST14.

[0093] In this manner, by repeating the processes from step ST14 to step ST20, the irradiation energy level R decreases by 1 from the maximum value 10 to the minimum value 1, and accordingly, the irradiation energy level R changes in 10 steps. As the irradiation energy level R changes by one step, the visible light images obtained by sequentially capturing images of the imaging area by the imaging unit 91, the distance measurement results by the distance measurement unit 92, the recognition results by the recognition unit 94, and the rating values ​​calculated by the processing unit 95 are stored in the image memory 42 for 10 frames.

[0094] In step ST24, the output unit 97 refers to the rating values ​​for the 10 frames, and outputs the visible light image, the distance measurement result, and the recognition result of the frame with the highest rating to the live view memory 42. The image is output to the image storage area 42A. After that, the imaging process proceeds to step ST26.

[0095] In step ST26, the display control unit 99 acquires the visible light image, the distance measurement result, and the recognition result of the frame with the highest rating from the live view image storage area 42A. Then, the display control unit 99 generates a superimposed image from the visible light image, the distance measurement result, and the recognition result of the frame with the highest rating acquired from the live view image storage area 42A, and causes the display 46 to display the generated superimposed image as a live view image. As a result, the live view image generated using the distance measurement result and the recognition result of the frame with the best overall accuracy of the distance measurement operation and the recognition operation among the distance measurement results and the recognition results for 10 frames acquired while changing the irradiation energy level R in 10 steps is displayed on the display 46. After that, the imaging process proceeds to step ST28.

[0096] In step ST28, the control unit 93 determines whether or not the soft key 28 (see FIG. 6) has been operated by the user. If the soft key 28 has not been operated in step ST28, the determination is negative, and the imaging process proceeds to step ST32. If the soft key 28 has been operated in step ST28, the determination is positive, and the imaging process proceeds to step ST30.

[0097] In step ST30, the control unit 93 causes the imaging unit 91 to capture a still image accompanied by a main exposure based on the distance measurement result and the recognition result of the frame with the highest rating value used in the live view image. After that, the imaging process proceeds to step ST32. Note that, although imaging for a still image is illustrated here, the technology of the present disclosure is not limited to this, and imaging for a moving image accompanied by a main exposure may be performed, as long as it is imaging for a recording image as described above.

[0098] In the still image capturing by the imaging unit 91 in step ST30, focusing is performed on the face of a person shown by a specific face image based on the distance measurement result and the recognition result of the frame with the highest rating used in the live view image. Here, the specific face image is, for example, a face image included in the recognition result. If the recognition result includes multiple face images, the face of a person closest to the center of the imaging area or the face of a person with a specific expression (for example, a smile) among the faces of multiple people shown by the multiple face images is selected as the focusing target. In addition, the face of a person specified by the user or the like via the touch panel 48 as the area to be focused among the faces of multiple people displayed on the display 46 surrounded by the frame line 29 may be selected as the focusing target.

[0099] The control unit 93 acquires the distance to the face of the person selected as the focus target from the distance measurement result, and derives the focus position corresponding to the acquired distance. The focus position is derived by the control unit 93 from, for example, a focus position derivation table (not shown) in which distance and focus position are associated with each other, or a focus position derivation arithmetic expression (not shown) in which distance is an independent variable and focus position is a dependent variable. The control unit 93 operates the focus control mechanism 31 to move the focus lens 30B to the derived focus position. Then, the control unit 93 drives the photoelectric conversion element driver 32 to cause the photoelectric conversion element 26 to perform main exposure. A still image is generated by the visible light pixel data processing circuit 34A by performing the main exposure, and the generated still image is stored in the image memory 42 by the visible light pixel data processing circuit 34A. Then, the control unit 93 acquires the still image from the image memory 42, and stores the acquired still image in a memory card (not shown) connected to the external I / F 52.

[0100] In step ST32, the control unit 93 judges whether or not a condition for ending the imaging process (hereinafter also referred to as an "imaging process ending condition") has been satisfied. One example of the imaging process ending condition is that an instruction to end the imaging process has been received via the touch panel 48. In step ST36, if the imaging process ending condition has not been satisfied, the judgment is If the result is negative, the imaging process proceeds to step ST 12. If the imaging process end condition is satisfied in step ST36, the determination is positive, and the imaging process ends.

[0101] As described above, according to the smart device 10 of the first embodiment, the control unit 93 causes the recognition unit 94 to recognize a specific subject (here, as an example, a person's face) included in the imaging area based on a visible light image obtained by imaging the imaging area by the imaging unit 91, and the distance measurement unit 92 to irradiate the imaging area with laser light and receive IR reflected light of the laser light on the imaging area to measure distance, in parallel multiple times. The change unit 96 changes the irradiation energy of the laser light on the imaging area for each distance measurement operation. Therefore, according to this configuration, it is possible to achieve both distance measurement and recognition of the specific subject with high accuracy, compared to a case where image recognition of the specific subject is performed while performing distance measurement using laser light with the same irradiation energy all the time. Note that, as an example of "changing the irradiation energy of the laser light on the imaging area for each distance measurement operation" according to the technology of the present disclosure, an aspect of changing the irradiation energy of the laser light for every distance measurement operation has been described, but the technology of the present disclosure is not limited thereto. For example, "changing the irradiation energy of the laser light on the imaging area for each distance measurement operation" includes cases where the irradiation energy of the laser light is the same in some consecutive or non-consecutive distance measurement operations. Of the multiple frames including distance measurement operations with the same irradiation energy of the laser light, only one of the frames may be set as a frame to be ranked, or all of the frames may be set as frames to be ranked.

[0102] According to the smart device 10 of the first embodiment, the output unit 97 outputs the recognition result and the distance measurement result for the frame with the highest rating value to a specific output destination (here, for example, the live view image storage area 42A and the display 46). Therefore, according to this configuration, the timing at which the recognition result and the distance measurement result were obtained can be easily identified compared to a case in which the recognition result and the distance measurement result are output at different times.

[0103] According to the smart device 10 of the first embodiment, the specific output destination is the display 46 capable of displaying at least one of the recognition result and the distance measurement result. Therefore, according to this configuration, the user can perceive the recognition result obtained by the recognition operation and the distance measurement result obtained by the distance measurement operation.

[0104] According to the smart device 10 of the first embodiment, the display 46 displays a specific subject image showing a specific subject and a frame 29 surrounding the specific subject image as a recognition result. Therefore, according to this configuration, the user can visually understand the recognition result obtained by the recognition operation.

[0105] According to the smart device 10 of the first embodiment, the control unit 93 moves the focus lens 30B to a focus position determined according to the distance measurement result. Therefore, according to this configuration, it is possible to easily achieve focus by the autofocus method compared to the case where focusing is performed by the manual focus method regardless of the distance measurement result.

[0106] According to the smart device 10 of the first embodiment, the processing unit 95 calculates ten rating values ​​from the recognition results and distance measurement results obtained by ten recognition operations and distance measurement operations. Therefore, according to this configuration, it is possible to identify a more balanced combination of the recognition results and distance measurement results than when no rating value is calculated.

[0107] According to the smart device 10 of the first embodiment, the recognition result obtained by the recognition operation is weighted according to the type and / or aspect of the specific subject. Therefore, according to this configuration, it is possible to specify the importance of the recognition result.

[0108] According to the smart device 10 of the first embodiment, the control unit 93 causes the imaging unit 91 to capture an image by main exposure based on a recognition result selected according to a rating value. Therefore, according to this configuration, it is possible to prevent the imaging unit 91 from capturing an image by main exposure based on an inappropriate recognition result, compared to a case where the imaging unit 91 is caused to capture an image by main exposure based on a recognition result randomly selected from a plurality of recognition results obtained by a plurality of recognition operations.

[0109] According to the smart device 10 of the first embodiment, the specific subject is a person's face. Therefore, according to this configuration, it is possible to achieve both distance measurement and face recognition with high accuracy, compared to a case where distance measurement based on directional light that is always irradiated with the same irradiation energy onto the imaging area and face recognition included in the imaging area are performed simultaneously.

[0110] According to the smart device 10 of the first embodiment, the face of the person may be a face with a specific expression. In this case, it is possible to simultaneously perform distance measurement based on laser light that is always irradiated with the same irradiation energy onto the imaging area and recognition of a face with a specific expression included in the imaging area with high accuracy.

[0111] In the above-mentioned first embodiment, a person's face is given as an example of the "specific subject" according to the technology of the present disclosure, but the technology of the present disclosure is not limited thereto. For example, the specific subject may be an object having a reflectance less than a threshold value. Here, the threshold value is, for example, a value derived in advance by sensory testing using an actual device and / or computer simulation as a lower limit value of reflectance at a level at which IR reflected light obtained by irradiating laser light at a preset irradiation energy level R appears as noise in a visible light image. The preset irradiation energy level R refers to, for example, the fifth irradiation energy level R out of the above-mentioned ten irradiation energy levels R. Therefore, according to this configuration, it is possible to achieve both distance measurement and recognition of an object having a reflectance less than the threshold value with high accuracy, compared to a case in which distance measurement based on laser light always irradiated with the same irradiation energy to an imaging area and recognition of an object having a reflectance less than the threshold value are performed simultaneously.

[0112] In the above-described first embodiment, the rating process is given as an example of the "specific process" according to the technology of the present disclosure, but the technology of the present disclosure is not limited thereto. For example, the processing unit 95 may perform a process of outputting at least the recognition result obtained by each recognition operation and the distance measurement result obtained by each distance measurement operation to an external device (not shown) such as a smart device other than the smart device 10, a personal computer, and / or a server via the external I / F 52. In addition, the processing unit 95 may perform a process of creating at least the recognition result obtained by each recognition operation and the distance measurement result obtained by each distance measurement operation and a visible light image as an image file in a specific format (for example, Exif format).

[0113] In the first embodiment, the recognition unit 94 recognizes a person's face as a specific subject, but the specific subject is not limited to a person's face, and may be a person's whole body, or a face with a specific expression (e.g., a smiling face) among a plurality of people's faces. The recognition unit 94 may recognize trees, flowers, historical buildings, and / or characters as specific subjects. In addition, when an object generally recognized as having a high reflectance (e.g., a mirror) is included in the subject, the object generally recognized as having a high reflectance is more likely to reflect laser light and appear as noise in a visible light image than an object that absorbs light, such as a black object, and therefore the recognition unit 94 may recognize only objects having a reflectance less than the above-mentioned threshold as specific subjects.

[0114] The recognition unit 94 may weight the specific subject depending on the type or state of the specific subject. For example, if the recognition unit 94 assigns weights to the faces of people who have been registered in advance and performs the main exposure based on the recognition results, an image focused on the person can be captured even if the person is not in the center of the imaging area. Also, if the recognition unit 94 assigns weights to unusual expressions or actions of animals, it becomes easier to focus on unusual expressions or actions. On the other hand, if the weighting for the expression of a person with closed eyes is set to zero, it becomes harder to focus on a person with closed eyes.

[0115] In the above first embodiment, the recognition result and distance measurement result for the frame with the highest rating (one example of a "specific time" according to the technology of the present disclosure) are output to a specific output destination (for example, the live view image storage area 42A and the display 46), but the technology of the present disclosure is not limited to this. For example, in continuous parallel processing, the recognition result and distance measurement result for the first frame, the tenth frame, and the frame with the highest rating out of 10 frames may be output to a specific output destination. In addition, since continuous parallel processing is performed repeatedly, in continuous parallel processing selected periodically or non-periodically, the recognition result and distance measurement result for at least one specific frame out of 10 frames may be output to a specific output destination.

[0116] [Second embodiment] 12, the smart device 100 according to the second embodiment differs from the smart device 10 according to the first embodiment in that it includes a zoom control mechanism 131. Other configurations of the smart device 100 are the same as those of the smart device 10 according to the first embodiment, so the same components as those described in the first embodiment are denoted by the same reference numerals and descriptions thereof will be omitted.

[0117] The smart device 100 differs from the smart device 10 described in the above embodiment in that it has a distance measuring imaging device 114 instead of the distance measuring imaging device 14. The distance measuring imaging device 114 differs from the distance measuring imaging device 14 in that it has a zoom control mechanism 131. The zoom control mechanism 131 includes a zoom lens 30D, a moving mechanism 160, a motor 162, and a motor driver 164. The zoom lens 30D is supported by the moving mechanism 160 so as to be slidable along the optical axis L2. The motor 162 is connected to the moving mechanism 160 and the motor driver 164. The motor driver 164 is connected to the input / output interface 40, and drives the motor 162 according to an instruction from the CPU 15A. The moving mechanism 160 is connected to a drive shaft (not shown) of the motor 162, and selectively moves the zoom lens 30D to the object side and the image side along the optical axis L2 by receiving power from the motor 162. That is, the CPU 15A controls the driving of the motor 162 via the motor driver 164 to adjust the angle of view of the image.

[0118] Next, the imaging process according to the second embodiment will be described with reference to Fig. 13. The imaging process shown in Fig. 13 differs from the imaging process shown in Fig. 11 in that it includes steps ST40 and ST42. Therefore, steps different from those in the flowchart shown in Fig. 11 will be described below, and steps that are the same as those in the flowchart shown in Fig. 11 will be given the same step numbers and their description will be omitted.

[0119] In the imaging process shown in Fig. 13, when step ST26 is completed, the imaging process proceeds to step ST40. In step ST40, the control unit 93 obtains a distance to a face of a person indicated by a specific face image (e.g., a face image closest to the center in the live view image) included in the frame with the highest rating used in the live view image from the distance measurement result of the frame with the highest rating used in the live view image, and determines whether the obtained distance is outside a preset range. The preset range may be a fixed value, a value that is changed according to an instruction accepted by the accepting device 47, or a value that is changed according to the operation mode of the smart device 10 and / or the imaging scene, etc.

[0120] In step ST40, if the distance to the face of the person represented by the specific facial image is within the predetermined range, the determination is negative, and the imaging process proceeds to step ST28. In step ST40, if the distance to the face of the person represented by the specific facial image is outside the predetermined range, the determination is positive, and the imaging process proceeds to step ST42.

[0121] In step ST42, the control unit 93 controls the motor 162 via the motor driver 164 to move the zoom lens 30D along the optical axis L2 by a specific movement amount so that the angle of view is determined according to the distance to the face of the person shown by the specific facial image. In other words, the control unit 93 moves the zoom lens 30D along the optical axis L2 to a zoom-in position or a zoom-out position on the optical axis L2 that is determined according to the distance measurement result obtained by the distance measurement operation.

[0122] Here, the specific movement amount is, for example, a movement amount derived by the control unit 93 from a movement amount derivation table (not shown) in which distances correspond to the movement amounts of the zoom lens 30D, or a movement amount derivation arithmetic expression (not shown) in which distance is an independent variable and the movement amount of the zoom lens 30D is a dependent variable. When the process of step ST42 ends, the imaging process proceeds to step ST12.

[0123] According to the smart device 100 of the second embodiment, the distance measuring imaging device 114 has a zoom lens 30D that is movable along the optical axis L2. The control unit 93 moves the zoom lens 30D along the optical axis L2 to a zoom-in position or a zoom-out position that is on the optical axis L2 and is determined according to the distance measurement result obtained by the distance measurement operation. Therefore, according to this configuration, it is possible to easily move the zoom lens 30D to a position according to the distance measurement result, compared to the case where the zoom lens 30D is moved manually.

[0124] In the second embodiment, the control unit 93 moves the zoom lens 30D in response to a distance measurement result obtained by a distance measurement operation, but the technology of the present disclosure is not limited to this. For example, the control unit 93 may move the zoom lens 30D in response to a zoom operation accepted by the accepting device 47.

[0125] In this case, for example, the imaging process shown in Fig. 14 is executed by the CPU 15A. The imaging process shown in Fig. 14 is different from the imaging process shown in Fig. 13 in that it has a process of step ST140 instead of the process of step ST40, and has a process of step ST142 instead of the process of step ST42.

[0126] 14, in step ST140, the control unit 93 determines whether or not a zoom operation has been performed on the receiving device 47. Examples of the zoom operation include a pinch-in operation and a pinch-out operation on the touch panel 48. For example, when zooming in on a person's face shown by a face image surrounded by a frame line 29, a pinch-out operation is performed on the touch panel 48 on the face image, and when zooming out from the person's face, a pinch-out operation is performed on the touch panel 48.

[0127] In step ST140, if a zoom operation has been performed on the receiving device 47, the determination is affirmative, and the imaging process proceeds to step ST142. In step ST140, if a zoom operation has not been performed on the receiving device 47, the determination is negative, and the imaging process proceeds to step ST28.

[0128] In step ST142, the control unit 93 controls the motor 162 via the motor driver 164 to move the zoom lens 30D along the optical axis L2 by an amount of movement corresponding to the zoom operation. After that, the imaging process proceeds to step ST12. By executing the process of step ST142, the angle of view is changed.

[0129] [Third embodiment] 8, the smart device 200 according to the third embodiment has the same configuration as the smart device 10 according to the first embodiment. The following describes the imaging process of the smart device 200 according to the third embodiment, focusing on differences from the imaging process of the smart device 10 according to the first embodiment.

[0130] As an example, as shown in FIG. 15, the imaging process according to the third embodiment differs from the imaging process shown in FIG. 11 in that steps ST50 and ST52 are provided between steps ST10 and ST12.

[0131] In the imaging process shown in FIG. 15, in step ST50, the control unit 93 performs distance measurement for focus control. By executing this step ST50, the light irradiator 16 irradiates laser light toward all or a part (for example, the center) of the imaging area, and the distance measurement division area 26N2 receives the IR reflected light. The pixel data obtained by receiving the IR reflected light by the distance measurement division area 26N2 is sent to the IR pixel data processing circuit 34B, and the distance from the smart device 200 to all or a part of the imaging area is measured based on the time required from the irradiation of the laser light to the reception of the IR reflected light and the speed of light. The distance measurement result obtained in this manner is stored in the image memory 42 by the IR pixel data processing circuit 34B.

[0132] In the next step ST52, the control unit 93 moves the focus lens 30B to a focus position determined according to the distance measurement result stored in the image memory 42 by executing step ST50. The focus position is derived by the control unit 93 from a focus position derivation table (not shown) or a focus position derivation arithmetic expression (not shown), similarly to the first embodiment.

[0133] According to the smart device 200 of the third embodiment, the distance measuring imaging device 14 (imaging section) has a focus lens 30B that can move along the optical axis L2. The distance measuring imaging device 14 (distance measuring section) performs distance measurement for focus control by irradiating the imaging area with laser light and receiving IR reflected light from the imaging area prior to the recognition operation. The control section 93 moves the focus lens 30B along the optical axis L2 to a focus position determined according to the distance measurement result obtained by the distance measurement for focus control. Therefore, according to this configuration, since the focus lens 30B is moved to the focus position before the recognition operation, the smart device 200 can improve the image quality of the visible light image obtained by performing parallel processing compared to a case where the focus lens 30B is not moved to the focus position before the recognition operation. As a result, the accuracy of image recognition for the visible light image obtained by performing parallel processing can be improved compared to a case where the focus lens 30B is not moved to the focus position before the recognition operation.

[0134] [Other embodiments] In each of the above embodiments, an example is described in which IR reflected light is received by the distance measurement partition area 26N2, but the technology disclosed herein is not limited to this, and IR reflected light may be received by a portion of the IR pixels within the distance measurement partition area 26N2.

[0135] In this case, for example, as shown in FIG. 16, the control unit 93 causes the display 46 to display a distance measurement range designation guidance screen 102. The distance measurement range designation guidance screen 102 is a screen that guides the user, etc., in designating a distance measurement range. The distance measurement range designation guidance screen 102 displays a message (hereinafter also referred to as a "guidance message") that asks the user whether or not to designate a distance measurement range. In the example shown in FIG. 16, the message "Do you want to designate a distance measurement range?" is displayed as an example of the guidance message. Also, the distance measurement range designation guidance screen 10 2, soft keys 102A and 102B are displayed. When the user or the like specifies the distance measurement range, the soft key 102A is turned on by the user or the like via the touch panel 48. When the user or the like does not specify the distance measurement range, that is, when the entire imaging area is to be the distance measurement range, the soft key 102B is turned on by the user or the like via the touch panel 48.

[0136] When the user or the like turns on the soft key 102A via the touch panel 48 while the ranging range designation guide screen 102 is displayed on the display 46, the control unit 93 causes the display 46 to display a visible light image as a live view image, as shown in Fig. 17 as an example. With the live view image displayed on the display 46, an image area (a rectangular area surrounded by a dashed line on the visible light image in the example shown in Fig. 17) is designated by the user or the like via the touch panel 48. A real space area corresponding to the image area designated by the user or the like (hereinafter also referred to as a "designated image area") is designated as a target for ranging by the ranging imaging device 14.

[0137] 18, area position specifying information (e.g., coordinates) capable of specifying the position of the designated image area on the visible light image is output from the touch panel 48 to the control unit 93. The control unit 93 outputs partition area position information (e.g., pixel address) capable of specifying the position of the designated partition area for distance measurement 26N2a in the partition area for distance measurement 26N2 to the photoelectric conversion element driver 32. Here, the designated partition area for distance measurement 26N2a refers to a partition area in the partition area for distance measurement 26N2 at a position corresponding to the position of the designated image area specified by the area position specifying information input from the touch panel 48.

[0138] In the distance measurement operation, the photoelectric conversion element driver 32 drives only the designated distance measurement segment area 26N2a of the distance measurement segment area 26N2, thereby causing the distance measurement imaging devices 14 and 114 to perform distance measurement using IR reflected light received only by the designated distance measurement segment area 26N2a. In other words, distance measurement is performed using IR reflected light received only by at least one IR pixel included in a designated area among the multiple IR pixels.

[0139] 16 to 18, the distance measurement section 26N2 is changed to the distance measurement designated section 26N2a so that the distance measurement is performed by focusing on the distance measurement target (so-called ROI) designated by the user, etc., but the method of focusing on the distance measurement target is not limited to this. For example, the control unit 93 may control to change the beam diameter and / or direction of the laser light according to an instruction accepted by the accepting device 47 so that the laser light is irradiated on the distance measurement target designated by the user, etc.

[0140] In the above-described embodiments, the smart devices 10, 100, and 200 equipped with the light receiver 18 are illustrated, but the technology of the present disclosure is not limited thereto. For example, as shown in FIG. 19, a smart device 300 equipped with the light receivers 18 and 350 may be used. As an example, as shown in FIG. 19, a light-transmitting window 352 is provided adjacent to the light-transmitting window 22 in the upper left part of the back surface 12A of the housing 12 when the smart device 300 is placed vertically (upper left part when viewed from the back surface of the smart device 300 placed vertically). The light-transmitting window 352 is an optical element (e.g., a lens) having translucency, similar to the light-transmitting windows 20 and 22, and the light-transmitting windows 20, 22, and 352 are arranged at a predetermined interval along the horizontal direction. The light-transmitting window 352 is also exposed from the back surface 12A, similar to the light-transmitting windows 20 and 22.

[0141] The receiver 350 includes a photoelectric conversion element 354. The photoelectric conversion element 354 is a photoelectric conversion element specialized for receiving reflected IR light, and has a plurality of IR pixels arranged in a matrix. One example of the plurality of IR pixels is a photodiode for IR pixels (e.g., InGaAs APD) of "4896 x 3265" pixels. The photoelectric conversion element 354 receives the reflected IR light taken into the receiver 350 through the light-transmitting window 352, and outputs the received light. The reflected IR light is reflected by the reflector 34. ... An electrical signal according to the amount of the reflected IR light is output to the signal processing circuit 34 (see FIG. 8).

[0142] In addition, in the example shown in FIG. 19, the smart device 300 equipped with the light receivers 18 and 350 is shown, but the technology of the present disclosure is not limited thereto. For example, as shown in FIG. 20, the smart device 400 equipped with the light receivers 18 and 450 may be used. As an example, as shown in FIG. 20, a light-transmitting window 452 is provided adjacent to the light-transmitting window 20 in the upper right part of the back surface 12A of the housing 12 when the smart device 400 is placed vertically (the upper right part when viewed from the back surface of the smart device 400 placed vertically). The light-transmitting window 452 is an optical element (for example, a lens) having translucency, similar to the light-transmitting windows 20 and 22, and the light-transmitting windows 452, 20, and 22 are arranged at a predetermined interval along the horizontal direction. The light-transmitting window 452 is also exposed from the back surface 12A, similar to the light-transmitting windows 20 and 22.

[0143] The optical receiver 450 includes a single photodiode 454. The photodiode 454 is, for example, a photodiode capable of receiving reflected IR light. An example of the photodiode 454 is an InGaAs APD. The photodiode 454 receives the reflected IR light taken into the optical receiver 450 through the light-transmitting window 452, and outputs an electrical signal according to the amount of the received reflected IR light to the signal processing circuit 34 (see FIG. 8).

[0144] In each of the above embodiments, laser light has been described as an example of "light" according to the technology of the present disclosure, but the technology of the present disclosure is not limited to this, and superluminescent light may be used instead of laser light, and distance measurement may be performed using light having a directionality that allows distance measurement.

[0145] In the above-described embodiments, the distance measuring imaging device 14 is built into the smart device 10, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 21, the distance measuring imaging device 14 may be externally attached to a general smart device 500, that is, a smart device 500 that does not have a built-in distance measuring imaging device 14.

[0146] In addition, in each of the above embodiments, an example in which the UI device 44 is incorporated in the smart device 10 has been described, but at least some of the components included in the UI device 44 may be externally attached to the smart device 10. In addition, at least some of the components included in the UI device 44 may be used by being connected to the external I / F 52 as separate entities.

[0147] 1 illustrates a smart device 10, but the technology of the present disclosure is not limited thereto. That is, the technology of the present disclosure can be applied to various electronic devices (e.g., interchangeable lens cameras, fixed lens cameras, personal computers, and / or wearable terminal devices) incorporating a distance measuring imaging device 14, and the same actions and effects as those of the smart device 10 can be obtained with these electronic devices.

[0148] In addition, in each of the above embodiments, the display 46 is exemplified, but the technology of the present disclosure is not limited to this. For example, a separate display attached to the smart device 10 may be used as the "display unit" according to the technology of the present disclosure.

[0149] In the above embodiment, the recognition unit 94 is mounted on the smart device 10, but the technology of the present disclosure is not limited to this. For example, an external device (e.g., another smart device, a personal computer, and / or a server) having the recognition unit 94 may be connected to the smart device 10. In this case, the image recognition result provided to the smart device 10 from the external device may be acquired by the CPU 15A of the smart device 10. In addition, in the case of cloud computing (FIG. The function of the recognition unit 94 may be performed by a cloud computing system (not shown) so that the image recognition result is provided to the smart device 10 from the cloud computing system. In this case, the image recognition result provided to the smart device 10 from the cloud computing system may be acquired by the CPU 15A of the smart device 10.

[0150] In addition, in each of the above-described embodiments, the imaging processing program 70 is stored in the storage 15B, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 22, the imaging processing program 70 may be stored in a storage medium 900. An example of the storage medium 900 is any portable storage medium such as an SSD or a USB memory.

[0151] The imaging processing program stored in the storage medium 900 is installed in the controller 15. The CPU 15A executes imaging processing in accordance with the imaging processing program .

[0152] In addition, the imaging processing program 70 may be stored in a memory unit of another computer or server device connected to the controller 15 via a communication network (not shown), and the imaging processing program 70 may be downloaded and installed in the controller 15 in response to a request from the smart device 10 described above.

[0153] It is not necessary to store the entire image capture processing program 70 in a storage unit of another computer or server device connected to the controller 15, or in the storage 15B; only a part of the image capture processing program 70 may be stored therein.

[0154] In the example shown in FIG. 22, an example is shown in which the controller 15 is built into the smart device 10, but the technology of the present disclosure is not limited to this, and for example, the controller 15 may be provided outside the smart device 10.

[0155] 22, the CPU 15A is a single CPU, but it may be a plurality of CPUs. Also, a GPU may be applied in place of the CPU 15A.

[0156] 22 illustrates a controller 15, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the controller 15. Also, instead of the controller 15, a combination of a hardware configuration and a software configuration may be used.

[0157] As the hardware resource for executing the imaging process described in each of the above embodiments, the following various processors can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing imaging processes by executing software, i.e., a program. Another example of the processor is a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically for executing specific processes, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor uses the memory to execute imaging processes.

[0158] The hardware resource for executing the imaging process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the imaging process may be a single processor.

[0159] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the imaging process. Second, as typified by SoC, a processor that realizes the functions of the entire system including multiple hardware resources that execute the imaging process in one IC chip is used. In this way, the imaging process is realized using one or more of the above various processors as hardware resources.

[0160] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above-mentioned imaging process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the invention.

[0161] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0162] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0163] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0164] The following supplementary notes are further disclosed regarding the above embodiment.

[0165] (Additional Note) A processor; A memory connected to or embedded in the processor; The processor is performing control to perform a recognition operation in which a recognition unit recognizes a specific subject included in an imaging area based on an image obtained by imaging an imaging area with an imaging unit, and a distance measurement operation in which a distance measurement unit irradiates the imaging area with light and receives reflected light of the light reflected by the imaging area, in parallel multiple times; and A process including changing the irradiation energy of the light on the imaging area is executed for each distance measurement operation. Processing unit.

Claims

1. A processor; A memory connected to or embedded in the processor, The processor, a recognition processor that recognizes a specific subject included in an imaging area based on an image captured by an imaging device at a preset frame rate, and a distance measurement device that performs a distance measurement by irradiating light onto the imaging area and receiving light reflected by the light onto the imaging area, the recognition processor performing a recognition operation multiple times in parallel; In the recognition operation and the distance measurement operation that are performed in parallel, a recognition result from the recognition operation and a distance measurement result from the distance measurement operation are integrated; In the distance measurement operation, the irradiation energy of the light for the different imaging regions is changed in multiple steps in accordance with the frame rate. Processing unit.

2. The processor outputs the recognition result obtained by the recognition operation and the distance measurement result obtained by the distance measurement operation to a specific output destination in accordance with a specific time out of the multiple times. The processing device of claim 1 .

3. The specific output destination is a display capable of displaying at least one of the recognition result and the distance measurement result. The processing device according to claim 2 .

4. The display displays a specific subject image showing the specific subject and an image surrounding the specific subject image as the recognition result. The processing device according to claim 3 .

5. The imaging device has a lens that is movable along an optical axis, The processor moves the lens along the optical axis to a position on the optical axis that is determined according to a distance measurement result obtained by the distance measurement operation. The processing apparatus according to any one of claims 1 to 4.

6. The position is a focus position. The processing device according to claim 5 .

7. The imaging device has a lens that is movable along an optical axis, the distance measuring device performs distance measurement for focus control by irradiating the imaging area with focus control light and receiving focus control reflected light due to the focus control light on the imaging area prior to the recognition operation; The processor moves the lens along the optical axis to a focus position determined in accordance with a result of the focus control distance measurement. The processing device according to any one of claims 1 to 6.

8. The processor performs a specific process using a plurality of recognition results obtained by the plurality of times of the recognition operation. The processing device according to any one of claims 1 to 7.

9. The specific processing is a processing in which the recognition result is output to the outside. The processing device of claim 8.

10. The specific processing is processing in which the recognition result and the captured image are created as a file in a specific format. The processing device of claim 8.

11. A processor; A memory connected to or embedded in the processor, The processor, a recognition processor that recognizes a specific subject included in an imaging area based on an image captured by an imaging device, and a distance measurement device that performs a distance measurement by irradiating light onto the imaging area and receiving light reflected by the light onto the imaging area, the recognition processor performing a recognition operation multiple times in parallel; In the recognition operation and the distance measurement operation that are performed in parallel, a recognition result from the recognition operation and a distance measurement result from the distance measurement operation are integrated; A distance measurement result obtained by the distance measurement operation is weighted according to the irradiation energy of the light on the imaging area that differs in the distance measurement operation. Processing unit.

12. The weight of the distance measurement result when the irradiation energy is a first irradiation energy is greater than the weight of the distance measurement result when the irradiation energy is a second irradiation energy smaller than the first irradiation energy. The processing device of claim 11.

13. A processor; A memory connected to or embedded in the processor, The processor, a recognition processor that recognizes a specific subject included in an imaging area based on an image captured by an imaging device, and a distance measurement device that performs a distance measurement by irradiating light onto the imaging area and receiving light reflected by the light onto the imaging area, the recognition processor performing a recognition operation multiple times in parallel; In the recognition operation and the distance measurement operation that are performed in parallel, a recognition result from the recognition operation and a distance measurement result from the distance measurement operation are integrated; The specific subject is a face. Processing unit.

14. The face is a face with a particular expression. The processing device of claim 13.

15. A processor; A memory connected to or embedded in the processor, The processor, a recognition processor that recognizes a specific subject included in an imaging area based on an image captured by an imaging device, and a distance measurement device that performs a distance measurement by irradiating light onto the imaging area and receiving light reflected by the light onto the imaging area, the recognition processor performing a recognition operation multiple times in parallel; In the recognition operation and the distance measurement operation that are performed in parallel, a recognition result from the recognition operation and a distance measurement result from the distance measurement operation are integrated; The specific object is an object having a reflectance less than a threshold value. Processing unit.

16. A processing device according to any one of claims 1 to 15; and at least one of the recognition processor and the ranging device. electronic equipment.

17. a control for performing a recognition operation in which a recognition processor recognizes a specific subject included in an imaging area based on an image captured by an imaging device at a preset frame rate, and a distance measurement operation in which a distance measurement device irradiates light onto the imaging area and measures distance by receiving light reflected by the light onto the imaging area, in parallel multiple times; In the recognition operation and the distance measurement operation that are performed in parallel, a recognition result from the recognition operation and a distance measurement result from the distance measurement operation are integrated; and and changing the irradiation energy of the light for the different imaging regions in the distance measuring operation in multiple steps in accordance with the frame rate. Processing methods.

18. On the computer, a control for performing a recognition operation in which a recognition processor recognizes a specific subject included in an imaging area based on an image captured by an imaging device at a preset frame rate, and a distance measurement operation in which a distance measurement device irradiates light onto the imaging area and measures distance by receiving light reflected by the light onto the imaging area, in parallel multiple times; In the recognition operation and the distance measurement operation that are performed in parallel, a recognition result from the recognition operation and a distance measurement result from the distance measurement operation are integrated; and A program for executing a process including changing the irradiation energy of the light for the different imaging regions in multiple stages in accordance with the frame rate during the distance measurement operation.

Images