Processing device, electronic device, processing method, and storage medium

Abstract

The processing device of the present application includes: a control section that performs control to perform a plurality of times in parallel a recognition operation in which a recognition section recognizes a specific subject included in an imaging region based on an imaging image obtained by imaging the imaging region by an imaging section, and a distance measurement operation in which distance measurement is performed by a distance measurement section irradiating the imaging region with light and receiving reflected light based on the light with respect to the imaging region; and a change section that changes the irradiation energy of the light with respect to the imaging region for each distance measurement operation.

Description

Technical Field

[0001] The technology of the present invention relates to a processing device, an electronic device, a processing method and a storage medium. Background Art

[0002] Japanese Patent No. 6321145 discloses a distance measuring device. The distance measuring device described in Japanese Patent No. 6321145 includes: a photographing unit that photographs a subject image formed by an imaging optical system that images a subject image representing the subject; an emitting unit that emits directional light as directional light along the optical axis direction of the imaging optical system, and is capable of adjusting the emission intensity of the directional light, and emits the directional light by adjusting the emission intensity according to at least one of focus state determination information, subject brightness, or exposure state determination information; a light receiving unit that receives reflected light of the directional light from the subject; a deriving unit that derives the distance to the subject based on the timing of emitting the directional light by the emitting unit and the timing of receiving the reflected light by the light receiving unit; an executing unit that performs at least one of focusing and exposure adjustment of the imaging optical system on the subject before photography by the photographing unit; and a receiving unit that receives a first pressing operation and a photography preparation instruction. The control unit is configured to control the execution of at least one of the focus adjustment and the exposure adjustment by the execution unit upon receiving the first press operation through the receiving unit, and to control the start of distance measurement based on the emitting unit, the light receiving unit, and the derivation unit on the condition that at least one of the focus adjustment and the exposure adjustment has been performed. After the distance measurement is completed, the prompting unit for prompting information related to the result of the distance measurement is prompted while the first press operation is maintained. Then, when the second press operation performed subsequent to the first press operation is received through the receiving unit without releasing the press operation on the receiving unit, the formal exposure control based on the photographic unit is performed.

[0003] Japanese Patent Application Laid-Open No. 2006-171120 discloses a photographic device. The photographic device described in Japanese Patent Application Laid-Open No. 2006-171120 detects subject contrast and adjusts focus before a photographic operation. The device is characterized by comprising: a first auxiliary light emitter that irradiates light having a relatively wide wavelength range toward the subject for subject contrast detection; a second auxiliary light emitter that irradiates light having a relatively narrow wavelength range toward the subject for subject contrast detection; and a switching mechanism that switches between the first auxiliary light emitter and the second auxiliary light emitter.

[0004] International Publication No. 2018 / 142993 discloses a light control device. The light control device described in International Publication No. 2018 / 142993 includes a light intensity setting unit for setting the light intensity of AF (auto focus) auxiliary light, and a light control unit for controlling the emission of the AF auxiliary light in accordance with the setting of the light intensity setting unit. Summary of the Invention

[0005] An embodiment of the technology involved in the present invention provides a processing device, electronic device, processing method and program that can achieve both ranging and identification of specific subjects with high precision, compared to a case where ranging based on light that always irradiates the camera area with the same irradiation energy and identification of specific subjects contained in the camera area are simultaneously performed.

[0006] Means for solving technical problems

[0007] A first embodiment of the technology of the present invention is a processing device comprising: a control unit that controls a plurality of parallel executions of a recognition operation and a ranging operation, wherein the recognition operation comprises recognizing a specific subject contained in a camera area based on a captured image obtained by capturing the camera area by a camera unit; and the ranging operation comprises measuring distance by irradiating the camera area with light by the ranging unit and receiving light reflected from the light directed to the camera area; and a changing unit that changes the amount of light irradiated onto the camera area for each ranging operation.

[0008] A second aspect of the present invention is the processing device according to the first aspect, further comprising: outputting a recognition result obtained by the recognition operation and a distance measurement result obtained by the distance measurement operation to a specific output destination according to a specific number of times among the multiple times.

[0009] A third aspect according to the technology of the present invention is the processing device according to the second aspect, wherein the specific output target is a display unit capable of displaying at least one of a recognition result and a distance measurement result.

[0010] A fourth aspect according to the technology of the present invention is the processing device according to the third aspect, wherein the display unit displays a specific subject image showing the specific subject and an image surrounding the specific subject image as a recognition result.

[0011] The fifth embodiment of the technology of the present invention is a processing device involved in any one of the first to fourth embodiments, wherein the camera unit has a lens that can move along the optical axis, and the control unit moves the lens along the optical axis to a position on the optical axis, which is a position determined based on a ranging result obtained by a ranging action.

[0012] A sixth aspect according to the technology of the present invention is the processing device according to the fifth aspect, wherein the position is a focus position.

[0013] The seventh mode involved in the technology of the present invention is a processing device involved in any one of the first to fourth modes, wherein the camera unit has a lens that can move along the optical axis, and the ranging unit performs focus control ranging by irradiating the camera area with focus control light and receiving focus control reflected light based on the focus control light for the camera area before recognizing the action, and the control unit causes the lens to move along the optical axis to a focus position determined according to the ranging result obtained by the focus control ranging.

[0014] An eighth aspect of the technology of the present invention is the processing device according to any one of the first to seventh aspects, further comprising a processing unit that performs a determination process using a plurality of recognition results obtained through a plurality of recognition operations.

[0015] The ninth aspect of the technology of the present invention is a processing device according to any one of the first to eighth aspects, wherein a recognition result obtained by a recognition action is weighted according to at least one of a specific subject and an irradiation energy used in a ranging action parallel to the recognition action.

[0016] A tenth aspect according to the technology of the present invention is the processing device according to the ninth aspect, wherein the control unit causes the imaging unit to perform imaging by main exposure based on the recognition result selected according to the weighting.

[0017] An eleventh aspect according to the technology of the present invention is the processing device according to any one of the first to tenth aspects, wherein the specific subject is a face.

[0018] A twelfth aspect according to the technology of the present invention is the processing device according to the eleventh aspect, wherein the face is a face with a specific expression.

[0019] A thirteenth aspect according to the technology of the present invention is the processing device according to any one of the first to tenth aspects, wherein the specific subject is an object having a reflectance smaller than a threshold value.

[0020] A fourteenth aspect according to the technology of the present invention is an electronic device including: the processing device according to any one of the first to thirteenth aspects; and at least one of a recognition unit and a distance measurement unit.

[0021] The fifteenth method involved in the technology of the present invention is a processing method comprising the following steps: controlling a recognition action and a ranging action to be performed multiple times in parallel, wherein in the recognition action, the recognition unit recognizes a specific subject included in the imaging area based on a camera image obtained by capturing the imaging area by the imaging unit; in the ranging action, the ranging unit performs ranging by irradiating the imaging area with light and receiving reflected light based on the light directed to the imaging area; and changing the irradiation energy of the light directed to the imaging area according to each ranging action.

[0022] A sixteenth method according to the present invention is a program for causing a computer to execute processing including the following steps: controlling a recognition operation and a ranging operation to be performed a plurality of times in parallel, wherein the recognition operation comprises a recognition unit recognizing a specific subject contained in a photographic area based on a photographic image obtained by photographing the photographic area by a photographic unit; and performing ranging by irradiating the photographic area with light by the ranging unit and receiving reflected light based on the light directed to the photographic area; and changing the irradiation energy of the light directed to the photographic area for each ranging operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic perspective view showing an example of how the smart device according to the first embodiment is used.

[0024] Figure 2 This is a rear perspective view showing an example of the appearance of the rear side of the smart device according to the first embodiment.

[0025] Figure 3 This is a schematic perspective view showing an example of the structure of a photoelectric conversion element included in the smart device according to the first embodiment.

[0026] Figure 4 Yes Figure 2 A front perspective view showing an example of the appearance of the front surface side of the smart device shown.

[0027] Figure 5 This is a conceptual diagram showing an example of a method in which an imaging area is imaged by the ranging imaging device included in the smart device according to the first embodiment, and a visible light image is displayed on a display.

[0028] Figure 6 This is a conceptual diagram showing an example of a method in which a distance measurement operation and a recognition operation are performed by the distance measurement camera included in the smart device according to the first embodiment, and a live view image is displayed on a display.

[0029] Figure 7 This is a conceptual diagram showing an example of a method of capturing a still image using the smart device according to the first embodiment.

[0030] Figure 8 This is a block diagram showing an example of the configuration of electrical system hardware of the smart device according to the first embodiment.

[0031] Figure 9 This is a conceptual diagram showing an example of reading an imaging processing program when the CPU included in the smart device according to the first embodiment executes imaging processing.

[0032] Figure 10 This is a functional block diagram showing an example of functions of the CPU and the distance measurement imaging device when the CPU included in the smart device according to the first embodiment performs imaging processing.

[0033] Figure 11 This is a flowchart showing an example of the flow of imaging processing according to the first embodiment.

[0034] Figure 12 This is a block diagram showing an example of the configuration of electrical system hardware of a smart device according to the second embodiment.

[0035] Figure 13 This is a flowchart showing an example of the flow of imaging processing according to the second embodiment.

[0036] Figure 14 This is a flowchart showing a modified example of the flow of the imaging process according to the second embodiment.

[0037] Figure 15 This is a flowchart showing an example of the flow of imaging processing according to the third embodiment.

[0038] Figure 16 This is a schematic screen diagram showing an example of a distance measurement range specification guidance screen displayed on a display included in a smart device according to another embodiment.

[0039] Figure 17 This is a schematic screen diagram showing an example of a visible light image and a designated image area displayed as a live view image on a display included in a smart device according to another embodiment.

[0040] Figure 18 This is a conceptual diagram for explaining an example of a method of reducing a distance-measuring segmented area to a distance-measuring designated segmented area in a photoelectric conversion element included in a smart device according to another embodiment.

[0041] Figure 19 It is a schematic perspective view showing a first modified example of the external structure of a smart device according to another embodiment.

[0042] Figure 20It is a schematic perspective view showing a second modified example of the external structure of a smart device according to another embodiment.

[0043] Figure 21 This is a schematic perspective view showing an example of an external configuration of a combination of a distance-measuring imaging device and a smart device according to another embodiment.

[0044] Figure 22 This is a conceptual diagram showing an example of how an image capture processing program is installed in a smart device. DETAILED DESCRIPTION

[0045] Hereinafter, an example of an embodiment of a distance measuring imaging device according to the technology of the present invention will be described with reference to the drawings.

[0046] First, the terms used in the following description are explained.

[0047] CPU stands for Central Processing Unit. RAM stands for Random Access Memory. ASIC stands for Application Specific Integrated Circuit. PLD stands for Programmable Logic Device. FPGA stands for Field-Programmable Gate Array. SoC stands for System-on-a-chip. SSD stands for Solid State Drive. USB stands for Universal Serial Bus. HDD stands for Hard Disk Drive. EEPROM stands for Electrically Erasable and Programmable Read Only Memory. EL stands for Electro-Luminescence. A / D stands for Analog / Digital. I / F stands for "Interface." UI stands for "User Interface." LTE stands for "Long Term Evolution." 5G stands for "5th Generation." LD stands for "Laser Diode." IR stands for "Infrared." APD stands for "Avalanche Photodiode." fps stands for "frame per second." LED stands for "Light Emitting Diode." ROI stands for "Region of Interest." LAN stands for "Local Area Network." Exif stands for "Exchangeable Image File Format."

[0048] In the description of this specification, "horizontal" means not only completely horizontal but also horizontal in the sense of including the error generally allowed in the technical field to which the technology of the present invention belongs. In the description of this specification, "parallel" means not only completely parallel but also parallel in the sense of including the error generally allowed in the technical field to which the technology of the present invention belongs. In the description of this specification, "perpendicular" means not only completely perpendicular but also perpendicular in the sense of including the error generally allowed in the technical field to which the technology of the present invention belongs. In the description of this specification, "identical" means not only completely identical but also identical in the sense of including the error generally allowed in the technical field to which the technology of the present invention belongs. Furthermore, in the description of this specification, a numerical range expressed with "to" indicates a range that includes the numerical values ​​recorded before and after "to" as the lower limit and upper limit.

[0049] [First embodiment]

[0050] As an example, Figure 1 As shown, the smart device 10 according to the first embodiment performs recognition and ranging operations. The recognition operation identifies a specific subject contained in the imaging area based on an image obtained by capturing the imaging area defined by a field of view angle θ1. The ranging operation measures the distance by irradiating the imaging area with a laser beam and receiving light reflected from the laser beam. The laser beam is an example of "light" within the technology of the present invention.

[0051] In this embodiment, “distance measurement” refers to the process of measuring the distance from the smart device 10 to the distance measurement object within the imaging area. Here, the “distance measurement object” refers to an object that reflects light. Figure 1 In the example shown, a person and a tree are shown as distance measurement targets within the imaging area. In addition, examples of the smart device 10 include a smartphone or a tablet terminal, which is an electronic device with an imaging function.

[0052] As an example, Figure 2 As shown, the smart device 10 includes a housing 12. Housing 12 houses a distance-measuring camera 14. Distance-measuring camera 14 includes a light irradiator 16 and a light receiver 18. Light irradiator 16 includes an LD 24, and light receiver 18 includes a photoelectric conversion element 26. In the smart device 10, imaging and distance-measuring operations are performed by distance-measuring camera 14. Distance-measuring camera 14 is an example of the "imaging unit (imaging device)" and "distance-measuring unit (distance-measuring device)" involved in the technology of the present invention.

[0053] A pointer key 13 is provided on the side of the smart device 10. The pointer key 13 receives various instructions. The "various instructions" mentioned here include, for example, instructions for displaying a menu screen from which various menus can be selected, instructions for selecting one or more menus, instructions for confirming selected content, and instructions for deleting selected content.

[0054] Light-transmitting windows 20 and 22 are provided on the upper portion of the back surface 12A of the frame 12 when the smart device 10 is set to a vertical position (the upper portion when the smart device 10 in the vertical position is viewed from the rear). The light-transmitting windows 20 and 22 are optical elements (e.g., lenses) with light transmittance, which are arranged at predetermined intervals (e.g., intervals of several millimeters) in the horizontal direction and exposed from the back surface 12A. The light irradiator 16 irradiates the laser beam emitted from the LD 24 onto the distance measurement object through the light-transmitting window 20. In this embodiment, a laser beam in the infrared wavelength region is used. However, the wavelength region of the laser beam is not limited thereto, and a laser beam in another wavelength region may also be used.

[0055] The light receiver 18 receives IR reflected light through the light-transmitting window 22. IR reflected light refers to light reflected from the laser beam irradiated by the light irradiator 16 onto the object being measured. Furthermore, the light receiver 18 receives visible reflected light through the light-transmitting window 22. Visible reflected light refers to light reflected from visible light irradiating the imaging area (e.g., visible light contained in sunlight). For ease of explanation, the following description will simply refer to "reflected light" when there is no need to distinguish between IR reflected light and visible reflected light.

[0056] The light receiver 18 includes a photoelectric conversion element 26 . The photoelectric conversion element 26 receives the reflected light from the light receiver 18 via the light-transmitting window 22 and outputs an electrical signal according to the amount of the received reflected light.

[0057] As an example, Figure 3 As shown, the photoelectric conversion element 26 includes a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896×3265" pixels.

[0058] Each photodiode included in the photoelectric conversion element 26 is equipped with a color filter. The color filters include a G filter corresponding to the G (green) wavelength region, which is most helpful in obtaining a luminance signal; an R filter corresponding to the R (red) wavelength region; a B filter corresponding to the B (blue) wavelength region; and an IR filter corresponding to the IR (infrared) wavelength region. Furthermore, in this embodiment, the G filter, the R filter, and the B filter also function as infrared light cutoff filters that cut off infrared light. For ease of explanation, the G filter, the R filter, and the B filter will be referred to as "visible light filters" when no distinction is made between them.

[0059] The photoelectric conversion element 26 has R pixels, G pixels, B pixels, and IR pixels. R pixels are pixels corresponding to photodiodes equipped with R filters, G pixels are pixels corresponding to photodiodes equipped with G filters, B pixels are pixels corresponding to photodiodes equipped with B filters, and IR pixels are pixels corresponding to photodiodes equipped with IR filters. R pixels, G pixels, B pixels, and IR pixels are arranged in a predetermined periodic pattern in the row direction (horizontal direction) and the column direction (vertical direction), respectively. In this embodiment, the arrangement of R pixels, G pixels, B pixels, and IR pixels is an arrangement obtained by replacing a portion of the G pixels in the X-Trans (registered trademark) arrangement with IR pixels. The IR pixels are arranged in a specific periodic pattern along the row direction and the column direction.

[0060] In addition, here, as an example of the arrangement of R pixels, G pixels, B pixels and IR pixels, an arrangement based on the X-Trans arrangement is illustrated, but the technology of the present invention is not limited to this. The arrangement of R pixels, G pixels, B pixels and IR pixels can also be an arrangement based on other arrangements such as the Bayer arrangement or the Honeycomb (registered trademark) arrangement.

[0061] Furthermore, while an arrangement of R pixels, G pixels, and B pixels is illustrated here as an arrangement obtained by replacing some of the G pixels in a conventionally known arrangement with IR pixels, the technology of the present invention is not limited to this arrangement. For example, each color filter corresponding to each of the R pixels, G pixels, and B pixels (hereinafter also referred to as "visible light pixels") can be a color filter that also transmits infrared light, and a pair of photodiodes, including a visible light pixel photodiode and an IR pixel photodiode (e.g., an InGaAs APD), can be arranged in one color filter.

[0062] In this embodiment, the photoelectric conversion element 26 is divided into two areas. Specifically, the photoelectric conversion element 26 includes a visible light image segmentation area 26N1 and a distance measurement segmentation area 26N2. The visible light image segmentation area 26N1 is a visible light pixel group based on multiple visible light pixels and is used to generate visible light images. The distance measurement segmentation area 26N2 is an IR pixel group based on multiple IR pixels and is used for distance measurement. The visible light image segmentation area 26N1 receives visible reflected light and outputs an electrical signal corresponding to the amount of light received. The distance measurement segmentation area 26N2 receives IR reflected light and outputs an electrical signal corresponding to the amount of light received.

[0063] As an example, Figure 4As shown, a touch panel display 59 is provided on the front surface 12B of the housing 12. The touch panel display 59 includes a display 46 and a touch panel 48. An example of the display 46 is an organic EL display. The display 46 may be another type of display such as a liquid crystal display instead of an organic EL display. The display 46 is an example of a "display unit" within the scope of the present invention.

[0064] Display 46 displays images, text information, and the like. Touch panel 48 is a transmissive touch panel that overlaps the surface of the display area of ​​display 46. Touch panel 48 receives user input by detecting contact with an indicator such as a finger or stylus. While an external touch panel display in which touch panel 48 overlaps the surface of the display area of ​​display 46 is described as an example of touch panel display 59, this is merely an example. For example, an external or internal touch panel display may also be used as touch panel display 59.

[0065] As an example, Figure 5 As shown, in smart device 10, when an instruction to start imaging is received via touch panel 48, the imaging area is captured by light receiver 18. Specifically, light receiver 18 receives visible reflected light and generates a visible light image representing the imaging area as an image corresponding to the received visible reflected light. This visible light image is an example of a "captured image" within the context of the present invention.

[0066] The visible light image is displayed on the display 46 according to the instruction received through the touch panel 48. Figure 5 In the example shown, the imaging area is defined by a viewing angle θ1 . The viewing angle θ1 is changed in accordance with an instruction received via the touch panel 48 .

[0067] The smart device 10 has an image recognition function. The smart device 10 recognizes a facial image representing a person's face from a visible light image by running the image recognition function. In the smart device 10, a recognition action (hereinafter also referred to as "recognition action") of recognizing the face of a person included in the camera area is performed. The recognition action is achieved by running the image recognition function, that is, recognizing a facial image from a visible light image. Figure 5 In the example shown, a facial image is recognized from a visible light image, and as a recognition result, the facial image is displayed with the facial image surrounded by a frame line 29. The recognition result includes, for example, the recognized facial image, the number of recognized facial images (hereinafter also referred to as the "number of facial images"), coordinates that can determine the position of the facial image, and frame line information representing the frame line 29. Furthermore, a person's face is an example of a "specific subject" involved in the technology of the present invention. Furthermore, the frame line 29 is an example of a "distance measurement result" and an "image surrounding a specific subject image" involved in the technology of the present invention.

[0068] Furthermore, the smart device 10 has a distance measurement function. The smart device 10 runs the distance measurement function, for example, Figure 6 As shown, the light irradiator 16 irradiates a laser beam at an irradiation angle θ2, and the distance measurement partitioned area 26N2 of the light receiver 18 receives the IR reflected light. The irradiation angle θ2 may be the same as or different from the field of view angle θ1. The distance from the smart device 10 to the distance measurement object is measured based on the time required from the irradiation of the laser beam to the reception of the IR reflected light and the speed of light. For example, if the distance to the distance measurement object is set to "L", the speed of light is set to "c", and the time required from the irradiation of the laser beam by the light irradiator 16 to the reception of the IR reflected light by the distance measurement partitioned area 26N2 is set to "t", the distance L is calculated according to the formula "L = c × t × 0.5".

[0069] The recognition result and the distance measurement result are superimposed on the visible light image. The superimposed image obtained by superimposing the recognition result and the distance measurement result on the visible light image is displayed on the display 46 as a live view image. Figure 6 In the example shown, a frame line 29 surrounding a facial image as a recognition result is superimposed on the visible light image, and a numerical value indicating the distance from the smart device 10 to the distance measurement object as a distance measurement result is superimposed (in Figure 6 In the example shown, images obtained at 1.6 m, 1.8 m, and 5.3 m) are used as overlapping images, and the overlapping images are displayed as live view images on the display 46. Figure 6 In the example shown, the smart device 10 is directed to a plurality of representative positions within the imaging area (in Figure 6 In the example shown, the distances to each of three positions (e.g., three positions) are superimposed on the visible light image. A representative example of multiple positions is a plurality of positions where the contrast difference between a specific subject (e.g., a subject and / or a person in the center area of ​​the image) within the imaging area is greater than a specified value.

[0070] The soft key 28 is displayed together with the live view image on the display 46. When the user or the like instructs to start recording the image, the soft key 28 is operated. As the recorded image, for example, a still image and / or a moving image can be cited. Figure 7 As shown, the user visually recognizes the live view image, determines the image capture area, and operates soft key 28. In the smart device 10, by operating soft key 28, the focus is adjusted based on the recognition and distance measurement results, and image capture for recording is performed. Hereinafter, the exposure performed during image capture for recording is also referred to as "main exposure."

[0071] refer to Figure 8, the structure of the smart device 10 will be described. In addition to the light irradiator 16 and the light receiver 18, the smart device 10 also includes a controller 15, an input / output interface 40, an image memory 42, a UI device 44, an external I / F 52, and a communication I / F 54. The controller 15 is an example of a "processing device" and a "computer" within the scope of the present invention.

[0072] The controller 15 includes a CPU 15A, a storage device 15B, and a memory 15C. The CPU 15A is an example of a "processor" and an "identification processor" involved in the technology of the present invention, and the memory 15C is an example of a "memory" involved in the technology of the present invention. The CPU 15A, the storage device 15B, and the memory 15C are connected via a bus 50, and the bus 50 is connected to the input / output interface 40. Figure 8 In the example shown, for ease of illustration, a single bus is shown as the bus 50, but multiple buses may be used. The bus 50 may be a serial bus or a parallel bus including an information bus, an address bus, and a control bus.

[0073] The storage device 15B stores various parameters and various programs. The storage device 15B is a non-volatile storage device. Here, as an example of the storage device 15B, a flash memory is used. The flash memory is just an example. As the storage device 15B, various non-volatile memories such as magnetoresistive memory and / or ferroelectric memory can be cited to replace the flash memory or used together with the flash memory. In addition, the non-volatile storage device can also be an EEPROM, HDD and / or SSD, etc. In addition, the memory 15C temporarily stores various information and is used as a working memory. As an example of the memory 15C, a RAM can be cited, but it is not limited to this, and other types of storage devices can also be cited.

[0074] Storage device 15B stores various programs, including an image capture processing program 70. Image capture processing program 70 is an example of a "program" within the meaning of the present invention. CPU 15A reads the required program from storage device 15B and executes it on memory 15C. CPU 15A controls the entire smart device 10 based on the program executed on memory 15C.

[0075] Multiple devices are connected to the input / output interface 40, which is responsible for sending and receiving various information between the multiple devices. Figure 8 In the illustrated example, a controller 15 , a light irradiator 16 , a light receiver 18 , an image memory 42 , a UI device 44 , an external I / F 52 , and a communication I / F 54 are shown as a plurality of devices connected to the input / output interface 40 .

[0076] The external I / F 52 is responsible for sending and receiving various information to and from devices external to the smart device 10 (hereinafter also referred to as "external devices"). An example of the external I / F 52 is a USB interface. External devices such as smart devices, personal computers, servers, USB memories, memory cards, and / or printers (not shown) can be connected directly or indirectly to the USB interface.

[0077] The communication I / F 54 has communication functions such as LTE, 5G, wireless LAN, and / or Bluetooth (registered trademark), and is responsible for sending and receiving various information between external devices and the CPU 15A. For example, the communication I / F 54 is communicatively connected to a network 56 (e.g., the Internet) via a base station (not shown), and is responsible for sending and receiving various information between external devices on the network 56 and the CPU 15A.

[0078] The UI device 44 includes a display 46, and the CPU 15A causes the display 46 to display various information. In addition, the UI device 44 includes a receiving device 47. The receiving device 47 includes a touch panel 48 and a hard key unit 53. The hard key unit 53 includes the indicator key 13 (see Figure 2 ). CPU 15A operates according to various instructions received via touch panel 48. Here, hard key unit 53 is included in UI device 44, but the technology of the present invention is not limited to this. For example, hard key unit 53 may be connected to external I / F 52.

[0079] The light irradiator 16 includes a light transmission window 20, a beam expander 21, a collimating lens 23, an LD 24, and an LD driver 25. The light transmission window 20, the beam expander 21, and the collimating lens 23 are arranged in this order along the optical axis L1 from the imaging area side (object side) to the LD 24. The LD driver 25 is connected to the LD 24 and the input / output interface 40, and drives the LD 24 according to the instruction of the CPU 15A to emit a laser beam from the LD 24.

[0080] The laser beam emitted from the LD 24 is converted into parallel light by the collimator lens 23 , has its optical diameter expanded by the beam expander 21 , and is irradiated toward the distance measurement object through the light transmission window 20 .

[0081] The light receiver 18 includes a light transmission window 22, an objective lens 30A, a focusing lens 30B, an aperture 30C, a photoelectric conversion element 26, a photoelectric conversion element driver 32, and a signal processing circuit 34. In the light receiver 18, the light transmission window 22, the objective lens 30A, the focusing lens 30B, and the aperture 30C are arranged in order from the imaging area side (object side) to the photoelectric conversion element 26 along the optical axis L2. The photoelectric conversion element driver 32 is connected to the photoelectric conversion element 26 and the input / output interface 40 and drives the photoelectric conversion element 26 according to instructions from the CPU 15A. For example, under the control of the CPU 15A, the photoelectric conversion element driver 32 supplies an imaging timing signal to the photoelectric conversion element 26 that specifies the timing of imaging by the photoelectric conversion element 26. The photoelectric conversion element 26 performs reset, exposure, and output of electrical signals based on the imaging timing signal supplied by the photoelectric conversion element driver 32. Examples of imaging timing signals include a vertical synchronization signal and a horizontal synchronization signal.

[0082] The light receiver 18 includes a focus control mechanism 31. The focus control mechanism 31 includes a focus lens 30B, a moving mechanism 60, a motor 62, and a motor driver 64. The focus lens 30B is supported by the moving mechanism 60 so that it can slide along the optical axis L2. The motor 62 is connected to the moving mechanism 60 and the motor driver 64. The motor driver 64 is connected to the input / output interface 40 and drives the motor 62 according to instructions from the CPU 15A. The moving mechanism 60 is connected to the drive shaft (not shown) of the motor 62 and receives power from the motor 62 to selectively move the focus lens 30B along the optical axis L2 toward the object side and the image side. In other words, the CPU 15A adjusts the focus position by controlling the drive of the motor 62 via the motor driver 64. Here, the "focus position" refers to the position of the focus lens 30B on the optical axis L2 when the focus is in focus (for example, when the contrast of the visible light image is maximized or when a specified depth of subject is achieved). Hereinafter, for convenience of explanation, the control for bringing the focus lens 30B into focus will also be referred to as “focus control”.

[0083] Aperture 30C is a fixed aperture with a constant opening. With a fixed aperture, exposure adjustment is performed by the electronic shutter of photoelectric converter 26. Aperture 30C may be a variable aperture rather than a fixed aperture. Furthermore, objective lens 30A, focus lens 30B, and aperture 30C included in light receiver 18 are merely examples; the present invention's technology is applicable even if the lens configuration and / or the position of aperture 30C are changed.

[0084] The reflected light enters the light receiver 18 from the light transmission window 22. The reflected light entering the light transmission window 22 is focused on the photoelectric conversion element 26 via the objective lens 30A, the focusing lens 30B, and the aperture 30C.

[0085] The photoelectric conversion element 26 is connected to the signal processing circuit 34 and outputs pixel data representing the pixel value of each of the visible light pixels and the IR pixels to the signal processing circuit 34. The signal processing circuit 34 performs A / D conversion on the pixel data input from the photoelectric conversion element 26 to digitize the pixel data, and then performs various signal processing on the digitized pixel data.

[0086] The signal processing circuit 34 includes a visible light pixel data processing circuit 34A and an IR pixel data processing circuit 34B. The visible light pixel data processing circuit 34A performs known signal processing on the pixel data of the visible light pixels, such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion, and chromatic aberration correction, thereby generating a visible light image. The visible light pixel data processing circuit 34A then stores the visible light image in the image memory 42. As will be described later, the image memory 42 stores 10 frames of visible light images captured sequentially while gradually varying the intensity of the distance-measuring laser beam.

[0087] In the distance measurement partitioned area 26N2, the IR reflected light is received by the IR pixels and used to measure the distance from the smart device 10 to the distance measurement target based on the emission timing and the light reception timing. The IR pixel data processing circuit 34B receives an emission timing signal from the CPU 15A, indicating the timing of laser beam emission from the LD 24 (hereinafter referred to as "emission timing"). The IR pixel data processing circuit 34B measures the distance from the smart device 10 to the distance measurement target for each IR pixel based on the emission timing indicated by the emission timing signal and the timing of reception of the IR reflected light by each IR pixel (hereinafter referred to as "light reception timing"). The IR pixel data processing circuit 34B then associates the distance measurement result, including the measured distance, with the visible light image and stores it in the image memory 42.

[0088] When the smart device 10 performs image recognition on a visible light image obtained by photographing a piece of paper with a person drawn on it as the subject, it may sometimes mistakenly identify the subject as a "person." However, by performing image recognition using distance measurement results obtained by measuring the distance in the longitudinal direction of the subject, the smart device 10 can determine whether the subject is "paper with a person drawn on it" or "person." In other words, by performing image recognition using distance measurement results, the smart device 10 can improve image recognition accuracy. Image recognition is performed on a visible light image generated by the visible light pixel data processing circuit 34A based on the light reception results in the visible light image segmented area 26N1. Ideally, in the visible light image segmented area 26N1, IR reflected light should be completely blocked by the visible light filter. However, depending on the intensity of the IR reflected light, this is difficult to achieve. When the IR reflected light reaches the visible light pixels, it may be reflected as noise in the visible light image. If the image quality of the visible light image deteriorates during image recognition, image recognition accuracy decreases. For example, if the intensity of the laser beam from LD 24 is set to zero, the IR reflected light also becomes zero. Therefore, the IR reflected light does not appear as noise in the visible light image, but distance measurement cannot be performed. If distance measurement cannot be performed, the distance measurement results cannot be used for image recognition, resulting in lower image recognition accuracy compared to when the distance measurement results are used for image recognition.

[0089] Therefore, in the smart device 10, as an example, Figure 9 As shown, the CPU 15A reads out the imaging processing program 70 from the storage device 15B, and executes imaging processing according to the read imaging processing program.

[0090] As an example, as shown in FIG10 , the imaging process is realized by the ranging imaging device 14 operating as an imaging unit 91 and a ranging unit 92, and the CPU 15A operating as a changing unit 96, a control unit 93, a recognition unit 94, a processing unit 95, an output unit 97, and a display control unit 99. The imaging unit 91 includes a light receiver 18, and the ranging unit 92 includes a light irradiator 16, a divided area 26N2 for ranging (see FIG10 ). Figure 3 and Figure 8 ) and IR pixel data processing circuit 34B, etc.

[0091] The changing unit 96 changes the irradiation energy of the laser beam (hereinafter also referred to as "irradiation energy") in multiple stages. In the present embodiment, as an example of "multiple stages", 10 stages are adopted. The changing unit 96 changes the irradiation energy by changing the intensity of the laser beam emitted from the LD24 via the LD driver 25. In the state where the irradiation energy is the lowest, the irradiation energy may be zero. The laser beam is irradiated in synchronization with the camera recording of the visible light image. Here, an example of a method of changing the intensity of the laser beam by the changing unit 96 is given, but the technology of the present invention is not limited thereto. For example, the irradiation energy of the laser beam may be changed by changing the emission time of the laser beam and / or the number of emission times of the laser beam per unit time together with or instead of the intensity of the laser beam.

[0092] The control unit 93 causes the camera unit 91 to capture the camera area at a predetermined frame rate (for example, 120 fps). The visible light image obtained by the camera unit 91 is stored in the image memory 42. Furthermore, the control unit 93 performs continuous parallel processing. Continuous parallel processing refers to parallel processing for multiple consecutive times. Here, multiple times refers to 10 times. 10 times, for example, means 10 frames of imaging performed by the camera unit 91. Furthermore, parallel processing refers to processing that causes the recognition unit 94 and the ranging unit 92 to perform recognition and ranging actions in parallel. The recognition action is performed by the recognition unit 94, and the ranging action is performed by the ranging unit 92. That is, the control unit 93 controls the recognition unit 94 and the ranging unit 92 in such a manner that the recognition and ranging actions for 10 frames are performed in parallel.

[0093] In addition, while the example described herein uses a method for performing recognition and ranging operations for 10 frames in parallel, the present invention is not limited thereto. Recognition and ranging operations for less than 10 frames or more than 11 frames can also be performed in parallel, as long as multiple recognition and ranging operations are performed in parallel. The number of times the recognition and ranging operations are performed in parallel can be a fixed value, a value that changes based on instructions received by the receiving device 47, or a value that changes based on the operating mode of the smart device 10 and / or the shooting scene.

[0094] Furthermore, in this embodiment, continuous parallel processing is repeated. That is, the recognition and ranging operations for 10 frames are repeated multiple times in parallel. In this embodiment, the continuous parallel processing is repeated several dozen times (for example, 30 times or more). The number of times the continuous parallel processing is repeated can be a fixed value, a value that changes based on instructions received by the receiving device 47, or a value that changes based on the operation mode of the smart device 10 and / or the shooting scene.

[0095] The IR pixel data processing circuit 34B processes the data from the time when the laser beam is irradiated by the light irradiator 16 to the time when the laser beam passes through the distance measurement divided area 26N2 (reference Figure 3 and Figure 8 ) to receive IR reflected light and the speed of light, and measures the distance from the smart device 10 to the measurement target for each IR pixel. At this point, depending on the laser beam's irradiation energy and the distance to the measurement target, IR pixels may be generated that are unable to fully receive IR reflected light. Here, "IR pixels that are unable to fully receive IR reflected light" refers to, for example, IR pixels where the amount of IR reflected light received from the measurement target several meters ahead does not meet a predetermined amount of light received, as determined through testing with actual equipment and / or computer simulations, as sufficient for distance measurement. Generally, the higher the laser beam's irradiation energy, the longer the measurable distance, while as the energy decreases, the measurable distance decreases. The IR pixel data processing circuit 34B includes a threshold for comparison with the amount of IR reflected light received by each IR pixel (e.g., the amount of light received per unit time). IR pixels whose IR reflected light received is less than the threshold are deemed to have failed to accurately measure the distance, and blank information is output. The distance measurement results for one frame based on the distance measurement unit 92 are associated with the visible light image and stored in the image memory 42.

[0096] The distance measurement result includes the distance measured by the distance measuring unit 92, that is, the distance from the smart device 10 to the distance measurement object, the number of IR pixels that accurately measured the distance (hereinafter also referred to as "accurately measured IR pixels") (hereinafter also referred to as "accurately measured IR pixel number"), and blank information. The number of accurately measured IR pixels is calculated by the IR pixel data processing circuit 34B. Accurately measured IR pixels are, for example, IR pixels that receive IR reflected light with a light intensity greater than a threshold value. Generally, the higher the irradiation energy of the laser beam, the more accurately measured IR pixels there are. Conversely, if the irradiation energy of the laser beam is high, the amount of laser beam light reflected in the visible light image increases, and the image quality deteriorates. Therefore, the higher the irradiation energy of the laser beam, the fewer facial images there are.

[0097] Recognition unit 94 reads the visible light image from image memory 42 and performs facial recognition on the read visible light image by referring to an image recognition dictionary (not shown). The image recognition dictionary contains, for example, a variety of facial images. The recognition results for one frame obtained through image recognition are associated with the visible light image and the distance measurement results and stored in image memory 42.

[0098] The processing unit 95 reads out the distance measurement results and recognition results of one frame from the image memory 42. Then, the processing unit 95 performs a rating process. Here, the rating process refers to a process of rating the corresponding frame for each frame (here, as an example, for each of 10 frames) based on the number of IR pixels and the number of facial images contained in the read distance measurement results. Rating each frame means weighting the recognition results corresponding to each frame. In addition, the number of facial images is an example of "a plurality of recognition results obtained through multiple recognition actions" involved in the technology of the present invention. Furthermore, the rating process is an example of "determination processing" involved in the technology of the present invention. Moreover, hereinafter, for the sake of convenience of explanation, one frame that is the object of rating will also be referred to as a "rating object frame."

[0099] The ranking process is performed by the processing unit 95 according to the following calculation formula (1). Specifically, as shown in the following calculation formula (1), the processing unit 95 calculates the sum of the value obtained by multiplying the number of accurate ranging IR pixels by coefficient A and the value obtained by multiplying the number of facial images by a coefficient B different from coefficient A as the ranking value. Here, positive values ​​are used as coefficients A and B.

[0100] Rating value = (number of accurate ranging IR pixels) × A + (number of face images) × B ... (1)

[0101] In addition, in the above calculation formula (1), coefficients A and B may not be positive values. For example, when obtaining a rating value that ignores the ranging result, coefficient A can be set to "0", and when obtaining a rating value that ignores the recognition result, coefficient B can be set to "0". In addition, coefficient A can also be determined according to the irradiation energy. For example, when the irradiation energy is "0", coefficient A can be set to "0", and the higher the irradiation energy, the greater the value of coefficient A. Thus, the rating value is calculated as a value corresponding to the irradiation energy used in the ranging action parallel to the recognition action. In addition, coefficient A and / or coefficient B can be fixed values, or values ​​that change according to instructions received through the receiving device 47, or values ​​that change according to the action mode and / or shooting scene of the smart device 10.

[0102] According to the above calculation formula (1), the more accurate IR pixels and facial images are, the higher the rating value is. The rating value is associated with the visible light image, the ranging result, and the recognition result and stored in the image memory 42.

[0103] The image memory 42 stores 10 frames of visible light images sequentially captured while changing the irradiation energy level R of the laser beam in 10 steps from 1 to 10, distance measurement results, recognition results, and rating values.

[0104] Output unit 97 outputs the recognition and ranging results for a specific frame among the multiple frames obtained by the imaging unit 91 capturing the imaging area multiple times at a predetermined frame rate. Specifically, output unit 97 refers to the ranking values ​​for ten frames and outputs the visible light image, ranging results, and recognition results for the frame with the highest ranking value among the ten frames to live view image storage area 42A of image memory 42. The frame with the highest ranking value among the ten frames is an example of a "specific number of times" within the scope of the present invention, and live view image storage area 42A is an example of a "specific output target" within the scope of the present invention.

[0105] The display control unit 99 acquires the visible light image, the distance measurement result, and the recognition result from the live view image storage area 42A, and generates an overlapped image based on the acquired visible light image, the distance measurement result, and the recognition result. The display control unit 99 outputs the generated overlapped image as a live view image to the display 46, thereby causing the display 46 to display the live view image (see FIG. 1 ). Figure 6 ). In addition, the output unit 97 and the display control unit 99 are examples of the "output unit" involved in the technology of the present invention. Furthermore, the display 46 is an example of the "specific output target" and the "display unit" involved in the technology of the present invention.

[0106] In addition, while the superimposed image is shown here as an example of an image obtained by superimposing both the distance measurement result and the frame line 29 on the visible light image, the technology of the present invention is not limited to this. An image may also be obtained by superimposing only one of the distance measurement result or the frame line 29 on the visible light image. Furthermore, the superimposed image and the visible light image can be selectively displayed on the display 46 as a live view image.

[0107] Next, refer to Figure 11 , the functions of the parts of the smart device 10 related to the technology of the present invention are explained. Figure 11 An example of the flow of the imaging process executed by the CPU 15A is illustrated in FIG.

[0108] exist Figure 11 In the imaging process shown, first, in step ST10, the control unit 93 determines whether a condition for starting the imaging process (hereinafter also referred to as the "imaging process starting condition") is satisfied. An example of an imaging process starting condition is that the touch panel 48 receives an instruction to start the imaging process. If the imaging process starting condition is not satisfied in step ST10, the determination in step ST10 is repeated. If the imaging process starting condition is satisfied in step ST10, the imaging process proceeds to step ST12.

[0109] In step ST12 , the changing unit 96 sets the irradiation energy level R of the laser beam to a maximum value of 10. Thereafter, the imaging process proceeds to step ST14 .

[0110] In step ST14, the control unit 93 determines whether the imaging timing has been reached. The imaging timing is reached, for example, at each cycle determined at a predetermined frame rate (e.g., 1 / 120 second). If the imaging timing has not been reached in step ST14, the determination is negative, and the determination in step ST14 is repeated. If the imaging timing has been reached in step ST14, the determination is positive, and the imaging process proceeds to step ST16.

[0111] In step ST16, the control unit 93 controls the imaging unit 91 to expose the photoelectric conversion element 26. Specifically, the control unit 93 causes the photoelectric conversion element driver 32 to output an imaging timing signal to the photoelectric conversion element 26, thereby resetting the photoelectric conversion element 26 and causing it to accumulate new charge. The control unit 93 then causes the visible light pixel data processing circuit 34A to generate a visible light image corresponding to the amount of charge accumulated in the photoelectric conversion element 26. The visible light image obtained by exposing the photoelectric conversion element 26 is stored in the image memory 42. Furthermore, the control unit 93 controls the distance measuring unit 92 to irradiate the laser beam. The imaging process then transitions to step ST18.

[0112] In step ST18, the control unit 93 performs parallel processing. Specifically, in step ST18, the distance measurement unit 92 performs distance measurement, and the recognition unit 94 performs recognition. The distance measurement unit 92 measures the distance from the smart device 10 to the distance measurement target for each IR pixel. The recognition unit 94 reads the visible light image from the image memory 42 and performs facial recognition on the read visible light image. The distance measurement and recognition results obtained for one frame are associated with the visible light image and stored in the image memory 42 by the control unit 93. The imaging process then transitions to step ST20.

[0113] In step ST20, the processing unit 95 reads the ranging and recognition results for the rating target frame from the image memory 42. Based on the number of accurately measured IR pixels and the number of facial images, it calculates a rating value for the ranging and recognition results for the rating target frame. The rating value is associated with the visible light image, ranging, and recognition results for the rating target frame and stored in the image memory 42. The imaging process then proceeds to step ST22.

[0114] In step ST22, the processing unit 95 determines whether the irradiation energy level R of the laser beam is equal to the minimum value 1. If R = 1 is not satisfied in step ST22, the determination is negative, and the imaging process proceeds to step ST23. If R = 1 is satisfied in step ST22, the determination is positive, and the imaging process proceeds to step ST24.

[0115] In step ST23 , the processing unit 95 subtracts “1” from the irradiation energy level R. Thereafter, the imaging process proceeds to step ST14 .

[0116] By repeating the processing from step ST14 to step ST20 in this manner, the irradiation energy level R decreases by 1 at a time from a maximum value of 10 to a minimum value of 1, and the irradiation energy level R changes in 10 steps. As the irradiation energy level R changes step by step, the visible light images obtained by the imaging unit 91 sequentially capturing the imaging area, the distance measurement results by the distance measuring unit 92, the recognition results by the recognition unit 94, and the rating values ​​calculated by the processing unit 95 are stored in the image memory 42 for 10 frames.

[0117] In step ST24, the output unit 97 refers to the ranking values ​​for 10 frames and outputs the best-ranked visible light image, the distance measurement result, and the recognition result to the live view image storage area 42A of the image memory 42. Thereafter, the imaging process proceeds to step ST26.

[0118] In step ST26, the display control unit 99 retrieves the visible light image, ranging results, and recognition results of the frame with the highest ranking value from the live view image storage area 42A. The display control unit 99 then generates an overlay image based on the visible light image, ranging results, and recognition results of the frame with the highest ranking value retrieved from the live view image storage area 42A, and displays the generated overlay image on the display 46 as the live view image. Thus, the live view image generated using the ranging results and recognition results of the frame with the best overall accuracy in ranging and recognition operations, out of the 10 frames of ranging and recognition results acquired while varying the irradiation energy level R in 10 steps, is displayed on the display 46. The imaging process then transitions to step ST28.

[0119] In step ST28, the control unit 93 determines whether the soft key 28 (see Figure 6 ). In step ST28, if the soft key 28 is not operated, the determination is negative, and the imaging process transitions to step ST32. In step ST28, if the soft key 28 is operated, the determination is positive, and the imaging process transitions to step ST30.

[0120] In step ST30, the control unit 93 controls the imaging unit 91 to capture a still image accompanying the main exposure based on the distance measurement and recognition results of the frame with the highest ranking value used in the live view image. The imaging process then transitions to step ST32. While the imaging of a still image is illustrated here, the technology of the present invention is not limited to this. The imaging of a moving image accompanying the main exposure or the aforementioned recording image can also be used.

[0121] In the still image shooting by the camera unit 91 in step ST30, the face of the person represented by the specific facial image is focused on based on the ranging result and the recognition result of the frame with the highest rating value used in the live view image. Here, the specific facial image is, for example, the facial image included in the recognition result. When a plurality of facial images are included in the recognition result, the face of the person closest to the center of the shooting area or the face of the person with a specific expression (for example, a smiling face) is selected as the focus object among the faces of the plurality of people represented by the plurality of facial images. Furthermore, among the faces of the plurality of people surrounded by the frame line 29 and displayed on the display 46, the face of the person specified by the user or the like via the touch panel 48 can be selected as the focus object as the area to be focused.

[0122] The control unit 93 obtains the distance to the face of the person selected as the focus object based on the distance measurement result, and derives the focus position corresponding to the obtained distance. The focus position is derived by the control unit 93 from a focus position derivation table (not shown) that establishes a correspondence between the distance and the focus position, or from a focus position derivation expression (not shown) that uses the distance as an independent variable and the focus position as a dependent variable. The control unit 93 operates the focus control mechanism 31 to move the focusing lens 30B to the derived focus position. Then, the control unit 93 drives the photoelectric conversion element driver 32 to formally expose the photoelectric conversion element 26. By performing the formal exposure, a still image is generated by the visible light pixel data processing circuit 34A, and the generated still image is stored in the image memory 42 by the visible light pixel data processing circuit 34A. Then, the control unit 93 obtains the still image from the image memory 42 and stores the obtained still image in a memory card (not shown) connected to the external I / F 52.

[0123] In step ST32, the control unit 93 determines whether a condition for terminating the imaging process (hereinafter also referred to as the "imaging process termination condition") is satisfied. An example of an imaging process termination condition is a condition where the touch panel 48 receives an instruction to terminate the imaging process. In step ST36, if the imaging process termination condition is not satisfied, the determination is negative, and the imaging process proceeds to step ST12. In step ST36, if the imaging process termination condition is satisfied, the determination is positive, and the imaging process ends.

[0124] As described above, according to the smart device 10 of the first embodiment, the control unit 93 causes the recognition operation and the distance measurement operation to be performed multiple times in parallel. In the recognition operation, the recognition unit 94 identifies a specific subject (here, for example, a person's face) contained in the imaging area based on a visible light image obtained by the imaging unit 91 capturing the imaging area. In the distance measurement operation, the distance measurement unit 92 irradiates the imaging area with a laser beam and receives IR light reflected from the laser beam. The changing unit 96 changes the irradiation energy of the laser beam directed at the imaging area for each distance measurement operation. Therefore, according to this configuration, compared to a case where the laser beam of the same irradiation energy is always used for distance measurement while simultaneously performing image recognition of the specific subject, both distance measurement and identification of the specific subject can be achieved with higher accuracy. Furthermore, while the method of changing the irradiation energy of the laser beam directed at the imaging area for each distance measurement operation has been described as an example of "changing the irradiation energy of the laser beam directed at the imaging area for each distance measurement operation" involved in the technology of the present invention, the technology of the present invention is not limited to this method. For example, "changing the laser beam irradiation energy for the imaging area for each ranging operation" also includes cases where the laser beam irradiation energy is the same in multiple continuous or discontinuous ranging operations. Among the multiple frames containing ranging operations with the same laser beam irradiation energy, only one frame or all frames may be selected as the rating target frame.

[0125] According to the smart device 10 of the first embodiment, the output unit 97 outputs the recognition result and ranging result for the frame with the highest ranking value to a specific output destination (here, for example, the live view image storage area 42A and the display 46). Therefore, according to this configuration, the timing at which the recognition result and ranging result were obtained can be more easily determined than in a case where the recognition result and ranging result are output at different timings.

[0126] According to the smart device 10 of the first embodiment, the specific output target is the display 46 capable of displaying at least one of the recognition result and the distance measurement result. Therefore, according to this configuration, the user can recognize the recognition result obtained by the recognition operation and the distance measurement result obtained by the distance measurement operation.

[0127] According to the smart device 10 of the first embodiment, the display 46 displays the specific subject image showing the specific subject and the frame line 29 surrounding the specific subject image as the recognition result. Therefore, this configuration allows the user to visually grasp the recognition result obtained by the recognition operation.

[0128] According to the smart device 10 of the first embodiment, the control unit 93 moves the focus lens 30B to the focus position determined based on the distance measurement result. Therefore, according to this configuration, focusing can be performed more easily using the automatic focusing method than when focusing is performed manually regardless of the distance measurement result.

[0129] In the smart device 10 of the first embodiment, the processing unit 95 calculates ten rating values ​​from the recognition and ranging results obtained through ten recognition and ranging operations. Therefore, this configuration allows for more balanced combinations of recognition and ranging results, compared to when no rating values ​​are calculated.

[0130] According to the smart device 10 of the first embodiment, the recognition results obtained by the recognition operation are weighted according to the type and / or form of the specific subject, etc. Therefore, according to this configuration, the importance of the recognition results can be determined.

[0131] In the smart device 10 of the first embodiment, the control unit 93 causes the imaging unit 91 to perform imaging based on the main exposure based on the recognition result selected according to the rating value. Therefore, according to this configuration, compared to a case where the imaging unit 91 performs imaging based on the main exposure based on a recognition result randomly selected from a plurality of recognition results obtained through multiple recognition operations, it is possible to suppress the imaging unit 91 from performing imaging based on the main exposure based on inappropriate recognition results.

[0132] According to the smart device 10 of the first embodiment, the specific subject is a person's face. Therefore, according to this configuration, both distance measurement and face recognition can be performed with high accuracy, compared to a case where distance measurement based on directional light that always illuminates the imaging area with the same irradiation energy and face recognition within the imaging area are performed simultaneously.

[0133] According to the smart device 10 of the first embodiment, a person's face can be a face with a specific expression. In this case, both distance measurement and recognition of a face with a specific expression contained in the imaging area can be performed with high accuracy, compared to a case where both distance measurement using a laser beam that always irradiates the imaging area with the same irradiation energy and recognition of a face with a specific expression contained in the imaging area are performed simultaneously.

[0134] In addition, in the first embodiment described above, a person's face was cited as an example of a "specific subject" involved in the technology of the present invention, but the technology of the present invention is not limited to this. For example, the specific subject may also be an object with a reflectivity less than a threshold value. Here, the threshold value is, for example, a value derived in advance through sensory testing based on actual equipment and / or computer simulation as a lower limit value of the reflectivity, which is a level at which IR reflected light obtained by irradiating a laser beam at a specified irradiation energy level R is visually recognized as noise in a visible light image. The specified irradiation energy level R refers to, for example, the irradiation energy level R of the fifth stage of the 10 stages of irradiation energy level R described above. Therefore, according to this configuration, compared to a case where ranging based on a laser beam that always irradiates the imaging area with the same irradiation energy and recognition of objects with a reflectivity less than a threshold value are simultaneously performed, ranging and recognition of objects with a reflectivity less than a threshold value can be achieved with high accuracy.

[0135] Furthermore, in the first embodiment described above, a rating process was cited as an example of the "determination process" involved in the technology of the present invention, but the technology of the present invention is not limited thereto. For example, the processing unit 95 may output, via the external I / F 52, at least the recognition result obtained by each recognition operation and the distance measurement result obtained by each distance measurement operation to an external device (not shown) such as a smart device different from the smart device 10, a personal computer, and / or a server. Furthermore, the processing unit 95 may create, as an image file in a specific format (e.g., Exif format), at least the recognition result obtained by each recognition operation and the distance measurement result obtained by each distance measurement operation, and the visible light image.

[0136] Furthermore, in the first embodiment described above, the recognition unit 94 recognizes the face of a person as a specific subject, but the specific subject is not limited to the face of a person, and may be the entire body of a person, or may be a face with a specific expression (e.g., a smiling face) among the faces of multiple people. The recognition unit 94 may also recognize trees, flowers, historical buildings, and / or characters as specific subjects. Furthermore, when the subject includes an object that is generally recognized as having a high reflectivity (e.g., a reflector), the object that is generally recognized as having a high reflectivity is more likely to reflect the laser beam than an object that absorbs light, such as a black object, and is more likely to appear as noise in the visible light image. Therefore, the recognition unit 94 may only recognize objects with a reflectivity less than the above-mentioned threshold as specific subjects.

[0137] The recognition unit 94 can also weight specific subjects based on their type or form. For example, if the recognition unit 94 weights the face of a pre-registered person and uses this recognition result for the main exposure, the image will be captured with the person in focus, even if the pre-registered person is not located in the center of the camera area. Furthermore, if the recognition unit 94 weights the rare expressions or movements of animals, it will be easier to focus on rare expressions or movements. On the other hand, if the weight for a person's closed eyes is set to zero, it will be difficult to focus on the person with closed eyes.

[0138] Furthermore, in the first embodiment described above, an example of outputting the recognition and ranging results for the frame with the highest ranking value (an example of a "specific ranking" as defined by the present invention) to a specific output destination (e.g., live view image storage area 42A and display 46) is provided. However, the present invention is not limited to this. For example, in continuous parallel processing, the recognition and ranging results for the first frame, the tenth frame, and the frame with the highest ranking value, out of ten frames, may be output to a specific output destination. Furthermore, since continuous parallel processing is repeated, the recognition and ranging results for at least one specific frame out of ten frames may also be output to a specific output destination in periodically or aperiodically selected continuous parallel processing.

[0139] [Second embodiment]

[0140] As an example, as shown in FIG12 , the smart device 100 according to the second embodiment includes a zoom control mechanism 131, which differs from the smart device 10 according to the first embodiment. The remaining structure of the smart device 100 is the same as that of the smart device 10 according to the first embodiment. Therefore, the same reference numerals are used for the same components as those described in the first embodiment, and their descriptions are omitted.

[0141] The smart device 100 differs from the smart device 10 described in the above embodiment in that it includes a range-finding camera 114 instead of the range-finding camera 14. The range-finding camera 114 differs from the range-finding camera 14 in that it includes a zoom control mechanism 131. The zoom control mechanism 131 includes a zoom lens 30D, a moving mechanism 160, a motor 162, and a motor driver 164. The zoom lens 30D is supported by the moving mechanism 160 so as to be slidable along the optical axis L2. The motor 162 is connected to the moving mechanism 160 and the motor driver 164. The motor driver 164 is connected to the input / output interface 40 and drives the motor 162 in response to instructions from the CPU 15A. The moving mechanism 160 is connected to a drive shaft (not shown) of the motor 162 and receives power from the motor 162 to selectively move the zoom lens 30D along the optical axis L2 toward the object side or the image side. That is, the CPU 15A adjusts the angle of view for imaging by controlling the driving of the motor 162 via the motor driver 164 .

[0142] Next, refer to Figure 13 The imaging process according to the second embodiment will be described. Figure 13 The camera processing shown is Figure 11 The difference between the imaging process shown in FIG. 1 and FIG. 2 is that the imaging process includes step ST40 and step ST42. Figure 11 The steps different from the flowchart shown in Figure 11 The same steps as those included in the flowchart shown in FIG. 1 are marked with the same step numbers and their descriptions are omitted.

[0143] exist Figure 13 In the illustrated imaging process, when step ST26 is completed, the imaging process transitions to step ST40. In step ST40, the control unit 93 obtains the distance to the face of a person represented by a specific facial image (e.g., the facial image closest to the center of the live view image) contained in the frame with the highest ranking value, based on the distance measurement results of the frame with the highest ranking value used in the live view image, and determines whether the obtained distance is outside a specified range. The specified range can be a fixed value, a value that changes based on instructions received by the receiving device 47, or a value that changes based on the operating mode of the smart device 10 and / or the shooting scene.

[0144] In step ST40, if the distance to the face of the person represented by the specific facial image is within the prescribed range, the determination is negative, and the imaging process proceeds to step ST28. In step ST40, if the distance to the face of the person represented by the specific facial image is outside the prescribed range, the determination is positive, and the imaging process proceeds to step ST42.

[0145] In step ST42, the control unit 93 controls the motor 162 via the motor driver 164 so as to achieve a field of view angle determined by the distance to the face of the person represented by the specific facial image, thereby moving the zoom lens 30D by a specific amount along the optical axis L2. In other words, the control unit 93 moves the zoom lens 30D along the optical axis L2 to a position on the optical axis L2 that is a zoom-in position or a zoom-out position determined based on the distance measurement result obtained by the distance measurement operation.

[0146] Here, the specific movement amount is, for example, derived by the control unit 93 from a movement amount derivation table (not shown) that associates distance with the movement amount of the zoom lens 30D, or derived from a movement amount derivation equation (not shown) that uses distance as an independent variable and the movement amount of the zoom lens 30D as a dependent variable. When the processing of step ST42 is completed, the imaging process proceeds to step ST12.

[0147] According to the smart device 100 of the second embodiment, the distance measurement camera 114 includes a zoom lens 30D that is movable along the optical axis L2. The control unit 93 moves the zoom lens 30D along the optical axis L2 to a position on the optical axis L2, which is a zoom-in position or a zoom-out position determined based on the distance measurement results obtained during the distance measurement operation. Therefore, according to this configuration, the zoom lens 30D can be moved to a position corresponding to the distance measurement results more easily than when the zoom lens 30D is manually moved.

[0148] Furthermore, in the second embodiment described above, the control unit 93 is described as moving the zoom lens 30D based on the distance measurement results obtained by the distance measurement operation. However, the technology of the present invention is not limited to this. For example, the control unit 93 may also move the zoom lens 30D based on a zoom operation received by the receiving device 47.

[0149] At this time, for example, CPU 15A executes Figure 14 The camera process shown. Figure 14 The camera processing shown is Figure 13 Compared with the imaging process shown in the figure, the difference lies in that the process of step ST140 is replaced by the process of step ST40, and the process of step ST142 is replaced by the process of step ST42.

[0150] exist Figure 14In the imaging process shown, in step ST140, the control unit 93 determines whether a zoom operation has been performed on the receiving device 47. Examples of zoom operations include pinch-in and pinch-out operations on the touch panel 48. For example, to zoom in on the face of a person represented by a facial image surrounded by a frame 29, the touch panel 48 is pinched out on the facial image, and to zoom out from the face of the person, the touch panel 48 is pinched in.

[0151] In step ST140, if the zoom operation is performed on the receiving device 47, the determination is affirmative, and the imaging process proceeds to step ST142. In step ST140, if the zoom operation is not performed on the receiving device 47, the determination is negative, and the imaging process proceeds to step ST28.

[0152] In step ST142, the control unit 93 controls the motor 162 via the motor driver 164 to move the zoom lens 30D along the optical axis L2 by an amount corresponding to the zoom operation. The imaging process then proceeds to step ST12. The angle of view is changed by executing the process of step ST142.

[0153] [Third embodiment]

[0154] As an example, as shown in FIG8 , the smart device 200 according to the third embodiment has the same structure as the smart device 10 according to the first embodiment. The following describes the differences between the imaging process of the smart device 200 according to the third embodiment and the imaging process of the smart device 10 according to the first embodiment.

[0155] As an example, Figure 15 As shown, the imaging process involved in this third embodiment is Figure 11 The imaging process shown in the figure is different in that steps ST50 and ST52 are provided between steps ST10 and ST12.

[0156] exist Figure 15 In the imaging process shown, in step ST50, the control unit 93 performs distance measurement for focus control. By executing step ST50, the light irradiator 16 irradiates the entire imaging area or a portion of the imaging area (e.g., the center), and the distance measurement partitioned area 26N2 receives the IR reflected light. The pixel data obtained by receiving the IR reflected light in the distance measurement partitioned area 26N2 is transmitted to the IR pixel data processing circuit 34B. Based on the time required from irradiation of the laser beam to reception of the IR reflected light and the speed of light, the distance from the smart device 200 to the entire imaging area or a portion of the imaging area is measured. The distance measurement result obtained in this manner is stored in the image memory 42 by the IR pixel data processing circuit 34B.

[0157] In the next step ST52, the control unit 93 moves the focus lens 30B to the focus position determined by executing step ST50 based on the distance measurement result stored in the image memory 42. As in the first embodiment described above, the control unit 93 derives the focus position from a focus position derivation table (not shown) or a focus position derivation expression (not shown).

[0158] According to the smart device 200 of the third embodiment, the ranging camera device 14 (camera unit) has a focus lens 30B that can move along the optical axis L2. Before the recognition action, the ranging camera device 14 (distance measuring unit) irradiates the imaging area with a laser beam and receives IR reflected light from the imaging area to perform distance measurement for focus control. The control unit 93 moves the focus lens 30B along the optical axis L2 to a focus position determined based on the distance measurement result obtained by the distance measurement for focus control. Therefore, according to this structure, the focus lens 30B moves to the focus position before the recognition action is performed, so the smart device 200 can improve the image quality of the visible light image obtained by parallel processing compared to the case where the focus lens 30B is not moved to the focus position before the recognition action is performed. As a result, the accuracy of image recognition of the visible light image obtained by parallel processing can be improved compared to the case where the focus lens 30B is not moved to the focus position before the recognition action is performed.

[0159] [Other embodiments]

[0160] In the above embodiments, the IR reflected light is received by the distance-measuring divided area 26N2 , but the technology of the present invention is not limited thereto. The IR reflected light may be received by a portion of the IR pixels within the distance-measuring divided area 26N2 .

[0161] At this time, for example, Figure 16 As shown, the control unit 93 causes the display 46 to display the distance measurement range designation guidance screen 102. The distance measurement range designation guidance screen 102 is a screen that guides the user to designate a distance measurement range. The distance measurement range designation guidance screen 102 displays a message (hereinafter also referred to as a "guidance message") asking the user whether to designate a distance measurement range. Figure 16 In the illustrated example, a message "Do you want to specify a distance measurement range?" is displayed as an example of a guidance message. Furthermore, soft keys 102A and 102B are displayed on distance measurement range specification guidance screen 102. When the user specifies a distance measurement range, the user activates soft key 102A via touch panel 48. When the user does not specify a distance measurement range, that is, when the user specifies the entire imaging area as the distance measurement range, the user activates soft key 102B via touch panel 48.

[0162] When the distance measurement range specification guidance screen 102 is displayed on the display 46, the user or the like turns on the soft key 102A via the touch panel 48. For example, Figure 17 As shown, the control unit 93 causes the display 46 to display the visible light image as a live view image. When the live view image is displayed on the display 46, the user or the like specifies the image area (in the Figure 17 In the example shown, a rectangular area enclosed by a dotted line on the visible light image. A real space area corresponding to an image area designated by a user or the like (hereinafter also referred to as a "designated image area") is designated as a distance measurement target by the distance measurement camera 14.

[0163] At this time, as an example, Figure 18 As shown, area position identification information (e.g., coordinates) capable of identifying the position of the designated image area on the visible light image is output from the touch panel 48 to the control unit 93. The control unit 93 outputs the divided area position information (e.g., pixel address) capable of identifying the position of the designated divided area 26N2a for ranging in the divided area 26N2 for ranging to the photoelectric conversion element driver 32. Here, the designated divided area 26N2a for ranging is the divided area at a position in the divided area 26N2 for ranging that corresponds to the position of the designated image area determined based on the area position identification information input from the touch panel 48.

[0164] During the distance measurement operation, the photoelectric conversion element driver 32 drives only the designated distance measurement divided area 26N2a within the distance measurement divided area 26N2. This causes the distance measurement camera devices 14 and 114 to perform distance measurement using IR reflected light received only by the designated distance measurement divided area 26N2a. In other words, distance measurement is performed using IR reflected light received only by at least one IR pixel included in the designated area among the plurality of IR pixels.

[0165] exist Figures 16 to 18 In the example shown, the distance measurement is performed by changing the distance measurement divided area 26N2 to the distance measurement designated divided area 26N2a, thereby narrowing down the distance measurement target (so-called ROI) designated by the user. However, the method of narrowing down the distance measurement target is not limited to this. For example, the control unit 93 may control the laser beam diameter and / or direction to be changed based on an instruction received by the receiving device 47, so as to irradiate the laser beam onto the distance measurement target designated by the user.

[0166] In the above embodiments, the smart devices 10, 100 and 200 equipped with the light receiver 18 are exemplified, but the technology of the present invention is not limited thereto. Figure 19 As shown, it can also be a smart device 300 equipped with light receivers 18 and 350. As an example, Figure 19As shown, a light-transmitting window 352 is provided adjacent to the light-transmitting window 22 at the upper left portion of the rear surface 12A of the housing 12 when the smart device 300 is in a vertical position (the upper left portion when viewed from behind the smart device 300 in the vertical position). Like the light-transmitting windows 20 and 22, the light-transmitting window 352 is a light-transmitting optical element (e.g., a lens). The light-transmitting windows 20, 22, and 352 are arranged at predetermined intervals along the horizontal direction. Like the light-transmitting windows 20 and 22, the light-transmitting window 352 is exposed from the rear surface 12A.

[0167] The photoreceiver 350 includes a photoelectric conversion element 354. The photoelectric conversion element 354 is a specialized photoelectric conversion element for receiving IR reflected light and has a plurality of IR pixels arranged in a matrix. An example of a plurality of IR pixels is a photodiode for IR pixels (e.g., InGaAs APD) with a pixel count of "4896×3265". The photoelectric conversion element 354 receives the IR reflected light from the photoreceiver 350 via the light-transmitting window 352 and outputs an electrical signal corresponding to the amount of the received IR reflected light to the signal processing circuit 34 (see FIG. 1 ). Figure 8 ).

[0168] And, in Figure 19 In the example shown, a smart device 300 equipped with light receivers 18 and 350 is shown, but the technology of the present invention is not limited to this. Figure 20 As shown, it can also be a smart device 400 equipped with light receivers 18 and 450. As an example, Figure 20 As shown, a light-transmitting window 452 is provided adjacent to light-transmitting window 20 at the upper right portion of rear surface 12A of housing 12 when smart device 400 is in a vertical position (the upper right portion when viewed from behind in a vertical position). Light-transmitting window 452, like light-transmitting windows 20 and 22, is a light-transmitting optical element (e.g., a lens). Light-transmitting windows 452, 20, and 22 are arranged horizontally at predetermined intervals. Light-transmitting window 452, like light-transmitting windows 20 and 22, is also exposed from rear surface 12A.

[0169] The light receiver 450 includes a single photodiode 454. The photodiode 454 is, for example, a photodiode capable of receiving IR reflected light. An example of the photodiode 454 is an InGaAs APD. The photodiode 454 receives the IR reflected light from the light receiver 450 via the light-transmitting window 452 and outputs an electrical signal corresponding to the amount of the received IR reflected light to the signal processing circuit 34 (see FIG. 1 ). Figure 8 ).

[0170] In the above-mentioned embodiments, a laser beam is cited as an example of "light" involved in the technology of the present invention, but the technology of the present invention is not limited to this. Superradiant light can also be used instead of the laser beam, as long as distance measurement is performed using light with directionality that can measure distance.

[0171] In the above embodiments, the distance measuring camera 14 is described as being built into the smart device 10, but the technology of the present invention is not limited thereto. Figure 21 As shown, the distance measuring camera 14 may be installed outside a normal smart device 500 , that is, a smart device 500 that does not have the distance measuring camera 14 built therein.

[0172] Furthermore, in the above-described embodiments, the UI device 44 is described as being assembled in the smart device 10. However, at least some of the multiple components included in the UI device 44 may be installed outside the smart device 10. Furthermore, at least some of the multiple components included in the UI device 44 may be used as independent units by being connected to the external I / F 52.

[0173] And, in Figure 1 While the example shown illustrates a smart device 10, the technology of the present invention is not limited thereto. Specifically, the technology of the present invention can be applied to various electronic devices (e.g., interchangeable-lens cameras, fixed-lens cameras, personal computers, and / or wearable terminal devices) that have a built-in range-finding camera 14. Even these electronic devices can achieve the same functions and effects as the smart device 10.

[0174] Furthermore, in the above embodiments, the display 46 is exemplified, but the technology of the present invention is not limited thereto. For example, an independent display later installed in the smart device 10 may be used as the "display unit" involved in the technology of the present invention.

[0175] Furthermore, in the above embodiment, an example in which the recognition unit 94 is mounted on the smart device 10 is given for explanation, but the technology of the present invention is not limited to this. For example, an external device (e.g., another smart device, a personal computer, and / or a server, etc.) having the recognition unit 94 may be connected to the smart device 10. In this case, the image recognition result provided from the external device to the smart device 10 can be obtained by the CPU 15A of the smart device 10. Furthermore, cloud computing (not shown) can be made to assume the function of the recognition unit 94, and the image recognition result can be provided to the smart device 10 from the cloud computing. In this case, the image recognition result provided from the cloud computing to the smart device 10 can be obtained by the CPU 15A of the smart device 10.

[0176] Furthermore, in the above-mentioned embodiments, the example in which the image processing program 70 is stored in the storage device 15B is described, but the technology of the present invention is not limited to this. Figure 22 As shown, the image capture processing program 70 may be stored in a storage medium 900. An example of the storage medium 900 is any portable storage medium such as an SSD or a USB memory.

[0177] The image pickup processing program 70 stored in the storage medium 900 is installed in the controller 15 . The CPU 15A executes the image pickup processing according to the image pickup processing program 70 .

[0178] Furthermore, the imaging processing program 70 may be stored in a storage unit of another computer or server device connected to the controller 15 via a communication network (not shown), downloaded in response to a request from the smart device 10 , and installed in the controller 15 .

[0179] Furthermore, it is not necessary to store all of the imaging processing program 70 in the storage unit of another computer or server device connected to the controller 15 or the storage device 15B, and a portion of the imaging processing program 70 may be stored.

[0180] exist Figure 22 In the illustrated example, the controller 15 is built into the smart device 10 , but the technology of the present invention is not limited thereto. For example, the controller 15 may be provided outside the smart device 10 .

[0181] exist Figure 22 In the example shown, the CPU 15A is a single CPU, but it may be a plurality of CPUs. Furthermore, a GPU may be used in place of the CPU 15A.

[0182] exist Figure 22 In the example shown, the controller 15 is illustrated, but the technology of the present invention is not limited thereto, and a device including an ASIC, FPGA, and / or PLD may be applied instead of the controller 15. Furthermore, a combination of hardware and software structures may be used instead of the controller 15.

[0183] As hardware resources for executing the imaging processing described in the above embodiments, various processors described below can be used. Examples of processors include general-purpose processors (CPUs) that execute software (programs) to function as hardware resources for executing imaging processing. Furthermore, examples of processors include processors (specialized circuits) with circuit structures specifically designed to execute specific processing, such as FPGAs, PLDs, and ASICs. All processors have built-in or connected memory, and all processors execute imaging processing using memory.

[0184] The hardware resource for executing the image processing can be composed of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resource for executing the image processing can also be a single processor.

[0185] Examples of systems comprised of a single processor include, firstly, a system in which a single processor is composed of a combination of one or more CPUs and software, with the processor functioning as a hardware resource for executing image processing. Secondly, a system in which a processor, such as a SoC, implements an entire system including multiple hardware resources for executing image processing on a single IC chip. In this manner, image processing is implemented as a hardware resource using one or more of the various processors described above.

[0186] Furthermore, as the hardware structure of these various processors, more specifically, circuits combining circuit elements such as semiconductor devices can be utilized. Furthermore, the above-described imaging process is merely an example. Therefore, it is of course possible to delete unnecessary steps, add new steps, or change the processing order without departing from the scope of the present invention.

[0187] The above-mentioned records and illustrated contents are detailed descriptions of the parts involved in the technology of the present invention, and are merely examples of the technology of the present invention. For example, the descriptions related to the above-mentioned structure, function, action and effect are descriptions related to an example of the structure, function, action and effect of the parts involved in the technology of the present invention. Therefore, it is undoubted that unnecessary parts can be deleted, new elements can be added, or replacements can be made in the above-mentioned records and illustrated contents without departing from the scope of the purpose of the technology of the present invention. Furthermore, in order to avoid complication and facilitate understanding of the parts involved in the technology of the present invention, descriptions related to technical common sense that does not need to be particularly explained in the implementation of the technology of the present invention have been omitted in the above-mentioned records and illustrated contents.

[0188] In this specification, "A and / or B" has the same meaning as "at least one of A and B." That is, "A and / or B" can mean just A, just B, or a combination of A and B. Furthermore, in this specification, when "and / or" is used to express a combination of three or more items, the same concept as "A and / or B" applies.

[0189] All documents, patent applications, and technical specifications described in this specification are incorporated herein by reference in the same manner as if each document, patent application, or technical specification were specifically and individually described as being incorporated by reference.

[0190] Regarding the above-mentioned embodiments, the following appendix is ​​also disclosed.

[0191] (appendix)

[0192] A processing device comprising:

[0193] processor; and

[0194] Memory, connected to or built into the above processor,

[0195] The processor performs a process comprising the following steps:

[0196] Control is performed to perform a plurality of times in parallel a recognition operation, wherein the recognition unit recognizes a specific subject included in the imaging area based on a captured image obtained by the imaging unit capturing the imaging area, and a distance measurement operation, wherein the distance measurement unit irradiates the imaging area with light and receives light reflected from the imaging area; and

[0197] The irradiation energy of the light with respect to the imaging area is changed for each of the distance measuring operations.

Claims

1. A processing device comprising: processor; and memory, connected to or built into the processor, The processor performs the following processing: Control is performed to perform a plurality of recognition operations and distance measurement operations in parallel, wherein in the recognition operation, the recognition processor recognizes a specific subject included in the imaging area based on a captured image obtained by capturing the imaging area by an imaging device, and in the distance measurement operation, the distance measurement is performed by irradiating the imaging area with light by the distance measurement device and receiving light reflected from the light relative to the imaging area. changing the irradiation energy of the light to the imaging area for each of the ranging operations, Whenever the recognition operation and the distance measurement operation are performed in parallel, a recognition result based on the recognition operation and a distance measurement result based on the distance measurement operation are integrated.

2. The processing device according to claim 1, wherein The processor ranks the rounds, ie, the execution times, of performing the recognition action and the ranging action in parallel using a ranking value obtained by integrating the recognition result and the ranging result.

3. The processing device according to claim 2, wherein The ranking value is a value obtained by integrating a value obtained by assigning a first weight indicating how good the recognition result is to the recognition result and a value obtained by assigning a second weight indicating how good the ranging result is to the ranging result.

4. The processing device according to claim 2, wherein the processor, Select the execution times according to the rating value, A first process is executed using the captured image, the recognition result, and / or the distance measurement result obtained at the selected execution time.

5. The processing device according to claim 4, wherein The processor selects the execution time having the highest rating value among the plurality of execution times.

6. The processing device according to claim 4, wherein The first processing includes camera processing, display processing and / or storage processing, The photographing process is a process of photographing using the recognition result and / or the distance measurement result. The display processing is processing of displaying the captured image, the recognition result and / or the distance measurement result on a display device. The storage process is a process of storing the captured image, the recognition result, and / or the distance measurement result in a storage area.

7. The processing device according to claim 6, wherein: The camera processing includes focusing processing, The focusing process is a process of performing focusing using the recognition result and / or the distance measurement result.

8. The processing device according to claim 1, wherein The processor outputs a recognition result obtained by the recognition action and a distance measurement result obtained by the distance measurement action to a specific output target according to a specific time among the multiple times.

9. The processing device according to claim 8, wherein The specific output object is a display capable of displaying at least one of the recognition result and the distance measurement result.

10. The processing device according to claim 9, wherein The display displays a specific subject image representing the specific subject and an image surrounding the specific subject image as a result of the recognition.

11. The processing device according to any one of claims 1 to 10, wherein: The imaging device includes a lens movable along an optical axis. The processor moves the lens along the optical axis to a position determined as a position on the optical axis in accordance with a distance measurement result obtained by the distance measurement operation.

12. The processing device according to claim 11, wherein The position is the focus position.

13. The processing device according to any one of claims 1 to 10, wherein The imaging device includes a lens movable along an optical axis. The distance measuring device performs distance measurement for focus control by irradiating the imaging area with focus control light and receiving focus control reflected light based on the focus control light with respect to the imaging area before the recognition operation. The processor moves the lens along the optical axis to a focus position determined according to a distance measurement result obtained by the distance measurement for focus control.

14. The processing device according to any one of claims 1 to 10, wherein The processor performs determination processing using a plurality of recognition results obtained by the plurality of recognition operations.

15. The processing device according to claim 14, wherein The determination process is a process of outputting the recognition result to the outside.

16. The processing device according to claim 14, wherein The determination process is a process of creating a file of a specific format of the recognition result and the captured image.

17. The processing device according to any one of claims 1 to 10, wherein The recognition result obtained by the recognition operation is weighted according to at least one of the specific object and the irradiation energy used in the distance measurement operation performed in parallel with the recognition operation.

18. The processing device according to any one of claims 1 to 10, wherein The recognition result obtained by the recognition operation is weighted according to the irradiation energy used in the distance measurement operation performed in parallel with the recognition operation.

19. The processing device according to claim 17, wherein The weight of the recognition result when the irradiation energy is the first irradiation energy is smaller than the weight of the recognition result when the irradiation energy is the second irradiation energy smaller than the first irradiation energy.

20. The processing device according to claim 17, wherein The processor causes the imaging device to perform imaging by main exposure based on the recognition result selected according to the weighting.

21. The processing device according to any one of claims 1 to 10, wherein The specific subject is a face.

22. The processing device according to claim 21, wherein The face is a face with a specific expression.

23. The processing device according to any one of claims 1 to 10, wherein The specific subject is an object having a reflectivity lower than a threshold value.

24. An electronic device comprising: The processing device according to any one of claims 1 to 23; and At least one of the identification processor and the distance measuring device.

25. A treatment method comprising the following steps: performing control to perform a plurality of recognition operations and a ranging operation in parallel, wherein the recognition operation comprises a recognition processor recognizing a specific subject contained in the imaging area based on a captured image obtained by an imaging device capturing the imaging area, and the ranging operation comprises a ranging operation wherein the ranging device irradiates the imaging area with light and receives light reflected from the imaging area; changing the irradiation energy of the light on the imaging area for each of the ranging operations; and Whenever the recognition operation and the distance measurement operation are performed in parallel, a recognition result based on the recognition operation and a distance measurement result based on the distance measurement operation are integrated.

26. A computer-readable non-volatile storage medium storing a program for causing a computer to execute a process comprising the following steps: performing control to perform a plurality of recognition operations and a ranging operation in parallel, wherein the recognition operation comprises a recognition processor recognizing a specific subject contained in the imaging area based on a captured image obtained by an imaging device capturing the imaging area, and the ranging operation comprises a ranging operation wherein the ranging device irradiates the imaging area with light and receives light reflected from the imaging area; changing the irradiation energy of the light on the imaging area for each of the ranging operations; Whenever the recognition operation and the distance measurement operation are performed in parallel, a recognition result based on the recognition operation and a distance measurement result based on the distance measurement operation are integrated.

Images