Control apparatus, image pickup apparatus, control method, storage medium, and computer program product

Abstract

The invention relates to a control apparatus, an image pickup apparatus, a control method, a storage medium, and a computer program product. The control apparatus is configured to control a first optical system for acquiring image information and a second optical system different from the first optical system. The control apparatus includes: a memory to store instructions; and a processor configured to execute instructions to: acquire at least one of first distance information and second distance information, the first distance information corresponding to the image information and obtained by using the first optical system; the second distance information corresponds to the image information and is obtained by using the second optical system, and in a case where a reliability of the first distance information is higher than a predetermined value, the first optical system is controlled to acquire the first distance information, and in a case where the reliability is lower than the predetermined value, the second optical system is controlled to acquire the second distance information. And controlling the second optical system to obtain the second distance information.

Description

Technical Field

[0001] One of the aspects of the embodiments relates to a control device, an imaging device, a control method, and a storage medium. Background Art

[0002] As a method for measuring the distance to an object, conventionally known are passive ranging methods for measuring the distance using image information and active ranging methods for measuring the distance by emitting auxiliary light. It is difficult for the passive ranging method to obtain the distance to a distant object with high accuracy. A known example of an active ranging method is light detection and ranging (LiDAR) based on the period from when an infrared laser beam is irradiated onto the object to when reflected light is received from the object to measure the distance to the object. LiDAR can obtain distance information with high accuracy regardless of the distance to the object, but consumes more power than the passive ranging method. Each ranging method has advantages and disadvantages. PCT International Publication WO 2015 / 083539 discloses a configuration for selecting any one of an active ranging method and a passive ranging method based on the average brightness of a captured image.

[0003] The configuration disclosed in PCT International Publication WO 2015 / 083539 does not switch from a passive ranging method to an active ranging method when acquiring the distance to a low-contrast object or acquiring the distance to an object in a blurred state (i.e., out of focus on the entire screen), so its ranging accuracy is reduced. Summary of the invention

[0004] A control device according to one aspect of the present disclosure is configured to control a first optical system for acquiring image information and a second optical system different from the first optical system. The control device includes: a memory for storing instructions; and a processor for executing the instructions to: acquire at least one of first distance information corresponding to the image information and obtained by using the first optical system, and second distance information corresponding to the image information and obtained by using the second optical system, and in the case where the reliability of the first distance information is higher than a predetermined value, control the first optical system to acquire the first distance information, and in the case where the reliability is lower than the predetermined value, control the second optical system to acquire the second distance information. An imaging device having the above-mentioned control device, a control method corresponding to the above-mentioned control device, and a storage medium storing a program for causing a computer to execute the above-mentioned control method also constitute other aspects of the present disclosure.

[0005] Further features of various embodiments of the present disclosure will become apparent from the following description of the embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram of the configuration of the image pickup apparatus according to the first embodiment.

[0007] Figure 2A , Figure 2B and Figure 2C The image sensor according to the first embodiment has been described.

[0008] Figure 3 is a cross-sectional view showing an imaging relationship of an optical image on the image sensor according to the first embodiment.

[0009] Figure 4A and Figure 4B The LiDAR ranging unit according to the first embodiment has been described.

[0010] Figure 5 is a flowchart showing the operation of the image pickup apparatus according to the first embodiment.

[0011] Figure 6 is a flowchart illustrating the second distance information acquisition process according to the first embodiment.

[0012] Figure 7 is a timing chart showing the timing of acquiring distance information according to the second embodiment.

[0013] Figure 8 is a block diagram of the configuration of an image pickup apparatus according to the third embodiment.

[0014] Fig. 9 is a flowchart showing the second distance information acquisition process according to the third embodiment.

[0015] Fig.10 is a block diagram of the configuration of an image pickup apparatus according to a fourth embodiment. DETAILED DESCRIPTION

[0016] Hereinafter, the term "unit" may refer to a software context, a hardware context, or a combination of software and hardware contexts. In the software context, the term "unit" refers to functionality, applications, software modules, functions, routines, instruction sets, or programs that can be executed by a programmable processor such as a microprocessor, a central processing unit (CPU), or a specially designed programmable device or controller. The memory contains instructions or programs, wherein the instructions or programs, when executed by the CPU, cause the CPU to perform operations corresponding to the unit or function. In the hardware context, the term "unit" refers to a hardware element, circuit, assembly, physical structure, system, module, or subsystem. According to a specific embodiment, the term "unit" may include mechanical, optical, or electronic components, or any combination thereof. The term "unit" may include active (e.g., transistors) or passive (e.g., capacitors) components. The term "unit" may include a semiconductor device having a substrate and other material layers, wherein the other material layers have different conductivity concentrations. The term "unit" may include a CPU or a programmable processor that can execute a program stored in a memory to perform a specific function. The term "unit" may include a logic element (e.g., AND, OR) implemented by a transistor circuit or any other switching circuit. In the combination of software and hardware contexts, the term "unit" or "circuit" refers to any combination of software and hardware contexts as described above. In addition, the term "element", "assembly", "component" or "device" may also refer to a "circuit" that is integrated with or not integrated with packaging materials.

[0017] Now, with reference to the accompanying drawings, a detailed description will be given of embodiments according to the present disclosure. Corresponding elements in the various drawings will be indicated by the same reference numerals, and repeated descriptions thereof will be omitted.

[0018] As a method of measuring the distance to an object, a passive distance measurement method that measures the distance using image information and an active distance measurement method that measures the distance by emitting auxiliary light are conventionally known.

[0019] The phase difference detection method as a passive distance measurement method arranges phase difference detection pixels that detect signals with different phases on an image sensor, and acquires distance information about a subject by calculating correlation between signals with different phases.

[0020] The active ranging method measures the distance to the object (subject) based on the time difference between the timing of light emission from an infrared laser and the detection timing of reflected light from the object (round trip time of the infrared laser beam). For example, the sensor that detects the reflected light can use a single photon avalanche diode (SPAD) sensor that can detect a single photon. The SPAD sensor detects the incident single photon as a very short detection pulse by avalanche multiplication. In the technology actually used, the time between the timing of light emission from the infrared laser and the timing of the detection pulse is measured by a time-to-digital converter (TDC). In fact, since the arrival time point of a photon fluctuates greatly, the infrared laser emission and single photon detection are repeated periodically, and the time measurement results are plotted into a histogram and statistically processed. As a result, the accuracy of time difference measurement (in other words, ranging (hereinafter referred to as LiDAR ranging)) can be improved. In addition, the infrared laser and SPAD sensor can be arranged in two dimensions to plot the results of LiDAR ranging in two dimensions, so that a so-called distance map can be generated. Recent imaging devices can realize generation of a three-dimensional computer graphics model and a plan view (hereinafter collectively referred to as a space model or a D model) and automatic focusing (AF) by using a distance map and a captured image.

[0021] First embodiment

[0022] This embodiment will discuss an example of determining the reliability of distance information obtained by a passive ranging method and switching the passive ranging method to an active ranging method when the reliability is low. More specifically, when the reliability of the distance information obtained by using the first optical system is low, the distance information is obtained by using the second optical system.

[0023] Figure 1 1 is a block diagram of the configuration of the imaging device 100 according to the present embodiment. The imaging device 100 is, for example, a digital camera, a smart phone, or a drone. The imaging device 100 includes a control unit 101, an imaging lens 102, an image sensor (imaging unit) 103, a sensor corrector 104, an image shift calculator 106, a reliability judgment unit 107, a defocus converter 108, and a lens driving unit 109. The imaging device 100 also includes a LiDAR ranging unit 110, a LiDAR corrector 113, a histogram calculator 114, and a viewpoint position corrector 115. The imaging lens 102, the image sensor 103, and the image shift calculator 106 are used as a first optical system, and acquire image information and first distance information corresponding to the image information. The LiDAR ranging unit 110 is used as a second optical system, and acquires second distance information corresponding to the image information acquired by using the first optical system.

[0024] Reference numeral s101 denotes incident light related to imaging, reference numeral s102 denotes visible light raw data, reference numeral s103 denotes image signals after various corrections, reference numeral s105 denotes an image shift amount, reference numeral s106 denotes a reliability determination result, and reference numeral s107 denotes a defocus amount. Reference numeral s108 denotes a laser beam, reference numeral s109 denotes reflected light from an object irradiated with the laser beam s108, reference numeral s110 denotes LiDAR ranging information, reference numeral s111 denotes various corrected distance information, reference numeral s112 denotes a distance map, and reference numeral s113 denotes distance information corrected for a viewpoint position. Reference numeral s114 denotes a lens drive amount.

[0025] The control unit 101 is a control device configured to control the entire imaging apparatus 100. The control unit 101 receives the reliability determination result s106 from the reliability determination unit 107, and controls the LiDAR ranging unit 110. The control unit 101 performs calculation processing and control processing according to various computer programs stored in a memory not shown.

[0026] The control unit 101 includes an acquisition unit 101a and an optical system control unit 101b. The acquisition unit 101a acquires at least one of the first distance information and the second distance information. The first distance information corresponds to the image information and is obtained by using the first optical system. The second distance information corresponds to the image information and is obtained by using the second optical system. In the case where the reliability of the first distance information is higher than a predetermined value, the optical system control unit 101b controls the first optical system so that the acquisition unit 101a acquires the first distance information. In the case where the reliability of the first distance information is lower than a predetermined value, the optical system control unit 101b controls the second optical system so that the acquisition unit 101a acquires the second distance information. In the case where the reliability of the first distance information is equal to the predetermined value, the optical system to be controlled can be arbitrarily set.

[0027] In other words, in the present embodiment, at least one processor functions as the acquisition unit 101a and the optical system control unit 101b when executing a computer program stored in at least one memory. More specifically, at least one processor performs a process for acquiring at least one of the first distance information and the second distance information and a process for controlling any one of the first optical system and the second optical system according to a duration during which one of the first distance information and the second distance information cannot be acquired.

[0028] The imaging lens 102 focuses incident light s101 onto the image sensor 103. The imaging lens 102 performs AF control by moving based on a lens driving amount s114 from the lens driving unit 109. The first optical system including the imaging lens 102 and the image sensor 103 shares at least a portion of the viewing angle with the second optical system including the LiDAR ranging unit 110. More specifically, the first optical system and the second optical system photograph at least one same object.

[0029] The image sensor 103 includes a plurality of pixels (each pixel includes a microlens and a photoelectric converter), and generates visible light raw data s102 by photoelectrically converting an image formed by the imaging lens 102 .

[0030] Figure 2A , Figure 2B and Figure 2C The image sensor 103 according to the present embodiment has been described. Figure 2A 2 shows the configuration of the image sensor 103. The image sensor 103 includes a pixel array 201, a vertical scanning circuit 202, a horizontal scanning circuit 203, and a timing generator TG204. The pixel array 201 includes a plurality of unit pixel units arranged in a two-dimensional matrix of rows and columns. The timing generator TG204 generates a timing of a photographing duration or a forwarding duration, etc., and transmits a timing signal to the vertical scanning circuit 202 and the horizontal scanning circuit 203. At the timing when the photographing duration ends, the vertical scanning circuit 202 transmits a signal output from the unit pixel unit to the vertical transmission path. The horizontal scanning circuit 203 sequentially outputs the accumulated signal to the outside through the output transmission path.

[0031] Figure 2B A unit pixel unit 205 is shown in the pixel array 201. The unit pixel unit 205 includes a microlens 206 and a pair of photoelectric converters 207a and 207b. The photoelectric converters 207a and 207b perform pupil division by receiving light beams that have passed through the common microlens 206 and mutually different pupil regions at the exit pupil of the imaging lens 102.

[0032] Figure 2C Pixel array 201 is shown. In image sensor 103, a plurality of unit pixel units are two-dimensionally arranged in row and column directions in pixel array 201 to provide a two-dimensional image signal. Unit pixel units 208, 209, 210, and 211 are arranged in a two-dimensional manner in row and column directions in pixel array 201 to provide a two-dimensional image signal. Figure 2B The photoelectric converters 208L, 209L, 210L and 211L correspond to the unit pixel unit 205 in FIG. Figure 2B The photoelectric converters 208R, 209R, 210R and 211R correspond to the photoelectric converter 207a in FIG. Figure 2B Corresponding to the photoelectric converter 207b in.

[0033] Reference now Figure 3 , a description will be given of the imaging relationship of the optical image (subject image) on the image sensor 103. Figure 3 1 is a cross-sectional view showing the imaging relationship of an optical image on the image sensor 103, and conceptually shows a situation where a light beam emitted from the exit pupil of the imaging lens 102 enters the image sensor 103. Reference numeral 301 denotes a microlens, and reference numeral 302 denotes a color filter. Reference numeral 303 denotes the exit pupil of the imaging lens 102.

[0034] The light beam emitted from the exit pupil 303 enters the image sensor 103 in a state centered on the optical axis 306. Reference numerals 304 and 305 denote partial regions of the exit pupil 303. Reference numerals 307 and 308 denote outermost rays passing through the partial region 304 of the exit pupil 303, and reference numerals 309 and 310 denote outermost rays passing through the partial region 305 of the exit pupil 303.

[0035] like Figure 3 As shown, among the light beams emitted from the exit pupil 303, the light beam above the optical axis 306 enters the photoelectric converter 207b, and the light beam below the optical axis 306 enters the photoelectric converter 207a. That is, the photoelectric converters 207a and 207b receive light passing through different regions of the exit pupil 303. The phase difference is detected by utilizing this characteristic.

[0036] Reference Figure 2C , a description will be given of the phase difference detection method. The photoelectric converter 207a in the unit pixel unit 205 serves as an A image pixel group that performs photoelectric conversion on the A image in a pair of subject images used for focus detection by the phase difference detection method. The photoelectric converter 207b serves as a B image pixel group that performs photoelectric conversion on the B image in a pair of subject images.

[0037] exist Figure 2CIn the pixel array 201, the A image pixel group is the row 212 to which the photoelectric converters 208L to 211L ... are referred to, and the B image pixel group is the row 213 to which the photoelectric converters 208R to 211R ... are referred to. The phase difference signal can be acquired by calculating the correlation between the signal obtained from the A image pixel group and the signal obtained from the B image pixel group. Rows such as rows 212 and 213 (from which the phase difference signal is output to the image shift calculator 106) will be referred to as phase difference detection pixel rows. AF that performs focus detection by the phase difference detection method in this way by using the A image pixel group and the B image pixel group set in the image sensor 103 will be referred to as imaging surface phase difference AF. On row 214, the image signal can be read by adding the signals from the two photoelectric converters of each unit pixel unit. Rows such as row 214 (from which the image signal is output to the sensor corrector 104) will be referred to as normal pixel rows. Each unit pixel unit on the normal pixel row may include only one photoelectric converter instead of two divided photoelectric converters.

[0038] Any method other than the above method in this embodiment may be used as a phase difference detection method. For example, a shading unit and a focus detection pixel may be arranged below a microlens that performs pupil division, and outputs from two types of focus detection pixels corresponding to different opening positions of the shading unit may be combined to form a pair of image signals of a subject image.

[0039] The sensor corrector 104 performs various correction processes such as shading correction and black level correction on the signal output from the image sensor 103 .

[0040] The image shift amount calculator 106 acquires information related to the distance to the subject (first distance information) by calculating the correlation between the image signals of the light beams received in different incident directions. In other words, the image shift amount calculator 106 functions as a distance measuring unit configured to acquire the first distance information based on the output from the phase difference detection pixel. The image shift amount calculator 106 calculates the image shift amount s105 based on the correlation calculation result.

[0041] The reliability judgment unit 107 judges the reliability of the image shift amount s105 output from the image shift amount calculator 106, thereby judging the reliability of the first distance information acquired by the image shift amount calculator 106. The reliability judgment unit 107 outputs the reliability judgment result s106 to the control unit 101. In the present embodiment, the reliability judgment unit 107 uses the contrast value of the captured image to judge the reliability of the image shift amount s105. More specifically, in the case where the contrast value of the captured image is higher than a predetermined contrast value, the reliability judgment unit 107 judges that the reliability of the image shift amount s105 is higher than the predetermined value, and in the case where the contrast value of the captured image is lower than the predetermined contrast value, the reliability judgment unit 107 judges that the reliability of the image shift amount s105 is lower than the predetermined value.

[0042] The defocus converter 108 calculates a defocus amount s107 by multiplying the image shift amount s105 output from the image shift amount calculator 106 by a predetermined conversion coefficient.

[0043] The lens driving unit 109 calculates a lens driving amount s114 by which the imaging lens 102 is to be moved, by using the defocus amount s107 from the defocus converter 108 or the distance information s113 from the viewpoint position corrector 115 .

[0044] The LiDAR ranging unit 110 includes a laser transmitter 112 and a laser receiver 111 .

[0045] Reference now Figure 4A and Figure 4B , a description will be given of the LiDAR ranging unit 110. Figure 4A and Figure 4B The LiDAR ranging unit 110 will be described. Figure 4A A laser receiver 111 is shown, and Figure 4B A laser emitter 112 is shown.

[0046] The laser emitter 112 includes a plurality of laser emitting elements 404 arranged two-dimensionally in the horizontal and vertical directions, and emits an infrared laser beam to the outside according to a laser pulse control signal from the control unit 101 .

[0047] The laser receiver 111 includes a plurality of two-dimensionally arranged SPAD elements 402 corresponding to the laser emitting elements 404, respectively, and generates LiDAR ranging information s110 by receiving reflected light of an infrared laser beam that has been emitted from the laser emitter 112 and irradiated onto an object. Ideally, distance information can be acquired by arranging one SPAD element 402 for one laser emitting element 404, but in practice, in some cases, the reflected light is offset relative to the expected point. Therefore, in the present embodiment, for one laser emitting element 404, a SPAD element group 403, which is a set of four SPAD elements 402, is used as one SPAD element. By averaging the output results from the four SPAD elements 402, high-precision distance information can be acquired.

[0048] The LiDAR corrector 113 performs various correction processes on the LiDAR ranging information s110, such as correction of positional offset between the laser receiver 111 and the laser transmitter 112, correction related to temperature characteristics, etc. The LiDAR corrector 113 outputs distance information s111 obtained by correcting the LiDAR ranging information s110 to the histogram calculator 114.

[0049] The histogram calculator 114 improves the distance measurement accuracy by applying histogram processing to the distance information s111 , and outputs the processed distance information s111 as a two-dimensional distance map s112 having the same number of elements as the laser emitters 112 .

[0050] The viewpoint position corrector 115 generates distance information s113 by correcting the viewpoint position deviation between the LiDAR distance measuring unit 110 and the imaging lens 102 with respect to the distance map s112 .

[0051] Reference now Figure 5 , a description will be given of the operation of the image pickup apparatus 100 according to the present embodiment. Figure 5 is a flowchart showing the operation of the image pickup apparatus 100 according to the present embodiment. When a shutter button included in an operation unit not shown is pressed by the user, an operation starts. Figure 5 Processing of the flowchart in .

[0052] In step S501 , the control unit 101 acquires visible light raw data s102 by driving the image sensor 103 .

[0053] At step S502 , the control unit 101 acquires an image signal s103 by driving the sensor corrector 104 to perform various correction processes.

[0054] At step S503 , the control unit 101 acquires the image shift amount s105 by driving the image shift amount calculator 106 .

[0055] At step S504, the control unit 101 acquires a determination result indicating whether the reliability of the image shift amount s105 is high by driving the reliability determination unit 107. In the present embodiment, in a case where the contrast value of the captured image is higher than a predetermined contrast value, the reliability determination unit 107 determines that the reliability of the image shift amount s105 is higher than the predetermined value, and in a case where the contrast value of the captured image is lower than the predetermined contrast value, the reliability determination unit 107 determines that the reliability of the image shift amount s105 is lower than the predetermined value. In a case where the contrast value of the captured image is equal to the predetermined contrast value, whether the reliability of the image shift amount s105 is determined to be high or low can be arbitrarily set. The control unit 101 performs the processing of step S505 in a case where the reliability of the image shift amount s105 is higher than the predetermined value, and performs the processing of step S506 in a case where the reliability of the image shift amount s105 is lower than the predetermined value.

[0056] At step S505 , the control unit 101 acquires the defocus amount s107 by driving the defocus converter 108 to defocus-convert the image shift amount s105 .

[0057] At step S506, since the reliability of the distance information (first distance information) obtained by controlling the first optical system is low, the control unit 101 obtains the distance information (second distance information) by controlling the second optical system. More specifically, the control unit 101 performs a process for obtaining the second distance information (second distance information acquisition process) by driving the LiDAR ranging unit 110.

[0058] At step S507 , the control unit 101 acquires a lens driving amount s114 based on the defocus amount s107 acquired at step S505 or the distance information s113 acquired at step S506 by driving the lens driving unit 109 .

[0059] Reference now Figure 6 , will give Figure 5 Description of the second distance information acquisition processing at step S507 in . Figure 6 is a flowchart showing the second distance information acquisition process.

[0060] At step S601 , the control unit 101 emits an infrared laser beam to the outside at certain intervals by driving the laser emitter 112 .

[0061] At step S602 , the control unit 101 receives reflected light from the object irradiated with the infrared laser beam at step S601 by driving the laser receiver 111 .

[0062] At step S603 , the control unit 101 acquires LiDAR ranging information s110 by extracting time-of-flight (TOF) information about an infrared laser beam that is emitted at a certain interval and then reflected by an object.

[0063] At step S604 , the control unit 101 drives the LiDAR corrector 113 to perform various correction processes on the LiDAR distance information s110 , thereby acquiring corrected distance information s111 .

[0064] At step S605 , the control unit 101 acquires a distance map s112 by driving the histogram calculator 114 to apply histogram processing to the distance information s111 and improve the distance measurement accuracy.

[0065] At step S606 , the control unit 101 corrects the viewpoint position deviation between the LiDAR ranging unit 110 and the imaging lens 102 by driving the viewpoint position corrector 115 , thereby acquiring distance information s113 .

[0066] As described above, in the case where the reliability of the distance information acquired by using the first optical system is low, the configuration according to the present embodiment can acquire the distance information with high accuracy by driving the LiDAR ranging unit 110. Compared with the case where the LiDAR ranging unit 110 is constantly driven, the configuration according to the present embodiment can also reduce power consumption.

[0067] In the present embodiment, the reliability judgment unit 107 uses the contrast value of the captured image to judge the reliability of the image shift amount s105, but the present embodiment is not limited to this example. For example, the reliability judgment unit 107 can judge the reliability of the image shift amount s105 based on whether the first optical system is in a blurred state as an out-of-focus state. In the case where the first optical system is in a blurred state, the phase difference detection method cannot correctly acquire the distance information. On the other hand, LiDAR uses a second optical system different from the first optical system, and can acquire the distance information with high accuracy without being affected by whether the first optical system is in a blurred state. Whether the first optical system is in a blurred state can be judged based on whether the focus position of the first optical system is outside a predetermined range. More specifically, in the case where the focus position of the first optical system is in a predetermined range, it can be judged that the first optical system is not in a blurred state, and in the case where the focus position of the first optical system is outside a predetermined range, it can be judged that the first optical system is in a blurred state.

[0068] Second embodiment

[0069] This embodiment will discuss an example of acquiring distance information using the second optical system during the period in which the first optical system acquires image information (still image) for recording. The configuration of the image pickup apparatus 100 according to this embodiment is the same as the configuration of the image pickup apparatus 100 according to the first embodiment, and this embodiment will discuss only the configuration different from the configuration of the first embodiment, and the description of the common configuration will be omitted.

[0070] Reference now Figure 7 , a description will be given of the operation of the image pickup apparatus 100 according to the present embodiment. Figure 7 : is a timing chart showing the timing of acquiring distance information according to the present embodiment. This embodiment will discuss an example of acquiring a live view image at 120fps (frames per second: frameper sec) and acquiring a still image for recording at 30fps. The live view image is an image displayed on an electronic viewfinder (EVF) not shown before actual imaging is performed.

[0071] Ordinary digital cameras use live view images to acquire distance information to be used for AF because the number of pixels required is not large compared to still images. That is, distance information is not acquired in a frame in which still image exposure is performed, and thus continuous distance information cannot be acquired. In this case, for example, when trying to use distance information to predict a moving subject for following a subject, defective frames may occur and the prediction accuracy of the moving subject may be reduced.

[0072] Therefore, the present embodiment drives the LiDAR ranging unit 110 included in the second optical system and acquires distance information during the still image acquisition by the first optical system. Therefore, distance information can be continuously acquired during still image exposure, and the prediction accuracy of moving objects can be improved.

[0073] The frame rate at which the distance information is acquired using the second optical system may be lower than the frame rate at which the distance information is acquired using the first optical system. As described in this embodiment, since the distance information is acquired at 120 fps by using the first optical system and at 30 fps by using the second optical system, power consumption can be reduced compared to a case where a plurality of optical systems are constantly driven at the same frame rate.

[0074] Third embodiment

[0075] This embodiment will discuss an example of detecting the remaining battery power and not performing LiDAR ranging when the detected remaining battery power is less than a predetermined power. For smartphones and digital cameras, suppressing battery consumption is one of the particularly important things. This embodiment will discuss only the configurations that are different from the configurations of the first and second embodiments, and the description of the common configurations will be omitted.

[0076] Figure 8 8 is a block diagram of the configuration of the image pickup apparatus 100 according to the present embodiment. The configuration of the image pickup apparatus 100 according to the present embodiment is basically the same as the configuration of the image pickup apparatus 100 according to the first embodiment. Unlike the image pickup apparatus 100 according to the first embodiment, the image pickup apparatus 100 according to the present embodiment includes a remaining battery level detector 801. The remaining battery level detector 801 detects the remaining power of a battery not shown that supplies power to the entire image pickup apparatus 100. Different batteries may be used for the first optical system and the second optical system.

[0077] Fig. 9 : is a flowchart showing the second distance information acquisition process according to the present embodiment.

[0078] At step S901, the control unit 101 obtains the remaining battery power by driving the remaining battery power detector 801, and determines whether the remaining battery power is greater than a predetermined power. If the control unit 101 determines that the remaining battery power is greater than the predetermined power, the control unit 101 executes the processing of step S902. Processing steps S902 to S907 are respectively Figure 6 The processing steps S601 to S606 in FIG. 1 are the same, so their description will be omitted. In the case where the control unit 101 determines that the remaining battery power is less than the predetermined power, the control unit 101 ends the process and then executes Figure 5 That is, the control unit 101 does not perform LiDAR ranging, but controls the first optical system to obtain the first distance information. The step to which the process proceeds when the remaining battery power is equal to the predetermined power can be arbitrarily set. In this embodiment, the processing of step S901 is performed before LiDAR ranging, but for example, the processing of step S901 can be performed in Figure 5 before or during treatment.

[0079] As described above, the configuration according to the present embodiment can reduce power consumption.

[0080] Fourth embodiment

[0081] This embodiment will discuss an example of detecting the amount of ambient light and not performing LiDAR ranging when the detected amount of ambient light is less than a predetermined amount value. This embodiment will discuss a configuration different from the configurations of the first to third embodiments, and description of the common configuration will be omitted.

[0082] Fig.101 is a block diagram of the configuration of the imaging device 100 according to the present embodiment. The configuration of the imaging device 100 according to the present embodiment is basically the same as the configuration of the imaging device 100 according to the first embodiment. In the present embodiment, unlike the first embodiment, the LiDAR ranging unit 110 includes an ambient light detector 1001. The ambient light detector 1001 detects the amount of ambient light. Ambient light is light other than the infrared laser beam emitted by the laser emitter 112. An IR filter not shown through which only infrared light passes is installed in the laser receiver 111, but in the case where the amount of ambient light is large, visible light other than infrared light and the like leak into the photoelectric converter and cause noise. Therefore, it is difficult to perform high-precision ranging. In addition, in the case where there are multiple devices equipped with LiDAR around, the infrared laser beam from any other device may erroneously enter the imaging device 100. Therefore, in the case where the amount of ambient light is large, it is difficult to obtain high-precision distance information by driving the LiDAR ranging unit 110 regardless of whether the light is visible or invisible.

[0083] Therefore, in this embodiment, when the amount of ambient light detected by the ambient light detector 1001 is greater than a predetermined amount, the control unit 101 does not drive the LiDAR ranging unit 110, but uses the first optical system to obtain distance information. This configuration can reduce power consumption while suppressing a decrease in ranging accuracy.

[0084] Other embodiments

[0085] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-mentioned embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0086] Although the present disclosure describes example embodiments, it is to be understood that some embodiments are not limited to the disclosed embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0087] Various embodiments can provide a control device capable of acquiring distance information with high accuracy and reduced power consumption.

Claims

1. A control device configured to control a first optical system for acquiring image information and a second optical system different from the first optical system, the control device include: A memory for storing instructions; as well as A processor that executes instructions to: acquiring at least one of first distance information corresponding to the image information and obtained by using the first optical system and second distance information corresponding to the image information and obtained by using the second optical system, and When the reliability of the first distance information is higher than a predetermined value, the first optical system is controlled to acquire the first distance information, and when the reliability is lower than the predetermined value, the second optical system is controlled to acquire the second distance information.

2. The control device according to claim 1, It is characterized in that In a case where the contrast of the image information is higher than a predetermined contrast value, the reliability is higher than the predetermined value, and in a case where the contrast is lower than the predetermined contrast value, the reliability is lower than the predetermined value.

3. The control device according to claim 1, It is characterized in that In a case where the focal position of the first optical system is within a predetermined range, the reliability is higher than the predetermined value, and in a case where the focal position is outside the predetermined range, the reliability is lower than the predetermined value.

4. The control device according to claim 1, It is characterized in that The processor is configured to, if the reliability is higher than the predetermined value and an image for recording is acquired by using the first optical system, control the second optical system to acquire the second distance information.

5. The control device according to claim 1, It is characterized in that The frame rate of acquiring the second distance information is lower than the frame rate of acquiring the first distance information.

6. The control device according to claim 1, It is characterized in that The processor is configured to control the first optical system to acquire the first distance information if the reliability is lower than the predetermined value and a remaining battery power for driving the first optical system and the second optical system is less than a predetermined power.

7. The control device according to claim 1, It is characterized in that The processor is configured to control the first optical system to acquire the first distance information if the reliability is lower than the predetermined value and the amount of ambient light different from light that has been emitted from the second optical system and then reflected by the object is greater than a predetermined amount.

8. The control device according to claim 1, It is characterized in that The processor is configured to acquire the first distance information based on outputs from phase difference detection pixels configured to detect a phase difference between images.

9. The control device according to claim 1, It is characterized in that The processor is configured to acquire the second distance information using a light transmitter configured to transmit light and a light receiver configured to receive light reflected by the object.

10. A camera device, include: A control device according to any one of claims 1 to 9; the first optical system; as well as The second optical system.

11. A control method configured to control a first optical system for acquiring image information and a second optical system different from the first optical system, the control method The following steps are involved: acquiring at least one of first distance information corresponding to the image information and obtained by using the first optical system and second distance information corresponding to the image information and obtained by using the second optical system, and When the reliability of the first distance information is higher than a predetermined value, the first optical system is controlled to acquire the first distance information, and when the reliability is lower than the predetermined value, the second optical system is controlled to acquire the second distance information. 12 . A non-transitory computer-readable storage medium for storing a computer program, the computer program causing a computer to execute the control method according to claim 11 .

13. A computer program product comprising a computer program, the computer program causing a computer to execute the control method according to claim 11.

Images