Imaging device and distance measurement method

The imaging device adjusts its distance measurement field of view in sync with the imaging angle by varying the irradiation range based on the imaging device's control unit, addressing mismatch issues and ensuring accurate distance measurements across different lens configurations and zoom levels.

WO2024228320A1PCT designated stage expired Publication Date: 2024-11-07SONY GROUP CORP
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/014492
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-01
Filing Date
2024-04-10
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Imaging devices with fixed distance measurement field of view suffer from mismatch issues when combined with interchangeable lenses or during zoom operations, leading to unnecessary processing and invalid distance measurement data due to differences between the camera's field of view and the distance measurement field of view.

Method used

An imaging device with a control unit that adjusts the irradiation range of the distance measurement module based on changes in the imaging angle of view, allowing the irradiation range to vary while maintaining a fixed irradiation direction, ensuring that the distance measurement field of view matches the imaging angle of view.

Benefits of technology

This approach optimizes distance measurement operations by reducing unnecessary processing, improving accuracy, and maintaining valid distance measurement data across the entire imaging area, even with changes in lens configurations or zoom levels.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024014492_07112024_PF_FP_ABST
    Figure JP2024014492_07112024_PF_FP_ABST
Patent Text Reader

Abstract

This imaging device comprises: an imaging unit; a distance measurement unit for obtaining distance information about the distance to a subject, the distance measurement unit having an irradiation part, which has a fixed irradiation direction, and a light-receiving part, which receives reflected light reflected from the subject after being emitted from the irradiation part; and a control unit for performing control for changing the irradiation range of light from the irradiation part in accordance with a change in imaging angle.
Need to check novelty before this filing date? Find Prior Art

Description

Imaging device and distance measurement method

[0001] The present technology relates to an imaging device and a distance measurement method using the imaging device.

[0002] Some imaging devices are equipped with a distance measurement module having a TOF (Time of Flight) sensor to detect distance information (depth information) to a subject. Patent Document 1 listed below discloses a technique related to an imaging device equipped with a distance measurement device.

[0003] JP 2016-14805 A

[0004] For example, in an active ranging module with an area ranging type, the angle of view of the ranging is fixed. Therefore, when combined with a camera with an interchangeable lens or when the camera is zoomed, a mismatch can occur between the camera's angle of view and the angle of view of the ranging. This mismatch can result in unnecessary processing. An area ranging type refers to a type in which the measuring points that can measure the distance simultaneously are distributed two-dimensionally.

[0005] For example, when a telephoto lens is attached to a camera, the camera's imaging angle of view narrows. In this case, if the ranging module's ranging angle of view is fixed, part of the ranging data will be distance information for depths outside the camera's angle of view. If distance information is used for focus control or as metadata for image depth information, the ranging data outside the camera's angle of view has no use, so it results in processing unnecessary data.

[0006] Therefore, the present disclosure proposes a technique that makes it possible to eliminate unnecessary distance measurement processing.

[0007] The imaging device according to the present technology includes a distance measuring unit that has an irradiation unit with a fixed irradiation direction and a light receiving unit that receives light reflected from the subject by the light irradiated from the irradiation unit, thereby determining distance information to the subject, and a control unit that controls changing the irradiation range of the light from the irradiation unit in accordance with changes in the imaging angle of view. In other words, the irradiation range of the irradiation unit in the distance measuring unit is not always kept constant, but is changed in accordance with the imaging angle of view. Note that an irradiation unit with a fixed irradiation direction refers to one whose irradiation direction is fixed during distance measurement. For example, even if the irradiation direction is adjustable, it is included in the above-mentioned irradiation unit as long as it does not change the direction during distance measurement.

[0008] 1 is a block diagram of a configuration example of an imaging device according to a first embodiment of the present technology. FIG. 1 is an explanatory diagram of a state in which an imaging angle of view and an illumination range for distance measurement match. FIG. 2 is an explanatory diagram of a state in which an imaging angle of view and an illumination range for distance measurement do not match. FIG. 3 is an explanatory diagram of an illumination spot pattern according to an embodiment. FIG. 4 is an explanatory diagram of a distance measurement operation according to an embodiment. FIG. 5 is an explanatory diagram of a distance measurement operation in which a frame time is changed according to an embodiment. FIG. 6 is an explanatory diagram of a distance measurement operation in which light emission intensity is changed according to an embodiment. FIG. 7 is an explanatory diagram of a distance measurement operation in which light emission intensity is changed and the frame time is kept constant according to an embodiment. FIG. 8 is a flowchart of an example of a control process in which an illumination area is changed by variable setting of a spot illumination area when a lens is attached according to an embodiment. FIG. 9 is a flowchart of an example of a control process related to a distance measurement operation when a lens is detached according to an embodiment. FIG. 10 is a block diagram of a configuration example of an imaging device according to a second embodiment. FIG. 11 is a block diagram of a configuration example in which a conversion lens is attached according to the second embodiment. FIG. 12 is an explanatory diagram of a correction process in accordance with a change in optical path length. FIG. 13 is an explanatory diagram of a correction process in accordance with a change in optical path length. FIG. 14 is a flowchart of an example of a control process in which an illumination area is changed by changing a zoom magnification according to an embodiment. 1 is a flowchart of an example of control processing for changing an illumination area by setting a zoom magnification and a spot illumination area according to an embodiment. FIG. 1 is an explanatory diagram of a difference in zoom range between an imaging unit and a ranging module. FIG. 2 is an explanatory diagram of an operation for changing an illumination area by setting a zoom magnification and a spot illumination area according to an embodiment. FIG. 1 is a flowchart of an example of control processing for changing an illumination area by setting a zoom magnification and a spot illumination area according to an embodiment. FIG. 2 is an explanatory diagram of an operation for changing an illumination area by setting a zoom magnification and a spot illumination area according to an embodiment. FIG. 3 is a flowchart of an example of processing for associating metadata according to an embodiment. FIG. 4 is an explanatory diagram of metadata contents according to an embodiment. FIG. 5 is a flowchart of changing an illumination area by a conversion lens according to an embodiment. FIG. 6 is an explanatory diagram of parameter control according to a change of an illumination area according to an embodiment.

[0009] The embodiments will be described below in the following order: <1. Configuration example of an imaging device according to a first embodiment> <2. Illumination range control by setting a spot illumination area> <3. Configuration example of an imaging device according to a second embodiment> <4. Illumination range control by changing the zoom magnification> <5. Illumination range control by setting the zoom magnification and the spot illumination area> <6. Illumination range control by a conversion lens> <7. Setting distance measurement parameters> <8. Summary and modified examples>

[0010] In this disclosure, the term "image" is used to include both still images and moving images.

[0011] 1 shows an example of the configuration of an imaging device 1 according to a first embodiment. The imaging device 1 includes an imaging unit 10, a camera signal processing unit 13, a recording unit 14, an output unit 15, a user interface unit 16, a memory unit 17, and a ranging module 20. Note that the configuration in Fig. 1 is merely an example showing the main parts of the imaging device 1, and the imaging device 1 may include components not shown, and does not necessarily include all of the components shown.

[0012] The imaging unit 10 includes a lens system 11 and an image sensor 12 (image sensor). The lens system 11 is configured as, for example, an interchangeable lens, and is detachable from the main body of the imaging device 1. However, the lens system 11 is not limited to an interchangeable lens, and may be a lens built into the main body of the imaging device 1.

[0013] The lens system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, and a drive mechanism for these optical elements. Light from a subject is guided by this lens system 11 and collected on an image sensor 12. The figure also shows a lens position encoder 11a, which outputs information on the lens positions of the focus lens and zoom lens.

[0014] The image sensor 12 is configured as, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type. The image sensor 12 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. on the electrical signal obtained by photoelectrically converting the received light, and further performs A / D (Analog / Digital) conversion processing. The captured image signal is then output as digital data to a camera signal processing unit 13 at a downstream stage.

[0015] The camera signal processing unit 13 is configured as an image processor using, for example, an LSI (Large Scale Integration), and includes, for example, a camera control unit 30, a RAW correction signal processing unit 31, a YC development signal processing unit 32, a YC codec unit 33, a RAW codec unit 34, a file generation processing unit 35, and a lens control unit 36.

[0016] The RAW correction signal processing unit 31 performs correction processing and the like on the captured image data from the image sensor 12, i.e., RAW image data. The YC development signal processing unit 32 performs various necessary development processing on the RAW image data, such as color separation processing, luminance signal processing, and color matrix processing. The YC codec unit 33 performs codec for recording and transmission on the developed captured image data. The RAW codec unit 34 performs codec for recording and transmission in the RAW image data state. The file generation processing unit 35 performs file generation processing required for recording and transmission on the developed captured image data or RAW image data.

[0017] Through the above processing, the camera signal processing unit 13 transfers image data to be recorded (developed captured image data or RAW image data) to the recording unit 14. Also, the camera signal processing unit 13 transfers image data to be transmitted and output (developed captured image data or RAW image data) to the output unit 15.

[0018] The recording unit 14 performs processing to record image files (content files) such as still image data and video data, attribute information of the image files, thumbnail images, etc., on a recording medium such as a non-volatile memory. The actual form of the recording unit 14 can be various. For example, the recording unit 14 may be a circuit unit that performs recording and playback on a flash memory built into the imaging device 1, or may be a card recording and playback unit that performs recording and playback access on a memory card (e.g., a portable flash memory) that can be attached to and detached from the imaging device 1. The recording unit 14 may also be realized as an HDD (Hard Disk Drive) or SSD (Solid State Drive) built into the imaging device 1.

[0019] The output unit 15 communicates with external devices via a wired or wireless connection. For example, the output unit 15 transmits and outputs captured image data (still image files and video files) to an external display device, recording device, playback device, etc. The output unit 15 may also be a network communication unit that communicates via various networks such as the Internet, a home network, a LAN (Local Area Network), etc., and transmits and receives various data to and from servers, terminals, etc. on the network.

[0020] The camera control unit 30 controls the entire system within the imaging device 1. The lens control unit 36 ​​performs drive control of the lenses of the lens system 11 based on instructions from the camera control unit 30. For example, it performs drive control of focus operation, zoom operation, aperture adjustment operation, etc. The lens system 11 includes motors that drive the focus lens and zoom lens, a motor that drives the aperture mechanism, and motor drive circuits that drive these motors. The lens control unit 36 ​​transmits drive control signals to the motor drive circuits. The lens control unit 36 ​​also acquires information from the lens position encoder 11a and transmits information such as the lens position, that is, the focus position and zoom angle of view, to the camera control unit 30.

[0021] In FIG. 1, the camera control unit 30 and the lens control unit 36 ​​are assumed to be built into a microprocessor that serves as the camera signal processing unit 13, but they may also be configured by a microprocessor separate from the camera signal processing unit 13.

[0022] The memory unit 17 stores information used for processing by the camera control unit 30. For example, this generically refers to a read-only memory (ROM), a random access memory (RAM), a flash memory, etc. The memory unit 17 may be a memory area built into a microcomputer chip that serves as the camera control unit 30, or may be configured as a separate memory chip.

[0023] The RAM in the memory unit 17 is used to temporarily store data, programs, etc. as a working area when the microprocessor functioning as the camera control unit 30 processes various types of data. The ROM and flash memory (non-volatile memory) in the memory unit 17 are used to store the OS (Operating System) used by the camera control unit 30 to control each unit, content files such as image files, application programs for various operations, firmware, etc.

[0024] The camera control unit 30 executes programs stored in the ROM or flash memory of the memory unit 17, thereby providing overall control over the entire imaging device 1. For example, the camera control unit 30 controls the operation of each necessary unit regarding the control of the shutter speed of the image sensor 12, instructions for various signal processing in the camera signal processing unit 13, imaging operations and recording operations in response to user operations, playback operations of recorded image files, camera operations such as zoom, focus, and exposure adjustment, and user interface operations.

[0025] The user interface unit 16 is configured to have, for example, a display unit and controls, and displays various information to the user, and also accepts user operations and notifies the camera control unit 30 of operation information.

[0026] The distance measurement module 20 includes a surface-emitting laser 21, an irradiation optical system 22, a light-receiving optical system 23, and a distance measurement sensor unit 24, and performs distance measurement processing using a time-of-flight (TOF) method to obtain distance information. The distance information in this case is the distance in the depth direction to the subject imaged by the imaging unit 10, i.e., so-called depth information.

[0027] The surface-emitting laser 21 is configured with laser elements, such as vertical cavity surface-emitting lasers (VCSELs), arranged in a plane, and emits multiple spot beams distributed over the plane. While the embodiments use a surface-emitting laser, it may also be configured with laser elements arranged in a line, emitting multiple spot beams distributed in a line. The distance measurement module 20 of the embodiments is described as an example in which the surface-emitting laser 21 emits spot beams, but it may also be configured to emit uniform light instead of forming spots on the subject. By emitting spot beams and localizing the light, the amount of light per spot can be increased, thereby extending the distance measurement, but the number of measurement points is limited to the number of spots. On the other hand, if uniform light is used, it is possible to have as many measurement points as there are pixels on the light-receiving side.

[0028] The irradiation optical system 22 is an optical system that irradiates a subject with light from the surface-emitting laser 21. In the first embodiment, the irradiation field angle of the irradiation optical system 22 is assumed to be fixed.

[0029] Furthermore, in each embodiment of the present disclosure, it is assumed that the irradiation direction of the laser light from the surface-emitting laser 21 and the irradiation optical system 22 is fixed. That is, the optical axis direction of the irradiated light is fixed, and the optical axis direction of the irradiated light is not changed, for example, by a horizontal or vertical scanning mechanism. The fixed irradiation direction (the optical axis direction of the irradiated light) is the direction of the subject imaged by the imaging unit 10. This is for obtaining distance information to the subject. Note that the term "fixed irradiation direction" as used herein means that the irradiation direction is fixed at least during distance measurement. For example, even if the irradiation direction can be adjusted using an adjustment mechanism or the irradiation direction can be changed by attachment, detachment, etc., as long as the optical axis direction of the irradiated light is not dynamically changed during distance measurement, this falls under the term "fixed irradiation direction" as used in the present disclosure.

[0030] The illumination light output from the surface-emitting laser 21 via the illumination optical system 22 is reflected by the subject and becomes reflected light. The reflected light is then incident on the light-receiving optical system 23 and received by the light-receiving pixel array 42 in the distance measurement sensor unit 24. The light-receiving optical system 23 is configured as an optical system with the same fixed angle of view as the illumination optical system 22.

[0031] The distance measurement sensor unit 24 is provided with a light-receiving pixel array 42, a sensor control unit 41, a distance estimation processing unit 43, and a timing generator 44. The timing generator 44 generates a light emission timing signal for the laser elements of the surface-emitting laser 21 and causes the surface-emitting laser 21 to emit light. The distance estimation processing unit 43 calculates distance information to the subject based on the difference between the light-receiving time of the light-receiving pixel array 42 and the light-emitting time of the surface-emitting laser 21. The sensor control unit 41 controls the processing of the distance estimation processing unit 43 and the timing generator 44 as communication with the camera control unit 30 and operation control of the distance measurement module 20.

[0032] The ranging module 20 outputs ranging coordinates (pixel coordinates on the light-receiving pixel array 42) and distance information for each ranging point. Based on this information, the camera control unit 30 can control the focus adjustment of the lens system 11, generate distance information for each region in the image, for example, as a depth map and add it to the recorded image as metadata, or estimate the self-position of the imaging device 1 and add it to the recorded image as camera position metadata.

[0033] In the present embodiment, for example, the camera control unit 30 calculates the relationship between the angle of view of the lens system 11, which is formed by the attached interchangeable lens, and the angle of view of the distance measurement by the distance measurement module 20, determines the area to be measured by the distance measurement module 20, and sends a control command to the sensor control unit 41. In response to this, the sensor control unit 41 sets the area where spot irradiation is to be performed by the surface-emitting laser 21. This makes it possible to change the irradiation range even if the angle of view of the irradiation optical system 22 is fixed.

[0034] 2. Illumination Range Control by Setting Spot Illumination Region The illumination range control by setting spot illumination regions, which is performed in the imaging device 1 described above, will be described.

[0035] In the first embodiment, when an active ranging module 20 of a surface ranging type is used in an interchangeable lens type imaging device 1, the ranging range of the ranging module 20 is controlled to match the angle of view of the imaging unit 10.

[0036] For example, in a ranging module 20 with a fixed ranging angle of view, a mismatch between the imaging angle of view and the ranging angle of view may occur depending on the angle of view of the lens system 11 using an interchangeable lens, and this mismatch may cause unnecessary processing.

[0037] In this disclosure, the term "imaging angle of view" refers to the angle of view corresponding to the range of the subject that appears in the captured image. This imaging angle of view mainly refers to the angle of view defined by the lens system 11 of the imaging unit 10, but may fluctuate if the imaging range of the image sensor 12 is changed or if image cropping processing is performed. The term "distance measurement angle of view" refers to the angle of view of the range in which spot illumination is performed by the distance measurement module 20, i.e., the angle of view corresponding to the range of the subject for which distance measurement is performed. If the distance measurement angle of view matches the imaging angle of view, distance measurement can be performed for substantially the entire area within the frame plane of the captured image, and all distance measurement data is valid information.

[0038] 2 schematically shows the relationship between the imaging field angle (dashed line) of the lens system 11 and the distance measurement field angle (solid line) of the illumination optical system 22. In this case, the imaging field angle and the distance measurement field angle are substantially the same at the focal position FP.

[0039] On the other hand, Figure 3 shows a state in which an interchangeable lens, such as a telephoto lens, is attached to the lens system 11. In this case, the imaging angle of view of the imaging unit 10 narrows, but the ranging angle of view of the ranging module 20 remains constant, resulting in a mismatch between the imaging angle of view and the ranging angle of view at the focal position FP. In this case, the peripheral portion of the light-receiving pixel array 42 receives reflected light from an area not included in the imaging angle of view, and ranging is performed on the background that is not included in the subject. In other words, unauthorized processing is occurring. Eliminating this unnecessary processing can reduce processing time and increase the frame rate, thereby improving ranging accuracy.

[0040] Conversely, if an interchangeable lens of the wide-angle lens system 11 is attached, the imaging angle of view of the imaging unit 10 becomes wider, contrary to Figure 3 (not shown), and since the ranging angle of view of the ranging module 20 is constant, there will be areas in the captured image that cannot be measured.

[0041] As described above, the ranging angle of view may not match the imaging angle of view of the imaging unit 10, making it impossible to optimize the ranging operation by the ranging module 20. For example, optimization may not be possible in terms of processing speed, frame rate, or ranging range. Therefore, the ranging area of ​​the ranging module 20 is changed according to the imaging angle of view of the imaging unit 10 to optimize the ranging operation.

[0042] 4 is a schematic diagram of the light-emitting surface of the surface-emitting laser 21. The circle in the diagram indicates the position of the laser element (= the spot light-emitting point of the laser element). In normal operation, the laser elements included in the area pattern PT0 of the surface-emitting laser 21 are driven to emit light. Laser elements outside the area pattern PT0 are not driven to emit light. The normal operation refers to a state in which the imaging angle of view and the ranging angle of view are substantially the same, as shown in FIG. 2.

[0043] When the lens system 11 is in the telephoto position, the laser elements included in the area pattern PT1 are driven to emit light, and the laser elements outside the area pattern PT1 are not driven to emit light. The light-receiving pixel array 42 receives reflected light from the laser elements included in the area pattern PT1, and the distance estimation processing unit 43 performs distance measurement processing on the received light. Reducing the number of irradiation spots can result in a power-saving effect. Alternatively, because the number of irradiation spots is reduced, it is possible to prevent an increase in the power consumption even if the number of times each spot is irradiated is increased. Furthermore, since the area (number of spots) targeted for distance measurement processing is reduced, the memory load and calculation load are reduced.

[0044] When the lens system 11 is at a wide angle, the laser elements included in the area pattern PT2 are driven to emit light. In this example, all of the laser elements are driven to emit light. This makes it possible to prevent areas in the captured image from being subject to distance measurement.

[0045] For example, by doing this, the irradiation range of the distance measurement module 20 can be changed according to the imaging angle of view of the lens system 11, so that distance measurement can be performed in a range suited to the imaging angle of view. In other words, the irradiation range can be adjusted by changing the number of irradiation spots of the laser light.

[0046] An example of distance measurement operation will be described with reference to Figures 5A, 5B, and 6. First, Figure 5A shows, on the time axis, time t, and light intensity, and illustrates spot irradiation for distance measurement, and the exposure and counting of reflected light.

[0047] During the irradiation period tTX, spot irradiation is performed multiple times from each laser element of the surface-emitting laser 21. On the light-receiving pixel array 42 side, during the exposure and count period tRX, the reflected light of each spot irradiation is exposed and the time from emission to exposure is counted. In this example, the first to fourth spot irradiations are recorded as 10 ns, 13 ns, 11 ns, and lost. Lost refers to a state in which reflected light could not be received significantly (a state in which time could not be counted). Note that while the figure shows four spot irradiations from one laser element, this is merely a schematic diagram for explanatory purposes, and in reality, the number of spot irradiations from each laser element is usually much greater.

[0048] In this way, the time is counted for each of the multiple spot irradiations for one measurement point, but because there is jitter in the arrival time of the photons, accurate distance information is obtained by examining statistics (histogram peaks). Figure 5B shows a histogram of the time from light emission to light reception obtained for one irradiation spot. Since the time is counted multiple times for each irradiation spot as described above, the histogram is created, and by taking the peak, accurate distance information Td for that spot irradiation is obtained.

[0049] This processing is performed by the distance estimation processing unit 43, but because time counting and histogram processing are performed for multiple spot irradiations for each laser element, the processing load increases as the number of laser elements (number of ranging points) that perform spot irradiation within the surface increases. In other words, the time required for peak detection processing increases in proportion to the number of ranging points. Also, memory (counters) and peak detection processing are required to check statistics (histogram peaks). The greater the count number (≒ number of emissions), the higher the accuracy, but the longer the counter word length per ranging point becomes.

[0050] Based on this, examples of distance measurement operations in normal, telephoto, and wide-angle modes will be described with reference to Fig. 6. Note that Fig. 6 shows an example in which distance measurement frames are asynchronous with the frame rate of image capture on the imaging unit 10 side.

[0051] In normal operation, spot irradiation is performed for laser elements in a range such as the area pattern PT0 in Fig. 4. In the case of Fig. 6, in normal operation, for example, the frame time tF0 = 1 / 60 sec. The number of laser elements (number of distance measurement points) in the area pattern PT0 is 200, and the irradiation / distance measurement angle is 20.0 degrees.

[0052] During the irradiation period tTX, spot irradiation is performed multiple times from each of the 200 laser elements within the range of the area pattern PT0 of the surface-emitting laser 21. During the exposure and count period tRX, the light-receiving pixel array 42 and distance estimation processing unit 43 expose the reflected light of each spot irradiation and count the time from emission to exposure. Then, during the calculation and output period tPR0, distance information is calculated for each measurement point by histogram processing and the distance information for each measurement point is output to the camera control unit 30.

[0053] In contrast, during telephoto shooting, spot irradiation is performed for laser elements in a range such as area pattern PT1 in Fig. 4. In the case of Fig. 6, during telephoto shooting, for example, the frame time tF1 is set to 1 / 100 sec. The number of laser elements (number of distance measurement points) within the range of area pattern PT1 is set to 50, and the irradiation / distance measurement angle is set to 9.8 degrees.

[0054] During the irradiation period tTX, spot irradiation is performed multiple times from each of the 50 laser elements within the range of the area pattern PT1 of the surface-emitting laser 21. The light-receiving pixel array 42 and distance estimation processing unit 43 expose the reflected light of each spot irradiation and count the time from emission to exposure during the exposure and count period tRX. Then, during the calculation and output period tPR1, distance information is calculated for each measurement point by histogram processing and the distance information for each measurement point is output to the camera control unit 30. Because the number of distance measurement points is smaller than normal, tPR0 > tPR1. This allows the frame time tF1 to be shorter than the normal frame time tF0.

[0055] When using telephoto, the image is easily affected by the movement of the imaging device 1 itself and the movement of the subject. Because the subject image is enlarged within the frame of the captured image, the amount of movement within the image is greater than normal, even if the amount of movement is the same. This means that the image moves quickly and the subject distance varies greatly from one measurement point to another. Therefore, it is preferable to increase the frame rate and output distance information more frequently.

[0056] At a wide angle, spot irradiation is performed for laser elements in a range such as the area pattern PT2 in Fig. 4. In the case of Fig. 6, at a wide angle, for example, the frame time tF1 is set to 1 / 30 sec. The number of laser elements (number of distance measurement points) in the area pattern PT2 is set to 800, and the irradiation / distance measurement angle is set to 43.2 degrees.

[0057] During the irradiation period tTX, spot irradiation is performed multiple times from each of the 800 laser elements within the range of the area pattern PT2 of the surface-emitting laser 21. The light-receiving pixel array 42 and distance estimation processing unit 43 expose the reflected light of each spot irradiation and count the time from emission to exposure during the exposure and count period tRX. Then, during the calculation and output period tPR2, distance information is calculated for each measurement point by histogram processing and the distance information for each measurement point is output to the camera control unit 30. Because the number of distance measurement points is greater than normal, tPR0<tPR2. As a result, the frame time tF2 is longer than the normal frame time tF0.

[0058] When the angle is wide, the image sensor is less susceptible to the movement of the image sensor 1 itself or the movement of the subject. Therefore, even if processing takes time and the frame time becomes longer, the accuracy is not affected much. Conversely, it is preferable to increase the number of ranging points so that the ranging range can be covered for the captured image.

[0059] Other examples of distance measurement operations will be described with reference to Figures 7A, 7B, and 8. Because Figures 7A and 7B are schematic diagrams of the same format as Figures 5A and 5B, detailed description will be omitted, but they show a case in which the light intensity of spot illumination is changed as indicated by the solid and dashed lines in Figure 7A. Increasing the light intensity reduces the possibility of reflected light being lost in the light-receiving pixel array 42. This improves distance measurement accuracy. However, power consumption increases. Taking this into consideration, distance measurement operations such as those shown in Figure 8 are performed in normal, telephoto, and wide-angle modes.

[0060] 8, the frame time tF of the ranging frame is constant and the frame rate is not changed. This is a suitable example when it is desired to synchronize with the frame rate of image capture on the imaging unit 10 side. However, it may be asynchronous with the frame rate of image capture.

[0061] In normal operation, spot irradiation is performed by laser elements in a range such as the area pattern PT0 in Fig. 4. In the example of Fig. 8, in normal operation, the number of laser elements (number of distance measurement points) in the area pattern PT0 is set to 200. The irradiation / distance measurement angle is 20.0 degrees, the emission intensity is P, and the number of times of emission is N.

[0062] During the irradiation period tTX0, spot irradiation is performed N times from each of the 200 laser elements within the range of the area pattern PT0 of the surface-emitting laser 21. During the exposure and count period tRX0, the light-receiving pixel array 42 and distance estimation processing unit 43 expose the reflected light of each spot irradiation and count the time from emission to exposure. Then, during the calculation and output period tPR0, distance information is calculated for each measurement point by histogram processing and the distance information for each measurement point is output to the camera control unit 30.

[0063] During telephoto shooting, spot irradiation is performed for laser elements in a range such as area pattern PT1 in Fig. 4. In the case of Fig. 8, during telephoto shooting, the number of laser elements (number of distance measurement points) in the range of area pattern PT1 is set to 50. In this example, the irradiation / distance measurement angle is 9.8 degrees, the emission intensity is 4 x P, and the number of emissions is 1.5 x N.

[0064] During the irradiation period tTX1, spot irradiation is performed 1.5×N times from each of the 50 laser elements within the area pattern PT1 of the surface-emitting laser 21. During the exposure and count period tRX1, the light-receiving pixel array 42 and distance estimation processing unit 43 expose the reflected light from each spot irradiation and count the time from emission to exposure. Here, the irradiation period tTX1 and the exposure and count period tRX1 are longer than the normal irradiation period tTX0 and the normal exposure and count period tRX0 due to the increased number of spot irradiation emissions. Then, during the calculation and output period tPR1, distance information is calculated for each measurement point using histogram processing, and the distance information for each measurement point is output to the camera control unit 30. Because the number of ranging points is smaller than normal, tPR0 > tPR1.

[0065] When using a telephoto lens, the depth of field is shallow, so it is necessary to accurately focus on distant subjects and measure distances accurately. Therefore, since the illumination range is narrow, the light emission intensity is increased to enable measurement over long distances. In addition, increasing the number of times light is emitted improves the accuracy of distance information obtained through histogram processing. At the same time, the frame time tF can be maintained.

[0066] At a wide angle, spot irradiation is performed for laser elements in a range such as area pattern PT2 in Fig. 4. In the case of Fig. 8, at a wide angle, the number of laser elements (number of distance measurement points) in the range of area pattern PT2 is set to 800. In this example, the irradiation / distance measurement angle is 43.2 degrees, the emission intensity is P, and the number of times of emission is N / 4.

[0067] During the irradiation period tTX2, spot irradiation is performed N / 4 times from each of the 800 laser elements within the area pattern PT2 of the surface-emitting laser 21. During the exposure and count period tRX2, the light-receiving pixel array 42 and distance estimation processing unit 43 expose the reflected light from each spot irradiation and count the time from emission to exposure. Here, the irradiation period tTX2 and exposure and count period tRX2 are shorter than the normal irradiation period tTX0 and exposure and count period tRX0 due to the reduced number of spot irradiation emissions. Then, during the calculation and output period tPR2, distance information is calculated for each measurement point using histogram processing, and the distance information for each measurement point is output to the camera control unit 30. Because there are more distance measurement points than normal, tPR0<tPR2.

[0068] When the angle of view is wide, the depth of field is deep, making it easier to achieve focus. Therefore, priority is given to measuring a wide range in a given time rather than distance measurement accuracy. In other words, the exposure time is reduced. To reduce the exposure time, the number of flashes is reduced. Reducing the number of flashes also reduces the counter word length required per distance measurement point. In this way, the frame time tF can be maintained and the number of distance measurement points corresponding to the wide angle can be set without increasing the hardware scale.

[0069] In this embodiment, distance measurement operations such as those shown in Figures 6 and 8 are possible. Examples of processing by the camera control unit 30 for executing such distance measurement operations are described with reference to Figures 9 and 10. As part of the processing shown in Figures 9 and 10, the camera control unit 30 issues various instruction controls to the sensor control unit 41 to execute the distance measurement operation.

[0070] The camera control unit 30 determines whether a lens is attached in step S101 of Fig. 9. That is, it determines whether an interchangeable lens is attached. If the camera control unit 30 determines that an interchangeable lens is attached, the process proceeds to step S102.

[0071] In step S102, the camera control unit 30 determines whether or not the current mode is one in which the illumination range of the ranging module 20 is changed in accordance with the imaging angle of view. The modes in which the illumination range is changed in accordance with the imaging angle of view and the modes in which it is not changed will be described later.

[0072] In a mode in which the illumination range is changed according to the imaging angle of view, the camera control unit 30 performs processing to acquire information about the angle of view of the interchangeable lens in step S103. For example, information about the imaging angle of view of the lens system 11 of the interchangeable lens is acquired from information from the lens position encoder 11a. In step S104, the camera control unit 30 checks whether or not the information about the imaging angle of view has been acquired.

[0073] If information on the imaging angle of view can be acquired from the interchangeable lens, the camera control unit 30 proceeds to step S105 and selects the area pattern of the smallest ranging area that covers the imaging angle of view of the lens system 11. For example, an area pattern that is adapted to the imaging angle of view of the current interchangeable lens is selected, such as area patterns PT0, PT1, and PT2 shown in Fig. 4. Note that if even the largest ranging area cannot cover the imaging angle of view, the area pattern of the largest ranging area is selected.

[0074] In step S106, the camera control unit 30 instructs the ranging module 20 on the area for spot irradiation according to the selected area pattern. That is, it instructs the laser elements inside the selected area pattern to be driven to emit light. This instructs the number of ranging points. The camera control unit 30 also sets operating parameters for the ranging module 20, such as the frame time (tF), the number of times each laser element emits light for spot irradiation (irradiation period tTX, exposure and count period tRX), and light emission intensity.

[0075] After these settings are made, the camera control unit 30 starts the distance measurement operation of the distance measurement module 20 in step S110. Then, in step S111, the camera control unit 30 starts receiving distance measurement data, i.e., information on the distance to the subject, from the distance measurement module 20. The camera control unit 30 uses the received distance information for focus control, or records it as metadata in association with the captured image as a depth map of the captured image or depth information of the main subject.

[0076] 6 or 8, a distance measurement operation is performed in an irradiation range (distance measurement angle of view) that matches the imaging angle of view of the lens system 11. By making it possible to variably control the irradiation range of the emitted laser light in accordance with the area distance measurement range, the distance measurement module 20 can adjust the irradiation range without having an optical system with a variable angle of view.

[0077] When an interchangeable lens of the lens system 11 with a narrow angle of view is attached, the irradiation range of the distance measurement module 20 is narrowed and the amount of calculation for distance calculation in the distance measurement module 20 is reduced, which can be utilized to improve the distance measurement frame rate. Furthermore, narrowing the irradiation range of the laser light emitted from the distance measurement module 20 has the effect of reducing power consumption.

[0078] Furthermore, when an interchangeable lens with a lens system 11 having a wide angle of view is attached, the irradiation range can be widened accordingly to measure the distance to the subject in the captured image. In this case, if the distance measurement frame rate needs to be maintained, control can be performed to reduce the irradiation time of the irradiated laser light (i.e., reduce the number of irradiations) as shown in the example of Figure 8. The wider the angle of view, the more distance measurement points there are, and the greater the amount of calculation in the distance measurement module 20. Therefore, the increase in calculation amount can be accommodated by increasing the time available for calculation processing within the frame time.

[0079] 9, if it is determined that the mode does not change the irradiation range of the ranging module 20, the camera control unit 30 sets the operating parameters of the ranging module 20 according to the mode in step S107, and starts the ranging operation of the ranging module 20 in step S110. Then, the camera control unit 30 starts receiving ranging data from the ranging module 20, that is, information on the distance to the subject, in step S111.

[0080] Also, if it is determined in step S104 that lens angle of view information has not been obtained, the process goes through step S107 and then steps S110 and S111.

[0081] In these cases, the illumination range control is not performed in accordance with the imaging angle of view. Examples of when illumination range control is performed and when it is not performed will be described.

[0082] First, cases in which illumination range control is performed according to the imaging angle of view include when there is information on the imaging angle of view from the lens system 11 of the imaging unit 10, and when ranging information is used for focus control or for recording depth metadata.

[0083] On the other hand, the case where the illumination range control according to the imaging angle of view is not performed is when information on the imaging angle of view by the lens system 11 of the imaging unit 10 cannot be obtained. This is the case where the process proceeds from step S104 to step S107, and if information on the imaging angle of view cannot be obtained, it is not possible to set the illumination range of the ranging module 20 accordingly. In such a case, it is conceivable to measure the distance only to the center of the captured image, for example.

[0084] In the above description, in step S103, the camera control unit 30 obtains information about the imaging angle of view from information from the lens position encoder 11a. However, there is other information that can be used to identify the imaging angle of view. For example, imaging lens information includes focal length, image magnification, and teleconverter magnification. In reality, the imaging angle of view may be related not only to the angle of view of the lens system 11, but also to the imaging range and crop magnification of the image sensor 12. Furthermore, it is advisable to include information about signal processing, such as distortion correction, breathing correction, image stabilization, and recording mode (RAW / YC), in the imaging angle of view information.

[0085] One case in which illumination range control is not performed regardless of the imaging angle of view is when the imaging device 1 is in calibration mode. This is because, when calibrating or testing the range finder module 20, if there are areas that are not illuminated, the task takes time and there is a possibility that the calibration will fail. If the entire area is illuminated, it is possible to detect any significant deviation in the illumination direction and to check at once whether there are any spots with poor illumination.

[0086] One case in which illumination range control is not performed regardless of the imaging angle of view is when the imaging device 1 is in a mode for estimating its own position or a mode for capturing images for subject shape estimation (modeling). When the imaging device 1 estimates its own position within a space in which imaging is performed, for example, it measures the distance to, for example, a wall that forms the space. When capturing an image of a subject and estimating its three-dimensional shape for modeling, it captures an image centered on a specific subject. In these cases, even distance measurement information outside the imaging angle of view is useful for improving estimation accuracy. Therefore, it is preferable not to perform illumination range control.

[0087] Therefore, if the imaging device 1 is in a calibration mode, an operation mode for estimating its own position, or a mode for capturing images for estimating the shape of a subject, the camera control unit 30 proceeds from step S102 to step S107.

[0088] In the above example, the irradiation range is controlled by selecting the laser elements to be emitted by the surface-emitting laser 21, that is, by selecting the area pattern. However, from the viewpoint of controlling the irradiation range, the irradiation range can also be changed by, for example, masking some of the laser elements with an aperture mechanism while emitting light from all the laser elements of the surface-emitting laser 21, or by changing the angle of view of the irradiation optical system 22. Possible methods for changing the angle of view of the irradiation optical system 22 include using a zoom lens, a varifocal lens, or a converter lens. In the second embodiment described below, a case will be described in which the irradiation optical system 22 includes a zoom lens to change the angle of view of the irradiation optical system 22. In the case of using such a zoom lens, the density of spot irradiation changes along with the irradiation range.

[0089] 10 shows an example of processing when an interchangeable lens is removed. While the distance measurement operation is being performed by the distance measurement module 20, the camera control unit 30 monitors in step S120 whether or not the interchangeable lens has been removed.

[0090] When it is detected that the interchangeable lens has been removed, the camera control unit 30 proceeds to step S121 and controls the distance measurement module 20 to stop the distance measurement operation. Then, in step S122, the camera control unit 30 performs processing to discard the distance measurement data received after the distance measurement operation has stopped.

[0091] After that, if a new interchangeable lens is attached, the camera control unit 30 will perform the process of FIG. 9 again.

[0092] In the first embodiment described above, an interchangeable lens is used, and a case has been described in which the angle of view on the imaging unit 10 side has changed due to lens replacement. However, regardless of whether the lens is an interchangeable lens or a built-in lens, the above processing can also be applied when the imaging angle of view changes due to the zoom operation of a fixed zoom lens or a variable zoom lens of the lens system 11, a change in the imaging range of the image sensor 12, or cropping control in the camera signal processing unit 13.

[0093] 11 shows a configuration example of an imaging device 1 according to a second embodiment. However, many parts are the same as those in FIG. 1, and the same components are denoted by the same reference numerals to avoid redundant explanation.

[0094] 11 , in the distance measurement module 20, the projection optical system 22 is equipped with a zoom lens 22Z, and the light receiving optical system 23 is also equipped with a zoom lens 23Z. The projection optical system 22 can change the irradiation range of the spot irradiation from the surface-emitting laser 21 by changing the zoom magnification of the zoom lens 22Z. The zoom lens 23Z of the light receiving optical system 23 is basically controlled to have the same angle of view as the zoom lens 22Z of the projection optical system 22, so that even if the angle of view of the projection optical system 22 is changed, the reflected light within the irradiation range can be received by the light receiving pixel array 42.

[0095] The lens system 11 in the imaging unit 10 is not limited to an interchangeable lens. In the second embodiment, the description will be made mainly assuming a case where the imaging angle of view is changed by, for example, a zoom lens in the lens system 11.

[0096] 11 , the range of spot irradiation by the surface-emitting laser 21 (range-finding angle of view) can be changed by changing the zoom magnification of the zoom lens 22Z in the range-finding module 20. Therefore, by changing the range of irradiation by setting the zoom magnification of the zoom lens 22Z in accordance with information on the lens system 11 in the imaging unit 10, i.e., the imaging angle of view, the operation of the range-finding module 20 can be optimized as described in the first embodiment.

[0097] 12, in addition to the configuration of FIG. 11, the distance measuring module 20 is provided with an additional optical system 25, that is, a conversion lens 18 on the light emitting side and a conversion lens 19 on the light receiving side.

[0098] For example, in the configuration of Fig. 11, even if the illumination range (distance measurement angle of view) is changed by changing the zoom magnification, if the angle of view cannot keep up with the imaging angle of view of the imaging unit 10, conversion lenses 18, 19 can be inserted as shown in Fig. 12 to make the distance measurement angle of view equal to the imaging angle of view. The additional optical system 25 may be manually attached by the user, or an additional optical system 25 module may be provided that allows the conversion lenses 18, 19 to be inserted into or removed from the illumination light path or the light receiving light path, and the conversion lenses 18, 19 may be used by automatic control.

[0099] Here, a description will be given of corrections that are required in accordance with changes in the optical path length when the distance measurement field angle is changed by the zoom lens 22Z or the conversion lens 18. The case where the conversion lens 18 is used will be described.

[0100] Figure 13 shows an example in which an additional optical system 25 is attached to the distance measurement module 20. Depth Z in the optical axis direction (distance from the image plane) corresponds to distance information to the subject. An optical path when the angle of view is not changed by the illumination optical system 22 is designated as optical path L0. For example, this can also be taken as the case when a conversion lens with image magnification = 1 is used. The angle of optical path L0 with respect to the optical axis is designated as θ0. Optical path L1 is the optical path when the angle of view is narrowed, and is the case when a conversion lens with image magnification > 1 is used. The angle of optical path L1 with respect to the optical axis is designated as θ1. Optical path L2 is the optical path when the angle of view is widened, and is the case when a conversion lens with image magnification < 1 is used. The angle of optical path L2 with respect to the optical axis is designated as θ2.

[0101] Here, considering the normal round trip optical path length L from light emission to light reception, the optical path length L = 2 × Z / cos θ, and the propagation time t = optical path length ÷ speed of light = 2 × Z / (cos θ * c), where c is the speed of light.

[0102] The depth Z estimated from the propagation time t is Z(t, θ) = t × c ÷ 2 × cos θ. If the angle θ value changes due to the conversion lens 18, the optical path length will change, so correction processing must be performed accordingly. Z(t, θ) = Z(t, θ0) × (cos θ / cos θ0) In this case, the above θ0, θ1, and θ2 are substituted for the cos θ term.

[0103] Specifically, to calculate the distance in an orthogonal coordinate system, calculate the depth from the reference plane R in Figure 14. For light path L0, the distance calculation for the light direction of θ0 is as follows: Propagation time: t0 Propagation distance: L0 = t0 x c ÷ 2 Depth Z: Z = L0 x cos θ0 In other words, Z(t) = t x C ÷ 2 x cos θ0

[0104] In the case of light path L1, the distance calculation for the ray direction of θ1 is as follows: Propagation time: t1 Propagation distance: L1 = t1 × c ÷ 2 Depth Z: Z = L1 × cos θ1 In other words, Z(t) = t × C ÷ 2 × cos θ1

[0105] In this way, if the distance measurement field angle of the distance measurement module 20 is originally the optical path L0 of θ0, and changes to the direction of θ1 due to a change in the lens, the calculation formula for depth Z(t) to be used will change. Therefore, it is necessary to detect the lens status of the distance measurement module 20 and change the distance calculation parameters according to the lens status, or perform processing to correct them later. The above has been explained in the case of the conversion lens 18, but the same applies when the distance measurement field angle is changed by the zoom lens 22Z.

[0106] 15 and 16 will be used to explain a processing example in which the range finder module 20 controls the illumination range by controlling the zoom magnification of the zoom lens 22Z in accordance with the imaging angle of view in the configuration of the second embodiment. Note that in the following flowcharts, the same steps as those already explained are assigned the same step numbers, and detailed overlapping explanations will be avoided.

[0107] 15 shows the control process for the distance measurement module 20 by the camera control unit 30. This process example is executed not only when the interchangeable lens is replaced, but also when the zoom magnification of the lens system 11 is changed.

[0108] In step S102, the camera control unit 30 determines whether or not the mode is one in which the illumination range of the ranging module 20 is changed in accordance with the imaging angle of view. If the mode is one in which the illumination range of the ranging module 20 is changed, the camera control unit 30 performs processing to acquire information about the angle of view of the lens system 11 in step S103. For example, information about the zoom magnification of the lens system 11 is acquired from information from the lens position encoder 11a. In step S104, the camera control unit 30 checks whether or not information about the imaging angle of view has been acquired.

[0109] If the information on the imaging angle of view has been acquired, the camera control unit 30 proceeds to step S150, where it sets the zoom magnification of the zoom lens 22Z so as to obtain a ranging angle of view corresponding to the imaging angle of view. Then, in step S151, the camera control unit 30 sets operating parameters of the ranging module 20, such as the frame time, the number of times each laser element emits a spot light, and the light emission intensity.

[0110] In step S152, the camera control unit 30 sets correction parameters for the distance measurement data. That is, since it is necessary to select or correct the calculation formula to be applied in accordance with the change in optical path length caused by the change in zoom magnification, as described with reference to FIGS. 13 and 14, the camera control unit 30 sets the correction parameters.

[0111] After these settings are made, the camera control unit 30 starts the ranging operation of the ranging module 20 in step S110. Then, in step S114, the camera control unit 30 starts receiving ranging data from the ranging module 20, i.e., information on the distance to the subject. In this case, the camera control unit 30 also performs a correction process on the ranging data in accordance with the change in the optical path length. Note that the ranging module 20 may transmit the corrected ranging data to the camera control unit 30.

[0112] Through the above processing, a ranging operation is performed in an illumination range (ranging angle of view) that is suited to the imaging angle of view. This prevents the generation of invalid ranging points and enables ranging that corresponds to the entire area within the plane of the captured image. For example, FIG. 16A shows the imaging angle of view 100 and ranging angle of view 101 (illumination range of the ranging module 20) in normal mode, FIG. 16B shows the telephoto mode, and FIG. 16C shows the wide-angle mode. The circles in the figures indicate the positions of spot illumination (measurement points). By changing the ranging angle of view 101, indicated by the dashed line, to match the imaging angle of view 100, indicated by the solid line, ranging that corresponds to the entire area within the plane of the captured image becomes possible.

[0113] If it is determined in step S102 that the mode does not change the irradiation range of the ranging module 20, or if it is determined in step S104 that lens angle of view information has not been obtained, the camera control unit 30 sets a zoom magnification according to the mode in step S153 and performs the processes of steps S151 to S114. In this case, ranging may be performed at a normal zoom magnification, or, in the case of the above-mentioned self-position estimation mode or 3D modeling mode, ranging may be performed at a zoom magnification that is wider than the imaging angle of view.

[0114] 5. Irradiation Range Control by Setting Zoom Magnification and Spot Irradiation Area> For example, in the configuration of the second embodiment, the irradiation range can be changed not only by the zoom magnification of the zoom lens 22Z, but also by selecting an area pattern for spot irradiation by the surface-emitting laser 21, as in the first embodiment. Therefore, an example using both of these will be described with reference to Fig. 17. Fig. 17 also includes a case where a conversion lens 18 is applied.

[0115] 17 considers a case where the range of variation in the zoom magnification of the zoom lens 22Z is ​​narrower than the range of variation in the zoom magnification of the lens system 11. For example, as shown in Fig. 18, the lens system 11 of the imaging unit 10 can change the zoom magnification within a range of focal lengths from 20 mm to 200 mm (35 mm equivalent angle of view), whereas the zoom lens 22Z of the range finder module 20 can change the zoom magnification within a range of focal lengths from 40 mm to 80 mm (35 mm equivalent angle of view).

[0116] In addition, in the processing examples of Figure 17 and Figures 20, 22, and 24 described below, the processing described in steps S102, S104, S107, and S153 in Figures 9 and 15 are omitted, but of course these processing may also be performed.

[0117] The camera control unit 30 periodically repeats the processing of Fig. 17. In step S300, the camera control unit 30 performs processing to acquire information about the angle of view of the lens system 11. For example, the camera control unit 30 acquires information about the zoom magnification of the lens system 11 from information from the lens position encoder 11a. Then, in step S301, the camera control unit 30 determines whether the image capture angle of view has changed since the previous acquisition, and if it has not changed, ends the processing of Fig. 17 from step S301.

[0118] If it is determined that the imaging angle of view has changed, then in step S302 the camera control unit 30 sets the zoom magnification of the zoom lens 22Z so that a distance measurement angle of view corresponding to the imaging angle of view is obtained.

[0119] In step S303, the camera control unit 30 determines whether the distance measurement angle of view is adapted to the imaging angle of view by the zoom magnification setting of the zoom lens 22Z. This is because, in the example of Fig. 18, the zoom lens 22Z may not be able to accommodate the zoom magnification of the imaging unit 10.

[0120] If the zoom magnification setting of the zoom lens 22Z allows the distance measurement angle of view to be adapted to the imaging angle of view, the camera control unit 30 proceeds from step S303 to step S307, and sets operating parameters for the distance measurement module 20, such as the frame time, the number of times each laser element emits a spot light, and the light emission intensity.

[0121] In step S308, the camera control unit 30 sets correction parameters for the distance measurement data. That is, since it is necessary to select or correct the calculation formula to be applied in accordance with the change in optical path length caused by the change in zoom magnification, as described with reference to Figs. 13 and 14, the camera control unit 30 sets the correction parameters.

[0122] After these settings are made, the camera control unit 30 starts the distance measurement operation of the distance measurement module 20 in step S110, and starts receiving distance measurement data, that is, information on the distance to the subject, from the distance measurement module 20 in step S114.

[0123] For example, Fig. 19A shows a state in which the distance measurement angle of view 101 is adapted to the imaging angle of view 100. At this time, suppose that the imaging angle of view 100 changes to Fig. 19B. The distance measurement angle of view 101 can also be adapted by changing the zoom magnification of the zoom lens 22Z. This is an example of a case in which it is determined in step S303 that adaptation has been achieved. In this case, the illumination range is optimized for the captured image by adjusting the distance measurement angle of view using the zoom lens 22Z.

[0124] On the other hand, suppose that the distance measurement angle of view 101 indicated by the dashed line in Fig. 19B is the most telephoto angle of view of the zoom lens 22Z. In this case, if the lens system 11 side has an even more telephoto angle of view, the zoom lens 22Z cannot follow. For example, Fig. 19C shows an example in which the distance measurement angle of view of the zoom lens 22Z is ​​not adapted to the imaging angle of view 100.

[0125] In such a case, the camera control unit 30 proceeds from step S303 to step S304 and selects the area pattern of the smallest ranging area that covers the imaging angle of view of the lens system 11. For example, an area pattern that is suitable for the current imaging angle of view is selected, such as the area patterns PT0, PT1, and PT2 shown in Fig. 4. For example, an area pattern that corresponds to the ranging angle of view 101 in Fig. 19D is selected.

[0126] Then, in step S305, the camera control unit 30 instructs the ranging module 20 on the area where spot irradiation will be performed, according to the selected area pattern. That is, it instructs the laser elements inside the selected area pattern to be driven to emit light. This is an instruction for the number of ranging points. The laser elements at the positions indicated by circles in Figure 19D are set to perform spot irradiation, and the laser elements indicated by circles are set not to emit light.

[0127] In step S306, the camera control unit 30 checks whether the ranging angle of view 101 is adapted to the imaging angle of view 100. For example, the camera control unit 30 sets an area pattern with the minimum number of ranging points as a settable area pattern. Then, by selecting an area pattern with the minimum number of ranging points as the limit, it is determined whether the ranging angle of view has been adapted to the imaging angle of view.

[0128] If the camera control unit 30 is adapted, it performs the processes from step S307 to step S114 in the same manner as described above. In this case, in addition to adjusting the distance measurement field angle by the zoom lens 22Z, the area pattern for emitting light from the laser element is set, so that the irradiation range is optimized for the captured image.

[0129] In step S306, it may be determined that the distance measurement angle of view is not adapted to the imaging angle of view. In that case, the camera control unit 30 proceeds to step S310, and, for example, issues an announcement to the user from the user interface unit 16 recommending the use of the conversion lens 18. This may prompt the user to attach the additional optical system 25 of the conversion lenses 18, 19. Note that, if the additional optical system 25 can be automatically inserted, the camera control unit 30 may control the distance measurement module 20 to insert the additional optical system 25 into the optical path.

[0130] If the additional optical system 25 is manually attached by the user, the user can choose to either attach it in response to an announcement or to perform imaging without attaching it. Depending on either option, the camera control unit 30 proceeds to step S311 or step S307, and executes the processes up to step S114. This makes it possible to further use the conversion lens 18 to adapt to the imaging angle of view when the distance measurement angle of view by the zoom lens 22Z and the area pattern for emitting light from the laser element are both set and the imaging angle of view cannot be adapted.

[0131] It should be noted that instead of proceeding from step S311 to step S307, for example, if the insertion of the conversion lens 18 is confirmed in step S311, the process may return to step S302 and start over from setting the magnification of the zoom lens 22Z with the conversion lens 18 applied.

[0132] The above process allows distance measurement to be performed within an illumination range (distance measurement angle of view) that matches the image capture angle of view, thereby preventing invalid distance measurement points and enabling distance measurement that covers the entire area within the captured image plane.

[0133] 20 shows an example in which both the change in the distance measurement field angle caused by the zoom lens 22Z and the change in the distance measurement field angle caused by setting the area pattern for emitting light from the laser element are combined, as in FIG. 17, but the order of setting is reversed.

[0134] The camera control unit 30 periodically repeats the processing of Fig. 20. In step S300, the camera control unit 30 performs processing to acquire information about the angle of view of the lens system 11. In step S301, the camera control unit 30 determines whether the image capture angle of view has changed since the previous acquisition, and if it has not changed, ends the processing of Fig. 20 from step S301.

[0135] If it is determined that the imaging angle of view has changed, then in step S304 the camera control unit 30 selects an area pattern of the smallest ranging area that covers the imaging angle of view of the lens system 11. Then, in step S305, the camera control unit 30 instructs the ranging module 20 on an area for spot irradiation in accordance with the selected area pattern. That is, the camera control unit 30 instructs the laser elements inside the selected area pattern to be driven to emit light.

[0136] In step S303, the camera control unit 30 checks whether the ranging angle of view is adapted to the imaging angle of view. For example, as shown in Fig. 21A, if the ranging angle of view 101 is adapted to the imaging angle of view 100, the camera control unit 30 proceeds from step S303 to step S307 and sets the operating parameters of the ranging module 20.

[0137] Then, in step S308, the camera control unit 30 sets correction parameters for the distance measurement data. However, in this case, the zoom lens 22Z remains set to the normal zoom magnification, and no correction is performed.

[0138] After these settings are made, the camera control unit 30 starts the distance measurement operation of the distance measurement module 20 in step S110, and starts receiving distance measurement data, that is, information on the distance to the subject, from the distance measurement module 20 in step S114.

[0139] On the other hand, there are cases where the ranging angle of view is not adapted to the imaging angle of view at the stage of step S303. For example, in Fig. 21B, even when all laser elements are set to emit light, the ranging angle of view is not adapted to the imaging angle of view. In such a case, in step S302, the camera control unit 30 sets the zoom magnification of the zoom lens 22Z so that the ranging angle of view corresponding to the imaging angle of view is obtained.

[0140] In step S306, the camera control unit 30 determines whether the distance measurement angle of view has been adapted to the imaging angle of view by setting the zoom magnification of the zoom lens 22Z.

[0141] 21C , the camera control unit 30 proceeds from step S306 to step S307 and subsequent steps. In this case, in step S308, the camera control unit 30 sets correction parameters for the distance measurement data in accordance with the change in zoom magnification.

[0142] If it is determined in step S306 that the distance measurement angle of view is not adapted to the imaging angle of view, the camera control unit 30 proceeds to step S310, and makes an announcement to the user from the user interface unit 16, for example, recommending the use of the conversion lens 18. If the user has attached the additional optical system 25 or has selected to perform imaging without attaching it, the camera control unit 30 proceeds to step S311, step S307, and executes the processing up to step S114.

[0143] Instead of proceeding from step S311 to step S307, for example, if the insertion of the conversion lens 18 is confirmed in step S311, the process may return to step S304 and start over from selecting the area pattern with the conversion lens 18 applied.

[0144] The above process allows distance measurement to be performed within an illumination range (distance measurement angle of view) that matches the image capture angle of view, thereby preventing invalid distance measurement points and enabling distance measurement that covers the entire area within the captured image plane.

[0145] In the processing example of FIG. 15, if the variable range of the zoom magnification of the zoom lens 22Z cannot fully accommodate the imaging angle of view, a conversion lens 18 may be applied as in the examples of FIG. 17 and FIG. 20.

[0146] Figure 22 shows an example in which both the zoom magnification of the zoom lens 22Z and the area pattern for emitting light in the surface-emitting laser 21 are set, and if an irradiation range (distance measurement angle of view) that matches the imaging angle of view cannot still be obtained, that information is recorded as metadata.

[0147] 22, steps S300 to S308 are the same as those in FIG. 17, and therefore description thereof will be omitted. In this case, if it is determined in step S306 that the distance measurement angle of view is not adapted to the image capture angle of view, the camera control unit 30 proceeds to step S320 and acquires information on the area within the captured image that is outside the illumination range. The camera control unit 30 then sets operating parameters and correction parameters in steps S307 and S308, and starts the distance measurement operation of the distance measurement module 20 in step S110.

[0148] In step S115, the camera control unit 30 starts receiving distance measurement data from the distance measurement module 20, i.e., information on the distance to the subject, and performing correction processing in response to changes in the optical path length. The camera control unit 30 also starts processing metadata related to the distance measurement operation. That is, the camera control unit 30 generates the following metadata related to distance measurement, associates it with the captured image, records it, and transmits it to an external device.

[0149] For example, metadata is generated so that the position on the captured image and the depth Z can be determined. For example, a set of (x, y, z) is recorded for each ranging point. x and y are coordinates within the plane of the captured image, and z is the measured depth (distance information).

[0150] 23A shows a case where the imaging angle of view 100 and the ranging angle of view 101 are the same, and in this case, the (x, y, z) values ​​for each measurement point are used as metadata. When the coordinate values ​​recorded as metadata are projected, they are distributed over almost the entire image.

[0151] 23B shows a case where the distance measurement angle of view 101 is narrower than the imaging angle of view 100. In this case, distance information cannot be obtained for a certain subject 102. Thus, there are areas in the periphery of the captured image where distance has not been measured. In such a case, for example, the (x, y, z) values ​​for each measurement point are used as metadata, and information on areas where distance could not be measured, i.e., information outside the illumination range acquired in step S320, is also included in the metadata. For example, the areas where distance could not be measured are indicated by x-coordinate values ​​and y-coordinate values.

[0152] 23C shows a case where the irradiable range is wider than the imaging angle of view of 100. In this case, the spots indicated by ● are not irradiated. Therefore, for the spots indicated by ○ where distance measurement has been performed, the set of (x, y, z) is associated with the captured image as metadata. The amount of metadata is reduced.

[0153] Regarding the metadata, for example, by adding information about the range of the distance measurement angle of view 101 shown by the dashed line in the figure, or information about the area where distance measurement was not possible as described above, it is possible to present the user with the range where distance measurement was possible when the captured image is displayed on the monitor. The total number of distance measurement points may also be included in the metadata. Information about the distance measurement angle of view may also be included in the metadata. This may be a focal length value, or a value relative to the imaging angle of view.

[0154] 6. Control of illumination range by conversion lens An example of control of illumination range by a conversion lens will be described. For example, this is an example of control in a configuration in which a conversion lens 18 can be applied, as shown in FIG. 12 . However, it is assumed here that the zoom lenses 22Z, 23Z in the illumination optical system 22 and the light receiving optical system 23 are not provided or have fixed angles of view. It is also assumed that the lens system 11 of the imaging unit 10 is an interchangeable lens.

[0155] 24 shows an example of processing by the camera control unit 30. In step S101, the camera control unit 30 determines whether a lens is attached, that is, whether an interchangeable lens is attached or not.

[0156] If the camera control unit 30 determines that an interchangeable lens is attached, the process proceeds to step S103, where it performs processing to acquire information about the angle of view of the interchangeable lens.

[0157] In step S160, the camera control unit 30 selects the optical system of the conversion lenses 18 and 19 with the minimum magnification that covers the imaging angle of view of the lens system 11, and inserts it into the optical path.

[0158] Although not shown in the flowchart, in this case, the camera control unit 30 may perform control to notify the user that the appropriate conversion lenses 18, 19 should be attached, or may perform control to insert the appropriate conversion lenses 18, 19 into the optical path using an automatic insertion / removal mechanism.

[0159] In step S161, the camera control unit 30 sets the operating parameters of the distance measurement module 20. In step S162, the camera control unit 30 sets correction parameters for the distance measurement data. That is, since it becomes necessary to select or correct the calculation formula to be applied as described with reference to FIGS. 13 and 14 in accordance with the change in the optical path length caused by the conversion lenses 18 and 19, the settings are made.

[0160] After these settings, the camera control unit 30 starts the distance measurement operation of the distance measurement module 20 in step S110, and starts reception of distance measurement data from the distance measurement module 20 and correction processing in step S114.

[0161] By the above processing, the conversion lenses 18 and 19 can be used to set the illumination range of the distance measurement module 20 so as to suit the imaging angle of view.

[0162] 7. Setting of Distance Measurement Parameters In the embodiments described above, examples have been shown in which the distance measurement angle of view is adjusted in accordance with the imaging angle of view to optimize the irradiation range of the distance measurement module 20. In these cases, examples of setting parameters for distance measurement operations will be described.

[0163] FIG. 25 shows the control operations of the distance measurement frame rate, the irradiation period tTX (=number of times of spot irradiation), and the light emission intensity as the operation parameters of the distance measurement module 20.

[0164] Increasing the ranging frame rate imposes time constraints on performance and increases power consumption, but it also increases time resolution and improves tracking of subject movement. Time constraints mean that the frame rate restricts the ranging time. Lowering the ranging frame rate can reduce power consumption, but is affected by moving objects. Alternatively, it can increase the number of times the spot light is emitted, improving ranging accuracy. Moving object influence means that ranging accuracy decreases for moving subjects.

[0165] Increasing the irradiation time and the number of spot irradiations increases the distance measurement time and power consumption, but the number of samples for histogram processing increases, improving distance measurement accuracy. Shortening the irradiation time and reducing the number of spot irradiations decreases the number of samples for histogram processing, reducing distance measurement accuracy, but power consumption. Alternatively, the shorter distance measurement time allows the frame rate to be increased, improving time resolution.

[0166] Increasing the light emission intensity increases power consumption but extends the distance that can be measured, making it possible to handle distant or low-reflectivity subjects. Decreasing the light emission intensity shortens the distance that can be measured but reduces power consumption.

[0167] Taking these factors into consideration, it is conceivable to carry out control as shown in Fig. 26. Fig. 26 shows the control contents according to the conditions and triggers for changing the operating parameters and the resulting effects. The following examples are conceivable.

[0168] The higher the frame rate on the imaging unit 10 side, the higher the ranging frame rate of the ranging module 20. When the imaging device 1 is in a mode for capturing images of fast-moving subjects, such as a sports photography mode, the ranging frame rate of the ranging module 20 is increased. When the exposure time is set to a short (narrow shutter opening) or in a photography mode, the ranging frame rate of the ranging module 20 is increased.

[0169] Increasing the ranging frame rate shortens the irradiation period tTX, thereby reducing the number of spot irradiations. In the above case, increasing the ranging frame rate improves the ability to follow the movement of the subject.

[0170] When the camera status is in standby mode, the ranging frame rate of the ranging module 20 is reduced. If ranging is performed mainly for focus control, ranging continues even during standby mode. However, if reducing power consumption is a priority, it may be possible to reduce the ranging frame rate.

[0171] The deeper the depth of field, the shorter the irradiation period (tTX). When the depth of field is deep, the focus range is wide, so the requirement for distance measurement accuracy is lower. Therefore, if reducing power consumption is a priority, it is effective to shorten the irradiation period and reduce the number of spot irradiations.

[0172] The longer the focal length of the lenses in the lens system 11, the stronger the emission intensity of the surface-emitting laser 21. The greater the amount of light from the imaging target (subject), the stronger the emission intensity of the surface-emitting laser 21. When imaging a distant subject with a long focal length or when there is a large amount of light on the subject side, it is preferable to increase the emission intensity in order to increase the detectability of reflected light.

[0173] When the power exceeds a predetermined upper limit in the distance measurement module 20, the number of spot irradiations and the light emission intensity are reduced to stay within the upper limit. This maintains the upper power limit, thereby complying with laser safety standards and preventing abnormal heating of the device.

[0174] - Power reduction control is not performed during external power supply. In other words, frame rate reduction, reduction in the number of spot shots, and reduction in light emission intensity are not performed, and distance measurement performance is given priority.

[0175] When the camera control unit 30 changes the irradiation range to adapt to the imaging angle of view, as shown in Figure 26 above, it further determines the above conditions and triggers and switches between the control of the ranging frame rate of the ranging module 20, the control of the irradiation period tTX of the surface-emitting laser 21, and the control of the light emission intensity, thereby realizing ranging operation that is appropriate for the situation.

[0176] 8. Summary and Modifications According to the above embodiment, the following effects can be obtained.

[0177] The imaging device 1 of the embodiment includes a ranging unit (ranging module 20) that includes an irradiating unit (surface-emitting laser 21) with a fixed irradiation direction and a light-receiving unit (light-receiving pixel array 42) that receives light irradiated from the irradiating unit and reflected from the subject, thereby determining distance information to the subject. The imaging device 1 also includes a control unit (camera control unit 30) that controls changing the irradiation range of light from the irradiating unit in response to changes in the imaging angle of view. This allows the ranging operation to be optimized according to the imaging angle of view. For example, in telephoto imaging, reducing the irradiation range enables ranging at a frame rate suitable for telephoto imaging and improved ranging accuracy suitable for telephoto imaging with a narrow depth of field. In addition, in wide-angle imaging, widening the irradiation range widens the ranging range, thereby enabling the ranging range to be appropriately covered for the angle of view. The imaging angle of view can change, for example, by changing an interchangeable lens, changing the magnification of the lens system 11 on the imaging unit 10, changing the imaging range of the image sensor 12, or changing the cropping range in the camera signal processing unit 13. It is appropriate to trigger changes in the imaging angle of view due to these factors to control the change in the irradiation range. For example, the processes in Figures 9, 15, 17, 20, 22, and 24 are examples in which the trigger is the replacement of an interchangeable lens or a change in magnification in the lens system 11 on the imaging unit 10 side, but the process can also be applied as a process in which the trigger is a change in the imaging range of the image sensor 12 or a change in the cropping range in the camera signal processing unit 13.

[0178] In the embodiment, the irradiation unit is configured with an array of multiple light-emitting elements, and the irradiation range is changed by variably setting the area within the area where the light-emitting elements are arranged, where the light-emitting elements are to perform irradiation. Specifically, the surface-emitting laser 21, which serves as the irradiation unit, is configured with an array of multiple light-emitting elements, each of which performs spot irradiation. The irradiation range is changed by variably setting the area within the area where the light-emitting elements are to perform spot irradiation (see Figures 4, 9, etc.). For example, in an active-type sensor for line distance measurement or area distance measurement, the irradiation range can be changed by changing the light-emitting area of ​​the light-emitting element that performs spot irradiation. This makes it possible to change the irradiation range regardless of the configuration of the optical system of the distance measurement module 20. Note that the distance measurement module 20 is not limited to a type that performs spot irradiation, and may also be a type that irradiates the subject with uniform light.

[0179] In the embodiment, the distance measuring module 20 includes the irradiation optical system 22 for the light from the surface-emitting laser 21, and the irradiation range is changed by changing the zoom magnification of the irradiation optical system 22 (see FIGS. 11 and 15, etc.). For example, by providing the zoom lens 22Z in the distance measuring module 20, the irradiation range can be changed by driving the zoom lens 22Z. This makes it possible to optimize the distance measuring operation.

[0180] In the embodiment, an example was given in which the illumination range of the distance measurement module 20 is changed by variably setting the area in the area where the light-emitting elements are arranged, where the light-emitting elements are to perform illumination, and also by changing the zoom magnification of the illumination optical system. Specifically, the illumination range is changed by variably setting the area in the surface-emitting laser 21, where the light-emitting elements are to perform spot illumination, and also by changing the zoom magnification of the illumination optical system 22 (see FIGS. 11 , 17 , 20 , 22 , etc.). Changing the light-emitting area of ​​the light-emitting element that performs spot illumination and changing the illumination range by driving the zoom lens 22Z are combined. This allows the illumination range (i.e., the distance measurement range) to be more flexibly adapted to changes in the imaging angle of view.

[0181] In the embodiment, an example has been given in which the camera control unit 30 associates predetermined metadata with a captured image when there is an area outside the irradiation range of the laser light within the image capture angle of view (see FIGS. 22 and 23). For example, this may be the case when the distance measurement range is narrower than the angle of view of the captured image. In this case, there is a subject for which distance measurement has not been possible. Therefore, predetermined metadata is associated with the captured image so that such a state can be recognized.

[0182] Furthermore, examples of metadata associated with a captured image include data indicating the ranging range, data indicating ranging coordinates, data indicating the number of ranging data, and data indicating the angle of view of the illumination range. The data indicating the ranging range on the captured image allows the range of the subject for which ranging was possible on the captured image to be identified. The data indicating the ranging coordinates allows the position for which ranging was possible on the captured image to be identified. The data indicating the number of ranging data allows the state in which the imaging angle of view has narrowed to the telephoto side to be understood. The data indicating the angle of view of the illumination range by the ranging module 20 allows the difference from the imaging angle of view to be understood. These make it possible to later determine whether the imaging angle of view and the ranging range match or deviate.

[0183] In the embodiment, an example has been given in which the camera control unit 30 performs control to notify the user when the image capture angle of view exceeds the angle of view corresponding to the range of control for changing the illumination range (see FIGS. 17 and 20, etc.). For example, when the ranging range cannot be optimized for the angle of view of the captured image, the user is notified so that some kind of action can be taken.

[0184] In the embodiment, an example was given in which the notification when the imaging angle of view exceeds the angle of view corresponding to the change control range of the illumination range is a notification recommending the attachment of the conversion lenses 18, 19 as the illumination optical system. Since it may be possible to optimize the ranging range by adjusting the illumination range to the captured image using the conversion lenses 18, 19, this is recommended to the user. Note that the notification may also be a notification that allows the user to select to continue ranging even if the illumination range cannot be adapted to the imaging angle of view.

[0185] In the embodiment, an example has been given in which, when the illumination range of the ranging module 20 is changed, distance information is calculated using a calculation process including correction according to predetermined conditions (see, for example, FIGS. 15, 17, 20, 22, and 24). A change in the illumination range may cause an error in the distance information obtained by the distance calculation process. Performing a calculation process including correction according to such an error maintains high-precision ranging. While the calculation as the correction process may be performed by the camera control unit 30, the ranging module 20, for example, the distance estimation processing unit 43, may perform the correction calculation and supply the distance information to the camera control unit 30.

[0186] Furthermore, the predetermined condition for performing calculation processing including correction is a condition where a change in the irradiation range causes a change in the optical path length from the emission of light from the surface-emitting laser 21 to the reception of light by the light-receiving pixel array 42. If changing the irradiation range causes the optical path length to change from the normal irradiation range, an error will occur in the calculated distance information. Therefore, calculation processing including correction is performed to cancel the change in the optical path length. This makes it possible to maintain high-precision distance measurement.

[0187] Furthermore, cases in which a change in the irradiation range causes a change in the optical path length are when the zoom magnification is changed by the zoom lens 22Z in the irradiation optical system 22, or when the conversion lens 18 is added to the irradiation optical system. For example, when the irradiation range is changed using the conversion lens 18 or the zoom lens 22Z, the optical path length changes compared to normal. Therefore, when the magnification of the zoom lens is changed or the conversion lens 18 is added, a calculation process is performed that includes a correction that takes into account the change in the optical path length, thereby maintaining high-precision distance measurement.

[0188] In the embodiment, the surface-emitting laser 21 is configured with an array of multiple light-emitting elements, each of which emits a spot light, and the spot illumination density for the captured image is changed in response to a change in the illumination range. For example, as shown in FIG. 4 , when the illumination range is changed by changing the light-emitting area of ​​the light-emitting element that emits the spot light, the spot density for the captured image changes. This allows distance measurement to be performed in an optimal range in response to a change in the imaging angle of view, even if the ranging module 20 has a fixed angle of view. On the other hand, when the illumination range is changed by varying the zoom magnification of the zoom lens 22Z in the ranging module 20, for example, the spot density for the captured image does not change. To maintain a constant measurement point density in the captured image, controlling the illumination range using the zoom lens 22Z is ​​preferable.

[0189] In the embodiment, an example has been given in which the camera control unit 30 determines whether to change the illumination range in response to a change in the imaging angle of view based on predetermined information (steps S102 and S104 in FIGS. 9 and 15). The illumination range of the ranging module 20 is not always changed in response to a change in the imaging angle of view, but is changed only when a change in the illumination range is appropriate. This allows the illumination range of the ranging module 20 to be changed when appropriate, depending on the application, state, mode, etc.

[0190] In this case, the predetermined information includes information on the presence or absence of lens information (step S104 in FIGS. 9 and 15). If information on the lens system 11 of the imaging unit 10, such as lens magnification information, cannot be obtained, the illumination range of the distance measuring module 20 cannot be set appropriately, so the illumination range is not changed. This prevents unnecessary changes to the illumination range.

[0191] The predetermined information also includes information on the purpose of the distance information obtained by the ranging module 20 (step S102 in FIGS. 9 and 15). For example, when the distance information is used for focus control or when it is associated with a captured image as metadata, it is preferable to change the irradiation range of the ranging module 20. On the other hand, when the distance information is used for calibrating the ranging module 20, when it is used for estimating the self-position of the imaging device 1, when it is used for estimating the shape of a subject (3D modeling), etc., it is preferable to obtain distance information over a range wider than the imaging angle of view, and therefore it is considered not to change the irradiation range in accordance with the imaging angle of view.

[0192] Furthermore, the information on the purpose of the distance information is assumed to be shooting mode information. For example, the shooting mode and the running application can determine the case in which imaging is performed. Therefore, the camera control unit 30 can determine the purpose of the distance information based on the shooting mode and the running application, and decide whether to change the irradiation range of the ranging module 20 in response to a change in the imaging angle of view by the imaging unit 10, etc. Note that if the imaging device 1 is configured as an information processing device such as a smartphone, the purpose of the distance information can be determined based on information on the running application. For example, if an application that captures an object for 3D modeling is running and imaging is performed, the shooting mode can be determined to be for object shape estimation.

[0193] In the embodiment, an example has been given in which the camera control unit 30 controls the ranging frame rate of the ranging module 20, or the irradiation period of the surface-emitting laser 21, or the light emission intensity of the surface-emitting laser 21 in response to a change in the irradiation range (see FIGS. 6, 8, 25, and 26). When the irradiation range of the ranging module 20 is changed, more optimized ranging processing may be achieved by controlling the ranging frame rate, the irradiation period tTX, and the light emission intensity.

[0194] In the above example, the camera control unit 30 switches between the distance measurement frame rate control, the irradiation period control, and the light emission intensity control depending on predetermined conditions (see FIG. 26). Which control is appropriate varies depending on the case. Therefore, predetermined conditions are determined to determine which of these controls to perform, thereby optimizing the control.

[0195] In addition, the predetermined conditions are exemplified as conditions relating to the shooting mode, exposure time, camera status (whether or not the camera is on standby for shooting), depth of field, lens focal length, subject light amount, and upper power limit (see FIG. 26). By determining which control is appropriate based on these conditions, it is possible to perform the optimal distance measurement operation according to the state and circumstances at the time.

[0196] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0197] The present technology may also be configured as follows. (1) An imaging device comprising: a distance measuring unit that has an irradiation unit with a fixed irradiation direction and a light receiving unit that receives light reflected from the subject of light emitted from the irradiation unit and that determines distance information to the subject; and a control unit that performs control to change the irradiation range of light from the irradiation unit in accordance with a change in an imaging angle of view. (2) The imaging device described in (1) above, in which the irradiation unit is configured with an array of multiple light-emitting elements, and the irradiation range is changed by variably setting an area within the area where the light-emitting elements are arranged where the light-emitting elements are caused to perform irradiation. (3) The imaging device described in (1) above, in which the distance measuring unit includes an irradiation optical system for light from the irradiation unit, and the irradiation range is changed by changing a zoom magnification of the irradiation optical system. (4) The imaging device described in (1) above, in which the distance measuring unit includes an irradiation optical system for light from the irradiation unit, which is configured with an array of multiple light-emitting elements, and the irradiation range is changed by variably setting an area within the area where the light-emitting elements are caused to perform irradiation where the light-emitting elements are caused to perform irradiation, and the irradiation range is also changed by changing a zoom magnification of the irradiation optical system. (5) The imaging device according to any one of (1) to (4) above, wherein the control unit associates predetermined metadata with the captured image when the imaging angle of view has an area outside the irradiation range of light from the irradiation unit. (6) The imaging device according to (5) above, wherein the predetermined metadata includes at least one of data indicating a ranging range, data indicating ranging coordinates, data indicating the number of ranging data, and data indicating the angle of view of the irradiation range of the irradiation unit. (7) The imaging device according to any one of (1) to (6) above, wherein the control unit performs control to execute a notification to a user when the imaging angle of view exceeds an angle of view corresponding to a control range for changing the irradiation range. (8) The imaging device according to (7) above, wherein the notification is a notification recommending the attachment of a conversion lens as an irradiation optical system for light from the irradiation unit. (9) The imaging device according to any one of (1) to (8) above, wherein when the irradiation range is changed, a process of calculating distance information by a calculation process including correction according to predetermined conditions is performed.(10) The imaging device according to (9) above, wherein the predetermined condition is a condition that a change in the illumination range causes a change in the optical path length from when light is emitted from the illumination unit to when it is received by the light receiving unit. (11) The imaging device according to (10) above, wherein a change in the illumination range causes a change in the optical path length when a zoom magnification is changed by a zoom lens in an illumination optical system for light from the illumination unit, or when a conversion lens is added as an illumination optical system for light from the illumination unit. (12) The imaging device according to any of (1) to (11) above, wherein the illumination unit is configured with an array of multiple light-emitting elements each performing spot illumination, and the spot illumination density for the captured image is changed in accordance with a change in the illumination range. (13) The imaging device according to any of (1) to (12) above, wherein the control unit determines whether or not to change the illumination range in accordance with a change in an imaging angle of view, based on predetermined information. (14) The imaging device according to (13) above, wherein the predetermined information includes information on the presence or absence of lens information. (15) The imaging device according to (13) or (14) above, wherein the predetermined information includes information on the purpose of the distance information obtained by the distance measuring unit. (16) The imaging device according to (15) above, wherein the purpose information is information determined based on shooting mode information or information on a running application. (17) The imaging device according to any of (1) to (16) above, wherein the control unit controls the ranging frame rate of the ranging unit, the irradiation period of the irradiation unit, or the light emission intensity of the irradiation unit, in accordance with a change in the irradiation range. (18) The imaging device according to (17) above, wherein the control unit switches between controlling the ranging frame rate of the ranging unit, the irradiation period of the irradiation unit, and the light emission intensity of the irradiation unit in accordance with a predetermined condition. (19) The imaging device according to (18) above, wherein the predetermined condition is a condition related to any one of the shooting mode, exposure time, camera status, depth of field, lens focal length, subject light amount, and upper power limit.(20) A distance measurement method for an imaging device including an imaging unit and a distance measurement unit having an imaging unit, an irradiation unit with a fixed irradiation direction, and a light receiving unit that receives light reflected from the subject of the light irradiated from the irradiation unit, and that obtains distance information to the subject, the distance measurement method performing control to change the irradiation range of light from the irradiation unit in accordance with a change in the imaging angle of view.

[0198] Furthermore, the present technology can also adopt the following configuration. (21) The imaging device according to (1) above, wherein the illumination unit is configured by an array of multiple light-emitting elements, each of which performs spot illumination, and the illumination range is changed by variably setting an area within the area where the light-emitting elements are arranged where the light-emitting elements are made to perform spot illumination. (22) The imaging device according to (1) above, wherein the distance measuring unit includes an illumination optical system for light from the illumination unit, which is configured by an array of multiple light-emitting elements, each of which performs spot illumination, and the illumination range is changed by variably setting an area within the area where the light-emitting elements are made to perform spot illumination, and the illumination range is also changed by changing a zoom magnification in the illumination optical system.

[0199] REFERENCE SIGNS LIST 1 Imaging device 10 Imaging unit 11 Lens system 12 Image sensor 13 Camera signal processing unit 14 Recording unit 15 Output unit 16 User interface unit 18, 19 Conversion lens 20 Distance measurement module 21 Surface emitting laser 22 Irradiation optical system 22Z Zoom lens 23 Light receiving optical system 23Z Zoom lens 24 Distance measurement sensor unit 25 Additional optical system 30 Camera control unit 41 Sensor control unit 42 Light receiving pixel array 43 Distance estimation processing unit

Claims

1. An imaging device comprising: an illumination unit with a fixed illumination direction; a light receiving unit that receives light reflected from the subject of the light illuminated by the illumination unit; a distance measuring unit that determines distance information to the subject; and a control unit that performs control to change the illumination range of the light from the illumination unit in response to changes in the imaging angle of view.

2. The imaging device according to claim 1, wherein the illumination section is configured with an array of a plurality of light-emitting elements, and the illumination range is changed by variably setting an area within the area in which the light-emitting elements are arranged where illumination is performed by the light-emitting elements.

3. The imaging device according to claim 1, wherein the distance measuring section includes an illumination optical system for the light from the illumination section, and the illumination range is changed by changing the zoom magnification of the illumination optical system.

4. The imaging device according to claim 1, wherein the distance measuring unit includes an irradiation optical system for the light from the irradiation unit configured with an array of a plurality of light-emitting elements, and the irradiation range is changed by variably setting an area within the area in which the light-emitting elements are arranged where the light-emitting elements are caused to irradiate, and the irradiation range is also changed by changing the zoom magnification in the irradiation optical system.

5. The imaging device according to claim 1, wherein the control unit associates predetermined metadata with the captured image when the imaging angle of view includes an area outside the range of light irradiation from the irradiation unit.

6. The imaging device according to claim 5, wherein the predetermined metadata includes at least one of data indicating a distance measurement range, data indicating distance measurement coordinates, data indicating the number of distance measurement data, or data indicating the angle of view of the irradiation range of the irradiation section.

7. The imaging device according to claim 1, wherein the control unit performs control to execute a notification to a user when an imaging angle of view exceeds an angle of view corresponding to a change control range of the irradiation range.

8. The imaging device according to claim 7, wherein the notification is a notification recommending the attachment of a conversion lens as an irradiation optical system for irradiating light from the irradiation unit.

9. The imaging device according to claim 1, further comprising a process for calculating distance information using a calculation process including correction in accordance with predetermined conditions when the illumination range is changed.

10. The imaging device according to claim 9, wherein the predetermined condition is that a change in the irradiation range causes a change in the optical path length from the emission of light from the irradiation section to the reception of the light by the light receiving section.

11. The imaging device according to claim 10, wherein a change in the irradiation range causes a change in the optical path length when the zoom magnification is changed by a zoom lens in the irradiation optical system for the light from the irradiation unit, or when a conversion lens is added as an irradiation optical system for the light from the irradiation unit.

12. The imaging device according to claim 1, wherein the illumination section is configured with an array of a plurality of light-emitting elements each performing spot illumination, and the spot illumination density for the captured image is changed in response to a change in the illumination range.

13. The imaging device according to claim 1, wherein the control unit determines whether or not to change the illumination range in response to a change in the imaging angle of view based on predetermined information.

14. The imaging device according to claim 13, wherein the predetermined information includes information on the presence or absence of lens information.

15. The imaging device according to claim 13, wherein the predetermined information includes information on the purpose of the distance information obtained by the distance measuring unit.

16. The imaging device according to claim 15, wherein the information on the purpose is determined based on shooting mode information or information on a running application.

17. The imaging device according to claim 1, wherein the control unit controls the distance measurement frame rate of the distance measurement unit, or the irradiation period of the irradiation unit, or the light emission intensity of the irradiation unit, in response to the change in the irradiation range.

18. The imaging device according to claim 17, wherein the control unit switches between controlling the ranging frame rate of the ranging unit, controlling the irradiation period of the irradiation unit, and controlling the light emission intensity of the irradiation unit according to predetermined conditions.

19. The imaging device according to claim 18, wherein the predetermined condition is any one of a shooting mode, an exposure time, a camera status, a depth of field, a lens focal length, an amount of light on a subject, and an upper limit of power.

20. A distance measurement method for an imaging device equipped with an imaging unit, an irradiation unit with a fixed irradiation direction, and a distance measurement unit having a light receiving unit that receives reflected light from the subject of the light irradiated from the irradiation unit, and that obtains distance information to the subject, the distance measurement method performing control to change the irradiation range of light from the irradiation unit in response to changes in the imaging angle of view.

Citation Information

Patent Citations

  • Stroboscopic photographing system, stroboscopic device and camera

    JP2006154368A

  • Distance detection camera

    JP2014157044A

  • Distance detector and imaging apparatus

    JP2021026236A

  • Imaging apparatus, imaging method, and program

    JP2023047714A

  • Method and system for welding quality control through automatic hot air heater temperature management

    KR102548955B1