Imaging device, image processing device, and image processing method

Abstract

To provide a 360-degree spherical imaging apparatus, an image processing apparatus, and an image processing method that certainly can cover a 360-degree spherical area and simultaneously measure all directions.SOLUTION: A 360-degree spherical imaging apparatus according to an embodiment is provided with at least two projectors, at least two light receiving units, and a housing on which the at least two projectors and the at least two light receiving units are arranged. On the housing, each of the at least two projectors has an optical axis substantially parallel to any one of the other projectors, and with the optical axis opposite to a light projection direction substantially above of any one of the other projectors, is arranged outside respective light receiving ranges of the at least two light receiving units. On the housing, each of the at least two light receiving units has an optical axis substantially parallel to any one of the other light receiving units, and with the optical axis opposite to a light receiving direction substantially above of any one of the other light receiving units, is arranged outside respective light projecting ranges of the at least two projectors.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention , taken The present invention relates to an image processing apparatus, an image processing method, and an image processing method. [Background technology]

[0002] One method for measuring the distance from an imaging device to a subject is the TOF (Time of Flight) method, which irradiates the subject with distance measuring light and calculates the distance from the time difference of the reflected light. In this method, infrared distance measuring light that is intensity-modulated according to a specified irradiation pattern is irradiated toward the subject, and the distance measuring light reflected by the subject is received by an infrared imaging element, and the time difference from irradiation to reception is detected for each pixel according to the irradiation pattern, and the distance is calculated. The calculated distance values ​​are collected in bitmap form for each pixel and saved as a "distance image." A distance image generating device (imaging device) that uses this type of method is called a TOF camera. Summary of the Invention [Problem to be solved by the invention]

[0003] However, when trying to measure all directions, including the vertical and horizontal directions, simultaneously using a conventional TOF camera, multiple optical systems are captured within each other's angles of view, making it impossible to measure distances related to the captured angles of view (directions).

[0004] The present invention reliably covers the entire celestial sphere and can measure all directions simultaneously. Shooting It is an object of the present invention to provide an image processing apparatus, an image processing method, and an image processing method. [Means for solving the problem]

[0005] An imaging device according to an embodiment of the present invention includes at least two light projecting units, at least two light receiving units, and a housing in which the at least two light projecting units and the at least two light receiving units are disposed. Each of the light projecting sections includes a light projecting lens that diffuses and projects light.The light projection range of two of the at least two light projection units is 360 degrees. U2 1 The above Light projection lens are arranged in different directions from each other. The at least two light receiving sections are arranged outside the light projection ranges of the at least two light projecting sections. Effect of the Invention

[0006] The present invention relates to Shooting According to the imaging device, the image processing device, and the image processing method, it is possible to reliably cover the entire celestial sphere and measure all directions simultaneously. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a perspective view that illustrates an example of the appearance of an imaging device 1a according to an embodiment. [Diagram 2] FIG. 2 is a top view of the imaging device 1a. [Diagram 3] FIG. 3 is a diagram for explaining the configuration of the light projection unit 11a (11b) having the TOF light projection lens 110a (or the TOF light projection lens 110b). [Figure 4] FIG. 4 is a diagram for explaining the configuration of the light receiving section 12a (12b) having the TOF light receiving lens 120a (or the TOF light receiving lens 120b). [Diagram 5] FIG. 5 is a block diagram showing an example of a hardware configuration of the imaging device 1 according to the embodiment. [Figure 6] FIG. 6 is a functional block diagram for explaining functions of the imaging device 1 according to the embodiment. [Figure 7] FIG. 7 is a diagram for explaining the distance calculation process executed by the imaging device 1a. [Figure 8] FIG. 8 is a flowchart showing the flow of the imaging process of the omnidirectional range image. [Figure 9] FIG. 9 is a flowchart showing the process of combining the range image data in step S4 of FIG. [Figure 10]FIGS. 10(a) and 10(b) are diagrams for explaining the fisheye image format. [Figure 11] 11(a) and 11(b) are diagrams for explaining the correspondence between a sphere and an equirectangular image in the equirectangular image format. [Figure 12] FIG. 12 is a diagram for explaining a change in the image format of a distance image following a process of combining distance image data. [Figure 13] FIG. 13 is a diagram illustrating a celestial sphere TOF camera based on a conventional TOF camera arrangement. [Figure 14] FIG. 14 is a diagram showing an example of an imaging device 1b according to a first modified example of the embodiment, as viewed from the top side of a housing 10b. [Figure 15] FIGS. 15(a) and 15(b) are diagrams for explaining an imaging device 1c according to a second modified example of the embodiment. [Figure 16] FIGS. 16(a) and 16(b) are diagrams for explaining an imaging device 1d according to a third modified example of the embodiment. [Figure 17] FIG. 17 is a diagram illustrating a hardware configuration of an imaging device 1d according to the third modified example. [Figure 18] FIG. 18 is a functional block diagram for explaining the functions of an imaging device 1d according to the third modified example. [Figure 19] FIG. 19 is a flowchart showing the flow of an RGB image combination process executed by an imaging device 1d according to the third modified example. [Figure 20] FIG. 20 is a diagram showing an example of an imaging device 1f according to a fourth modified example of the embodiment, as viewed from the top side of a housing 10f. [Figure 21] 21(a) and 21(b) are diagrams for explaining an imaging device 1g according to a fifth modified example of the embodiment. [Figure 22] FIG. 22 is a diagram showing an example of an imaging device 1h according to a sixth modified example of the embodiment, as viewed from the top side of a housing 10h. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Please refer to the attached drawing below. , taken Embodiments of an image device, an image processing apparatus and an image processing method are described in detail below.

[0009] Fig. 1 is a perspective view showing an example of the appearance of an imaging device 1a according to an embodiment. Fig. 2 shows a top view of the imaging device 1. The imaging device 1a according to the embodiment shown in Figs. 1 and 2 is an omnidirectional imaging device serving as an omnidirectional camera capable of capturing a 360-degree distance image.

[0010] The imaging device 1a includes at least two light projecting units 11a and 11b, at least two light receiving units 12a and 12b, and a housing in which the at least two light projecting units 11a and 11b and the at least two light receiving units 12a and 12b are arranged. In the imaging device 1a, each of the at least two light projecting units 11a and 11b is arranged outside the light receiving range of each of the at least two light receiving units in the housing 1a, with the optical axis of each of the at least two light projecting units being approximately parallel to that of any one of the other light projecting units and with the light receiving direction of the optical axis of each of the at least two light receiving units being approximately opposite to that of any one of the other light projecting units. In the imaging device 1a, each of the at least two light receiving units 12a and 12b is arranged outside the light projecting range of each of the at least two light projecting units in the housing, with the optical axis of each of the at least two light receiving units being approximately parallel to that of any one of the other light receiving units and with the light receiving direction of the optical axis of each of the at least two light receiving units being approximately opposite to that of any one of the other light receiving units.

[0011] 1 and 2, the imaging device 1a includes a light projecting unit 11a having a TOF projecting lens 110a, a light projecting unit 11b having a TOF projecting lens 110b, a light receiving unit 12a having a TOF receiving lens 120a, a light receiving unit 12b having a TOF receiving lens 120b, a housing 10a, a grip unit 14, and an imaging button 15. In addition, since the TOF projecting lens 110b and the TOF receiving lens 120b are on the rear side of the housing 10a in the figure, only the reference symbols are shown in the figure.

[0012] The housing 10a has a cube shape, for example, as shown in FIG. 1, and houses therein modules corresponding to the optical system and image processing system of the imaging device 1a.

[0013] The TOF projector lens 110a and the light projecting unit 11a having the TOF projector lens 110a are disposed on one of the side surfaces of the housing 10a, while the TOF projector lens 110b and the light projecting unit 11b having the TOF projector lens 110b are disposed on a surface opposite to the surface on which the TOF projector lens 110a is disposed.

[0014] The TOF projection lenses 110a and 110b are fisheye lenses with a full angle of more than 180 degrees. Therefore, the projection range realized by using the TOF projection lenses 110a and 110b is more than 180 degrees, like the projection ranges 110a and 110b shown in FIG.

[0015] The TOF receiving lens 120a and the light receiving unit 12a having the TOF receiving lens 120a are disposed on one of the side surfaces of the housing 10a on which the TOF projector lenses 110a and 110b are not disposed. The TOF receiving lens 120b and the light receiving unit 12b having the TOF receiving lens 120b are disposed on the surface opposite to the surface on which the TOF receiving lens 120a is disposed.

[0016] The TOF receiving lenses 120a and 120b are fisheye lenses with a full angle of more than 180 degrees. Therefore, the light receiving range realized by using the TOF receiving lenses 120a and 120b is more than 180 degrees, like the light receiving ranges 120a and 120b shown in FIG.

[0017] 3 is a diagram for explaining the configuration of the light projecting unit 11a (11b) having the TOF projecting lens 110a (or the TOF projecting lens 110b). Note that the arrows in the figure are schematic diagrams showing the traveling direction of light.

[0018] As shown in the figure, the light projecting unit 11a (11b) has a surface-emitting light source 111a (111b) and a TOF light projecting lens 110a (110b) having a light projection area of ​​more than 180 degrees.

[0019] The surface-emitting light source 111a (111b) is a surface-shaped light source and is built into the housing 10a. For example, the surface-emitting light source 111a (111b) may be a two-dimensional array of vertical cavity surface-emitting lasers (VCSELs). The light emitted from the surface-emitting light source 111a (111b) is diffused by the TOF projection lens 110 (110b) and projected over a projection range of 180 degrees or more.

[0020] 4 is a diagram for explaining the configuration of the light receiving section 12a (12b) having the TOF light receiving lens 120a (or the TOF light receiving lens 120b). Note that the arrows in the figure are schematic diagrams showing the traveling direction of light.

[0021] As shown in the figure, the light receiving section 12 has a TOF sensor 121a (121b) and a TOF light receiving lens 120a (120b) having a light receiving area of ​​more than 180 degrees.

[0022] The light received by the TOF light receiving lens 120a (120b) is converged by the lens as shown in FIG. 4 and enters the TOF sensor 121a (121b).

[0023] The TOF sensor 121a (121b) is a two-dimensional area sensor similar to an image sensor used in a digital camera, etc., and is built into the housing 10a. The TOF sensor 121a (121b) calculates distance data for each pixel and generates distance image data.

[0024] Returning to FIG. 1, the grip portion 14 is a holding portion for holding the imaging device 1a by hand or the like.

[0025] An imaging button 15 for issuing an instruction to capture an image is provided on the grip section 14. The grip section 14 may further be provided with an operation section for performing operations such as setting the imaging device 1a, and a display section for displaying a status and the like.

[0026] Fig. 5 is a block diagram showing an example of a hardware configuration of an image capturing device 1a according to an embodiment of the present invention, which includes a distance measurement control unit 23 in addition to surface-emitting light sources 111a and 111b and TOF sensors 121a and 121b.

[0027] The distance measurement control unit 23 is connected to the surface-emitting light sources 111a and 111b and the TOF sensors 121a and 121b, and is built into the housing 10a. The distance measurement control unit 23 has a CPU (Central Processing Unit) 231, a ROM (Read Only Memory) 232, a RAM (Random Access Memory) 233, an SSD (Solid State Drive) 234, a light source drive circuit 235, a sensor I / F (Interface) 236, and an input / output I / F 237. These are electrically connected to each other via a system bus 238.

[0028] The CPU 231 reads programs and data from a storage device such as a ROM 232 or an SSD 234. The CPU 231 reads the data into the RAM 233 and executes processing to control the entire distance measurement control unit 23 and to realize functions described later. Note that some or all of the functions of the CPU 231 may be realized by electronic circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0029] The ROM 232 is a non-volatile semiconductor memory (storage device) capable of retaining programs and data even when the power is turned off. The ROM 232 stores programs and data such as a BIOS (Basic Input / Output System) executed when the processing unit 100 is started up, and OS settings.

[0030] The RAM 233 is a volatile semiconductor memory (storage device) that temporarily stores programs and data.

[0031] The SSD 234 is a non-volatile memory that stores various data and programs for executing processes by the distance measurement control unit 23. The SSD may be a hard disk drive (HDD) or the like.

[0032] The light source driving circuit 235 is an electric circuit electrically connected to the surface-emitting light sources 111a, 111b, and outputs a driving signal such as a driving voltage to the surface-emitting light sources 111a, 111b in response to a control signal input from the CPU 231 or the like. The light source driving circuit 235 drives a plurality of light-emitting units included in the surface-emitting light sources 111a, 111b to emit light in response to the control signal. A rectangular wave, a sine wave, or a voltage waveform of a predetermined waveform can be used as the driving signal. The light source driving circuit 235 can change the frequency of the voltage waveform to modulate the frequency of the driving signal.

[0033] The sensor I / F 236 is an interface that is electrically connected to the TOF sensors 121a and 121b and inputs the phase signals output by the TOF sensors 121a and 121b. The input / output I / F 237 is an interface for connecting to an external device such as the main controller 80 or a PC (Personal Computer).

[0034] 6 is a functional block diagram for explaining the functions of the imaging device 1a according to the embodiment. In FIG. 6, the imaging device 1a includes a distance measurement control unit 23 as one of its functions.

[0035] The distance measurement control unit 23 includes a light emission control unit 238 and a light reception processing unit 239. The distance measurement control unit 23 controls light emission by the surface-emitting light sources 111a and 111b via the light emission control unit 238 and light reception by the TOF sensors 121a and 121b via the light reception processing unit 239 in synchronization with each other.

[0036] The light emission control unit 238 has at least a drive signal output unit 238 as a function of the imaging device 1a.

[0037] The drive signal output unit 238 outputs a drive signal to the surface-emitting light sources 111a and 111b to cause them to emit light simultaneously. The drive signal output unit 238 also outputs a drive signal with a predetermined voltage waveform and a predetermined emission frequency, thereby enabling time modulation (temporal control) of the light emitted by the surface-emitting light sources 111a and 111b. In the present embodiment, as an example, a square wave or sine wave drive signal with a frequency of about MHz (megahertz) is output to the surface-emitting light sources 111a and 111b at a predetermined timing.

[0038] The light receiving processing section 239 has at least a phase signal input section 239a, a distance image acquisition section 239b, a storage section 239c, and a distance image combination section 239d as functions of the imaging device 1a.

[0039] The phase signal input unit 239a is realized by the sensor I / F 236 and the like, and inputs the phase signals output by the TOF sensors 121a and 121b. The phase signal input unit 239a can input the phase signals of each pixel two-dimensionally arranged in the TOF sensors 121a and 121b. The phase signal input unit 239a outputs the input phase signals to the distance image acquisition unit 239b. In this embodiment, two TOF sensors 121a and 121b are connected to the phase signal input unit 239a. Therefore, two sets of phase signals are output.

[0040] The distance image acquisition unit 239b acquires distance image data from the imaging device 1a to the target object based on the phase signals of the TOF sensors 121a and 121b for each pixel input from the phase signal input unit 239a. Here, the distance image is an image generated by arranging the distance data acquired for each pixel two-dimensionally according to the position of the pixel, for example, an image generated by converting the distance into the brightness of the pixel. The distance image acquisition unit 239b outputs the two acquired distance image data to the storage unit 239c.

[0041] The storage unit 239c is realized by the RAM 233 or the like, and temporarily stores the distance image data input from the distance image acquisition unit 239b.

[0042] The distance image combination unit 239d reads out the two distance image data temporarily stored in the storage unit 239c, and combines them to generate one piece of omnidirectional distance image data.

[0043] The distance image combination unit 239d is realized by executing a control program by the CPU 231. However, without being limited to this example, a part or all of the distance image combination unit 239d may be realized by dedicated hardware designed to execute similar functions, for example, a semiconductor integrated circuit such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), or an FPGA (field programmable gate array), or a conventional circuit module.

[0044] In the above description, the configuration shown in Fig. 6 is described as being built into the imaging device 1a, but this is not limited to this example. For example, the configuration shown in Fig. 6 may be provided in an external information processing device that can be connected to the imaging device 1a.

[0045] (Distance calculation process) Next, the distance calculation process executed by the imaging device 1a will be described.

[0046] TOF sensors 121a and 121b each have two charge accumulation sections (first charge accumulation A and second charge accumulation B) in one light receiving element, and can quickly switch which one accumulates the charge. This makes it possible to simultaneously detect two opposite phase signals for one square wave. For example, it is possible to simultaneously detect a combination of 0 degrees and 180 degrees, or a combination of 90 degrees and 270 degrees. Therefore, distance measurement is possible through two light projection and light reception processes.

[0047] 7 is a diagram for explaining the distance calculation process executed by the imaging device 1a, and is a graph showing the accumulation timing of the first charge accumulation unit α and the second charge accumulation unit β with respect to the projected light and the reflected light. In the figure, the hatched area indicates the period during which the electric charge is accumulated.

[0048] In order to increase the amount of accumulated charge, it is preferable to project light in a repeating pattern of square waves with a duty ratio of 50% rather than a single square wave, and to repeatedly switch between the first charge accumulation A and the second charge accumulation B accordingly.

[0049] Since the four phase signals A0, A90, A180, and A270 are phase signals that are divided into four phases of 0°, 90°, 180°, and 270° in time with respect to the pulse period of the irradiated light, respectively, the phase difference angle φ can be calculated using the following equation (1). φ=Arctan{(A90-A270) / (A0-A180)} (1)

[0050] Using the phase difference angle φ, the delay time Td can be calculated from the following equation (2). Td = φ / 2π × T (T = 2T0, T0: pulse width of the irradiated light) (2)

[0051] The distance value d to the target object can be calculated using the delay time Td according to the following equation (3). d = Td × c ÷ 2 (c: speed of light) … (3)

[0052] Next, imaging processing (of a celestial sphere distance image) including distance calculation processing executed by the imaging device 1a will be described.

[0053] 8 is a flowchart showing the flow of the imaging process of the omnidirectional distance image. As shown in the figure, the drive signal output unit 238a outputs a drive signal to the light projecting units 11a and 11b (step S1), and causes the surface-emitting light sources 111a and 111b in the light projecting units 11a and 11b to emit light simultaneously.

[0054] In step S1, the drive signal output unit 238a outputs a drive signal with a predetermined voltage waveform and a predetermined light emission frequency to time-modulate (temporally control) the light emission by the light-emitting unit. For example, the drive signal output unit 238a outputs a square wave or sine wave drive signal with a frequency of about MHz (megahertz) to the light-projecting units 11a and 11b at a predetermined timing.

[0055] The phase signal input unit 239a inputs the phase signals corresponding to the light emission of the first group outputted by the TOF sensors 121a and 121b of the light receiving units 12a and 12b, and outputs the input phase signals to the distance image acquisition unit 239b (step S2).

[0056] The distance image acquisition unit 239b acquires distance image data from the imaging device 1a to the object corresponding to the light emission of the first group in accordance with the phase signals for each pixel of the TOF sensors 121a, 121b input from the phase signal input unit 239a (step S3). The distance image acquisition unit 239b outputs the acquired distance image data to the storage unit 239c. The storage unit 239c temporarily stores the distance image data input from the distance image acquisition unit storage unit 239b.

[0057] The distance image combination unit 239d reads out the two distance image data temporarily stored in the storage unit 239c, and combines them to generate one omnidirectional distance image data (step S4).

[0058] FIG. 9 is a flowchart showing the process of combining the range image data in step S4 of FIG.

[0059] As shown in the figure, first, the distance image combination unit 239d acquires two distance image data in fisheye image format (step S401).

[0060] Here, the fisheye image format will be described. Fig. 10(a) is a diagram for explaining the fisheye image format, and is a diagram for explaining how the fisheye lens (i.e., the TOF receiving lenses 120a and 120b) projects three-dimensional incident light into two dimensions. Fig. 10(b) is a diagram showing a two-dimensional plane onto which the three-dimensional incident light is projected in the relationship of Fig. 10(a).

[0061] As shown in Fig. 10(a), the incidence angle φ is expressed as the angle with respect to the optical axis when the intersection point between the plane tangent to the edge of the fisheye lens and the optical axis is taken as the vertex. A fisheye image captured by a fisheye lens with a field angle of more than 180° is an image of a scene approximately equivalent to a hemisphere from the capture position.

[0062] As shown in Figures 10(a) and 10(b), a fisheye image is generated at image height h corresponding to the incidence angle φ, whose relationship is determined by the projection function f(φ). The projection function f(φ) varies depending on the properties of the fisheye lens. For example, there is a fisheye lens that has a projection method called the equidistant projection method, which is expressed by the following equation (4) where φ is the image height h, the focal length fL, and the angle between the incidence direction and the optical axis (incident angle), and this is the one we will use here. h = fL × φ … (4)

[0063] 9, the distance image combination unit 239d converts each of the two distance image data in the fisheye image format into an equirectangular image format (step S402). In this step S402, the two distance image data are converted into the equirectangular image format based on the same coordinate system, which makes it easier to combine the images in the next step S403.

[0064] Here, the equirectangular image format will be described. The equirectangular image format is a method capable of expressing a celestial sphere image, and is a format of an image (equirectangular image) created using equirectangular projection. Equirectangular projection is a projection that expresses three-dimensional directions with two variables, such as the latitude and longitude of a globe, and displays the image on a plane so that the latitude and longitude are orthogonal. Therefore, an equirectangular image is an image generated using equirectangular projection, and is expressed in coordinates with two axes that are two angular variables of a spherical coordinate system.

[0065] Fig. 11(a) is a diagram showing an example of coordinates in an equirectangular image. Fig. 11(b) is a diagram showing an example of coordinates in a spherical surface. The correspondence between the spherical surface and the equirectangular image in the equirectangular image format will be described with reference to Fig. 11(a) and Fig. 11(b).

[0066] An equirectangular image is an image with pixel values ​​corresponding to angular coordinates, with horizontal angles of -180 to 180° and vertical angles of -90 to 90°, as shown in Fig. 11(a). The angular coordinates correspond to each point on the spherical surface, and are similar to the latitude and longitude coordinates of a globe.

[0067] On the other hand, the relationship between the coordinates of the fisheye image shown in Fig. 11(b) and the coordinates of the equirectangular image shown in Fig. 11(a) can be established by using the projection function f(φ) as described in Fig. 10(b). Therefore, by using this correspondence, image conversion can be performed from two pieces of image data in the fisheye image format to the equirectangular image format.

[0068] 9, distance image combination unit 239d combines the two distance image data in the equirectangular image format to generate one omnidirectional distance image data (step S403). Since the two distance image data input in step S401 cover an area with a full angle of view of more than 180 degrees, the omnidirectional distance image data generated in this way covers the omnidirectional area by appropriately combining the two distance image data. Note that an existing technique for combining multiple images can be used, and the method is not particularly limited.

[0069] FIG. 12 is a diagram for explaining a change in the image format of a distance image following a process of combining distance image data.

[0070] As shown in the figure, distance images 1 and 2 input in step S401 are images in the fisheye image format. The fisheye image format is a so-called circular fisheye, which has a circular effective area, and the area outside of that is an invalid area, which is usually a black area as shown in FIG.

[0071] In step S402, the distance images 1 and 2 are converted into equirectangular image format images. The orientations (directions of light entering the lenses) of the two TOF receiving lenses 120a and 120b are opposite to each other. Therefore, when converted into the equirectangular image format based on the same spherical coordinate system, the effective area and the ineffective area are roughly reversed (shifted by 180 degrees). The two TOF receiving lenses 120a and 120b are fisheye lenses with an angle of view of more than 180 degrees, so they cover more than half of the sphere. Therefore, the effective areas of the distance images 1 and 2 as equirectangular image format images partially overlap.

[0072] In step S403, distance images 1 and 2 are combined to generate a celestial sphere distance image in equirectangular image format. The two distance images 1 and 2 in fisheye image format input in step S401 cover an area with a total angle of view of more than 180 degrees. Therefore, when the two distance images 1 and 2 in equirectangular image format generated from the distance images in fisheye image format are combined, they can cover the celestial sphere.

[0073] (effect) According to the omnidirectional imaging device according to the present embodiment described above, TOF projector lenses 110a, 110b (light projecting units 11a, 11b) with a light projection range of 180 degrees or more are arranged outside the light receiving ranges of TOF receiver lenses 120a, 120b (light receiving units 12a, 12b) with their optical axes substantially parallel and their light projection direction substantially opposite to that of any other TOF projector lens (light projector). Also, TOF receiver lenses 120a, 120b (light receiving units 12a, 12b) with a light reception range of 180 degrees or more are arranged outside the light projection ranges of TOF projector lenses 110a, 110b (light projecting units 11a, 11b) with their optical axes substantially parallel and their light projection direction substantially opposite to that of any other TOF receiver lens.

[0074] Therefore, the above-described arrangement of TOF projector lenses 110a, 110b, which makes the light projection range 180 degrees or more, can make the light projection range 360 ​​degrees. Also, the above-described arrangement of TOF receiver lenses 120a, 120b, which makes the light reception range 180 degrees or more, can make the light reception range 360 ​​degrees.

[0075] Furthermore, TOF projector lenses 110a, 110b are arranged so as not to enter (be captured in) light receiving ranges 110aR, 110bR (i.e., the angle of view of TOF receiver lenses 120a, 120b), and TOF receiver lenses 120a, 120b are arranged so as not to enter (be captured in) light projection ranges 120bR, 120bR (i.e., the angle of view of TOF projector lenses 110a, 110b). Therefore, in each of the light projection range and light receiving range, the light projecting section and the light receiving section do not interfere with each other, and imaging can be achieved in a full celestial sphere with no blind spots.

[0076] (Comparative Example) 13 is a diagram illustrating omnidirectional TOF camera 30 based on the arrangement of a conventional TOF camera. As shown in the figure, in omnidirectional TOF camera 30, TOF projector lens 110a and TOF receiver lens 120a are arranged side by side on one of the opposing side surfaces of housing 105 having a rectangular parallelepiped shape, for example, and TOF projector lens 110b and TOF receiver lens 120b are arranged side by side on the other of the opposing side surfaces of housing 105.

[0077] In the case of such a conventional configuration, the TOF receiving lens 120a is reflected in the light projection range 110aR of the TOF projector lens 110a, and the TOF projector lens 110a is reflected in the light reception range 120aR of the TOF receiver lens 120a. Similarly, the TOF receiver lens 120b is reflected in the light projection range 110bR of the TOF projector lens 110b, and the TOF projector lens 110b is reflected in the light reception range 120bR of the TOF receiver lens 120b. Therefore, in the conventional configuration, the angles of view of the TOF receiver lens and the TOF projector lens interfere with each other, making it impossible to obtain a full-sphere distance image covering a full angle of more than 180 degrees.

[0078] In contrast, in the imaging device 1a according to this embodiment, the light projecting section and the light receiving section do not interfere with each other in the light projection range and the light receiving range, respectively, and therefore imaging can be achieved in the entire celestial sphere without blind spots.

[0079] [First Modification of the Embodiment] Next, a first modified example of the embodiment will be described. Fig. 14 is a diagram showing an example of an imaging device 1b according to a first modified example of the embodiment, viewed from the top side of a housing 10b. In Fig. 14, the same reference numerals are used for parts common to Figs. 1 and 2 described above, and detailed description thereof will be omitted.

[0080] The imaging device 1a shown in Fig. 1 and Fig. 2 has a configuration in which TOF projector lenses 110a, 110b and TOF receiver lenses 120a, 120b are arranged on four side surfaces of a cubic housing 10a. In contrast, the imaging device 1b according to the present first modification is configured to include a housing 10b in which the boundary sides of each side surface of the cubic housing 10a in the imaging device 1a shown in Fig. 1 and Fig. 2 are chamfered. That is, the housing 10b has a chamfered shape such that the corners between the TOF projector lenses 110a, 110b and the TOF receiver lenses 120a, 120b of the cubic housing 10a are cut off.

[0081] By giving housing 10b such a shape, even if housing 10b is made large relative to the sizes of TOF projector lenses 110a, 110b and TOF receiver lenses 120a, 120b, it is possible to prevent housing 10b from being reflected within the angle of view of the lenses.

[0082] In addition, a solid body obtained by chamfering a cube can be regarded as a cube by considering each face that is not chamfered as a virtual face that constitutes the solid body.

[0083] [Second Modification of the Embodiment] Next, a second modified example of the embodiment will be described. FIG. 15(a) is a perspective view that shows a schematic example of the appearance of an imaging device 1c according to a second modified example of the embodiment. FIG. 15(b) is a view of the imaging device 1c according to the second modified example of the embodiment seen from the top side of the housing 10c. In FIG. 15(a) and FIG. 15(b), the same reference numerals are used for the parts common to FIG. 1 and FIG. 2 described above, and detailed description will be omitted. In FIG. 15(a), the TOF projection lens 120a and the TOF receiving lens 120b are on the rear side of the housing 10c in the figure, so only the reference numerals are shown in the figure.

[0084] The imaging device 1a shown in Figures 1 and 2 has a cubic housing 10a. In contrast, the imaging device 1c according to the second modification has a housing 10c whose upper surface is a rectangle with two opposing sides of equal length and two sides that meet at an apex of different lengths, as shown in Figures 15(a) and 15(b).

[0085] The housing 10b may be a rectangular parallelepiped as long as the rectangular top surface has a ratio of the length of the short side to the length of the long side that is equal to or less than a predetermined value, that is, the shape is such that each lens does not reflect another lens within its angle of view.

[0086] Furthermore, the housing in which the TOF projector lenses 110a, 110b and the TOF receiver lenses 120a, 120b are arranged may have any shape as long as it is surrounded by a six-sided rectangle, has two opposing parallel surfaces, and has at least four of the six surfaces on each of which either the TOF projector lenses 110a, 110b or the TOF receiver lenses 120a, 120b is arranged. For example, the housing may have a shape called a parallelepiped, in which all six surfaces are parallelograms.

[0087] [Third Modification of the Embodiment] Next, a third modified example of the embodiment will be described. An imaging device 1d according to the third modified example of the embodiment has a configuration capable of acquiring a celestial sphere RGB image in addition to a celestial sphere distance image.

[0088] Fig. 16(a) is a perspective view showing an example of the appearance of an imaging device 1d according to a third modified example of the embodiment, and Fig. 16(b) is a view showing the imaging device 1d according to the third modified example of the embodiment as viewed from the top side of the housing 10d.

[0089] 16(a) and 16(b), the imaging device 1d includes RGB sensors 13a and 13b arranged on the top and bottom surfaces of the housing 10d, respectively, in addition to the configuration of the imaging device 1a shown in Fig. 1 and Fig. 2. The RGB sensors 13a and 13b are devices that include RGB light receiving lenses 130a and 130b with a full angle of more than 180 degrees and an area sensor, and obtain visible light imaging (RGB image) in a fisheye image format using visible light.

[0090] The RGB light receiving lenses 130a, 130b are disposed in the housing 10d outside the light projection ranges 110aR, 110bR of the at least two TOF light projecting lenses 110a, 110b, and outside the light receiving ranges 120aR, 120bR of the at least two TOF light receiving units. The RGB light receiving lenses 130a, 130b are disposed in the housing 10d so that their angle of view ranges are completely opposite to each other.

[0091] That is, as shown in Fig. 16(a) and Fig. 16(b), in the housing 10d of the imaging device 1d, the same lenses are arranged on two opposing faces of a cube or rectangular parallelepiped. In other words, each lens is arranged so that none of the lenses is captured in the field of view of another lens (the TOF projector lenses 110a, 110b, the TOF receiver lenses 120a, 120b, and the RGB receiver lenses 130a, 130b are not captured in each other). Therefore, it is possible to acquire a omnidirectional distance image and a omnidirectional RGB image with no blind spots.

[0092] Fig. 17 is a diagram for explaining a hardware configuration of an imaging device 1d according to the third modified example. As shown in the figure, the imaging device 1d further includes an RGB sensor I / F 240 in addition to the hardware configuration of the imaging device 1a shown in Fig. 5, in addition to the two area sensors 131a and 131b.

[0093] The area sensors 131a and 131b are sensors that are used in general digital cameras, and are capable of acquiring visible light images (RGB images).

[0094] The RGB sensor I / F 240 is an interface that is electrically connected to the area sensors 131a and 131b and receives the RGB signals output from the area sensors 131a and 131b.

[0095] 18 is a functional block diagram for explaining the functions of an imaging device 1d according to the third modified example. As shown in the figure, the imaging device 1d further includes an RGB image processing unit 241 in addition to the functions of the imaging device 1a shown in FIG.

[0096] The RGB image processing unit 241 has an RGB image input unit 241a, an RGB image storage unit 241b, and an RGB image combining unit 241c.

[0097] The RGB image input unit 241a inputs the RGB image output by the area sensors 131a and 131b. For example, the RGB image input unit 241a can input RGB signals for each pixel arranged two-dimensionally in the area sensors 131a and 131b. The RGB image input unit 241a outputs the input FRGB image to the RGB image storage unit 241b (in this modification, since two area sensors 131a and 131b are connected to the RGB image processing unit 241, two RGB images are output). The RGB image input unit 241a is realized by the RGB sensor I / F 240, etc.

[0098] The RGB image storage unit 241b is realized by the RAM 233 or the like, and temporarily stores the RGB image data input from the RGB image input unit 241a.

[0099] The RGB image combining unit 241c reads out two pieces of RGB image data temporarily stored in the RGB image storage unit 241b, and combines them to generate one piece of spherical RGB image data. The RGB image combining unit 241c is realized by the CPU 231 executing a control program. However, without being limited to this example, a part or the whole of the RGB image combining unit 241c may be realized by dedicated hardware designed to execute similar functions, such as a semiconductor integrated circuit such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), or an FPGA (field programmable gate array), or a conventional circuit module.

[0100] The distance measurement control unit 23 can synchronize and control the light emission of the surface-emitting light sources 111a, 111b and the light reception by the TOF sensors 121a, 121b, and can also synchronize the light reception by the area sensors 131a, 131b, thereby enabling distance images and RGB images to be acquired at the same time.

[0101] FIG. 19 is a flowchart showing the flow of an RGB image combination process executed by an imaging device 1d according to the third modified example.

[0102] As shown in the figure, the RGB image input unit 241a inputs two RGB image data in a fisheye image format (step S11). The RGB image storage unit 241b temporarily stores the two RGB image data input by the RGB image input unit 241a.

[0103] The RGB image storage unit 241b converts each RGB image data into an equirectangular image format (step S12). As described above, the equirectangular image format is a format capable of expressing a spherical image. The RGB image storage unit 241b converts the two RGB image data into the equirectangular image format based on the same coordinate system, thereby facilitating image combination in the next step.

[0104] The RGB image storage unit 241b combines the two RGB image data generated in step S12 to generate one piece of spherical RGB image data (step S13). The two pieces of input RGB image data cover an area with a full angle of view of more than 180 degrees. Therefore, the spherical RGB image data generated by appropriately connecting the two pieces of RGB image data can cover the spherical area.

[0105] The combining process in step S13 can use existing techniques for joining multiple images, and there are no particular limitations on the method.

[0106] [Fourth Modification of the Embodiment] Next, a fourth modified example of the embodiment will be described. Fig. 20 is a diagram showing an example of an imaging device 1f according to the fourth modified example of the embodiment, viewed from the top side of a housing 10f. In Fig. 20, the same reference numerals are used for the parts common to Figs. 1 and 2 described above, and detailed description thereof will be omitted.

[0107] The imaging device 1a shown in Fig. 1 and Fig. 2 is configured to include a cubic housing 10a. In contrast, as shown in Fig. 20, the imaging device 1f according to the second modification has housing 10f configured as a hexagonal prism, and TOF projector lenses 110a and 110b with a projection range of 180 degrees or more are arranged on six faces (side faces) of the hexagonal prism in housing 10f, with their optical axes approximately parallel to each other and with their optical axes approximately opposite to the projection direction of any one of the other TOF projector lenses, outside the light receiving ranges of the TOF receiver lenses 120a and 120b, and outside the light receiving ranges of the RGB receiver lenses 130a and 130b. TOF receiving lenses 120a, 120b, which have a light receiving range of 180 degrees or more, are arranged outside the light projection ranges of TOF projector lenses 110a, 110b and outside the light receiving ranges of RGB receiving lenses 130a, 130b, with their optical axes approximately parallel and their light projection directions approximately opposite to those of any other TOF receiving lens. Furthermore, RGB receiving lenses 130a, 130b, which have a light receiving range of 180 degrees or more, are arranged outside the light projection ranges of TOF projector lenses 110a, 110b and outside the light receiving ranges of TOF receiving lenses 120a, 120b, with their angle of view ranges opposite to each other.

[0108] That is, by forming the housing 10f in a hexagonal prism shape, the TOF projector lenses 110a and 110b, the TOF receiver lenses 120a and 120b, and the RGB receiver lenses 130a and 130b can be arranged so that they do not appear in each other's images. Therefore, it is possible to obtain a omnidirectional distance image and a omnidirectional RGB image with no blind spots.

[0109] Furthermore, compared to the third modified example of the embodiment, the absence of a lens on the underside makes it easier to ensure space for arranging a grip and an image capture button.

[0110] [Fifth Modification of the Embodiment] Next, a fifth modified example of the embodiment will be described. An image pickup device 1g according to the fifth modified example of the embodiment has a configuration in which the number of (TOF) light receiving units is increased and the angle of view covered by one light receiving unit is narrowed, thereby improving the resolution of the omnidirectional range image.

[0111] Fig. 21(a) is a perspective view that shows an example of the appearance of an imaging device 1g according to a fifth modified example of the embodiment. Fig. 21(b) is a perspective view that shows an example of the appearance of an imaging device 1g according to a fifth modified example of the embodiment, seen from a viewpoint on the opposite side to the viewpoint of Fig. 21(a). That is, Figs. 21(a) and (b) show the same imaging device 1g seen from different directions (an xyz coordinate system is shown on the left, and the relationship is such that they are shown from directions rotated 180 degrees around the z axis).

[0112] As shown in Figures 21(a) and 21(b), the imaging device 1g includes two TOF projector lenses 110a, 110b (i.e., two light projecting units 11a, 11b), RGB receiving lenses 130a, 130b (i.e., two RGB sensors 13a, 13b), six TOF receiving lenses 120a, 120b, 120c, 120d, 120e, and 120f (i.e., six TOF sensors 12a, 12b, 12c, 12d, 12e, and 12f), and a housing 10g.

[0113] The housing 10g has a shape in which each vertex of a cube is cut off (chamfered) by a plane passing through the midpoint of the three sides intersecting at the vertex. The six TOF receiving lenses 120a, 120b, 120c, 120d, 120e, and 120f are disposed on each face (six faces) of the housing 10g, which is a cube shape before chamfering. The two TOF projector lenses 110a and 110b are disposed on the faces of the housing 10g formed by chamfering the cube, with their optical axes approximately parallel to each other, with their optical axes approximately opposite to the projection direction of any one of the other TOF projector lenses, outside the light receiving ranges of the TOF receiving lenses, and outside the light receiving ranges of the RGB light receiving lenses 130a and 130b (so that their angle of view ranges are opposite to each other). The two RGB light receiving lenses 130a, 130b are disposed on the housing 10g on the surfaces formed by chamfering a cube so that their angle of view ranges are opposite to each other.

[0114] That is, in the imaging device 1g, the two TOF projecting lenses 110a and 110b that cover the omnidirectional region, the two RGB receiving lenses 130a and 130b that cover the omnidirectional region, and the six TOF receiving lenses 120a, 120b, 120c, 120d, 120e, and 120f are arranged so as not to be captured by each other. Therefore, it is possible to obtain a omnidirectional RGB image with no blind spots, and a TOF-type omnidirectional distance image with no blind spots and higher resolution.

[0115] [Sixth Modification of the Embodiment] Next, a sixth modified example of the embodiment will be described. The imaging device 1h according to the sixth modified example of the embodiment is capable of acquiring a celestial sphere distance image by a stereo ranging method by adding two more RGB sensors to the imaging device 1f according to the fourth modified example.

[0116] Fig. 22 is a diagram showing an example of an imaging device 1h according to a sixth modified example of the embodiment, viewed from the top side of a housing 10h. In Fig. 22, parts common to Figs. 1, 2, and 20 described above are given the same reference numerals, and detailed description thereof will be omitted.

[0117] The imaging device 1g shown in FIG. 22 has a configuration in which, compared to the imaging device 1h shown in FIG. 20, an RGB light receiving lens 130c (RGB sensor 13c) is further provided on the top surface of a housing 10h having a hexagonal prism shape, and an RGB light receiving lens 130d (RGB sensor 13d) is further provided on the bottom surface.

[0118] That is, the imaging device 1g includes four RGB receiving lenses 130a, 130b, 130c, and 130d (that is, four RGB sensors 13a, 13b, 13c, and 13d). The RGB receiving lens 130a (RGB sensor 13a) and the RGB receiving lens 130b (RGB sensor 13b) are arranged so that their angle of view ranges are opposite to each other, and the RGB receiving lens 130c (RGB sensor 13c) and the RGB receiving lens 130d (RGB sensor 13d) are arranged so that their angle of view ranges are opposite to each other. Therefore, a celestial sphere distance image can be generated by the four RGB sensors 13a, 13b, 13c, and 13d using the stereo distance measurement method. Details of the generation of the celestial sphere distance image using the stereo distance measurement method will be omitted here.

[0119] In general, the characteristics of a spherical distance image produced by a stereo ranging system are different from those of a spherical distance image produced by a TOF system. For example, a spherical distance image produced by a TOF system is highly accurate and can measure distances in areas without texture, but tends to have low resolution. On the other hand, a spherical distance image produced by a stereo ranging system can measure distances in high resolution, but has poor accuracy at long distances and can only measure distances in areas with texture.

[0120] According to the imaging device 1h of the sixth modified example, a celestial sphere distance image obtained by a stereo ranging method having no blind spots and a celestial sphere distance image obtained by a TOF method having no blind spots are appropriately combined to obtain a better celestial sphere distance image in a complementary manner.

[0121] It should be noted that the above-described embodiment and each of the modified examples are preferred examples of the present invention, but the present invention is not limited to these, and various modifications can be made without departing from the gist of the present invention. [Explanation of symbols]

[0122] 1a, 1b, 1c, 1d, 1f, 1g, 1h Imaging device 10a, 10b, 10c, 10d, 10f, 10g, 10h, 105 housing 11a, 11b Light projecting section 12a, 12b Light receiving section 13a, 13b RGB sensor 14 Grip section 15 Image capture button 23 Distance control section 110a, 110b TOF projection lens 110aR, 110bR Light projection range 120a, 120b, 120c, 120d, 120e, 120f TOF receiving lens 120aR, 120bR light receiving range 130a, 130b RGB receiving lens 130aR, 130bR light receiving range 131a, 131b Area sensor 231 CPU 232 ROM 233 RAM 234 SSD 235 Light source driving circuit 236 Sensor I / F 237 Input / Output Interface 238 Light Emission Control Unit 238a Drive signal output section 239 Light receiving processing section 239a Phase signal input section 239b Distance image acquisition unit 239c Storage 239d Range image combination section 240 RGB sensor I / F 241 RGB image processing section 241a RGB image input section 241b RGB image storage section 241c RGB image combiner [Prior art documents] [Patent documents]

[0123] [Patent Document 1] JP 2018-152632 A

Claims

1. At least two light projecting units; At least two light receiving units; a housing in which the at least two light projecting units and the at least two light receiving units are disposed; Equipped with Each of the light projecting units includes a light projecting lens that diffuses and projects light, the two light-projecting lenses are arranged in different directions so that the light-projecting range of the two light-projecting units of the at least two light-projecting units is 360 degrees; The at least two light receiving units are disposed outside the light projection ranges of the at least two light projecting units. Imaging device.

2. At least two light projecting units; At least two light receiving units; a housing in which the at least two light projecting units and the at least two light receiving units are disposed; Equipped with Each of the light projecting units includes a light projecting lens that diffuses and projects light, the at least two light projection lenses are arranged in different directions so that the light projection range of the at least two light projection units is 360 degrees; the at least two light-projecting units include a light-projecting unit that is arranged in a different orientation from one of the at least two light-receiving units and has a light-projecting range that partially overlaps with a light-receiving range of the one light-receiving unit, the at least two light receiving units are disposed outside the light projection ranges of the at least two light projecting units, Two of the at least two light receiving units are disposed on two mutually orthogonal planes, respectively. Imaging device.

3. At least two imaging units for capturing visible light images are further provided, The at least two imaging units are disposed outside the light projection ranges of the at least two light projecting units. The imaging device according to claim 1 .

4. The at least two imaging units are arranged in different directions so that an imaging range of the at least two imaging units is 360 degrees. The imaging device according to claim 3 .

5. The imaging range of each of the at least two imaging units is 180 degrees or more.

5. The imaging device according to claim 3.

6. The at least two light receiving units are arranged in different directions so that the light receiving range of the at least two light receiving units is 360 degrees. The imaging device according to claim 1 .

7. Two of the at least two light receiving units are disposed on two planes perpendicular to each other, 7. The imaging device according to claim 1, 3 or 6.

8. The projection range of each of the at least two light projecting units is 180 degrees or more. The imaging device according to claim 1 .

9. a distance image acquisition unit that acquires a distance image based on signals output from the at least two light receiving units, The imaging device according to claim 1 .

10. An imaging device according to any one of claims 1 to 9; an acquisition unit that acquires captured images corresponding to the at least two light receiving units based on light receiving signals acquired from the at least two light receiving units; an image processing unit that generates a celestial sphere range image based on a plurality of captured images; An image processing device comprising:

11. a step of acquiring an image corresponding to each of the at least two light receiving units based on a light receiving signal acquired from each of the at least two light receiving units using an imaging device, the imaging device including at least two light projecting units each including a light projecting lens that diffuses and projects light, the two light projecting lenses being arranged in different directions so that the light projection range of the two light projecting units is 360 degrees, and the at least two light receiving units being arranged outside the light projection ranges of the at least two light projecting units; generating a distance image based on the captured image; An image processing method comprising:

12. a step of acquiring an image corresponding to each of the at least two light receiving sections based on a light receiving signal acquired from each of the at least two light receiving sections using an imaging device, the at least two light receiving sections including a light projecting lens that diffuses and projects light are arranged in different orientations from each other so that the light projection range of the at least two light projecting sections is 360 degrees, the at least two light projecting sections are arranged in a different orientation from one of the at least two light receiving sections and include a light projecting section having a light projection range that partially overlaps with the light receiving range of the one light receiving section, the at least two light receiving sections are arranged outside the light projection ranges of the at least two light projecting sections, and two of the at least two light receiving sections are respectively arranged on two planes perpendicular to each other; generating a distance image based on the captured image; An image processing method comprising:

Images