Information processing device and information processing method

Abstract

To provide an information processing device and information processing method that allows for easy confirmation of the presence of the photographer or tripod, etc., in the captured image, and that the desired 3D information has not been acquired. [Solution] The imaging device 1 comprises an imaging unit 11 that outputs three-dimensional information determined based on received light, and a display control unit 170 that outputs two-dimensional image information G captured by the imaging unit 11 separately from the three-dimensional information. The display control unit 170 outputs the two-dimensional image information G before determining the three-dimensional information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing method.

Background Art

[0002] Patent Document 1 describes a distance measuring device that can stably and accurately measure the distance to an object.

[0003] Patent Document 2 describes an imaging device that performs image processing to reduce the influence of reflections when reflections such as fingers are generated.

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to provide an information processing apparatus and an information processing method that can easily confirm that desired three-dimensional information has been acquired.

Means for Solving the Problems

[0005] The information processing apparatus according to the present invention includes an imaging unit that outputs three-dimensional information determined based on received light, and an output unit that outputs two-dimensional image information captured by the imaging unit separately from the three-dimensional information.

Effects of the Invention

[0006] According to the present invention, it is possible to provide an information processing apparatus and an information processing method that can easily confirm that desired three-dimensional information has been acquired.

Brief Description of the Drawings

[0007] [Figure 1] It is a diagram showing an example of the appearance of an imaging device according to an embodiment of the present invention. [Figure 2] It is a diagram for explaining the configuration of the imaging device in the same embodiment. [Figure 3]This diagram illustrates the usage of the imaging device in the same embodiment. [Figure 4] This figure shows an example of the configuration of the processing block in the processing circuit in the same embodiment. [Figure 5] This flowchart shows an example of the operation of the processing circuit of the imaging device in the same embodiment. [Figure 6] This is a flowchart for generating 360-degree image data in the same embodiment. [Figure 7] This is a flowchart for determining nearby objects in the same embodiment. [Figure 8] This is a diagram illustrating the display content of the display unit in the same embodiment. [Figure 9] This figure shows the external appearance of an imaging device according to a modified embodiment of the same. [Figure 10] This figure shows the configuration of the processing block in the processing circuit in a modified example. [Figure 11] This figure shows the external appearance of an imaging device according to a second modified embodiment of the present invention. [Figure 12] This figure shows the configuration of the processing block in the processing circuit in the second modified example. [Figure 13] This is a flowchart for determining proximity to objects in the second modified example. [Figure 14] This figure illustrates the configuration of an imaging device according to a third modified embodiment of the present invention. [Modes for carrying out the invention]

[0008] Embodiments of the information processing device and information processing method will be described in detail below with reference to the attached drawings.

[0009] Figure 1 shows an example of the external appearance of an imaging device according to an embodiment of the present invention. Figure 2 is a diagram illustrating the configuration of the imaging device. Figure 2 shows the internal configuration of the imaging device shown in Figure 1.

[0010] The imaging device 1 is an example of an information processing device that outputs three-dimensional information determined based on received light. It comprises an imaging unit (camera) 11, a projection unit (corresponding to the light-emitting part of a distance sensor) 12 that projects light other than visible light, and a distance information acquisition unit (corresponding to the light-receiving part of a distance sensor) 13 that acquires distance information based on the light projected by the projection unit 12, all integrated into a housing 10. Each unit is electrically connected to a processing circuit 14 inside the housing 10 by a synchronization signal line L, and they operate in synchronization with each other.

[0011] The shooting switch 15 is for the user to input a shooting instruction signal to the processing circuit 14. The display unit 20 displays content corresponding to the output signal of the processing circuit 14 and is composed of an LCD screen or the like. The display unit 20 may be composed of a touch panel or the like and may accept user operation input. Based on the shooting instruction, the processing circuit 14 controls each part to acquire RGB image and distance information data, and performs processing to reconstruct the acquired distance information data into high-density 3D point cloud data based on the RGB image and distance information data. It is possible to construct 3D point cloud data even if the distance information data is used as is, but in that case, the accuracy of the 3D point cloud data is limited by the number of pixels (resolution) of the distance information acquisition unit 13. In this example, the processing when reconstructing it into high-density 3D point cloud data is also shown. The reconstructed data is output to an external PC or the like via a portable recording medium or communication and used to display a 3D reconstruction model.

[0012] Each component and processing circuit 14 is powered by a battery housed inside the enclosure 10. Alternatively, power may be supplied from outside the enclosure 10 via a connecting cord.

[0013] The imaging unit 11 includes imaging elements 11a, 11A, and fisheye lenses (wide-angle lenses) 11b, 11B, etc. The projection unit 12 includes light source units 12a, 12A, and wide-angle lenses 12b, 12B, etc. The distance information acquisition unit 13 includes TOF (Time Of Flight) sensors 13a, 13A, and wide-angle lenses 13b, 13B, etc. Although the illustration of each unit is omitted, it may constitute an optical system such as a prism or a lens group. For example, an optical system for forming an image of the light collected by the fisheye lenses 11b, 11B on the imaging elements 11a, 11A may be configured in the imaging unit 11. Also, an optical system for guiding the light of the light source units 12a, 12A to the wide-angle lenses 12b, 12B may be configured in the projection unit 12. Further, an optical system for forming an image of the light collected by the wide-angle lenses 13b, 13B on the TOF sensors 13a, 13A may be configured in the distance information acquisition unit 13. Each optical system may be appropriately determined according to the configuration and arrangement of the imaging elements 11a, 11A, the light source units 12a, 12A, the TOF sensors 13a, 13A, etc. Here, the description of the optical system such as a prism or a lens group is omitted.

[0014] The imaging elements 11a, 11A, the light source units 12a, 12A, and the TOF sensors 13a, 13A are integrally housed inside the housing 10. The fisheye lens 11b, the wide-angle lens 12b, the wide-angle lens 13b, and the display unit 20 are provided on the first surface on the front side of the housing 10, respectively. On the first surface, the inner ranges of the fisheye lens 11b, the wide-angle lens 12b, and the wide-angle lens 13b are open.

[0015] The fisheye lens 11B, the wide-angle lens 12B, the wide-angle lens 13B, and the shooting switch 15 are provided on the second surface on the back side of the housing 10, respectively. On the second surface, the inner ranges of the fisheye lens 11B, the wide-angle lens 12B, and the wide-angle lens 13B are open.

[0016] The imaging elements 11a and 11A are two-dimensional resolution image sensors (area sensors). The imaging elements 11a and 11A have an imaging area in which a large number of light receiving elements (photodiodes) of each pixel are arranged in two-dimensional directions. A color filter of R (Red), G (Green), and B (Blue) such as a Bayer array is provided in the imaging area to receive visible light, and the light passing through the color filter is stored in the photodiodes. Here, an image sensor with a large number of pixels is used so that a two-dimensional image with a wide angle (for example, a range of a 180-degree half sphere with the imaging direction shown in FIG. 2 as the front) can be acquired with high resolution. The imaging elements 11a and 11A convert the light imaged in their imaging areas into electrical signals by pixel circuits of each pixel and output a high-resolution RGB image. The fisheye lenses 11b and 11B collect light from a wide angle (for example, a range of a hemisphere of 180 degrees with the imaging direction shown in FIG. 2 as the front) and image the light on the imaging areas of the imaging elements 11a and 11A.

[0017] The light source units 12a and 12A are semiconductor lasers that emit laser light in a wavelength band other than the visible light region used for distance measurement (here, infrared is used as an example). One semiconductor laser may be used for the light source units 12a and 12A, or a combination of a plurality of semiconductor lasers may be used. Also, a surface-emitting semiconductor laser such as VCSEL (Vertical Cavity Surface Emitting LASER) may be used as the semiconductor laser. Further, the light of the semiconductor laser may be shaped by an optical lens to be vertically long, and the vertically long light may be scanned in the one-dimensional direction of the measurement range by a light deflection element such as a MEMS (Micro Electro Mechanical Systems) mirror. In the present embodiment, as the light source units 12a and 12A, a form is shown in which the light of the semiconductor laser LA is spread over a wide angle range through wide angle lenses 12b and 12B without using a light deflection element such as a MEMS mirror.

[0018] The wide-angle lenses 12b and 12B of the light source units 12a and 12A have the function of spreading the light emitted by the light source units 12a and 12A over a wide angle range (for example, a hemispherical area of ​​180 degrees around the imaging direction shown in Figure 2).

[0019] The wide-angle lenses 13b and 13B of the distance information acquisition unit 13 capture reflected light from the light sources 12a and 12A projected by the projection unit 12 from various directions within the wide-angle measurement range (for example, a 180-degree hemispherical area with the imaging direction shown in Figure 2 as the front), and image this light onto the light-receiving areas of the TOF sensors 13a and 13A. The measurement range includes one or more objects to be projected (for example, buildings), and the light reflected by these objects (reflected light) enters the wide-angle lenses 13b and 13B. The reflected light may be captured by, for example, providing a filter that cuts out light with wavelengths greater than the infrared region across the entire surface of the wide-angle lenses 13b and 13B. However, this is not the only method; as long as infrared light enters the light-receiving area, a means for passing infrared wavelength light, such as a filter, may be provided in the optical path from the wide-angle lenses 13b and 13B to the light-receiving area.

[0020] TOF sensors 13a and 13A are two-dimensional resolution optical sensors. TOF sensors 13a and 13A have a light-receiving area in which a large number of light-receiving elements (photodiodes) are arranged in a two-dimensional direction. In this sense, they can be called a "second imaging and light-receiving means." TOF sensors 13a and 13A receive reflected light from each area (each area is also called a position) within the measurement range using the light-receiving element corresponding to each area, and measure (calculate) the distance to each area based on the light detected by each light-receiving element.

[0021] In this embodiment, distance is measured using a phase difference detection method. In the phase difference detection method, a laser beam amplitude-modulated at the fundamental frequency is shone onto the measurement range, the reflected light is received, and the phase difference between the shone light and the reflected light is measured to determine the time, which is then multiplied by the speed of light to calculate the distance. A strength of this method is that a certain degree of resolution can be expected.

[0022] TOF sensors 13a and 13A are driven in synchronization with the irradiation of light by the projection unit 12, and each light-receiving element (corresponding to a pixel) calculates the distance corresponding to each pixel from the phase difference with the reflected light, and outputs distance information image data (hereinafter also referred to as "distance image" or "TOF image") which associates information indicating the distance to each area within the measurement range with the pixel information. TOF sensors 13a and 13A may also output phase information image data which associates phase information with the pixel information, and distance information image data may be acquired based on the phase information image data in post-processing.

[0023] The number of areas that can be divided into the measurement range is determined by the resolution of the light-receiving area. Therefore, if a lower resolution is used for miniaturization, the number of pixels in the distance image data decreases, and consequently, the number of 3D point clouds also decreases.

[0024] Alternatively, distance can be measured using a pulse method instead of the phase difference detection method. In this case, for example, the light sources 12a and 12A emit an ultrashort pulse P1 with a rise time of a few nanoseconds (ns) and a high peak optical power, and the TOF sensors 13a and 13A measure the time (t) it takes for the reflected pulse P2, which is the reflected light of the irradiation pulse P1 emitted by the light sources 12a and 12A, to be received. When this method is adopted, for example, the TOF sensors 13a and 13A are equipped with a circuit for measuring time on the output side of the light-receiving element. In each circuit, for each light-receiving element, the time taken from when the light sources 12a and 12A emit the irradiation pulse P1 until the reflected pulse P2 is received is converted into distance, and the distance to each area is obtained.

[0025] This method is suitable for widening the imaging device 1 because it can output powerful light using peak light. Furthermore, if the configuration uses MEMS mirrors or the like to deflect (scan) the light, powerful light can be emitted over long distances while suppressing its spread, leading to an increase in the measurement distance. In this case, the laser light emitted from the light sources 12a and 12A is scanned (blinded) by the MEMS mirrors toward the wide-angle lenses 12b and 12B.

[0026] It is desirable that the effective field of view of the imaging unit 11 and the effective field of view of the distance information acquisition unit 13 coincide, for example, at 180 degrees or more, but they do not necessarily have to coincide. The effective field of view of the imaging unit 11 and the effective field of view of the distance information acquisition unit 13 may be reduced as needed. In this embodiment, the imaging unit 11 and the distance information acquisition unit 13 have their effective pixels reduced to a range such as 100 degrees to 180 degrees so that there is nothing interfering with the field of view. In addition, the resolution of the TOF sensors 13a and 13A may be set lower than the resolution of the image sensors 11a and 11A in order to prioritize miniaturization of the imaging device 1. By making the TOF sensors 13a and 13A have a lower resolution than the image sensors 11a and 11A, the size of the light-receiving area can be suppressed, which leads to miniaturization of the imaging device 1. Therefore, the TOF sensors 13a and 13A have low resolution, and the 3D point clouds obtained by the TOF sensors 13a and 13A are low density. However, because a processing circuit 14, which is the "acquisition means," is provided, it can be converted into a high-density 3D point cloud. The process of converting to a high-density 3D point cloud in the processing circuit 14 will be described later.

[0027] In this embodiment, as an example, the image sensor 11a, the light source unit 12a, and the TOF sensor 13a are arranged in a straight line along the longitudinal direction of the housing 10. Similarly, the image sensor 11A, the light source unit 12A, and the TOF sensor 13A are arranged in a straight line along the longitudinal direction of the housing 10. The following explanation will use the image sensor 11a, the light source unit 12a, and the TOF sensor 13a as examples.

[0028] The imaging area (imaging surface) of the image sensor 11a and the light-receiving area (light-receiving surface) of the TOF sensor 13a may be arranged in a direction perpendicular to the longitudinal direction, as shown in Figure 2, or they may be arranged in the longitudinal direction by providing a prism or the like that which changes the direction of light propagation (optical path) by 90 degrees before incident light. In addition, they may be arranged in any direction depending on the configuration. In other words, the image sensor 11a, the light source unit 12a, and the TOF sensor 13a are arranged so that they target the same measurement range. The imaging unit 11, the projection unit 12, and the distance information acquisition unit 13 are arranged from one side of the housing 10 toward its measurement range. At this time, it is sufficient if the image sensor 11a and the TOF sensor 13a are arranged on the same baseline so as to create parallel stereo. By arranging them to create parallel stereo, even if there is only one image sensor 11a, it is possible to obtain parallax data using the output of the TOF sensor 13a. The light source unit 12a is configured to irradiate the measurement range of the TOF sensor 13a with light.

[0029] (Processing circuit) Next, the processing of processing circuit 14 will be explained. TOF images obtained using only TOF sensors 13a and 13A have low resolution. Therefore, in this example, we will show an example of reconstructing high-resolution, high-density 3D point cloud data using processing circuit 14. Note that some or all of the following processing as "information processing means" in processing circuit 14 may be performed by an external device.

[0030] As mentioned above, the 3D point cloud data reconstructed by the imaging device 1 is output to an external device such as a PC via a portable recording medium or communication, and is used to display a 3D reconstruction model.

[0031] This makes it possible to provide an imaging device 1 that is faster, smaller, and lighter, resulting in greater portability, compared to a case where the imaging device 1 itself displays the 3D reconstruction model.

[0032] However, after leaving the site where the 3D information was acquired and restoring it using external equipment, it may become apparent that the photographer themselves or the tripod are visible in the captured image, or that the desired layout of the 3D information has not been acquired. In such cases, it becomes necessary to revisit the site where the 3D information was acquired.

[0033] One possible solution to this problem is to bring a 3D reconstruction device to the site, but doing so would eliminate the advantages of speed, miniaturization, and weight reduction.

[0034] Alternatively, it is conceivable to transmit the acquired 3D information to an external device via a communication line and receive the reconstructed 3D information. However, this would negate the benefits of high speed, and since 3D information contains a large amount of data, it would be difficult to visually confirm the presence of the photographer or tripod, etc., in the captured image.

[0035] In particular, with full-spherical 3D information, it is extremely difficult to visually confirm whether the photographer themselves or the tripod, etc., are reflected in the captured image.

[0036] In view of the above problems, this embodiment aims to provide an imaging device 1 that allows for easy, in-real-time confirmation of whether the photographer themselves, a tripod, or other objects are visible in the captured image, or whether three-dimensional information of the desired layout has not been acquired.

[0037] Figure 3 is a diagram illustrating the usage of the imaging device in the same embodiment.

[0038] In the state shown in Figure 3(a), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the 360-degree imaging range R, and therefore neither the photographer M nor the selfie stick 1A appear in the 360-degree image.

[0039] In the situation shown in Figure 3(b), the photographer M is included in the 360-degree imaging range R, and therefore photographer M is captured in the 360-degree image.

[0040] In the state shown in Figure 3(c), the tripod 1B supporting the imaging device 1 is included in the 360-degree imaging range R, and the tripod 1B is captured in the 360-degree image.

[0041] In the state shown in Figure 3(d), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the 360-degree imaging range R, and therefore the photographer M and the selfie stick 1A do not appear in the 360-degree image. However, due to the strong ambient light, there is a possibility of misjudging their presence.

[0042] Furthermore, in the conditions shown in Figures 3(b) and 3(c), the color, type, and appearance of the reflected objects varied, making it difficult to uniformly determine whether or not reflections were present.

[0043] In the situations described above, when determining the presence or absence of specific objects (nearby objects) such as the photographer or a tripod based on distance information image data output from TOF sensors 13a and 13A, it was difficult to distinguish whether a specific object was actually present or if the ambient light was simply too strong.

[0044] In other words, when the charge stored in a specific pixel of the TOF sensors 13a and 13A was saturated, it was difficult to distinguish from the output of the TOF sensors 13a and 13A whether this was due to the presence of a specific object or excessive ambient light intensity.

[0045] In view of the above problems, this embodiment also aims to provide an imaging device 1 that can accurately confirm whether or not the photographer themselves or specific objects such as a tripod are reflected in the captured image, distinct from the influence of ambient light.

[0046] Figure 4 shows an example of the configuration of the processing blocks of the processing circuit 14. The processing circuit 14 shown in Figure 4 includes a control unit 141, an RGB image data acquisition unit 142, a monochrome processing unit 143, a TOF image data acquisition unit 144, a high-resolution processing unit 145, a matching processing unit 146, a reprojection processing unit 147, a semantic segmentation unit 148, a disparity calculation unit 149, a 3D reconstruction processing unit 150, a judgment unit 160, a display control unit 170 which is an example of an output unit, and a transmitting / receiving unit 180 which is an example of an output unit. In Figure 4, solid arrows indicate the flow of signals, and dashed arrows indicate the flow of data.

[0047] When the control unit 141 receives an ON signal (shooting start signal) from the shooting switch 15, it outputs synchronization signals to the image sensors 11a, 11A, the light source units 12a, 12A, and the TOF sensors 13a, 13A, thereby controlling the entire processing circuit 14. First, the control unit 141 outputs a signal to the light source units 12a, 12A instructing them to emit ultrashort pulses, and at the same time, it outputs a signal to the TOF sensors 13a, 13A instructing them to generate TOF image data. Furthermore, the control unit 141 outputs a signal to the image sensors 11a, 11A instructing them to take images. Note that the imaging by the image sensors 11a, 11A may occur during the period when light is being emitted from the light source units 12a, 12A, or in the most recent period before or after that.

[0048] The RGB image data acquisition unit 142 acquires RGB image data captured by the image sensors 11a and 11A based on the imaging instruction from the control unit 141, and outputs 360-degree RGB image data. The monochrome processing unit 143 performs processing to align the data types for matching with TOF image data obtained from the TOF sensors 13a and 13A. In this example, the monochrome processing unit 143 performs processing to convert the 360-degree RGB image data into a 360-degree monochrome image.

[0049] The TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on the TOF image data generation instruction from the control unit 141, and outputs the 360-degree TOF image data.

[0050] The high-resolution enhancement unit 145 treats the 360-degree TOF image data as a monochrome image and enhances its resolution. Specifically, the high-resolution enhancement unit 145 replaces the distance values ​​associated with each pixel of the 360-degree TOF image data with the values ​​(grayscale values) of the 360-degree monochrome image. Furthermore, the high-resolution enhancement unit 145 enhances the resolution of the 360-degree monochrome image to the resolution of the 360-degree RGB image data obtained from the image sensors 11a and 11A. The conversion to high resolution is performed, for example, by applying a normal upconversion process. As for other conversion methods, for example, multiple frames of continuously generated 360-degree TOF image data may be acquired, and these may be used to add the distances of adjacent points and perform super-resolution processing.

[0051] The matching processing unit 146 extracts feature quantities from parts of the texture of the spherical monochrome image obtained by refining the spherical TOF image data and the spherical monochrome image obtained by refining the spherical RGB image data, and performs matching processing using the extracted feature quantities. For example, the matching processing unit 146 extracts edges from each monochrome image and performs matching processing using the extracted edge information. Alternatively, matching processing may be performed using a method that features texture changes, such as SIFT. Here, matching processing means searching for corresponding pixels.

[0052] One specific method for matching is block matching. Block matching calculates the similarity between pixel values ​​extracted as M×M (where M is a positive integer) pixel blocks around the reference pixel and pixel values ​​extracted as M×M pixel blocks around the central pixel of the other image, and identifies the central pixel with the highest similarity as the corresponding pixel.

[0053] There are various methods for calculating similarity. For example, one can use the formula for the Normalized Correlation Coefficient (CNCC). A higher CNCC value indicates greater similarity, and a value of 1 indicates that the pixel values ​​of the blocks are perfectly identical.

[0054] Furthermore, since distance data for textureless regions can also be obtained from the 360-degree TOF image data, weights may be applied to the matching process depending on the region. For example, in the calculation of the formula for CNCC, a calculation may be performed that adds weight to areas other than edges (textureless regions).

[0055] Alternatively, instead of using an equation to show NCC, you may use a formula such as Selective Correlation Coefficient (SCC).

[0056] The reprojection processing unit 147 performs a process to reproject the 360-degree TOF image data, which shows the distance to each position in the measurement range, onto the 2D coordinate system (screen coordinate system) of the imaging unit 11. Reprojection means determining the coordinates in the images of the image sensors 11a and 11A where the 3D points calculated by the TOF sensors 13a and 13A are located. The 360-degree TOF image data shows the position of the 3D points in a coordinate system centered on the distance information acquisition unit 13 (mainly the wide-angle lenses 13b and 13B). Therefore, the 3D points shown in the 360-degree TOF image data are reprojected onto a coordinate system centered on the imaging unit 11 (mainly the fisheye lenses 11b and 11B). For example, the reprojection processing unit 147 translates the coordinates of the 3D points in the 360-degree TOF image data to the coordinates of the 3D points centered on the imaging unit 11, and then converts them to the 2D coordinate system (screen coordinate system) shown in the 360-degree RGB image data.

[0057] The parallax calculation unit 149 calculates the parallax at each position from the distance difference with the corresponding pixel obtained through the matching process.

[0058] Furthermore, the parallax matching process utilizes the reprojected coordinates converted by the reprojection processing unit 147 to search for surrounding pixels at the position of the reprojected coordinates, thereby shortening processing time and enabling the acquisition of more detailed and high-resolution distance information.

[0059] Furthermore, the segmentation data obtained by the semantic segmentation processing of the semantic segmentation unit 148 may be used for the parallax matching process. In that case, it becomes possible to obtain even more detailed and high-resolution distance information.

[0060] Alternatively, disparity matching may be performed only on edges or areas with strong features, while propagation processing may be performed on other parts using the full-spherical TOF image data, for example, by utilizing the full-spherical RGB image features or probabilistic methods.

[0061] The semantic segmentation unit 148 uses deep learning to assign segmentation labels indicating objects to the input image within the measurement range. This allows each pixel of the 360-degree TOF image data to be constrained to one of multiple distance regions divided by distance, further increasing the reliability of the calculation.

[0062] The 3D reconstruction processing unit 145 acquires RGB image data of the entire sphere from the RGB image data acquisition unit 142, reconstructs 3D data of the entire sphere based on the distance information output by the disparity calculation unit 149, and outputs a high-density 3D point cloud of the entire sphere with color information added to each 3D point. The 3D reconstruction processing unit 150 is an example of a 3D information determination unit that determines 3D information.

[0063] The determination unit 160 acquires 360-degree RGB image data from the RGB image data acquisition unit 142, and also acquires 360-degree TOF image data converted from the 360-degree RGB image data to a 2D coordinate system, from the reprojection processing unit 147. Based on this data, it determines whether or not a specific object is reflected in the captured image and outputs the determination result to the display control unit 170.

[0064] The display control unit 170 acquires 360-degree RGB image data from the RGB image data acquisition unit 142 and displays 2D image information based on the acquired 360-degree RGB image data on the display unit 20. The display control unit 170 also superimposes information indicating the judgment result acquired from the judgment unit 160 onto the 2D image information and displays it on the display unit 20.

[0065] The display control unit 170 is an example of an output unit that outputs 2D image information captured by the imaging unit 11 separately from 3D information, and the display unit 20 is an example of an output destination that outputs 2D image information.

[0066] The display control unit 170 may acquire 3D data of the entire sphere from the 3D reconstruction processing unit 145 and display the 3D information on the display unit 20. Specifically, the display control unit 170 may select whether to display 2D image information or 3D information on the display unit 20 according to predetermined conditions. This allows the display control unit 170 to output 2D image information separately from 3D information.

[0067] The transmitting / receiving unit 180 communicates with an external device using wired or wireless technology, and transmits (outputs) the 3D data of the entire sphere output from the 3D reconstruction processing unit 145 and the 2D image information of the entire sphere output from the RGB image data acquisition unit 142 to an external device 300 that performs 3D reconstruction processing via the network 400.

[0068] The transmitting / receiving unit 180 is an example of an output unit that outputs 3D information, and the external device 300 is an example of an output destination that outputs 3D information.

[0069] The transmitting / receiving unit 180 may transmit only 3D data of the 360-degree sphere, without transmitting 2D image information of the 360-degree sphere. Furthermore, the transmitting / receiving unit 180 may be configured with an interface circuit for a portable storage medium such as an SD card or a personal computer.

[0070] (Operation of the processing circuit) Figure 5 is a flowchart showing an example of the operation of the processing circuit 14 of the imaging device 1. When the user turns on the shooting switch 15 and a shooting instruction signal is input, the control unit 141 of the processing circuit 14 performs the operation to generate a high-density 3D point cloud in the following manner (an example of an imaging processing method and an information processing method).

[0071] First, the control unit 141 drives the light source units 12a and 12A, the TOF sensors 13a and 13A, and the image sensors 11a and 11A to capture an image of the measurement range (step S1). Driven by the control unit 141, the light source units 12a and 12A emit infrared light (an example of a projection step), and the TOF sensors 13a and 13A receive the reflected light (an example of a light receiving step). In addition, the image sensors 11a and 11A capture an image of the measurement range at the timing of the start of the drive of the light source units 12a and 12A or in the period immediately preceding it (an example of an imaging step).

[0072] Next, the RGB image data acquisition unit 142 acquires RGB image data within the measurement range from the image sensors 11a and 11A (step S2). Then, the display control unit 170 acquires 360-degree RGB image data from the RGB image data acquisition unit 142 and displays 2D image information based on the acquired 360-degree RGB image data on the display unit 20 (an example of a 2D image information output step) (step S3).

[0073] The display control unit 170 displays 2D image information of a portion of the acquired 360-degree RGB image data on the display unit 20, and changes the area of ​​the 2D image information displayed on the display unit 20 based on various user inputs. Various user inputs can be implemented by providing operation switches other than the shooting switch 15, or by configuring the display unit 20 as an input unit such as a touch panel.

[0074] At this stage, the photographer can use the two-dimensional image information displayed on the display unit 20 to confirm that they themselves, the tripod, or other objects are not visible in the captured image, and that no two-dimensional image information with the desired layout has been acquired.

[0075] Next, the TOF image data acquisition unit 144 acquires TOF image data from the TOF sensors 13a and 13A that shows the distance to each position in the two-dimensional region (step S4).

[0076] Next, the monochrome processing unit 143 converts the RGB image data to a monochrome image (step S5). TOF image data and RGB image data have different data types, distance data and RGB data, respectively, so matching cannot be performed directly. Therefore, each data is first converted to a monochrome image. For TOF image data, the high-resolution unit 145 converts it by replacing the values ​​indicating the distance of each pixel with the values ​​of the monochrome image before increasing the resolution.

[0077] Next, the resolution enhancement unit 145 enhances the resolution of the TOF image data (step S6).

[0078] Next, the matching processing unit 146 extracts feature quantities from the textured parts of each monochrome image and performs matching processing using the extracted feature quantities (step S7).

[0079] Next, the parallax calculation unit 149 calculates the distance by calculating the parallax of each position from the distance difference of the corresponding pixels (step S8).

[0080] Next, the determination unit 160 acquires 360-degree RGB image data from the RGB image data acquisition unit 142 and also acquires 360-degree TOF image data converted to a two-dimensional coordinate system indicated by the RGB image data from the reprojection processing unit 147. Based on this data, it determines whether or not a nearby object, as a specific object, is reflected in the captured image and outputs the determination result to the display control unit 170. The display control unit 170 superimposes the information indicating the determination result acquired from the determination unit 160 onto the two-dimensional image information and displays it on the display unit 20 (an example of a display step) (step S9).

[0081] Then, the 3D reconstruction processing unit 145 acquires RGB image data from the RGB image data acquisition unit 142, reconstructs the 3D data based on the distance information output by the disparity calculation unit 149, and outputs a high-density 3D point cloud with color information added to each 3D point (step S10).

[0082] Next, the transmitting / receiving unit 180 transmits the 3D data output from the 3D reconstruction processing unit 145 and the 2D image information output from the RGB image data acquisition unit 142 to an external device 300 that performs 3D reconstruction processing via the network 400 (an example of a 3D information output step) (step S11).

[0083] The transmitting / receiving unit 180 may transmit the 3D data output from the 3D reconstruction processing unit 145 without transmitting the 2D image information output from the RGB image data acquisition unit 142.

[0084] As described above, the imaging device 1 comprises an imaging unit 11 and a display control unit 170 that outputs two-dimensional image information captured by the imaging unit 11 separately from the three-dimensional information.

[0085] This makes it easy to check from 2D image data whether the photographer or tripod is visible in the captured image, or whether the desired 3D layout information has not been acquired, without having to check the 3D information.

[0086] Therefore, it becomes possible to reacquire 3D information while remaining at the site where the 3D information was acquired. This reduces the effort required to revisit the site to acquire the 3D information compared to realizing after leaving the site that the photographer or tripod, etc., are included in the captured image, or that 3D information of the desired layout has not been acquired.

[0087] The 3D information includes 3D information of the entire sphere. In this case, even with 3D information of the entire sphere, where it is difficult to confirm whether the photographer themselves or the tripod are reflected in the captured image, or whether 3D information of the desired layout has not been acquired, it becomes easy to confirm from the 2D image information captured by the imaging unit 11 that the photographer themselves or the tripod are reflected in the captured image, or whether 3D information of the desired layout has not been acquired.

[0088] The display control unit 170 outputs 2D image information G in step S3 before the transmitting / receiving unit 180 transmits (outputs) 3D information in step S11. The display control unit 170 outputs 2D image information G in step S3 before the 3D reconstruction processing unit 150 determines the 3D information in step S10.

[0089] This makes it possible to check from the 2D image information whether the photographer themselves or the tripod are visible in the captured image, or whether the desired 3D layout information has not been acquired, before checking the 3D information.

[0090] The display control unit 170 causes the display unit 20 to display two-dimensional image information. The imaging device 1 includes the display unit 20.

[0091] This makes it easy to check from the 2D image information displayed on the display unit 20 whether the photographer themselves or the tripod are visible in the captured image, or whether the desired 3D layout information has not been acquired.

[0092] The display control unit 170 outputs two-dimensional image information to a display unit 20 that is different from the external device 300 to which the transmitting / receiving unit 180 outputs three-dimensional information.

[0093] This makes it possible to confirm, from the 2D image information output to the display unit 20 (which is different from the external device 300), whether the photographer themselves or the tripod are visible in the captured image, or whether the desired 3D layout information has not been acquired, without having to check the 3D information output to the external device 300.

[0094] The imaging device 1 includes a 3D reconstruction processing unit 150 that determines 3D information based on the output of the distance information acquisition unit 13. The 3D reconstruction processing unit 150 determines 3D information based on the output of the distance information acquisition unit 13 and 2D image information.

[0095] This makes it possible to confirm from the 2D image information captured by the imaging unit 11 that the photographer themselves, the tripod, or other objects are not visible in the captured image, and that the desired 3D layout information has not been acquired, without having to check the 3D information determined by the 3D reconstruction processing unit 150.

[0096] Figure 6 is a flowchart illustrating the generation of 360-degree image data in the same embodiment.

[0097] Figure 6(a) is a flowchart showing the process for generating 360-degree RGB image data corresponding to step S2 described in Figure 5.

[0098] The RGB image data acquisition unit 142 receives two RGB image data in fisheye image format (step S201).

[0099] The RGB image data acquisition unit 142 converts each RGB image data into an equirectangular image format (step S202). The RGB image data acquisition unit 142 converts two RGB image data into an equirectangular image format based on the same coordinate system, thereby facilitating image stitching in the next step.

[0100] Here, we will explain the equirectangular image format. The equirectangular image format is a method that can represent a full-sphere image, and it is the format of an image (equisthenticular image) created using the equirectangular projection. The equirectangular projection is a projection that represents three-dimensional directions with two variables, such as the latitude and longitude of a globe, and displays them in a plane so that the latitude and longitude are orthogonal. Therefore, an equirectangular image is an image generated using the equirectangular projection and is represented by coordinates with two angular variables of the spherical coordinate system as the two axes.

[0101] The RGB image data acquisition unit 142 combines the two RGB image data generated in step S202 to generate a single 360-degree RGB image data (step S203). The two input RGB image data cover an area with a field of view exceeding 180 degrees. Therefore, the 360-degree RGB image data generated by appropriately connecting these two RGB image data can cover the entire 360-degree area.

[0102] The merging process in step S203 can utilize existing techniques for joining multiple images, and the method is not particularly limited.

[0103] Figure 6(b) shows the raw 360° TOF image data corresponding to step S4 described in Figure 5. This is a flowchart showing the processing steps.

[0104] The TOF image data acquisition unit 144 acquires two distance image data in fisheye image format (step S401).

[0105] The TOF image data acquisition unit 144 converts two TOF image data in fisheye image format into equirectangular image format (step S402). As mentioned above, the equirectangular image format is a format that can represent a full-sphere image. In step S402, converting the two TOF image data into equirectangular image format based on the same coordinate system facilitates image stitching in the next step S403.

[0106] The TOF image data acquisition unit 144 combines the two TOF image data generated in step S402 to generate a single 360-degree TOF image data (step S403). The two input TOF image data cover an area with a field of view exceeding 180 degrees. Therefore, the 360-degree TOF image data generated by appropriately connecting these two TOF image data can cover the entire 360-degree area.

[0107] The merging process in step S403 can utilize existing techniques for joining multiple images, and the method is not particularly limited.

[0108] Figure 7 is a flowchart illustrating the proximity object detection process in the same embodiment.

[0109] Figure 7 is a flowchart showing the process for determining whether or not a nearby object is captured in the image, corresponding to step S9 described in Figure 5.

[0110] The determination unit 160 determines, based on the spherical TOF image data acquired from the reprojection processing unit 147, whether there are any pixels in the spherical TOF image data whose stored energy has reached saturation (step S801).

[0111] If the determination unit 160 finds that there is a pixel whose charge has saturated in step S801, it determines whether the charge has saturated for a pixel at the same coordinate as the pixel whose charge was saturated in step S801, based on the 360-degree RGB image data acquired from the RGB image data acquisition unit 142 (step S802).

[0112] If the charge level is saturated in step S802, the determination unit 160 determines that the pixels whose charge level is saturated in step S801 are due to ambient light, and outputs error information to the display control unit 170. Based on the error information obtained from the determination unit 160, the display control unit 170 superimposes the error information onto the two-dimensional image information and displays it on the display unit 20 (step S803).

[0113] If the charge level is not saturated in step S802, the determination unit 160 determines that the pixels whose charge level is saturated in step S801 are due to the presence of nearby objects, and outputs the coordinate position information of the pixels whose charge level was saturated in step S801 to the display control unit 170. Based on the coordinate position information of the pixels obtained from the determination unit 160, the display control unit 170 superimposes identification information for identifying nearby objects onto the 2D image information and displays it on the display unit 20 (step S804).

[0114] If no pixels have reached saturation in step S801, the determination unit 160 determines, based on the spherical TOF image data acquired from the reprojection processing unit 147, whether there are any pixels in the spherical TOF image data that indicate a distance of 0.5m or less (step S805).

[0115] If the determination unit 160 finds no pixels indicating distance information of 0.5m or less in step S805, it terminates the process.

[0116] If the determination unit 160 finds a pixel showing distance information of 0.5m or less in step S805, it proceeds to step S804, where it determines that the pixel showing distance information of 0.5m or less in step S805 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel showing distance information of 0.5m or less in step S805 to the display control unit 170. Based on the coordinate position information of the pixel obtained from the determination unit 160, the display control unit 170 overlays identification information for identifying nearby objects onto the 2D image information and displays it on the display unit 20.

[0117] As explained above, the display control unit 170 superimposes the identification information onto the two-dimensional image information if it determines that a nearby object exists, and does not superimpose the identification information onto the two-dimensional image information if it does not determine that a nearby object exists.

[0118] In other words, the display control unit 170 causes the display unit 20 to display different information depending on whether or not there is an object nearby.

[0119] Furthermore, the display control unit 170 overlays identification information for identifying nearby objects onto the two-dimensional image information based on the pixel coordinate position information obtained from the determination unit 160 and displays it on the display unit 20.

[0120] In other words, the display control unit 170 causes the display unit 20 to display different positions depending on the position of nearby objects.

[0121] Figure 8 is a diagram illustrating the display content of the display unit in the same embodiment.

[0122] Figure 8 is an explanatory diagram corresponding to step S2 shown in Figure 5, and steps S803 and S804 shown in Figure 7.

[0123] The display unit 20 displays two-dimensional image information G, which is controlled by the display control unit 170. In addition, the display unit 20 displays identification information G1 and G2 for identifying nearby objects and error information G3, which are superimposed on the two-dimensional image information G, which are also controlled by the display control unit 170.

[0124] As described above, the imaging device 1 comprises an imaging unit 11 that images an object, a projection unit 12 that projects light onto the object, a distance information acquisition unit 13 that receives light reflected from the object, and a display control unit 170 that causes the display unit 20 to display different information depending on the presence or absence of nearby objects, which is determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

[0125] This allows the photographer to accurately check whether they or nearby objects such as tripods are reflected in the captured image, distinguishing them from the effects of ambient light.

[0126] The imaging device 1 includes a display unit 20. This allows the photographer to reliably confirm whether or not nearby objects are included in the captured image.

[0127] The display control unit 170 causes the display unit 20 to display different positions depending on the position of the nearby object. This allows the photographer to confirm the position of the nearby object in the captured image.

[0128] The display control unit 170 displays the image information G captured by the imaging unit 11 on the display unit 20, and also displays identification information G1 and G2 for identifying nearby objects superimposed on the image information on the display unit 20. This allows the photographer to reliably confirm the position of nearby objects appearing in the captured image.

[0129] The imaging device 1 includes a determination unit 160 that determines the presence of a nearby object when the amount of charge stored by the distance information acquisition unit 13 due to light received is saturated, and the amount of charge stored in the pixels of the imaging unit 11 is not saturated.

[0130] This allows the photographer to accurately determine whether or not nearby objects are appearing in the captured image, distinguishing them from the effects of ambient light.

[0131] Figure 9 shows the external appearance of an imaging device according to a modified example of the same embodiment. Figure 10 shows the configuration of the processing block of the processing circuit in the modified example.

[0132] In this modified example, the display control unit 170 acquires 360-degree RGB image data from the RGB image data acquisition unit 142 and displays 2D image information based on the acquired 360-degree RGB image data on the display unit 520 of the display device 500. The display unit 520 is an example of an output destination for outputting 2D image information.

[0133] This makes it easy to check from the 2D image information displayed on the display unit 520 whether the photographer themselves or the tripod are visible in the captured image, or whether the desired 3D layout information has not been acquired.

[0134] The display control unit 170 outputs two-dimensional image information to a display unit 520 that is different from the external device 300 to which the transmitting / receiving unit 180 outputs three-dimensional information.

[0135] This makes it possible to confirm, from the 2D image information output to the display unit 520 (which is different from the external device 300), whether the photographer themselves or the tripod are visible in the captured image, or whether the desired 3D layout information has not been acquired, without having to check the 3D information output to the external device 300.

[0136] The display control unit 170 may acquire 3D data of the entire sphere from the 3D reconstruction processing unit 145 and display the 3D information on the display unit 520. Specifically, the display control unit 170 may select whether to display 2D image information or 3D information on the display unit 520 according to predetermined conditions. This allows the display control unit 170 to output 2D image information separately from 3D information.

[0137] The display control unit 170, based on the error information acquired from the judgment unit 160, superimposes the error information onto the two-dimensional image information and displays it on the display unit 520.

[0138] The display control unit 170 overlays identification information for identifying nearby objects onto the two-dimensional image information based on the pixel coordinate position information obtained from the determination unit 160 and displays it on the display unit 520.

[0139] In other words, the display control unit 170 causes the display unit 520 to display different information depending on the presence or absence of nearby objects, which is determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

[0140] This allows the photographer to accurately check whether they or nearby objects such as tripods are reflected in the captured image, distinguishing them from the effects of ambient light.

[0141] The display control unit 170 causes the display unit 520 to display different positions depending on the position of the nearby object. This allows the photographer to confirm the position of the nearby object in the captured image.

[0142] The display control unit 170 displays the image information captured by the imaging unit 11 on the display unit 520, and also displays identification information for identifying nearby objects superimposed on the image information on the display unit 520. This allows the photographer to reliably confirm the position of nearby objects appearing in the captured image.

[0143] Figure 11 shows the external appearance of an imaging device according to a second modified embodiment of the present invention. Figure 12 shows the configuration of the processing block of the processing circuit in the second modified embodiment.

[0144] In the second modified example shown in Figure 11, the imaging device 1 is equipped with multiple display units 20A, 20a instead of the display unit 20 shown in Figure 1. The display units 20A, 20a are composed of LEDs or the like and blink or light up in response to the output signal of the processing circuit 14.

[0145] The display unit 20a is provided on the first front surface of the housing 10, and the display unit 20A is provided on the second rear surface of the housing 10.

[0146] In the second modified example shown in Figure 12, the display control unit 170 displays information indicating the judgment result obtained from the judgment unit 160 on the display units 20A and 20a.

[0147] Furthermore, the transmitting / receiving unit 180 transmits (outputs) the 2D spherical image information output from the RGB image data acquisition unit 142 to the display device 500 via the network 400. The display device 500 is an example of an output destination for outputting 2D image information.

[0148] In other words, in the second modified example, in step S3 shown in Figure 5, the transmitting / receiving unit 180 acquires 360-degree RGB image data from the RGB image data acquisition unit 142 and transmits (outputs) two-dimensional image information based on the acquired 360-degree RGB image data to the display device 500.

[0149] The transmitting / receiving unit 510 of the display device 500 receives two-dimensional image information transmitted from the transmitting / receiving unit 180 of the imaging device 1.

[0150] The control unit 530 of the display device 500 displays the two-dimensional image information received by the transmitting / receiving unit 510 on the display unit 520.

[0151] As described above, the imaging device 1 comprises an imaging unit 11 and a transmitting / receiving unit 180 that outputs two-dimensional image information captured by the imaging unit 11 separately from three-dimensional information.

[0152] This makes it easy to check from 2D image data whether the photographer or tripod is visible in the captured image, or whether the desired 3D layout information has not been acquired, without having to check the 3D information.

[0153] Therefore, it becomes possible to reacquire 3D information while remaining at the site where the 3D information was acquired. This reduces the effort required to revisit the site to acquire the 3D information compared to realizing after leaving the site that the photographer or tripod, etc., are included in the captured image, or that 3D information of the desired layout has not been acquired.

[0154] The transmitting / receiving unit 180 transmits (outputs) 2D image information G in step S3 before transmitting (outputting) 3D information in step S11. The transmitting / receiving unit 180 transmits (outputs) 2D image information G in step S3 before the 3D reconstruction processing unit 150 determines the 3D information in step S10.

[0155] This makes it possible to check from the 2D image information whether the photographer themselves or the tripod are visible in the captured image, or whether the desired 3D layout information has not been acquired, before checking the 3D information.

[0156] The transmitting / receiving unit 180 transmits two-dimensional image information to the display device 500, and the display device 500 displays the two-dimensional image information on the display unit 520.

[0157] This makes it easy to check from the 2D image information displayed on the display unit 520 whether the photographer themselves or the tripod are visible in the captured image, or whether the desired 3D layout information has not been acquired.

[0158] The transmitting / receiving unit 180 transmits two-dimensional image information to a display device 500 that is different from the external device 300 that outputs three-dimensional information.

[0159] This makes it possible to confirm, from the 2D image information output to the display unit 520 of a display device 500 (which is different from the external device 300), that the photographer themselves or the tripod, etc., are not included in the captured image, and that the desired 3D layout information has not been acquired, without having to check the 3D information output to the external device 300.

[0160] The transmitting / receiving unit 180 may transmit 3D information to the display device 500. Specifically, the transmitting / receiving unit 180 may select whether to transmit 2D image information to the display device 500 or 3D information to the display device 500 according to predetermined conditions. This allows the transmitting / receiving unit 180 to transmit 2D image information to the display device 500 separately from 3D information.

[0161] Figure 13 is a flowchart for determining proximity in the second modified example.

[0162] Figure 13 is a flowchart showing the process for determining whether or not a nearby object is captured in the image, corresponding to step S9 described in Figure 5, in the second modified example.

[0163] The determination unit 160 determines, based on the spherical TOF image data acquired from the reprojection processing unit 147, whether there are any pixels in the spherical TOF image data whose stored energy has reached saturation (step S811).

[0164] If the determination unit 160 finds that there is a pixel whose charge has saturated in step S811, it determines whether the charge has saturated for the pixel at the same coordinate as the pixel whose charge was saturated in step S811, based on the 360-degree RGB image data acquired from the RGB image data acquisition unit 142 (step S812).

[0165] If the charge storage capacity is saturated in step S812, the determination unit 160 determines in step S811 that the pixels with saturated charge storage capacity are due to ambient light and outputs error information to the display control unit 170. Based on the error information obtained from the determination unit 160, the display control unit 170 displays the error information on the display units 20A and 20a (step S813).

[0166] If the charge level is not saturated in step S812, the determination unit 160 determines that the saturated charge level in the pixels in step S811 is due to the presence of nearby objects, and outputs the coordinate position information of the saturated pixels to the display control unit 170 in step S811. Based on the coordinate position information of the pixels obtained from the determination unit 160, the display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 (step S814).

[0167] If no pixels have reached saturation in step S811, the determination unit 160 determines, based on the spherical TOF image data acquired from the reprojection processing unit 147, whether there are any pixels in the spherical TOF image data that indicate a distance of 0.5 m or less (step S815).

[0168] If the determination unit 160 finds no pixels indicating distance information of 0.5m or less in step S815, it terminates the process.

[0169] If the determination unit 160 finds a pixel showing distance information of 0.5m or less in step S815, it proceeds to step S814, where it determines that the pixel showing distance information of 0.5m or less in step S815 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel showing distance information of 0.5m or less in step S815 to the display control unit 170. Based on the coordinate position information of the pixel obtained from the determination unit 160, the display control unit 170 determines whether the coordinate position information is on the front side of the housing 10.

[0170] If the display control unit 170 determines in step S814 that it is the front side, it flashes the display unit 20a located on the front side of the housing 10 (step S816).

[0171] If the display control unit 170 does not determine in step S814 that it is the front side, it flashes the display unit 20A located on the rear side of the housing 10 (step S817).

[0172] As explained above, the display control unit 170 flashes the display unit 20a or display unit 20A if it determines that an object is nearby, and does not flash the display unit 20a or display unit 20A if it does not determine that an object is nearby.

[0173] In other words, the display control unit 170 causes the display unit 20a and the display unit 20A to display different information depending on whether or not there is an object nearby.

[0174] This allows the photographer to accurately check whether they or nearby objects such as tripods are reflected in the captured image, distinguishing them from the effects of ambient light.

[0175] Furthermore, the display control unit 170 blinks the display unit 20a or the display unit 20A based on the pixel coordinate position information obtained from the determination unit 160.

[0176] In other words, the display control unit 170 causes the display units 20a and 20A to display different positions depending on the position of the nearby object. This allows the photographer to confirm the position of the nearby object in the captured image.

[0177] The display control unit 170 then causes the display unit 20A, 20a that is closer to the nearby object to display different information depending on whether or not the nearby object is present. This allows the photographer to reliably confirm the location of a specific object in the captured image.

[0178] Figure 14 is a diagram illustrating the configuration of an imaging device according to a third modified embodiment of the present invention.

[0179] In the third modified example shown in Figure 14, the imaging device 1 includes, in addition to the configuration shown in Figure 2, other imaging units 111 having other image sensors 111a, 111A, other fisheye lenses (wide-angle lenses) 111b, 111B, etc.

[0180] In the third modification, the RGB imaging unit 11 and the other imaging units 111 are located on the same baseline. In this case, multi-view processing becomes possible in the processing circuit 14. That is, by simultaneously driving the imaging unit 11 and the other imaging units 111 located at a predetermined distance apart on one plane, RGB images from two viewpoints can be obtained. Therefore, it becomes possible to use the parallax calculated based on the two RGB images, and further improve the distance accuracy of the entire measurement range.

[0181] Specifically, when an RGB imaging unit 11 and other imaging units 111 are provided, multi-baseline stereo (MSB) and EPI processing using SSSD become available, similar to conventional parallax calculations. Therefore, utilizing this improves the reliability of parallax, enabling the achievement of high spatial resolution and accuracy.

[0182] As described above, the imaging device 1 is equipped with another imaging unit 111, and the 3D reconstruction processing unit 150 determines 3D information based on the output of the distance information acquisition unit 13, 2D image information, and other 2D image information captured by the other imaging unit 111.

[0183] The imaging device 1 may also include another imaging unit 111 and a 3D information determination unit that determines 3D information based on 2D image information and other 2D image information captured by the other imaging unit 111, without relying on the output of the distance information acquisition unit 13.

[0184] This makes it possible to confirm from the 2D image information captured by the imaging unit 11 that the photographer themselves, the tripod, or other objects are not visible in the captured image, and that 3D information of the desired layout has not been acquired, without having to check the 3D information determined by the 3D reconstruction processing unit 150 based on the 2D image information.

[0185] As described above, an imaging device 1 (an example of an information processing device) according to one embodiment of the present invention comprises an imaging unit 11 that outputs three-dimensional information determined based on received light, and a display control unit 170 (an example of an output unit) or a transmitting / receiving unit 180 (an example of an output unit) that outputs two-dimensional image information G captured by the imaging unit 11 separately from the three-dimensional information.

[0186] This makes it easy to check from 2D image data whether the photographer or tripod is visible in the captured image, or whether the desired 3D layout information has not been acquired, without having to check the 3D information.

[0187] Therefore, it becomes possible to reacquire 3D information while remaining at the site where the 3D information was acquired. This reduces the effort required to revisit the site to acquire the 3D information compared to realizing after leaving the site that the photographer or tripod, etc., are included in the captured image, or that 3D information of the desired layout has not been acquired.

[0188] The display control unit 170 or the transmitting / receiving unit 180 outputs 2D image information G before outputting 3D information. The display control unit 170 or the transmitting / receiving unit 180 outputs 2D image information G before determining 3D information.

[0189] This makes it possible to check from the 2D image information whether the photographer themselves or the tripod are visible in the captured image, or whether the desired 3D layout information has not been acquired, before checking the 3D information.

[0190] The display control unit 170 causes the display units 20 and 520 to display two-dimensional image information G. The imaging device 1 includes a display unit 20.

[0191] This makes it easy to check from the 2D image information displayed on the display unit whether the photographer themselves or the tripod are visible in the captured image, or whether the desired 3D layout information has not been acquired.

[0192] The display control unit 170 or the transmitting / receiving unit 180 outputs 2D image information G to display units 20 and 520 (examples of output destinations) that are different from the external device 300 (example of output destination) that outputs 3D information.

[0193] This makes it possible to confirm, from the 2D image information output to the display units 20 and 520 (an example of an output destination) that is different from the external device 300, that the photographer themselves or the tripod, etc., are not included in the captured image, and that the desired layout's 3D information has not been acquired, without having to check the 3D information output to the external device 300.

[0194] The imaging device 1 comprises a projection unit 12 that projects light onto an object, a distance information acquisition unit 13 (an example of a light receiving unit) that receives light reflected from the object, and a 3D reconstruction processing unit 150 (an example of a 3D information determination unit) that determines 3D information based on the output of the distance information acquisition unit 13. The 3D reconstruction processing unit 150 determines 3D information based on the output of the distance information acquisition unit 13 and 2D image information.

[0195] This makes it possible to confirm from the 2D image information captured by the imaging unit 11 that the photographer themselves, the tripod, or other objects are not visible in the captured image, and that the desired 3D layout information has not been acquired, without having to check the 3D information determined by the 3D reconstruction processing unit 150.

[0196] The imaging device 1 includes another imaging unit 111, and the 3D reconstruction processing unit 150 determines 3D information based on the output of the distance information acquisition unit 13, 2D image information, and other 2D image information captured by the other imaging unit 111.

[0197] The imaging device 1 may also include another imaging unit 111 and a 3D information determination unit that determines 3D information based on 2D image information and other 2D image information captured by the other imaging unit 111, without relying on the output of the distance information acquisition unit 13.

[0198] This makes it possible to confirm from the 2D image information captured by the imaging unit 11 that the photographer themselves, the tripod, or other objects are not visible in the captured image, and that 3D information of the desired layout has not been acquired, without having to check the 3D information determined by the 3D reconstruction processing unit 150 based on the 2D image information.

[0199] The 3D information includes 3D information of the entire sphere. In this case, even with 3D information of the entire sphere, where it is difficult to confirm whether the photographer themselves or the tripod are reflected in the captured image, or whether 3D information of the desired layout has not been acquired, it becomes easy to confirm from the 2D image information captured by the imaging unit 11 that the photographer themselves or the tripod are reflected in the captured image, or whether 3D information of the desired layout has not been acquired. [Explanation of symbols]

[0200] 1. Imaging device (an example of an information processing device) 10 cabinets 11 Imaging Unit 11a, 11A image sensor 11b, 11B fisheye lens 12 Projection part 12a, 12A light source section 12b, 12B wide-angle lens 13. Distance information acquisition unit (an example of a light receiving unit) 13a, 13A TOF sensors 13b, 13B wide-angle lens 14 Processing Circuit 15. Shooting switch 20 Display section 20A, 20a display section 111 Other imaging units 150 3D Reconstruction Processing Unit (An example of a 3D Information Determination Unit) 160 Judgment Department 170 Display Control Unit (Example of Output Unit) 180 Transmitter / Receiver Unit (Example of Output Unit) 300 External devices (example of output destination) 500 Display device (example of output destination) 520 Display Unit (Example of Output Destination) L Sync signal line [Prior art documents] [Patent Documents]

[0201] [Patent Document 1] Japanese Patent Publication No. 2018-077071 [Patent Document 2] Patent No. 5423287

Claims

1. An information processing device that outputs three-dimensional information determined based on received light, An information processing device comprising an imaging unit and an output unit that outputs two-dimensional image information captured by the imaging unit, separately from the three-dimensional information.

2. The information processing apparatus according to claim 1, wherein the output unit outputs the two-dimensional image information before outputting the three-dimensional information.

3. The information processing apparatus according to claim 2, wherein the output unit outputs the two-dimensional image information before determining the three-dimensional information.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the output unit displays the two-dimensional image information on the display unit.

5. The information processing apparatus according to claim 4, comprising the display unit.

6. The information processing apparatus according to any one of claims 1 to 5, wherein the output unit outputs the two-dimensional image information to an output destination different from the output destination to which the three-dimensional information is output.

7. A projection unit that projects light onto the target, A light receiving unit that receives the light reflected from the aforementioned object, An information processing apparatus according to any one of claims 1 to 6, comprising a three-dimensional information determination unit that determines the three-dimensional information based on the output of the light receiving unit.

8. The information processing apparatus according to claim 7, wherein the three-dimensional information determination unit determines the three-dimensional information based on the output of the light receiving unit and the two-dimensional image information.

9. Equipped with other imaging units, The information processing apparatus according to claim 8, wherein the three-dimensional information determination unit determines the three-dimensional information based on the output of the light receiving unit, the two-dimensional image information, and other two-dimensional image information captured by the other imaging unit.

10. Other imaging units, An information processing apparatus according to any one of claims 1 to 9, comprising: a three-dimensional information determination unit that determines the three-dimensional information based on the two-dimensional image information and other two-dimensional image information captured by the other imaging unit.

11. The information processing apparatus according to any one of claims 1 to 10, wherein the three-dimensional information includes all-sphere three-dimensional information.

12. Imaging step, A 3D information output step that outputs 3D information determined based on the received light, In addition to the three-dimensional information output step, there is a two-dimensional image information output step that outputs the two-dimensional image information captured by the imaging step, An information processing method equipped with [a specific feature / feature].

Images