Image capture device

Abstract

To provide an information processing device and an information processing method that makes it easy to confirm that a photographer himself or herself, a tripod or the like do not appear in a captured image and that desired three-dimensional information is not acquired.SOLUTION: An image capture device 1, which outputs three-dimensional information determined based on received light, comprises an image capture unit 11, and a display control unit 170 for outputting two-dimensional image information G captured by the image capture unit 11 apart from the three-dimensional information. The display control unit 170 causes the two-dimensional image information G to be output before the determination of the three-dimensional information.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Patent Document 1 describes a distance measuring device that can stably and accurately measure the distance to an object.

[0003] Patent Document 2 describes an imaging device that performs image processing to reduce the influence of a finger or other object captured in the image when that object is captured in the image. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide an information processing apparatus and an information processing method that allow easy confirmation that desired three-dimensional information has been acquired. [Means for solving the problem]

[0005] The information processing device according to the present invention includes an imaging section that outputs three-dimensional information determined based on received light, and an output section that outputs two-dimensional image information captured by the imaging section separately from the three-dimensional information. Effect of the Invention

[0006] According to the present invention, it is possible to provide an information processing apparatus and an information processing method that allow easy confirmation that desired three-dimensional information has been acquired. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram showing an example of the appearance of an imaging device according to an embodiment of the present invention. [Diagram 2] 2 is a diagram for explaining a configuration of an imaging device in the embodiment. FIG. [Diagram 3]2A to 2C are diagrams for explaining a usage state of the imaging device in the embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a configuration of a processing block of a processing circuit in the embodiment. [Diagram 5] 10 is a flow chart showing an example of the operation of a processing circuit of the imaging device according to the embodiment. [Figure 6] FIG. 11 is a flowchart showing a generation process of spherical image data in the embodiment. [Figure 7] FIG. 11 is a flowchart showing a nearby object determination process according to the embodiment. [Figure 8] 4A to 4C are diagrams for explaining display contents of a display unit in the embodiment. [Figure 9] 13 is a diagram showing the appearance of an imaging device according to a modified example of the embodiment. FIG. [Figure 10] FIG. 13 is a diagram showing a configuration of a processing block of a processing circuit in a modified example. [Figure 11] FIG. 13 is a diagram showing the appearance of an imaging device according to a second modified example of the embodiment of the present invention. [Figure 12] FIG. 13 is a diagram showing a configuration of a processing block of a processing circuit in a second modified example. [Figure 13] FIG. 11 is a flowchart of a nearby object determination process according to a second modified example. [Figure 14] FIG. 13 is a diagram illustrating a configuration of an imaging device according to a third modified example of an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of an information processing apparatus and an information processing method will be described in detail with reference to the accompanying drawings.

[0009] Fig. 1 is a diagram showing an example of the external appearance of an imaging device according to an embodiment of the present invention. Fig. 2 is a diagram for explaining the configuration of the imaging device. Fig. 2 shows the internal configuration of the imaging device of Fig. 1.

[0010] The imaging device 1 is an example of an information processing device that outputs three-dimensional information determined based on received light, and includes an imaging section (camera) 11, a projection section (corresponding to a light-emitting section of a distance sensor) 12 that projects light other than visible light, and a distance information acquisition section (corresponding to a light-receiving section of a distance sensor) 13 that acquires distance information based on the light projected by the projection section 12, all of which are provided integrally with a housing 10. Each section is electrically connected to a processing circuit 14 inside the housing 10 by a synchronization signal line L, and operates synchronously with each other.

[0011] The photographing switch 15 is for the user to input a photographing instruction signal to the processing circuit 14. The display unit 20 displays contents according to the output signal of the processing circuit 14, and is composed of a liquid crystal screen or the like. The display unit 20 may be composed of a touch panel or the like, and may receive operation input from the user. Based on the photographing instruction, the processing circuit 14 controls each unit to acquire RGB image and distance information data, and performs processing to reconstruct the acquired distance information data into high-density three-dimensional point cloud data based on the RGB image and distance information data. The distance information data can be used as is to construct three-dimensional point cloud data, but in that case, the accuracy of the three-dimensional point cloud data is limited by the number of pixels (resolution) of the distance information acquisition unit 13. In this example, processing for reconstructing it into high-density three-dimensional point cloud data is also shown. The reconstructed data is output to an external PC or the like via a portable recording medium or communication, and is used to display a three-dimensional restoration model.

[0012] Power is supplied to each unit and the processing circuit 14 from a battery housed inside the housing 10. Alternatively, power may be supplied from outside the housing 10 via a connection cord.

[0013] The imaging unit 11 includes imaging elements 11a and 11A, fisheye lenses (wide-angle lenses) 11b and 11B, etc. The projection unit 12 includes light source units 12a and 12A, wide-angle lenses 12b and 12B, etc. The distance information acquisition unit 13 includes TOF (Time Of Flight) sensors 13a and 13A, wide-angle lenses 13b and 13B, etc. Although not shown, each unit may be configured as an optical system such as a prism or a lens group. For example, the imaging unit 11 may be configured as an optical system for forming an image on the imaging elements 11a and 11A using light collected by the fisheye lenses 11b and 11B. The projection unit 12 may be configured as an optical system for guiding light from the light source units 12a and 12A to the wide-angle lenses 12b and 12B. Further, an optical system for forming an image on TOF sensors 13a, 13A of light collected by wide-angle lenses 13b, 13B may be configured in distance information acquisition unit 13. Each optical system may be appropriately determined according to the configuration and arrangement of image pickup elements 11a, 11A, light source units 12a, 12A, TOF sensors 13a, 13A, etc., and here, the description will be omitted for the optical system such as a prism and a lens group.

[0014] Imaging elements 11a, 11A, light source units 12a, 12A, and TOF sensors 13a, 13A are integrally housed inside housing 10. Fisheye lens 11b, wide-angle lens 12b, wide-angle lens 13b, and display unit 20 are each provided on a first surface on the front side of housing 10. On the first surface, the inner ranges of fisheye lens 11b, wide-angle lens 12b, and wide-angle lens 13b are each open.

[0015] Fisheye lens 11B, wide-angle lens 12B, wide-angle lens 13B, and shooting switch 15 are each provided on a second surface on the rear side of housing 10. On the second surface, the inner ranges of fisheye lens 11B, wide-angle lens 12B, and wide-angle lens 13B are each open.

[0016] The imaging elements 11a and 11A are image sensors (area sensors) with two-dimensional resolution. The imaging elements 11a and 11A have an imaging area in which a large number of light receiving elements (photodiodes) for each pixel are arranged in a two-dimensional direction. In the imaging area, R (Red), G (Green), and B (Blue) color filters such as a Bayer array are provided to receive visible light, and light passing through the color filters is stored in the photodiodes. Here, an image sensor with a large number of pixels is used so that a two-dimensional image with a wide angle (for example, a range of a hemisphere of 180 degrees around the imaging direction shown in FIG. 2) can be acquired with high resolution. The imaging elements 11a and 11A convert the light imaged in the imaging area into an electric signal by the pixel circuit of each pixel, and output a high-resolution RGB image. The fisheye lenses 11b and 11B collect light from a wide angle (for example, a range of a hemisphere of 180 degrees around the imaging direction shown in FIG. 2) and image the light in the imaging area of ​​the imaging elements 11a and 11A.

[0017] The light source units 12a and 12A are semiconductor lasers, and emit laser light in a wavelength band other than the visible light region used for distance measurement (here, infrared is used as an example). The light source units 12a and 12A may use one semiconductor laser, or may use a combination of multiple semiconductor lasers. In addition, a surface-emitting semiconductor laser such as a VCSEL (Vertical Cavity Surface Emitting LASER) may be used as the semiconductor laser. In addition, the light of the semiconductor laser may be shaped to be elongated vertically by an optical lens, and the elongated light may be scanned in a one-dimensional direction in the measurement range by an optical deflection element such as a MEMS (Micro Electro Mechanical Systems) mirror. In this embodiment, the light source units 12a and 12A show a form in which the light of the semiconductor laser LA is expanded to a wide angle range via wide-angle lenses 12b and 12B without using an optical deflection element such as a MEMS mirror.

[0018] The wide-angle lenses 12b, 12B of the light source units 12a, 12A have the function of expanding the light emitted by the light source units 12a, 12A to a wide-angle range (for example, a hemispherical range of 180 degrees around the imaging direction shown in FIG. 2).

[0019] The wide-angle lenses 13b and 13B of the distance information acquisition unit 13 capture reflected light of the light from the light source units 12a and 12A projected by the projection unit 12 from each direction of a wide angle (for example, a hemispherical range of 180 degrees around the imaging direction shown in FIG. 2) that is the measurement range, and form an image of the light on the light receiving area of ​​the TOF sensors 13a and 13A. The measurement range includes one or more projection targets (for example, buildings, etc.), and light reflected by the projection targets (reflected light) enters the wide-angle lenses 13b and 13B. The reflected light may be captured by providing a filter that cuts off light with wavelengths in the infrared range or longer on the entire surface of the wide-angle lenses 13b and 13B, for example. Note that this is not limited to this, and since it is only necessary that light in the infrared range enters the light receiving area, a means for passing light with wavelengths in the infrared range, such as a filter, may be provided on the optical path from the wide-angle lenses 13b and 13B to the light receiving area.

[0020] The TOF sensors 13a and 13A are optical sensors with two-dimensional resolution. The TOF sensors 13a and 13A have a light receiving area in which a large number of light receiving elements (photodiodes) are arranged in a two-dimensional direction. In this sense, they can be called "second imaging light receiving means." The TOF sensors 13a and 13A receive reflected light from each area (each area is also called a position) in the measurement range with the light receiving elements corresponding to each area, and measure (calculate) the distance to each area based on the light detected by each light receiving element.

[0021] In this embodiment, the distance is measured using a phase difference detection method. In the phase difference detection method, a laser beam that is amplitude modulated at a fundamental frequency is irradiated onto a measurement range, and the reflected light is received to measure the phase difference between the irradiated light and the reflected light to obtain time, which is then multiplied by the speed of light to calculate distance. The advantage of this method is that it can provide a certain degree of resolution.

[0022] The TOF sensors 13a and 13A are driven in synchronization with the irradiation of light by the projection unit 12, calculate the distance corresponding to each pixel from the phase difference with the reflected light at each light receiving element (corresponding to a pixel), and output distance information image data (hereinafter also referred to as a "distance image" or "TOF image") in which information indicating the distance to each area within the measurement range is associated with the pixel information. The TOF sensors 13a and 13A may output phase information image data in which phase information is associated with pixel information, and acquire distance information image data based on the phase information image data in post-processing.

[0023] The number of areas into which the measurement range can be divided is determined by the resolution of the light receiving area. Therefore, if a low-resolution sensor is used to reduce size, the number of pixels in the range image data will decrease, and the number of 3D points will also decrease.

[0024] As another embodiment, distance may be measured by a pulse method instead of the phase difference detection method. In that case, for example, the light source units 12a and 12A emit an ultrashort pulse irradiation pulse P1 with a rise time of several nanoseconds (ns) and a strong optical peak power, and the TOF sensors 13a and 13A measure the time (t) required to receive the reflected pulse P2, which is the reflected light of the irradiation pulse P1 emitted by the light source units 12a and 12A, in synchronization with the irradiation pulse P1. When this method is adopted, for example, the TOF sensors 13a and 13A are used that are equipped with a circuit for measuring time on the output side of the light receiving element. In each circuit, the time required from the light source units 12a and 12A emitting the irradiation pulse P1 to receiving the reflected pulse P2 is converted into a distance for each light receiving element, and the distance to each area is obtained.

[0025] This method is suitable for widening the angle of the imaging device 1 because it can output strong light using peak light. In addition, when a MEMS mirror or the like is used to deflect (scan) the light, strong light can be irradiated far away while suppressing the spread, which leads to an increase in the measurement distance. In this case, the laser light emitted from the light source units 12a and 12A is arranged so that it is scanned (deflected) by the MEMS mirror toward the wide-angle lenses 12b and 12B.

[0026] It is desirable that the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 are the same, for example, at 180 degrees or more, but they do not necessarily have to be the same. If necessary, the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 may be reduced. In this embodiment, the effective pixels of the imaging unit 11 and the distance information acquisition unit 13 are reduced to, for example, within a range of 100 degrees to 180 degrees so that there is no interference with the angle of view. In addition, the resolution of the TOF sensors 13a and 13A may be set lower than the resolution of the imaging elements 11a and 11A by prioritizing the miniaturization of the imaging device 1. By making the TOF sensors 13a and 13A lower resolution than the imaging elements 11a and 11A, the size expansion of the light receiving area can be suppressed, which leads to the miniaturization of the imaging device 1. For this reason, the TOF sensors 13a and 13A have a low resolution, and the 3D point clouds obtained by the TOF sensors 13a and 13A have a low density, but the processing circuit 14, which is an "acquisition means," can be used to convert the 3D point cloud into a high density one. The process of converting the 3D point cloud into a high density one in the processing circuit 14 will be described later.

[0027] In the present embodiment, as an example, the imaging element 11a, the light source unit 12a, and the TOF sensor 13a are arranged so as to be aligned in a straight line in the longitudinal direction of the housing 10. Similarly, the imaging element 11A, the light source unit 12A, and the TOF sensor 13A are arranged so as to be aligned in a straight line in the longitudinal direction of the housing 10. Below, an example of the imaging element 11a, the light source unit 12a, and the TOF sensor 13a will be described.

[0028] The imaging area (imaging surface) of the imaging element 11a and the light receiving area (light receiving surface) of the TOF sensor 13a may be arranged in a direction perpendicular to the longitudinal direction as shown in FIG. 2, or may be arranged in the longitudinal direction by providing a prism that changes the linear direction (optical path) of light by 90 degrees and makes it incident. In addition, they may be arranged in any direction depending on the configuration. In other words, the imaging element 11a, the light source unit 12a, and the TOF sensor 13a are arranged so that the same measurement range is the target. The imaging unit 11, the projection unit 12, and the distance information acquisition unit 13 are arranged from one side of the housing 10 toward the measurement range. In this case, it is sufficient that the imaging element 11a and the TOF sensor 13a are arranged on the same base line so as to achieve parallel stereo. By arranging them so as to achieve parallel stereo, even if there is only one imaging element 11a, it is possible to obtain parallax data using the output of the TOF sensor 13a. The light source unit 12a is configured so as to be able to irradiate light onto the measurement range of the TOF sensor 13a.

[0029] (Processing circuit) Next, the processing of the processing circuit 14 will be described. The TOF image obtained only by the TOF sensors 13a and 13A has low resolution as it is. For this reason, in this example, the resolution is increased by the processing circuit 14, and high-density 3D point cloud data is reconstructed. Note that some or all of the processing described below as "information processing means" in the processing circuit 14 may be performed by an external device.

[0030] As described above, the three-dimensional point cloud data reconstructed by the imaging device 1 is output to an external device such as a PC via a portable recording medium or communication, and is used to display a three-dimensional restored model.

[0031] This makes it possible to provide an imaging device 1 that is faster, smaller, and lighter, and therefore more portable, than when the imaging device 1 itself displays the 3D reconstruction model.

[0032] However, after leaving the site where the 3D information was acquired and restoring the 3D information using an external device, it may be noticed that the photographer or the tripod is captured in the captured image, or that the 3D information was not captured in the desired layout, which requires the effort of revisiting the site where the 3D information was acquired.

[0033] One solution to this problem would be to bring in a 3D reconstruction device on-site, but then the benefits of high speed, compactness, and light weight would be lost.

[0034] It is also possible to transmit the acquired 3D information to an external device via a communication line and then receive the restored 3D information, but this would eliminate the advantage of high speed, and in addition, 3D information contains a large amount of information, making it difficult to visually confirm whether the photographer, the tripod, etc. are reflected in the captured image.

[0035] In particular, in the case of spherical three-dimensional information, it is extremely difficult to visually check whether the photographer, a tripod, or other objects are captured in the captured image.

[0036] In view of the above problems, the present embodiment aims to provide an imaging device 1 that allows easy confirmation in real time of whether the photographer himself / herself, a tripod, etc. are captured in the captured image, or whether 3D information of the desired layout has not been acquired.

[0037] FIG. 3 is a diagram for explaining a usage state of the imaging device in the embodiment.

[0038] In the state shown in FIG. 3(a), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the omnidirectional imaging range R, and the photographer M and the selfie stick 1A are not captured in the omnidirectional captured image.

[0039] In the state shown in FIG. 3(b), the photographer M is included in the omnidirectional imaging range R, and the photographer M ends up being captured in the omnidirectional captured image.

[0040] In the state shown in FIG. 3(c), the tripod 1B supporting the imaging device 1 is included in the omnidirectional imaging range R, and the tripod 1B ends up appearing in the captured image of the omnidirectional imaging range R.

[0041] In the state shown in FIG. 3(d), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the spherical imaging range R, and the photographer M and the selfie stick 1A are not captured in the spherical captured image. However, due to strong external light, there is a possibility that the photographer M and the selfie stick 1A are erroneously determined to be captured in the spherical captured image.

[0042] In addition, in the conditions shown in Figures 3(b) and 3(c), the colors and types of objects reflected in the image, as well as the way they appear, are diverse, making it difficult to uniformly determine whether or not an object is reflected.

[0043] In the above situation, when determining the presence or absence of a specific object (nearby object) such as the photographer himself or a tripod based on the distance information image data output from the TOF sensors 13a and 13A, it was difficult to distinguish whether a specific object was actually present or whether the external light was too strong.

[0044] In other words, when the charge storage amount of a particular pixel of the TOF sensors 13a and 13A becomes saturated, it is difficult to distinguish from the output of the TOF sensors 13a and 13A alone whether this is due to the presence of a particular object or because the intensity of external light is too strong.

[0045] In view of the above-mentioned problems, another object of the present embodiment is to provide an imaging device 1 that can accurately check whether or not a specific object, such as the photographer himself or herself or a tripod, is reflected in a captured image, distinguishing it from the effects of external light.

[0046] Fig. 4 is a diagram showing an example of the configuration of the processing block of the processing circuit 14. The processing circuit 14 shown in Fig. 4 has a control unit 141, an RGB image data acquisition unit 142, a monochrome processing unit 143, a TOF image data acquisition unit 144, a resolution increasing unit 145, a matching processing unit 146, a reprojection processing unit 147, a semantic segmentation unit 148, a parallax calculation unit 149, a 3D reconstruction processing unit 150, a determination unit 160, a display control unit 170 which is an example of an output unit, and a transmission / reception unit 180 which is an example of an output unit. In Fig. 4, the solid line arrows indicate the flow of signals, and the dashed line arrows indicate the flow of data.

[0047] When the control unit 141 receives an ON signal (photographing start signal) from the photographing switch 15, it outputs a synchronization signal to the image pickup elements 11a, 11A, the light source units 12a, 12A, and the TOF sensors 13a, 13A, and controls the entire processing circuit 14. The control unit 141 first outputs a signal to the light source units 12a, 12A to instruct them to emit ultrashort pulses, and at the same timing outputs a signal to the TOF sensors 13a, 13A to instruct them to generate TOF image data. Furthermore, the control unit 141 outputs a signal to instruct the image pickup elements 11a, 11A to capture an image. Note that the image pickup in the image pickup elements 11a, 11A may be performed during the period when light is emitted from the light source units 12a, 12A, or during the period immediately before or after that.

[0048] The RGB image data acquisition unit 142 acquires RGB image data captured by the imaging elements 11a and 11A based on an imaging instruction from the control unit 141, and outputs RGB image data of the celestial sphere. The monochrome processing unit 143 performs processing to align data types for matching processing with TOF image data obtained from the TOF sensors 13a and 13A. In this example, the monochrome processing unit 143 performs processing to convert the RGB image data of the celestial sphere into a monochrome image of the celestial sphere.

[0049] The TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on an instruction to generate TOF image data from the control unit 141, and outputs TOF image data of the entire celestial sphere.

[0050] The resolution increasing unit 145 increases the resolution of the TOF image data of the celestial sphere by regarding it as a monochrome image. Specifically, the resolution increasing unit 145 replaces the distance value associated with each pixel of the TOF image data of the celestial sphere with the value (grayscale value) of the monochrome image of the celestial sphere and uses it. Furthermore, the resolution increasing unit 145 increases the resolution of the monochrome image of the celestial sphere to the resolution of the RGB image data of the celestial sphere obtained from the imaging elements 11a and 11A. The conversion to high resolution is performed, for example, by performing a normal up-conversion process. As another conversion method, for example, multiple frames of TOF image data of the celestial sphere generated continuously may be obtained, and the distances of adjacent points may be added using the frames to perform super-resolution processing.

[0051] The matching processing unit 146 extracts features of textured parts of a monochrome image of the celestial sphere obtained by increasing the resolution of the TOF image data of the celestial sphere and a monochrome image of the celestial sphere obtained by increasing the resolution of the celestial sphere's RGB image data, and performs a matching process using the extracted features. For example, the matching processing unit 146 extracts edges from each monochrome image, and performs a matching process between the extracted edge information. As another method, the matching process may be performed using a method of converting changes in texture into features, such as SIFT. Here, the matching process means searching for corresponding pixels.

[0052] A specific matching method is, for example, block matching, which calculates the similarity between pixel values ​​cut out as a block of M x M pixels (M is a positive integer) around a reference pixel and pixel values ​​cut out as a block of M x M pixels around a pixel that is the center of the search in another image, and determines the center pixel with the highest similarity as the corresponding pixel.

[0053] There are various methods for calculating similarity. For example, the formula showing the normalized autocorrelation coefficient (CNCC) may be used. The higher the normalized autocorrelation coefficient (CNCC) value, the higher the similarity. When the pixel values ​​of the blocks are completely the same, the value is 1.

[0054] In addition, since the distance data of the textureless area can be obtained from the TOF image data of the celestial sphere, the matching process may be weighted according to the area. For example, in the calculation of the formula showing CNCC, a calculation may be performed in which weight is applied to the areas other than the edges (textureless areas).

[0055] Moreover, instead of the formula indicating the NCC, the selective correlation coefficient (SCC) or the like may be used.

[0056] The reprojection processing unit 147 performs a process of reprojecting the TOF image data of the celestial sphere, which indicates the distance of each position in the measurement range, onto the two-dimensional coordinates (screen coordinate system) of the imaging unit 11. The reprojection is to obtain the coordinates at which the three-dimensional points calculated by the TOF sensors 13a and 13A are reflected in the images of the imaging elements 11a and 11A. The TOF image data of the celestial sphere indicates the position of the three-dimensional point in a coordinate system centered on the distance information acquisition unit 13 (mainly the wide-angle lenses 13b and 13B). Therefore, the three-dimensional point indicated by the TOF image data of the celestial sphere is reprojected onto a coordinate system centered on the imaging unit 11 (mainly the fisheye lenses 11b and 11B). For example, the reprojection processing unit 147 translates the coordinates of the three-dimensional point of the TOF image data of the celestial sphere to the coordinates of the three-dimensional point centered on the imaging unit 11, and after the translation, performs a process of converting the coordinates into the two-dimensional coordinate system (screen coordinate system) indicated by the RGB image data of the celestial sphere.

[0057] A parallax calculation unit 149 calculates the parallax at each position from the deviation in distance to the corresponding pixel obtained by the matching process.

[0058] In addition, the parallax matching process uses the reprojection coordinates converted by the reprojection processing unit 147 to search for surrounding pixels around the position of the reprojection coordinates, making it possible to shorten the processing time and obtain more detailed, high-resolution distance information.

[0059] Furthermore, the disparity matching process may use segmentation data obtained by the semantic segmentation process of the semantic segmentation unit 148. In this case, it becomes possible to obtain more detailed and high-resolution distance information.

[0060] Alternatively, disparity matching processing may be performed only on edges or only on parts with strong features, and propagation processing may be performed for other parts by using spherical TOF image data, for example, by using spherical RGB image features or a probabilistic method.

[0061] The semantic segmentation unit 148 uses deep learning to assign segmentation labels indicating objects to the input image in the measurement range. This makes it possible to constrain each pixel of the omnidirectional TOF image data to one of a plurality of distance regions divided by distance, further improving the reliability of the calculation.

[0062] The 3D reconstruction processing unit 145 acquires the RGB image data of the celestial sphere from the RGB image data acquisition unit 142, reconstructs three-dimensional data of the celestial sphere based on the distance information output by the parallax calculation unit 149, and outputs a high-density three-dimensional point cloud of the celestial sphere in which color information is added to each three-dimensional point. The 3D reconstruction processing unit 150 is an example of a three-dimensional information determination unit that determines three-dimensional information.

[0063] The determination unit 160 acquires RGB image data of the celestial sphere from the RGB image data acquisition unit 142, and also acquires TOF image data of the celestial sphere converted into a two-dimensional coordinate system indicated by the RGB image data of the celestial sphere from the reprojection processing unit 147. Based on these pieces of data, the determination unit 160 determines whether or not a specific object is reflected in the captured image, and outputs the determination result to the display control unit 170.

[0064] The display control unit 170 acquires RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and causes two-dimensional image information based on the acquired RGB image data of the entire celestial sphere to be displayed on the display unit 20. In addition, the display control unit 170 causes information indicating the determination result acquired from the determination unit 160 to be superimposed on the two-dimensional image information and displayed on the display unit 20.

[0065] The display control unit 170 is an example of an output unit that outputs two-dimensional image information captured by the imaging unit 11 separately from the three-dimensional information, and the display unit 20 is an example of an output destination that outputs the two-dimensional image information.

[0066] The display control unit 170 may obtain three-dimensional data of the entire celestial sphere from the three-dimensional reconstruction processing unit 145, and cause the display unit 20 to display the three-dimensional information. Specifically, the display control unit 170 may select, in accordance with a predetermined condition, whether to display two-dimensional image information on the display unit 20 or to display three-dimensional information on the display unit 20. This allows the display control unit 170 to output the two-dimensional image information separately from the three-dimensional information.

[0067] The transmission / reception unit 180 communicates with an external device by wired or wireless technology, and transmits (outputs) the three-dimensional data of the celestial sphere output from the 3D reconstruction processing unit 145 and the two-dimensional image information of the celestial sphere output from the RGB image data acquisition unit 142 via the network 400 to the external device 300 that performs three-dimensional reconstruction processing.

[0068] The transmitting / receiving unit 180 is an example of an output unit that outputs three-dimensional information, and the external device 300 is an example of an output destination that outputs the three-dimensional information.

[0069] The transmitting / receiving unit 180 may transmit only the three-dimensional data of the celestial sphere without transmitting the two-dimensional image information of the celestial sphere. The transmitting / receiving unit 180 may be configured by an interface circuit with a portable storage medium such as an SD card, a personal computer, or the like.

[0070] (Operation of the processing circuit) 5 is a flow diagram showing an example of the operation of the processing circuit 14 of the imaging device 1. When the user turns on the imaging switch 15 and inputs an imaging instruction signal, the control unit 141 of the processing circuit 14 performs an operation to generate a high-density three-dimensional point cloud in the following manner (an example of an imaging processing method and an information processing method).

[0071] First, the control unit 141 drives the light source units 12a and 12A, the TOF sensors 13a and 13A, and the image pickup elements 11a and 11A to capture an image of the measurement range (step S1). Driven by the control unit 141, the light source units 12a and 12A irradiate infrared light (an example of a projection step), and the TOF sensors 13a and 13A receive the reflected light (an example of a light receiving step). In addition, the image pickup elements 11a and 11A capture an image of the measurement range at the timing when the light source units 12a and 12A start to be driven or in a period immediately thereafter (an example of an image pickup step).

[0072] Next, the RGB image data acquisition unit 142 acquires RGB image data of the measurement range from the imaging elements 11a and 11A (step S2). Then, the display control unit 170 acquires the RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and displays two-dimensional image information based on the acquired RGB image data of the entire celestial sphere on the display unit 20 (an example of a two-dimensional image information output step) (step S3).

[0073] The display control unit 170 displays two-dimensional image information of a partial area of ​​the acquired RGB image data of the entire celestial sphere on the display unit 20, and changes the area of ​​the two-dimensional image information displayed on the display unit 20 according to various inputs from the user. Various inputs from the user can be realized by providing an operation switch other than the photographing switch 15 or by configuring the display unit 20 as an input unit such as a touch panel.

[0074] At this stage, the photographer can check from the 2D image information displayed on the display unit 20 that the photographer himself or the tripod, etc., is not reflected in the captured image, and that 2D image information of the desired layout has not been obtained.

[0075] Next, the TOF image data acquisition unit 144 acquires TOF image data indicating the distance from the TOF sensors 13a and 13A to each position in the two-dimensional area (step S4).

[0076] Next, the monochrome processing unit 143 converts the RGB image data into a monochrome image (step S5). The TOF image data and the RGB image data are different data types, namely distance data and RGB data, respectively, and matching cannot be performed as is. Therefore, first, each data is converted into a monochrome image. For the TOF image data, the resolution increasing unit 145 converts the values ​​indicating the distance of each pixel before resolution increase by directly replacing them with the values ​​of the monochrome image.

[0077] Next, the resolution increasing unit 145 increases the resolution of the TOF image data (step S6).

[0078] Next, the matching processing unit 146 extracts the feature amount of a portion having texture for each monochrome image, and performs matching processing using the extracted feature amount (step S7).

[0079] Next, the parallax calculation unit 149 calculates the parallax at each position from the deviation in distance between corresponding pixels, and calculates the distance (step S8).

[0080] Next, the determination unit 160 acquires RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and acquires TOF image data of the entire celestial sphere converted into a two-dimensional coordinate system indicated by the RGB image data from the reprojection processing unit 147, and determines whether or not a nearby object as a specific target is reflected in the captured image based on these pieces of data, and outputs the determination result to the display control unit 170. The display control unit 170 superimposes information indicating the determination result acquired from the determination unit 160 on the two-dimensional image information and displays it on the display unit 20 (an example of a display step) (step S9).

[0081] Then, the 3D reconstruction processing unit 145 acquires the RGB image data from the RGB image data acquisition unit 142, reconstructs the 3D data based on the distance information output by the disparity calculation unit 149, and outputs a high-density 3D point cloud with color information added to each 3D point (step S10).

[0082] Next, the transmitter / receiver unit 180 transmits the three-dimensional data output from the three-dimensional reconstruction processing unit 145 and the two-dimensional image information output from the RGB image data acquisition unit 142 via the network 400 to an external device 300 that performs three-dimensional restoration processing (an example of a three-dimensional information output step) (step S11).

[0083] The transmitting / receiving unit 180 may transmit the three-dimensional data output from the three-dimensional reconstruction processing unit 145 without transmitting the two-dimensional image information output from the RGB image data acquisition unit 142 .

[0084] As described above, the imaging device 1 includes the imaging section 11 and the display control section 170 that outputs two-dimensional image information captured by the imaging section 11 separately from three-dimensional information.

[0085] This makes it possible to easily check from the 2D image information whether the photographer himself / herself, a tripod, etc., is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information.

[0086] Therefore, it becomes possible to re-acquire 3D information while remaining at the site where the 3D information is acquired, which reduces the effort of re-visiting the site where the 3D information is acquired compared to realizing, after leaving the site where the 3D information is acquired, that the photographer or the tripod, etc., is captured in the captured image or that 3D information with the desired layout has not been acquired.

[0087] The three-dimensional information includes spherical three-dimensional information. In this case, even in the case of spherical three-dimensional information in which it is difficult to confirm that the photographer himself / herself, a tripod, etc. are included in the captured image or that three-dimensional information of a desired layout has not been acquired, it becomes possible to easily confirm from the two-dimensional image information captured by the imaging unit 11 that the photographer himself / herself, a tripod, etc. are included in the captured image or that three-dimensional information of a desired layout has not been acquired.

[0088] The display control unit 170 outputs the two-dimensional image information G in step S3 before the transmitting / receiving unit 180 transmits (outputs) the three-dimensional information in step S11. The display control unit 170 outputs the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

[0089] This makes it possible to check from the 2D image information whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been acquired, before checking the 3D information.

[0090] The display control unit 170 causes the display unit 20 to display the two-dimensional image information.

[0091] This makes it possible to easily check from the 2D image information displayed on the display unit 20 whether the photographer himself / herself, a tripod, etc., is captured in the captured image, or whether 3D information of the desired layout has not been obtained.

[0092] The display control unit 170 outputs the two-dimensional image information to a display unit 20 that is different from the external device 300 to which the transmitting and receiving unit 180 outputs the three-dimensional information.

[0093] This makes it possible to check whether the photographer himself / herself, the tripod, etc. are reflected in the captured image, or whether 3D information of the desired layout has not been obtained, from the 2D image information output to a display unit 20 different from the external device 300, without checking the 3D information output to the external device 300.

[0094] The imaging device 1 includes a 3D reconstruction processor 150 that determines 3D information based on the output of the distance information acquisition unit 13. The 3D reconstruction processor 150 determines the 3D information based on the output of the distance information acquisition unit 13 and the 2D image information.

[0095] This makes it possible to check from the 2D image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information determined by the 3D reconstruction processing unit 150.

[0096] FIG. 6 is a flowchart showing a process for generating spherical image data in the embodiment.

[0097] FIG. 6A is a flowchart showing the process of generating RGB image data of the entire celestial sphere, which corresponds to step S2 described in FIG.

[0098] The RGB image data acquisition unit 142 receives two pieces of RGB image data in a fisheye image format (step S201).

[0099] The RGB image data acquisition unit 142 converts each RGB image data into an equirectangular image format (step S202). The RGB image data acquisition unit 142 converts the two RGB image data into an equirectangular image format based on the same coordinate system, which makes it easier to combine the images in the next step.

[0100] Here, the equirectangular image format will be described. The equirectangular image format is a method capable of expressing a celestial sphere image, and is a format of an image (equirectangular image) created using equirectangular projection. Equirectangular projection is a projection that expresses three-dimensional directions with two variables, such as the latitude and longitude of a globe, and displays the image on a plane so that the latitude and longitude are orthogonal. Therefore, an equirectangular image is an image generated using equirectangular projection, and is expressed in coordinates with two axes that are two angular variables of a spherical coordinate system.

[0101] The RGB image data acquisition unit 142 combines the two RGB image data generated in step S202 to generate one piece of spherical RGB image data (step S203). The two pieces of input RGB image data cover an area with a full angle of view of more than 180 degrees. Therefore, the spherical RGB image data generated by appropriately connecting the two pieces of RGB image data can cover the spherical area.

[0102] The combining process in step S203 can use existing techniques for joining multiple images, and there are no particular limitations on the method.

[0103] FIG. 6(b) shows the generation of TOF image data of the celestial sphere corresponding to step S4 described in FIG. 1 is a flowchart showing a generation process.

[0104] The TOF image data acquisition unit 144 acquires two pieces of range image data in a fisheye image format (step S401).

[0105] The TOF image data acquisition unit 144 converts the two TOF image data in the fisheye image format into an equirectangular image format (step S402). As described above, the equirectangular image format is a method capable of expressing a celestial sphere image. In this step S402, the two TOF image data are converted into the equirectangular image format based on the same coordinate system, which facilitates image combination in the next step S403.

[0106] The TOF image data acquisition unit 144 combines the two TOF image data generated in step S402 to generate one piece of spherical TOF image data (step S403). The two input TOF image data cover an area with a full angle of view of more than 180 degrees. Therefore, the spherical TOF image data generated by appropriately connecting the two TOF image data can cover the spherical area.

[0107] The combining process in step S403 can use existing techniques for joining multiple images, and there are no particular limitations on the method.

[0108] FIG. 7 is a flowchart showing a process for determining a nearby object in the embodiment.

[0109] FIG. 7 is a flowchart showing the process of determining whether or not a nearby object is captured in a captured image, which corresponds to step S9 described in FIG.

[0110] The determination unit 160 determines whether or not there is a pixel whose charged amount is saturated in the TOF image data of the celestial sphere, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S801).

[0111] When there is a pixel whose charge amount is saturated in step S801, the determination unit 160 determines, based on the RGB image data of the celestial sphere acquired from the RGB image data acquisition unit 142, whether or not the charge amount is saturated for a pixel that has the same coordinates as the pixel whose charge amount is saturated in step S801, among the RGB image data of the celestial sphere (step S802).

[0112] If the amount of stored power is saturated in step S802, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S801 is due to external light, and outputs error information to the display control unit 170. Based on the error information acquired from the determination unit 160, the display control unit 170 superimposes the error information on the two-dimensional image information and displays it on the display unit 20 (step S803).

[0113] If the amount of stored power is not saturated in step S802, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S801 is due to the presence of a nearby object, and outputs coordinate position information of the pixel whose amount of stored power is saturated in step S801 to the display control unit 170. Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 superimposes identification information for identifying the nearby object on the two-dimensional image information and displays it on the display unit 20 (step S804).

[0114] When there is no pixel whose charge amount is saturated in step S801, the determination unit 160 determines whether or not there is a pixel indicating distance information of 0.5 m or less in the TOF image data of the celestial sphere, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S805).

[0115] If there is no pixel indicating distance information of 0.5 m or less in step S805, the determination unit 160 ends the process.

[0116] If there is a pixel showing distance information of 0.5 m or less in step S805, the determination unit 160 proceeds to the above-mentioned step S804, determines that the pixel showing distance information of 0.5 m or less in step S805 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel showing distance information of 0.5 m or less in step S805 to the display control unit 170. Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 causes the display unit 20 to display the identification information identifying the nearby object by superimposing it on the two-dimensional image information.

[0117] As described above, when the display control unit 170 determines that a nearby object exists, it superimposes the identification information on the two-dimensional image information, and when it does not determine that a nearby object exists, it does not superimpose the identification information on the two-dimensional image information.

[0118] That is, the display control unit 170 causes the display unit 20 to display differently depending on the presence or absence of a nearby object.

[0119] Furthermore, based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 causes the display unit 20 to display identification information for identifying the nearby object superimposed on the two-dimensional image information.

[0120] That is, the display control unit 170 causes the display unit 20 to display a different position depending on the position of the nearby object.

[0121] FIG. 8 is a diagram for explaining the display content of the display unit in the embodiment.

[0122] FIG. 8 is an explanatory diagram corresponding to step S2 shown in FIG. 5 and steps S803 and S804 shown in FIG.

[0123] On the display unit 20, the two-dimensional image information G is displayed by the display control unit 170. In addition, on the display unit 20, the display control unit 170 displays identification information G1 and G2 for identifying nearby objects and error information G3, each of which is superimposed on the two-dimensional image information G.

[0124] As described above, the imaging device 1 comprises an imaging unit 11 that images a target, a projection unit 12 that projects light onto the target, a distance information acquisition unit 13 that receives light reflected from the target, and a display control unit 170 that causes the display unit 20 to display different images depending on the presence or absence of a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

[0125] This allows the photographer to accurately check whether or not the photographer himself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the effects of external light.

[0126] The imaging device 1 includes a display unit 20. This allows the photographer to reliably check whether or not a nearby object appears in the captured image.

[0127] The display control unit 170 causes the display unit 20 to display a different position depending on the position of the nearby object, thereby enabling the photographer to confirm the position of the nearby object reflected in the captured image.

[0128] The display control unit 170 causes the display unit 20 to display the image information G captured by the imaging unit 11, and also causes the identification information G1, G2 for identifying nearby objects to be superimposed on the image information and displayed on the display unit 20. This allows the photographer to reliably confirm the position of the nearby object in the captured image.

[0129] The imaging device 1 includes a determination unit 160 that determines that there is a nearby object when the amount of charge stored by the light received by the distance information acquisition unit 13 is saturated and the amount of charge stored in the pixels of the imaging unit 11 is not saturated.

[0130] This allows the photographer to accurately check whether or not a nearby object is reflected in the captured image, distinguishing it from the effects of external light.

[0131] Fig. 9 is a diagram showing the external appearance of an imaging device according to a modified example of the embodiment, and Fig. 10 is a diagram showing the configuration of a processing block of a processing circuit in the modified example.

[0132] In this modification, display control unit 170 acquires RGB image data of the celestial sphere from RGB image data acquisition unit 142, and causes two-dimensional image information based on the acquired RGB image data of the celestial sphere to be displayed on display unit 520 of display device 500. Display unit 520 is an example of an output destination that outputs the two-dimensional image information.

[0133] This makes it possible to easily check from the 2D image information displayed on the display unit 520 whether the photographer himself / herself, a tripod, etc., is captured in the captured image, or whether 3D information of the desired layout has not been obtained.

[0134] The display control unit 170 outputs the two-dimensional image information to a display unit 520 that is different from the external device 300 to which the transmitting and receiving unit 180 outputs the three-dimensional information.

[0135] This makes it possible to check whether the photographer himself / herself, the tripod, etc. are reflected in the captured image, or whether 3D information of the desired layout has not been obtained, from the 2D image information output to the display unit 520, which is different from the external device 300, without checking the 3D information output to the external device 300.

[0136] The display control unit 170 may obtain three-dimensional data of the entire celestial sphere from the three-dimensional reconstruction processing unit 145, and display the three-dimensional information on the display unit 520. Specifically, the display control unit 170 may select, in accordance with a predetermined condition, whether to display two-dimensional image information on the display unit 520 or to display three-dimensional information on the display unit 520. This allows the display control unit 170 to output the two-dimensional image information separately from the three-dimensional information.

[0137] Based on the error information acquired from the determination unit 160, the display control unit 170 causes the display unit 520 to display the error information superimposed on the two-dimensional image information.

[0138] The display control unit 170 superimposes identification information for identifying the nearby object on the two-dimensional image information based on the coordinate position information of the pixel acquired from the determination unit 160, and causes the display unit 520 to display the two-dimensional image information.

[0139] That is, the display control unit 170 causes the display unit 520 to display differently depending on the presence or absence of a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

[0140] This allows the photographer to accurately check whether or not the photographer himself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the effects of external light.

[0141] The display control unit 170 causes the display unit 520 to display a different position depending on the position of the nearby object, thereby enabling the photographer to confirm the position of the nearby object reflected in the captured image.

[0142] The display control unit 170 causes the display unit 520 to display the image information captured by the imaging unit 11, and also causes the display unit 520 to display identification information for identifying the nearby object superimposed on the image information. This allows the photographer to reliably confirm the position of the nearby object in the captured image.

[0143] Fig. 11 is a diagram showing the external appearance of an imaging device according to a second modified example of the embodiment of the present invention, Fig. 12 is a diagram showing the configuration of a processing block of a processing circuit in the second modified example.

[0144] In a second modified example shown in Fig. 11, the imaging device 1 includes a plurality of display units 20A, 20a instead of the display unit 20 shown in Fig. 1. The display units 20A, 20a are configured with LEDs or the like, and blink or light up in response to an output signal from the processing circuit 14.

[0145] The display unit 20a is provided on a first surface on the front side of the housing 10, and the display unit 20A is provided on a second surface on the rear side of the housing 10.

[0146] In the second modified example shown in FIG. 12, the display control unit 170 causes information indicating the determination result acquired from the determination unit 160 to be displayed on the display units 20A and 20a.

[0147] Furthermore, the transmitting / receiving unit 180 transmits (outputs) the two-dimensional image information of the celestial sphere output from the RGB image data acquiring unit 142 to the display device 500 via the network 400. The display device 500 is an example of an output destination to which the two-dimensional image information is output.

[0148] That is, in the second modified example, in step S3 shown in FIG. 5, the transmission / reception unit 180 acquires RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and transmits (outputs) two-dimensional image information based on the acquired RGB image data of the entire celestial sphere to the display device 500.

[0149] The transmitting / receiving unit 510 of the display device 500 receives the two-dimensional image information transmitted from the transmitting / receiving unit 180 of the imaging device 1.

[0150] The control unit 530 of the display device 500 causes the two-dimensional image information received by the transmitting / receiving unit 510 to be displayed on the display unit 520 .

[0151] As described above, the imaging device 1 includes the imaging section 11 and the transmitting / receiving section 180 that outputs two-dimensional image information captured by the imaging section 11 separately from three-dimensional information.

[0152] This makes it possible to easily check from the 2D image information whether the photographer himself / herself, a tripod, etc., is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information.

[0153] Therefore, it becomes possible to re-acquire 3D information while remaining at the site where the 3D information is acquired, which reduces the effort of re-visiting the site where the 3D information is acquired compared to realizing, after leaving the site where the 3D information is acquired, that the photographer or the tripod, etc., is captured in the captured image or that 3D information with the desired layout has not been acquired.

[0154] The transmitting / receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before transmitting (outputting) the three-dimensional information in step S11. The transmitting / receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

[0155] This makes it possible to check from the 2D image information whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been acquired, before checking the 3D information.

[0156] The transmitting / receiving unit 180 transmits the two-dimensional image information to the display device 500, and the display device 500 causes the display unit 520 to display the two-dimensional image information.

[0157] This makes it possible to easily check from the 2D image information displayed on the display unit 520 whether the photographer himself / herself, a tripod, etc., is captured in the captured image, or whether 3D information of the desired layout has not been obtained.

[0158] The transmitting / receiving unit 180 transmits the two-dimensional image information to a display device 500 that is different from the external device 300 that outputs the three-dimensional information.

[0159] This makes it possible to check whether the photographer himself / herself, the tripod, etc. are reflected in the captured image, or whether 3D information of the desired layout has not been obtained, from the 2D image information output to the display unit 520 of the display device 500, which is different from the external device 300, without checking the 3D information output to the external device 300.

[0160] The transmitting / receiving unit 180 may transmit the three-dimensional information to the display device 500. Specifically, the transmitting / receiving unit 180 may select between transmitting the two-dimensional image information to the display device 500 and transmitting the three-dimensional information to the display device 500 in accordance with a predetermined condition. This allows the transmitting / receiving unit 180 to transmit the two-dimensional image information to the display device 500 separately from the three-dimensional information.

[0161] FIG. 13 is a flowchart showing a process for determining an adjacent object in the second modified example.

[0162] FIG. 13 is a flowchart showing the process of determining whether or not a nearby object is captured in a captured image, which corresponds to step S9 described with reference to FIG. 5 in the second modified example.

[0163] The determination unit 160 determines whether or not there is a pixel in which the amount of charged energy is saturated in the TOF image data of the celestial sphere, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S811).

[0164] When there is a pixel whose charge amount is saturated in step S811, the determination unit 160 determines, based on the RGB image data of the celestial sphere acquired from the RGB image data acquisition unit 142, whether or not the charge amount is saturated for a pixel that has the same coordinates as the pixel whose charge amount is saturated in step S811, among the RGB image data of the celestial sphere (step S812).

[0165] If the amount of stored power is saturated in step S812, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S811 is due to external light, and outputs error information to the display control unit 170. The display control unit 170 causes the display units 20A and 20a to display the error information based on the error information acquired from the determination unit 160 (step S813).

[0166] If the amount of stored power is not saturated in step S812, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S811 is due to the presence of a nearby object, and outputs coordinate position information of the pixel whose amount of stored power is saturated in step S811 to the display control unit 170. The display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 based on the coordinate position information of the pixel acquired from the determination unit 160 (step S814).

[0167] When there is no pixel whose charge amount is saturated in step S811, the determination unit 160 determines whether or not there is a pixel indicating distance information of 0.5 m or less in the TOF image data of the celestial sphere, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S815).

[0168] If there is no pixel showing distance information of 0.5 m or less in step S815, the determination unit 160 ends the process.

[0169] If there is a pixel showing distance information of 0.5 m or less in step S815, the determination unit 160 proceeds to the above-mentioned step S814, determines that the pixel showing distance information of 0.5 m or less in step S815 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel showing distance information of 0.5 m or less in step S815 to the display control unit 170. The display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 based on the pixel coordinate position information acquired from the determination unit 160.

[0170] If it is determined in step S814 that the display is on the front side, the display control unit 170 causes the display unit 20a arranged on the front side of the housing 10 to blink (step S816).

[0171] If it is not determined in step S814 that the display is on the front side, the display control unit 170 causes the display unit 20A arranged on the back side of the housing 10 to blink (step S817).

[0172] As described above, when the display control unit 170 determines that a nearby object is present, it blinks the display unit 20a or the display unit 20A, and when it does not determine that a nearby object is present, it does not blink the display unit 20a or the display unit 20A.

[0173] That is, the display control unit 170 causes the display unit 20a and the display unit 20A to display different images depending on whether or not a nearby object is present.

[0174] This allows the photographer to accurately check whether or not the photographer himself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the effects of external light.

[0175] Furthermore, the display control unit 170 causes the display unit 20a or the display unit 20A to blink based on the coordinate position information of the pixel acquired from the determination unit 160.

[0176] That is, the display control unit 170 causes the display unit 20a and the display unit 20A to display different positions depending on the position of the nearby object, thereby enabling the photographer to confirm the position of the nearby object reflected in the captured image.

[0177] Then, the display control unit 170 causes one of the display units 20A and 20a, which is closer to the nearby object, to display differently depending on the presence or absence of the nearby object. This allows the photographer to reliably confirm the position of the specific object in the captured image.

[0178] FIG. 14 is a diagram for explaining the configuration of an imaging device according to a third modified example of the embodiment of the present invention.

[0179] In a third modified example shown in FIG. 14, the imaging device 1 includes, in addition to the configuration shown in FIG. 2, another imaging section 111 having other imaging elements 111a, 111A, other fisheye lenses (wide-angle lenses) 111b, 111B, etc.

[0180] In the third modified example, the RGB imaging unit 11 and the other imaging unit 111 are provided on the same baseline. In this case, multi-eye processing becomes possible in the processing circuit 14. That is, RGB images from two viewpoints can be obtained by simultaneously driving the imaging unit 11 and the other imaging unit 111, which are provided at a predetermined distance apart on one surface. This makes it possible to use parallax calculated based on the two RGB images, and further improves the distance accuracy over the entire measurement range.

[0181] Specifically, when an RGB imaging unit 11 and another imaging unit 111 are provided, it becomes possible to use multi-baseline stereo (MSB) using SSSD and EPI processing, as in conventional parallax calculations. Therefore, by using this, the reliability of parallax increases, making it possible to achieve high spatial resolution and accuracy.

[0182] As described above, the imaging device 1 is equipped with another imaging section 111, and the 3D reconstruction processing section 150 determines 3D information based on the output of the distance information acquisition section 13, the 2D image information, and the other 2D image information captured by the other imaging section 111.

[0183] The imaging device 1 may also be provided with another imaging unit 111 and a three-dimensional information determination unit that determines three-dimensional information based on the two-dimensional image information and other two-dimensional image information captured by the other imaging unit 111, regardless of the output of the distance information acquisition unit 13.

[0184] This makes it possible to check from the two-dimensional image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image or whether three-dimensional information of the desired layout has not been obtained, without checking the three-dimensional information determined by the three-dimensional reconstruction processing unit 150 based on the two-dimensional image information.

[0185] As described above, the imaging device 1 (an example of an information processing device) according to one embodiment of the present invention includes an imaging unit 11 that outputs three-dimensional information determined based on received light, and a display control unit 170 (an example of an output unit) or a transmission / reception unit 180 (an example of an output unit) that outputs two-dimensional image information G captured by the imaging unit 11 separately from the three-dimensional information.

[0186] This makes it possible to easily check from the 2D image information whether the photographer himself / herself, a tripod, etc., is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information.

[0187] Therefore, it becomes possible to re-acquire 3D information while remaining at the site where the 3D information is acquired, which reduces the effort of re-visiting the site where the 3D information is acquired compared to realizing, after leaving the site where the 3D information is acquired, that the photographer or the tripod, etc., is captured in the captured image or that 3D information with the desired layout has not been acquired.

[0188] Before outputting the three-dimensional information, the display control unit 170 or the transmission / reception unit 180 outputs the two-dimensional image information G. Before determining the three-dimensional information, the display control unit 170 or the transmission / reception unit 180 outputs the two-dimensional image information G.

[0189] This makes it possible to check from the 2D image information whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been acquired, before checking the 3D information.

[0190] The display control unit 170 causes the display units 20 and 520 to display the two-dimensional image information G. The imaging device 1 includes the display unit 20.

[0191] This makes it possible to easily check from the 2D image information displayed on the display unit whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained.

[0192] The display control unit 170 or the transmission / reception unit 180 outputs the two-dimensional image information G to the display unit 20, 520 (an example of an output destination) different from the external device 300 (an example of an output destination) that outputs the three-dimensional information.

[0193] This makes it possible to check whether the photographer himself / herself, the tripod, etc. are reflected in the captured image, or whether 3D information of the desired layout has not been obtained, from the 2D image information output to a display unit 20, 520 other than the external device 300 (an example of an output destination), without checking the 3D information output to the external device 300.

[0194] Imaging device 1 includes projection unit 12 that projects light onto an object, distance information acquisition unit 13 (an example of a light receiving unit) that receives light reflected from the object, and 3D reconstruction processing unit 150 (an example of a 3D information determination unit) that determines 3D information based on the output of distance information acquisition unit 13. 3D reconstruction processing unit 150 determines the 3D information based on the output of distance information acquisition unit 13 and 2D image information.

[0195] This makes it possible to check from the 2D image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information determined by the 3D reconstruction processing unit 150.

[0196] The imaging device 1 is equipped with another imaging section 111, and the 3D reconstruction processing section 150 determines 3D information based on the output of the distance information acquisition section 13, the 2D image information, and other 2D image information captured by the other imaging section 111.

[0197] The imaging device 1 may also be provided with another imaging unit 111 and a three-dimensional information determination unit that determines three-dimensional information based on the two-dimensional image information and other two-dimensional image information captured by the other imaging unit 111, regardless of the output of the distance information acquisition unit 13.

[0198] This makes it possible to check from the two-dimensional image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image or whether three-dimensional information of the desired layout has not been obtained, without checking the three-dimensional information determined by the three-dimensional reconstruction processing unit 150 based on the two-dimensional image information.

[0199] The three-dimensional information includes spherical three-dimensional information. In this case, even in the case of spherical three-dimensional information in which it is difficult to confirm that the photographer himself / herself, a tripod, etc. are included in the captured image or that three-dimensional information of a desired layout has not been acquired, it becomes possible to easily confirm from the two-dimensional image information captured by the imaging unit 11 that the photographer himself / herself, a tripod, etc. are included in the captured image or that three-dimensional information of a desired layout has not been acquired. [Explanation of symbols]

[0200] 1. Imaging device (an example of an information processing device) 10. Chassis 11 Imaging unit 11a, 11A Image sensor 11b, 11B Fisheye Lens 12 Projection section 12a, 12A light source section 12b, 12B wide-angle lens 13 Distance information acquisition unit (an example of a light receiving unit) 13a, 13A TOF sensor 13b, 13B Wide-angle lens 14 Processing circuit 15 Shooting switch 20 Display section 20A, 20a display section 111 Other imaging units 150 3D reconstruction processing unit (an example of a 3D information determination unit) 160 Judgment Department 170 Display control unit (an example of an output unit) 180 Transmitter / receiver (an example of an output unit) 300 External device (example of output destination) 500 Display device (example of output destination) 520 Display unit (example of output destination) L Synchronous signal line [Prior art documents] [Patent documents]

[0201] [Patent Document 1] JP 2018-077071 A [Patent Document 2] Patent No. 5423287

Claims

1. An information processing device that outputs three-dimensional information determined based on received light, An information processing device comprising: an imaging unit; and an output unit that outputs two-dimensional image information captured by the imaging unit separately from the three-dimensional information.

2. The information processing apparatus according to claim 1 , wherein the output unit outputs the two-dimensional image information before outputting the three-dimensional information.

3. The information processing apparatus according to claim 2 , wherein the output unit outputs the two-dimensional image information before determining the three-dimensional information.

4. 4. The information processing apparatus according to claim 1, wherein the output unit causes a display unit to display the two-dimensional image information.

5. 5. An information processing apparatus according to claim 4, further comprising the display unit.

6. 6. The information processing apparatus according to claim 1, wherein the output unit outputs the two-dimensional image information to an output destination different from an output destination to which the three-dimensional information is output.

7. A projection unit that projects light onto a target; A light receiving unit that receives the light reflected from the object; 7. The information processing apparatus according to claim 1, further comprising: a three-dimensional information determining section that determines the three-dimensional information based on an output from the light receiving section.

8. The information processing apparatus according to claim 7 , wherein the three-dimensional information determination section determines the three-dimensional information based on an output from the light receiving section and the two-dimensional image information.

9. Another imaging unit is provided, 9. The information processing apparatus according to claim 8, wherein the three-dimensional information determination section determines the three-dimensional information based on an output from the light receiving section, the two-dimensional image information, and other two-dimensional image information captured by the other imaging section.

10. Other imaging units, 10. The information processing apparatus according to claim 1, further comprising: a three-dimensional information determination section that determines the three-dimensional information based on the two-dimensional image information and other two-dimensional image information captured by the other imaging section.

11. The information processing device according to claim 1 , wherein the three-dimensional information includes spherical three-dimensional information.

12. An imaging step; a three-dimensional information output step of outputting three-dimensional information determined based on the received light; a two-dimensional image information output step for outputting two-dimensional image information captured by the imaging step, separately from the three-dimensional information output step; An information processing method comprising:

Images