Information processing apparatus and information processing method

Abstract

To provide an information processing apparatus that allows a photographer to easily check, from a three-dimensional image, the cause of non-acquisition of desired three-dimensional information.SOLUTION: An imaging apparatus and a display device each comprise a display control unit that displays, on a display 520, a three-dimensional image 3G determined on the basis of output from a distance information acquisition unit, which is an example of a light receiving unit that receives light projected on an object and reflected from the object. The display control unit displays, on the display, a display image including the three-dimensional image and pieces of position identification information 3Ga, 3Gb, and 3Gc for identifying positions, on the basis of, in the three-dimensional image, a distant object distant from the distance information acquisition unit when receiving the light reflected from the object, a low reflectance object having low reflectance to the projected light, or position information indicating a position determined to be at least one of blind spots relative to the distance information acquisition unit when receiving the light reflected from the object.SELECTED DRAWING: Figure 22

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Patent Document 1 describes a distance measuring device that can stably and accurately measure the distance to an object.

[0003] Patent Document 2 describes an imaging device that performs image processing to reduce the influence of a finger or other object that is captured in the image.

[0004] Patent document 3 describes a three-dimensional synthesis processing system that has a measurement position presentation unit that extracts blocks where the density of measurement data is lower than a predetermined threshold and presents coordinates within the range of the extracted blocks as a presented measurement position, which is the position where a three-dimensional measurement device should be installed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-077071 [Patent Document 2] Patent No. 5423287 [Patent Document 3] Patent No. 6192938 Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to provide an information processing device and an information processing method that can easily check, using a three-dimensional image, the cause of why desired three-dimensional information has not been acquired. [Means for solving the problem]

[0007] The information processing device of the present invention includes a display control unit that causes a display unit to display a three-dimensional image determined based on the output of a light receiving unit that receives light projected onto an object and reflected from the object, and the display control unit causes the display unit to display a display image including position identification information that identifies the position of at least one of a distant object, a low-reflecting object, and a blind spot, and the three-dimensional image, based on position information indicating a position in the three-dimensional image that is determined to be at least one of a distant object that is far away from the light receiving unit when the light reflected from the object is received, a low-reflecting object that has a low reflectivity to the projected light, and a blind spot for the light receiving unit when the light reflected from the object is received. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an information processing device and an information processing method that can easily check, using a three-dimensional image, the cause of why desired three-dimensional information has not been acquired. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the appearance of an imaging device according to an embodiment of the present invention. [Figure 2] 2 is a diagram illustrating a configuration of an imaging device according to the embodiment. FIG. [Figure 3] 2A and 2B are diagrams for explaining a usage situation of the imaging device in the embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a processing block of a processing circuit according to the embodiment. [Figure 5] 10 is a flowchart showing an example of the operation of a processing circuit of the imaging device according to the embodiment. [Figure 6] FIG. 10 is a flowchart showing a generation of spherical image data in the embodiment. [Figure 7] FIG. 10 is a flowchart illustrating a nearby object determination process according to the embodiment. [Figure 8] 3A and 3B are diagrams for explaining the display content of the display unit in the embodiment. [Figure 9] FIG. 10 is a diagram showing the appearance of an imaging device according to a modified example of the embodiment. [Figure 10]FIG. 10 is a diagram showing a configuration of a processing block of a processing circuit according to a modified example. [Figure 11] FIG. 10 is a diagram showing the appearance of an imaging device according to a second modified example of the embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing a configuration of a processing block of a processing circuit in a second modified example. [Figure 13] FIG. 10 is a flowchart showing a nearby object determination process according to a second modified example. [Figure 14] FIG. 10 is a diagram illustrating the configuration of an imaging device according to a third modified example of the embodiment of the present invention. [Figure 15] FIG. 10 is a flowchart for determining whether a highly reflective object is present in the embodiment of the present invention. [Figure 16] FIG. 10 is a flowchart for determining whether a distant object or a low-reflecting object is present in the embodiment. [Figure 17] FIG. 10 is a flowchart showing a determination of image blur in the embodiment. [Figure 18] FIG. 10 is a decision flow diagram according to a fourth modified example of the embodiment of the present invention. [Figure 19] FIG. 10 is a diagram illustrating an example of the configuration of a processing block of a processing circuit according to a fifth modified example of the embodiment of the present invention. [Figure 20] FIG. 10 is a diagram illustrating an example of a configuration of an information processing system according to a sixth modified example of the embodiment of the present invention. [Figure 21] FIG. 13 is a diagram illustrating an example of a configuration of an information processing system according to a seventh modified example of the embodiment of the present invention. [Figure 22] FIG. 10 is a diagram for explaining the display contents of the display unit in the fifth to seventh modified examples. [Figure 23] 3A and 3B are diagrams for explaining a three-dimensional image displayed by a display unit according to an embodiment of the present invention. [Figure 24] FIG. 10 is a decision flow diagram in the fifth to seventh modified examples. [Figure 25] FIG. 10 is another diagram for explaining the display contents of the display unit in the fifth to seventh modified examples. [Figure 26] FIG. 10 is a flowchart illustrating the processing in the fifth to seventh modified examples. [Figure 27]FIG. 10 is another flowchart illustrating the processing in the fifth to seventh modified examples. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of an imaging device and an imaging processing method will be described in detail with reference to the accompanying drawings.

[0011] Fig. 1 is a diagram showing an example of the appearance of an imaging device according to an embodiment of the present invention. Fig. 2 is a diagram for explaining the configuration of the imaging device. Fig. 2 shows the internal configuration of the imaging device of Fig. 1.

[0012] The imaging device 1 is an example of an information processing device that outputs three-dimensional information determined based on received light, and has an imaging unit (camera) 11, a projection unit (corresponding to the light-emitting unit of a distance sensor) 12 that projects light other than visible light, and a distance information acquisition unit (corresponding to the light-receiving unit of a distance sensor) 13 that acquires distance information based on the light projected by the projection unit 12, all of which are provided integrally with a housing 10. Each unit is electrically connected to a processing circuit 14 inside the housing 10 by a synchronization signal line L, and operates synchronously with each other.

[0013] The photographing switch 15 is used by the user to input a photographing instruction signal to the processing circuit 14. The display unit 20 displays content according to the output signal of the processing circuit 14 and is configured with a liquid crystal screen or the like. The display unit 20 may be configured with a touch panel or the like and be configured to accept user operation input. Based on the photographing instruction, the processing circuit 14 controls each unit to acquire RGB image and distance information data, and performs processing to reconstruct the acquired distance information data into high-density 3D point cloud data based on the RGB image and distance information data.

[0014] It is possible to construct 3D point cloud data by using distance information data as is, but in that case, the accuracy of the 3D point cloud data is limited by the number of pixels (resolution) of the distance information acquisition unit 13. This example also shows the process of reconstructing it into high-density 3D point cloud data. The reconstructed data is output to an external PC or the like via a portable recording medium or communication, and is used to display the 3D reconstruction model.

[0015] Power is supplied to each unit and the processing circuit 14 from a battery housed inside the housing 10. Alternatively, power may be supplied from outside the housing 10 via a connection cord.

[0016] The imaging unit 11 captures two-dimensional image information and includes imaging elements 11a and 11A, fisheye lenses (wide-angle lenses) 11b and 11B, etc. The projection unit 12 includes light source units 12a and 12A, wide-angle lenses 12b and 12B, etc. The distance information acquisition unit 13 includes TOF (Time Of Flight) sensors 13a and 13A, wide-angle lenses 13b and 13B, etc. Although not shown, each unit may constitute an optical system such as a prism or a group of lenses.

[0017] For example, the imaging unit 11 may be configured with an optical system for focusing light collected by fisheye lenses 11b and 11B on the imaging elements 11a and 11A. The projection unit 12 may be configured with an optical system for guiding light from the light source units 12a and 12A to the wide-angle lenses 12b and 12B. The distance information acquisition unit 13 may be configured with an optical system for focusing light collected by the wide-angle lenses 13b and 13B on the TOF sensors 13a and 13A. The optical systems may be appropriately determined depending on the configurations and arrangements of the imaging elements 11a and 11A, the light source units 12a and 12A, the TOF sensors 13a and 13A, etc., and the optical systems such as prisms and lens groups will not be described here.

[0018] Image sensors 11a and 11A, light source units 12a and 12A, and TOF sensors 13a and 13A are housed integrally inside housing 10. Fisheye lens 11b, wide-angle lens 12b, wide-angle lens 13b, and display unit 20 are each provided on a first surface on the front side of housing 10. On the first surface, the inner ranges of fisheye lens 11b, wide-angle lens 12b, and wide-angle lens 13b are each open.

[0019] Fisheye lens 11B, wide-angle lens 12B, wide-angle lens 13B, and shooting switch 15 are each provided on a second surface on the rear side of housing 10. On the second surface, the inner ranges of fisheye lens 11B, wide-angle lens 12B, and wide-angle lens 13B are each open.

[0020] The imaging elements 11a and 11A are two-dimensional resolution image sensors (area sensors). The imaging elements 11a and 11A have an imaging area in which a large number of light-receiving elements (photodiodes) for each pixel are arranged in a two-dimensional direction. The imaging area is provided with R (Red), G (Green), and B (Blue) color filters in a Bayer array or the like to receive visible light, and light that passes through the color filters is stored in the photodiodes. Here, an image sensor with a large number of pixels is used so that a wide-angle (for example, a range of a hemisphere of 180 degrees around the imaging direction as shown in FIG. 2) two-dimensional image can be acquired with high resolution.

[0021] The image sensors 11a and 11A convert the light focused on their imaging areas into electrical signals using the pixel circuits of each pixel, and output a high-resolution RGB image. The fisheye lenses 11b and 11B collect light from a wide angle (for example, a hemispherical range of 180 degrees around the imaging direction as shown in Figure 2) and focus the light on the imaging areas of the image sensors 11a and 11A.

[0022] The light source units 12a and 12A are semiconductor lasers that emit laser light in a wavelength band other than the visible light range used for distance measurement (infrared light is used here as an example). The light source units 12a and 12A may use one semiconductor laser or a combination of multiple semiconductor lasers. Furthermore, the semiconductor laser may be a surface-emitting semiconductor laser such as a VCSEL (Vertical Cavity Surface Emitting Laser).

[0023] Alternatively, the light from the semiconductor laser may be shaped by an optical lens to be elongated vertically, and the elongated light may be scanned in one dimension of the measurement range by an optical deflection element such as a MEMS (Micro Electro Mechanical Systems) mirror. In this embodiment, the light source units 12a and 12A are configured to widen the light from the semiconductor laser LA to a wide angle range via wide-angle lenses 12b and 12B without using an optical deflection element such as a MEMS mirror.

[0024] The wide-angle lenses 12b and 12B of the light source units 12a and 12A have the function of expanding the light emitted by the light source units 12a and 12A to a wide-angle range (for example, a hemispherical range of 180 degrees around the imaging direction shown in Figure 2).

[0025] The wide-angle lenses 13b and 13B of the distance information acquisition unit 13 capture reflected light from the light source units 12a and 12A projected by the projection unit 12 from each direction within a wide-angle measurement range (e.g., a 180-degree hemispherical range around the imaging direction as shown in FIG. 2 ), and form an image of that light on the light-receiving area of ​​the TOF sensors 13a and 13A. The measurement range includes one or more projection targets (e.g., buildings), and light reflected from the projection targets (reflected light) enters the wide-angle lenses 13b and 13B. The reflected light may be captured by, for example, providing a filter on the entire surface of the wide-angle lenses 13b and 13B that cuts out light with wavelengths in the infrared range or longer. However, this is not limiting; as long as light in the infrared range enters the light-receiving area, a means for transmitting light with wavelengths in the infrared range, such as a filter, may be provided on the optical path from the wide-angle lenses 13b and 13B to the light-receiving area.

[0026] The TOF sensors 13a and 13A are optical sensors with two-dimensional resolution. The TOF sensors 13a and 13A have a light-receiving area in which a large number of light-receiving elements (photodiodes) are arranged in a two-dimensional direction. In this sense, they can be called "second image-receiving means." The TOF sensors 13a and 13A receive reflected light from each area (each area is also called a position) in the measurement range using the light-receiving elements corresponding to each area, and measure (calculate) the distance to each area based on the light detected by each light-receiving element.

[0027] In this embodiment, distance is measured using a phase difference detection method. In this method, a laser beam amplitude-modulated at a fundamental frequency is irradiated onto the measurement area, and the reflected light is received. The phase difference between the irradiated light and the reflected light is measured to determine the time, and the distance is calculated by multiplying this time by the speed of light. The advantage of this method is that it can be expected to have a certain degree of resolution.

[0028] The TOF sensors 13a and 13A are driven in synchronization with the irradiation of light by the projection unit 12, calculate the distance corresponding to each pixel from the phase difference with the reflected light at each light receiving element (corresponding to a pixel), and output distance information image data (hereinafter also referred to as a "distance image" or "TOF image") in which pixel information is associated with information indicating the distance to each area within the measurement range. The TOF sensors 13a and 13A output phase information image data in which phase information is associated with pixel information, and may acquire distance information image data based on the phase information image data in post-processing.

[0029] The number of areas into which the measurement range can be divided is determined by the resolution of the light-receiving area. Therefore, if a low-resolution sensor is used to reduce size, the number of pixel information pieces in the range image data will decrease, and the number of 3D point clouds will also decrease.

[0030] Alternatively, distance may be measured using a pulse method instead of the phase difference detection method. In this case, for example, the light source units 12a and 12A emit an irradiation pulse P1, which is an ultrashort pulse having a rise time of several nanoseconds (ns) and a high optical peak power, and the TOF sensors 13a and 13A measure the time (t) required to receive a reflected pulse P2, which is the reflected light of the irradiation pulse P1 emitted by the light source units 12a and 12A, in synchronization with the irradiation pulse P1.

[0031] When this method is adopted, for example, TOF sensors 13a and 13A are used that have a circuit for measuring time mounted on the output side of the light receiving element. Each circuit converts the time it takes for each light receiving element, from when the light source unit 12a or 12A emits the irradiation pulse P1 to when it receives the reflected pulse P2, into a distance, and obtains the distance to each area.

[0032] This method is suitable for widening the angle of the imaging device 1 because it can output powerful light using peak light. Also, if a MEMS mirror or the like is used to deflect (scan) the light, powerful light can be emitted over a long distance while suppressing its spread, which leads to an increase in the measurement distance. In this case, the laser light emitted from the light source units 12a and 12A is positioned so that it is scanned (deflected) by the MEMS mirror toward the wide-angle lenses 12b and 12B.

[0033] It is desirable that the effective angle of view of imaging unit 11 and the effective angle of view of distance information acquisition unit 13 are the same, for example, at 180 degrees or more, but they do not necessarily have to be the same. If necessary, the effective angle of view of imaging unit 11 and the effective angle of view of distance information acquisition unit 13 may each be reduced. In this embodiment, imaging unit 11 and distance information acquisition unit 13 reduce the effective pixels to within a range of, for example, 100 degrees to 180 degrees so that the imaging device 1 main body, distance information acquisition unit 13, etc. are not included in the angle of view.

[0034] Furthermore, the resolution of the TOF sensors 13a and 13A may be set lower than the resolution of the image sensors 11a and 11A, prioritizing the miniaturization of the imaging device 1. By setting the resolution of the TOF sensors 13a and 13A lower than that of the image sensors 11a and 11A, it is possible to suppress an increase in the size of the light receiving area, which leads to a miniaturization of the imaging device 1. As a result, the TOF sensors 13a and 13A have a low resolution, and the 3D point clouds obtained by the TOF sensors 13a and 13A have a low density, but the provision of the processing circuit 14, which is an "acquisition means," allows them to be converted into a high-density 3D point cloud. The process of converting into a high-density 3D point cloud in the processing circuit 14 will be described later.

[0035] In the present embodiment, as an example, the image sensor 11a, the light source unit 12a, and the TOF sensor 13a are arranged so as to be aligned in a straight line in the longitudinal direction of the housing 10. Similarly, the image sensor 11A, the light source unit 12A, and the TOF sensor 13A are arranged so as to be aligned in a straight line in the longitudinal direction of the housing 10. Below, an example of the image sensor 11a, the light source unit 12a, and the TOF sensor 13a will be described.

[0036] The imaging area (imaging surface) of the imaging element 11a and the light receiving area (light receiving surface) of the TOF sensor 13a may be arranged to face a direction perpendicular to the longitudinal direction as shown in FIG. 2, or may be arranged to face the longitudinal direction by providing a prism or the like that changes the linear direction (optical path) of light by 90 degrees before making it incident. They may also be arranged in any other direction depending on the configuration. In other words, the imaging element 11a, light source unit 12a, and TOF sensor 13a are arranged so that they cover the same measurement range. The imaging unit 11, projection unit 12, and distance information acquisition unit 13 are arranged from one side of the housing 10, facing the measurement range.

[0037] In this case, it is sufficient that the image sensor 11a and the TOF sensor 13a are arranged on the same baseline so as to achieve parallel stereo. By arranging them so as to achieve parallel stereo, it becomes possible to obtain parallax data using the output of the TOF sensor 13a even with only one image sensor 11a. The light source unit 12a is configured so as to be able to irradiate light onto the measurement range of the TOF sensor 13a.

[0038] (processing circuit) Next, the processing of the processing circuit 14 will be described. The TOF images obtained only by the TOF sensors 13a and 13A have low resolution as they are. For this reason, this example shows an example in which the resolution is increased by the processing circuit 14 to reconstruct high-density 3D point cloud data. Note that some or all of the following processing performed by the processing circuit 14 as "information processing means" may be performed by an external device.

[0039] As described above, the three-dimensional point cloud data reconstructed by the imaging device 1 is output to an external device such as a PC via a portable recording medium or communication, and is used to display a three-dimensional reconstruction model.

[0040] This makes it possible to provide an imaging device 1 that is faster, smaller, and lighter, and therefore more portable, than when the imaging device 1 itself displays the 3D reconstruction model.

[0041] However, after leaving the site where the 3D information was acquired and restoring the 3D information using an external device, it may be noticed that the photographer or the tripod is reflected in the captured image, or that the 3D information was not acquired in the desired layout. In such cases, it is necessary to go through the trouble of revisiting the site where the 3D information was acquired.

[0042] One solution to this problem would be to bring an external device such as a PC to the site, but this would eliminate the benefits of higher speed, smaller size, and lighter weight.

[0043] It is also possible to transmit the acquired 3D information to an external device via a communication line and receive the restored 3D information, but this would eliminate the benefit of high speed, and in addition, 3D information contains a large amount of information, making it difficult to visually confirm whether the photographer, the tripod, etc. are reflected in the captured image.

[0044] In particular, in the case of spherical 3D information, it is extremely difficult to visually check whether the photographer himself, the tripod, etc. are reflected in the captured image.

[0045] In view of the above-mentioned problems, the present embodiment aims to provide an imaging device 1 that can easily check in real time whether the photographer himself or herself, a tripod, etc., are captured in the captured image, or whether 3D information of the desired layout has not been acquired.

[0046] FIG. 3 is a diagram for explaining a usage state of the imaging device in the embodiment.

[0047] In the state shown in FIG. 3(a), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the spherical imaging range R, and the photographer M and the selfie stick 1A are not captured in the spherical captured image.

[0048] In the state shown in FIG. 3(b), the photographer M is included in the spherical imaging range R, and the photographer M appears in the spherical captured image.

[0049] In the state shown in FIG. 3(c), the tripod 1B supporting the imaging device 1 is included in the spherical imaging range R, and the tripod 1B appears in the captured image of the celestial sphere.

[0050] In the state shown in FIG. 3(d), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the spherical imaging range R, and the photographer M and the selfie stick 1A will not appear in the spherical captured image; however, due to strong external light (such as the sun or lighting), there is a possibility that the photographer M and the selfie stick 1A may be erroneously determined to be in the image.

[0051] Furthermore, in the situations shown in Figures 3(b) and 3(c), the colors and types of objects reflected in the image, as well as their appearance, vary widely, making it difficult to uniformly determine whether or not an object is reflected.

[0052] In the above situation, when determining whether or not a specific object (nearby object) such as the photographer himself or a tripod is present based on the distance information image data output from the TOF sensors 13a and 13A, it is difficult to distinguish whether a specific object is actually present or whether the external light is too strong.

[0053] In other words, when the charge storage amount of a particular pixel of the TOF sensors 13a and 13A is saturated, it is difficult to distinguish from the output of the TOF sensors 13a and 13A alone whether this is due to the presence of a particular object or to the intensity of external light being too strong.

[0054] In view of the above-mentioned problems, another object of this embodiment is to provide an imaging device 1 that can accurately check whether or not a specific object, such as the photographer himself or herself or a tripod, is reflected in a captured image, while distinguishing it from the influence of external light. Another object of this embodiment is to be able to check that not only nearby objects but also highly reflective objects, distant objects, low reflective objects, image blur, and other objects are included in the image.

[0055] Fig. 4 is a diagram showing an example of the configuration of processing blocks of the processing circuit 14. The processing circuit 14 shown in Fig. 4 includes a control unit 141, an RGB image data acquisition unit 142, a monochrome processing unit 143, a TOF image data acquisition unit 144, a resolution enhancement unit 145, a matching processing unit 146, a reprojection processing unit 147, a semantic segmentation unit 148, a parallax calculation unit 149, a 3D reconstruction processing unit 150, a determination unit 160, a display control unit 170 which is an example of an output unit, and a transmission / reception unit 180 which is an example of an output unit. In Fig. 4, solid arrows indicate the flow of signals, and dashed arrows indicate the flow of data.

[0056] When the control unit 141 receives an ON signal (photography start signal) from the photography switch 15, it outputs synchronization signals to the image pickup elements 11a, 11A, the light source units 12a, 12A, and the TOF sensors 13a, 13A, thereby controlling the entire processing circuit 14. The control unit 141 first outputs a signal to the light source units 12a, 12A to instruct them to emit ultrashort pulses, and at the same timing outputs a signal to the TOF sensors 13a, 13A to instruct them to generate TOF image data. The control unit 141 then outputs a signal to the image pickup elements 11a, 11A to capture an image. Note that the image pickup at the image pickup elements 11a, 11A may be taken during the period when light is emitted from the light source units 12a, 12A, or during the period immediately before or after that period.

[0057] The RGB image data acquisition unit 142 acquires RGB image data captured by the image capture elements 11a and 11A and outputs RGB image data of the celestial sphere based on an image capture instruction from the control unit 141. The monochrome processing unit 143 performs processing to align data types for matching processing with TOF image data obtained from the TOF sensors 13a and 13A. In this example, the monochrome processing unit 143 performs processing to convert the RGB image data of the celestial sphere into a monochrome image of the celestial sphere.

[0058] The TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on an instruction to generate TOF image data from the control unit 141, and outputs TOF image data of the entire celestial sphere.

[0059] The resolution increasing unit 145 regards the TOF image data of the celestial sphere as a monochrome image and increases the resolution thereof. Specifically, the resolution increasing unit 145 replaces the distance value associated with each pixel of the TOF image data of the celestial sphere with the value (grayscale value) of the monochrome image of the celestial sphere and uses the result. Furthermore, the resolution increasing unit 145 increases the resolution of the monochrome image of the celestial sphere to the resolution of the RGB image data of the celestial sphere obtained from the image sensors 11a and 11A. The conversion to high resolution is performed by, for example, performing a normal up-conversion process. As another conversion method, for example, multiple frames of continuously generated TOF image data of the celestial sphere may be acquired, and the distances of adjacent points may be added using the frames to perform super-resolution processing.

[0060] The matching processing unit 146 extracts feature amounts of textured portions of a monochrome image of the celestial sphere obtained by increasing the resolution of the TOF image data of the celestial sphere and a monochrome image of the celestial sphere obtained by increasing the resolution of the celestial sphere's RGB image data, and performs matching processing using the extracted feature amounts. For example, the matching processing unit 146 extracts edges from each monochrome image and performs matching processing between the extracted edge information. As an alternative method, the matching processing may be performed using a method that converts texture changes into feature amounts, such as SIFT. Here, the matching processing refers to searching for corresponding pixels.

[0061] A specific matching technique is block matching, which calculates the similarity between pixel values ​​extracted as an M x M (M is a positive integer) pixel block around a reference pixel and pixel values ​​extracted as an M x M pixel block around a pixel that is the center of the search in another image, and then determines the center pixel with the highest similarity as the corresponding pixel.

[0062] There are various ways to calculate similarity. For example, the formula for the normalized autocorrelation coefficient (NCC) can be used. The higher the normalized autocorrelation coefficient (NCC), the higher the similarity, and the value is 1 when the pixel values ​​of the blocks are perfectly matched.

[0063] Furthermore, since distance data for textureless regions can also be obtained from spherical TOF image data, weighting of the matching process may be performed depending on the region. For example, in the calculation of the formula for CNCC, weighting may be applied to areas other than edges (textureless regions).

[0064] Moreover, instead of the formula showing the NCC, a selective normalized correlation coefficient (SCC) or the like may be used.

[0065] The reprojection processing unit 147 performs a process of reprojecting the TOF image data of the celestial sphere, which indicates the distance to each position in the measurement range, onto the two-dimensional coordinates (screen coordinate system) of the imaging unit 11. Reprojection means determining at what coordinates in the images of the imaging elements 11a and 11A the three-dimensional points calculated by the TOF sensors 13a and 13A are shown. The TOF image data of the celestial sphere indicates the positions of the three-dimensional points in a coordinate system centered on the distance information acquisition unit 13 (mainly the wide-angle lenses 13b and 13B). Therefore, the three-dimensional points indicated by the TOF image data of the celestial sphere are reprojected onto a coordinate system centered on the imaging unit 11 (mainly the fisheye lenses 11b and 11B).

[0066] For example, the reprojection processing unit 147 translates the coordinates of three-dimensional points in the TOF image data of the celestial sphere to coordinates of three-dimensional points centered on the imaging unit 11, and after the translation, converts the coordinates of the three-dimensional points in the TOF image data of the celestial sphere into a two-dimensional coordinate system (screen coordinate system) indicated by the RGB image data of the celestial sphere. This causes the coordinates of the three-dimensional points in the TOF image data of the celestial sphere and the coordinates of the two-dimensional image information of the celestial sphere captured by the imaging unit 11 to correspond to each other. The reprojection processing unit 147 causes the coordinates of the three-dimensional points in the TOF image data of the celestial sphere and the coordinates of the two-dimensional image information of the celestial sphere captured by the imaging unit 11 to correspond to each other.

[0067] The parallax calculation unit 149 calculates the parallax at each position from the difference in distance to the corresponding pixel obtained by the matching process. Note that the parallax matching process uses the reprojection coordinates converted by the reprojection processing unit 147 to search for pixels surrounding the position of the reprojection coordinates, thereby shortening the processing time and enabling the acquisition of more detailed and high-resolution distance information.

[0068] Furthermore, the disparity matching process may use segmentation data obtained by the semantic segmentation process of the semantic segmentation unit 148. In this case, it becomes possible to acquire more detailed and higher resolution distance information.

[0069] Alternatively, the parallax matching process may be performed only on edges or only on parts with strong features, and the propagation process may be performed on other parts by using spherical TOF image data, for example, by using spherical RGB image features or a probabilistic method.

[0070] The semantic segmentation unit 148 uses deep learning to assign segmentation labels indicating objects to the input image in the measurement range. This allows each pixel of the omnidirectional TOF image data to be constrained to one of multiple distance regions divided by distance, further increasing the reliability of calculations.

[0071] 3D reconstruction processing unit 150 acquires the RGB image data of the celestial sphere from RGB image data acquisition unit 142, reconstructs 3D data of the celestial sphere based on the distance information output by parallax calculation unit 149, and outputs a high-density 3D point cloud of the celestial sphere in which color information is added to each 3D point. 3D reconstruction processing unit 150 is an example of a 3D information determination unit that determines 3D information.

[0072] The determination unit 160 acquires the RGB image data of the celestial sphere from the RGB image data acquisition unit 142, and also acquires the TOF image data of the celestial sphere converted into a two-dimensional coordinate system indicated by the RGB image data of the celestial sphere from the reprojection processing unit 147, and determines whether or not a specific object is reflected in the captured image based on these pieces of data, and outputs the determination result to the display control unit 170.

[0073] The display control unit 170 acquires the RGB image data of the celestial sphere from the RGB image data acquisition unit 142, and causes two-dimensional image information based on the acquired RGB image data of the celestial sphere to be displayed on the display unit 20. Furthermore, the display control unit 170 causes the display unit 20 to display a display image including information indicating the determination result acquired from the determination unit 160 and the two-dimensional image information.

[0074] The display control unit 170 is an example of an output unit that outputs two-dimensional image information captured by the imaging unit 11 separately from the three-dimensional information, and the display unit 20 is an example of an output destination that outputs the two-dimensional image information.

[0075] The display control unit 170 may acquire three-dimensional data of the celestial sphere from the three-dimensional reconstruction processing unit 150, and display the three-dimensional information on the display unit 20. Specifically, the display control unit 170 may select, in accordance with a predetermined condition, whether to display two-dimensional image information on the display unit 20 or three-dimensional information on the display unit 20. This allows the display control unit 170 to output the two-dimensional image information separately from the three-dimensional information.

[0076] The transmitting / receiving unit 180 communicates with an external device via wired or wireless technology, and transmits (outputs) the three-dimensional data of the celestial sphere output from the three-dimensional reconstruction processing unit 150 and the two-dimensional image information of the celestial sphere output from the RGB image data acquisition unit 142 via the network 400 to the external device 300 that performs three-dimensional reconstruction processing.

[0077] In this embodiment, the 2D image information captured by the imaging unit 11 refers to "original 2D image information" for creating "2D image data for display" or "2D image data for display." For example, there are cases where "2D image data for display" is created from "original 2D image information" inside the imaging device 1, or where "original 2D image information" is transmitted from the imaging device 1 to an external device, and "2D image data for display" is created from the "original 2D image information" in the external device.

[0078] The transmitting / receiving unit 180 is an example of an output unit that outputs three-dimensional information, and the external device 300 is an example of an output destination that outputs three-dimensional information.

[0079] The transmitting / receiving unit 180 may transmit only the three-dimensional data of the celestial sphere, without transmitting the two-dimensional image information of the celestial sphere. The transmitting / receiving unit 180 may also be configured by an interface circuit with a portable storage medium such as an SD card, a personal computer, or the like.

[0080] (Operation of processing circuit) 5 is a flow diagram showing an example of the operation of the processing circuit 14 of the imaging device 1. When the user turns on the imaging switch 15 and inputs an imaging instruction signal, the control unit 141 of the processing circuit 14 performs an operation to generate a high-density three-dimensional point cloud in the following manner (an example of an imaging processing method and an information processing method).

[0081] First, the control unit 141 drives the light source units 12a and 12A, the TOF sensors 13a and 13A, and the image pickup elements 11a and 11A to capture an image of the measurement range (step S1). Driven by the control unit 141, the light source units 12a and 12A irradiate infrared light (an example of a projection step), and the TOF sensors 13a and 13A receive the reflected light (an example of a light reception step). Furthermore, the image pickup elements 11a and 11A capture an image of the measurement range at the timing when the light source units 12a and 12A start to be driven or in a period immediately thereafter (an example of an image pickup step).

[0082] Next, the RGB image data acquisition unit 142 acquires RGB image data of the measurement range from the image sensors 11a and 11A (step S2). Then, the display control unit 170 acquires the RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and displays two-dimensional image information based on the acquired RGB image data of the entire celestial sphere on the display unit 20 (an example of a display step) (step S3).

[0083] The display control unit 170 displays two-dimensional image information of a partial area of ​​the acquired spherical RGB image data on the display unit 20, and changes the area of ​​the two-dimensional image information displayed on the display unit 20 in response to various inputs from the user. Various inputs from the user can be realized by providing an operation switch other than the photographing switch 15 or by configuring the display unit 20 as an input unit such as a touch panel.

[0084] At this stage, the photographer can check the 2D image information displayed on the display unit 20 to ensure that the photographer himself or herself, the tripod, etc., are not reflected in the captured image, and that 2D image information with the desired layout has not been acquired.

[0085] Next, the TOF image data acquisition unit 144 acquires TOF image data indicating the distance from the TOF sensors 13a and 13A to each position in the two-dimensional area (step S4).

[0086] Next, the monochrome processing unit 143 converts the RGB image data into a monochrome image (step S5). The TOF image data and the RGB image data are different data types, namely distance data and RGB data, respectively, and matching cannot be performed as is. Therefore, each data is first converted into a monochrome image. For the TOF image data, the resolution increasing unit 145 converts the values ​​indicating the distance of each pixel before resolution increase by directly replacing them with the values ​​of the monochrome image.

[0087] Next, the resolution increasing unit 145 increases the resolution of the TOF image data (step S6). Next, the matching processing unit 146 extracts feature amounts of textured portions of each monochrome image and performs matching processing using the extracted feature amounts (step S7).

[0088] Next, the parallax calculation unit 149 calculates the parallax at each position from the difference in distance between corresponding pixels, thereby calculating the distance (step S8).

[0089] Next, the determination unit 160 acquires the RGB image data of the celestial sphere from the RGB image data acquisition unit 142, and also acquires the TOF image data of the celestial sphere converted into the two-dimensional coordinate system indicated by the RGB image data from the reprojection processing unit 147, and determines whether or not a nearby object as a specific target is reflected in the captured image based on these pieces of data, and outputs the determination result to the display control unit 170 (an example of a determination step).

[0090] The display control unit 170 displays information indicating the determination result acquired from the determination unit 160 on the display unit 20 by superimposing it on or including it in the two-dimensional image information (an example of a display step) (step S9). In step S9, the determination unit 160 determines whether or not there are not only nearby objects as specific targets but also highly reflective objects, distant objects, low reflective objects, and image blur, and outputs the determination result to the display control unit 170.

[0091] Then, the 3D reconstruction processing unit 150 acquires the RGB image data from the RGB image data acquisition unit 142, reconstructs the 3D data based on the distance information output by the parallax calculation unit 149, and outputs a high-density 3D point cloud with color information added to each 3D point (step S10).

[0092] Next, the transmitter / receiver 180 transmits the three-dimensional data output from the three-dimensional reconstruction processing unit 150 and the two-dimensional image information output from the RGB image data acquisition unit 142 via the network 400 to an external device 300 that performs three-dimensional reconstruction processing (an example of a three-dimensional information output step) (step S11).

[0093] The transmitting / receiving unit 180 may transmit the three-dimensional data output from the three-dimensional reconstruction processing unit 150 without transmitting the two-dimensional image information output from the RGB image data acquisition unit 142.

[0094] As described above, the imaging device 1 includes the imaging unit 11 and the display control unit 170 that outputs two-dimensional image information captured by the imaging unit 11 separately from three-dimensional information.

[0095] This makes it possible to easily check from the 2D image information whether the photographer or the tripod is captured in the captured image, or whether the 3D information of the desired layout has not been acquired, without checking the 3D information.

[0096] Therefore, it becomes possible to reacquire three-dimensional information while remaining at the site where the three-dimensional information is being acquired, which reduces the effort required to visit the site where the three-dimensional information is being acquired again, compared to when, after leaving the site where the three-dimensional information is being acquired, one realizes that the photographer, the tripod, etc., are reflected in the captured image, or that three-dimensional information with the desired layout has not been acquired.

[0097] The three-dimensional information includes spherical three-dimensional information. In this case, even in the case of spherical three-dimensional information in which it is difficult to confirm whether the photographer himself / herself, the tripod, etc. are included in the captured image or whether three-dimensional information of a desired layout has not been acquired, it becomes possible to easily confirm whether the photographer himself / herself, the tripod, etc. are included in the captured image or whether three-dimensional information of a desired layout has not been acquired from the two-dimensional image information captured by the imaging unit 11.

[0098] The display control unit 170 outputs the two-dimensional image information G in step S3 before the transmitting / receiving unit 180 transmits (outputs) the three-dimensional information in step S11. The display control unit 170 outputs the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

[0099] This makes it possible to check from the 2D image information whether the photographer or the tripod is captured in the captured image, or whether the 3D information of the desired layout has not been acquired, before checking the 3D information.

[0100] The display control unit 170 causes the display unit 20 to display the two-dimensional image information.

[0101] This makes it possible to easily check from the 2D image information displayed on the display unit 20 whether the photographer himself or the tripod is reflected in the captured image, or whether 3D information of the desired layout has not been acquired.

[0102] The display control unit 170 outputs the two-dimensional image information to a display unit 20 that is different from the external device 300 to which the transmitting and receiving unit 180 outputs the three-dimensional information.

[0103] This makes it possible to check whether the photographer himself or the tripod is reflected in the captured image or whether three-dimensional information of the desired layout has not been obtained, from the two-dimensional image information output to a display unit 20 different from the external device 300, without checking the three-dimensional information output to the external device 300.

[0104] The imaging device 1 includes a 3D reconstruction processing unit 150 that determines 3D information based on the output of the distance information acquisition unit 13. The 3D reconstruction processing unit 150 determines the 3D information based on the output of the distance information acquisition unit 13 and the 2D image information.

[0105] This makes it possible to check from the 2D image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been acquired, without checking the 3D information determined by the 3D reconstruction processing unit 150.

[0106] FIG. 6 is a flowchart showing the generation of spherical image data in the embodiment.

[0107] FIG. 6(a) is a flowchart showing the process of generating RGB image data of the celestial sphere, which corresponds to step S2 described in FIG.

[0108] The RGB image data acquisition unit 142 receives two sets of RGB image data in a fisheye image format (step S201).

[0109] The RGB image data acquisition unit 142 converts each RGB image data into an equirectangular image format (step S202). The RGB image data acquisition unit 142 converts the two RGB image data into an equirectangular image format based on the same coordinate system, thereby facilitating image combination in the next step. Note that the RGB image data can be converted into image data using one or more image formats other than the equirectangular image format as needed. For example, the RGB image data can also be converted into coordinates of an image perspectively projected onto an arbitrary surface or an image perspectively projected onto each surface of an arbitrary polyhedron.

[0110] Here, the equirectangular image format will be described. The equirectangular image format is a method capable of representing a celestial sphere image, and is a format of an image (equirectangular image) created using equirectangular projection. Equirectangular projection is a projection that represents three-dimensional directions using two variables, such as the latitude and longitude of a globe, and displays the image on a plane so that the latitude and longitude are orthogonal. Therefore, an equirectangular image is an image created using equirectangular projection, and is represented by coordinates with two axes consisting of two angular variables in a spherical coordinate system.

[0111] The RGB image data acquisition unit 142 combines the two RGB image data generated in step S202 to generate one piece of spherical RGB image data (step S203). The two pieces of input RGB image data cover an area with a full angle of view of more than 180 degrees. Therefore, the spherical RGB image data generated by appropriately connecting these two pieces of RGB image data can cover the entire spherical area.

[0112] The joining process in step S203 can use existing techniques for joining multiple images, and there are no particular limitations on the method.

[0113] FIG. 6B is a flowchart showing the process of generating TOF image data of the entire celestial sphere, which corresponds to step S4 described in FIG.

[0114] The TOF image data acquisition unit 144 acquires two pieces of depth image data in a fisheye image format (step S401).

[0115] The TOF image data acquisition unit 144 converts each of the two TOF image data in the fisheye image format into an equirectangular image format (step S402). As described above, the equirectangular image format is a method capable of expressing a spherical image. In step S402, converting the two TOF image data into the equirectangular image format based on the same coordinate system facilitates image combination in the next step S403.

[0116] The TOF image data acquisition unit 144 combines the two TOF image data generated in step S402 to generate one piece of spherical TOF image data (step S403). The two input TOF image data cover an area with a full angle of view of more than 180 degrees. Therefore, the spherical TOF image data generated by appropriately joining the two TOF image data can cover the entire spherical area.

[0117] The joining process in step S403 can use existing techniques for joining multiple images, and there are no particular limitations on the method.

[0118] FIG. 7 is a flowchart of nearby object determination in the same embodiment.

[0119] FIG. 7 is a flowchart showing the process of determining whether or not a nearby object is captured in a captured image, which corresponds to step S9 described in FIG.

[0120] The determination unit 160 determines whether or not there is a pixel whose charge amount is saturated, as an example of a pixel whose charge amount is equal to or greater than a predetermined value, in the TOF image data of the celestial sphere, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S801).

[0121] When there is a pixel whose amount of stored power is saturated in step S801, the determination unit 160 determines, based on the RGB image data of the celestial sphere acquired from the RGB image data acquisition unit 142, whether the amount of stored power is saturated, as an example of a pixel whose amount of stored power is equal to or greater than a predetermined value, for pixels that have the same coordinates as the pixel whose amount of stored power is saturated in step S801, among the RGB image data of the celestial sphere (step S802).

[0122] If the amount of stored power is saturated in step S802, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S801 is due to external light (for example, the sun or lighting), and outputs error information to the display control unit 170. Based on the error information acquired from the determination unit 160, the display control unit 170 causes the display unit 20 to display a display image including the error information and two-dimensional image information (step S803).

[0123] If the amount of stored power is not saturated in step S802, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S801 is due to the presence of a nearby object, and outputs coordinate position information of the pixel whose amount of stored power is saturated in step S801 to the display control unit 170. Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 causes the display unit 20 to display a display image including identification information that identifies the nearby object and two-dimensional image information (step S804).

[0124] If there is no pixel whose charge amount is saturated in step S801, the determination unit 160 determines whether there is a pixel in the TOF image data of the celestial sphere that indicates distance information of 0.5 m or less, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S805).

[0125] If there is no pixel showing distance information of 0.5 m or less in step S805, the determination unit 160 ends the process.

[0126] If there is a pixel showing distance information of 0.5 m or less in step S805, the determination unit 160 proceeds to step S804 described above, determines that the pixel showing distance information of 0.5 m or less in step S805 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel showing distance information of 0.5 m or less in step S805 to the display control unit 170. Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 causes the display unit 20 to display a display image including identification information that identifies the nearby object and two-dimensional image information.

[0127] As described above, when the display control unit 170 determines that a nearby object exists, it superimposes or includes the identification information in the two-dimensional image information, and when it does not determine that a nearby object exists, it does not superimpose or include the identification information in the two-dimensional image information.

[0128] That is, the display control unit 170 causes the display unit 20 to display different images depending on whether or not a nearby object exists.

[0129] Furthermore, based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 causes the display unit 20 to display a display image including identification information for identifying the nearby object and two-dimensional image information.

[0130] That is, the display control unit 170 causes the display unit 20 to display different images at different positions depending on the position of the nearby object.

[0131] FIG. 8 is a diagram for explaining the display content of the display unit in the embodiment.

[0132] FIG. 8 is an explanatory diagram corresponding to step S2 shown in FIG. 5 and steps S803 and S804 shown in FIG.

[0133] On the display unit 20, the display control unit 170 displays two-dimensional image information G. Also, on the display unit 20, the display control unit 170 displays a display image including identification information G1, G2 (for example, a finger, a tripod) that identify an object such as a nearby object, error information G3, and the two-dimensional image information G. The error information G3 can be displayed using a mark such as "sun, lighting," as shown in FIG.

[0134] As described above, the imaging device 1 includes an imaging unit 11 that images an object, a projection unit 12 that projects light onto the object, a distance information acquisition unit 13 that receives light reflected from the object, and a display control unit 170 that causes the display unit 20 to display different images depending on the presence or absence of an object such as a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

[0135] This allows the photographer to accurately check whether or not the photographer himself or herself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the influence of external light.

[0136] The imaging device 1 includes a display unit 20. This allows the photographer to reliably check whether or not a nearby object appears in the captured image.

[0137] The display control unit 170 displays different images at different positions on the display unit 20 depending on the position of the nearby object, thereby allowing the photographer to confirm the position of the nearby object reflected in the captured image.

[0138] The display control unit 170 causes the display unit 20 to display the image information G captured by the imaging unit 11, and also causes the display unit 20 to display a display image including the identification information G1, G2 that identifies the nearby object and the image information. This allows the photographer to reliably confirm the position of the nearby object reflected in the captured image.

[0139] The imaging device 1 includes a judgment unit 160 that judges that there is an object nearby when the charge amount is saturated as an example of a pixel where the charge amount due to light received by the distance information acquisition unit 13 is equal to or greater than a predetermined value, and when the charge amount is not saturated as an example of a pixel where the charge amount is equal to or less than the predetermined value in the imaging unit 11.

[0140] This allows the photographer to accurately check whether or not a nearby object is reflected in the captured image, distinguishing it from the influence of external light.

[0141] Fig. 9 is a diagram showing the appearance of an imaging device according to a modified example of the embodiment, and Fig. 10 is a diagram showing the configuration of a processing block of a processing circuit in the modified example.

[0142] In this modification, display control unit 170 acquires RGB image data of the celestial sphere from RGB image data acquisition unit 142, and displays two-dimensional image information based on the acquired RGB image data of the celestial sphere on display unit 520 of display device 500. Display unit 520 is an example of an output destination that outputs the two-dimensional image information.

[0143] This makes it possible to easily check from the 2D image information displayed on the display unit 520 whether the photographer or the tripod is reflected in the captured image, or whether 3D information of the desired layout has not been obtained.

[0144] The display control unit 170 outputs the two-dimensional image information to a display unit 520 that is different from the external device 300 to which the transmitting and receiving unit 180 outputs the three-dimensional information.

[0145] This makes it possible to check whether the photographer himself or the tripod is reflected in the captured image or whether three-dimensional information of the desired layout has not been obtained, from the two-dimensional image information output to a display unit 520 different from the external device 300, without checking the three-dimensional information output to the external device 300.

[0146] The display control unit 170 may acquire three-dimensional data of the celestial sphere from the three-dimensional reconstruction processing unit 150, and display the three-dimensional information on the display unit 520. Specifically, the display control unit 170 may select, in accordance with a predetermined condition, whether to display two-dimensional image information on the display unit 520 or to display three-dimensional information on the display unit 520. This allows the display control unit 170 to output the two-dimensional image information separately from the three-dimensional information.

[0147] The display control unit 170 causes the display unit 520 to display a display image including the error information and the two-dimensional image information based on the error information acquired from the determination unit 160.

[0148] The display control unit 170 causes the display unit 520 to display a display image including identification information for identifying the nearby object and two-dimensional image information based on the coordinate position information of the pixel acquired from the determination unit 160.

[0149] That is, the display control unit 170 causes the display unit 520 to display different images depending on the presence or absence of a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

[0150] This allows the photographer to accurately check whether or not the photographer himself or herself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the influence of external light.

[0151] The display control unit 170 displays different images at different positions on the display unit 520 depending on the position of the nearby object, thereby allowing the photographer to confirm the position of the nearby object reflected in the captured image.

[0152] The display control unit 170 causes the display unit 520 to display the image information captured by the imaging unit 11, and also causes the display unit 520 to display a display image including identification information for identifying the nearby object and the image information. This allows the photographer to reliably confirm the position of the nearby object reflected in the captured image.

[0153] Fig. 11 is a diagram showing the appearance of an imaging device according to a second modified example of the embodiment of the present invention, and Fig. 12 is a diagram showing the configuration of a processing block of a processing circuit in the second modified example.

[0154] 11, the imaging device 1 includes a plurality of display units 20A, 20a instead of the display unit 20 shown in Fig. 1. The display units 20A, 20a are configured with LEDs or the like, and flash or light up in response to an output signal from the processing circuit 14.

[0155] The display unit 20a is provided on a first surface on the front side of the housing 10, and the display unit 20A is provided on a second surface on the rear side of the housing 10.

[0156] 12, the display control unit 170 causes the display units 20A and 20a to display information indicating the determination result obtained from the determination unit 160. For example, the display units 20a and 20b may flash red when there is an object close to each side of the imaging device 1.

[0157] Furthermore, the transmitting / receiving unit 180 transmits (outputs) the spherical two-dimensional image information output from the RGB image data acquiring unit 142 to the display device 500 via the network 400. The display device 500 is an example of an output destination to which the two-dimensional image information is output.

[0158] That is, in the second modified example, in step S3 shown in FIG. 5 , the transmitting / receiving unit 180 acquires RGB image data of the entire celestial sphere from the RGB image data acquiring unit 142, and transmits (outputs) two-dimensional image information based on the acquired RGB image data of the entire celestial sphere to the display device 500.

[0159] The transmitting / receiving unit 510 of the display device 500 receives the two-dimensional image information transmitted from the transmitting / receiving unit 180 of the imaging device 1.

[0160] The display control unit 530 of the display device 500 causes the two-dimensional image information received by the transmitting and receiving unit 510 to be displayed on the display unit 520. The display device 500 including the display control unit 530 is an example of an information processing device.

[0161] As described above, the imaging device 1 includes the imaging unit 11 and the transmitting / receiving unit 180 that outputs two-dimensional image information captured by the imaging unit 11 separately from three-dimensional information.

[0162] This makes it possible to easily check from the 2D image information whether the photographer or the tripod is captured in the captured image, or whether the 3D information of the desired layout has not been acquired, without checking the 3D information.

[0163] Therefore, it becomes possible to reacquire three-dimensional information while remaining at the site where the three-dimensional information is being acquired, which reduces the effort required to visit the site where the three-dimensional information is being acquired again, compared to when, after leaving the site where the three-dimensional information is being acquired, one realizes that the photographer, the tripod, etc., are reflected in the captured image, or that three-dimensional information with the desired layout has not been acquired.

[0164] The transmitting / receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before transmitting (outputting) the three-dimensional information in step S11. The transmitting / receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

[0165] This makes it possible to check from the 2D image information whether the photographer or the tripod is captured in the captured image, or whether the 3D information of the desired layout has not been acquired, before checking the 3D information.

[0166] The transmitting / receiving unit 180 transmits the two-dimensional image information to the display device 500, and the display device 500 causes the display unit 520 to display the two-dimensional image information.

[0167] This makes it possible to easily check from the 2D image information displayed on the display unit 520 whether the photographer or the tripod is reflected in the captured image, or whether 3D information of the desired layout has not been obtained.

[0168] The transmitting / receiving unit 180 transmits the two-dimensional image information to a display device 500 that is different from the external device 300 that outputs the three-dimensional information.

[0169] This makes it possible to check whether the photographer himself or the tripod is reflected in the captured image or whether three-dimensional information of the desired layout has not been obtained, from the two-dimensional image information output to the display unit 520 of the display device 500, which is different from the external device 300, without checking the three-dimensional information output to the external device 300.

[0170] The transmitting / receiving unit 180 may transmit the three-dimensional information to the display device 500. Specifically, the transmitting / receiving unit 180 may select, in accordance with a predetermined condition, whether to transmit two-dimensional image information to the display device 500 or three-dimensional information to the display device 500. This allows the transmitting / receiving unit 180 to transmit the two-dimensional image information to the display device 500 separately from the three-dimensional information.

[0171] FIG. 13 is a flowchart of nearby object determination in the second modified example.

[0172] FIG. 13 is a flowchart showing the process of determining whether or not a nearby object is captured in a captured image, which corresponds to step S9 described with reference to FIG. 5 in the second modified example.

[0173] The determination unit 160 determines whether or not there is a pixel whose charge amount is saturated, as an example of a pixel whose charge amount is equal to or greater than a predetermined value, in the TOF image data of the celestial sphere, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S811).

[0174] When there is a pixel whose amount of stored power is saturated in step S811, the determination unit 160 determines, based on the RGB image data of the celestial sphere acquired from the RGB image data acquisition unit 142, whether the amount of stored power is saturated, as an example of a pixel whose amount of stored power is equal to or greater than a predetermined value, for pixels that have the same coordinates as the pixel whose amount of stored power is saturated in step S811, among the RGB image data of the celestial sphere (step S812).

[0175] If the amount of stored power is saturated in step S812, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S811 is due to external light, and outputs error information to the display control unit 170. The display control unit 170 causes the display units 20A and 20a to display the error information based on the error information acquired from the determination unit 160 (step S813).

[0176] If the amount of stored power is not saturated in step S812, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S811 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel whose amount of stored power is saturated in step S811 to the display control unit 170. Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 (step S814).

[0177] If there is no pixel whose charged amount is saturated in step S811, the determination unit 160 determines whether there is a pixel in the TOF image data of the celestial sphere that indicates distance information of 0.5 m or less, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S815).

[0178] If there is no pixel showing distance information of 0.5 m or less in step S815, the determination unit 160 ends the process.

[0179] If there is a pixel showing distance information of 0.5 m or less in step S815, the determination unit 160 proceeds to step S814 described above, determines that the pixel showing distance information of 0.5 m or less in step S815 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel showing distance information of 0.5 m or less in step S815 to the display control unit 170. The display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 based on the coordinate position information of the pixel acquired from the determination unit 160.

[0180] If it is determined in step S814 that the display control unit 170 is on the front side, it causes the display unit 20a arranged on the front side of the housing 10 to blink (step S816).

[0181] If it is not determined in step S814 that the display control unit 170 is on the front side, the display control unit 170 causes the display unit 20A arranged on the back side of the housing 10 to blink (step S817).

[0182] As described above, the display control unit 170 blinks the display unit 20a or the display unit 20A when it determines that a nearby object is present, and does not blink the display unit 20a or the display unit 20A when it does not determine that a nearby object is present.

[0183] That is, the display control unit 170 causes the display unit 20a and the display unit 20A to display different images depending on whether or not a nearby object is present.

[0184] This allows the photographer to accurately check whether or not the photographer himself or herself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the influence of external light.

[0185] Furthermore, the display control unit 170 blinks the display unit 20a or the display unit 20A based on the coordinate position information of the pixel acquired from the determination unit 160.

[0186] That is, the display control unit 170 displays a different image on the display unit 20a or the display unit 20A depending on the position of the nearby object, thereby enabling the photographer to confirm the position of the nearby object in the captured image.

[0187] Then, the display control unit 170 controls the display unit 20A, 20a that is closer to the nearby object to display different images depending on whether or not the nearby object is present, thereby enabling the photographer to reliably confirm the position of a specific object in the captured image.

[0188] FIG. 14 is a diagram illustrating the configuration of an imaging device according to a third modified example of the embodiment of the present invention.

[0189] In a third variant shown in FIG. 14, the imaging device 1 includes, in addition to the configuration shown in FIG. 2, another imaging unit 111 having other imaging elements 111a, 111A, other fisheye lenses (wide-angle lenses) 111b, 111B, etc.

[0190] In the third modified example, the RGB imaging unit 11 and another imaging unit 111 are provided on the same baseline. In this case, multi-view processing becomes possible in the processing circuit 14. That is, by simultaneously driving the imaging unit 11 and another imaging unit 111, which are provided a predetermined distance apart on one surface, RGB images from two viewpoints can be obtained. This makes it possible to use parallax calculated based on the two RGB images, further improving the distance accuracy over the entire measurement range.

[0191] Specifically, when an RGB imaging unit 11 and another imaging unit 111 are installed, it becomes possible to use multi-baseline stereo (MSB) using SSD and EPI processing, as in conventional parallax calculations. Therefore, by using this, the reliability of parallax increases, making it possible to achieve high spatial resolution and accuracy.

[0192] As described above, the imaging device 1 includes another imaging unit 111, and the 3D reconstruction processing unit 150 determines 3D information based on the output of the distance information acquisition unit 13, the 2D image information, and other 2D image information captured by the other imaging unit 111.

[0193] The imaging device 1 may also include another imaging unit 111 and a three-dimensional information determination unit that determines three-dimensional information based on two-dimensional image information and other two-dimensional image information captured by the other imaging unit 111, without relying on the output of the distance information acquisition unit 13.

[0194] This makes it possible to check from the two-dimensional image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image or whether three-dimensional information of the desired layout has not been obtained, without checking the three-dimensional information determined by the three-dimensional reconstruction processing unit 150 based on the two-dimensional image information.

[0195] FIG. 15 is a flowchart showing a determination flow for a highly reflective object in an embodiment of the present invention, and is a flowchart showing the process of determining whether or not a highly reflective object is reflected in a captured image, which corresponds to step S9 described in FIG.

[0196] The determination unit 160 determines whether or not there is a pixel whose charge amount is saturated, as an example of a pixel whose charge amount is equal to or greater than a predetermined value, in the TOF image data of the celestial sphere, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S21).

[0197] If a pixel with saturated charge is found in step S21, the determination unit 160 determines whether RGB image data including pixels at the same coordinates as the pixel with saturated charge in step S21, among the RGB image data of the celestial sphere, matches reference information indicating a highly reflective object, based on the RGB image data of the celestial sphere acquired from the RGB image data acquisition unit 142 (step S22). A model image may be used as the reference information indicating a highly reflective object, and the degree of match between the RGB image data and the model image may be determined by image recognition. Alternatively, parameters such as spectrum and color tone may be used as the reference information and RGB image data indicating a highly reflective object, and the degree of match may be determined based on a predetermined threshold. Furthermore, the reference information may be stored in a table, or a learning model may be used.

[0198] The processing circuit 14 stores images of highly reflective objects such as metals and mirrors as model image information, and in step S22, the judgment unit 160 uses a judger such as AI to determine whether the acquired image matches the stored image of the highly reflective object.

[0199] If the determination unit 160 determines that the image acquired in step S22 matches the stored image of the highly reflective object, it outputs the coordinate position information of the pixel determined in step S22 to the display control unit 170 (step S23). The display control unit 170 causes the display unit 20, 520 to display a display image including identification information for identifying the highly reflective object and two-dimensional image information based on the coordinate position information of the pixel acquired from the determination unit 160 (step S24), and ends the process.

[0200] Steps S22 and S23 are an example of a determination step, and step S24 is an example of a display step.

[0201] If the judgment unit 160 determines that the image acquired in step S22 does not match the stored image of a highly reflective object, it proceeds to judgment of a nearby object (step S23) and executes the nearby object judgment flow shown in Figure 7 (step S25).

[0202] As described above, the imaging device 1 includes a determination unit 160 that determines whether a highly reflective object is present based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a display control unit 170 that causes the display units 20, 520 to display different images depending on whether a highly reflective object is present or not.

[0203] This allows the photographer to accurately confirm that a highly reflective object such as a mirror is included in the captured image, distinguishing it from nearby objects and the effects of external light.

[0204] The imaging device 1 includes a display unit 20. This allows the photographer to reliably confirm that a highly reflective object is included in the captured image.

[0205] The display control unit 170 displays different images at different positions on the display units 20 and 520 depending on the position of the highly reflective object, thereby allowing the photographer to confirm the position of the highly reflective object.

[0206] 13, the display unit 20 includes a plurality of display units 20A, 20a, and the display control unit 170 controls the display unit that is closer to the highly reflective object to display different information depending on whether or not the target object is present. This allows the photographer to reliably confirm the position of the highly reflective object.

[0207] 3, the display control unit 170 causes the display unit 20, 520 to display the image information G captured by the imaging unit 11, and also causes the display unit 20, 520 to display a display image including identification information for identifying the highly reflective object and the image information G. This allows the photographer to reliably confirm the position of the highly reflective object.

[0208] The judgment unit 160 judges that a highly reflective object is present when the charge stored in the pixel due to the light received by the distance information acquisition unit 13 is saturated, which is an example of a pixel where the charge stored in the pixel due to the light received is greater than a predetermined value, and when the image information captured by the imaging unit matches model image information, which is an example of reference information indicating a highly reflective object.

[0209] This allows the photographer to accurately confirm that a highly reflective object is included in the captured image, distinguishing it from nearby objects and the effects of external light.

[0210] The imaging device 1 acquires distance information to the target based on the light received by the distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired distance information is not being acquired is not due to a nearby object or external light, but due to a highly reflective object.

[0211] The imaging device 1 includes a transmitter / receiver 180 that outputs three-dimensional information determined based on distance information acquired from the distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired three-dimensional information is not being acquired is not due to a nearby object or external light, but due to a highly reflective object.

[0212] FIG. 16 is a flowchart showing the determination flow for distant objects and low-reflecting objects in this embodiment, and is a flowchart showing the process for determining whether distant objects and low-reflecting objects are reflected in the captured image, which corresponds to step S9 described in FIG.

[0213] Based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147, the determination unit 160 determines whether or not there is a pixel in the TOF image data of the celestial sphere whose amount of charge is equal to or less than a threshold at which distance information can be acquired (step S41).

[0214] If there is no pixel with a charged amount equal to or less than the threshold in step S41, the determination unit 160 determines whether or not there is a pixel in the TOF image data of the celestial sphere that indicates distance information of 10 m or more, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147, and if there is a pixel that indicates distance information of 10 m or more, it determines that the object is a distant object, and outputs coordinate position information of the pixel to the display control unit 170 (step S42).

[0215] The display control unit 170 displays a display image including identification information for identifying distant objects and two-dimensional image information on the display unit 20, 520 based on the coordinate position information of the pixel acquired from the judgment unit 160 (step S43), and ends the processing.

[0216] If there is no pixel showing distance information of 10 m or more in step S42, the decision unit 160 ends the process.

[0217] When there is a pixel whose amount of stored power is equal to or less than the threshold in step S41, the determination unit 160 determines, based on the RGB image data of the celestial sphere acquired from the RGB image data acquisition unit 142, whether the amount of stored power is equal to or less than the threshold at which an object can be recognized for pixels that have the same coordinates as the pixel whose amount of stored power is equal to or less than the threshold in step S41 (step S44).

[0218] If the determination unit 160 determines in step S44 that the amount of stored power is equal to or less than the threshold value at which an object can be recognized, it determines that the object is a low-reflectivity object, and outputs the coordinate position information of the pixel to the display control unit 170.

[0219] The display control unit 170 displays a display image including identification information for identifying low-reflectivity objects and two-dimensional image information on the display unit 20, 520 based on the coordinate position information of the pixel acquired from the judgment unit 160 (step S45), and ends the processing.

[0220] If the determination unit 160 determines in step S44 that the amount of stored power is not equal to or less than the threshold for object recognition, it determines the distance for the RGB image data including the pixel determined in step S44 based on model information, which is an example of reference information that associates images with distances (step S46). If a model image is used as the reference information, the degree of match between the RGB image data and the model image may be determined by image recognition. Alternatively, parameters may be used as the reference information and the RGB image data to determine the degree of match based on a predetermined threshold. Furthermore, the reference information may be stored in a table, or a learning model may be used.

[0221] The processing circuit 14 stores images associated with distances for each of a plurality of distances as model information, and in step S46, the judgment unit 160 uses a judger such as AI to determine whether the acquired image matches the image for each stored distance.

[0222] The judgment unit 160 judges whether the distance associated with the image that matches the image acquired in step S46 is 10 m or more, and if it is 10 m or more, it judges it to be a distant object, outputs the coordinate position information of the pixel to the display control unit 170 (step S47), and proceeds to step S43.

[0223] If the distance associated with the image that matches the image acquired in step S46 is not 10 m or more, the judgment unit 160 judges that it is a low-reflecting object, outputs the coordinate position information of the pixel to the display control unit 170 (step S47), and proceeds to step S45.

[0224] Steps S41, S42, S44, and S47 are examples of a determination step, and steps S43 and S45 are examples of a display step.

[0225] As described above, the imaging device 1 includes a determination unit 160 that determines whether a distant object or a low-reflectivity object is present based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a display control unit 170 that causes the display units 20, 520 to display different images depending on whether a distant object or a low-reflectivity object is present.

[0226] This allows the photographer to accurately confirm that distant objects or low-reflectivity objects such as black objects are included in the captured image.

[0227] The imaging device 1 includes a display unit 20. This allows the photographer to reliably confirm that a distant object or a low-reflectivity object is included in the captured image.

[0228] The display control unit 170 displays different images at different positions on the display units 20 and 520 depending on the position of the distant object or low-reflectivity object, thereby allowing the photographer to confirm the position of the distant object or low-reflectivity object.

[0229] 13, the display unit 20 includes a plurality of display units 20A, 20a, and the display control unit 170 controls the display unit that is closer to the distant object or low-reflectivity object to display different information depending on whether the object is present or not. This allows the photographer to reliably confirm the position of the distant object or low-reflectivity object.

[0230] 3, the display control unit 170 causes the display unit 20, 520 to display the image information G captured by the imaging unit 11, and also causes the display unit 20, 520 to display a display image including identification information for identifying the distant object or low-reflectivity object and the image information G. This allows the photographer to reliably confirm the position of the distant object or low-reflectivity object.

[0231] When the amount of charge stored in a pixel due to light received by distance information acquisition unit 13 is equal to or less than a threshold, determination unit 160 determines whether the object is a low-reflecting object or a distant object based on the output of imaging unit 11. This allows the photographer to accurately confirm whether the captured image contains a low-reflecting object or a distant object.

[0232] The determining unit 160 determines that a low-reflectivity object is present when the amount of charge stored in the pixel due to the light received by the distance information acquiring unit 13 is equal to or less than a threshold and the amount of charge stored in the pixel of the imaging unit 11 is equal to or less than a threshold. This allows the photographer to accurately confirm that a low-reflectivity object is included in the captured image.

[0233] The judgment unit 160 judges that there is a distant object if the amount of charge stored in the pixel due to the light received by the distance information acquisition unit 13 is below a threshold, the amount of charge stored in the pixel of the imaging unit 11 is above a threshold, and the distance judged based on the pixel is above a threshold.

[0234] This allows the photographer to accurately confirm that the distant object is included in the captured image.

[0235] Imaging device 1 acquires distance information to the target based on the light received by distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired distance information is not being acquired is because of a distant object or a low-reflecting object.

[0236] Imaging device 1 includes transmitting / receiving unit 180, which is an example of an output unit that outputs three-dimensional information determined based on distance information acquired from distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired three-dimensional information is not being acquired is because of a distant object or a low-reflecting object.

[0237] FIG. 17 is a flowchart showing the process of determining whether or not there is image blur in the captured image, which corresponds to step S9 described in FIG.

[0238] The determination unit 160 determines whether or not there is a pixel of an image including an edge peripheral region in the RGB image of the celestial sphere, based on the RGB image data of the celestial sphere acquired from the RGB image data acquisition unit 142 (step S51).

[0239] Here, the judgment unit 160 detects edges contained in the image by comparing changes in luminance values ​​in pixels and their first and second derivative values ​​with thresholds, and identifies pixels in the image that include areas surrounding the edges, but edges may also be detected using other methods.

[0240] Next, if there is a pixel in the image that includes an edge peripheral region in step S51, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147, it is determined whether or not the edge of the TOF phase image is misaligned for TOF image data that includes pixels at the same coordinates as the pixels of the image that have been determined to include the edge peripheral region in step S51, among the TOF image data of the celestial sphere.If it is determined that the edge is misaligned, the coordinate position information of the pixel determined in step S51 is output to the display control unit 170 (step S52).

[0241] Based on the pixel coordinate position information acquired from the judgment unit 160, the display control unit 170 causes the display unit 20, 520 to display a display image including identification information for identifying image blur and two-dimensional image information (step S53), and ends the processing.

[0242] Steps S51 and S52 are an example of a determination step, and step S53 is an example of a display step.

[0243] If there are no pixels in the image that include the edge peripheral region in step S51, and if there is no edge shift in the TOF phase image in step S52, the determination unit 160 ends the process.

[0244] In this embodiment, the distance is measured by a phase difference detection method, and the imaging device 1 acquires and adds N TOF phase images of the same phase for each of the phases of 0°, 90°, 180°, and 270°.

[0245] In this way, by adding N phase images of the same phase, the dynamic range of the phase image of that phase is expanded. Furthermore, the time required to capture N phase images of each phase is shortened, resulting in a phase image with excellent positional accuracy that is less affected by blurring and the like. Therefore, the phase image with an expanded dynamic range can be used to accurately detect the amount of image misalignment described below.

[0246] The determination unit 160 calculates the amount of pixel shift for each phase using a general optical flow calculation process or the machine learning method disclosed in the reference paper listed below, and compares the sum of the pixel shift amounts for each phase for all phases with a threshold value to make a final determination as to whether or not the image is blurred; however, the determination as to whether or not the image is blurred may be made using other methods.

[0247] Paper title Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset author Qi Guo (SEAS, Harvard University), Iuri Frosio Orazio Gallo Todd Zickler(SEAS, Harvard University) Jan Kautz Release date Monday, September 10, 2018 Published by ECCV(European Conference on Computer Vision) 2018 URL (Uniform Resource Locator) https: / / research.nvidia.com / publication / 2018-09_Tackling-3D-ToF

[0248] As described above, the imaging device 1 includes a determination unit 160 that determines whether or not there is image blur based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a display control unit 170 that causes the display units 20, 520 to display different images depending on whether or not there is image blur.

[0249] This allows the photographer to accurately confirm that the captured image contains image blur.

[0250] The imaging device 1 includes a display unit 20. This allows the photographer to reliably confirm that the captured image contains image blur.

[0251] The display control unit 170 displays different images at different positions on the display units 20 and 520 according to the position of the blur in the image, thereby enabling the photographer to confirm the position of the blur in the image.

[0252] 13, the display unit 20 includes a plurality of display units 20A, 20a, and the display control unit 170 controls the display unit that is closer to the position of the blurred image to display differently depending on whether or not there is an object. This allows the photographer to reliably confirm the position of the blurred image.

[0253] 3, the display control unit 170 causes the display unit 20, 520 to display the image information G captured by the imaging unit 11, and also causes the display unit 20, 520 to display a display image including identification information for identifying blurring of the image and the image information G. This allows the photographer to reliably confirm the position of blurring of the image.

[0254] The determining unit 160 detects the edge of the image based on the image information captured by the imaging unit 11, and determines that the image is blurred if there is a pixel shift due to the light received by the distance information acquiring unit 13.

[0255] This allows the photographer to accurately confirm that the captured image contains image blur.

[0256] Imaging device 1 acquires distance information to the target based on the light received by distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired distance information is not acquired is due to image blur.

[0257] Imaging device 1 includes transmitting / receiving unit 180, which is an example of an output unit that outputs three-dimensional information determined based on distance information acquired from distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired three-dimensional information is not acquired is due to image blur.

[0258] FIG. 18 is a decision flow diagram in the fourth modified example of the embodiment of the present invention.

[0259] In step S9 described in FIG. 5, the judgment unit 160 judges whether or not a specific object such as a nearby object is present, and the display control unit 170 causes the display units 20 and 520 to display different images depending on whether or not the specific object is present. However, in the fourth variant, the judgment unit 160 does not judge whether or not a specific object is present, and the display control unit 170 does not cause the display units 20 and 520 to display different images depending on whether or not the specific object is present, but the user is able to recognize the specific object.

[0260] In the flow shown in FIG. 18(a), the determination unit 160 determines, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147, whether or not there is a pixel in the TOF image data of the celestial sphere whose amount of stored charge is saturated as an example of a pixel whose amount of stored charge is equal to or greater than a predetermined value and whose amount of stored charge is equal to or greater than a threshold from which distance information can be acquired, and if there is a pixel whose amount of stored charge is equal to or greater than the threshold, outputs coordinate position information of the pixel to the display control unit 170 (step S31).

[0261] Based on the coordinate position information of the pixel acquired from the judgment unit 160, the display control unit 170 causes the display unit 20, 520 to display a display image including position identification information for identifying the position and two-dimensional image information, similar to the nearby object shown in Figure 3 (step S32), and then ends the processing.

[0262] If the amount of stored power is not equal to or greater than the threshold in step S31, the determination unit 160 ends the process.

[0263] In the flow shown in FIG. 18(b), the determination unit 160 determines whether or not there is a pixel in the TOF image data of the celestial sphere, the amount of stored charge of which is equal to or less than a threshold at which distance information can be acquired, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147, and if there is a pixel in which the amount of stored charge is equal to or less than the threshold, outputs coordinate position information of the pixel to the display control unit 170 (step S33).

[0264] Based on the coordinate position information of the pixel acquired from the judgment unit 160, the display control unit 170 causes the display unit 20, 520 to display a display image including position identification information for identifying the position and two-dimensional image information, similar to the nearby object shown in Figure 3 (step S34), and then ends the processing.

[0265] If the amount of stored power is not equal to or less than the threshold in step S33, the determination unit 160 ends the process.

[0266] In the flow illustrated in FIG. 18(c), the determination unit 160 determines whether or not there is a pixel in the TOF image data of the celestial sphere for which distance information cannot be acquired due to a shift in the TOF phase image, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147. If there is a pixel for which the TOF phase image is shifted, the determination unit 160 outputs coordinate position information of the pixel to the display control unit 170 (step S35).

[0267] Here, the determining unit 160 determines the deviation of the TOF phase images using the same method as that described in step S52 of FIG.

[0268] Based on the coordinate position information of the pixel acquired from the judgment unit 160, the display control unit 170 causes the display unit 20, 520 to display a display image including position identification information for identifying the position and two-dimensional image information, similar to the nearby object shown in Figure 3 (step S36), and then ends the processing.

[0269] If there is no pixel in the TOF phase image that is shifted, the determination unit 160 ends the process.

[0270] As described above, the imaging device 1 includes a display control unit 170 that causes the display unit 20, 520 to display a display image including position identification information that identifies a position and two-dimensional image information G captured by the imaging unit 11 that captures an image of the target, based on position information indicating a position determined by the determination unit 160 as to whether the output of the distance information acquisition unit 13 is above or below a threshold value.

[0271] This allows the causes of the desired output not being obtained to be identified by checking the two-dimensional image G for positions where the output of the distance information acquisition unit 13 is above or below the threshold, i.e., positions where the output of the distance information acquisition unit 13 is too strong or too weak and the desired output cannot be obtained.

[0272] The imaging device 1 also includes a display control unit 170 that causes the display unit 20, 520 to display a display image including position identification information that identifies the position and two-dimensional image information G captured by the imaging unit 11 that captures the target, based on position information indicating the position determined by the judgment unit 160 that distance information to the target cannot be acquired based on the output of the distance information acquisition unit 13.

[0273] This allows the cause of the inability to acquire distance information to the target to be confirmed by checking the position where distance information to the target cannot be acquired in the two-dimensional image G.

[0274] The judgment units 160, 560, and 660 may determine that distance information to the target cannot be acquired not only when the output of the distance information acquisition unit 13 is above or below the threshold, but also when image blur is detected based on the output of the distance information acquisition unit 13.

[0275] FIG. 19 is a diagram showing an example of the configuration of a processing block of a processing circuit according to the fifth modified example of the embodiment of the present invention.

[0276] The processing blocks of the processing circuit in the fifth modified example illustrated in FIG. 19 are different from the processing blocks of the processing circuit 14 of the present embodiment illustrated in FIG. 4 in that the determination unit 160 outputs the determination result to the transmission / reception unit 180, the determination unit 160 acquires three-dimensional data of the celestial sphere from the three-dimensional reconstruction processing unit 150 and outputs the determination result to the transmission / reception unit 180, and the display control unit 170 acquires the three-dimensional data of the celestial sphere from the three-dimensional reconstruction processing unit 150.

[0277] The transmitting / receiving unit 180 transmits (outputs) the determination result of the determining unit 160 via the network 400 to the external device 300 that performs three-dimensional reconstruction processing, in addition to the three-dimensional data of the celestial sphere output from the three-dimensional reconstruction processing unit 150 and the two-dimensional image information of the celestial sphere output from the RGB image data acquiring unit 142.

[0278] Display control unit 170 causes display unit 20 to display a three-dimensional image based on the three-dimensional data of the celestial sphere acquired from three-dimensional reconstruction processing unit 150, and also causes display unit 20 to display a display image including identification information for identifying the specific object and the three-dimensional image based on the determination result by determination unit 160, which has determined whether a specific object is present based on both the output of imaging unit 11 and the output of distance information acquisition unit 13. The specific object also includes a nearby object, a highly reflective object, a distant object, a low-reflective object, and a blurred area of ​​the image.

[0279] This allows the user to view the three-dimensional image 3G and confirm whether the reason the desired three-dimensional image 3G is not displayed is due to a nearby object, a highly reflective object, a distant object, a low-reflective object, or image blur.

[0280] FIG. 20 is a diagram showing an example of the configuration of an information processing system according to the sixth modified example of the embodiment of the present invention.

[0281] The information processing system according to the sixth modified example shown in FIG.

[0282] The imaging device 1 shown in FIG. 20 includes imaging elements 11a and 11A, TOF sensors 13a and 13A, light source units 12a and 12A, and a shooting switch 15, which are configured in the same manner as those shown in FIG.

[0283] 20 includes a control unit 141, an RGB image data acquisition unit 142, a TOF image data acquisition unit 144, and a transmission / reception unit 180. The control unit 141 has the same configuration as that shown in FIG.

[0284] Similar to FIG. 4, the RGB image data acquisition unit 142 acquires RGB image data captured by the image capture elements 11a and 11A based on an image capture instruction from the control unit 141, and outputs RGB image data of the entire celestial sphere. However, it differs from FIG. 4 in that the output destination is the transmitting / receiving unit 180.

[0285] Similar to FIG. 4 , the TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on an instruction to generate TOF image data from the control unit 141, and outputs the TOF image data of the entire celestial sphere. However, it differs from FIG. 4 in that the output destination is the transmitting / receiving unit 180.

[0286] Unlike in FIG. 4, the transmitting / receiving unit 180 transmits (outputs) the RGB image data of the entire celestial sphere output from the RGB image data acquiring unit 142 and the TOF image data of the entire celestial sphere output from the TOF image data acquiring unit 144 to the display device 500.

[0287] The display device 500 shown in FIG. 20 includes a transmitting / receiving unit 510, a display unit 520, and a display control unit 530, similar to the second modification shown in FIG. 12. The image processing device further includes an RGB image data acquisition unit 542, a monochrome processing unit 543, a TOF image data acquisition unit 544, a high resolution unit 545, a matching processing unit 546, a reprojection processing unit 547, a semantic segmentation unit 548, a disparity calculation unit 549, a 3D reconstruction processing unit 550, and a judgment unit 560.

[0288] The transmitting / receiving unit 180 receives the spherical RGB image data and the spherical TOF image data transmitted from the imaging device 1.

[0289] The RGB image data acquisition unit 542 acquires RGB image data of the entire celestial sphere from the transmission / reception unit 180, and the TOF image data acquisition unit 544 acquires RGB image data of the entire celestial sphere from the transmission / reception unit 180. Otherwise, they are configured in the same manner as the RGB image data acquisition unit 142 and the TOF image data acquisition unit 144 shown in FIG. 4, respectively.

[0290] The monochrome processing unit 543, TOF image data acquisition unit 544, high resolution unit 545, matching processing unit 546, reprojection processing unit 547, semantic segmentation unit 548, parallax calculation unit 549, 3D reconstruction processing unit 550, and judgment unit 560 are configured in the same manner as the monochrome processing unit 143, TOF image data acquisition unit 144, high resolution unit 145, matching processing unit 146, reprojection processing unit 147, semantic segmentation unit 148, parallax calculation unit 149, 3D reconstruction processing unit 150, and judgment unit 160 shown in Figure 4.

[0291] The display control unit 530 may acquire RGB image data of the celestial sphere from the RGB image data acquisition unit 542 and cause the display unit 520 to display a two-dimensional image based on the acquired RGB image data of the celestial sphere, or may acquire three-dimensional data of the celestial sphere from the three-dimensional reconstruction processing unit 545 and cause the display unit 520 to display a three-dimensional image.

[0292] The display control unit 530 causes the display unit 520 to display a display image including information indicating the determination result acquired from the determination unit 160 and a two-dimensional image or a three-dimensional image.

[0293] As described above, the display device 500 includes a transmitting / receiving unit 510, which is an example of a receiving unit that receives the output of the imaging unit 11 that images an object and the output of the distance information acquisition unit 13 that projects light onto the object and receives the light reflected from the object, a judgment unit 560 that judges whether a specific object is present based on both the output of the distance information acquisition unit 13 received by the transmitting / receiving unit 510 and the output of the imaging unit 11, and a display control unit 530 that causes the display unit to display different images depending on the presence or absence of the specific object based on the judgment result of the judgment unit 560.

[0294] The specific objects include close objects, high reflectivity objects, distant objects, low reflectivity objects and blurred areas of the image.

[0295] The display device 500 also includes a display control unit 530 that causes the display unit 520 to display a display image including identification information that identifies the specific target and a three-dimensional image 3G determined by the three-dimensional reconstruction processing unit 550, based on the judgment result of a judgment unit 560 that determines whether a specific target is present based on both the output of the imaging unit 11 that captures an image of the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives the light reflected from the target.

[0296] FIG. 21 is a diagram showing an example of the configuration of an information processing system according to the seventh modified example of the embodiment of the present invention.

[0297] The information processing system according to the seventh modified example shown in FIG. 21 includes the imaging device 1, a display device 500, and a server 600.

[0298] The imaging device 1 shown in FIG. 21 is configured similarly to the imaging device 1 shown in FIG. 20, and the display device 500 shown in FIG. 21 is configured similarly to the display device 500 shown in FIG.

[0299] The server 600 shown in FIG. 21 includes a receiving unit 610, an RGB image data acquisition unit 642, a monochrome processing unit 643, a TOF image data acquisition unit 644, a high resolution unit 645, a matching processing unit 646, a reprojection processing unit 647, a semantic segmentation unit 648, a disparity calculation unit 649, a 3D reconstruction processing unit 650, a judgment unit 660, and a transmission unit 680.

[0300] The receiving unit 610 receives the RGB image data of the celestial sphere and the TOF image data of the celestial sphere transmitted from the imaging device 1 via the network 400.

[0301] The RGB image data acquisition unit 642 acquires RGB image data of the entire celestial sphere from the receiving unit 610, and the TOF image data acquisition unit 644 acquires RGB image data of the entire celestial sphere from the receiving unit 610. Otherwise, they are configured in the same manner as the RGB image data acquisition unit 142 and the TOF image data acquisition unit 144 shown in FIG. 4, respectively.

[0302] The monochrome processing unit 643, TOF image data acquisition unit 644, high resolution unit 645, matching processing unit 646, reprojection processing unit 647, semantic segmentation unit 648, parallax calculation unit 649, 3D reconstruction processing unit 650, and judgment unit 660 are configured in the same manner as the monochrome processing unit 143, TOF image data acquisition unit 144, high resolution unit 145, matching processing unit 146, reprojection processing unit 147, semantic segmentation unit 148, parallax calculation unit 149, 3D reconstruction processing unit 150, and judgment unit 160 shown in Figure 4.

[0303] The transmission unit 680 transmits (outputs) the three-dimensional data of the celestial sphere output from the three-dimensional reconstruction processing unit 150, the two-dimensional image information of the celestial sphere output from the RGB image data acquisition unit 142, and the determination result of the determination unit 160 to the display device 500 via the network 400.

[0304] The transmitter / receiver 510 of the display device 510 receives the three-dimensional data of the celestial sphere, the two-dimensional image information, and the determination result of the determination unit 160 transmitted from the server 600.

[0305] Display control unit 530 of display device 510 may acquire RGB image data of the celestial sphere from transmitting / receiving unit 510 and cause display unit 520 to display a two-dimensional image based on the acquired RGB image data of the celestial sphere, or may acquire three-dimensional data of the celestial sphere from transmitting / receiving unit 510 and cause display unit 520 to display a three-dimensional image.

[0306] The display control unit 530 causes the display unit 520 to display a display image including information indicating the determination result acquired from the transmitting / receiving unit 510 and a two-dimensional image or a three-dimensional image.

[0307] As described above, the display device 500 includes a transmission / reception unit 510 that receives the determination result by the determination unit 660 of the server 600 as to whether a specific target is present based on both the output of the imaging unit 11 that captures an image of the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives light reflected from the target, and a display control unit 530 that causes the display unit 520 to display different images depending on the presence or absence of the specific target based on the determination result received by the transmission / reception unit 510. The specific targets include nearby objects, highly reflective objects, distant objects, low reflective objects, and blurred areas of the image.

[0308] The display device 500 also includes a display control unit 530 that causes the display unit 520 to display a display image including identification information that identifies the specific target and a three-dimensional image 3G determined by the three-dimensional reconstruction processing unit 650 based on the judgment result of a judgment unit 660 that determines whether a specific target is present based on both the output of the imaging unit 11 that captures an image of the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives the light reflected from the target.

[0309] FIG. 22 is a diagram for explaining the display contents of the display unit in the fifth to seventh modified examples.

[0310] 22, a three-dimensional image 3G including identification information 3Ga, 3Gb, and 3Gc that identify specific objects is displayed on the display unit 520 by the display control unit 530. 3Ga, 3Gb, and 3Gc may be position identification information that identifies the position of the specific object.

[0311] Although FIG. 22 shows the display unit 520, the display control unit 170 also displays the three-dimensional image 3G including the identification information 3Ga, 3Gb, and 3Gc that identify specific objects on the display unit 20 in the same manner.

[0312] Here, the identification information 3Ga indicates a blind spot and is displayed in pink or the like, the identification information 3Gb indicates a low-reflectivity object and is displayed in orange or the like, and the identification information 3Gc indicates a distant object and is displayed in mosaic or the like.

[0313] These pieces of identification information 3Ga, 3Gb, and 3Gc may all be displayed at the same time, or any one or two of them may be displayed.

[0314] FIG. 23 is a diagram for explaining a three-dimensional image displayed by a display unit in an embodiment of the present invention.

[0315] 23(a) is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is displayed as a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user viewing the celestial sphere image CE displayed as a three-dimensional sphere.

[0316] FIG. 23(b) is a three-dimensional perspective view of FIG. 23(a), and FIG. 23(c) is a diagram showing the predetermined area image when displayed on a display.

[0317] Fig. 23(b) represents the celestial sphere image CE shown in Fig. 23(a) as a three-dimensional sphere CS. If the celestial sphere image CE generated in this way is a 3D sphere CS, then the virtual camera IC is located inside the celestial sphere image CE as shown in Fig. 23(a).

[0318] The predetermined area T in the omnidirectional image CE is an imaging area of ​​the virtual camera IC, and is specified by predetermined area information indicating the imaging direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE.

[0319] Furthermore, zooming of the predetermined region T can also be expressed by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined region image Q is an image of the predetermined region T in the celestial sphere image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the celestial sphere image CE.

[0320] That is, the display control units 170, 530 change the position and orientation of the virtual camera IC located at the viewpoint position for viewing the three-dimensional image 3G, thereby changing the display area of ​​the three-dimensional image 3G displayed on the display units 20, 520.

[0321] Although the above description has been given using an example in which a spherical image is used as the 3D image displayed by the display unit, the same applies when 3D point cloud data is used. A 3D point cloud is placed in a virtual space, and a virtual camera is placed in that virtual space. A 3D image is obtained by projecting the 3D point cloud onto a predetermined projection surface in the virtual space based on predetermined area information indicating the viewpoint position, shooting direction, and angle of view of the virtual camera. In addition, the display area of ​​the 3D image can be changed by changing the viewpoint position and direction of the virtual camera.

[0322] 24 is a determination flow diagram in the fifth to seventh modified examples. Based on the three-dimensional data of the celestial sphere acquired from the three-dimensional reconstruction processing units 150, 550, 650, the determination units 160, 560, 660 determine whether or not there is an area (coordinates) in the three-dimensional data of the celestial sphere where the density of point cloud data is less than a threshold value (step S61).

[0323] If the judgment unit 160, 560, 660 determines in step S61 that there is an area (coordinate) where the density of the point cloud data is below the threshold, it determines whether multiple pixels at the same coordinates as the area where the density of the point cloud data is below the threshold include pixels that have been determined to be distant objects based on the output of the imaging unit 11 in the flow shown in Figure 16, and if a pixel determined to be a distant object is included, it outputs the coordinate position information of the pixel to the display control unit 170, 530 (step S62).

[0324] Based on the coordinate position information of the pixel acquired from the judgment unit 160, 560, 660, the display control unit 170, 530 displays a display image including position identification information 3Gc that identifies the position of a distant object and a three-dimensional image G on the display unit 20, 520, as shown in Figure 22 (step S63), and ends the processing.

[0325] In step S62, if the judgment unit 160, 560, 660 does not include any pixels judged to be distant objects for multiple pixels at the same coordinates as an area where the density of the point cloud data is less than the threshold, it judges whether or not any pixels judged to be low-reflectivity objects are included based on the output of the imaging unit 11 in the flow shown in Figure 16, and if any pixels judged to be low-reflectivity objects are included, it outputs the coordinate position information of the pixels to the display control unit 170, 530 (step S64).

[0326] Based on the pixel coordinate position information acquired from the judgment units 160, 560, 660, the display control units 170, 530 display a display image including position identification information 3Gb that identifies the position of the low-reflectivity object and a three-dimensional image G on the display units 20, 520, as shown in Figure 22 (step S65), and then terminate the processing.

[0327] In step S64, if the area where the density of the point cloud data is less than the threshold and the multiple pixels at the same coordinates do not include any pixels judged to be low-reflectivity objects, the judgment unit 160, 560, 660 judges that the area is a blind spot and outputs the coordinate position information of the pixel to the display control unit 170, 530.

[0328] Based on the pixel coordinate position information acquired from the determination unit 160, 560, 660, the display control unit 170, 530 displays a display image including position identification information 3Ga that identifies the position of the blind spot and a three-dimensional image G on the display unit 20, 520 (step S66), as shown in Fig. 22, and ends the process. Steps S61, S62, and S64 are examples of determination steps, and steps S63, S65, and S66 are examples of display steps.

[0329] As described above, the imaging device 1 and the display device 500 are equipped with a display control unit 170, 530 that causes the display unit 20, 520 to differently display display images including identification information 3Ga, 3Gb, 3Gc that identifies the specific target and a three-dimensional image 3G that is determined by a three-dimensional reconstruction processing unit 150, 550, 650, which is an example of a three-dimensional information determination unit, based on the judgment result of the judgment unit 160, 560, 660, which determines whether a specific target is present based on both the output of the imaging unit 11 that captures an image of the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives the light reflected from the target.

[0330] Specific targets include not only distant objects, low reflectivity objects and blind spots, but also close objects, high reflectivity objects and blurred areas of the image.

[0331] This allows the user to view the three-dimensional image 3G and determine whether the reason the desired three-dimensional image 3G is not displayed is due to a distant object, a low-reflecting object, a blind spot, a close object, a high-reflecting object, or image blur.

[0332] The imaging device 1 and the display device 500 also include a display control unit 170, 530 that causes the display unit 20, 520 to display a three-dimensional image 3G determined based on the output of the distance information acquisition unit 13, which receives light projected onto and reflected from the object. The display control unit 170, 530 causes the display unit 20, 520 to display a display image including the three-dimensional image 3G and position identification information 3Ga, 3Gb, or 3Gc that identifies the position of at least one of a distant object, a low-reflecting object, and a blind spot, based on position information indicating a position in the three-dimensional image 3G that is determined to be at least one of a distant object that is far away from the distance information acquisition unit 13 when the light reflected from the object is received, a low-reflecting object that has a low reflectivity to the projected light, and a blind spot for the distance information acquisition unit 13 when the light reflected from the object is received.

[0333] This allows the user to view the three-dimensional image 3G and determine whether the reason the desired three-dimensional image 3G is not displayed is due to a distant object, a low-reflecting object, or a blind spot, and makes it possible to take appropriate measures such as re-imaging depending on the cause.

[0334] The three-dimensional image 3G is determined by three-dimensional reconstruction processing units 150, 550, and 650, which are examples of three-dimensional information determination units.

[0335] The display control unit 170, 530 may cause the display unit 20, 520 to display a display image including any one of the position identification information 3Ga, 3Gb, and 3Gc and a three-dimensional image 3G based on the position information of any one of distant objects, low-reflecting objects, and blind spots, or may cause the display unit 20, 520 to display a display image including any two or all of the position identification information 3Ga, 3Gb, and 3Gc and a three-dimensional image 3G based on the position information of any two or all of the distant objects, low-reflecting objects, and blind spots.

[0336] When the information processing device is an imaging device 1, the imaging device 1 includes a distance information acquisition unit 13 and a three-dimensional reconstruction processing unit 150, as shown in FIG.

[0337] When the information processing device is a display device 500, as shown in Figures 20 and 21, the display device 500 does not have a distance information acquisition unit 13, and the imaging device 1 has the distance information acquisition unit 13 and transmits the output of the distance information acquisition unit 13 to the display device 500 or the server 600.

[0338] The display device 500 may or may not include a three-dimensional reconstruction processing unit 550 as shown in FIG.

[0339] If the display device 500 does not have a 3D reconstruction processing unit 550, the imaging device 1 may have a 3D reconstruction processing unit 150 and transmit a 3D image to the display device 500, or as shown in Figure 21, the server 600 may have a 3D reconstruction processing unit 650 and transmit a 3D image to the display device 500.

[0340] The display control unit 170, 530 causes the display unit 20, 520 to display a display image including the position identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G based on position information indicating a position where the density of the point cloud data contained in the three-dimensional image 3G is lower than a threshold and which is determined to be at least one of a distant object, a low-reflecting object, or a blind spot.

[0341] This allows the user to view the three-dimensional image 3G and confirm whether the cause of the density of the point cloud data being lower than the threshold is a distant object, a low-reflecting object, or a blind spot.

[0342] The display control unit 170, 530 causes the display unit 20, 520 to display a display image including position identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G based on the output of the imaging unit 11 that images the target and based on position information indicating a position in the three-dimensional image 3G that is determined to be at least one of a distant object, a low-reflecting object, or a blind spot.

[0343] This makes it possible to accurately determine, based on the output of the imaging unit 11, whether the cause of the desired three-dimensional image 3G not being displayed is a distant object, a low-reflecting object, or a blind spot.

[0344] When the information processing device is an imaging device 1, the imaging device 1 includes an imaging unit 11 as shown in Fig. 19. When the information processing device is a display device 500, the display device 500 does not include an imaging unit 11 as shown in Figs. 20 and 21, but the imaging device 1 includes the imaging unit 11 and transmits the output of the imaging unit 11 to the display device 500 or the server 600.

[0345] The imaging device 1 and the display device 500 are provided with a judgment unit 160, 560, 660 that judges a position in the three-dimensional image 3G that is at least one of a distant object, a low-reflecting object, or a blind spot, and the display control unit 170, 530 causes the display unit 20, 520 to display a display image including position identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G based on the judgment result of the judgment unit 160, 560, 660.

[0346] When the information processing device is the imaging device 1, the imaging device 1 includes a determination unit 160 as shown in FIG.

[0347] When the information processing device is a display device 500, the display device 500 may or may not include a determination unit 560 as shown in FIG.

[0348] If the display device 500 does not have a judgment unit 560, the imaging device 1 may have a judgment unit 160 and transmit the judgment result to the display device 500, or the server 600 may have a judgment unit 660 and transmit the judgment result to the display device 500 as shown in Figure 21.

[0349] FIG. 25 is another diagram for explaining the display contents of the display unit in the fifth to seventh modified examples.

[0350] As shown in Figure 25, the display control unit 530 displays a three-dimensional image 3G on the display unit 520, which includes position identification information 3G1 and 3G2 that identify the position of the distance information acquisition unit 13 when it receives light reflected from the object.

[0351] The three-dimensional image 3G is determined based on the output of a distance information acquisition unit 13 located at a first position and the output of a distance information acquisition unit 13 located at a second position different from the first position, and the position identification information 3G1 is an example of first position identification information that identifies the first position, and the position identification information 3G2 is an example of first position identification information that identifies the second position.

[0352] In Figure 25, display unit 520 is shown, but display unit 20 also displays a three-dimensional image 3G including position identification information 3G1, 3G2 that identifies the position of distance information acquisition unit 13 when light reflected from the object is received by display control unit 170.

[0353] The display control unit 170, 530 displays on the display unit 20, 520 a display image including a three-dimensional image 3G and identification information 3Ga, 3Gb, and 3Gc, which are examples of low-density identification information, as shown in FIG. 22, and may also include position identification information 3G1, 3G2 in the display image, which identifies the position of the distance information acquisition unit 13 when it receives light reflected from the target, as shown in FIG. 25.

[0354] FIG. 26 is a flowchart illustrating the processing in the fifth to seventh modified examples.

[0355] The 3D reconstruction processing units 150, 550, and 650 read high-density 3D point cloud data of the entire celestial sphere (step S71), and acquire the origin of the 3D point cloud data as position information indicating the imaging position of the distance information acquisition unit 13 when light reflected from the target is received (step S72).

[0356] The 3D reconstruction processing unit 150, 550, 650 checks whether there is any 3D point cloud data that has been pre-loaded, and if there is no 3D point cloud data that has been pre-loaded, outputs the 3D point cloud data that has been loaded in step S71 and the position information that has been acquired in step S72 to the display control unit 170, 530 (step S73).

[0357] Based on the three-dimensional point cloud data and position information acquired from the three-dimensional reconstruction processing unit 150, 550, 650, the display control unit 170, 530 displays a display image including position identification information 3G1 that identifies the position of the distance information acquisition unit 13 when it receives light reflected from the object and a three-dimensional image 3G on the display unit 20, 520, as shown in Figure 25 (step S74), and then terminates the processing.

[0358] In step S73, if there is pre-loaded three-dimensional point cloud data, the three-dimensional reconstruction processing unit 150, 550, 650 integrates the three-dimensional point cloud data loaded in step S71 with the pre-loaded three-dimensional point cloud data (step S75).

[0359] The 3D reconstruction processing unit 150, 550, 650 calculates the coordinates in the 3D point cloud data integrated in step S75 as position information of the imaging position for each of the origins of the 3D point cloud data read in step S71 and the origins of the 3D point cloud data read in advance, and outputs the 3D point cloud data integrated in step S75 and the calculated multiple pieces of position information to the display control unit 170, 530 (step S76).

[0360] In step S74, based on the three-dimensional point cloud data and multiple pieces of position information acquired from the three-dimensional reconstruction processing unit 150, 550, 650, the display control unit 170, 530 displays on the display unit 20, 520 a display image including multiple pieces of position identification information 3G1, 3G2 that identify the position of the distance information acquisition unit 13 when it receives light reflected from the object, and a three-dimensional image 3G, as shown in Figure 25.

[0361] FIG. 27 is another flowchart illustrating the processing in the fifth to seventh modified examples.

[0362] The 3D reconstruction processing units 150, 550, and 650 read high-density three-dimensional point cloud data of the celestial sphere (step S81), and the determination units 160, 560, and 660 execute steps S61, S62, and S64 of the flow illustrated in FIG. 24 based on the three-dimensional data of the celestial sphere acquired from the 3D reconstruction processing units 150, 550, and 650, to extract low-density parts where the density of the point cloud data is lower than a threshold (step S82).

[0363] When the virtual camera IC shown in FIG. 23 is located at the position of the position identification information 3G1 or 3G2 shown in FIG. 25, the display control unit 170, 530 executes steps S63, S65 and S66 of the flow shown in FIG. 24 to change the orientation of the virtual camera IC so that at least one of the identification information 3Ga, 3Gb and 3Gc, which are examples of low-density identification information shown in FIG. 22, is included in the displayed image (step S83).

[0364] As described above, the imaging device 1 and the display device 500 are equipped with display control units 170, 530 that cause the display units 20, 520 to display a three-dimensional image 3G determined based on the output of the distance information acquisition unit 13, and the display control units 170, 530 cause the display units 20, 520 to display a display image including position identification information 3G1, 3G2 that identifies the position of the distance information acquisition unit 13 when it receives light reflected from the object, and the three-dimensional image 3G, based on position information that indicates the position of the distance information acquisition unit 13 when it receives light reflected from the object.

[0365] This allows the positional relationship between the imaging position, which indicates the position of distance information acquisition unit 13 when light reflected from the object is received, and the specific object to be grasped in the three-dimensional image 3G.

[0366] The three-dimensional image 3G and position information are determined by the three-dimensional reconstruction processors 150, 550, and 650.

[0367] When the information processing device is an imaging device 1, the imaging device 1 includes a distance information acquisition unit 13 and a three-dimensional reconstruction processing unit 150, as shown in FIG.

[0368] When the information processing device is a display device 500, as shown in Figures 20 and 21, the display device 500 does not have a distance information acquisition unit 13, and the imaging device 1 has the distance information acquisition unit 13 and transmits the output of the distance information acquisition unit 13 to the display device 500 or the server 600.

[0369] The display device 500 may or may not include a three-dimensional reconstruction processing unit 550 as shown in FIG.

[0370] If the display device 500 does not have a 3D reconstruction processing unit 550, the imaging device 1 may have a 3D reconstruction processing unit 150 and transmit the 3D image and position information to the display device 500, or as shown in Figure 21, the server 600 may have a 3D reconstruction processing unit 650 and transmit the 3D image and position information to the display device 500.

[0371] The display control unit 170, 530 causes the display unit 20, 520 to display a display image including identification information 3Ga, 3Gb, 3Gc, which is an example of low-density identification information for identifying a region, and the three-dimensional image 3G, based on region information indicating a region where the density of point cloud data in the three-dimensional image 3G is lower than a threshold value.

[0372] In this case, since it is possible to grasp the positional relationship between the imaging position and the area where the density of the point cloud data is lower than the threshold, it is possible to identify the cause of the density of the point cloud data being lower than the threshold. For example, if the area is farther away from the imaging position, it is determined that the cause is a distant object; if the area is in a blind spot of the imaging position, it is determined that the cause is a blind spot; and if the area is neither a distant object nor a blind spot, it is determined that the cause is a low-reflecting object.

[0373] The display control unit 170, 530 changes the display area of ​​the three-dimensional image 3G displayed on the display unit 20, 520 by changing the position and orientation of the virtual camera IC located at the viewpoint position for viewing the three-dimensional image 3G.

[0374] When the position of the virtual camera IC is at position 3G1 or 3G2 identified by the position identification information, the display control unit 170 or 530 changes the orientation of the virtual camera IC to a predetermined orientation. The predetermined orientation is an area that includes, in the display area, areas that may cause re-imaging, such as low-density point cloud areas, areas that meet pre-set conditions, such as areas to be checked during a site inspection, or any areas focused on by the photographer or other inspectors. Examples of areas to be checked during a site inspection, such as a construction site, include areas where changes occur continuously (material storage areas), the positions of each object in the permanent structure (the building itself), the gap distances between objects, spaces for new installations, temporary structures (such as material storage areas and scaffolding that are installed and removed during the construction process), storage areas for heavy equipment (forks and cranes), workspaces (rotation ranges, delivery routes), and resident traffic routes (detours during construction).

[0375] This makes it possible to direct the gaze of a user positioned at an imaging position to a specific object that the user wants to see on-site.

[0376] The display control unit 170, 530 changes the orientation of the virtual camera IC so that predetermined coordinates in the 3D image 3G or low-density areas where the density of point cloud data is lower than a threshold are included in the display area. The predetermined coordinates do not specify the image, and are maintained even if, for example, the image at the predetermined coordinates changes before and after integrating the 3D point cloud data in step S75 of FIG. 26 .

[0377] This makes it possible to direct the line of sight of the user positioned at the imaging position to a specific object that is a low-density area in the three-dimensional image 3G.

[0378] The display control unit 170, 530 causes the display unit 20, 520 to display a three-dimensional image 3G determined based on the output of the distance information acquisition unit 13 located at a first position and the output of the distance information acquisition unit 13 located at a second position different from the first position, and causes the display unit 20, 520 to display a display image including first position identification information 3G1 that identifies the first position and second position identification information 3G2 that identifies the second position, and the three-dimensional image 3G.

[0379] This allows the positional relationship between the first and second imaging positions and the specific object to be grasped in the three-dimensional image 3G.

[0380] ●Summary● As described above, the imaging device 1 according to one embodiment of the present invention comprises an imaging unit 11 that images an object, a projection unit 12 that projects light onto the object, a distance information acquisition unit 13 (an example of a light receiving unit) that receives light reflected from the object, a judgment unit 160 that determines whether a highly reflective object is present based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a display control unit 170 that causes the display units 20, 520 to display different images depending on whether a highly reflective object is present or not.

[0381] This allows the photographer to accurately confirm that a highly reflective object such as a mirror is included in the captured image, distinguishing it from nearby objects and the effects of external light.

[0382] The imaging device 1 includes a display unit 20. This allows the photographer to reliably confirm that a highly reflective object is included in the captured image.

[0383] The display control unit 170 displays different images at different positions on the display units 20 and 520 depending on the position of the highly reflective object, thereby allowing the photographer to confirm the position of the highly reflective object.

[0384] The display unit 20 includes a plurality of display units 20A, 20a, and the display control unit 170 controls the display unit closest to the highly reflective object to display different information depending on whether or not the object is present, thereby enabling the photographer to reliably confirm the position of the highly reflective object.

[0385] The display control unit 170 causes the display unit 20, 520 to display the image information G captured by the imaging unit 11, and also causes the display unit 20, 520 to display a display image including identification information for identifying the highly reflective object and the image information G. This allows the photographer to reliably confirm the position of the highly reflective object.

[0386] The judgment unit 160 judges that a highly reflective object is present when the charge storage amount is saturated, which is an example of a pixel where the charge storage amount due to light received by the distance information acquisition unit 13 is greater than a predetermined value, and the image information captured by the imaging unit matches the model image information, which is an example of reference information indicating a highly reflective object.

[0387] This allows the photographer to accurately confirm that a highly reflective object is included in the captured image, distinguishing it from nearby objects and the effects of external light.

[0388] The imaging device 1 acquires distance information to the target based on the light received by the distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired distance information is not being acquired is not due to a nearby object or external light, but due to a highly reflective object.

[0389] The imaging device 1 includes a transmitter / receiver 180, which is an example of an output unit that outputs three-dimensional information determined based on distance information acquired from the distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired three-dimensional information is not being acquired is not due to a nearby object or external light, but due to a highly reflective object.

[0390] An imaging processing method according to one embodiment of the present invention includes an imaging step of imaging an object using the imaging unit 11, a projection step of projecting light onto the object using the projection unit 12, a light receiving step of receiving light reflected from the object using the distance information acquisition unit 13, a determination step of determining whether a highly reflective object is present using the determination unit 160, 560, 660 based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a display step of causing the display control unit 170, 530 to display different images on the display unit 20, 520 depending on whether a highly reflective object is present or not.

[0391] The imaging device 1 and display device 500, which are examples of information processing devices according to one embodiment of the present invention, are equipped with display control units 170, 530 that cause the display units 20, 520 to display different images depending on the presence or absence of a highly reflective object, based on the judgment results of judgment units 160, 560, 660 that determine whether a highly reflective object is present based on both the output of an imaging unit 11 that captures an image of an object and the output of a distance information acquisition unit 13 that projects light onto the object and receives the light reflected from the object.

[0392] A display device 500, which is an example of an information processing device according to an embodiment of the present invention, includes a transceiver 510, which is an example of a receiver, that receives a determination result made by the determination unit 160 of the imaging device 1 or the determination unit 660 of the server 600 as to whether a specific target is present, based on both the output of the imaging unit 11 that captures an image of the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives light reflected from the target, and a display control unit 530 that causes the display unit 520 to display different images depending on the presence or absence of the specific target, based on the determination result received by the transceiver 510. The specific targets include nearby objects, highly reflective objects, distant objects, low reflective objects, blind spots, and blurred areas of the image.

[0393] A display device 500, which is an example of an information processing device according to an embodiment of the present invention, includes a transceiver 510, which is an example of a receiver that receives the output of an imaging unit 11 that images an object and the output of a distance information acquisition unit 13 that projects light onto the object and receives light reflected from the object, a determination unit 560 that determines whether a specific object is present based on both the output of the distance information acquisition unit 13 received by the transceiver 510 and the output of the imaging unit 11, and a display control unit 530 that causes the display unit to display different images depending on the presence or absence of the specific object based on the determination result of the determination unit 560. The specific objects include nearby objects, highly reflective objects, distant objects, low reflective objects, blind spots, and blurred image areas.

[0394] The imaging device 1 and display device 500, which are examples of an information processing device according to an embodiment of the present invention, include a display control unit 170, 530 that causes the display unit 20, 520 to display a display image including identification information 3Ga, 3Gb, 3Gc that identifies the specific object and a three-dimensional image 3G, based on the determination result by a determination unit 160, 560 that determines whether a specific object is present based on both the output of an imaging unit 11 that images an object and the output of a distance information acquisition unit 13 that projects light onto the object and receives light reflected from the object. The specific object includes not only distant objects, low-reflectivity objects, and blind spots, but also close objects, high-reflectivity objects, and blurred areas of the image.

[0395] The three-dimensional image 3G is determined by the three-dimensional reconstruction processing units 150, 550, and 650, which are examples of three-dimensional information determination units, based on the output of the distance information acquisition unit 13.

[0396] This allows the user to view the three-dimensional image 3G and determine whether the reason the desired three-dimensional image 3G is not displayed is due to a distant object, a low-reflecting object, a blind spot, a close object, a high-reflecting object, or image blur.

[0397] The imaging device 1 and display device 500, which are examples of information processing devices according to one embodiment of the present invention, include a display control unit 170, 530 that causes the display unit 20, 520 to display a display image including position identification information that identifies a position and two-dimensional image information G captured by the imaging unit 11 that captures the target, based on position information indicating a position determined by a judgment unit 160, 560 as to whether the output of a distance information acquisition unit 13, which projects light onto a target and receives light reflected from the target, is above or below a threshold value.

[0398] This allows the causes of the desired output not being obtained to be identified by checking the two-dimensional image G for positions where the output of the distance information acquisition unit 13 is above or below the threshold, i.e., positions where the output of the distance information acquisition unit 13 is too strong or too weak and the desired output cannot be obtained.

[0399] The imaging device 1 and display device 500, which are examples of information processing devices according to one embodiment of the present invention, are equipped with a display control unit 170, 530 that causes the display unit 20, 520 to display a display image including position identification information that identifies the position and two-dimensional image information G captured by the imaging unit 11 that captures the target, based on position information indicating the position determined by the judgment unit 160, 560 that distance information to the target cannot be acquired based on the output of the distance information acquisition unit 13 that projects light onto the target and receives light reflected from the target.

[0400] This allows the cause of the inability to acquire distance information to the target to be confirmed by checking the position where distance information to the target cannot be acquired in the two-dimensional image G.

[0401] The judgment units 160, 560, and 660 may determine that distance information to the target cannot be acquired not only when the output of the distance information acquisition unit 13 is above or below the threshold, but also when image blur is detected based on the output of the distance information acquisition unit 13.

[0402] In the above, when the information processing device is the imaging device 1, the imaging device 1 includes an imaging unit 11, a distance information acquisition unit 13, a 3D reconstruction processing unit 150, and a determination unit 160, as shown in FIG.

[0403] When the information processing device is a display device 500, as shown in Figures 20 and 21, the display device 500 does not have an imaging unit 11 and a distance information acquisition unit 13, and the imaging device 1 has an imaging unit 11 and a distance information acquisition unit 13, and transmits the outputs of these to the display device 500 or the server 600.

[0404] The display device 500 may or may not include a determination unit 560 as shown in FIG.

[0405] If the display device 500 does not have a judgment unit 560, the imaging device 1 may have a judgment unit 160 and transmit the judgment result to the display device 500, or the server 600 may have a judgment unit 660 and transmit the judgment result to the display device 500 as shown in Figure 21.

[0406] Similarly, the display device 500 may or may not include a three-dimensional reconstruction processor 550 as shown in FIG.

[0407] If the display device 500 does not have a 3D reconstruction processing unit 550, the imaging device 1 may have a 3D reconstruction processing unit 150 and transmit a 3D image to the display device 500, or as shown in Figure 21, the server 600 may have a 3D reconstruction processing unit 650 and transmit a 3D image to the display device 500.

[0408] As described above, the imaging device 1 according to one embodiment of the present invention comprises an imaging unit 11 that images an object, a projection unit 12 that projects light onto the object, a distance information acquisition unit 13 (an example of a light receiving unit) that receives light reflected from the object, a determination unit 160 that determines whether a distant object or a low-reflecting object is present based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a display control unit 170 that causes the display unit 20, 520 to display different images depending on whether a distant object or a low-reflecting object is present.

[0409] This allows the photographer to accurately confirm that distant objects or low-reflectivity objects such as black objects are included in the captured image.

[0410] The imaging device 1 includes a display unit 20. This allows the photographer to reliably confirm that a distant object or a low-reflectivity object is included in the captured image.

[0411] The display control unit 170 displays different images at different positions on the display units 20 and 520 depending on the position of the distant object or low-reflectivity object, thereby allowing the photographer to confirm the position of the distant object or low-reflectivity object.

[0412] The display unit 20 includes a plurality of display units 20A, 20a, and the display control unit 170 controls the display unit that is closer to the distant object or low-reflectivity object to display different information depending on whether the object is present or not. This allows the photographer to reliably confirm the position of the distant object or low-reflectivity object.

[0413] The display control unit 170 causes the display unit 20, 520 to display the image information G captured by the imaging unit 11, and also causes the display unit 20, 520 to display a display image including identification information for identifying the distant object or low-reflectivity object and the image information G. This allows the photographer to reliably confirm the position of the distant object or low-reflectivity object.

[0414] When the amount of charge stored in a pixel due to light received by distance information acquisition unit 13 is equal to or less than a threshold, determination unit 160 determines whether the object is a low-reflecting object or a distant object based on the output of imaging unit 11. This allows the photographer to accurately confirm whether the captured image contains a low-reflecting object or a distant object.

[0415] The determination unit 160 determines that a low-reflectivity object is present when the amount of charge stored in the pixel due to the light received by the distance information acquisition unit 13 is equal to or less than a threshold and the amount of charge stored in the pixel of the imaging unit 11 is equal to or less than a threshold. This allows the photographer to accurately confirm that a low-reflectivity object is included in the captured image.

[0416] The judgment unit 160 judges that there is a distant object when the amount of charge stored in the pixel due to the light received by the distance information acquisition unit 13 is below a threshold, the amount of charge stored in the pixel of the imaging unit 11 is above a threshold, and the distance judged based on the pixel is above a threshold.

[0417] This allows the photographer to accurately confirm that the distant object is included in the captured image.

[0418] Imaging device 1 acquires distance information to the target based on the light received by distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired distance information is not being acquired is because of a distant object or a low-reflecting object.

[0419] Imaging device 1 includes transmitting / receiving unit 180, which is an example of an output unit that outputs three-dimensional information determined based on distance information acquired from distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired three-dimensional information is not being acquired is because of a distant object or a low-reflecting object.

[0420] An imaging processing method according to one embodiment of the present invention includes an imaging step of imaging an object using the imaging unit 11, a projection step of projecting light onto the object using the projection unit 12, a light receiving step of receiving light reflected from the object using the distance information acquisition unit 13, a determination step of determining whether a distant object or a low-reflecting object is present using the determination unit 160, 560, 660 based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a display step of causing the display control unit 170, 530 to display different images on the display unit 20, 520 depending on whether a distant object or a low-reflecting object is present.

[0421] The imaging device 1 and display device 500, which are examples of information processing devices according to one embodiment of the present invention, are equipped with display control units 170, 530 that cause the display units 20, 520 to display different images depending on the presence or absence of a distant object or a low-reflectivity object, based on the judgment results of judgment units 160, 560, 660 that determine whether or not there is a distant object or a low-reflectivity object based on both the output of the imaging unit 11 that captures an image of the object and the output of the distance information acquisition unit 13 that projects light onto the object and receives the light reflected from the object.

[0422] As described above, the imaging device 1 according to one embodiment of the present invention comprises an imaging unit 11 that images an object, a projection unit 12 that projects light onto the object, a distance information acquisition unit 13 (an example of a light receiving unit) that receives light reflected from the object, a judgment unit 160 that judges whether or not there is image blur based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a display control unit 170 that causes the display unit 20, 520 to display different images depending on whether or not there is image blur.

[0423] This allows the photographer to accurately confirm that the captured image contains image blur.

[0424] The imaging device 1 includes a display unit 20. This allows the photographer to reliably confirm that the captured image contains image blur.

[0425] The display control unit 170 displays different images at different positions on the display units 20 and 520 according to the position of the blur in the image, thereby enabling the photographer to confirm the position of the blur in the image.

[0426] The display unit 20 includes a plurality of display units 20A, 20a, and the display control unit 170 controls the display unit that is closer to the position of the blurred image to display different information depending on whether or not an object is present, thereby enabling the photographer to reliably confirm the position of the blurred image.

[0427] The display control unit 170 causes the display unit 20, 520 to display the image information G captured by the imaging unit 11, and also causes the display unit 20, 520 to display a display image including identification information for identifying blurring of the image and the image information G. This allows the photographer to reliably confirm the position of blurring of the image.

[0428] The determining unit 160 detects the edge of the image based on the image information captured by the imaging unit 11, and determines that the image is blurred if there is a pixel shift due to the light received by the distance information acquiring unit 13.

[0429] This allows the photographer to accurately confirm that the captured image contains image blur.

[0430] Imaging device 1 acquires distance information to the target based on the light received by distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired distance information is not acquired is due to image blur.

[0431] Imaging device 1 includes transmitting / receiving unit 180, which is an example of an output unit that outputs three-dimensional information determined based on distance information acquired from distance information acquisition unit 13. In this case, the photographer can confirm that the reason the desired three-dimensional information is not acquired is due to image blur.

[0432] An imaging processing method according to one embodiment of the present invention includes an imaging step of imaging an object using the imaging unit 11, a projection step of projecting light onto the object using the projection unit 12, a light receiving step of receiving light reflected from the object using the distance information acquisition unit 13, a determination step of determining whether or not there is image blur using the determination unit 160, 560, 660 based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a display step of causing the display unit 20, 520 to display different images using the display control unit 170, 530 depending on whether or not there is image blur.

[0433] The imaging device 1 and display device 500, which are examples of information processing devices according to one embodiment of the present invention, are equipped with display control units 170, 530 that cause the display units 20, 520 to display different images depending on whether or not the image is blurred, based on the determination results of determination units 160, 560, 660 that determine whether or not the image is blurred based on both the output of the imaging unit 11 that images the object and the output of the distance information acquisition unit 13 that projects light onto the object and receives the light reflected from the object.

[0434] The imaging device 1 and the display device 500, which are examples of an information processing device according to one embodiment of the present invention, are equipped with a display control unit 170, 530 that causes the display unit 20, 520 to display a three-dimensional image 3G determined based on the output of a distance information acquisition unit 13, which is an example of a light receiving unit that receives light projected onto an object and reflected from the object. The display control unit 170, 530 causes the display unit 20, 520 to display a display image including position identification information 3Ga, 3Gb, 3Gc that identifies the positions of at least one of a distant object, a low-reflecting object, and a blind spot, based on position information indicating a position in the three-dimensional image 3G that is determined to be at least one of a distant object that is far away from the distance information acquisition unit 13 when light reflected from the object is received, a low-reflecting object that has a low reflectivity to the projected light, and a blind spot for the distance information acquisition unit 13 when light reflected from the object is received.

[0435] This allows the user to view the three-dimensional image 3G and determine whether the reason the desired three-dimensional image 3G is not displayed is due to a distant object, a low-reflecting object, or a blind spot, and makes it possible to take appropriate measures such as re-imaging depending on the cause.

[0436] The three-dimensional image 3G is determined by three-dimensional reconstruction processing units 150, 550, and 650, which are examples of three-dimensional information determination units.

[0437] The display control unit 170, 530 may cause the display unit 20, 520 to display a display image including any one of the position identification information 3Ga, 3Gb, and 3Gc and a three-dimensional image 3G based on the position information of any one of distant objects, low-reflecting objects, and blind spots, or may cause the display unit 20, 520 to display a display image including any two or all of the position identification information 3Ga, 3Gb, and 3Gc and a three-dimensional image 3G based on the position information of any two or all of the distant objects, low-reflecting objects, and blind spots.

[0438] When the information processing device is an imaging device 1, the imaging device 1 includes a distance information acquisition unit 13 and a three-dimensional reconstruction processing unit 150, as shown in FIG.

[0439] When the information processing device is a display device 500, as shown in Figures 20 and 21, the display device 500 does not have a distance information acquisition unit 13, and the imaging device 1 has a distance information acquisition unit 13 and transmits the output of the distance information acquisition unit 13 to the display device 500 or the server 600.

[0440] The display device 500 may or may not include a 3D reconstruction processing unit 550. If the display device 500 does not include a 3D reconstruction processing unit 550, the imaging device 1 may include a 3D reconstruction processing unit 150 and transmit 3D images to the display device 500, or, as shown in FIG. 21, the server 600 may include a 3D reconstruction processing unit 650 and transmit 3D images to the display device 500.

[0441] The display control unit 170, 530 controls the display unit 20, 520 to display a display image including the position identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G based on the position information indicating a position where the density of the point cloud data included in the three-dimensional image 3G is lower than a threshold and is determined to be at least one of a distant object, a low-reflecting object, or a blind spot.

[0442] This allows the user to view the three-dimensional image 3G and confirm whether the cause of the density of the point cloud data being lower than the threshold is a distant object, a low-reflecting object, or a blind spot.

[0443] The display control unit 170, 530 causes the display unit 20, 520 to display a display image including position identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G based on the output of the imaging unit 11 that images the target and based on position information indicating a position in the three-dimensional image 3G that is determined to be at least one of a distant object, a low-reflecting object, or a blind spot.

[0444] This makes it possible to accurately determine, based on the output of the imaging unit 11, whether the cause of the desired three-dimensional image 3G not being displayed is a distant object, a low-reflecting object, or a blind spot.

[0445] When the information processing device is an imaging device 1, the imaging device 1 includes an imaging unit 11 as shown in Fig. 19. When the information processing device is a display device 500, the display device 500 does not include an imaging unit 11 as shown in Figs. 20 and 21, but the imaging device 1 includes the imaging unit 11 and transmits the output of the imaging unit 11 to the display device 500 or the server 600.

[0446] The imaging device 1 and the display device 500 are provided with a judgment unit 160, 560 that judges the position in the three-dimensional image 3G as being at least one of a distant object, a low-reflecting object, or a blind spot, and the display control unit 170, 530 causes the display unit 20, 520 to display a display image including the position identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G based on the judgment result of the judgment unit 160, 560.

[0447] When the information processing device is the imaging device 1, the imaging device 1 includes a determination unit 160 as shown in FIG.

[0448] When the information processing device is a display device 500, the display device 500 may or may not include a determination unit 560 as shown in FIG.

[0449] If the display device 500 does not have a judgment unit 560, the imaging device 1 may have a judgment unit 160 and transmit the judgment result to the display device 500, or the server 600 may have a judgment unit 660 and transmit the judgment result to the display device 500 as shown in Figure 21.

[0450] The display control unit 170, 530 changes the display area of ​​the three-dimensional image 3G displayed on the display unit 20, 520 by changing the position and orientation of the virtual camera IC located at the viewpoint position for viewing the three-dimensional image 3G.

[0451] The imaging device 1 and display device 500, which are examples of information processing devices according to one embodiment of the present invention, are equipped with a display control unit 170, 530 that causes the display unit 20, 520 to display a three-dimensional image 3G determined based on the output of the distance information acquisition unit 13, which is an example of a light receiving unit that receives light projected onto an object and reflected from the object. The display control unit 170, 530 causes the display unit 20, 520 to display a display image including position identification information 3G1, 3G2 that identifies the position of the distance information acquisition unit 13 when it received the light reflected from the object, and the three-dimensional image 3G, based on position information indicating the position of the distance information acquisition unit 13 when it received the light reflected from the object.

[0452] This makes it possible to grasp the positional relationship between the imaging position, which indicates the position of distance information acquisition unit 13 when light reflected from the target is received, and the specific target in the three-dimensional image 3G. In other words, it is possible to easily compare the positional relationship between the imaging position and the specific target at the site where the three-dimensional image was acquired with the positional relationship between the imaging position and the specific target in the three-dimensional image.

[0453] The three-dimensional image 3G and the position information are determined by three-dimensional reconstruction processing units 150, 550, and 650, which are examples of three-dimensional information determination units.

[0454] When the information processing device is the imaging device 1, the imaging device 1 includes a distance information acquisition unit 13 and a three-dimensional reconstruction processing unit 150.

[0455] When the information processing device is a display device 500, the display device 500 does not have a distance information acquisition unit 13, and the imaging device 1 has a distance information acquisition unit 13 and transmits the output of the distance information acquisition unit 13 to the display device 500 or the server 600.

[0456] The display device 500 may or may not include a 3D reconstruction processing unit 550. If the display device 500 does not include a 3D reconstruction processing unit 550, the imaging device 1 may include a 3D reconstruction processing unit 150 and transmit 3D images and position information to the display device 500, or the server 600 may include a 3D reconstruction processing unit 650 and transmit 3D images and position information to the display device 500.

[0457] The display control unit 170, 530 causes the display unit 20, 520 to display a display image including identification information 3Ga, 3Gb, 3Gc, which is an example of low-density identification information for identifying a region, and the three-dimensional image 3G, based on region information indicating a region where the density of point cloud data in the three-dimensional image 3G is lower than a threshold value.

[0458] In this case, since it is possible to grasp the positional relationship between the imaging position and the area where the density of the point cloud data is lower than the threshold, it is possible to identify the cause of the density of the point cloud data being lower than the threshold. For example, if the area is farther away from the imaging position, it is determined that the cause is a distant object; if the area is in a blind spot of the imaging position, it is determined that the cause is a blind spot; and if the area is neither a distant object nor a blind spot, it is determined that the cause is a low-reflecting object.

[0459] The display control unit 170, 530 changes the display area of ​​the three-dimensional image 3G displayed on the display unit 20, 520 by changing the position and orientation of the virtual camera IC located at the viewpoint position for viewing the three-dimensional image 3G.

[0460] When the position of the virtual camera IC is at the position 3G1 or 3G2 identified by the position identification information, the display control unit 170 or 530 changes the orientation of the virtual camera IC to a predetermined orientation.

[0461] This makes it possible to direct the gaze of a user positioned at an imaging position to a specific object that the user wants to see on-site.

[0462] The display control units 170 and 530 change the orientation of the virtual camera IC so that predetermined coordinates or low-density areas in the three-dimensional image 3G where the density of point cloud data is lower than a threshold value are included in the display area.

[0463] This makes it possible to direct the line of sight of the user positioned at the imaging position to predetermined coordinates or a specific object that is a low-density area in the three-dimensional image 3G.

[0464] The display control unit 170, 530 causes the display unit 20, 520 to display a three-dimensional image 3G determined based on the output of the distance information acquisition unit 13 located at a first position and the output of the distance information acquisition unit 13 located at a second position different from the first position, and causes the display unit 20, 520 to display a display image including first position identification information 3G1 that identifies the first position and second position identification information 3G2 that identifies the second position, and the three-dimensional image 3G.

[0465] This allows the positional relationship between the first and second imaging positions and the specific object to be grasped in the three-dimensional image 3G. [Explanation of symbols]

[0466] 1. Imaging device (an example of an information processing device) 3G 3D images 3Ga, 3Gb, 3Gc identification information 3G1, 3G2 location identification information 10. Cabinet 11 Imaging unit 11a, 11A image sensor 11b, 11B fisheye lens 12 Projection section 12a, 12A light source section 12b, 12B wide-angle lens 13 Distance information acquisition unit (an example of a light receiving unit) 13a, 13A TOF sensor 13b, 13B wide-angle lens 14 Processing circuit 15 Shooting switch 20 Display section 20A, 20a display section 111 Other imaging units 150, 550, 650 3D reconstruction processing unit (an example of a 3D information determination unit) 160, 560, 660 Judgment section 170 Display control unit (an example of an output unit) 180 Transmitting and receiving unit (an example of an output unit) 300 External device (example of output destination) 500 Display device (output destination, example of information processing device) 520 Display unit (example of output destination) 530 Display control unit (an example of an output unit) 600 servers L Sync signal line

Claims

1. a display control unit that causes a display unit to display a three-dimensional image determined based on an output of a light receiving unit that receives light projected onto an object and reflected from the object; The display control unit An information processing device that displays on the display unit a display image including the three-dimensional image and position identification information that identifies the position of at least one of a distant object that is far away from the light receiving unit when light reflected from the object is received, a low-reflecting object that has a low reflectivity to the projected light, and a blind spot for the light receiving unit when light reflected from the object is received, based on position information indicating a position that is determined to be at least one of the distant object, the low-reflecting object, and the blind spot for the light receiving unit when light reflected from the object is received.

2. The display control unit 2. The information processing device according to claim 1, wherein the display image is displayed on the display unit based on position information indicating a position where the density of point cloud data contained in the three-dimensional image is lower than a threshold and is determined to be at least one of the distant object, the low-reflecting object, or the blind spot.

3. The display control unit 3. An information processing device according to claim 1, wherein the display image is displayed on the display unit based on position information indicating a position in the three-dimensional image that is determined to be at least one of the distant object, the low-reflecting object, or the blind spot, based on the output of an imaging unit that images the object.

4. a determination unit that determines a position in the three-dimensional image that is at least one of the distant object, the low-reflection object, or the blind spot; 4. The information processing apparatus according to claim 1, wherein the display control unit causes the display unit to display the display image based on the determination result of the determination unit.

5. The display control unit changes the display area of ​​the three-dimensional image to be displayed on the display unit by changing the position and orientation of a virtual camera located at the viewpoint position from which the three-dimensional image is viewed.

6. An information processing method for displaying a three-dimensional image on a display unit, the three-dimensional image being determined based on an output of a light receiving unit that receives light projected onto an object and reflected from the object, the method comprising: a determining step of determining, based on an output of an imaging unit that images the object, a position in the three-dimensional image that is at least one of a distant object that is far away from the light receiving unit when light reflected from the object is received, a low-reflectivity object that has a low reflectivity for the projected light, and a blind spot for the light receiving unit when light reflected from the object is received; a display step of displaying, on the display unit, a display image including position identification information that identifies the position of at least one of the distant object, the low-reflecting object, and the blind spot based on position information indicating the position determined in the determination step, and the three-dimensional image; An information processing method comprising:

Images