Imaging apparatus and imaging processing method

Abstract

To provide an imaging apparatus and an imaging processing method capable of accurately confirming the presence / absence of reflection of an object on a picked-up image.SOLUTION: The imaging device 1 includes the imaging unit 11 that images a target, the projection unit 12 that projects light onto the target, the distance information acquisition unit 13 that receives light reflected from the target, and the display control unit 170 that causes the display units 20 and 520 to perform different displays according to the presence or absence of a specific target object determined based on an output of the distance information acquisition unit 13 and an output of the imaging unit 11.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an imaging device and an imaging processing method. [Background technology]

[0002] Patent Document 1 describes a distance measuring device that can stably and accurately measure the distance to an object.

[0003] Patent Document 2 describes an imaging device that performs image processing to reduce the influence of a finger or other object that is captured in the image. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide an imaging device and an imaging processing method that can accurately check whether or not an object is captured in a captured image. [Means for solving the problem]

[0005] The imaging device of the present invention comprises an imaging unit that images an object, a projection unit that projects light onto the object, a light receiving unit that receives light reflected from the object, and a display control unit that causes the display unit to display different images depending on the presence or absence of a specific object determined based on the output of the light receiving unit and the output of the imaging unit. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide an imaging device and an imaging processing method that can accurately check whether or not an object is captured in a captured image. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of the appearance of an imaging device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating the configuration of the imaging device according to the embodiment. [Figure 3] FIG. 3 is a diagram for explaining a usage state of the imaging device in the embodiment. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of a processing block of a processing circuit according to the embodiment. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the processing circuit of the imaging device in the embodiment. [Figure 6] FIG. 6 is a flowchart showing the generation of spherical image data in the embodiment. [Figure 7] FIG. 7 is a flowchart of nearby object determination in the same embodiment. [Figure 8] FIG. 8 is a diagram for explaining the display content of the display unit in the embodiment. [Figure 9] FIG. 9 is a diagram showing the appearance of an imaging device according to a modified example of the embodiment. [Figure 10] FIG. 10 is a diagram showing the configuration of a processing block of a processing circuit in a modified example. [Figure 11] FIG. 11 is a diagram showing the appearance of an imaging device according to a second modified example of the embodiment of the present invention. [Figure 12] FIG. 12 is a diagram showing the configuration of a processing block of a processing circuit in the second modified example. [Figure 13] FIG. 13 is a flowchart of nearby object determination in the second modified example. [Figure 14] FIG. 14 is a diagram illustrating the configuration of an imaging device according to a third modified example of the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of an imaging device and an imaging processing method will be described in detail with reference to the accompanying drawings.

[0009] Fig. 1 is a diagram showing an example of the appearance of an imaging device according to an embodiment of the present invention. Fig. 2 is a diagram for explaining the configuration of the imaging device. Fig. 2 shows the internal configuration of the imaging device of Fig. 1.

[0010] The imaging device 1 is an example of an information processing device that outputs three-dimensional information determined based on received light, and has an imaging unit (camera) 11, a projection unit (corresponding to the light-emitting unit of a distance sensor) 12 that projects light other than visible light, and a distance information acquisition unit (corresponding to the light-receiving unit of a distance sensor) 13 that acquires distance information based on the light projected by the projection unit 12, all of which are provided integrally with a housing 10. Each unit is electrically connected to a processing circuit 14 inside the housing 10 by a synchronization signal line L, and operates synchronously with each other.

[0011] The photographing switch 15 allows the user to input a photographing instruction signal to the processing circuit 14. The display unit 20 displays content corresponding to the output signal of the processing circuit 14 and is configured with an LCD screen or the like. The display unit 20 may also be configured with a touch panel or the like and be configured to accept user input. Based on the photographing instruction, the processing circuit 14 controls each component to acquire RGB image and distance information data, and performs processing to reconstruct the acquired distance information data into high-density 3D point cloud data based on the RGB image and distance information data. The distance information data can be used as is to construct 3D point cloud data, but in this case, the accuracy of the 3D point cloud data is limited by the number of pixels (resolution) of the distance information acquisition unit 13. This example also illustrates processing to reconstruct it into high-density 3D point cloud data. The reconstructed data is output to an external PC or the like via a portable recording medium or communication, and is used to display a 3D reconstruction model.

[0012] Power is supplied to each unit and the processing circuit 14 from a battery housed inside the housing 10. Alternatively, power may be supplied from outside the housing 10 via a connection cord.

[0013] The imaging unit 11 includes imaging elements 11a and 11A, fisheye lenses (wide-angle lenses) 11b and 11B, etc. The projection unit 12 includes light sources 12a and 12A, wide-angle lenses 12b and 12B, etc. The distance information acquisition unit 13 includes time-of-flight (TOF) sensors 13a and 13A, wide-angle lenses 13b and 13B, etc. Although not shown, each component may be configured with an optical system such as a prism or a lens group. For example, the imaging unit 11 may be configured with an optical system for focusing light collected by the fisheye lenses 11b and 11B onto the imaging elements 11a and 11A. The projection unit 12 may be configured with an optical system for guiding light from the light sources 12a and 12A to the wide-angle lenses 12b and 12B. Furthermore, an optical system for forming an image on TOF sensors 13a and 13A from light collected by wide-angle lenses 13b and 13B may be configured in distance information acquisition unit 13. Each optical system may be determined appropriately depending on the configuration and arrangement of image pickup elements 11a and 11A, light source units 12a and 12A, TOF sensors 13a and 13A, etc., and the following description will omit explanation of optical systems such as prisms and lens groups.

[0014] Image sensors 11a and 11A, light source units 12a and 12A, and TOF sensors 13a and 13A are housed integrally inside housing 10. Fisheye lens 11b, wide-angle lens 12b, wide-angle lens 13b, and display unit 20 are each provided on a first surface on the front side of housing 10. On the first surface, the inner ranges of fisheye lens 11b, wide-angle lens 12b, and wide-angle lens 13b are each open.

[0015] Fisheye lens 11B, wide-angle lens 12B, wide-angle lens 13B, and shooting switch 15 are each provided on a second surface on the rear side of housing 10. On the second surface, the inner ranges of fisheye lens 11B, wide-angle lens 12B, and wide-angle lens 13B are each open.

[0016] The image sensors 11a and 11A are two-dimensional resolution image sensors (area sensors). The image sensors 11a and 11A have an imaging area in which a large number of light-receiving elements (photodiodes) are arranged in a two-dimensional direction. The imaging area is provided with R (Red), G (Green), and B (Blue) color filters in a Bayer pattern or similar to receive visible light, and light passing through the color filters is stored in the photodiodes. Here, an image sensor with a large number of pixels is used to capture a high-resolution two-dimensional image with a wide angle (e.g., a 180-degree hemispherical range with the imaging direction facing forward as shown in FIG. 2). The image sensors 11a and 11A convert the light focused on the imaging area into an electrical signal using the pixel circuit of each pixel, and output a high-resolution RGB image. The fisheye lenses 11b and 11B collect light from a wide angle (e.g., a 180-degree hemispherical range with the imaging direction facing forward as shown in FIG. 2) and focus the light on the imaging area of ​​the image sensors 11a and 11A.

[0017] The light source units 12a and 12A are semiconductor lasers that emit laser light in a wavelength band outside the visible light range used for distance measurement (infrared, for example). The light source units 12a and 12A may each include a single semiconductor laser or a combination of multiple semiconductor lasers. Furthermore, the semiconductor laser may be a surface-emitting semiconductor laser, such as a vertical cavity surface-emitting laser (VCSEL). Alternatively, the light from the semiconductor laser may be shaped vertically by an optical lens, and the vertically elongated light may be scanned in one dimension within the measurement range by an optical deflection element such as a micro-electro-mechanical systems (MEMS) mirror. In this embodiment, the light source units 12a and 12A are configured to expand the light from the semiconductor laser LA to a wide-angle range via wide-angle lenses 12b and 12B without using an optical deflection element such as a MEMS mirror.

[0018] The wide-angle lenses 12b and 12B of the light source units 12a and 12A have the function of expanding the light emitted by the light source units 12a and 12A to a wide-angle range (for example, a hemispherical range of 180 degrees around the imaging direction shown in Figure 2).

[0019] The wide-angle lenses 13b and 13B of the distance information acquisition unit 13 capture reflected light from the light source units 12a and 12A projected by the projection unit 12 from each direction within a wide-angle measurement range (e.g., a 180-degree hemispherical range around the imaging direction as shown in FIG. 2 ), and form an image of that light on the light-receiving area of ​​the TOF sensors 13a and 13A. The measurement range includes one or more projection targets (e.g., buildings), and light reflected from the projection targets (reflected light) enters the wide-angle lenses 13b and 13B. The reflected light may be captured by, for example, providing a filter on the entire surface of the wide-angle lenses 13b and 13B that cuts out light with wavelengths in the infrared range or longer. However, this is not limiting; as long as light in the infrared range enters the light-receiving area, a means for transmitting light with wavelengths in the infrared range, such as a filter, may be provided on the optical path from the wide-angle lenses 13b and 13B to the light-receiving area.

[0020] The TOF sensors 13a and 13A are optical sensors with two-dimensional resolution. The TOF sensors 13a and 13A have a light-receiving area in which a large number of light-receiving elements (photodiodes) are arranged in a two-dimensional direction. In this sense, they can be called "second image-receiving means." The TOF sensors 13a and 13A receive reflected light from each area (each area is also called a position) in the measurement range using the light-receiving elements corresponding to each area, and measure (calculate) the distance to each area based on the light detected by each light-receiving element.

[0021] In this embodiment, distance is measured using a phase difference detection method. In this method, a laser beam amplitude-modulated at a fundamental frequency is irradiated onto the measurement area, and the reflected light is received. The phase difference between the irradiated light and the reflected light is measured to determine the time, and the distance is calculated by multiplying this time by the speed of light. The advantage of this method is that it can be expected to have a certain degree of resolution.

[0022] The TOF sensors 13a and 13A are driven in synchronization with the irradiation of light by the projection unit 12, calculate the distance corresponding to each pixel from the phase difference with the reflected light at each light receiving element (corresponding to a pixel), and output distance information image data (hereinafter also referred to as a "distance image" or "TOF image") in which pixel information is associated with information indicating the distance to each area within the measurement range. The TOF sensors 13a and 13A output phase information image data in which phase information is associated with pixel information, and may acquire distance information image data based on the phase information image data in post-processing.

[0023] The number of areas into which the measurement range can be divided is determined by the resolution of the light-receiving area. Therefore, if a low-resolution sensor is used to reduce size, the number of pixel information pieces in the range image data will decrease, and the number of 3D point clouds will also decrease.

[0024] Alternatively, distance may be measured using a pulse method instead of the phase difference detection method. In this case, for example, the light source units 12a and 12A emit an ultrashort irradiation pulse P1 with a rise time of several nanoseconds (ns) and a high optical peak power. In synchronization with this, the TOF sensors 13a and 13A measure the time (t) required for receiving a reflected pulse P2, which is the light reflected from the irradiation pulse P1 emitted by the light source units 12a and 12A. When this method is employed, for example, the TOF sensors 13a and 13A may be implemented with a circuit for measuring time on the output side of the light receiving element. Each circuit converts the time required for each light receiving element from emitting the irradiation pulse P1 to receiving the reflected pulse P2 into distance, thereby obtaining the distance to each area.

[0025] This method is suitable for widening the angle of the imaging device 1 because it can output powerful light using peak light. Also, if a MEMS mirror or the like is used to deflect (scan) the light, powerful light can be emitted over a long distance while suppressing its spread, which leads to an increase in the measurement distance. In this case, the laser light emitted from the light source units 12a and 12A is positioned so that it is scanned (deflected) by the MEMS mirror toward the wide-angle lenses 12b and 12B.

[0026] While it is desirable for the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 to be equal, for example, at 180 degrees or more, they do not necessarily have to be equal. If necessary, the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 may be reduced. In this embodiment, the effective pixels of the imaging unit 11 and the distance information acquisition unit 13 are reduced to, for example, a range of 100 degrees to 180 degrees so as not to interfere with the angle of view. Furthermore, the resolution of the TOF sensors 13a and 13A may be set lower than the resolution of the image sensors 11a and 11A, prioritizing the miniaturization of the imaging device 1. By setting the resolution of the TOF sensors 13a and 13A lower than that of the image sensors 11a and 11A, the increase in the size of the light receiving area can be suppressed, leading to a miniaturization of the imaging device 1. Therefore, the TOF sensors 13a and 13A have low resolution, and the 3D point clouds obtained by the TOF sensors 13a and 13A have low density, but the processing circuit 14, which is an "acquisition means," is provided, so that the 3D point clouds can be converted into high-density 3D point clouds. The process of converting the 3D point clouds into high-density 3D point clouds in the processing circuit 14 will be described later.

[0027] In the present embodiment, as an example, the image sensor 11a, the light source unit 12a, and the TOF sensor 13a are arranged so as to be aligned in a straight line in the longitudinal direction of the housing 10. Similarly, the image sensor 11A, the light source unit 12A, and the TOF sensor 13A are arranged so as to be aligned in a straight line in the longitudinal direction of the housing 10. Below, an example of the image sensor 11a, the light source unit 12a, and the TOF sensor 13a will be described.

[0028] The imaging area (imaging surface) of the image sensor 11a and the light receiving area (light receiving surface) of the TOF sensor 13a may be arranged to face a direction perpendicular to the longitudinal direction as shown in FIG. 2, or may be arranged to face the longitudinal direction by providing a prism or the like that changes the linear direction (optical path) of light by 90 degrees before it is incident. They may also be arranged in any other direction depending on the configuration. That is, the image sensor 11a, the light source unit 12a, and the TOF sensor 13a are arranged so that they cover the same measurement range. The image sensor 11a, the projection unit 12, and the distance information acquisition unit 13 are arranged from one side of the housing 10 toward the measurement range. In this case, it is sufficient that the image sensor 11a and the TOF sensor 13a are arranged on the same baseline so as to achieve parallel stereo. By arranging them so as to achieve parallel stereo, even with a single image sensor 11a, it is possible to obtain parallax data using the output of the TOF sensor 13a. The light source unit 12a is configured to irradiate light onto the measurement range of the TOF sensor 13a.

[0029] (processing circuit) Next, the processing of the processing circuit 14 will be described. The TOF images obtained only by the TOF sensors 13a and 13A have low resolution as they are. For this reason, this example shows an example in which the resolution is increased by the processing circuit 14 to reconstruct high-density 3D point cloud data. Note that some or all of the following processing performed by the processing circuit 14 as "information processing means" may be performed by an external device.

[0030] As described above, the three-dimensional point cloud data reconstructed by the imaging device 1 is output to an external device such as a PC via a portable recording medium or communication, and is used to display a three-dimensional reconstruction model.

[0031] This makes it possible to provide an imaging device 1 that is faster, smaller, and lighter, and therefore more portable, than when the imaging device 1 itself displays the 3D reconstruction model.

[0032] However, after leaving the site where the 3D information was acquired and restoring the 3D information using an external device, it may be noticed that the photographer or the tripod is reflected in the captured image, or that the 3D information was not acquired in the desired layout. In such cases, it is necessary to go through the trouble of revisiting the site where the 3D information was acquired.

[0033] One possible solution to this problem would be to bring a 3D reconstruction device to the site, but this would eliminate the benefits of high speed, compactness, and light weight.

[0034] It is also possible to transmit the acquired 3D information to an external device via a communication line and receive the restored 3D information, but this would eliminate the benefit of high speed, and in addition, 3D information contains a large amount of information, making it difficult to visually confirm whether the photographer, the tripod, etc. are reflected in the captured image.

[0035] In particular, in the case of spherical 3D information, it is extremely difficult to visually check whether the photographer himself, the tripod, etc. are reflected in the captured image.

[0036] In view of the above-mentioned problems, the present embodiment aims to provide an imaging device 1 that can easily check in real time whether the photographer himself or herself, a tripod, etc., are captured in the captured image, or whether 3D information of the desired layout has not been acquired.

[0037] FIG. 3 is a diagram for explaining a usage state of the imaging device in the embodiment.

[0038] In the state shown in FIG. 3(a), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the spherical imaging range R, and the photographer M and the selfie stick 1A are not captured in the spherical captured image.

[0039] In the state shown in FIG. 3(b), the photographer M is included in the spherical imaging range R, and the photographer M appears in the spherical captured image.

[0040] In the state shown in FIG. 3(c), the tripod 1B supporting the imaging device 1 is included in the spherical imaging range R, and the tripod 1B appears in the captured image of the celestial sphere.

[0041] In the state shown in FIG. 3(d), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the spherical imaging range R, and the photographer M and the selfie stick 1A will not appear in the spherical captured image; however, due to strong external light, there is a possibility that the photographer M and the selfie stick 1A may be erroneously determined to be in the image.

[0042] Furthermore, in the situations shown in Figures 3(b) and 3(c), the colors and types of objects reflected in the image, as well as their appearance, vary widely, making it difficult to uniformly determine whether or not an object is reflected.

[0043] In the above situation, when determining whether or not a specific object (nearby object) such as the photographer himself or a tripod is present based on the distance information image data output from the TOF sensors 13a and 13A, it is difficult to distinguish whether a specific object is actually present or whether the external light is too strong.

[0044] In other words, when the charge storage amount of a particular pixel of the TOF sensors 13a and 13A is saturated, it is difficult to distinguish from the output of the TOF sensors 13a and 13A alone whether this is due to the presence of a particular object or to the intensity of external light being too strong.

[0045] In view of the above-mentioned problems, another object of this embodiment is to provide an imaging device 1 that can accurately check whether or not a specific object, such as the photographer himself or herself or a tripod, is reflected in a captured image, distinguishing it from the influence of external light.

[0046] Fig. 4 is a diagram showing an example of the configuration of processing blocks of the processing circuit 14. The processing circuit 14 shown in Fig. 4 includes a control unit 141, an RGB image data acquisition unit 142, a monochrome processing unit 143, a TOF image data acquisition unit 144, a resolution enhancement unit 145, a matching processing unit 146, a reprojection processing unit 147, a semantic segmentation unit 148, a parallax calculation unit 149, a 3D reconstruction processing unit 150, a determination unit 160, a display control unit 170 which is an example of an output unit, and a transmission / reception unit 180 which is an example of an output unit. In Fig. 4, solid arrows indicate the flow of signals, and dashed arrows indicate the flow of data.

[0047] When the control unit 141 receives an ON signal (photography start signal) from the photography switch 15, it outputs synchronization signals to the image pickup elements 11a, 11A, the light source units 12a, 12A, and the TOF sensors 13a, 13A, thereby controlling the entire processing circuit 14. The control unit 141 first outputs a signal to the light source units 12a, 12A to instruct them to emit ultrashort pulses, and at the same timing outputs a signal to the TOF sensors 13a, 13A to instruct them to generate TOF image data. The control unit 141 then outputs a signal to the image pickup elements 11a, 11A to capture an image. Note that the image pickup at the image pickup elements 11a, 11A may be taken during the period when light is emitted from the light source units 12a, 12A, or during the period immediately before or after that period.

[0048] The RGB image data acquisition unit 142 acquires RGB image data captured by the image capture elements 11a and 11A and outputs RGB image data of the celestial sphere based on an image capture instruction from the control unit 141. The monochrome processing unit 143 performs processing to align data types for matching processing with TOF image data obtained from the TOF sensors 13a and 13A. In this example, the monochrome processing unit 143 performs processing to convert the RGB image data of the celestial sphere into a monochrome image of the celestial sphere.

[0049] The TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on an instruction to generate TOF image data from the control unit 141, and outputs TOF image data of the entire celestial sphere.

[0050] The resolution increasing unit 145 regards the TOF image data of the celestial sphere as a monochrome image and increases the resolution thereof. Specifically, the resolution increasing unit 145 replaces the distance value associated with each pixel of the TOF image data of the celestial sphere with the value (grayscale value) of the monochrome image of the celestial sphere and uses the result. Furthermore, the resolution increasing unit 145 increases the resolution of the monochrome image of the celestial sphere to the resolution of the RGB image data of the celestial sphere obtained from the image sensors 11a and 11A. The conversion to high resolution is performed by, for example, performing a normal up-conversion process. As another conversion method, for example, multiple frames of continuously generated TOF image data of the celestial sphere may be acquired, and the distances of adjacent points may be added using the frames to perform super-resolution processing.

[0051] The matching processing unit 146 extracts feature amounts of textured portions of a monochrome image of the celestial sphere obtained by increasing the resolution of the TOF image data of the celestial sphere and a monochrome image of the celestial sphere obtained by increasing the resolution of the celestial sphere's RGB image data, and performs matching processing using the extracted feature amounts. For example, the matching processing unit 146 extracts edges from each monochrome image and performs matching processing between the extracted edge information. As an alternative method, the matching processing may be performed using a method that converts texture changes into feature amounts, such as SIFT. Here, the matching processing refers to searching for corresponding pixels.

[0052] A specific matching technique is block matching, which calculates the similarity between pixel values ​​extracted as an M x M (M is a positive integer) pixel block around a reference pixel and pixel values ​​extracted as an M x M pixel block around a pixel that is the center of the search in another image, and then determines the center pixel with the highest similarity as the corresponding pixel.

[0053] There are various ways to calculate similarity. For example, an equation showing the normalized autocorrelation coefficient (CNCC) can be used. The higher the value of the normalized autocorrelation coefficient (CNCC), the higher the similarity, and the value is 1 when the pixel values ​​of the blocks are perfectly matched.

[0054] Furthermore, since distance data for textureless regions can also be obtained from spherical TOF image data, weighting of the matching process may be performed depending on the region. For example, in the calculation of the formula for CNCC, weighting may be applied to areas other than edges (textureless regions).

[0055] Moreover, instead of the formula showing the NCC, a selective normalized correlation coefficient (SCC) or the like may be used.

[0056] The reprojection processing unit 147 performs a process of reprojecting the TOF image data of the celestial sphere, which indicates the distance to each position in the measurement range, onto the two-dimensional coordinates (screen coordinate system) of the imaging unit 11. Reprojection means determining at what coordinates in the images of the imaging elements 11a and 11A the three-dimensional points calculated by the TOF sensors 13a and 13A are projected. The TOF image data of the celestial sphere indicates the positions of the three-dimensional points in a coordinate system centered on the distance information acquisition unit 13 (mainly the wide-angle lenses 13b and 13B). Therefore, the three-dimensional points indicated by the TOF image data of the celestial sphere are reprojected onto a coordinate system centered on the imaging unit 11 (mainly the fisheye lenses 11b and 11B). For example, the reprojection processing unit 147 translates the coordinates of the three-dimensional points in the TOF image data of the celestial sphere to the coordinates of the three-dimensional points centered on the imaging unit 11, and after the translation, performs a process of converting the coordinates into the two-dimensional coordinate system (screen coordinate system) indicated by the RGB image data of the celestial sphere.

[0057] The parallax calculation unit 149 calculates the parallax at each position from the difference in distance to the corresponding pixel obtained by the matching process.

[0058] In addition, the parallax matching process uses the reprojection coordinates converted by the reprojection processing unit 147 to search for surrounding pixels at the position of the reprojection coordinates, thereby making it possible to reduce processing time and obtain more detailed and high-resolution distance information.

[0059] Furthermore, the disparity matching process may use segmentation data obtained by the semantic segmentation process of the semantic segmentation unit 148. In this case, it becomes possible to acquire more detailed and higher resolution distance information.

[0060] Alternatively, the parallax matching process may be performed only on edges or only on parts with strong features, and the propagation process may be performed on other parts by using spherical TOF image data, for example, by using spherical RGB image features or a probabilistic method.

[0061] The semantic segmentation unit 148 uses deep learning to assign segmentation labels indicating objects to the input image in the measurement range. This allows each pixel of the omnidirectional TOF image data to be constrained to one of multiple distance regions divided by distance, further increasing the reliability of calculations.

[0062] 3D reconstruction processing unit 145 acquires the RGB image data of the celestial sphere from RGB image data acquisition unit 142, reconstructs 3D data of the celestial sphere based on the distance information output by parallax calculation unit 149, and outputs a high-density 3D point cloud of the celestial sphere in which color information is added to each 3D point. 3D reconstruction processing unit 150 is an example of a 3D information determination unit that determines 3D information.

[0063] The determination unit 160 acquires the RGB image data of the celestial sphere from the RGB image data acquisition unit 142, and also acquires the TOF image data of the celestial sphere converted into a two-dimensional coordinate system indicated by the RGB image data of the celestial sphere from the reprojection processing unit 147, and determines whether or not a specific object is reflected in the captured image based on these pieces of data, and outputs the determination result to the display control unit 170.

[0064] The display control unit 170 acquires the RGB image data of the celestial sphere from the RGB image data acquisition unit 142, and causes two-dimensional image information based on the acquired RGB image data of the celestial sphere to be displayed on the display unit 20. Furthermore, the display control unit 170 causes information indicating the determination result acquired from the determination unit 160 to be superimposed on the two-dimensional image information and displayed on the display unit 20.

[0065] The display control unit 170 is an example of an output unit that outputs two-dimensional image information captured by the imaging unit 11 separately from the three-dimensional information, and the display unit 20 is an example of an output destination that outputs the two-dimensional image information.

[0066] The display control unit 170 may acquire three-dimensional data of the celestial sphere from the three-dimensional reconstruction processing unit 145, and display the three-dimensional information on the display unit 20. Specifically, the display control unit 170 may select, in accordance with a predetermined condition, whether to display two-dimensional image information on the display unit 20 or three-dimensional information on the display unit 20. This allows the display control unit 170 to output the two-dimensional image information separately from the three-dimensional information.

[0067] The transmitting / receiving unit 180 communicates with an external device via wired or wireless technology, and transmits (outputs) the three-dimensional data of the celestial sphere output from the three-dimensional reconstruction processing unit 145 and the two-dimensional image information of the celestial sphere output from the RGB image data acquisition unit 142 via the network 400 to the external device 300 that performs three-dimensional reconstruction processing.

[0068] The transmitting / receiving unit 180 is an example of an output unit that outputs three-dimensional information, and the external device 300 is an example of an output destination that outputs three-dimensional information.

[0069] The transmitting / receiving unit 180 may transmit only the three-dimensional data of the celestial sphere, without transmitting the two-dimensional image information of the celestial sphere. The transmitting / receiving unit 180 may also be configured by an interface circuit with a portable storage medium such as an SD card, a personal computer, or the like.

[0070] (Operation of processing circuit) 5 is a flow diagram showing an example of the operation of the processing circuit 14 of the imaging device 1. When the user turns on the imaging switch 15 and inputs an imaging instruction signal, the control unit 141 of the processing circuit 14 performs an operation to generate a high-density three-dimensional point cloud in the following manner (an example of an imaging processing method and an information processing method).

[0071] First, the control unit 141 drives the light source units 12a and 12A, the TOF sensors 13a and 13A, and the image pickup elements 11a and 11A to capture an image of the measurement range (step S1). Driven by the control unit 141, the light source units 12a and 12A irradiate infrared light (an example of a projection step), and the TOF sensors 13a and 13A receive the reflected light (an example of a light reception step). Furthermore, the image pickup elements 11a and 11A capture an image of the measurement range at the timing when the light source units 12a and 12A start to be driven or in a period immediately thereafter (an example of an image pickup step).

[0072] Next, the RGB image data acquisition unit 142 acquires RGB image data of the measurement range from the image sensors 11a and 11A (step S2). Then, the display control unit 170 acquires the RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and displays two-dimensional image information based on the acquired RGB image data of the entire celestial sphere on the display unit 20 (an example of a two-dimensional image information output step) (step S3).

[0073] The display control unit 170 displays two-dimensional image information of a partial area of ​​the acquired spherical RGB image data on the display unit 20, and changes the area of ​​the two-dimensional image information displayed on the display unit 20 in response to various inputs from the user. Various inputs from the user can be realized by providing an operation switch other than the photographing switch 15 or by configuring the display unit 20 as an input unit such as a touch panel.

[0074] At this stage, the photographer can check the 2D image information displayed on the display unit 20 to ensure that the photographer himself or herself, the tripod, etc., are not reflected in the captured image, and that 2D image information with the desired layout has not been acquired.

[0075] Next, the TOF image data acquisition unit 144 acquires TOF image data indicating the distance from the TOF sensors 13a and 13A to each position in the two-dimensional area (step S4).

[0076] Next, the monochrome processing unit 143 converts the RGB image data into a monochrome image (step S5). The TOF image data and the RGB image data are different data types, namely distance data and RGB data, respectively, and matching cannot be performed as is. Therefore, each data is first converted into a monochrome image. For the TOF image data, the resolution increasing unit 145 converts the values ​​indicating the distance of each pixel before resolution increase by directly replacing them with the values ​​of the monochrome image.

[0077] Next, the resolution increasing unit 145 increases the resolution of the TOF image data (step S6).

[0078] Next, the matching processing unit 146 extracts the feature amount of the textured portion of each monochrome image, and performs matching processing using the extracted feature amount (step S7).

[0079] Next, determination unit 160 acquires RGB image data of the celestial sphere from RGB image data acquisition unit 142, and also acquires TOF image data of the celestial sphere converted into a two-dimensional coordinate system indicated by the RGB image data from reprojection processing unit 147, and determines whether or not a nearby object as a specific target is reflected in the captured image based on these pieces of data, and outputs the determination result to display control unit 170. Display control unit 170 superimposes information indicating the determination result acquired from determination unit 160 on the two-dimensional image information and displays it on display unit 20 (an example of a display step) (step S8).

[0080] Next, the parallax calculation unit 149 calculates the parallax at each position from the difference in distance between corresponding pixels (step S9).

[0081] Then, the 3D reconstruction processing unit 145 acquires the RGB image data from the RGB image data acquisition unit 142, reconstructs the 3D data based on the distance information output by the parallax calculation unit 149, and outputs a high-density 3D point cloud with color information added to each 3D point (step S10).

[0082] Next, the transmitter / receiver 180 transmits the three-dimensional data output from the three-dimensional reconstruction processing unit 145 and the two-dimensional image information output from the RGB image data acquisition unit 142 via the network 400 to an external device 300 that performs three-dimensional reconstruction processing (an example of a three-dimensional information output step) (step S11).

[0083] The transmitting / receiving unit 180 may transmit the three-dimensional data output from the three-dimensional reconstruction processing unit 145 without transmitting the two-dimensional image information output from the RGB image data acquisition unit 142.

[0084] As described above, the imaging device 1 includes the imaging unit 11 and the display control unit 170 that outputs two-dimensional image information captured by the imaging unit 11 separately from three-dimensional information.

[0085] This makes it possible to easily check from the 2D image information whether the photographer or the tripod is captured in the captured image, or whether the 3D information of the desired layout has not been acquired, without checking the 3D information.

[0086] Therefore, it becomes possible to reacquire three-dimensional information while remaining at the site where the three-dimensional information is being acquired, which reduces the effort required to visit the site where the three-dimensional information is being acquired again, compared to when, after leaving the site where the three-dimensional information is being acquired, one realizes that the photographer, the tripod, etc., are reflected in the captured image, or that three-dimensional information with the desired layout has not been acquired.

[0087] The three-dimensional information includes spherical three-dimensional information. In this case, even in the case of spherical three-dimensional information in which it is difficult to confirm whether the photographer himself / herself, the tripod, etc. are included in the captured image or whether three-dimensional information of a desired layout has not been acquired, it becomes possible to easily confirm whether the photographer himself / herself, the tripod, etc. are included in the captured image or whether three-dimensional information of a desired layout has not been acquired from the two-dimensional image information captured by the imaging unit 11.

[0088] The display control unit 170 outputs the two-dimensional image information G in step S3 before the transmitting / receiving unit 180 transmits (outputs) the three-dimensional information in step S11. The display control unit 170 outputs the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

[0089] This makes it possible to check from the 2D image information whether the photographer or the tripod is captured in the captured image, or whether the 3D information of the desired layout has not been acquired, before checking the 3D information.

[0090] The display control unit 170 causes the display unit 20 to display the two-dimensional image information.

[0091] This makes it possible to easily check from the 2D image information displayed on the display unit 20 whether the photographer himself or the tripod is reflected in the captured image, or whether 3D information of the desired layout has not been acquired.

[0092] The display control unit 170 outputs the two-dimensional image information to a display unit 20 that is different from the external device 300 to which the transmitting and receiving unit 180 outputs the three-dimensional information.

[0093] This makes it possible to check whether the photographer himself or the tripod is reflected in the captured image or whether three-dimensional information of the desired layout has not been obtained, from the two-dimensional image information output to a display unit 20 different from the external device 300, without checking the three-dimensional information output to the external device 300.

[0094] The imaging device 1 includes a 3D reconstruction processing unit 150 that determines 3D information based on the output of the distance information acquisition unit 13. The 3D reconstruction processing unit 150 determines the 3D information based on the output of the distance information acquisition unit 13 and the 2D image information.

[0095] This makes it possible to check from the 2D image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been acquired, without checking the 3D information determined by the 3D reconstruction processing unit 150.

[0096] FIG. 6 is a flowchart showing the generation of spherical image data in the embodiment.

[0097] FIG. 6(a) is a flowchart showing the process of generating RGB image data of the celestial sphere, which corresponds to step S2 described in FIG.

[0098] The RGB image data acquisition unit 142 receives two sets of RGB image data in a fisheye image format (step S201).

[0099] The RGB image data acquisition unit 142 converts each RGB image data into an equirectangular image format (step S202). The RGB image data acquisition unit 142 converts the two RGB image data into an equirectangular image format based on the same coordinate system, which makes it easier to combine the images in the next step.

[0100] Here, the equirectangular image format will be described. The equirectangular image format is a method capable of representing a celestial sphere image, and is a format of an image (equirectangular image) created using equirectangular projection. Equirectangular projection is a projection that represents three-dimensional directions using two variables, such as the latitude and longitude of a globe, and displays the image on a plane so that the latitude and longitude are orthogonal. Therefore, an equirectangular image is an image created using equirectangular projection, and is represented by coordinates with two axes consisting of two angular variables in a spherical coordinate system.

[0101] The RGB image data acquisition unit 142 combines the two RGB image data generated in step S202 to generate one piece of spherical RGB image data (step S203). The two pieces of input RGB image data cover an area with a full angle of view of more than 180 degrees. Therefore, the spherical RGB image data generated by appropriately connecting these two pieces of RGB image data can cover the entire spherical area.

[0102] The joining process in step S203 can use existing techniques for joining multiple images, and there are no particular limitations on the method.

[0103] FIG. 6B is a flowchart showing the process of generating TOF image data of the entire celestial sphere, which corresponds to step S4 described in FIG.

[0104] The TOF image data acquisition unit 144 acquires two pieces of depth image data in a fisheye image format (step S401).

[0105] The TOF image data acquisition unit 144 converts each of the two TOF image data in the fisheye image format into an equirectangular image format (step S402). As described above, the equirectangular image format is a method capable of expressing a spherical image. In step S402, converting the two TOF image data into the equirectangular image format based on the same coordinate system facilitates image combination in the next step S403.

[0106] The TOF image data acquisition unit 144 combines the two TOF image data generated in step S402 to generate one piece of spherical TOF image data (step S403). The two input TOF image data cover an area with a full angle of view of more than 180 degrees. Therefore, the spherical TOF image data generated by appropriately joining the two TOF image data can cover the entire spherical area.

[0107] The joining process in step S403 can use existing techniques for joining multiple images, and there are no particular limitations on the method.

[0108] FIG. 7 is a flowchart of nearby object determination in the same embodiment.

[0109] FIG. 7 is a flowchart showing the process of determining whether or not a nearby object is captured in a captured image, which corresponds to step S8 described in FIG.

[0110] The determination unit 160 determines whether or not there is a pixel in which the amount of stored charge is saturated in the TOF image data of the celestial sphere, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S801).

[0111] When there is a pixel whose amount of stored power is saturated in step S801, the determination unit 160 determines, based on the RGB image data of the entire celestial sphere acquired from the RGB image data acquisition unit 142, whether or not the amount of stored power is saturated for pixels that have the same coordinates as the pixel whose amount of stored power is saturated in step S801, among the RGB image data of the entire celestial sphere (step S802).

[0112] If the amount of stored power is saturated in step S802, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S801 is due to external light, and outputs error information to the display control unit 170. Based on the error information acquired from the determination unit 160, the display control unit 170 superimposes the error information on the two-dimensional image information and displays it on the display unit 20 (step S803).

[0113] If the amount of stored power is not saturated in step S802, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S801 is due to the presence of a nearby object, and outputs coordinate position information of the pixel whose amount of stored power is saturated in step S801 to the display control unit 170. Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 superimposes identification information that identifies the nearby object on the two-dimensional image information and displays it on the display unit 20 (step S804).

[0114] When there are no pixels with saturated power storage in step S801, the determination unit 160 determines whether there are pixels indicating distance information of 0.5 m or less in the omnidirectional TOF image data acquired from the reprojection processing unit 147 (step S805).

[0115] If there are no pixels indicating distance information of 0.5 m or less in step S805, the determination unit 160 ends the process.

[0116] If there are pixels indicating distance information of 0.5 m or less in step S805, the determination unit 160 proceeds to step S804 described above, determines that the pixels indicating distance information of 0.5 m or less are due to the presence of a nearby object, and outputs the coordinate position information of the pixels indicating distance information of 0.5 m or less in step S805 to the display control unit 170. Based on the coordinate position information of the pixels acquired from the determination unit 160, the display control unit 170 superimposes identification information for identifying the nearby object on the two-dimensional image information and causes the display unit 20 to display it.

[0117] As described above, when it is determined that a nearby object exists, the display control unit 170 superimposes the identification information on the two-dimensional image information, and when it is not determined that a nearby object exists, the display control unit 170 does not superimpose the identification information on the two-dimensional image information.

[0118] That is, the display control unit 170 causes the display unit 20 to display differently according to the presence or absence of a nearby object.

[0119] Also, based on the coordinate position information of the pixels acquired from the determination unit 160, the display control unit 170 superimposes identification information for identifying the nearby object on the two-dimensional image information and causes the display unit 20 to display it.

[0120] That is, the display control unit 170 causes the display unit 20 to display at different positions according to the position of the nearby object.

[0121] FIG. 8 is a diagram for explaining the display content of the display unit in the same embodiment.

[0122] FIG. 8 is an explanatory diagram corresponding to step S2 shown in FIG. 5 and steps S803 and S804 shown in FIG.

[0123] On the display unit 20, the two-dimensional image information G is displayed by the display control unit 170. Also, on the display unit 20, the display control unit 170 displays identification information G1 and G2 for identifying nearby objects and error information G3, each superimposed on the two-dimensional image information G.

[0124] As described above, the imaging device 1 includes an imaging unit 11 that images an object, a projection unit 12 that projects light onto the object, a distance information acquisition unit 13 that receives light reflected from the object, and a display control unit 170 that causes the display unit 20 to display different images depending on the presence or absence of a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

[0125] This allows the photographer to accurately check whether or not the photographer himself or herself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the influence of external light.

[0126] The imaging device 1 includes a display unit 20. This allows the photographer to reliably check whether or not a nearby object appears in the captured image.

[0127] The display control unit 170 causes the display unit 20 to display different positions depending on the position of the nearby object, thereby enabling the photographer to confirm the position of the nearby object reflected in the captured image.

[0128] The display control unit 170 displays the image information G captured by the imaging unit 11 on the display unit 20, and also displays the identification information G1, G2 for identifying nearby objects superimposed on the image information on the display unit 20. This allows the photographer to reliably confirm the position of the nearby object reflected in the captured image.

[0129] The imaging device 1 includes a determination unit 160 that determines that there is a nearby object when the amount of charge stored by the light received by the distance information acquisition unit 13 is saturated and the amount of charge stored in the pixels of the imaging unit 11 is not saturated.

[0130] This allows the photographer to accurately check whether or not a nearby object is reflected in the captured image, distinguishing it from the influence of external light.

[0131] Fig. 9 is a diagram showing the appearance of an imaging device according to a modified example of the embodiment, and Fig. 10 is a diagram showing the configuration of a processing block of a processing circuit in the modified example.

[0132] In this modification, display control unit 170 acquires RGB image data of the celestial sphere from RGB image data acquisition unit 142, and displays two-dimensional image information based on the acquired RGB image data of the celestial sphere on display unit 520 of display device 500. Display unit 520 is an example of an output destination that outputs the two-dimensional image information.

[0133] This makes it possible to easily check from the 2D image information displayed on the display unit 520 whether the photographer or the tripod is reflected in the captured image, or whether 3D information of the desired layout has not been obtained.

[0134] The display control unit 170 outputs the two-dimensional image information to a display unit 520 that is different from the external device 300 to which the transmitting and receiving unit 180 outputs the three-dimensional information.

[0135] This makes it possible to check whether the photographer himself or the tripod is reflected in the captured image or whether three-dimensional information of the desired layout has not been obtained, from the two-dimensional image information output to a display unit 520 different from the external device 300, without checking the three-dimensional information output to the external device 300.

[0136] The display control unit 170 may acquire three-dimensional data of the celestial sphere from the three-dimensional reconstruction processing unit 145, and display the three-dimensional information on the display unit 520. Specifically, the display control unit 170 may select, in accordance with a predetermined condition, whether to display two-dimensional image information on the display unit 520 or to display three-dimensional information on the display unit 520. This allows the display control unit 170 to output the two-dimensional image information separately from the three-dimensional information.

[0137] Based on the error information acquired from the determining unit 160, the display control unit 170 causes the display unit 520 to display the error information superimposed on the two-dimensional image information.

[0138] Based on the coordinate position information of the pixel acquired from the determining unit 160, the display control unit 170 causes the display unit 520 to display the identification information for identifying the nearby object superimposed on the two-dimensional image information.

[0139] That is, the display control unit 170 causes the display unit 520 to display different images depending on the presence or absence of a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

[0140] This allows the photographer to accurately check whether or not the photographer himself or herself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the influence of external light.

[0141] The display control unit 170 causes the display unit 520 to display different positions depending on the position of the nearby object, thereby enabling the photographer to confirm the position of the nearby object reflected in the captured image.

[0142] The display control unit 170 causes the display unit 520 to display the image information captured by the imaging unit 11, and also causes the display unit 520 to superimpose identification information for identifying the nearby object on the image information, thereby allowing the photographer to reliably confirm the position of the nearby object reflected in the captured image.

[0143] Fig. 11 is a diagram showing the appearance of an imaging device according to a second modified example of the embodiment of the present invention, and Fig. 12 is a diagram showing the configuration of a processing block of a processing circuit in the second modified example.

[0144] 11, the imaging device 1 includes a plurality of display units 20A, 20a instead of the display unit 20 shown in Fig. 1. The display units 20A, 20a are configured with LEDs or the like, and flash or light up in response to an output signal from the processing circuit 14.

[0145] The display unit 20a is provided on a first surface on the front side of the housing 10, and the display unit 20A is provided on a second surface on the rear side of the housing 10.

[0146] In the second modified example shown in FIG. 12, the display control unit 170 causes the display units 20A and 20a to display information indicating the determination result obtained from the determination unit 160.

[0147] Furthermore, the transmitting / receiving unit 180 transmits (outputs) the spherical two-dimensional image information output from the RGB image data acquiring unit 142 to the display device 500 via the network 400. The display device 500 is an example of an output destination to which the two-dimensional image information is output.

[0148] That is, in the second modified example, in step S3 shown in FIG. 5 , the transmitting / receiving unit 180 acquires RGB image data of the entire celestial sphere from the RGB image data acquiring unit 142, and transmits (outputs) two-dimensional image information based on the acquired RGB image data of the entire celestial sphere to the display device 500.

[0149] The transmitting / receiving unit 510 of the display device 500 receives the two-dimensional image information transmitted from the transmitting / receiving unit 180 of the imaging device 1.

[0150] The control unit 530 of the display device 500 causes the display unit 520 to display the two-dimensional image information received by the transmitting / receiving unit 510 .

[0151] As described above, the imaging device 1 includes the imaging unit 11 and the transmitting / receiving unit 180 that outputs two-dimensional image information captured by the imaging unit 11 separately from three-dimensional information.

[0152] This makes it possible to easily check from the 2D image information whether the photographer or the tripod is captured in the captured image, or whether the 3D information of the desired layout has not been acquired, without checking the 3D information.

[0153] Therefore, it becomes possible to reacquire three-dimensional information while remaining at the site where the three-dimensional information is being acquired, which reduces the effort required to visit the site where the three-dimensional information is being acquired again, compared to when, after leaving the site where the three-dimensional information is being acquired, one realizes that the photographer, the tripod, etc., are reflected in the captured image, or that three-dimensional information with the desired layout has not been acquired.

[0154] The transmitting / receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before transmitting (outputting) the three-dimensional information in step S11. The transmitting / receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

[0155] This makes it possible to check from the 2D image information whether the photographer or the tripod is captured in the captured image, or whether the 3D information of the desired layout has not been acquired, before checking the 3D information.

[0156] The transmitting / receiving unit 180 transmits the two-dimensional image information to the display device 500, and the display device 500 causes the display unit 520 to display the two-dimensional image information.

[0157] This makes it possible to easily check from the 2D image information displayed on the display unit 520 whether the photographer or the tripod is reflected in the captured image, or whether 3D information of the desired layout has not been obtained.

[0158] The transmitting / receiving unit 180 transmits the two-dimensional image information to a display device 500 that is different from the external device 300 that outputs the three-dimensional information.

[0159] This makes it possible to check whether the photographer himself or the tripod is reflected in the captured image or whether three-dimensional information of the desired layout has not been obtained, from the two-dimensional image information output to the display unit 520 of the display device 500, which is different from the external device 300, without checking the three-dimensional information output to the external device 300.

[0160] The transmitting / receiving unit 180 may transmit the three-dimensional information to the display device 500. Specifically, the transmitting / receiving unit 180 may select, in accordance with a predetermined condition, whether to transmit two-dimensional image information to the display device 500 or three-dimensional information to the display device 500. This allows the transmitting / receiving unit 180 to transmit the two-dimensional image information to the display device 500 separately from the three-dimensional information.

[0161] FIG. 13 is a flowchart of nearby object determination in the second modified example.

[0162] FIG. 13 is a flowchart showing the process of determining whether or not a nearby object is captured in a captured image, which corresponds to step S8 described with reference to FIG. 5 in the second modified example.

[0163] The determination unit 160 determines whether or not there is a pixel in which the amount of stored charge is saturated in the TOF image data of the celestial sphere, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S811).

[0164] When there is a pixel whose amount of stored power is saturated in step S811, the determination unit 160 determines, based on the RGB image data of the entire celestial sphere acquired from the RGB image data acquisition unit 142, whether the amount of stored power is saturated for pixels that have the same coordinates as the pixel whose amount of stored power is saturated in step S811, among the RGB image data of the entire celestial sphere (step S812).

[0165] If the amount of stored power is saturated in step S812, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S811 is due to external light, and outputs error information to the display control unit 170. The display control unit 170 causes the display units 20A and 20a to display the error information based on the error information acquired from the determination unit 160 (step S813).

[0166] If the amount of stored power is not saturated in step S812, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S811 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel whose amount of stored power is saturated in step S811 to the display control unit 170. Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 (step S814).

[0167] If there is no pixel whose charged amount is saturated in step S811, the determination unit 160 determines whether there is a pixel in the TOF image data of the celestial sphere that indicates distance information of 0.5 m or less, based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S815).

[0168] If there is no pixel showing distance information of 0.5 m or less in step S815, the determination unit 160 ends the process.

[0169] If there is a pixel showing distance information of 0.5 m or less in step S815, the determination unit 160 proceeds to step S814 described above, determines that the pixel showing distance information of 0.5 m or less in step S815 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel showing distance information of 0.5 m or less in step S815 to the display control unit 170. The display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 based on the coordinate position information of the pixel acquired from the determination unit 160.

[0170] If it is determined in step S814 that the display control unit 170 is on the front side, it causes the display unit 20a arranged on the front side of the housing 10 to blink (step S816).

[0171] If it is not determined in step S814 that the display control unit 170 is on the front side, the display control unit 170 causes the display unit 20A arranged on the back side of the housing 10 to blink (step S817).

[0172] As described above, the display control unit 170 blinks the display unit 20a or the display unit 20A when it determines that a nearby object is present, and does not blink the display unit 20a or the display unit 20A when it does not determine that a nearby object is present.

[0173] That is, the display control unit 170 causes the display unit 20a and the display unit 20A to display different images depending on whether or not a nearby object is present.

[0174] This allows the photographer to accurately check whether or not the photographer himself or herself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the influence of external light.

[0175] Furthermore, the display control unit 170 blinks the display unit 20a or the display unit 20A based on the coordinate position information of the pixel acquired from the determination unit 160.

[0176] That is, the display control unit 170 causes the display unit 20a and the display unit 20A to display different positions depending on the position of the nearby object, thereby allowing the photographer to confirm the position of the nearby object reflected in the captured image.

[0177] Then, the display control unit 170 controls the display unit 20A, 20a that is closer to the nearby object to display different images depending on whether or not the nearby object is present, thereby enabling the photographer to reliably confirm the position of a specific object in the captured image.

[0178] FIG. 14 is a diagram illustrating the configuration of an imaging device according to a third modified example of the embodiment of the present invention.

[0179] In a third variant shown in FIG. 14, the imaging device 1 includes, in addition to the configuration shown in FIG. 2, another imaging unit 111 having other imaging elements 111a, 111A, other fisheye lenses (wide-angle lenses) 111b, 111B, etc.

[0180] In the third modified example, the RGB imaging unit 11 and another imaging unit 111 are provided on the same baseline. In this case, multi-view processing becomes possible in the processing circuit 14. That is, by simultaneously driving the imaging unit 11 and another imaging unit 111, which are provided a predetermined distance apart on one surface, RGB images from two viewpoints can be obtained. This makes it possible to use parallax calculated based on the two RGB images, further improving the distance accuracy over the entire measurement range.

[0181] Specifically, when an RGB imaging unit 11 and another imaging unit 111 are installed, it becomes possible to use multi-baseline stereo (MSB) using SSSD and EPI processing, as in conventional parallax calculations. Therefore, by using this, the reliability of parallax increases, making it possible to achieve high spatial resolution and accuracy.

[0182] As described above, the imaging device 1 includes another imaging unit 111, and the 3D reconstruction processing unit 150 determines 3D information based on the output of the distance information acquisition unit 13, the 2D image information, and other 2D image information captured by the other imaging unit 111.

[0183] The imaging device 1 may also include another imaging unit 111 and a three-dimensional information determination unit that determines three-dimensional information based on two-dimensional image information and other two-dimensional image information captured by the other imaging unit 111, without relying on the output of the distance information acquisition unit 13.

[0184] This makes it possible to check from the two-dimensional image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image or whether three-dimensional information of the desired layout has not been obtained, without checking the three-dimensional information determined by the three-dimensional reconstruction processing unit 150 based on the two-dimensional image information.

[0185] As described above, the imaging device 1 according to one embodiment of the present invention comprises an imaging unit 11 that images an object, a projection unit 12 that projects light onto the object, a distance information acquisition unit 13 (an example of a light receiving unit) that receives light reflected from the object, and a display control unit 170 that causes the display units 20, 520 to display different images depending on the presence or absence of a specific object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

[0186] This allows the photographer to accurately check whether or not a particular object, such as the photographer himself or herself or a tripod, is reflected in the captured image, distinguishing it from the influence of external light.

[0187] The imaging device 1 includes a display unit 20. This allows the photographer to reliably check whether or not a particular object appears in the captured image.

[0188] The display control unit 170 causes the display units 20 and 520 to display different positions depending on the position of the specific object, thereby enabling the photographer to confirm the position of the specific object in the captured image.

[0189] The display unit 20 includes a plurality of display units 20A, 20a, and the display control unit 170 controls the display unit that is closer to a specific object among the plurality of display units 20A, 20a to display different images depending on whether the object is present or not. This allows the photographer to reliably confirm the position of the specific object in the captured image.

[0190] The display control unit 170 displays the image information G captured by the imaging unit 11 on the display unit 20, 520, and also superimposes the identification information G1, G2 that identifies the specific object on the image information and displays it on the display unit 20, 520. This allows the photographer to reliably confirm the position of the specific object in the captured image.

[0191] The imaging device 1 includes a determination unit 160 that determines that a specific object is present when the amount of charge stored by the light received by the distance information acquisition unit 13 is saturated and the amount of charge stored in the pixels of the imaging unit is not saturated.

[0192] This allows the photographer to accurately check whether or not a specific object is captured in the captured image, distinguishing it from the influence of external light. [Explanation of symbols]

[0193] 1. Imaging device (an example of an information processing device) 10. Cabinet 11 Imaging unit 11a, 11A image sensor 11b, 11B fisheye lens 12 Projection section 12a, 12A light source section 12b, 12B wide-angle lens 13 Distance information acquisition unit (an example of a light receiving unit) 13a, 13A TOF sensor 13b, 13B wide-angle lens 14 Processing circuit 15 Shooting switch 20 Display section 20A, 20a display section 111 Other imaging units 150 3D reconstruction processing unit (an example of a 3D information determination unit) 160 Judgment Department 170 Display control unit (an example of an output unit) 180 Transmitting and receiving unit (an example of an output unit) 300 External device (example of output destination) 500 Display device (example of output destination) 520 Display unit (example of output destination) L Sync signal line [Prior art documents] [Patent documents]

[0194] [Patent Document 1] Japanese Patent Application Publication No. 2018-077071 [Patent Document 2] Patent No. 5423287

Claims

1. an imaging unit that captures an image of a target; a projection unit that projects light onto the target; a light receiving unit that receives the light reflected from the object; a display control unit that causes a display unit to display different images depending on the presence or absence of a specific object determined based on the output of the light receiving unit and the output of the imaging unit; An imaging device comprising:

2. The imaging device according to claim 1 , further comprising the display unit.

3. 3. The imaging device according to claim 1, wherein the display control unit causes the display unit to display a different position depending on the position of the specific object.

4. the display unit includes a plurality of display units, The imaging device according to claim 4 , wherein the display control unit causes one of the plurality of display units that is closer to the specific object to display different images depending on whether the object is present or not.

5. The imaging device according to any one of claims 1 to 4, wherein the display control unit displays image information captured by the imaging unit on the display unit, and also displays identification information that identifies the specific object on the display unit by superimposing it on the image information.

6. An imaging device according to any one of claims 1 to 5, further comprising a determination unit that determines that the specific object is present when the amount of charge stored by the light received by the light receiving unit is saturated and the amount of charge stored in the pixels of the imaging unit is not saturated.

7. an imaging step of imaging the target by an imaging unit; a projection step of projecting light onto the target; a light receiving step of receiving the light reflected from the object by a light receiving unit; a display step of displaying different images depending on the presence or absence of a specific object determined based on the output of the light receiving unit and the output of the imaging unit.

Images