Information processing device and information processing method

Abstract

An information processing device (1) outputs three-dimensional information determined based on received light, comprising: a camera unit (11) configured to capture two-dimensional image information, and an output unit (170, 180) configured to output the two-dimensional image information for display and the three-dimensional information associated with the coordinates of the two-dimensional image information.

Description

Technical Field

[0001] The present disclosure relates to an information processing device and an information processing method. Background Art

[0002] JP-2018-077071-A discloses a ranging device capable of stably and correctly measuring the range or distance to an object.

[0003] JP-2011-077606-A discloses an image pickup apparatus that, when a finger or the like is included in a captured image, performs image processing to reduce the influence of the finger or the like included in the captured image.

[0004] Citation List

[0005] Patent Literature

[0006] [Patent Document 1] JP-2018-077071-A

[0007] [Patent Document 2] JP-2011-077606-A Summary of the invention

[0008] Technical issues

[0009] The present invention discloses an embodiment designed in view of the above-mentioned problems, in order to provide an information processing device and an information processing method which can easily confirm that desired three-dimensional information is acquired without an unexpected object.

[0010] Solutions to the problem

[0011] In one aspect of the present invention, an information processing device outputs three-dimensional information determined based on received light, comprising: a camera unit configured to capture two-dimensional image information, and an output unit configured to output the two-dimensional image information for display and the three-dimensional information associated with the coordinates of the two-dimensional image information.

[0012] Effects of the Invention

[0013] According to one or more embodiments of the present disclosure, it is possible to provide an information processing device and an information processing method that can easily confirm that desired three-dimensional information is acquired without an unexpected object. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are intended to illustrate exemplary embodiments of the present invention and should not be interpreted as limiting its scope. Unless explicitly stated, the accompanying drawings should not be considered to be drawn to scale. In addition, in all the drawings, the same or similar reference numerals represent the same or similar parts.

[0015] Figure 1FIG. 1 is an example of the appearance of an image pickup device according to an embodiment of the present invention.

[0016] Figure 2 FIG. 2 is a cross-sectional view showing the structure of an imaging device according to an embodiment of the present invention.

[0017] Figure 3A FIG. 1 is an example of a usage state of the imaging device according to the embodiment of the present invention.

[0018] Figure 3B The figure shows an example of a usage state of the imaging device according to the embodiment of the present invention.

[0019] Figure 3C The figure shows an example of a usage state of the imaging device according to the embodiment of the present invention.

[0020] Figure 3D FIG. 1 is an example of a usage state of the imaging device according to the embodiment of the present invention.

[0021] Figure 4 FIG. 1 is an example of a functional block diagram of a processing circuit according to an embodiment of the present invention.

[0022] Figure 5 FIG. 1 is an example of an operation flow chart of a processing circuit of the image pickup device according to the embodiment of the present invention.

[0023] Fig. 6A FIG. 1 is an example of a flowchart for generating image data of a panoramic spherical image according to an embodiment of the present invention.

[0024] Figure 6B FIG. 1 is an example of a flowchart for generating image data for generating a panoramic spherical image according to an embodiment of the present invention.

[0025] Figure 7 FIG. 1 is an example of a flowchart for determining a nearby object according to an embodiment of the present invention.

[0026] Figure 8 FIG. 1 is an example of displaying contents of a view using a display unit according to an embodiment of the present invention.

[0027] Fig. 9 FIG. 1 is an example of the appearance of an imaging device according to a first modified example of the embodiment of the present invention.

[0028] Fig.10 FIG. 1 is an example of a functional block diagram of a processing circuit according to a first modification example of the embodiment of the present invention.

[0029] Fig.11FIG. 1 is an example of the appearance of an imaging device according to a second modified example of the embodiment of the present invention.

[0030] Fig.12 FIG. 1 is an example of a functional block diagram of a processing circuit according to a second modification example of the embodiment of the present invention.

[0031] Fig.13 FIG. 1 is an example of a flowchart for determining a nearby object according to a second variation of the embodiment of the present invention.

[0032] Fig.14 FIG. 1 is a cross-sectional view showing the structure of an imaging device according to a third modified example of the embodiment of the present invention. DETAILED DESCRIPTION

[0033] The words used herein are only used for the purpose of illustrating specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are intended to also include plural forms, unless the context clearly states otherwise. When describing the embodiments shown in the accompanying drawings, specific words are used for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the selected specific words, and it should be understood that each specific element includes all technical equivalents having similar functions, operating in a similar manner and achieving similar results. Below, the composition of the present invention is described with reference to the accompanying drawings.

[0034] Hereinafter, one or more embodiments of an image capturing device, an image capturing / processing method, an information processing device, and an information processing method will be described in detail with reference to the accompanying drawings.

[0035] Figure 1 1 is an example of the appearance of the imaging device 1 according to the embodiment of the present invention. Figure 2 FIG. 1 is an example of a cross-sectional view showing the structure of the imaging device 1 . Figure 2 Shown is Figure 1 The internal structure of the camera device 1 is shown in FIG.

[0036] The imaging device 1 is an example of an information processing device that outputs three-dimensional information determined based on received light.

[0037] like Figure 1 As shown, the imaging device 1 includes, for example, an imaging unit 11 (ie, a camera), a projection unit 12 , and a distance information acquisition unit 13 , which are integrally accommodated in a housing 10 (or a box 10 ).

[0038] The projection unit 12 corresponds to a light emitting unit of the distance sensor, and projects light other than visible light.

[0039] The distance information acquisition unit 13 corresponds to a light receiving unit of the distance sensor, which acquires distance information based on the light projected by the projection unit 12 and reflected from the object. These units are configured to communicate with each other via a synchronization signal line L (see Figure 2 ) and the processing circuit 14 (see Figure 2 ) are electrically connected and are configured to act synchronously with each other.

[0040] like Figure 1 As shown, the camera device 1 further includes a camera switch 15 , which is used by a user to input a camera instruction signal to the processing circuit 14 .

[0041] like Figure 1 As shown, the camera device 1 may further include a display unit 20 that displays content corresponding to the output signal received from the processing circuit 14. The display unit 20 may be, for example, a liquid crystal screen, etc. The display unit 20 may be, for example, a touch panel to receive operations input by a user.

[0042] The processing circuit 14 controls the above-mentioned various units to acquire information such as red / green / blue (RGB) image data and distance information data, and reconstructs the acquired distance information data into high-density three-dimensional point group data based on the acquired RGB image data and distance information data. The distance information data may be used to construct the three-dimensional point group data without processing the distance information data, but in this case, the accuracy of the three-dimensional point group data is limited to the number of pixels (i.e., resolution) of the distance information acquisition unit 13.

[0043] In the embodiment of the present invention, the process of reconstructing distance information data into high-density three-dimensional point group data is described. The reconstructed data is output to an external device or apparatus such as a personal computer (PC) via a portable recording medium or a communication network, and then used to display a three-dimensional restoration model.

[0044] Each of the above-described respective units and the processing circuit 14 is supplied with power from a battery included in the housing 10. Alternatively, power may be supplied from the outside of the housing 10 via a connection line or wirelessly.

[0045] The image pickup unit 11 that captures two-dimensional image information includes, for example, image pickup elements 11a, 11A and fisheye lenses 11b, 11B. The fisheye lens is an example of a wide-angle lens.

[0046] The projection unit 12 includes, for example, light source units 12 a and 12A and wide-angle lenses 12 b and 12B.

[0047] The distance information acquisition unit 13 includes, for example, time-of-flight (TOF) sensors 13 a , 13A and wide-angle lenses 13 b , 13B.

[0048] Although not shown, it can be configured to use an optical system including a prism and a lens group. For example, the imaging unit 11 can be configured using an optical system for focusing the light collected by the fisheye lenses 11b, 11B onto the imaging elements 11a, 11A. In addition, the projection unit 12 can be configured to guide the light from the light source units 12a, 12A to the wide-angle lenses 12b, 12B using an optical system. In addition, the distance information acquisition unit 13 can be configured to focus the light collected by the wide-angle lenses 13b, 13B onto the TOF sensors 13a, 13A using an optical system. Each optical system can be appropriately designed according to the composition and configuration of the imaging elements 11a, 11A, the light source units 12a, 12A, and the TOF sensors 13a, 13A. In this description, the optical system including a prism and a lens group is omitted.

[0049] The imaging elements 11a, 11A, the light source units 12a, 12A, and the TOF sensors 13a, 13A are integrally housed in the housing 10. The fisheye lens 11b, the wide-angle lens 12b, the wide-angle lens 13b, and the display unit 20 are provided on the first surface (i.e., the front side) of the housing 10. With respect to the first surface, each of the fisheye lens 11b, the wide-angle lens 12b, and the wide-angle lens 13b has an internal open space.

[0050] Furthermore, a fisheye lens 11B, a wide-angle lens 12B, a wide-angle lens 13B, and an image pickup switch 15 are provided on the second surface (ie, rear side) of the housing 10. With respect to the second surface, each of the fisheye lens 11B, the wide-angle lens 12B, and the wide-angle lens 13B has an internal open space.

[0051] The imaging elements 11a and 11A are image sensors (or area sensors) with a predetermined two-dimensional resolution. Each imaging element 11a and 11A has an imaging area in which a large number of light-receiving elements (e.g., photodiodes) corresponding to each pixel are arranged in a two-dimensional direction. The imaging area is equipped with red (R), green (G), and blue (B) color filters that receive visible light using, for example, a Bayer arrangement, and the light that passes through the color filters is stored in the photodiode as electricity. In this specification, an image sensor has a large number of pixels, so that it can acquire a wide-angle area (e.g., with a large angle) with high resolution. Figure 2 The imaging direction shown is a two-dimensional image of the front side (a hemispherical range with a circumference of 180 degrees).

[0052] Each of the imaging elements 11 a and 11A converts light focused on an imaging area into an electrical signal using a pixel circuit of each pixel, and outputs high-resolution RGB image data.

[0053] Each fisheye lens 11b, 11B collects images from a wide-angle area (e.g., Figure 2The imaging direction shown is the light of a hemispherical range having a circumference of 180 degrees as the front side, and the light is focused on the imaging area of ​​each imaging element 11a, 11A.

[0054] Each light source unit 12a, 12A is, for example, a semiconductor laser that emits a laser beam light of a wavelength band (e.g., infrared light) outside the visible light region to measure a range or distance. Each light source unit 12a, 12A can use a single semiconductor laser or a combination of multiple semiconductor lasers. In addition, the semiconductor laser can use a surface-emitting semiconductor laser, such as a vertical cavity surface emitting laser (VCSEL).

[0055] Furthermore, an optical lens may be provided to form the light of the semiconductor laser into a longitudinal light, and then the longitudinal light may be scanned in a one-dimensional direction of the measurement range using an optical deflection element such as a micro-electromechanical system (MEMS) mirror.

[0056] In this specification, the light source units 12 a and 12A do not use optical deflection elements such as MEMS mirrors, but emit light from the semiconductor laser LA to a wide angle area through the wide-angle lenses 12 b and 12B.

[0057] Wide-angle lens of light source unit 12a, 12A The wide-angle lens has a function of expanding the light emitted from light source unit 12a, 12A to a wide angle region (for example, Figure 2 The camera direction is shown as a function of the front hemispherical range with a circumference of 180 degrees.

[0058] The wide-angle lenses 13b and 13B of the distance information acquisition unit 13 measure the distance from a wide-angle region (for example, Figure 2 The reflected light of the light source units 12a and 12A corresponding to the light projected by the projection unit 12 is received in each direction (a hemispherical range with a circumference of 180 degrees on the front side) and is focused on the light receiving areas of the TOF sensors 13a and 13A respectively.

[0059] The measurement range represents a space including one or more objects (for example, a target object such as a building) to be projected with light. Light reflected from the target object (reflected light) enters each wide-angle lens 13b, 13B. For example, the reflected light can be received by setting a filter for cutting off light having a wavelength range shorter than the infrared wavelength range on the entire surface of each wide-angle lens 13b, 13b. In addition, the configuration is not limited to this. For example, since it is sufficient as long as light in the infrared region enters the light receiving area, a unit such as a filter for passing light in the infrared region can be set in the optical path from the wide-angle lenses 13b, 13b to the light receiving area.

[0060] Each TOF sensor 13a, 13A is an optical sensor with a two-dimensional resolution. Each TOF sensor 13a, 13A has a light receiving area in which a large number of light receiving elements (such as photodiodes) are arranged in a two-dimensional direction. In this description, each TOF sensor 13a, 13A is sometimes referred to as a "second camera light receiving unit". Each TOF sensor 13a, 13A uses a light receiving element corresponding to each area to receive the reflected light of each area in the measurement range (each area is also called a position), and then measures or calculates the distance or range to each area based on the light detected by each light receiving element.

[0061] In this embodiment, a phase difference detection method is used to measure the distance. In the phase difference detection method, a laser beam light with amplitude modulated at a basic frequency is irradiated toward the measurement range to receive reflected light, and then the phase difference between the irradiated light and the reflected light is measured to obtain or obtain the time period between the emission start time of the irradiated light and the reception time of the reflected light, and the distance is calculated by multiplying the time period by the speed of light. Phase difference detection can ensure a certain degree of resolution.

[0062] Each TOF sensor 13a, 13A is driven synchronously with the projection of the irradiated light of the projection unit 12, calculates the distance corresponding to each pixel from the phase difference of the reflected light at each photosensitive element (corresponding to each pixel), and outputs image data including distance information (hereinafter referred to as "distance image" or "TOF image") that is associated with the pixel information and represents the distance to each area within the measurement range.

[0063] Furthermore, each TOF sensor 13 a , 13A may be configured to output phase information image data in which phase information is associated with pixel information, and acquire image data including distance information based on the phase information image data by performing post-processing.

[0064] In addition, the number of divided areas in the measurement range can be determined in consideration of the resolution of the light receiving area. Therefore, when a low resolution is used to reduce the size of the device, the amount of pixel information of the distance image data is reduced, and the number of three-dimensional point groups is also reduced.

[0065] In addition, instead of the phase difference detection method, a pulse method may be used to measure the distance. In the pulse method, for example, each light source unit 12a, 12A emits an ultra-short pulse irradiation pulse P1 having a rise time of several nanoseconds (ns) and a relatively strong intensity peak power, and in synchronization with the emission of the irradiation pulse P1, each TOF sensor 13a, 13A measures the time (t) required to receive a reflection pulse P2, which is the reflected light corresponding to the irradiation pulse P1 emitted from the light source unit 12a, 12A. When the pulse method is adopted, for example, each TOF sensor 13a, 13A includes one or more circuits for measuring time on the output side of the light receiving element. Each circuit converts the time period required to receive the reflection pulse P2 after each light source unit 12a, 12A emits the irradiation pulse P1 into the distance information of each light receiving element to acquire or obtain the distance to each area.

[0066] The pulse method is suitable for setting a wide-angle area for the camera 1 because the pulse method can output light with a stronger intensity using peak light. In addition, when a MEMS mirror or the like is used to oscillate or scan light, while reducing the diffusion angle of the light, it is possible to irradiate light with a stronger intensity to a distant place, thereby increasing the measurement distance. In this case, the light source units 12a, 12A, the wide-angle lenses 12b, 12B, and the MEMS mirror are configured in a prescribed configuration so that the laser beam light emitted from each light source unit 12a, 12A is scanned or deflected toward the wide-angle lenses 12b, 12B through the MEMS mirror.

[0067] In addition, the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 are preferably matched with each other, for example, 180 degrees or more, but the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 do not necessarily have to match each other. The effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 can be reduced as needed.

[0068] In the present embodiment, the imaging unit 11 and the distance information acquisition unit 13 reduce the number of effective pixels within a range of 100 to 180 degrees, for example, so that the subject of the imaging device 1 and the distance information acquisition unit 13 are not included in the angle of view.

[0069] In addition, the resolution of the TOF sensors 13a, 13A can be set to be lower than the resolution of the imaging elements 11a, 11A to give priority to the size reduction or miniaturization of the imaging device 1. By setting the resolution of the TOF sensors 13a, 13A to be lower than the resolution of the imaging elements 11a, 11A, the size of the imaging device 1 can be reduced because the size of the light receiving area can be reduced. Therefore, each TOF sensor 13a, 13A can be set to a low resolution, and the three-dimensional point group acquired or obtained by each TOF sensor 13a, 13A has a low density, but because the processing circuit 14 used as an "acquisition unit" is provided, the three-dimensional point group can be converted into a high-density three-dimensional point group. The process of converting to a high-density three-dimensional point group using the processing circuit 14 will be described later.

[0070] In this embodiment, as an example, the imaging element 11a, the light source unit 12a, and the TOF sensor 13a are arranged in a straight line along the long side direction of the housing 10. Similarly, the imaging element 11A, the light source unit 12A, and the TOF sensor 13A are arranged in a straight line along the long side direction of the housing 10. Hereinafter, an example of the imaging element 11a, the light source unit 12a, and the TOF sensor 13a will be described.

[0071] like Figure 2 As shown, the imaging area (imaging surface) of the imaging element 11a and the light receiving area (light receiving surface) of the TOF sensor 13a can be arranged in a direction orthogonal to the long side direction, or can be arranged in the long side direction by setting a prism that changes the path of the light direction (optical path) by 90 degrees. In addition, the imaging element 11a, the light source unit 12a and the TOF sensor 13a can be arranged in any direction according to various configurations. That is, the imaging element 11a, the light source unit 12a and the TOF sensor 13a are arranged in any direction so that the same measurement range can be set to measure the distance to the target object. The imaging unit 11, the projection unit 12 and the distance information acquisition unit 13 are arranged on one side of the housing 10 to face the measurement range. In this configuration, the imaging element 11a and the TOF sensor 13a can be arranged along the same baseline to form a parallel three-dimensional configuration. By configuring the imaging element 11a and the TOF sensor 13a using a parallel stereo configuration, even if the number of the imaging element 11a is one, parallax data can be acquired or obtained using the output of the TOF sensor 13a. The light source unit 12a is configured to irradiate light to the measurement range of the TOF sensor 13a.

[0072] (Processing Circuit)

[0073] The processing of the processing circuit 14 is described below. The TOF image acquired or obtained by the TOF sensor 13a, 13A has a low resolution. Therefore, in this embodiment, the resolution is improved by the processing circuit 14, and then high-density three-dimensional point group data is reconstructed. In addition, part or all of the following processing performed by the "information processing unit" such as the processing circuit 14 can be performed by an external device or apparatus.

[0074] As described above, the three-dimensional point group data reconstructed by the imaging device 1 is output to an external device (eg, a PC) via a portable recording medium or a communication network, and is used to display a three-dimensional restoration model.

[0075] According to this configuration, compared with the case where the imaging device 1 displays a three-dimensional restoration model on the imaging device 1 , it is possible to provide the imaging device 1 having a higher speed for processing data, a smaller size, and a lighter weight.

[0076] However, after leaving the location where the user acquired the three-dimensional information and restoring the three-dimensional information through an external device, the user may recognize that the user or the tripod is included in the captured image and that the three-dimensional information having the desired layout has not been acquired. In this case, the user needs additional time and effort to revisit the location where the three-dimensional information was acquired.

[0077] This problem can be solved by bringing a three-dimensional restoration device to the site, but if the user carries the three-dimensional restoration device, it is impossible to design the imaging device 1 which can process data at a high processing speed and has a small size and light weight.

[0078] In addition, this problem can be solved by sending the acquired three-dimensional information to an external device via a communication line and receiving the restored three-dimensional information from the external device. However, this method does not have the advantage of high processing speed. In addition, since the three-dimensional information has a large amount of information, it is difficult for the user to visually confirm that the user himself, the tripod, etc. are included in the captured image.

[0079] Particularly, in the case of full-view spherical three-dimensional information, it is extremely difficult for a user to visually confirm that the user himself, a tripod, and the like are included in a captured image.

[0080] The imaging device 1 of the present embodiment can easily confirm in real time the above-mentioned problem that the three-dimensional information of the user himself, the tripod, etc., is included in the captured image and the desired layout has not been acquired.

[0081] Figure 3A , 3B 3C and 3D show examples of using the imaging device 1 according to the present embodiment.

[0082] exist Figure 3AIn the illustrated situation, the user M and the selfie stick 1A holding the camera 1 are not included in the panoramic spherical image capturing range R, so that the user M and the selfie stick 1A are not included in the panoramic spherical image (ie, the captured image). In this illustration, the user M operates the camera 1 to capture an image.

[0083] exist Figure 3B In the illustrated situation, since the user M is included in the panoramic spherical image capturing range R, the user M is included in the panoramic spherical image (ie, the captured image).

[0084] exist Figure 3C In the illustrated case, since the tripod 1B holding the image pickup apparatus 1 is included in the panoramic spherical image pickup range R, the tripod 1B is included in the panoramic spherical image (ie, the captured image).

[0085] exist Figure 3D In the illustrated situation, the user M and the selfie stick 1A holding the camera device 1 are not included in the panoramic spherical image capturing range R, so the user M and the selfie stick 1A are not included in the panoramic spherical image (i.e., the captured image). However, since external light (e.g., sunlight, light from lighting equipment) has a strong intensity, it is possible to erroneously determine that an object is included in the captured image.

[0086] In addition, Figure 3B and 3C In the illustrated situation, since the colors, kinds, and appearances of the objects included in the captured image are various, it is difficult for the conventional method to determine whether a target object such as a user or the like is included in the captured image.

[0087] If it happens Figure 3B and 3C In the above-mentioned case, when determining the presence or absence of a specific object (e.g., a nearby object) such as the user himself or the tripod based on the image data including distance information output from the TOF sensors 13a, 13A, it is difficult to distinguish whether the specific object really exists or the external light is too strong.

[0088] That is, when the charge amount of a specific pixel of the TOF sensor 13a, 13A is saturated, it is difficult to distinguish whether the saturated charge amount is caused by the presence of a specific object or by excessive intensity of external light only from the output of the TOF sensor 13a, 13A.

[0089] In view of the above, the imaging apparatus 1 is configured to correctly confirm whether a specific object such as the user himself or a nearby object (eg, a tripod) is included in a captured image by distinguishing the specific object from the influence of external light.

[0090] Figure 41 is an example of a functional block diagram of the processing circuit 14. Figure 4 As shown, the processing circuit 14 includes, for example, a control unit 141, an RGB image data acquisition unit 142, a monochrome processing unit 143, a TOF image data acquisition unit 144, a high-resolution processing unit 145, a matching processing unit 146, a re-projection processing unit 147, a semantic segmentation unit 148, a parallax calculation unit 149, a three-dimensional reconstruction processing unit 150, a determination unit 160, a display control unit 170 (an example of an output unit), and a transmission-reception unit 180 (an example of an output unit). Figure 4 In the figure, the solid arrows represent the flow of signals, and the dotted arrows represent the flow of data.

[0091] In response to receiving an ON signal (imaging start signal) from the imaging switch 15 , the control unit 141 outputs a synchronization signal to the imaging elements 11 a , 11A, the light source units 12 a , 12A, and the TOF sensors 13 a , 13A, and controls the entire processing circuit 14 .

[0092] First, the control unit 141 outputs a signal instructing the light source units 12a and 12A to emit ultrashort pulse light, and at the same time, outputs a signal instructing the TOF sensors 13a and 13A to generate TOF image data. In addition, the control unit 141 outputs a signal instructing the imaging elements 11a and 11A to capture an image. In this configuration, the imaging performed by the imaging elements 11a and 11A can be performed during the period when light is emitted from the light source units 12a and 12A, or before and after the light is emitted from the light source units 12a and 12A.

[0093] The RGB image data acquisition unit 142 acquires the RGB image data captured by the image pickup elements 11 a and 11A according to the image pickup instruction of the control unit 141 , and outputs the RGB image data of the panoramic spherical image.

[0094] The monochrome processing unit 143 performs a process of setting the data type for matching processing with the TOF image data acquired or obtained from the TOF sensor 13a, 13A. In this example, the monochrome processing unit 143 converts the RGB image data of the panoramic spherical image into monochrome image data of the panoramic spherical image.

[0095] The TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13 a and 13A according to the TOF image data generation instruction of the control unit 141 , and outputs the TOF image data of the panoramic spherical image.

[0096] The high-resolution processing unit 145 assumes the TOF image data of the panoramic spherical image to be monochrome image data and increases the resolution of the TOF image data. Specifically, the high-resolution processing unit 145y replaces the distance value associated with each pixel of the TOF image data of the panoramic spherical image with the value (grayscale value) of the monochrome image data of the panoramic spherical image.

[0097] In addition, the high-resolution processing unit 145 increases the resolution of the monochrome image data of the panoramic spherical image to the resolution of the RGB image data of the panoramic spherical image acquired or obtained from the imaging element 11a, 11A. For example, conversion to high resolution is performed by performing normal conversion increase processing.

[0098] In addition, other conversion methods may be applied, such as acquiring multiple frames of TOF images of a continuously generated panoramic spherical image, and then performing super-resolution processing by appending the distances between adjacent points using these frames.

[0099] The matching processing unit 146 extracts a feature quantity at a portion having a texture from the monochrome image data of the panoramic spherical image acquired or obtained by converting the TOF image data of the panoramic spherical image into high-resolution image data and the monochrome image data of the panoramic spherical image corresponding to the RGB image data of the panoramic spherical image, and then performs matching processing based on the extracted feature quantity. For example, the matching processing unit 146 extracts an edge from each monochrome image data and then performs matching processing on the extracted edge information.

[0100] In addition, other conversion methods may be applied. For example, a method of quantifying texture changes such as Scale Invariant Feature Transform (SIFT) may be used to perform the matching process. In this specification, the matching process means a search for corresponding pixels.

[0101] As a method of matching processing, for example, block matching can be used. Block matching is a method of calculating the similarity between the pixel values ​​of a block cut into M×M (M is a positive integer) pixel size near the pixel to be referenced and the pixel values ​​of a block cut into M×M pixel size near the center pixel to be searched in the corresponding image, and taking the center pixel with the highest similarity as the corresponding pixel.

[0102] Various methods can be used to calculate similarity. For example, a formula representing a normalized correlation coefficient (NCC) such as a normalized autocorrelation coefficient can be used. The higher the NCC value, the higher the similarity, and when the pixel values ​​of the blocks are completely matched, the NCC value becomes 1.

[0103] In addition, since the distance data of the texture-free area is also acquired or obtained from the TOF image data of the panoramic spherical image, the matching process can be performed by setting weights according to the area. For example, when the calculation is performed using the formula representing the NCC, the calculation can be performed by setting weights to the part other than the edge (texture-free area).

[0104] In addition, instead of the formula representing NCC, the selective correlation coefficient (SCC) may be used.

[0105] The re-projection processing unit 147 re-projects the TOF image data of the panoramic spherical image representing the distance of each position within the measurement range onto the two-dimensional coordinates (screen coordinate system) of the imaging unit 11. The re-projection is performed to determine to which two-dimensional coordinates of the image captured by the imaging element 11a, 11A the three-dimensional point calculated by the TOF sensor 13a, 13A will be projected.

[0106] The TOF image data of the panoramic spherical image indicates the position of each three-dimensional point in a coordinate system centered on the distance information acquisition unit 13 (mainly the wide-angle lenses 13 b and 13B).

[0107] Therefore, the three-dimensional point represented by the TOF image data of the panoramic spherical image is reprojected onto a coordinate system centered on the imaging unit 11 (mainly the fisheye lenses 11b, 11B). For example, the reprojection processing unit 147 parallel-shifts the coordinates of the three-dimensional point of the TOF image data of the panoramic spherical image to the coordinates of the three-dimensional point centered on the imaging unit 11. After the parallel shift, the reprojection processing unit 147 converts the coordinates of the three-dimensional point of the TOF image data into a two-dimensional coordinate system (screen coordinate system) represented by the RGB image data of the panoramic spherical image.

[0108] According to this configuration, the coordinates of the three-dimensional points of the TOF image data of the panoramic spherical image and the coordinates of the two-dimensional image information can be associated with each other.

[0109] The re-projection processing unit 147 associates the coordinates of the three-dimensional points of the TOF image data of the panoramic spherical image with the coordinates of the two-dimensional image information of the panoramic spherical image captured by the imaging unit 11 .

[0110] The parallax calculation unit 149 calculates the parallax of each position from the deviation or fluctuation regarding the distance of the corresponding pixels acquired by the matching process.

[0111] Furthermore, the reprojection coordinates converted by the reprojection processing unit 147 can be used to perform parallax matching processing by searching for surrounding pixels at the position of the reprojection coordinates, so that the processing time can be shortened and more detailed and higher-resolution distance information can be acquired or obtained.

[0112] In addition, segmentation data acquired or obtained by performing semantic segmentation processing using the semantic segmentation unit 148 can be used for disparity matching processing. In this case, more detailed and higher-resolution distance information can be acquired or obtained.

[0113] Furthermore, parallax matching processing may be performed only on edges or portions having a large feature amount, and propagation processing may be performed on other portions using TOF image data of a panoramic spherical image and, for example, RGB image features of the panoramic spherical image and a random technique.

[0114] The semantic segmentation unit 148 uses deep learning to apply the segmentation label representing the target object to the input image of the measurement range. Through this processing, each pixel of the TOF image data of the panoramic spherical image can be fixed to any one of the multiple distance areas divided for each distance, which can further improve the reliability of the calculation.

[0115] The three-dimensional reconstruction processing unit 150 obtains the RGB image data of the panoramic spherical image from the RGB image data acquisition unit 142, reconstructs the three-dimensional data of the panoramic spherical image based on the distance information output from the parallax calculation unit 149, and then outputs a high-density three-dimensional point group of the panoramic spherical image with color information added to each three-dimensional point.

[0116] The three-dimensional reconstruction processing unit 150 is an example of a three-dimensional information specifying unit that specifies three-dimensional information.

[0117] The determination unit 160 acquires the RGB image data of the panoramic spherical image from the RGB image data acquisition unit 142, and also acquires the TOF image data of the panoramic spherical image converted into a two-dimensional coordinate system represented by the RGB image data of the panoramic spherical image from the re-projection processing unit 147, and then determines whether a specific object is included in the captured image based on these data, and outputs the determination result to the display control unit 170.

[0118] The display control unit 170 acquires the RGB image data of the panoramic spherical image from the RGB image data acquisition unit 142 , and instructs the display unit 20 to display the two-dimensional image information based on the acquired RGB image data of the panoramic spherical image.

[0119] Furthermore, the display control unit 170 superimposes the information indicating the determination result acquired or obtained from the determination unit 160 on the two-dimensional image information, and then instructs the display unit 20 to display the two-dimensional image information.

[0120] The display control unit 170 is an example of an output unit that outputs the two-dimensional image information captured by the imaging unit 11 separately from the three-dimensional information. The display unit 20 is an example of an output destination to which the two-dimensional image information is output.

[0121] The display control unit 170 may be configured to acquire or obtain the three-dimensional data of the panoramic spherical image from the three-dimensional reconstruction processing unit 150, and instruct the display unit 20 to display the three-dimensional information. Specifically, the display control unit 170 may select a mode in which the display control unit 170 instructs the display unit 20 to display two-dimensional image information, and another mode in which the display control unit 170 instructs the display unit 20 to display three-dimensional information according to a preset condition. According to this configuration, the display control unit 170 may output the two-dimensional image information separately from the three-dimensional information.

[0122] The transmitting-receiving unit 180 communicates with the external device by wire or wirelessly. The transmitting-receiving unit 180 transmits or outputs the three-dimensional data of the panoramic spherical image output from the three-dimensional reconstruction processing unit 150 and the two-dimensional image information of the panoramic spherical image output from the RGB image data acquisition unit 142 to the external device 300 that performs the three-dimensional restoration processing via the network 400.

[0123] In this description, the two-dimensional image information captured by the camera unit 11 refers to "raw two-dimensional image information" or "two-dimensional image data for display" used to create "two-dimensional image data for display." For example, the two-dimensional image information captured by the camera unit 11 (i.e., raw two-dimensional image information) may be processed inside the camera device 1 to create "two-dimensional image data for display," or the two-dimensional image information captured by the camera unit 11 (i.e., raw two-dimensional image information) may be sent from the camera device 1 to an external device, and the external device may then create "two-dimensional image data for display" from the "raw two-dimensional image information."

[0124] The transmission / reception unit 180 is an example of an output unit that outputs three-dimensional information. The external device 300 is an example of an output destination to which the three-dimensional information is output.

[0125] In addition, the transmitting-receiving unit 180 may be configured not to transmit the two-dimensional image information of the panoramic spherical image, but to transmit only the three-dimensional data of the panoramic spherical image. In addition, the transmitting-receiving unit 180 may be configured to use an interface circuit capable of communicating with a portable storage medium such as a secure digital (SD) card and a personal computer.

[0126] (Processing circuit operation)

[0127] Figure 5 The figure shows an example of a flowchart of the operation of the processing circuit 14 of the camera device 1. When the camera switch 15 is turned on by the user and the camera instruction signal is input, the control unit 141 of the processing circuit 14 uses the following method (an example of the camera / processing method and the information processing method) to perform the operation of generating a high-density three-dimensional point group.

[0128] First, the control unit 141 drives the light source units 12a, 12A, the TOF sensors 13a, 13A, and the imaging elements 11a, 11A to capture an image of the measurement range (step S1). The control unit 141 instructs the light source units 12a, 12A to emit infrared light (an example of a light projection step), and then the TOF sensors 13a, 13A receive light reflected from an object (an example of a light receiving step). In addition, the imaging elements 11a, 11A capture an image of the measurement range at the time when the light source units 12a, 12A start to be driven or during a period near the time when the light source units 12a, 12A start to be driven (an example of an imaging step).

[0129] Then, the RGB image data acquisition unit 142 acquires the RGB image data of the measurement range from the imaging elements 11 a and 11A (step S2 ).

[0130] Then, the display control unit 170 acquires the RGB image data of the panoramic spherical image from the RGB image data acquisition unit 142, and then instructs the display unit 20 to display the two-dimensional image information based on the acquired RGB image data of the panoramic spherical image (an example of a two-dimensional image information output step) (step S3).

[0131] The display control unit 170 instructs the display unit 20 to display the two-dimensional image information of a partial area of ​​the RGB image data of the acquired panoramic spherical image, and then changes the area of ​​the two-dimensional image information displayed by the display unit 20 in response to receiving various instructions input by the user. For example, if the imaging device 1 is provided with one or more operation switches other than the imaging switch 15, or the display unit 20 such as a touch panel as an input unit, the user can input various instructions using these switches and units.

[0132] In step S3 , the user can confirm whether the user himself, the tripod, etc. are included in the photographed image and whether the two-dimensional image information of a desired layout has been acquired by viewing the two-dimensional image information displayed using the display unit 20 .

[0133] Then, the TOF image data acquisition unit 144 acquires TOF image data indicating the distance of each position in the two-dimensional area from the TOF sensors 13 a and 13A (step S4 ).

[0134] Then, the monochrome processing unit 143 converts the RGB image data into monochrome image data (step S5). Because the TOF image data is distance data and the RGB image data is RGB data (that is, the data types of the TOF image data and the RGB image data are different), the TOF image data and the RGB image data cannot be matched. Therefore, first, each TOF image data and each RGB image data are converted into monochrome image data. Regarding the TOF image data, before performing resolution enhancement, the high-resolution processing unit 145 converts the value representing the distance of each pixel by directly replacing the value representing the distance of each pixel with the value of the monochrome image data.

[0135] Next, the high-resolution processing unit 145 increases the resolution of the TOF image data (step S6 ).

[0136] Then, the matching processing unit 146 extracts the feature amount of the portion having the texture for each monochrome image data, and then performs matching processing using the extracted feature amount (step S7 ).

[0137] Then, the parallax calculation unit 149 calculates the parallax of each position from the deviation of the distances of the corresponding pixels to calculate the distance (step S8 ).

[0138] Then, the determination unit 160 acquires the RGB image data of the panoramic spherical image from the RGB image data acquisition unit 142, and also acquires the TOF image data of the panoramic spherical image converted into a two-dimensional coordinate system represented by the RGB image data from the re-projection processing unit 147, and then determines whether there is a nearby object (i.e., a specific object) in the captured image based on the RGB image data of the panoramic spherical image and the TOF image data of the panoramic spherical image, and then outputs the determination result to the display control unit 170.

[0139] Then, the display control unit 170 superimposes the information indicating the determination result acquired or obtained from the determination unit 160 on the two-dimensional image information, and instructs the display unit 20 to display the two-dimensional image information (an example of a display step) (step S9).

[0140] Then, the 3D reconstruction processing unit 150 acquires RGB image data from the RGB image data acquisition unit 142, reconstructs 3D data based on the distance information output from the parallax calculation unit 149, and outputs a high-density 3D point group in which color information is added to each 3D point (step S10).

[0141] Then, the sending-receiving unit 180 sends the three-dimensional data output from the three-dimensional reconstruction processing unit 150 and the two-dimensional image information output from the RGB image data acquisition unit 142 to the external device 300 that performs the three-dimensional restoration processing (an example of a three-dimensional information output step) via the network 400 (step S11).

[0142] The transmitting-receiving unit 180 may be configured to transmit the three-dimensional data output from the three-dimensional reconstruction processing unit 150 without transmitting the two-dimensional image information output from the RGB image data acquiring unit 142 .

[0143] As described above, the imaging device 1 includes the imaging unit 11 and the display control unit 170 that outputs two-dimensional image information captured by the imaging unit 11 separately from three-dimensional information.

[0144] According to this configuration, without confirming three-dimensional information, the user can easily confirm from two-dimensional image information whether the user himself, a tripod, etc. are included in the captured image and whether three-dimensional information having a desired layout has been acquired.

[0145] Therefore, the user can reacquire the three-dimensional information while staying at the place where the three-dimensional information is acquired or obtained, which can reduce the work or labor of revisiting the place where the three-dimensional information is acquired or obtained, compared to a situation where the user recognizes that the captured image includes the user himself, a tripod, etc. or that the three-dimensional information with the desired layout is not acquired after leaving the place where the three-dimensional information is acquired.

[0146] In this description, the three-dimensional information includes, for example, three-dimensional information of a panoramic spherical image. In this case, if the panoramic spherical three-dimensional information is captured and the user feels it is difficult to confirm whether the captured image includes the user, a tripod, etc., and whether the three-dimensional information with the desired layout has been acquired, the user can easily confirm whether the captured image includes the user, a tripod, etc., and whether the three-dimensional information with the desired layout has been acquired from the two-dimensional image information captured by the camera unit 11.

[0147] Before the transmission-reception unit 180 transmits or outputs the three-dimensional information in step S11, the display control unit 170 outputs the two-dimensional image information G in step S3. Also, before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10, the display control unit 170 outputs the two-dimensional image information G in step S3.

[0148] According to this configuration, before confirming three-dimensional information, the user can confirm from the two-dimensional image information whether the user himself, the tripod, etc. are included in the captured image and whether three-dimensional information having a desired layout has been acquired.

[0149] The camera device 1 may include a display unit 20 , wherein the display control unit 170 instructs the display unit 20 to display the two-dimensional image information.

[0150] According to this configuration, the user can easily confirm whether the user himself, the tripod, etc. are included in the captured image from the two-dimensional image information displayed using the display unit 20, and whether three-dimensional information having a desired layout has been acquired.

[0151] The display control unit 170 may output the two-dimensional image information to the display unit 20 different from the external device 300 to which the transmission-reception unit 180 outputs the three-dimensional information.

[0152] According to this configuration, the user can confirm whether the captured image includes the user himself, a tripod, etc., and whether three-dimensional information with a desired layout has been obtained from the two-dimensional image information output to the display unit 20 different from the external device 300, without confirming the three-dimensional information output to the external device 300.

[0153] The imaging device 1 may include a 3D reconstruction processing unit 150 that determines 3D information based on the output of the distance information acquisition unit 13. The 3D reconstruction processing unit 150 determines 3D information based on the output received from the distance information acquisition unit 13 and the 2D image information.

[0154] According to this configuration, the user can confirm whether the captured image includes the user himself, a tripod, etc., and whether the two-dimensional image information captured from the camera unit 11 has acquired three-dimensional information with the desired layout, without confirming the three-dimensional information determined by the three-dimensional reconstruction processing unit 150.

[0155] FIG. 6 shows an example of a flowchart of a process for generating image data of a panoramic spherical image according to the present embodiment.

[0156] Fig. 6A The shown corresponds to the reference Figure 5 An example of a flowchart of the process of generating RGB image data of a panoramic spherical image in step S2 will be described.

[0157] The RGB image data acquisition unit 142 inputs two frames of RGB image data in a fisheye image format (step S201 ).

[0158] Then, the RGB image data acquisition unit 142 converts each of the two frames of RGB image data into image data having an equidistant cylindrical image format (step S202). The RGB image data acquisition unit 142 converts the two frames of RGB image data into image data having an equidistant cylindrical image format based on the same coordinate system, so as to facilitate image combination or splicing in the next step S203. It should be noted that the RGB image data can also be converted into image data using one or more image formats other than the equidistant cylindrical image format as needed. For example, the RGB image data can be converted into the coordinates of an image perspectively projected on any surface or the coordinates of an image perspectively projected on each surface of an arbitrary polyhedron.

[0159] The following describes the equidistant cylindrical image format. The equidistant cylindrical image format is a format capable of expressing a panoramic spherical image, and is a format of an image (i.e., an equidistant cylindrical image) created using the equidistant cylindrical method. The equidistant cylindrical method is a method of representing a three-dimensional direction using two variables such as the latitude and longitude of the earth, and displaying a planar image in which the latitude and longitude are orthogonal to each other. Therefore, an equidistant cylindrical image is an image generated using the equidistant cylindrical method, and is represented using two angular variables of a spherical coordinate system as coordinates of two axes.

[0160] Then, the RGB image data acquisition unit 142 combines or splices the two frames of RGB image data generated in step S202 to generate one image of RGB image data of a panoramic spherical image (step S203). Each of the two frames of RGB image data inputted covers an area having a full viewing angle greater than 180 degrees. Therefore, the panoramic spherical image RGB image data generated by appropriately combining or splicing the two frames of RGB image data can cover the panoramic spherical image area.

[0161] In addition, the combining or splicing process in step S203 may use any known technique for combining or splicing a plurality of images, and the method is not particularly limited.

[0162] Figure 6B The shown corresponds to the reference Figure 5 An example of a flowchart of the process of generating TOF image data of a panoramic spherical image in step S4 will be described.

[0163] The TOF image data acquisition unit 144 acquires two frames of range image data in a fisheye image format (step S401 ).

[0164] Then, the TOF image data acquisition unit 144 converts the two frames of TOF image data in the fisheye image format into image data in the equidistant cylindrical image format (step S402). As described above, the equidistant cylindrical image format is a format capable of representing a panoramic spherical image. In step S402, the two frames of TOF image data are converted into image data in the equidistant cylindrical image format based on the same coordinate system to facilitate the combination or splicing of the images in the next step S403.

[0165] Then, the TOF image data acquisition unit 144 combines or splices the two frames of TOF image data generated in step S402 to generate TOF image data of a panoramic spherical image (step S403). Each of the two frames of TOF image data input covers an area with a full viewing angle greater than 180 degrees. Therefore, the panoramic spherical image TOF image data generated by appropriately combining or splicing the two frames of TOF image data can cover the panoramic spherical image area.

[0166] In addition, the combining or splicing process in step S403 may use any known technique for combining or splicing a plurality of images, and the method is not particularly limited.

[0167] Figure 7 FIG. 1 is an example of a flowchart for determining a nearby object according to the present embodiment.

[0168] Figure 7 FIG. 1 is a flowchart of a process for determining whether a nearby object is included in a captured image. The process corresponds to the process of referring to FIG. Figure 5 Step S9 is described.

[0169] The determination unit 160 determines whether the TOF image data of the panoramic spherical image has one or more pixels with saturated charge amounts based on the TOF image data of the panoramic spherical image acquired or obtained from the re-projection processing unit 147 (step S801 ).

[0170] If the determination unit 160 determines in step S801 that the TOF image data of the panoramic spherical image has more than one pixel with a saturated charge amount (S801: "Yes"), based on the RGB image data of the panoramic spherical image acquired or obtained from the RGB image data acquisition unit 142, the determination unit 160 determines whether the RGB image data of the panoramic spherical image has one or more pixels having the same coordinates as the one or more pixels with a saturated charge amount in step S801, and the pixels have a saturated charge amount (step S802).

[0171] If the determination unit 160 determines in step S802 that the RGB image data of the panoramic spherical image has more than one pixel with a saturated charge amount (S802: "Yes"), the determination unit 160 determines that the pixel with a saturated charge amount in step S801 is caused by external light (e.g., sunlight, light from a lighting device), and outputs an error message to the display control unit 170.

[0172] Then, based on the error information acquired or obtained from the determination unit 160, the display control unit 170 superimposes the error information on the two-dimensional image information, and then instructs the display unit 20 to display the two-dimensional image information superimposed with the error information (step S803).

[0173] If the determination unit 160 determines in step S802 that the RGB image data of the panoramic spherical image does not have pixels with saturated charge amounts (S802: "No"), the determination unit 160 determines that the pixels with saturated charge amounts in step S801 are caused by the presence of nearby objects, and outputs the coordinate position information of the pixels with saturated charge amounts to the display control unit 170 in step S801.

[0174] Then, based on the coordinate position information of the pixel acquired or obtained from the determination unit 160, the display control unit 170 superimposes the identification information for identifying the nearby objects on the two-dimensional image information, and then instructs the display unit 20 to display the identification information for identifying the nearby objects superimposed on the two-dimensional image information (step S804).

[0175] If the determination unit 160 determines in step S801 that the TOF image data of the panoramic spherical image does not have pixels with saturated charge amount (S801: "No"), based on the TOF image data of the panoramic spherical image acquired or obtained from the re-projection processing unit 147, the determination unit 160 determines whether the TOF image data of the panoramic spherical image has specified distance information in the TOF image data representing the panoramic spherical image, for example, one or more pixels below 0.5 m (step S805).

[0176] If the determination unit 160 determines in step S805 that the TOF image data of the panoramic spherical image has no pixels indicating distance information of less than 0.5 m (S805: No), the determination unit 160 ends the sequence.

[0177] If the determination unit 160 determines in step S805 that the TOF image data of the panoramic spherical image has one or more pixels indicating distance information of 0.5 m or less (S805: “Yes”), the determination unit 160 continues the sequence to step S804.

[0178] If step S805 is “yes”, the determination unit 160 determines that the pixel representing the distance information less than 0.5 m is caused by the presence of a nearby object, and outputs the coordinate position information of the pixel representing the distance information less than 0.5 m to the display control unit 170 in step S805.

[0179] Then, in step S804 , based on the coordinate position information of the pixel acquired or obtained from the determination unit 160 , the display control unit 170 instructs the display unit 20 to display the identification information identifying the nearby object by superimposing the identification information identifying the nearby object on the two-dimensional image information.

[0180] As described above, if the determination unit 160 determines that a nearby object exists, the display control unit 170 superimposes the identification information on the two-dimensional image information, and if the determination unit 160 determines that a nearby object does not exist, the display control unit 170 does not superimpose the identification information on the two-dimensional image information.

[0181] That is, the display control unit 170 instructs the display unit 20 to output different displays depending on the presence or absence of a nearby object.

[0182] Furthermore, based on the coordinate position information of the pixel acquired or obtained from the determination unit 160 , the display control unit 170 instructs the display unit 20 to display the identification information identifying the nearby object on the two-dimensional image information.

[0183] That is, the display control unit 170 instructs the display unit 20 to set the display position of the nearby objects according to the positions of the nearby objects in the space, which means that the display position of each nearby object becomes different according to the position of each nearby object in the space.

[0184] Figure 8 FIG. 1 is an example of the content of the view displayed using the display unit according to the present embodiment.

[0185] Figure 8 The corresponding Figure 5 Step S3 and Figure 7 FIG. 8 is a diagram of steps S803 and S804 in FIG.

[0186] like Figure 8 As shown, the display control unit 170 instructs the display unit 20 to display the two-dimensional image information G. In addition, as shown Figure 8 As shown, by superimposing the recognition information G1, G2 and the error information G3 on the two-dimensional image information G, the display control unit 170 instructs the display unit 20 to display the information G1, G2 and the error information G3 for recognizing each nearby object (e.g., finger, tripod). The error information G3 may be Figure 8 The symbol shown representing "sunlight" can also mean "lighting equipment".

[0187] As described above, the camera device 1 includes a camera unit 11 for capturing an image of a target object, a projection unit 12 for projecting light onto the target object, a distance information acquisition unit 13 for receiving light reflected from the target object, and a display control unit 170 for instructing the display unit 20 to output different displays according to the presence or absence of a nearby object determined based on output information received from the distance information acquisition unit 13 and output information from the camera unit 11.

[0188] According to this configuration, by distinguishing the nearby object from the influence of external light, the user can correctly confirm whether the user himself and the nearby object (for example, a tripod) are included in the captured image.

[0189] As described above, the image pickup apparatus 1 is provided with the display unit 20. According to this configuration, the user can correctly check whether a nearby object is included in a captured image.

[0190] As described above, the display control unit 170 instructs the display unit 20 to set the display position of the nearby object according to the position of the nearby object in space, which means that the display position of the nearby object becomes different according to the position of each nearby object in space. According to this configuration, the user can confirm the position of each nearby object included in the captured image.

[0191] As described above, the display control unit 170 instructs the display unit 20 to display the image information G captured by the camera unit 11, and instructs the display unit 20 to display the identification information G1 and G2 for identifying each nearby object by superimposing the identification information G1 and G2 on the image information. According to this configuration, the user can correctly confirm the position of each nearby object included in the captured image.

[0192] As described above, the imaging device 1 includes the determination unit 160 that determines that there is a nearby object when the charge amount due to the light received by the distance information acquisition unit 13 is saturated and the charge amount of the pixel in the imaging unit 11 is not saturated.

[0193] According to this configuration, by distinguishing the nearby object from the influence of external light, the user can correctly confirm whether the nearby object is included in the captured image.

[0194] Fig. 9 1 is an example of the appearance of an image pickup device 1 according to a first modified example of the present embodiment. Fig.10 FIG. 1 is an example of a functional block diagram of a processing circuit according to a first variation of the present embodiment.

[0195] In the first modification, the display control unit 170 acquires the RGB image data of the panoramic spherical image from the RGB image data acquisition unit 142, and instructs the display unit 520 of the display device 500 to display the two-dimensional image information based on the acquired RGB image data of the panoramic spherical image. The display unit 520 is an example of an output destination for outputting the two-dimensional image information.

[0196] According to this configuration, the user can easily confirm whether the user himself, the tripod, etc. are included in the captured image from the two-dimensional image information displayed by the display unit 520, and whether three-dimensional information having a desired layout has been acquired.

[0197] The display control unit 170 outputs the two-dimensional image information to the display unit 520 which is different from the external device 300 to which the transmission-reception unit 180 outputs the three-dimensional information.

[0198] According to this configuration, the user can confirm whether the captured image includes the user himself, a tripod, etc., and whether three-dimensional information with a desired layout has been obtained from the two-dimensional image information output to the display unit 520 different from the external device 300, without confirming the three-dimensional information output to the external device 300.

[0199] In addition, the display control unit 170 may be configured to acquire or obtain the three-dimensional data of the panoramic spherical image from the three-dimensional reconstruction processing unit 150, and instruct the display unit 520 to display the three-dimensional information. Specifically, the display control unit 170 may select a case where the display control unit 170 instructs the display unit 520 to display the two-dimensional image information, and another case where the display control unit 170 instructs the display unit 520 to display the three-dimensional information according to a preset condition. According to this configuration, the display control unit 170 may output the two-dimensional image information separately from the three-dimensional information.

[0200] The display control unit 170 superimposes error information on the two-dimensional image information based on the error information acquired or obtained from the determination unit 160 , and instructs the display unit 520 to display the two-dimensional image information superimposed with the error information.

[0201] The display control unit 170 instructs the display unit 520 to display identification information identifying nearby objects superimposed on the two-dimensional image information based on the coordinate position information of the pixels acquired or obtained from the determination unit 160 .

[0202] That is, the display control unit 170 instructs the display unit 520 to output a different display according to the presence or absence of a nearby object determined based on the output information received from the distance information acquisition unit 13 and the output information of the imaging unit 11 .

[0203] According to this configuration, by distinguishing the nearby object from the influence of external light, the user can correctly confirm whether the user himself and the nearby object (for example, a tripod) are included in the captured image.

[0204] As described above, the display control unit 170 instructs the display unit 520 to set the display position of the nearby object according to the position of the nearby object in space, which means that the display position of the nearby object becomes different according to the position of each nearby object in space. According to this configuration, the user can confirm the position of each nearby object included in the captured image.

[0205] The display control unit 170 instructs the display unit 520 to display the image information G captured by the camera unit 11, and also instructs the display unit 520 to display identification information for identifying nearby objects by superimposing the identification information on the image information. According to this configuration, the user can correctly confirm the position of each nearby object included in the captured image.

[0206] Fig.11 1 is an example of the appearance of an image pickup device 1 according to a second modification example of the present embodiment. Fig.12 FIG. 1 is an example of a functional block diagram of a processing circuit according to a second variation of the present embodiment.

[0207] exist Fig.11 In the second modification shown, the camera device 1 has, for example, a plurality of display units 20a, 20A instead of Figure 1 The display unit 20 shown in FIG. Each of the display units 20 a and 20A can be formed of, for example, a light emitting diode (LED) and flashes based on an output signal of the processing circuit 14 .

[0208] The display unit 20 a is provided on a first face of the housing 10 (eg, the front side of the housing 10 ), and the display unit 20 a is provided on a second face of the housing 10 (eg, the rear side of the housing 10 ).

[0209] exist Fig.12 In the second modification shown, the display control unit 170 instructs the display units 20a, 20A to display information or signals indicating the determination results acquired or obtained from the determination unit 160. For example, if an object approaches on each side of the camera 1, the display units 20a and 20b may flash red.

[0210] Furthermore, the transmission-reception unit 180 transmits or outputs the two-dimensional image information of the panoramic spherical image output from the RGB image data acquisition unit 142 to the display device 500 via the network 400. The display device 500 is an example of an output destination of the two-dimensional image information.

[0211] That is, regarding the second modification, Figure 5 In step S3 of , the transmitting-receiving unit 180 acquires the RGB image data of the panoramic spherical image from the RGB image data acquiring unit 142 , and then transmits or outputs the two-dimensional image information based on the acquired RGB image data of the panoramic spherical image to the display device 500 .

[0212] The transmission-reception unit 510 of the display device 500 receives the two-dimensional image information transmitted from the transmission-reception unit 180 of the camera 1 .

[0213] The control unit 530 of the display device 500 instructs the display unit 520 to display the two-dimensional image information received by the transmitting-receiving unit 510 .

[0214] As described above, the imaging device 1 includes the imaging unit 11 and the transmission-reception unit 180 that outputs two-dimensional image information captured by the imaging unit 11 separately from three-dimensional information.

[0215] According to this configuration, without confirming three-dimensional information, the user can easily confirm from two-dimensional image information whether the user himself, a tripod, etc. are included in the captured image and whether three-dimensional information having a desired layout has been acquired.

[0216] Therefore, the user can reacquire the three-dimensional information while staying at the place where the three-dimensional information is acquired or obtained, which can reduce the work or labor of revisiting the place where the three-dimensional information is acquired or obtained, compared to a situation where the user recognizes that the captured image includes the user himself, a tripod, etc., or that the three-dimensional information with the desired layout is not acquired or obtained after leaving the place where the three-dimensional information is acquired.

[0217] Before transmitting or outputting the three-dimensional information in step S11, the transmitting-receiving unit 180 transmits or outputs the two-dimensional image information G in step S3. Also, before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10, the transmitting-receiving unit 180 outputs the two-dimensional image information G in step S3.

[0218] According to this configuration, before confirming three-dimensional information, the user can confirm from the two-dimensional image information whether the user himself, the tripod, etc. are included in the captured image and whether three-dimensional information having a desired layout has been acquired.

[0219] The transmitting-receiving unit 180 transmits the two-dimensional image information to the display device 500, and the display device 500 instructs the display unit 520 to display the two-dimensional image information.

[0220] According to this configuration, the user can easily confirm whether the user himself, the tripod, etc. are included in the captured image from the two-dimensional image information displayed by the display unit 520, and whether three-dimensional information having a desired layout has been acquired.

[0221] The transmitting-receiving unit 180 transmits the two-dimensional image information to the display device 500 which is different from the external device 300 which outputs the three-dimensional information.

[0222] According to this configuration, the user can confirm whether the captured image includes the user himself, a tripod, etc., and whether three-dimensional information with a desired layout has been obtained from the two-dimensional image information output to the display unit 520 different from the external device 300, without confirming the three-dimensional information output to the external device 300.

[0223] In addition, the transmitting-receiving unit 180 may be configured to transmit the three-dimensional information to the display device 500. Specifically, the transmitting-receiving unit 180 may select a case where the transmitting-receiving unit 180 transmits the two-dimensional image information to the display device 500 and another case where the transmitting-receiving unit 180 transmits the three-dimensional information to the display device 500 according to a preset condition. According to this configuration, the transmitting-receiving unit 180 may transmit the two-dimensional image information to the display device 500 separately from the three-dimensional information.

[0224] Fig.13 FIG. 1 is an example of a flowchart for determining a nearby object according to the second variation of the present embodiment.

[0225] Fig.13 FIG. 1 is a flowchart of a process for determining whether a nearby object is included in a captured image according to a second modification example. The process corresponds to the process of referring to FIG. Figure 5 Step S9 is described.

[0226] The determination unit 160 determines whether the TOF image data of the panoramic spherical image has one or more pixels with saturated charge amounts, based on the TOF image data of the panoramic spherical image acquired or obtained from the re-projection processing unit 147 (step S811 ).

[0227] If the determination unit 160 determines in step S811 that the TOF image data of the panoramic spherical image has one or more pixels with saturated charge amounts (S811: "Yes"), based on the RGB image data of the panoramic spherical image acquired or obtained from the RGB image data acquisition unit 142, the determination unit 160 determines whether the RGB image data of the panoramic spherical image has one or more pixels with the same coordinates as the pixels with saturated charge amounts in step S811, and the pixels have saturated charge amounts (step S812).

[0228] If the determination unit 160 determines in step S812 that the RGB image data of the panoramic spherical image has more than one pixel with saturated charge amount (S812: "Yes"), the determination unit 160 determines in step S811 that the pixel with saturated charge amount is caused by external light, and outputs error information to the display control unit 170.

[0229] Then, based on the error information acquired or obtained from the determination unit 160, the display control unit 170 superimposes the error information on the two-dimensional image information, and then instructs the display unit 20a or the display unit 20A to display the error information (step S813).

[0230] If the determination unit 160 determines in step S812 that the RGB image data of the panoramic spherical image does not have pixels with saturated charge amounts (S812: "No"), the determination unit 160 determines in step S801 that the pixels with saturated charge amounts are caused by nearby objects, and in step S811 outputs the coordinate position information of the pixels with saturated charge amounts to the display control unit 170.

[0231] Then, the display control unit 170 determines whether the coordinate position information indicates the front side of the casing 10 based on the coordinate position information of the pixel acquired or obtained from the determination unit 160 (step S814 ).

[0232] If the determination unit 160 determines in step S811 that the TOF image data of the panoramic spherical image does not have pixels with saturated charge amount (S811: "No"), based on the TOF image data of the panoramic spherical image acquired or obtained from the re-projection processing unit 147, the determination unit 160 determines whether the TOF image data of the panoramic spherical image has one or more pixels representing specified distance information (for example, less than 0.5 m) in the TOF image data of the panoramic spherical image (step S815).

[0233] If the determination unit 160 determines in step S815 that the TOF image data of the panoramic spherical image does not have a pixel indicating predetermined distance information (eg, less than 0.5 m) (S815: No), the determination unit 160 ends the sequence.

[0234] If the determination unit 160 determines in step S815 that the TOF image data of the panoramic spherical image has one or more pixels indicating prescribed distance information (eg, less than 0.5 m) (S815: Yes), the determination unit 160 continues the sequence to step S814.

[0235] If step S815 is “yes”, the determination unit 160 determines that the pixel representing the distance information less than 0.5 m is caused by the presence of a nearby object, and outputs the coordinate position information of the pixel representing the distance information less than 0.5 m to the display control unit 170 in step S815.

[0236] Then, in step S814 , the display control unit 170 determines whether the coordinate position information of the pixel indicates the front side of the casing 10 based on the coordinate position information of the pixel acquired or obtained from the determination unit 160 .

[0237] If the display control unit 170 determines in step S814 that the coordinate position information of the pixel indicates the front side (S814: YES), the display control unit 170 instructs the display unit 20a arranged on the front side of the casing 10 to blink light (step S816).

[0238] If the display control unit 170 determines in step S814 that the coordinate position information of the pixel does not indicate the front side (S814: No), the display control unit 170 instructs the display unit 20A arranged on the rear side of the casing 10 to blink light (step S817).

[0239] As described above, when the display control unit 170 determines that there is a nearby object, the display control unit 170 instructs the display unit 20a or the display unit 20A to flash the light, and when the display control unit 170 determines that there is no nearby object, it does not instruct the display unit 20a or the display unit 20A to flash the light.

[0240] That is, the display control unit 170 instructs the display unit 20 a or the display unit 20A to output a different display depending on the presence or absence of a nearby object.

[0241] According to this configuration, by distinguishing a nearby object from the influence of external light, the user can correctly confirm whether the user himself or a nearby object (for example, a tripod) is included in the captured image.

[0242] Furthermore, the display control unit 170 instructs the display unit 20 a or the display unit 20A to blink light based on the coordinate position information of the pixel acquired or obtained from the determination unit 160 .

[0243] That is, the display control unit 170 instructs the display unit 20a or the display unit 20A to display the positions of the nearby objects according to the positions of the nearby objects in space, which means that the positions of the nearby objects are displayed differently according to the positions of each nearby object in space. According to this configuration, the user can confirm the position of each nearby object included in the captured image.

[0244] Then, the display control unit 170 instructs the display unit 20a, 20A whichever is closer to the nearby object to output a different display depending on the presence or absence of the nearby object. According to this configuration, the user can accurately confirm the position of a specific object included in the captured image.

[0245] Fig.14 FIG. 2 is a cross-sectional view showing the configuration of an imaging device 1 according to a third modified example of the present embodiment.

[0246] exist Fig.14 In the third variant shown, in addition to Figure 2 In addition to the illustrated configuration, the imaging device 1 includes another imaging unit 111 including, for example, another imaging element 111 a , 111 a and another fisheye lens (wide-angle lens) 111 b , 111 b .

[0247] In the third modified example, the RGB imaging unit 11 and the other imaging unit 111 are arranged on the same baseline. In this configuration, the processing circuit 14 is capable of performing processing on a configuration having multiple eyes. That is, RGB images of two viewpoints can be acquired or obtained by simultaneously driving the imaging unit 11 and the other imaging unit 111, wherein a prescribed preset distance is provided between the imaging unit 11 and the other imaging unit 111 along one surface. Therefore, the parallax calculated from the two RGB images can be used, and the distance accuracy of the entire measurement range can be improved.

[0248] Specifically, when the imaging unit 11 and another imaging unit 111 for capturing RGB image data are provided, multi-baseline stereo (MSB) and epipolar plane image (EPI) processing using the sum of squared differences (SSD) can be used as in conventional parallax calculation. Therefore, by using the imaging unit 11 and another imaging unit 111, the reliability of parallax can be increased, and high spatial resolution and high precision can be achieved.

[0249] As described above, the imaging device 1 includes another imaging unit 111 , and the 3D reconstruction processing unit 150 that determines 3D information based on the output information received from the distance information acquisition unit 13 , the 2D image information, and another 2D image information photographed by the another imaging unit 111 .

[0250] Furthermore, the imaging device 1 may include another imaging unit 111 and a three-dimensional information determination unit that determines three-dimensional information based on two-dimensional image information and another two-dimensional image information captured by the another imaging unit 111 , without using output information from the distance information acquisition unit 13 .

[0251] According to this configuration, the user can confirm whether the captured image includes the user himself, a tripod, etc., and whether the two-dimensional image information captured from the camera unit 11 has obtained three-dimensional information with the desired layout, without confirming the three-dimensional information determined by the three-dimensional reconstruction processing unit 150 based on the two-dimensional image information.

[0252] Regarding the above-described embodiment, the imaging device 1 (an example of an information processing device) includes an imaging unit 11 that captures two-dimensional image information, an output unit that outputs the two-dimensional image information G for display, and three-dimensional information associated with coordinates of the two-dimensional image information.

[0253] The output unit may be implemented as the display control unit 170. In this case, the display control unit 170 outputs two-dimensional image information and three-dimensional information for display.

[0254] In addition, the output unit may be implemented as the transmission-reception unit 180. In this case, the transmission-reception unit 180 outputs two-dimensional image information for display to the external device, and outputs three-dimensional information to the external device.

[0255] In addition, the output unit may be implemented as a display control unit 170 and a transmission-reception unit 180. In this case, the display control unit 170 outputs two-dimensional image information for display, and the transmission-reception unit 180 outputs three-dimensional information to an external device. Alternatively, the display control unit 170 outputs three-dimensional information to an external device, and the transmission-reception unit 180 outputs two-dimensional image information for display.

[0256] According to this configuration, without confirming three-dimensional information, the user can easily confirm from two-dimensional image information whether the user himself, a tripod, etc. are included in the captured image and whether three-dimensional information having a desired layout has been acquired.

[0257] Therefore, the user can reacquire the three-dimensional information while staying at the place where the three-dimensional information is acquired or obtained, which can reduce the work or labor of revisiting the place where the three-dimensional information is acquired or obtained, compared to a situation where the user recognizes that the captured image includes the user himself, a tripod, etc., or that the three-dimensional information with the desired layout is not acquired or obtained after leaving the place where the three-dimensional information is acquired.

[0258] The display control unit 170 or the transmission-reception unit 180 outputs the two-dimensional image information G before outputting the three-dimensional information. In addition, the display control unit 170 or the transmission-reception unit 180 outputs the two-dimensional image information G before determining the three-dimensional information.

[0259] According to this configuration, before confirming three-dimensional information, the user can confirm from the two-dimensional image information whether the user himself, the tripod, etc. are included in the captured image and whether three-dimensional information having a desired layout has been acquired.

[0260] The display control unit 170 instructs the display unit 20 and the display unit 520 to display the two-dimensional image information G. The imaging device 1 includes the display unit 20 .

[0261] According to this configuration, the user can easily confirm whether the user himself, the tripod, etc. are included in the captured image from the two-dimensional image information displayed by the display unit, and whether three-dimensional information having a desired layout has been acquired.

[0262] The display control unit 170 or the transmission / reception unit 180 outputs the two-dimensional image information G to the display unit 20 and the display unit 520 (an example of an output destination) which are different from the external device 300 (an example of an output destination) that outputs the three-dimensional information.

[0263] According to this configuration, the user can confirm whether the captured image includes the user himself, a tripod, etc., and whether three-dimensional information having a desired layout has been acquired from the two-dimensional image information output to the display unit 20 and the display unit 520 different from the external device 300, without confirming the three-dimensional information output to the external device 300 (an example of an output destination).

[0264] The imaging device 1 includes a projection unit 12 for projecting light onto a target object, a distance information acquisition unit 13 (an example of a light receiving unit) for receiving light reflected from the target object, and a three-dimensional reconstruction processing unit 150 (an example of a three-dimensional information determination unit) for determining three-dimensional information based on output information received from the distance information acquisition unit 13. The three-dimensional reconstruction processing unit 150 determines three-dimensional information based on the output information received from the distance information acquisition unit 13 and two-dimensional image information.

[0265] According to this configuration, the user can confirm whether the captured image includes the user himself, a tripod, etc., and whether the two-dimensional image information captured from the camera unit 11 has acquired three-dimensional information with the desired layout, without confirming the three-dimensional information determined by the three-dimensional reconstruction processing unit 150.

[0266] The imaging apparatus 1 includes another imaging unit 111 , and a three-dimensional reconstruction processing unit 150 that determines three-dimensional information based on output information received from the distance information acquisition unit 13 , two-dimensional image information, and another two-dimensional image information photographed by the another imaging unit 111 .

[0267] As described above, the imaging device 1 may include another imaging unit 111 and a three-dimensional information determination unit that determines three-dimensional information based on two-dimensional image information and another two-dimensional image information captured by the another imaging unit 111 , without using output information from the distance information acquisition unit 13 .

[0268] According to this configuration, the user can confirm whether the captured image includes the user himself, a tripod, etc., and whether the two-dimensional image information captured from the camera unit 11 has obtained three-dimensional information with the desired layout, without confirming the three-dimensional information determined by the three-dimensional reconstruction processing unit 150 based on the two-dimensional image information.

[0269] The three-dimensional information includes, for example, panoramic spherical three-dimensional information. In this case, even if it is difficult for the user to confirm whether the user, a tripod, etc. are included in the captured image, and whether the three-dimensional information with a desired layout has been acquired in the panoramic spherical three-dimensional information, the user can easily confirm whether the user, a tripod, etc. are included in the captured image, and whether the three-dimensional information with a desired layout has been acquired from the two-dimensional image information captured by the camera unit 11.

[0270] Regarding the above-mentioned embodiment, the camera device 1 (an example of an information processing device) includes a camera unit 11 for capturing an image of a target object, a projection unit 12 for projecting light onto the target object, a distance information acquisition unit 13 (an example of a light receiving unit) for receiving light reflected from the target object, and a display control unit 170 for instructing the display unit 20 and the display unit 520 to output different displays according to the presence or absence of a specific object determined based on the output information of the distance information acquisition unit 13 and the output information of the camera unit 11.

[0271] According to this configuration, the user can correctly confirm whether a specific object, such as the user himself or herself or a nearby object (for example, a tripod), is included in a captured image by distinguishing the specific object from the influence of external light.

[0272] The image pickup apparatus 1 includes the display unit 20. According to this configuration, the user can correctly confirm whether a specific object is included in a captured image.

[0273] That is, the display control unit 170 instructs the display unit 20 and the display unit 520 to set the display position of the specific object according to the position of the specific object in the space, which means that the display position of each specific object becomes different according to the position of each specific object in the space. According to this configuration, the user can correctly confirm the position of the specific object included in the captured image.

[0274] The display unit 20 includes a plurality of display units 20A, 20a. That is, the display control unit 170 instructs the display unit 20A or the display unit 20a close to the nearby object to output different displays according to the presence or absence of a specific object. According to this configuration, the user can correctly confirm the position of the specific object included in the captured image.

[0275] The display control unit 170 instructs the display unit 20 and the display unit 520 to display the image information G photographed by the camera unit 11, and instructs the display unit 20 and the display unit 520 to display the identification information G1 and G2 for identifying the specific object by superimposing the identification information G1 and G2 on the image information G. According to this configuration, the user can correctly confirm the position of the specific object included in the photographed image.

[0276] The imaging device 1 includes a determination unit 160 that determines that a specific object exists when the charge amount due to the light received by the distance information acquisition unit 13 is saturated and the charge amount of the pixel in the imaging unit 11 is not saturated.

[0277] According to this configuration, by distinguishing the specific object from the influence of external light, the user can correctly confirm whether the specific object is included in the captured image.

[0278] In addition, each function of the above-described embodiments can be implemented by one or more processing circuits or circuits. The processing circuit includes a programmed processor, and the processor includes a circuit system. The processing circuit also includes an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), and conventional circuit elements arranged to perform the functions. The program for executing the above-described embodiments can be stored in a carrier device or a storage medium.

[0279] This patent application is based upon and claims the benefit of priority from Japanese Patent Application No. 2020-050539 filed on March 23, 2020, and Japanese Patent Application No. 2020-050608 filed on March 23, 2020, the disclosures of which are incorporated herein in their entirety by reference.

[0280] Reference numerals list

[0281] 1. Imaging Device (Example / Embodiment of Information Processing Device)

[0282] 10 Housing

[0283] 11 Camera unit

[0284] 11a, 11A imaging element

[0285] 11b, 11B fisheye lens

[0286] 12 projection units

[0287] 12a, 12A light source unit

[0288] 12b, 12B wide angle lens

[0289] 13Distance information acquisition unit (example of light receiving unit)

[0290] 13a, 13A TOF sensor

[0291] 13b, 13B wide angle lens

[0292] 14 Processing circuit

[0293] 15 Camera switch

[0294] 20 Display unit

[0295] 20A, 20a display unit

[0296] 111 Another camera unit

[0297] 150 Three-dimensional reconstruction processing unit (an example of a three-dimensional information determination unit)

[0298] 160 judgment unit

[0299] 170 Display control unit (example of output unit)

[0300] 180 Transmit-Receive Unit (Example of Output Unit)

[0301] 300 External device (Example of output destination)

[0302] 500 Display device (Example of output destination)

[0303] 520 Display unit (Example of output destination)

[0304] L Sync signal line

Claims

1. An information processing device (1) for outputting three-dimensional information determined based on received light, the information processing device (1) comprising: A camera unit (11) is configured to capture two-dimensional image information. a determination unit (160) configured to determine whether a specific object exists based on an output of a light receiving unit (13) that projects light from a projection unit (12) to an object and receives the light reflected from the object, and an output from the imaging unit, and an output unit (170, 180) configured to output the two-dimensional image information for display and the three-dimensional information associated with the coordinates of the two-dimensional image information; The output unit includes a display control unit (170) configured to instruct the display unit (20, 520, 20A, 20a) to display differently according to the presence or absence of the specific object; The determination unit (160) is configured to determine that the specific object exists when the charge amount caused by the light received by the light receiving unit (13) is saturated and the charge amount of the pixel of the imaging unit (11) is not saturated.

2. The information processing device (1) according to claim 1, in, The output unit (170, 180) outputs the two-dimensional image information before outputting the three-dimensional information.

3. The information processing device (1) according to claim 2, in, The output unit (170, 180) outputs the two-dimensional image information before determining the three-dimensional information.

4. The information processing device (1) according to claim 1, in, The output unit (170, 180) instructs the display unit (20, 520) to display the two-dimensional image information.

5. The information processing device (1) according to claim 1, in, The output unit (170, 180) is configured to output the two-dimensional image to an output destination different from another output destination to which the three-dimensional information is output.

6. The information processing device (1) according to claim 1, further comprising: A projection unit (12) configured to project light into space; a light receiving unit (13) configured to receive light reflected from one or more objects existing in the space, and A three-dimensional information determination unit (150) is configured to determine the three-dimensional information based on the output from the light receiving unit (13).

7. The information processing device (1) according to claim 6, in, The three-dimensional information determination unit (150) determines the three-dimensional information based on the output of the light receiving unit (13) and the two-dimensional image information.

8. The information processing device (1) according to claim 7, further comprising another camera unit (111): in, The three-dimensional information determination unit (150) determines the three-dimensional information based on the output of the light receiving unit (13), the two-dimensional image information, and another two-dimensional image information captured by the another imaging unit (111).

9. The information processing device (1) according to claim 1, further comprising: another camera unit (111), and A three-dimensional information determination unit (150) is configured to determine the three-dimensional information based on the two-dimensional image information and another two-dimensional image information photographed by the another imaging unit (111).

10. The information processing device (1) according to claim 1, in, The three-dimensional information includes panoramic spherical three-dimensional information.

11. The information processing device (1) according to any one of claims 1 to 10, wherein: The projection unit (12) is configured to project the light into space; The light receiving unit (13) is configured to receive light reflected from one or more objects existing in the space.

12. The information processing device (1) according to claim 11, in, The display control unit (170) instructs the display unit (20, 520) to set the display position of the specific object according to the position of the specific object in space.

13. The information processing device (1) according to claim 11, in, The display control unit (170) instructs the display unit (20, 520) to display the image information captured by the camera unit (11), and instructs the display unit (20, 520) to display the identification information identifying the specific object on the image information by superimposing the identification information on the image information.

14. An information processing method, comprising: The step of capturing two-dimensional image information by means of a camera unit (11); A step of determining the presence or absence of a specific object by a determination unit (160) based on output information of a light receiving unit (13) that projects light from a projection unit (12) to an object and receives light reflected from the object, and output information of the photographing step; The step of outputting the two-dimensional image information for display via an output unit (170, 180), and The step of outputting three-dimensional information determined based on the light received by the light receiving unit (13) and associated with the coordinates of the two-dimensional image information through an output unit (170, 180) includes the step of instructing a display unit (20, 520, 20A, 20a) to output different displays based on the presence or absence of the specific object through a display control unit (170) included in the output unit; The step of determining the presence or absence of a specific object by the determination unit (160) includes determining that the specific object exists when the charge amount caused by the light received by the light receiving unit (13) is saturated and the charge amount of the pixel of the imaging unit (11) is not saturated.

15. The information processing method according to claim 14, wherein the determining step comprises: A step of projecting light into space by a projection unit (12); A step of receiving light reflected from one or more objects existing in the space by a light receiving unit (13).

Images