Information processing system and imaging device

Abstract

The present invention provides an information processing system and an imaging device that can obtain continuous position and orientation information even when the imaging device is moved to a dark place. [Solution] The imaging device comprises a first light source installed in the housing of the imaging device and irradiating the space with light in the first wavelength region, a first light receiving unit installed in the housing and receiving light in the first wavelength region, a bottom located at the longitudinal end of the housing, and an estimation unit that estimates the position and orientation of the imaging device from the video data from the first light receiving unit. The first light source is installed at the same height as the first light receiving unit, or higher than the first light receiving unit, with the bottom as the reference point in the longitudinal direction, and the first light receiving unit acquires video data while the first light source is irradiating with light in the first wavelength region.

Description

Technical Field

[0001] The present invention relates to an information processing system and an imaging device.

Background Art

[0002] Conventionally, as one method of distance measurement from an imaging device to an object, a distance measurement system called the TOF (Time of Flight) method, which calculates the distance to the object based on the time from when light is irradiated until the reflected light is received, is known. Such a TOF distance measurement system is used, for example, to acquire spatial information within a structure. For this purpose, in a TOF distance measurement system, not only distance information of the surrounding space but also an RGB camera for acquiring color information is separately required. Furthermore, a light source that irradiates light in the visible light region is required to acquire color information in a dark place with this RGB camera. In addition, in order to integrate (registration) distance information captured at multiple locations into one, information on the position and orientation of the imaging device itself at each time of imaging is required, and various derivative technologies based on SLAM (Simultaneous Localization and Mapping) exist as means for this. Among such SLAM, VSLAM (Visual SLAM) is known as a method that estimates its own position by optimizing the position and orientation of the imaging device and the three-dimensional structure of the environment so that geometric constraints on the object position relationship in a plurality of frame images of a moving image are satisfied using a classical camera image.

[0003] As a technique related to such VSLAM, map information is generated based on a plurality of images captured by an imaging device along a path between a predetermined start position and an end position of the imaging device, and based on the virtual distance between a start point corresponding to the start position on the map information and an end point corresponding to the end position, and the actually measured distance given in advance for the pair of the start position and the end position of the imaging device, a technique for generating map information with corrected scale is disclosed (for example, Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technology described in Patent Document 1 has a problem in that it becomes difficult to obtain continuous position and orientation information when the imaging device is moved to a dark place such as an attic.

[0005] The present invention has been made in view of the above, and aims to provide an information processing system and an imaging device that can obtain continuous position and orientation information even when the imaging device is moved to a dark place. [Means for solving the problem]

[0006] To solve the above-mentioned problems and achieve the objective, the present invention comprises: a first light source installed in the housing of an imaging device and irradiating space with light in a first wavelength region; a first light receiving unit installed in the housing and receiving light in the first wavelength region; a bottom portion located at the longitudinal end of the housing; and an estimation unit that estimates the position and orientation of the imaging device from video data obtained by the first light receiving unit, wherein the first light source is installed at the same height as the first light receiving unit, or higher than the first light receiving unit, with the bottom portion as the reference in the longitudinal direction, and the first light receiving unit acquires the video data while the first light source is irradiating with light in the first wavelength region. [Effects of the Invention]

[0007] According to the present invention, continuous position and orientation information can be obtained even when the imaging device is moved to a dark place. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the configuration of an information processing system according to the first embodiment. [Figure 2] Figure 2 shows an example of a support configuration for the imaging device according to the first embodiment. [Figure 3] Figure 3 illustrates the LED light projection angle and RGB light reception angle of the imaging device according to the first embodiment. [Figure 4] Figure 4 is a diagram illustrating the general operation of the comparative distance measuring system. [Figure 5] Figure 5 shows an example of the hardware configuration of the information processing device according to the first embodiment. [Figure 6] Figure 6 shows an example of the configuration of a functional block in the information processing device according to the first embodiment. [Figure 7] Figure 7 is a diagram illustrating the operation of the information processing system according to the first embodiment. [Figure 8] Figure 8 illustrates the LED light projection angle and RGB light reception angle of the imaging device according to the second embodiment. [Figure 9] Figure 9 illustrates the LED light projection angle and RGB light reception angle of the imaging device according to the third embodiment. [Figure 10] Figure 10 shows an example of the configuration of a functional block in the information processing device according to the fifth embodiment. [Figure 11] Figure 11 is a flowchart showing an example of the operation flow of the information processing system according to the fifth embodiment. [Figure 12] Figure 12 shows an example of the configuration of an information processing system according to the sixth embodiment. [Figure 13] Figure 13 shows an example of the configuration of a functional block in the information processing apparatus according to the sixth embodiment. [Figure 14] Figure 14 is a flowchart showing an example of the operation flow of an information processing system according to the sixth embodiment. [Figure 15] Figure 15 shows an example of the configuration of a functional block in the information processing device according to the seventh embodiment. [Figure 16] Figure 16 is a flowchart showing an example of the operation flow of the information processing system according to the seventh embodiment. [Figure 17] Figure 17 illustrates the LED light projection angle and RGB light reception angle of the imaging device according to the eighth embodiment. [Figure 18] Figure 18 shows an example of the configuration of an imaging device according to the ninth embodiment. [Figure 19] FIG. 19 is a diagram showing an example of the configuration of an imaging device according to the tenth embodiment. [Figure 20] FIG. 20 is a diagram for explaining the operation when the imaging device according to the tenth embodiment is placed in a dark place. [Figure 21] FIG. 21 is a diagram showing another example of the configuration of an imaging device according to the tenth embodiment. [Figure 22] FIG. 22 is a diagram showing an example of the schematic configuration of a smartphone or the like according to the twelfth embodiment.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of an information processing system and an imaging device according to the present invention will be described in detail with reference to the drawings. Further, the present invention is not limited by the following embodiments, and the constituent elements in the following embodiments include those that can be easily conceived by those skilled in the art, substantially the same ones, and those within the so-called equivalent range. Furthermore, various omissions, substitutions, changes, and combinations of constituent elements can be made without departing from the gist of the following embodiments.

[0010] [First Embodiment] (Configuration of Information Processing System) FIG. 1 is a diagram showing an example of the outline of the configuration of an information processing system according to the first embodiment. FIG. 2 is a diagram showing an example of the support mode of an imaging device according to the first embodiment. FIG. 3 is a diagram for explaining the LED light projection angle and the RGB light reception angle of the imaging device according to the first embodiment. The outline of the configuration of the information processing system 1 according to the present embodiment will be described with reference to FIGS. 1 to 3.

[0011] The information processing system 1 shown in FIG. 1 is a TOF distance measurement system having a self-position estimation function by VSLAM. As shown in FIG. 1, the information processing system 1 includes an imaging device 10 and an information processing device 20. This information processing system 1 is mainly used to acquire spatial shape information of the surrounding environment and objects / humans.

[0012] The imaging device 10 is a device for receiving infrared light to measure distance by the TOF method and capturing video data by an RGB camera to estimate the position and orientation of the imaging device 10 based on VSLAM. As shown in FIG. 1, the imaging device 10 includes an infrared light source 11 (second light source), an infrared light receiving unit 12 (second light receiving unit), a visible light source 13 (first light source), an RGB camera 14 (first light receiving unit), and a housing 50.

[0013] The infrared light source 11 is a light source for irradiating infrared light in the infrared region (an example of the second wavelength region) to the surroundings. The infrared light source 11 includes, for example, an infrared light source substrate on which a light source for irradiating infrared light is mounted, and an optical system such as a lens, a deflection element, or a diffraction element. As shown in FIG. 1, a plurality of infrared light sources 11 are arranged on the side surface of the housing 50. The infrared light receiving unit 12 is a unit that receives the reflected light reflected by surrounding objects among the infrared light irradiated from the infrared light source 11. As shown in FIG. 1, a plurality of infrared light receiving units 12 are arranged on the upper side of the housing 50. In the present embodiment, a plurality of infrared light receiving units 12 are provided on the side surface of the housing 50 and one infrared light receiving unit 12 is provided on the upper surface of the housing 50. The infrared light source 11 and the infrared light receiving unit 12 function as a TOF camera for calculating the distance based on the time from irradiating infrared light to receiving the reflected light.

[0014] In addition, since the angle of the infrared light irradiated by the infrared light source 11 (light projection angle) and the angle of the infrared light received by the infrared light receiving unit 12 (light receiving angle) are both wide-angle angles, the distance measurement range can be the entire spherical range centered on the imaging device 10. Note that the distance measurement range does not necessarily need to cover the entire sphere, and it may be such that irradiation or reception is not performed for some regions.

[0015] As described above, the imaging device 10 according to this embodiment is equipped with an infrared light source 11 and an infrared light receiving unit 12 that have a distance measurement range covering the entire sphere. Therefore, distance information can be obtained in the entire sphere without changing the orientation of the imaging device 10, eliminating the need to change the orientation of the imaging device 10 itself during measurement, and enabling imaging of the space with minimal effort. Furthermore, since it is equipped with a solid-state infrared light source 11 and infrared light receiving unit 12 rather than a method that mechanically changes the direction of infrared light irradiation, the generation of vibrations caused by the operation of movable parts is suppressed, and stable measurements can be performed even when the imaging device 10 is fixed to the end of a long-extended tripod or monopod. For this reason, the imaging device 10 according to this embodiment is extremely useful, for example, when performing distance measurement by inserting the imaging device into a dark space such as an attic space with a maintenance hatch.

[0016] Multiple visible light sources 13 are arranged on the lower side of the housing 50 (on the side of the support parts 19 and 19a described later) than the infrared light source 11 and the infrared light receiving unit 12, and are light sources such as LEDs (Light Emitting Diodes) that emit light (visible light) in the visible light region (an example of the first wavelength region).

[0017] Multiple RGB cameras 14 are positioned on the side of the housing 50 and are devices that capture (receive) visible light from the surroundings. The RGB cameras 14 output video data composed of information such as R (red), G (green), and B (blue) from the captured visible light. The video data captured by the RGB cameras 14 is used to estimate the position and orientation information of the imaging device 10 by VSLAM, as will be described later. The RGB cameras 14 are positioned on the side of the housing 50 at approximately the same height as the visible light source 13. That is, the RGB cameras 14 are positioned at approximately the same position as the visible light source 13 in the longitudinal direction of the housing 50 (the direction from the support parts 19 and 19a, described later, toward the side where the infrared light receiving unit 12 is located). As described above, because the surroundings are illuminated by the visible light source 13, the RGB cameras 14 can acquire video data even in dark places.

[0018] Furthermore, the RGB camera 14 is not limited to being positioned at approximately the same location as the visible light source 13 in the longitudinal direction of the housing 50, but may be positioned closer to the support parts 19, 19a than the visible light source 13. The support parts 19, 19a are the bottom of the housing 50, that is, the part at one end in the longitudinal direction of the housing 50. In other words, the visible light source 13 may be installed at the same height as the RGB camera 14, or higher than the RGB camera 14, when the support parts 19, 19a (bottom) are used as a reference in the longitudinal direction of the housing 50.

[0019] Furthermore, the visible light source 13 and RGB camera 14 shown in Figure 3(a) are arranged alternately on the four sides of the housing 50, as shown in Figure 3(b). As a result, one of the two visible light sources 13 is located on the opposite side of the housing 50 from the side on which the other is located. Similarly, one of the two RGB cameras 14 is located on the opposite side of the housing 50 from the side on which the other is located. In addition, the angle of visible light emitted by the visible light source 13 (LED projection angle) is slightly greater than 180 degrees, as shown in Figure 3(b), thus covering the entire sphere. Note that this illumination range does not necessarily have to cover the entire sphere; some areas may not be illuminated. Also, the angle of visible light received by the RGB camera 14 (RGB reception angle) is slightly greater than 180 degrees, as shown in Figure 3(b), thus covering the entire sphere. Furthermore, the light-receiving range does not necessarily need to cover the entire sphere, and some areas may not receive light. Also, as shown in Figure 3(b), the visible light source 13 is positioned so as not to be within the RGB light-receiving field of view of the RGB camera 14, thereby avoiding adverse effects such as flare and ghosting caused by unwanted visible light reflection from the visible light source 13 on the video data acquired by the RGB camera 14.

[0020] The housing 50 forms the main body of the imaging device 10 and is a rectangular parallelepiped housing, as shown in Figure 3(b), for example. As described above, the infrared light source 11, infrared light receiving unit 12, visible light source 13, and RGB camera 14 are arranged on the side of the housing 50.

[0021] Now, with reference to Figure 2, the support configuration of the imaging device 10 will be described.

[0022] Figure 2 shows an example of how the imaging device 10 is supported. As shown in Figure 2(a), the imaging device 10 may be provided with a support part 19 for supporting the housing 50, which has screw holes or the like formed at the bottom of the housing 50 for attaching a tripod or other device 30. In this case, the imaging device 10 is supported by attaching a tripod or other device 30 to the support part 19 provided on the housing 50. In Figure 2(a), a tripod is shown as the device 30, but a monopod or the like may be used instead.

[0023] Furthermore, as shown in Figure 2(b), in the imaging device 10, the support portion 19a, which is a part of the housing 50 on one longitudinal end for supporting the housing 50 by being grasped by the user's hand, may be positioned below the visible light source 13 and the RGB camera 14. In this case, the imaging device 10 is supported by the user grasping the support portion 19a side of the housing 50 with their hand.

[0024] The information processing device 20 is a PC (Personal Computer) or tablet terminal, etc., that performs information processing based on information received from the imaging device 10. The information processing device 20 communicates with the imaging device 10 via wired or wireless data communication. The information processing device 20 calculates the distance to surrounding objects from the light received data based on the TOF method by the infrared light source 11 and infrared light receiving unit 12 of the imaging device 10 and generates distance information. The information processing device 20 also receives video data captured by the RGB camera 14, estimates the position and orientation of the imaging device 10 based on VSLAM from the video data, and generates position and orientation information.

[0025] (Operational overview of the comparative distance measuring system) Figure 4 is a diagram illustrating the general operation of the comparative distance measuring system. The general operation of the comparative distance measuring system will be explained with reference to Figure 4.

[0026] As shown in Figure 4(a), in the comparative example distance measuring system, distance information (1A) is obtained in a wide space from light received by the infrared light source 1011 and infrared light receiving unit 1012 of the imaging device 1000 using the TOF method. Then, after obtaining the distance information (1A), as shown in Figures 4(b) and 4(c), the imaging device 1000 is moved into a dark space partitioned by a perforated member (for example, a ceiling space with a maintenance hatch), and as shown in Figure 4(d), distance information (2A) is obtained by the infrared light source 1011 and infrared light receiving unit 1012 with the entire imaging device 1000 inside the dark space. During this operation, position and orientation information is obtained in parallel from video data (such as video) captured by the RGB camera 1014 through estimation based on VSLAM. Post-processing by the information processing device integrates the aforementioned distance information (1A) and distance information (2A) using the position and orientation information to form continuous spatial information.

[0027] In this case, in the comparative distance measuring system, if the visible light source 1013, such as an LED that emits visible light, is located below the position where the RGB camera 1014 is positioned in the housing 1050 of the imaging device 1000, as shown in Figure 4, then, as shown in Figure 4(c), the visible light source 1013 may be in the space below and the RGB camera 1014 may be in the dark space above. In this case, the video data captured by the RGB camera 1014 will include information about the dark space where the illumination effect of the visible light source 1013 is not obtained, resulting in a temporary blackout. Subsequently, as shown in Figure 4(d), the dark space is illuminated by the visible light source 1013 as it enters the dark space up to the visible light source 1013, and the RGB camera 1014 becomes able to capture images of the dark space where the illumination effect is obtained. However, because the video data includes information about the blackout state, it becomes difficult to obtain continuous position and orientation information. As a result, continuous position and orientation information is included between the state of distance information (1A) and the state of distance information (2A), making it impossible to integrate distance information (1A) and distance information (2A) as continuous spatial information.

[0028] The following describes in detail the configuration and operation of the information processing system 1 according to this embodiment, which allows continuous position and orientation information to be obtained even when the imaging device 10 is moved to a dark place, and thereby integrated as continuous spatial information.

[0029] (Hardware configuration of information processing equipment) Figure 5 shows an example of the hardware configuration of the information processing device according to the first embodiment. The hardware configuration of the information processing device 20 according to this embodiment will be described with reference to Figure 5.

[0030] As shown in Figure 5, the information processing device 20 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an auxiliary storage device 505, a media drive 507, a display 508, a network interface 509, a keyboard 511, a mouse 512, and a DVD (Digital Versatile Disc) drive 514.

[0031] The CPU 501 is an arithmetic unit that controls the operation of the entire information processing unit 20. The ROM 502 is a non-volatile memory device that stores programs such as the IPL (Initial Program Loader) that are first executed by the CPU 501. The RAM 503 is a volatile memory device used as the work area of ​​the CPU 501.

[0032] The auxiliary storage device 505 is a non-volatile storage device that stores various data such as programs. The auxiliary storage device 505 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0033] The media drive 507 is a device that controls the reading or writing of data to or from a recording medium 506, such as flash memory.

[0034] Display 508 is a liquid crystal display (LCD) or an organic EL (electro-luminescence) display, etc., that displays various information such as cursors, menus, windows, characters, or images.

[0035] Network I / F509 is an interface for data communication with external devices (e.g., imaging device 10) using a network. Network I / F509 is, for example, a NIC (Network Interface Card) that enables communication using the TCP (Transmission Control Protocol) / IP (Internet Protocol) protocol. Network I / F509 may also be a communication interface with wireless communication functionality based on standards such as Wi-Fi (registered trademark).

[0036] Keyboard 511 is an example of an input device equipped with multiple keys for inputting characters, numbers, various instructions, etc. Mouse 512 is a type of input device used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc.

[0037] The DVD drive 514 is a device that controls the reading or writing of various types of data to a DVD 513, which is an example of a removable storage medium. The DVD 513 is, for example, a DVD-RW (Digital Versatile Disk Rewritable), a DVD-R (Digital Versatile Disk Recordable), a CD-RW (Compact Disc Rewritable), or a CD-R (Compact Disc Recordable).

[0038] The CPU 501, ROM 502, RAM 503, auxiliary storage device 505, media drive 507, display 508, network interface 509, keyboard 511, mouse 512, and DVD drive 514 are all connected to each other via bus lines 510, such as an address bus and a data bus, enabling them to communicate with one another.

[0039] Note that the hardware configuration of the information processing device 20 shown in Figure 5 is just one example, and it is not necessary to have all of the components, nor is it necessary to have other components.

[0040] (Configuration and operation of functional blocks in an information processing device) Figure 6 shows an example of the configuration of the functional blocks of the information processing device according to the first embodiment. The configuration and operation of the functional blocks of the information processing device 20 according to this embodiment will be described with reference to Figure 6.

[0041] As shown in Figure 6, the information processing device 20 includes a first acquisition unit 201, a second acquisition unit 202, a distance calculation unit 203 (calculation unit), an estimation unit 204, and an integration unit 205.

[0042] The first acquisition unit 201 is a functional unit that acquires video data (such as video) continuously captured by the RGB camera 14 in a space illuminated with visible light by the visible light source 13, via the network I / F 509.

[0043] The second acquisition unit 202 is a functional unit that acquires TOF-type light reception data, which is received by the infrared light receiving unit 12 from the infrared light source 11, via the network I / F 509. For example, when a release switch or the like provided on the imaging device 10 is pressed, or at a predetermined period, the infrared light source 11 emits infrared light and the infrared light receiving unit 12 receives the reflected light, and the first acquisition unit 201 acquires the light reception data for that reception.

[0044] The distance calculation unit 203 is a functional unit that calculates the distance to the group of objects surrounding the imaging device 10 from the light reception data acquired by the second acquisition unit 202 and generates distance information.

[0045] The estimation unit 204 is a functional unit that estimates the position and orientation of the imaging device 10 based on VSLAM from the video data acquired by the first acquisition unit 201, and generates position and orientation information.

[0046] The integration unit 205 is a functional unit that integrates multiple distance information points at discontinuous locations of the imaging device 10 generated by the distance calculation unit 203, using the position and orientation information of the imaging device 10 generated by the estimation unit 204, into continuous spatial information (for example, 3D point cloud data).

[0047] The first acquisition unit 201, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, and the integration unit 205 described above are implemented, for example, by the execution of a program by the CPU 501 shown in Figure 5. At least a portion of the first acquisition unit 201, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, and the integration unit 205 may be implemented by hardware such as an FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), or by a combination of software and hardware.

[0048] It should be noted that the functional units of the information processing device 20 shown in Figure 6 are conceptual representations of their functions and are not limited to this configuration. For example, multiple functional units shown as independent functional units in the information processing device 20 shown in Figure 6 may be configured as a single functional unit. Alternatively, the functions of a single functional unit in the information processing device 20 shown in Figure 6 may be divided into multiple functions and configured as multiple functional units. Furthermore, each functional unit of the information processing device 20 does not need to be configured as a clear software module as a block as shown in Figure 6; the individual functions of each functional unit may be realized as a whole when a program is executed on the information processing device 20.

[0049] (Operation of the information processing system) Figure 7 is a diagram illustrating the operation of the information processing system according to the first embodiment. The operation of the information processing system 1 according to this embodiment will be explained with reference to Figure 7.

[0050] First, as shown in Figure 7(a), in a wide space, the infrared light receiving unit 12 of the imaging device 10 receives reflected infrared light emitted from the infrared light source 11 based on the TOF method, at the timing when a release switch or the like provided on the imaging device 10 is pressed by the user. Next, the second acquisition unit 202 of the information processing device 20 acquires the received light data received by the infrared light receiving unit 12. Then, the distance calculation unit 203 calculates the distance to the group of objects around the imaging device 10 from the received light data acquired by the second acquisition unit 202 and generates distance information (1).

[0051] Then, after the distance information (1) is generated, the user moves the imaging device 10 into a dark space partitioned by a perforated member (for example, a ceiling space with a maintenance hatch) as shown in Figures 7(b) and 7(c).

[0052] Then, as shown in Figure 7(d), with the entire imaging device 10 in a dark space, the infrared light receiving unit 12 of the imaging device 10 receives reflected infrared light emitted from the infrared light source 11 based on the TOF method, at the timing when a release switch or the like provided on the imaging device 10 is pressed by the user. Next, the second acquisition unit 202 of the information processing device 20 acquires the received light data received by the infrared light receiving unit 12. Then, the distance calculation unit 203 of the information processing device 20 calculates the distance of the group of objects around the imaging device 10 from the received light data acquired by the second acquisition unit 202 and generates distance information (2). In this way, multiple distance information (distance information (1) and distance information (2) described above) is generated at discontinuous positions of the imaging device 10, so the data capacity can be significantly reduced compared to the case where it is generated continuously as in a video.

[0053] Furthermore, the RGB camera 14 continuously captures video data based on visible light from the surroundings, throughout the states shown in Figures 7(a) to 7(d). Here, as described above, the RGB camera 14 is positioned at approximately the same location in the longitudinal direction of the housing 50 (the direction from the support parts 19, 19a toward the side where the infrared light receiving unit 12 is located). Therefore, as shown in Figure 7(c), when the RGB camera 14 enters the dark space, the visible light source 13 also enters the dark space at the same time to illuminate the dark space (to irradiate it with visible light), so there is no blackout, and the video data captured by the RGB camera 14 does not contain information of the blackout state, resulting in continuous video data. As a result, the information necessary for the estimation of the position and orientation of the imaging device 10 by VSLAM by the estimation unit 204 is not interrupted, and continuous position and orientation information can be obtained, as described below.

[0054] Next, the first acquisition unit 201 of the information processing device 20 acquires video data continuously captured by the RGB camera 14 via the network I / F 509. Then, the estimation unit 204 of the information processing device 20 estimates the position and orientation of the imaging device 10 based on VSLAM from the video data acquired by the first acquisition unit 201 and generates position and orientation information. Then, the integration unit 205 of the information processing device 20 can integrate multiple distance information (distance information (1) and distance information (2) above) at discontinuous positions of the imaging device 10 generated by the distance calculation unit 203 into continuous spatial information using the continuous position and orientation information of the imaging device 10 generated by the estimation unit 204.

[0055] As described above, in the information processing system 1 according to this embodiment, the visible light source 13 is installed in the housing 50 of the imaging device 10 and irradiates the space with light in the visible light range, the RGB camera 14 is installed in the housing 50 and receives light in the visible light range, the support parts 19 and 19a are located at the ends in the longitudinal direction of the housing 50 and support the housing 50, and the visible light source 13 is installed at the same height as the RGB camera 14, or higher than the RGB camera 14, when the support parts 19 and 19a are considered as the bottom (referenced to the bottom) in the longitudinal direction of the housing 50. This makes it possible to obtain continuous position and orientation information even when the imaging device 10 is moved to a dark place.

[0056] Furthermore, in the information processing system 1 according to this embodiment, the infrared light source 11 is installed in the housing 50 and emits light in the infrared region with wavelengths longer than the visible light region. The infrared light receiving unit 12 receives reflected light from objects among the light emitted from the infrared light source 11. The distance calculation unit 203 calculates the distance to objects around the imaging device 10 from the light received by the infrared light receiving unit 12. The estimation unit 204 estimates the position and orientation of the imaging device 10 from the video data from the RGB camera 14. The integration unit 205 integrates the distance calculated by the distance calculation unit 203 and the position and orientation estimated by the estimation unit 204 into spatial information. As a result, the information necessary for the estimation of the position and orientation of the imaging device 10 by the estimation unit 204 is not interrupted, and continuous spatial information can be obtained.

[0057] Furthermore, in the information processing system 1 according to this embodiment, the height of the infrared light receiving unit 12, when the bottom of the imaging device 10 is used as a reference, is higher than the height of the visible light source 13 and the height of the RGB camera 14. This makes it easier for the infrared light receiving unit 12 to receive reflected light from above the housing 50, enabling both high-precision acquisition of distance information and acquisition of video data in dark space. Also, when the bottom of the imaging device 10 is used as a reference, the visible light source 13 is positioned at a height closer to the height of the RGB camera 14 than the height of the infrared light receiving unit 12, so the visible light source 13 can be provided without narrowing the effective field of view of the infrared light receiving unit 12.

[0058] [Second Embodiment] The information processing system according to the second embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, the LED light projection angle and RGB light reception angle will be described in a configuration that differs from that of the first embodiment. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration and functional block configuration of the information processing device 20, are the same as those described in the first embodiment.

[0059] Figure 8 illustrates the LED light projection angle and RGB light reception angle of the imaging device according to the second embodiment. The LED light projection angle and RGB light reception angle of the imaging device 10a according to this embodiment will be described with reference to Figure 8.

[0060] In the first embodiment, the angle of visible light emitted by the visible light source 13 (LED projection angle) was slightly greater than 180 degrees, thus covering the entire sphere. In this embodiment, as shown in Figure 8, the imaging device 10a is equipped with multiple (four in Figure 8) visible light sources 13a (first light sources) in the housing 50 at positions corresponding to the position where the visible light source 13 was located in the first embodiment. The visible light sources 13a are light sources that diffuse the visible light they emit using a diffusing member or the like. Furthermore, as shown in Figure 8, the visible light sources 13a have an LED projection angle of less than 180 degrees, but as described above, the entire sphere is covered by the multiple visible light sources 13a.

[0061] In the first embodiment, the housing 50 is provided with four visible light sources 13a at positions corresponding to where the visible light source 13 was located. However, the number of visible light sources 13a is not limited to four.

[0062] The configuration of the imaging device 10a in this embodiment, as described above, also produces the same effects as the first embodiment described above.

[0063] [Third Embodiment] The information processing system according to the third embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, the configuration in which the visible light source and RGB camera are arranged will differ from that of the first embodiment. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration and functional block configuration of the information processing device 20, are the same as those described in the first embodiment.

[0064] Figure 9 illustrates the LED light projection angle and RGB light reception angle of the imaging device according to the third embodiment. The LED light projection angle and RGB light reception angle of the imaging device 10b according to this embodiment will be described with reference to Figure 9.

[0065] In the first embodiment, the visible light source 13 was positioned at approximately the same location in the longitudinal direction of the housing 50 (the direction from the support parts 19, 19a toward the side where the infrared light receiving unit 12 is located). In this embodiment, as shown in Figure 9, the imaging device 10b has a visible light source 13b (first light source) positioned in the longitudinal direction of the housing 50 toward the infrared light receiving unit 12 (a position above the RGB camera 14) than the position where the RGB camera 14 is located.

[0066] Even in such cases, as shown in Figure 7 above, when the imaging device 10b enters the dark space, the visible light source 13b enters before the RGB camera 14 and illuminates the dark space, so there is no blackout, and the video data captured by the RGB camera 14 does not contain information about the blackout state, resulting in continuous video data.

[0067] Furthermore, as shown in Figure 9, the housing 50 has shoulder portions 15 that extend outward from the side where the visible light source 13b is located. Therefore, in the housing 50, the shoulder portions 15 are interposed between the visible light source 13b and the RGB camera 14. As a result, a portion of the visible light emitted from the visible light source 13b is blocked by the shoulder portions 15, preventing the visible light emitted from the visible light source 13b from directly entering the RGB camera 14. This makes it possible to avoid adverse effects such as flare and ghosting caused by unwanted visible light reflections from the visible light source 13b on the video data acquired by the RGB camera 14.

[0068] [Fourth Embodiment] The information processing system according to the fourth embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, the configuration in which the infrared light receiving unit 12 has a bandpass filter will be described. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration and functional block configuration of the information processing device 20, are the same as those described in the first embodiment.

[0069] In this embodiment, the wavelength of infrared light emitted from the infrared light source 11 is, for example, 940 [nm], and the infrared light receiving unit 12 has a bandpass filter that has a transmission wavelength band for light of, for example, 940 ± 50 [nm]. Here, the transmission wavelength band refers to the wavelength band in which the transmittance is 50% or more of the maximum transmittance. The visible light source 13 emits visible light having a projection wavelength band of, for example, less than 890 [nm]. Here, the projection wavelength band refers to the wavelength band in which the illuminance is 50% or more of the maximum illuminance. In other words, the wavelength range of the light received by the infrared light receiving unit 12 and the wavelength range of the light emitted by the visible light source 13 do not overlap.

[0070] This configuration suppresses the direct or indirect incidence of light emitted from the visible light source 13 onto the infrared light receiving unit 12, thereby reducing noise during distance measurement.

[0071] [Fifth Embodiment] The information processing system according to the fifth embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, the operation of controlling the on / off operation of the visible light source 13 when distance measurement is performed using the TOF method will be described. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration of the information processing device, are the same as those described in the first embodiment.

[0072] (Configuration and operation of functional blocks in an information processing device) Figure 10 shows an example of the configuration of a functional block in the information processing device according to the fifth embodiment. The configuration and operation of the functional block in the information processing device 20c of the information processing system 1c according to this embodiment will be described with reference to Figure 10.

[0073] As shown in Figure 10, the information processing device 20c includes a first acquisition unit 201, a second acquisition unit 202, a distance calculation unit 203 (calculation unit), an estimation unit 204, an integration unit 205, and a determination unit 206. The functions of the first acquisition unit 201, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, and the integration unit 205 are as described in the first embodiment above.

[0074] The determination unit 206 is a functional unit that determines whether the position or orientation of the imaging device 10 is moving (i.e., whether the imaging device 10 is moving) based on the position and orientation information of the imaging device 10 generated by the estimation unit 204. The determination unit 206 outputs the determination result of whether the position or orientation of the imaging device 10 is moving to the imaging device 10. Alternatively, the determination unit 206 may determine whether the position or orientation of the imaging device 10 is moving based on the video data of the RGB camera 14 acquired by the first acquisition unit 201.

[0075] As shown in Figure 10, the imaging device 10 has a light source control unit 101.

[0076] The light source control unit 101 is a functional unit that controls the on / off operation of the visible light source 13 based on the determination result of the determination unit 206. For example, when distance measurement (imaging) is performed using the infrared light source 11 and the infrared light receiving unit 12, the light source control unit 101 turns off the visible light source 13 if the determination result of the determination unit 206 indicates that the position or orientation of the imaging device 10 is not moving. Also, if the determination result of the determination unit 206 indicates that the position or orientation of the imaging device 10 is moving, the light source control unit 101 cancels the distance measurement using the infrared light source 11 and the infrared light receiving unit 12 (the light receiving operation by the infrared light receiving unit 12). The light source control unit 101 is implemented, for example, by a program executed by the CPU of the imaging device 10, or by a hardware circuit.

[0077] The first acquisition unit 201, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, the integration unit 205, and the determination unit 206 described above are implemented, for example, by the execution of a program by the CPU 501 shown in Figure 5. At least a part of the first acquisition unit 201, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, the integration unit 205, and the determination unit 206 may be implemented by hardware such as an FPGA or ASIC, or by a combination of software and hardware.

[0078] It should be noted that the functional units of the information processing device 20c shown in Figure 10 are conceptual representations of their functions and are not limited to this configuration. For example, multiple functional units shown as independent functional units in the information processing device 20c shown in Figure 10 may be configured as a single functional unit. On the other hand, the functions of a single functional unit in the information processing device 20c shown in Figure 10 may be divided into multiple functions and configured as multiple functional units. Furthermore, each functional unit of the information processing device 20c does not need to be configured as a clear software module as a block as shown in Figure 10; the individual functions of each functional unit may be realized as a whole when a program is executed on the information processing device 20c.

[0079] (Operation flow of an information processing system) Figure 11 is a flowchart showing an example of the operation flow of the information processing system according to the fifth embodiment. The operation flow of the information processing system 1c according to this embodiment will be explained with reference to Figure 11.

[0080] <Step S11> The user operates the imaging device 10 to start capturing video data with the RGB camera 14. In this case, the light source control unit 101 turns on the visible light source 13. Then, the process proceeds to step S12.

[0081] <Step S12> Furthermore, the first acquisition unit 201 starts acquiring video data continuously captured by the RGB camera 14 in a space illuminated with visible light by the visible light source 13, and the estimation unit 204 starts estimating the position and orientation of the imaging device 10 based on VSLAM from the video data and generates position and orientation information. Then, the process proceeds to step S13.

[0082] <Step S13> The user presses the release switch or similar on the imaging device 10. Then, the process proceeds to step S14.

[0083] <Step S14> The determination unit 206 then determines whether the position or orientation of the imaging device 10 is moving, based on the position and orientation information for the imaging device 10 generated by the estimation unit 204. The determination unit 206 outputs the determination result of whether the position or orientation of the imaging device 10 is moving to the imaging device 10. If the determination result of the determination unit 206 indicates that the position or orientation of the imaging device 10 is not moving (step S14: "not moving"), the process proceeds to step S15. If it indicates that it is moving (step S14: "moving"), the process proceeds to step S19.

[0084] <Step S15> When the visible light source 13 is turned off in a dark place, the video data captured by the RGB camera 14 will temporarily contain information about a darkened state. However, if the position and orientation of the imaging device 10 are fixed at the time the visible light source 13 is turned off, there is no problem in ignoring the frames in the video data that contain information about the darkened state and performing the estimation operation by the estimation unit 204. Therefore, since the position and orientation of the imaging device 10 are not moving, the light source control unit 101 turns off the visible light source 13. Then, the process proceeds to step S16.

[0085] <Step S16> The infrared light source 11 emits infrared light, and the infrared light receiving unit 12 receives the reflected light using the TOF method to obtain reception data. The second acquisition unit 202 then acquires the TOF reception data, which is the reflected infrared light emitted from the infrared light source 11 and received by the infrared light receiving unit 12, via the network I / F 509. Then, the process proceeds to step S17.

[0086] <Step S17> The distance calculation unit 203 calculates the distance to the group of objects surrounding the imaging device 10 from the light reception data acquired by the second acquisition unit 202 and generates distance information. The integration unit 205 then integrates the multiple distance information at discontinuous positions of the imaging device 10 generated by the distance calculation unit 203, using the position and orientation information of the imaging device 10 generated by the estimation unit 204, into continuous spatial information (for example, 3D point cloud data). The integration unit 205 then stores the generated spatial information in the auxiliary storage device 505. The process then proceeds to step S18.

[0087] <Step S18> Then, the light source control unit 101 turns on the visible light source 13. Then, the process returns to step S13.

[0088] <Step S19> The light source control unit 101 cancels the distance measurement using the infrared light source 11 and the infrared light receiving unit 12 (the light receiving operation by the infrared light receiving unit 12). Then, the process proceeds to step S20.

[0089] <Step S20> The imaging device 10 then displays an error by, for example, lighting up an indicator. The error display by the indicator may be represented by the color of the light emitted or by the lighting pattern. Alternatively, the information processing device 20 may display the error on an application that controls the imaging device 10. Then, the process returns to step S13.

[0090] In the fourth embodiment described above, the bandpass filter provided in the infrared light receiving unit 12 reduces noise caused by light from the visible light source 13 entering the infrared light receiving unit 12. However, in this embodiment, by turning off the visible light source 13 only when measuring distance using the infrared light source 11 and the infrared light receiving unit 12, it is possible to prevent light from the visible light source 13 from directly or indirectly entering the infrared light receiving unit 12, thereby reducing noise during distance measurement.

[0091] (Modification 1 of the fifth embodiment) In Figure 10 above, the determination unit 206 is shown to be provided in the information processing device 20c, but it may also be provided in the imaging device 10. In this case, the estimation unit 204 outputs the generated position and orientation information to the imaging device 10, and the determination unit 206 of the imaging device 10 determines from the position and orientation information whether or not the position or orientation of the imaging device 10 is moving (i.e., whether or not the imaging device 10 is moving). The light source control unit 101 then controls the on and off operation of the visible light source 13 based on the determination result of the determination unit 206. The configuration of the determination unit 206 in this modified example is also applicable to the sixth and seventh embodiments described later.

[0092] (Modification 2 of the fifth embodiment) In the modified example 1 described above, the imaging device 10 equipped with the determination unit 206 may further include an estimation unit that estimates the position and orientation of the imaging device 10 based on VSLAM from the video data obtained by the imaging device 10 and generates position and orientation information. In this case, the imaging device 10 may be equipped with an estimation unit separately from the estimation unit 204 in the information processing device 20c, or the imaging device 10 may be equipped with an estimation unit instead of the information processing device 20c being equipped with an estimation unit 204.

[0093] If the imaging device 10 has an estimation unit separate from the estimation unit 204 in the information processing device 20c, the estimation unit should output the generated position and orientation information to the determination unit 206 of the imaging device 10. In this case, the estimation unit, separate from the estimation unit 204, may prioritize calculation speed over accuracy when estimating for determination by the determination unit 206.

[0094] If the imaging device 10 has an estimation unit instead of the information processing device 20c having an estimation unit 204, the estimation unit outputs the generated position and orientation information to the determination unit 206 and also outputs the position and orientation information to the information processing device 20c. The first acquisition unit 201 of the information processing device 20c then acquires the video data and position and orientation information and outputs the acquired position and orientation information to the integration unit 205.

[0095] Furthermore, the configuration of the determination unit 206 in this modified example is also applicable to the sixth and seventh embodiments described later.

[0096] [Sixth Embodiment] The information processing system according to the sixth embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, another example of the operation of controlling the on / off operation of the visible light source 13 when distance measurement is performed by the TOF method will be described. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration of the information processing device, are the same as the configuration described in the first embodiment.

[0097] (Configuration of the information processing system) Figure 12 is a diagram showing an example of the configuration of an information processing system according to the sixth embodiment. The configuration of the information processing system 1d according to this embodiment will be described with reference to Figure 12.

[0098] As shown in Figure 12, the information processing system 1d includes an imaging device 10d and an information processing device 20d.

[0099] As shown in Figure 12, the imaging device 10d includes an infrared light source 11 (second light source), an infrared light receiving unit 12 (second light receiving unit), a visible light source 13 (first light source), an RGB camera 14 (first light receiving unit), an IMU (Inertial Measurement Unit) 16, and a housing 50. The infrared light source 11, infrared light receiving unit 12, visible light source 13, RGB camera 14, and housing 50 are as described in the first embodiment above.

[0100] The IMU16 is a device that detects three-dimensional inertial motion using an acceleration sensor and a gyroscope, etc. The IMU16 outputs the detection results to the information processing device 20d.

[0101] (Configuration and operation of functional blocks in an information processing device) Figure 13 is a diagram showing an example of the configuration of a functional block in the information processing apparatus according to the sixth embodiment. The configuration and operation of the functional block in the information processing apparatus 20d according to this embodiment will be described with reference to Figure 13.

[0102] As shown in Figure 13, the information processing device 20d includes a first acquisition unit 201, a second acquisition unit 202, a distance calculation unit 203 (calculation unit), an estimation unit 204, an integration unit 205, a determination unit 206, a third acquisition unit 208, and an IMU estimation unit 209. The functions of the first acquisition unit 201, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, the integration unit 205, and the determination unit 206 are as described in the first embodiment above.

[0103] The third acquisition unit 208 is a functional unit that acquires the detection results from the IMU 16 via the network I / F 509.

[0104] The IMU estimation unit 209 is a functional unit that estimates the position and orientation of the imaging device 10d from the detection results of the IMU 16 acquired by the third acquisition unit 208, and generates position and orientation information.

[0105] The determination unit 206 is a functional unit that determines whether the position or orientation of the imaging device 10d is moving (i.e., whether the imaging device 10d is moving) based on the position and orientation information of the imaging device 10d generated by the IMU estimation unit 209. The determination unit 206 outputs the determination result of whether the position or orientation of the imaging device 10d is moving to the imaging device 10d. Alternatively, the determination unit 206 may integrate the position and orientation information from the estimation unit 204 and the position and orientation information from the IMU estimation unit 209 to determine whether the position or orientation of the imaging device 10d is moving.

[0106] As shown in Figure 13, the imaging device 10d includes an IMU 16 and a light source control unit 101.

[0107] The light source control unit 101 is a functional unit that controls the on / off operation of the visible light source 13 based on the determination result of the determination unit 206. For example, when distance measurement (imaging) is performed using the infrared light source 11 and the infrared light receiving unit 12, the light source control unit 101 turns off the visible light source 13 if the determination result of the determination unit 206 indicates that the position or orientation of the imaging device 10d is not moving. Also, if the determination result of the determination unit 206 indicates that the position or orientation of the imaging device 10d is moving, the light source control unit 101 cancels the distance measurement using the infrared light source 11 and the infrared light receiving unit 12 (the light receiving operation by the infrared light receiving unit 12). The light source control unit 101 is implemented, for example, by a program executed by the CPU of the imaging device 10, or by a hardware circuit.

[0108] The first acquisition unit 201, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, the integration unit 205, the determination unit 206, the third acquisition unit 208, and the IMU estimation unit 209 described above are implemented, for example, by a program executed by the CPU 501 shown in Figure 5. At least a portion of the first acquisition unit 201, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, the integration unit 205, the determination unit 206, the third acquisition unit 208, and the IMU estimation unit 209 may be implemented by hardware such as an FPGA or ASIC, or by a combination of software and hardware.

[0109] The functional units of the information processing device 20d shown in Figure 13 are conceptual representations of their functions and are not limited to this configuration. For example, multiple functional units shown as independent functional units in the information processing device 20d in Figure 13 may be configured as a single functional unit. Alternatively, the functions of a single functional unit in the information processing device 20d shown in Figure 13 may be divided into multiple functions and configured as multiple functional units. Furthermore, each functional unit of the information processing device 20d does not need to be configured as a clear software module as a block as shown in Figure 13; the individual functions of each functional unit may be realized as a whole when a program is executed on the information processing device 20d.

[0110] (Operation flow of an information processing system) Figure 14 is a flowchart showing an example of the operation flow of the information processing system according to the sixth embodiment. The operation flow of the information processing system 1d according to this embodiment will be explained with reference to Figure 14.

[0111] <Step S31> The user operates the imaging device 10d to start capturing video data with the RGB camera 14 and the detection operation with the IMU 16. In this case, the light source control unit 101 turns on the visible light source 13. Then, the process proceeds to step S32.

[0112] <Step S32> Furthermore, the first acquisition unit 201 starts acquiring video data continuously captured by the RGB camera 14 in a space illuminated with visible light by the visible light source 13, and the estimation unit 204 starts estimating the position and orientation of the imaging device 10d based on VSLAM from the video data and generates position and orientation information. Also, the third acquisition unit 208 starts acquiring detection results from the IMU 16, and the IMU estimation unit 209 starts estimating the position and orientation of the imaging device 10d from the detection results of the IMU 16 acquired by the third acquisition unit 208 and generates position and orientation information. Then, the process proceeds to step S33.

[0113] <Step S33> The user presses the release switch or similar on the imaging device 10d. Then, the process proceeds to step S34.

[0114] <Step S34> The determination unit 206 then determines whether the position or orientation of the imaging device 10d is moving, based on the position and orientation information for the imaging device 10d generated by the IMU estimation unit 209. The determination unit 206 outputs the determination result of whether the position or orientation of the imaging device 10d is moving to the imaging device 10d. If the determination result of the determination unit 206 indicates that the position or orientation of the imaging device 10d is not moving (step S34: "not moving"), the process proceeds to step S35. If it indicates that it is moving (step S34: "moving"), the process proceeds to step S39.

[0115] <Step S35> When the visible light source 13 is turned off in a dark place, the video data captured by the RGB camera 14 will temporarily contain information about a darkened state. However, if the position and orientation of the imaging device 10d are fixed at the time the visible light source 13 is turned off, there is no problem in ignoring the frames in the video data that contain information about the darkened state and performing the estimation operation by the estimation unit 204. Therefore, since the position and orientation of the imaging device 10d are not moving, the light source control unit 101 turns off the visible light source 13. Then, the process proceeds to step S36.

[0116] <Steps S36~S38> The processes in steps S36 to S38 are the same as those in steps S16 to S18 shown in Figure 11 above. Then, the process returns to step S33.

[0117] <Steps S39 and S40> The processes in steps S39 and S40 are the same as those in steps S19 and S20 shown in Figure 11 above. Then, the process returns to step S33.

[0118] This provides the same effects as the fifth embodiment described above.

[0119] Furthermore, the integration unit 205 may integrate multiple distance information points at discontinuous locations of the imaging device 10 generated by the distance calculation unit 203, using not only the position and orientation information of the imaging device 10 generated by the estimation unit 204 but also the position and orientation information generated by the IMU estimation unit 209, into continuous spatial information (e.g., 3D point cloud data). This allows for obtaining highly accurate spatial information.

[0120] [Seventh Embodiment] The information processing system according to the seventh embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, the operation of controlling the on / off operation of the visible light source 13 from the acquired light reception amount data will be described. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration and functional block configuration of the information processing device, are the same as those described in the first embodiment.

[0121] (Configuration and operation of functional blocks in an information processing device) Figure 15 shows an example of the configuration of a functional block in the information processing device according to the seventh embodiment. The configuration and operation of the functional block of the information processing device 20e of the information processing system 1e according to this embodiment will be described with reference to Figure 15.

[0122] As shown in Figure 15, the information processing device 20e includes a first acquisition unit 201e, a second acquisition unit 202, a distance calculation unit 203, an estimation unit 204, and an integration unit 205. The functions of the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, and the integration unit 205 are as described in the first embodiment above.

[0123] The first acquisition unit 201e is a functional unit that acquires video data (such as video) continuously captured by the RGB camera 14 in a space illuminated with visible light by the visible light source 13, via the network I / F 509.

[0124] As shown in Figure 15, the imaging device 10 includes a light source control unit 101 and a light reception amount determination unit 102.

[0125] The light reception amount determination unit 102 is a functional unit that acquires one frame from the video data obtained by the RGB camera 14 and determines whether a sufficient amount of visible light is being received by the RGB camera 14 based on the pixel values ​​(luminance values) of that frame. For example, the light reception amount determination unit 102 determines whether the amount of light obtained from the pixel values ​​of one frame is greater than a predetermined amount, and if it is greater than the predetermined amount, it determines that a sufficient amount of light is being received. The light reception amount determination unit 102 outputs the determination result of whether a sufficient amount of visible light is being received by the RGB camera 14 to the light source control unit 101. The light reception amount determination unit 102 can be implemented, for example, by a program executed by the CPU of the imaging device 10, or by a hardware circuit. Furthermore, the light reception amount determination unit 102 is not limited to acquiring one frame from the video data obtained by the RGB camera 14, but may also determine whether a sufficient amount of visible light is being received from data detected by an optical sensor (light receiving unit) provided separately from the RGB camera 14. In this case, by positioning the light sensor higher than the RGB camera 14 with respect to the bottom of the housing 50, the amount of light received by the light sensor decreases before the amount of light received by the RGB camera 14 decreases when the imaging device 10 is moved to a dark place. This makes it possible to determine more quickly that sufficient visible light is not being received.

[0126] The light source control unit 101 is a functional unit that controls the on / off operation of the visible light source 13 based on the determination result of the light reception amount determination unit 102. For example, if the determination result of the light reception amount determination unit 102 indicates that sufficient light is not being received by the RGB camera 14, the light source control unit 101 turns on the visible light source 13. Also, if the determination result of the light reception amount determination unit 102 indicates that sufficient light is being received by the RGB camera 14, the light source control unit 101 turns off the visible light source 13. The light source control unit 101 can be implemented, for example, by a program executed by the CPU of the imaging device 10, or by a hardware circuit.

[0127] Furthermore, the light source control unit 101 may control the on / off operation of the visible light source 13 based on the determination result of the light reception amount determination unit 102 regarding whether a sufficient amount of visible light is received by the RGB camera 14, as well as the determination result of the determination unit 206 described in the fifth or sixth embodiment above regarding whether the position or orientation is moving. For example, the light source control unit 101 can control the visible light source 13 to turn on when the position or orientation of the imaging device 10 is moving and a sufficient amount of light is not received.

[0128] The first acquisition unit 201e, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, and the integration unit 205 described above are implemented, for example, by the execution of a program by the CPU 501 shown in Figure 5. At least a portion of the first acquisition unit 201e, the second acquisition unit 202, the distance calculation unit 203, the estimation unit 204, and the integration unit 205 may be implemented by hardware such as an FPGA or ASIC, or by a combination of software and hardware.

[0129] It should be noted that the functional units of the information processing device 20e shown in Figure 15 are conceptual representations of their functions and are not limited to this configuration. For example, multiple functional units shown as independent functional units in the information processing device 20e shown in Figure 15 may be configured as a single functional unit. On the other hand, the functions of one functional unit in the information processing device 20e shown in Figure 15 may be divided into multiple functions and configured as multiple functional units. Furthermore, each functional unit of the information processing device 20e does not need to be configured as a clear software module as a block as shown in Figure 15; the individual functions of each functional unit may be realized as a whole when a program is executed on the information processing device 20e.

[0130] (Operation flow of an information processing system) Figure 16 is a flowchart showing an example of the operation flow of the information processing system according to the seventh embodiment. The operation flow of the information processing system 1e according to this embodiment will be explained with reference to Figure 16.

[0131] <Step S51> The light reception amount determination unit 102 acquires one frame from video data (video, etc.) continuously captured by the RGB camera 14 in a space illuminated with visible light by the visible light source 13. Then, it proceeds to step S52.

[0132] <Step S52> The light reception amount determination unit 102 determines whether a sufficient amount of visible light is being received by the RGB camera 14 based on the pixel values ​​(luminance values) of one frame acquired. For example, the light reception amount determination unit 102 determines whether the amount of light received obtained from the pixel values ​​of one frame is greater than a predetermined amount, and if it is greater than the predetermined amount, it determines that a sufficient amount of light is being received. The light reception amount determination unit 102 outputs the determination result of whether a sufficient amount of visible light is being received by the RGB camera 14 to the light source control unit 101. If a sufficient amount of light is not being received (step S52: not being received), the process proceeds to step S53, and if a sufficient amount of light is being received (step S52: being received), the process proceeds to step S54.

[0133] <Step S53> Because the amount of light received from one frame of video data is insufficient, the light source control unit 101 turns on the visible light source 13. Then, the process returns to step S51.

[0134] <Step S54> From the perspective of reducing power consumption, there is little point in turning on the visible light source 13 when the RGB camera 14 is capturing images in a brightly lit space. Here, since the amount of light received from one frame of video data is sufficient, the light source control unit 101 turns off the visible light source 13. Then, the process returns to step S51.

[0135] As described above, in the information processing system 1e according to this embodiment, when lighting is required, such as when the imaging device 10 moves from a bright space to a dark space, the visible light source 13 is automatically turned on, and turned off when it is not needed. This reduces power consumption.

[0136] In this configuration, the visible light source 13 is turned on or off according to the amount of light received from the RGB camera 14's video data. However, this is not the only option; the intensity of the light emitted from the visible light source 13 may also be adjusted according to the amount of light received.

[0137] Furthermore, the configuration of the imaging device 10 according to this embodiment can also be applied to the first to sixth embodiments described above.

[0138] [Eighth Embodiment] The information processing system according to the eighth embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, a configuration will be described in which the LED illumination angle and RGB reception angle cover the entire sphere by providing three or more visible light sources and RGB cameras. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration and functional block configuration of the information processing device 20, are the same as those described in the first embodiment.

[0139] Figure 17 is a diagram illustrating the LED light projection angle and RGB light reception angle of the imaging device according to the eighth embodiment. The configuration of the imaging device 10e according to this embodiment will be described with reference to Figure 17.

[0140] In the first embodiment, the angles of visible light emitted by the two visible light sources 13 (LED projection angles) were each slightly greater than 180 degrees, thus covering the entire sphere. Similarly, the angles of visible light received by the two RGB cameras 14 (RGB reception angles) were each slightly greater than 180 degrees, thus covering the entire sphere.

[0141] In this embodiment, as shown in Figure 17, the imaging device 10e includes, for example, three visible light sources 13e (first light sources) and three RGB cameras 14e (first light receiving units) on the side of the housing 50. The three visible light sources 13e and the three RGB cameras 14e are arranged alternately on the side of the housing 50, as shown in Figure 17.

[0142] Furthermore, as shown in Figure 17(b), the angles of visible light emitted by the three visible light sources 13e (LED projection angles) are each slightly greater than 120 degrees, thus covering the entire sphere's illumination range. Similarly, as shown in Figure 17(a), the angles of visible light received by the three RGB cameras 14e (RGB reception angles) are each slightly greater than 120 degrees, thus covering the entire sphere's reception range.

[0143] The configuration of the information processing system according to this embodiment described above can achieve the same effects as the first embodiment described above.

[0144] Furthermore, the system is not limited to three visible light sources 13e and three RGB cameras 14e; it may also include four or more visible light sources 13e and four or more RGB cameras 14e. For example, if four visible light sources 13e and four RGB cameras 14e are provided, the LED light emission angle and the RGB light reception angle should each be set to an angle slightly greater than 90 degrees.

[0145] [Ninth Embodiment] The information processing system according to the ninth embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, a configuration will be described in which the LED illumination angle and RGB reception angle cover the entire sphere by providing three or more visible light sources and RGB cameras. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration and functional block configuration of the information processing device 20, are the same as those described in the first embodiment.

[0146] Figure 18 shows an example of the configuration of the imaging device according to the ninth embodiment. The configuration of the imaging device 10f according to this embodiment will be described with reference to Figure 18.

[0147] As shown in Figure 18, the imaging device 10f includes an infrared light source substrate 110, a visible light source substrate 130, a prism 17, and a lens system 18. Of these, the infrared light source substrate 110, the prism 17, and the lens system 18 constitute the infrared light source 11, while the visible light source substrate 130, the prism 17, and the lens system 18 constitute the visible light source 13. In other words, the prism 17 and the lens system 18 are shared by the infrared light source 11 and the visible light source 13.

[0148] The infrared light source substrate 110 is a substrate on which a semiconductor light-emitting element that emits infrared light is mounted. The infrared light emitted from the infrared light source substrate 110 is incident on the prism 17, as shown in Figure 18.

[0149] The visible light source substrate 130 is a substrate on which a semiconductor light-emitting element that emits visible light is mounted. The visible light emitted from the visible light source substrate 130 is incident on the prism 17, as shown in Figure 18.

[0150] The prism 17 is an optical component that reflects and bends infrared light emitted from the infrared light source substrate 110, directing it toward the lens system 18. The prism 17 also transmits infrared light emitted from the visible light source substrate 130, directing it toward the lens system 18.

[0151] The lens system 18 is an optical component that projects infrared light and visible light emitted through the prism 17 at a predetermined angle of view after optical processing.

[0152] Thus, in the imaging device 10f according to this embodiment, the optical components (prism 17 and lens system 18) used in the infrared light source 11 and the visible light source 13 are shared. This reduces the number of optical components, thereby reducing the size, weight, and cost of the imaging device 10f.

[0153] [Tenth Embodiment] The information processing system according to the tenth embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, the configuration when the imaging device is a handheld laser scanner will be described. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration and functional block configuration of the information processing device 20, are the same as those described in the first embodiment.

[0154] Figure 19 shows an example of the configuration of the imaging device according to the tenth embodiment. Figure 20 is a diagram illustrating the operation when the imaging device according to the tenth embodiment is placed in a dark place. Figure 21 shows another example of the configuration of the imaging device according to the tenth embodiment. The configuration of the imaging device 10g according to this embodiment will be described with reference to Figures 19 to 21.

[0155] The imaging device 10g shown in Figure 19 is a handheld laser scanner that scans information about the surrounding space while the user moves. As shown in Figure 19, the imaging device 10g comprises an infrared light source 11g-1 (second light source), a mirror 11g-2 (scanning member), an infrared light receiving unit 12 (second light receiving unit), a visible light source 13g (first light source), an RGB camera 14 (first light receiving unit), and a support unit 19g.

[0156] The infrared light source 11g-1 is a light source for emitting collimated laser light (infrared light). The mirror 11g-2 is a polygon mirror or MEMS (Micro Electro Mechanical Systems) mirror, etc., that reflects or refracts the laser light emitted from the infrared light source 11g-1 and scans it in the surrounding space. As shown in Figure 19, the infrared light source 11g-1 and the mirror 11g-2 are covered and protected by a cover material, for example, a material that transmits infrared light.

[0157] The infrared light receiving unit 12 receives reflected light from surrounding objects among the laser beam scanned by the mirror 11g-2. In the example shown in Figure 19, four infrared light receiving units 12 are arranged on the side of the housing 50.

[0158] The visible light source 13g is positioned lower to the housing 50 (towards the support part 19g, which will be described later) than the infrared light source 11 and the infrared light receiving unit 12, and is a ring light-shaped light source such as an LED (Light Emitting Diode) that emits light in the visible light region (visible light). The visible light source 13g may also be a light source having a lens-shaped or diffusing member as described above.

[0159] Multiple RGB cameras 14 are arranged on the side of the housing 50 and capture (receive) visible light from the surroundings. From the captured visible light, the RGB cameras 14 output video data consisting of information such as R (red), G (green), and B (blue). The RGB cameras 14 are positioned on the side of the support part 19g rather than the visible light source 13g in the longitudinal direction of the housing 50 (the direction from the support part 19g, described later, toward the side where the infrared light receiving unit 12 is located). In the example shown in Figure 19, four RGB cameras 14 are arranged on the side of the housing 50. Furthermore, because the surroundings are illuminated by the visible light source 13g, the RGB cameras 14 can acquire video data even in dark places.

[0160] The support portion 19g is a part that can be grasped by the user and is formed at the end of the housing 50 opposite to the side where the infrared light receiving unit 12 is located.

[0161] As shown in Figure 20, when a user holding the support part 19g moves the imaging device 10g into a dark space such as an attic, the visible light source 13g enters the dark space before the RGB camera 14 and illuminates the dark space (by irradiating it with visible light). As a result, there is no blackout, and the video data captured by the RGB camera 14 does not include information about the blackout state, resulting in continuous video data. Consequently, as described below, the information necessary for the estimation of the position and orientation of the imaging device 10g by the estimation unit 204 using VSLAM is not interrupted, and continuous position and orientation information can be obtained.

[0162] Furthermore, if four infrared light receiving units 12 and four RGB cameras 14 are arranged as shown in Figure 19 to perform 360-degree imaging, the area in which the user is captured will be large when the infrared light receiving units 12 and RGB cameras 14 are located on the user's side. Therefore, a configuration that excludes the infrared light receiving units 12 and RGB cameras 14 located on the user's side may be adopted, as shown in the top view of Figure 21(a) and the front view of Figure 21(b). This contributes to cost and weight reduction by reducing some of the infrared light receiving units 12 and RGB cameras 14. However, in this case, it is desirable that the RGB camera 14 has a light receiving range of at least 180 degrees along the longitudinal axis of the imaging device 10g, the visible light source 13g also has an illumination range of at least 180 degrees, and the infrared light receiving unit 12 also has a light receiving range of at least 180 degrees. Furthermore, it is desirable for the user to cover blind spots by once changing the orientation of the imaging device 10g by 180 degrees.

[0163] [Embodiment 11] The information processing system according to the eleventh embodiment will be described, focusing on the differences from the information processing system 1 according to the first embodiment. In this embodiment, a structured camera type imaging device 10 will be described. The overall configuration of the information processing system according to this embodiment, as well as the hardware configuration and functional block configuration of the information processing device 20, are the same as those described in the first embodiment.

[0164] In the first embodiment described above, a TOF (Time-of-Flight) type imaging device 10 was described, but the imaging device 10 according to this embodiment is a structured camera type imaging device. In this embodiment, the infrared light source 11 emits patterned light (structured light) in the infrared region having a pattern such as a grid, stripes, or random. The infrared light receiving unit 12 receives reflected light from the patterned light emitted from the infrared light source 11, which includes distortion of unevenness reflected by surrounding objects. The second acquisition unit 202 then acquires received light data including information on distortion of unevenness from the infrared light receiving unit 12, and the distance calculation unit 203 calculates the distance of the group of objects surrounding the imaging device 10 from the received light data and generates distance information.

[0165] With the configuration of the imaging device 10 according to this embodiment described above, the same effects as those of the first embodiment described above can be achieved.

[0166] [Embodiment 12] Figure 22 is a diagram showing an example of the schematic configuration of a smartphone or the like according to the 12th embodiment. The configuration of the imaging device 10h according to this embodiment will be described with reference to Figure 22.

[0167] In the embodiments described above, an imaging device 10 having a rod-shaped housing with support parts 19, 19a as shown in Figure 2, etc., was described, but an imaging device 10h such as a smartphone or tablet terminal as shown in Figure 22 can also be used.

[0168] The imaging device 10h shown in Figure 22, like the imaging device 10 described above, receives infrared light to measure distance using the TOF method and captures video data with an RGB camera to estimate the position and orientation of the imaging device 10h based on VSLAM. The imaging device 10h includes an infrared light source 11h (second light source), an infrared light receiving unit 12h (second light receiving unit), a visible light source 13h (first light source), and an RGB camera 14h (first light receiving unit). In this case, the part of the housing of the imaging device 10h below the position where the infrared light source 11h, infrared light receiving unit 12h, visible light source 13h, and RGB camera 14h are located becomes the support part corresponding to the support part 19 described above that is grasped by the user.

[0169] In the example shown in Figure 22, the visible light source 13h is positioned at approximately the same height as the RGB camera 14h, but it may be positioned higher than the RGB camera 14h when the support base is used as the reference point. Furthermore, the positions of the infrared light source 11h and the infrared light receiving unit 12h are not limited to those shown in Figure 22 and can be changed to any desired position.

[0170] Thus, when applying the imaging device 10h to the information processing system 1 described above, for example, when applied to the 5th to 7th embodiments described above, the power consumption of the imaging device 10h can be reduced by controlling the light source by the light source control unit 101 based on the determination result of the determination unit 206. Therefore, this is effective for imaging devices 10h that are smartphones or tablet terminals with limited battery capacity.

[0171] Furthermore, at least a portion of the functional units of the information processing devices 20, 20c to 20e according to each of the above embodiments may be implemented by the imaging devices 10, 10a to 10h. That is, the imaging devices 10, 10a to 10h may be equipped with a CPU, and at least a portion of the functional units of the information processing devices 20, 20c to 20e may be implemented by the execution of a program.

[0172] Furthermore, in each of the embodiments described above, if at least one of the functions of the information processing devices 20, 20c to 20e is realized by program execution, the program is provided pre-installed in ROM or the like. Also, in each of the embodiments described above, the program executed by the information processing devices 20, 20c to 20e may be configured to be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk-Recordable), or a DVD (Digital Versatile Disc). Also, in each of the embodiments described above, the program executed by the information processing devices 20, 20c to 20e may be configured to be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Also, in each of the embodiments described above, the program executed by the information processing devices 20, 20c to 20e may be configured to be provided or distributed via a network such as the Internet. Furthermore, in each of the embodiments described above, the programs executed by the information processing devices 20, 20c to 20e are configured as modules that include at least one of the functional units described above. In actual hardware, the CPU reads the program from the storage device and executes it, thereby loading and generating the functional units described above onto the main memory.

[0173] The embodiments of the present invention are as follows. <1> A first light source is installed in the housing of the imaging device and irradiates the space with light in the first wavelength range, A first light receiving unit, which is installed in the housing and receives light in the first wavelength range, The bottom portion located at the longitudinal end of the housing, An estimation unit that estimates the position and orientation of the imaging device from the video data from the first light-receiving unit, Equipped with, The first light source is installed at the same height as the first light receiving unit, or at a higher position than the first light receiving unit, with reference to the bottom in the longitudinal direction. The first light receiving unit is an information processing system that acquires the video data while the light in the first wavelength range is irradiated by the first light source. <2> A second light source is installed in the housing and emits light in a second wavelength region that is longer than the first wavelength region, A second light-receiving unit that receives reflected light from an object among the light emitted from the aforementioned second light source, The aforementioned further comprising <1> This is the information processing system described in [the relevant document]. <3> A calculation unit calculates the distance to an object around the imaging device from the light-receiving data by the second light-receiving unit, An integration unit integrates the distance calculated by the calculation unit and the position and orientation estimated by the estimation unit as spatial information, The aforementioned further comprising <2> This is the information processing system described in [the relevant document]. <4> The following further comprises a light source control unit that controls the turning on and off of the first light source according to the amount of light received by the first light receiving unit. <1> ~ <3> It is an information processing system described in any one of the items. <5> The following further comprises a light source control unit that turns off the first light source when the second light receiving unit performs a light receiving operation. <3> This is the information processing system described in [the relevant document]. <6> The system further includes a determination unit that determines whether or not the imaging device is moving based on the position and orientation of the imaging device estimated by the estimation unit when the second light receiving unit performs a light receiving operation. If the determination unit determines that the imaging device is not operating, the light source control unit turns off the first light source. <5> This is the information processing system described in [the relevant document]. <7> An inertial measuring device for detecting the inertial motion of the imaging device, When the second light receiving unit performs a light receiving operation, a determination unit determines whether or not the imaging device is moving based on the detection result from the inertial measuring device, Furthermore, If the determination unit determines that the imaging device is not operating, the light source control unit turns off the first light source. <5> This is the information processing system described in [the relevant document]. <8> The first wavelength region and the second wavelength region are the non-overlapping <2> or <3> This is the information processing system described in [the relevant document]. <9> The first light source and the second light source share an optical member for irradiating light. <2> or <3> This is the information processing system described in [the relevant document]. <10> The second light source has a diffusing member that diffuses the light being irradiated. <2> or <3> This is the information processing system described in [the relevant document]. <11> The device further comprises a scanning member that dynamically reflects or refracts light from the second light source to scan it. <2> or <3> This is the information processing system described in [the relevant document]. <12> The second light source irradiates structured light <2> or <3> This is the information processing system described in [the relevant document]. <13> The casing and A first light source installed in the aforementioned housing irradiates the space with light in the first wavelength range, A first light receiving unit, which is installed in the housing and receives light in the first wavelength range, The bottom portion located at the longitudinal end of the housing, Equipped with, The light-receiving range of the first light-receiving unit is 180 degrees or more with respect to the longitudinal direction. The first light source is an imaging device installed at the same height as the first light-receiving unit, or at a higher position than the first light-receiving unit, with reference to the bottom in the longitudinal direction. [Explanation of symbols]

[0174] 1. 1c~1e Information Processing Systems 10, 10a~10h Imaging device 11 Infrared light source 11g-1 infrared light source 11g-2 Mirror 11h infrared light source 12, 12h Infrared light receiving unit 13, 13a, 13b, 13e, 13g, 13h visible light source 14, 14e, 14h RGB cameras 15 Shoulder 16 IMU 17 Prism 18 Lens System 19, 19a, 19g Support part 20, 20c~20e Information Processing Equipment 30 devices 50 cabinets 101 Light source control unit 102 Received light amount determination section 110 Infrared Light Source Substrate 130 Visible light source board 201, 201e 1st acquisition part 202 Second Acquisition Department 203 Distance Calculation Unit 204 Estimation section 205 Integration Department 206 Judgment section 208 Third Acquisition Department 209 IMU estimation section 501 CPU 502 ROM 503 RAM 505 Auxiliary storage 506 Recording media 507 Media Drive 508 displays 509 Network Interface 510 Bus Line 511 keyboard 512 mice 513 DVD 514 DVD drive 1000 Imaging device 1011 Infrared light source 1012 Infrared light receiving unit 1013 Visible light source 1014 RGB Camera 1050 cabinets [Prior art documents] [Patent Documents]

[0175] [Patent Document 1] Japanese Patent Publication No. 2023-137739

Claims

1. A first light source is installed in the housing of the imaging device and irradiates the space with light in the first wavelength range, A first light receiving unit, which is installed in the housing and receives light in the first wavelength range, The bottom portion located at the longitudinal end of the housing, An estimation unit that estimates the position and orientation of the imaging device from the video data obtained by the first light receiving unit, Equipped with, The first light source is installed at the same height as the first light receiving unit, or at a position higher than the first light receiving unit, with reference to the bottom in the longitudinal direction. The first light receiving unit is an information processing system that acquires the video data while the first light source is irradiating the image with light in the first wavelength range.

2. A second light source is installed in the housing and irradiates light in a second wavelength region that is longer in wavelength than the first wavelength region, A second light-receiving unit that receives reflected light from an object among the light irradiated from the aforementioned second light source, The information processing system according to claim 1, further comprising:

3. A calculation unit calculates the distance to an object around the imaging device from the light-receiving data by the second light-receiving unit, An integration unit integrates the distance calculated by the calculation unit and the position and orientation estimated by the estimation unit as spatial information, The information processing system according to claim 2, further comprising:

4. The information processing system according to any one of claims 1 to 3, further comprising a light source control unit that controls the turning on and off of the first light source according to the amount of light received by the first light receiving unit.

5. The information processing system according to claim 3, further comprising a light source control unit that turns off the first light source when the second light receiving unit performs a light receiving operation.

6. The system further includes a determination unit that determines whether or not the imaging device is moving based on the position and orientation of the imaging device estimated by the estimation unit when the second light receiving unit performs a light receiving operation. The information processing system according to claim 5, wherein the light source control unit turns off the first light source when the determination unit determines that the imaging device is not operating.

7. An inertial measuring device for detecting the inertial motion of the imaging device, When the second light receiving unit performs a light receiving operation, a determination unit determines whether or not the imaging device is moving based on the detection result from the inertial measuring device, Furthermore, The information processing system according to claim 5, wherein the light source control unit turns off the first light source when the determination unit determines that the imaging device is not operating.

8. The information processing system according to claim 2 or 3, wherein the first wavelength region and the second wavelength region do not overlap.

9. The information processing system according to claim 2 or 3, wherein the first light source and the second light source share an optical member for irradiating light.

10. The information processing system according to claim 2 or 3, wherein the second light source has a diffusing member that diffuses the irradiated light.

11. The information processing system according to claim 2 or 3, further comprising a scanning member that dynamically reflects or refracts light from the second light source for scanning.

12. The information processing system according to claim 2 or 3, wherein the second light source irradiates structured light.

13. The casing and A first light source installed in the aforementioned housing irradiates the space with light in the first wavelength range, A first light receiving unit, which is installed in the housing and receives light in the first wavelength range, The bottom portion located at the longitudinal end of the housing, Equipped with, The light-receiving range of the first light-receiving unit is 180 degrees or more with respect to the longitudinal direction. The first light source is an imaging device installed at the same height as the first light-receiving unit, or higher than the first light-receiving unit, with reference to the bottom in the longitudinal direction.

Images