Information processing apparatus, information processing method, and program

Abstract

There is provided an information processing system, including: a feature point extraction unit configured to extract a feature point from an image of an environment using the image or point cloud data of the environment; and an emission region determination unit configured to determine, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame.

Description

INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Japanese Priority Patent Application JP 2023-174573 filed October 6, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

[0003] In recent years, the development of technology for sensing an environment using a camera that images the environment and a sensor that is capable of detecting the position of an object present in the environment using sensing light, such as light detection and ranging (LiDAR), has progressed.

[0004] For example, in simultaneous localization and mapping (SLAM), a combination of Visual-SLAM using an image taken by a camera and LiDAR-SLAM using point cloud data measured by LiDAR is being considered. In Visual-SLAM, highly accurate estimation can be performed in an environment with little structural information, because texture information is used. However, the estimation accuracy decreases in the case where there is a change in illumination. Meanwhile, in LiDAR-SLAM, highly accurate estimation can be performed even in the case where there is a change in illumination, because sensing light is used. However, the estimation accuracy decreases in an environment with little structural information. For this reason, the combination of Visual-SLAM and LiDAR-SLAM makes it possible to further improve the robustness of estimation for the environment.

[0005] Further, in recent years, as disclosed in the following Patent Literature 1,an imaging device that is capable of measuring the distance to an object present in an environment by detecting reflected light of light applied to the object and generating an image of the environment has been developed.

[0006] WO 2020 / 255999Summary

[0007] However, since the imaging device disclosed in Patent Literature 1 emits sensing light to an object present in an environment in order to measure the distance to the object, power consumption is higher than that of an imaging device that performs only imaging. In this regard, it has been desired to reduce the power consumption of the imaging device that is capable of imaging an environment and measuring the distance to an object present in the environment.

[0008] In accordance with the present disclosure, there is provided an information processing system, including: a feature point extraction unit configured to extract a feature point from an image of an environment using the image or point cloud data of the environment; and an emission region determination unit configured to determine, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame, the feature point extraction unit and the emission region determination unit being each implemented via at least one processor.

[0009] Further, in accordance with the present disclosure, there is provided an information processing method executed by a computer, including: extracting a feature point from an image of an environment using the image or point cloud data of the environment; and determining, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame.

[0010] Further, in accordance with the present disclosure, there is provided a non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method including: extracting a feature point from an image of an environment using the image or point cloud data of the environment; and determining, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame.

[0011] Fig. 1A is a schematic diagram describing an imaging device that is controlled by an information processing apparatus according to an embodiment of the present disclosure.Fig. 1B is an explanatory diagram for describing coaxial measurement for imaging and ranging.Fig. 2 is a schematic diagram schematically showing an example of the mechanism of the imaging device shown in Fig. 1A.Fig. 3 is a schematic diagram schematically showing an example of the mechanism of the imaging device shown in Fig. 1A.Fig. 4 is a schematic diagram schematically showing an example of the mechanism of the imaging device shown in Fig. 1A.Fig. 5 is a block diagram showing a functional configuration of the information processing apparatus according to the embodiment.Fig. 6 is a schematic diagram describing a method of determining an emission region.Fig. 7 is a flowchart showing a flow of operations of the information processing apparatus and a sensor unit according to the embodiment.Fig. 8 is a flowchart showing a flow of operations of an information processing apparatus and a sensor unit according to a first modified example.Fig. 9A is an explanatory diagram describing observation accuracy of a feature point relative to the distance from the sensor unit.Fig. 9B is an explanatory diagram describing observation accuracy of a feature point relative to the distance from the sensor unit.Fig. 10A is a schematic diagram showing a change in the size of an emission region in a second modified example.Fig. 10B is a schematic diagram showing a change in the size of the emission region in the second modified example.Fig. 11 is a flowchart showing a flow of operations of an information processing apparatus and a sensor unit according to the second modified example.Fig. 12 is a block diagram showing a functional configuration of an information processing apparatus according to a third modified example.Fig. 13 is an explanatory diagram describing a method of estimating a position of a feature point from a position of a sensor unit.Fig. 14 is a flowchart showing a flow of operations of the information processing apparatus and a sensor unit according to the third modified example.Fig. 15 is a block diagram showing a functional configuration of an information processing apparatus according to a fourth modified example.Fig. 16 is a graph diagram showing an example of a photon count histogram in the case where the sensor unit is SPAD LiDAR.Fig. 17 is a flowchart showing a flow of operations of the information processing apparatus and a sensor unit according to the fourth modified example.Fig. 18 is a block diagram showing a configuration example of a computer.

[0012] An embodiment of the present disclosure will be described below in detail with reference to the accompanying drawings. Note that in the present specification and drawings, components having substantially the same functional configurations will be denoted by the same reference symbols and redundant description will be omitted.

[0013] Note that description will be made in the following order. 1. Imaging device 2. Configuration of information processing apparatus 3. Operation example 4. Modified example 4.1. First modified example 4.2. Second modified example 4.3. Third modified example 4.4. Fourth modified example 5. Hardware configuration example

[0014] <1. Imaging device> An imaging device that is controlled by an information processing apparatus according to an embodiment of the present disclosure will be described first with reference to Fig. 1A to Fig. 4. Fig. 1A is a schematic diagram describing an imaging device CS that is controlled by an information processing apparatus according to this embodiment. Fig. 1B is an explanatory diagram for describing coaxial measurement for imaging and ranging. Fig. 2 to Fig. 4 are each a schematic diagram schematically showing an example of the mechanism of the imaging device CS shown in Fig. 1A.

[0015] The imaging device CS that is controlled by an information processing apparatus according to this embodiment is a sensor that is capable of imaging an environment and measuring the distance to an object present in the environment. Specifically, the imaging device CS has an imaging function for taking an RGB color image or monochrome image and a ranging function for measuring the distance to an object present in an environment on the basis of the flight time until the sensing light emitted to the object returns.

[0016] As shown in Fig. 1A, the imaging device CS is capable of coaxially performing imaging of an environment and measurement of the distance to an object present in the environment. That is, the imaging device CS is capable of performing imaging of the same region and distance measurement from the same position. In such a case, since an imaging region IF where an environment is imaged and a ranging region DF where the distance to an object present in the environment is measured are the same, the imaging device CS is capable of acquiring an image and ranging information that are not misaligned with each other.

[0017] Meanwhile, as shown in Fig. 1B, an imaging device IS and a distance measuring device DS are separate devices, the imaging device IS and the distance measuring device DS perform observation from different positions. For this reason, misalignment occurs between the imaging region IF where the imaging device IS images the environment and the ranging region DF where the distance measuring device DS measures the distance to an object present in the environment. In such a case, in order to integrate the image taken by the imaging device IS and ranging information of the distance measuring device DS, it is necessary to correct the positional relationship between the imaging device IS and the distance measuring device DS, and correct the coordinates of the image and the coordinates of the ranging information. Further, since misalignment has occurred between the imaging region IF and the ranging region DF, the image and ranging information can be integrated for a region where the imaging region IF and the ranging region DF are superimposed on each other.

[0018] Since the imaging device CS that is controlled by an information processing apparatus according to this embodiment is capable of coaxially performing imaging of an environment and measurement of the distance to an object present in the environment, the image and ranging information can be integrated without performing correction. Further, since the imaging device CS is capable of performing imaging of an environment and measurement of the distance to an object present in the environment in synchronization with each other, it is possible to acquire the image and ranging information that are completely synchronized in time. Further, the imaging device CS makes it possible to make the hardware and the configuration of the entire system more compact as compared with the case where the imaging device IS and the distance measuring device DS are separately provided.

[0019] For example, as shown in Fig. 2, the imaging device CS may be a sensor that includes a ranging pixel 25d and an imaging pixel 25i in a pixel region 20. The ranging pixel 25d is a pixel that receives sensing light (e.g., infrared rays) emitted to an object present in an environment. The imaging device CS is capable of calculating the distance to an object on the basis of the flight time of sensing light received by the ranging pixel 25d. The imaging pixel 25i includes, for example, a red pixel 25r that receives red light, a green pixel 25g that receives green light, and a blue pixel 25b that receives blue light. The imaging device CS is capable of generating an image of an environment on the basis of the amount of received light of each of the red pixel 25r, the green pixel 25g, and the blue pixel 25b. Since both the ranging pixel 25d the imaging pixel 25i are present in the pixel region 20, the imaging device CS is capable of coaxially performing imaging of an environment and measurement of the distance to an object present in the environment.

[0020] Further, as shown in Fig. 3, the imaging device CS may be a sensor obtained by stacking a first substrate 21 in which the imaging pixel 25i is provided and a second substrate 22 in which the ranging pixel 25d is provided. The first substrate 21 includes the red pixel 25r that receives red light, the green pixel 25g that receives green light, and the blue pixel 25b that receives blue light. The imaging device CS is capable of generating an image of an environment on the basis of the amount of received light of each of the red pixel 25r, the green pixel 25g, and the blue pixel 25b provided in the first substrate 21. The second substrate 22 includes the ranging pixel 25d that receives sensing light (e.g., infrared rays) emitted to an object present in the environment. The imaging device CS is capable of calculating the distance to the object on the basis of the flight time of sensing light received by the ranging pixel 25d provided in the first substrate 21. The second substrate 22 may be stacked on the surface of the first substrate 21 opposite to the light-receiving surface and receive sensing light (e.g., infrared rays) that has been transmitted through the first substrate 21 by the ranging pixel 25d. By stacking the first substrate 21 that includes the imaging pixel 25i and the second substrate 22 that includes the ranging pixel 25d, the imaging device CS is capable of coaxially performing imaging of an environment and measurement of the distance to an object present in the environment.

[0021] Further, as shown in Fig. 4, the imaging device CS may be SPAD LiDAR that emits sensing light Le to an object 30 present in an environment and receives, by single-photon avalanche diode (SPAD) 23, reflected light Lr reflected by the object 30. The SPAD 23 is a sensor that is capable of detecting reception of light in units of photons. The imaging device CS is capable of acquiring ranging information to the object 30 on the basis of the flight time of the sensing light Le by receiving the reflected light Lr by the SPAD 23. Further, the SPAD 23 receives not only the reflected light Lr but also ambient light Lb from a lighting 31. For this reason, the imaging device CS is also capable of acquiring an image of the object 30 by integrating the histogram of photos received by the SPAD 23 (i.e., the reflected light Lr + the ambient light Lb). In accordance with this, the imaging device CS is capable of coaxially performing imaging of an environment and measurement of the distance to an object present in the environment by changing the temporal histogram operations of photons received by the SPAD 23.

[0022] However, in the imaging device CS, since the imaging region IF and the ranging region DF are the same, power consumption increases in the case where sensing light is emitted to all the regions of the ranging region DF.

[0023] In this regard, the information processing apparatus according to this embodiment controls the emission region where sensing light is emitted, on the basis of the feature point extracted from an image taken by the imaging device CS. In accordance with this, the information processing apparatus is capable of emitting sensing light only to the region where a feature point to be used for processing such as SLAM is estimated to be present, and thus, it is possible to reduce power consumption due to emission of sensing light.

[0024] <2. Configuration of information processing apparatus> Next, the configuration of the information processing apparatus according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing a functional configuration of the information processing apparatus 100 according to this embodiment.

[0025] As shown in Fig. 5, the information processing apparatus 100 includes an image acquisition unit 110, a point cloud acquisition unit 120, a feature point extraction unit 130, a self-position estimation unit 140, and an emission region determination unit 150. Further, a sensor unit 200 includes an image light-receiving unit 210, a ranging light-receiving unit 220, a light-emitting unit 230, and a light-emission control unit 240. The sensor unit 200 is capable of imaging an environment and measuring the distance to an object present in the environment. The information processing apparatus 100 and the sensor unit 200 may be included in a single apparatus or may be included in a system in which the information processing apparatus 100 and the sensor unit 200 are included in different apparatuses.

[0026] (Sensor unit 200) The image light-receiving unit 210 generates an image of an environment by receiving incident light. Specifically, the image light-receiving unit 210 may generate an image of an environment by receiving incident light by each of a plurality of pixels arrayed in a two-dimensional matrix. The image light-receiving unit 210 may be, for example, a complementary metal-oxide-semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, or a single-photon avalanche diode (SPAD) image sensor.

[0027] The ranging light-receiving unit 220 measures the distance to an object present in an environment by receiving reflected light of sensing light emitted from the light-emitting unit 230 to the object. Specifically, the ranging light-receiving unit 220 may measure the distance to an object by measuring the flight time until sensing light emitted from the light-emitting unit 230 is reflected by the object, returns, and received by the pixel. Further, the ranging light-receiving unit 220 may generate point cloud data of an environment on the basis of the measured ranging information to the object.

[0028] Note that the ranging light-receiving unit 220 measures the distance to an object present in an environment coaxially with the image light-receiving unit 210. As a result, arithmetic processing can be performed on the image of an environment generated by the image light-receiving unit 210 and the ranging information to an object present in the environment measured by the ranging light-receiving unit 220, by the information processing apparatus 100 without correction.

[0029] The light-emitting unit 230 emits sensing light to an object present in an environment. Specifically, the light-emitting unit 230 may be a laser light source that emits a laser beam on the basis of control from the light-emission control unit 240. The light-emitting unit 230 may be, for example, a laser light source that emits infrared rays.

[0030] The light-emission control unit 240 controls the light-emitting unit 230 such that sensing light is emitted to the emission region input from the information processing apparatus 100. Specifically, the light-emission control unit 240 may control the light-emitting unit 230 such that sensing light is emitted to the emission region where a feature point is estimated to be present by the information processing apparatus 100. Details of generation of the emission region by the information processing apparatus 100 will be described below.

[0031] The above sensor unit 200 may be included in a moving object such as a trolley, a drone, or an automobile. In accordance with this, a moving object such as a trolley, a drone, or an automobile is capable of performing automated movement or maneuvering support with higher accuracy by estimating the self-position on the basis of the image and ranging information obtained by the sensor unit 200.

[0032] Further, the sensor unit 200 may be included in a device such as a smartphone or a head mounted display (HMD). In accordance with this, the device such as a smartphone or an HMD is capable of superimposing a virtual space of augmented reality (AR) on a real space with higher accuracy by estimating the self-position on the basis of the image and ranging information obtained by the sensor unit 200. Therefore, these devices are capable of providing augmented reality (AR) with higher quality to users.

[0033] (Information processing apparatus 100) The image acquisition unit 110 acquires an image of an environment from the sensor unit 200. Specifically, the image acquisition unit 110 may be a connection port that acquires an image of an environment from the sensor unit 200. Further, the image acquisition unit 110 may acquire a received light signal from the image light-receiving unit 210 and generate an image of an environment on the basis of the acquired received light signal.

[0034] The point cloud acquisition unit 120 acquires point cloud data of an environment from the sensor unit 200. Specifically, the point cloud acquisition unit 120 may be a connection port that acquires point cloud data of an environment from the sensor unit 200. Further, the point cloud acquisition unit 120 may acquire ranging information to an object present in an environment from the ranging light-receiving unit 220 and generate point cloud data of an environment on the basis of the acquired ranging information.

[0035] The feature point extraction unit 130 extracts a feature point from an image on the basis of an image of an environment acquired by the image acquisition unit 110 and point cloud data of an environment acquired by the point cloud acquisition unit 120. Specifically, first, the feature point extraction unit 130 integrates a captured image and point cloud data to generate an RGB-D image that includes color data and depth data. Next, the feature point extraction unit 130 may extract a feature point from an image by applying a known feature point extraction algorithm to the generated RGB-D image. Examples of the feature point extraction algorithm include scale-invariant feature transform (SIFT), speeded up robust features (SURF), features from accelerated segment test (FAST), maximally stable extremal regions (MSER), and Harris operator.

[0036] The self-position estimation unit 140 estimates the self-position of the sensor unit 200 using the feature point extracted by the feature point extraction unit 130. Specifically, the self-position estimation unit 140 may estimate the self-position of the sensor unit 200 by tracking the position and depth of the feature points extracted from the image between frames of the image. For example, the self-position estimation unit 140 may estimate the change in the self-position of the sensor unit 200 by identifying the corresponding feature point between the frames of the image and tracking the change in the position and depth of the corresponding feature point between the respective frames. As a detailed method of estimating the change in the self-position from the change in the position and depth of the feature point, a known SLAM algorithm can be used.

[0037] The emission region determination unit 150 determines an emission region where sensing light is emitted from the light-emitting unit 230, on the basis of the position of the feature point of the image. The method of determining an emission region ER by the emission region determination unit 150 will be described in detail with reference to Fig. 6. Fig. 6 is a schematic diagram describing the method of determining the emission region ER.

[0038] As shown in Fig. 6, first, the emission region determination unit 150 acquires a feature point FP1 in the current frame of the image extracted by the feature point extraction unit 130. Next, the emission region determination unit 150 estimates the position of a feature point FP2 in a next frame FV2 of the image on the basis of the feature point FP1 in the current frame of the image and the self-position of the sensor unit 200 estimated by the self-position estimation unit 140. Subsequently, the emission region determination unit 150 determines the emission region ER having a predetermined size in the peripheral region around the position of the feature point FP2 in the next frame FV2 of the image. As shown in Fig. 6, the shape of the emission region ER may be, for example, a rectangular shape, a circular shape, or an elliptical shape.

[0039] In accordance with this, the light-emission control unit 240 of the sensor unit 200 is capable of controlling the light-emitting unit 230 such that sensing light is emitted only to the emission region ER where a feature point is estimated to be present in the next frame. Therefore, the information processing apparatus 100 is capable of reducing power consumption by the sensor unit 200 because the region where sensing light is emitted from the light-emitting unit 230 can be limited. Meanwhile, since the sensor unit 200 is capable of increasing the intensity of sensing light to be emitted to the emission region ER with reduced power consumption, it is possible to widen the detection range of the depth of the feature point.

[0040] For example, in the case where the device that includes the sensor unit 200 is a trolley, the automatedly moving trolley is desired to estimate the self-position with high accuracy in delivery of articles or patrolling the site. Further, the automatedly moving trolley is desired to estimate the self-position with lower power consumption in order to move for a long time. In accordance with the information processing apparatus 100 according to this embodiment, the above trolley is capable of estimating the self-position with lower power consumption and higher accuracy while maintaining the moving time.

[0041] For example, in the case where the device that includes the sensor unit 200 is a drone, the automatedly flying drone is desired to estimate the self-position with high accuracy when performing hovering flights or waypoint flights. Further, the automatedly flying drone is desired to estimate the self-position with lower power consumption in order to increase the flight time. In accordance with the information processing apparatus 100 according to this embodiment, the above drone is capable of realizing automated flight with higher accuracy while maintaining the flight time.

[0042] For example, in the case where the device that includes the sensor unit 200 is a vehicle, a large number of sensors are mounted on the vehicle that performs automated driving or driving assistance in order to ensure safety. For this reason, each of the sensors mounted on the vehicle is desired to have lower power consumption in order to suppress battery consumption and improve fuel efficiency. In accordance with the information processing apparatus 100 according to this embodiment, the above vehicle is capable of estimating the self-position by SLAM or the like with lower power consumption.

[0043] For example, in the case where the device that includes the sensor unit 200 is a portable device such as a smartphone, the portable device that realizes augmented reality (AR) is desired to estimate the self-position with high accuracy in order to suppress the superimposition gap between the virtual space and the real space. Further, the portable device that realizes AR is desired to estimate the self-position with lower power consumption in order to increase the operating time of the battery and suppress the heat generation of the battery. In accordance with the information processing apparatus 100 according to this embodiment, the above portable device is capable of estimating the self-position with lower power consumption and higher accuracy and thus suppressing the heat generation of the battery.

[0044] For example, in the case where the device that includes the sensor unit 200 is a head mounted display (HMD), the HMD that realizes virtual reality (VR) is desired to estimate the self-position with high accuracy in order to suppress deviation between the movement of the head wearing the HMD and the estimated self-position. Further, the HMD that realizes VR is desired to suppress heat generation of the battery, because it is worn on the head. In accordance with the information processing apparatus 100 according to this embodiment, the above HMD is capable of estimating the self-position with lower power consumption and higher accuracy and thus suppressing heat generation of the battery.

[0045] <3. Operation example> Next, operations of the information processing apparatus 100 and the sensor unit 200 according to this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing a flow of operations the information processing apparatus 100 and the sensor unit 200 according to this embodiment.

[0046] As shown in Fig. 7, first, the light-emission control unit 240 determines whether or not an emission region has been determined by the emission region determination unit 150 (S101).

[0047] In the case where the position of a feature point has not been estimated and an emission region has not been determined by the emission region determination unit 150 (S101 / NO), the light-emission control unit 240 controls the light-emitting unit 230 such that sensing light is emitted at the entire angle of view of the image in the next frame (S102). Meanwhile, in the case where the position of a feature point has been estimated and an emission region has been determined with the feature point as the center thereof by the emission region determination unit 150 (S101 / YES), the light-emission control unit 240 controls the light-emitting unit 230 such that sensing light is emitted only to the determined emission region (S103).

[0048] Next, reflected light of the emitted sensing light is received by the ranging light-receiving unit 220, and thus, point cloud data of an environment is acquired by the point cloud acquisition unit 120 (S105). Subsequently, the feature point extraction unit 130 extracts a feature point from the point cloud data of an environment acquired by the point cloud acquisition unit 120 and the image of the environment acquired by the image acquisition unit 110 (S107).

[0049] Subsequently, the self-position estimation unit 140 determines whether or not depth data of the extracted feature point has been obtained (S109). That is, the self-position estimation unit 140 determines whether or not sensing light has been emitted to the extracted feature point and the distance to the feature point has been measured.

[0050] In the case where depth data of the feature point has been obtained (S109 / YES), the self-position estimation unit 140 estimates the self-position of the sensor unit 200 on the basis of the feature point (S111). Specifically, the self-position estimation unit 140 estimates the self-position of the sensor unit 200 by tracking the position of the corresponding feature point and the change in the depth between the respective frames of the image. Meanwhile, in the case where depth data of a feature point has not been obtained (S109 / NO), the self-position estimation unit 140 skips estimation of the self-position of the sensor unit 200.

[0051] After that, the emission region determination unit 150 estimates the position of the feature point in the next frame on the basis of the feature point in the current frame of the image and determines, as an emission region, a peripheral region with the estimated position of the feature point as the center (S113).

[0052] In accordance with the above operation, the information processing apparatus 100 is capable of determining an emission region around the estimated feature point and causes the light-emitting unit 230 to emit sensing light only to an emission region where a feature point is estimated to be present. Therefore, the information processing apparatus 100 makes it possible to reduce power consumption of the sensor unit 200.

[0053] <4. Modified example> (4.1. First modified example) The information processing apparatus 100 according to a first modified example will be described with reference to Fig. 8. Fig. 8 is a flowchart showing a flow of operations of the information processing apparatus 100 and the sensor unit 200 according to the first modified example. In the first modified example, in the case where an emission region has not been determined, emission of sensing light is skipped instead of emitting sensing light at the entire angle of view of the image.

[0054] For example, in the case where an emission region has not been determined by the emission region determination unit 150 due to initialization of the information processing apparatus 100, or the like, the light-emission control unit 240 controls the light-emitting unit 230 not to emit sensing light. At this time, since point cloud data of an environment has not been obtained, as shown in Fig. 8, the feature point extraction unit 130 extracts a feature point from only the image of the environment acquired by the image acquisition unit 110 (S201).

[0055] Subsequently, the emission region determination unit 150 estimates the position of the feature point in the next frame of the image on the basis of the extracted feature point in the current frame of the image (S203). The emission region determination unit 150 determines whether or not the position of the feature point in the next frame has been estimated (S205). In the case where the position of the feature point in the next frame has been estimated (S205 / YES), the emission region determination unit 150 determines, as an emission region, a peripheral region with the estimated position of the feature point as the center (S207).

[0056] Meanwhile, in the case where the position of the feature point in the next frame of the image has not been estimated (S205 / NO), the emission region determination unit 150 does not determine an emission region. For this reason, the light-emission control unit 240 controls the light-emitting unit 230 not to emit sensing light also in the next frame.

[0057] The information processing apparatus 100 according to the first modified example is capable of performing control such that sensing light is not emitted at the entire angle of view of the image in the case where an emission region has not been determined by the emission region determination unit 150. Therefore, the information processing apparatus 100 according to the first modified example is capable of further reducing power consumption of the sensor unit 200.

[0058] (4.2. Second modified example) The information processing apparatus 100 according to a second modified example will be described with reference to Fig. 9A to Fig. 11. Fig. 9A and Fig. 9B are each an explanatory diagram describing the observation accuracy of a feature point FP relative to the distance from the sensor unit 200. Fig. 10A and Fig. 10B are each a schematic diagram showing the change in the size of the emission region in the second modified example. Fig. 11 is a flowchart showing a flow of operations of the information processing apparatus 100 and the sensor unit 200 according to the second modified example.

[0059] The observation accuracy (particularly, the observation accuracy of the depth) of the feature point FP by the sensor unit 200 is affected by the distance between the sensor unit 200 and the feature point FP.

[0060] For example, as shown in Fig. 9A, regarding the feature point FP where the distance from the sensor unit 200 is short, since the observation accuracy is high, the position of the feature point FP in the next frame is estimated with high accuracy. As a result, a possible region OB where the feature point FP is estimated to be present in the next frame is relatively narrow. Meanwhile, as shown in Fig. 9B, regarding the feature point FP where the distance from the sensor unit 200 is long, since the observation accuracy is low, the position of the feature point FP in the next frame is estimated with low accuracy. As a result, the possible region OB where the feature point FP is estimated to be present in the next frame is relatively wide. That is, since the observation accuracy (particularly, the observation accuracy of the depth) of the feature point FP is lower as the distance from the sensor unit 200 increases, the estimation accuracy of the position of the feature point FP in the next frame is also lower.

[0061] In the second modified example, the emission region determination unit 150 changes the size of the emission region where sensing light is emitted, in accordance with the distance between the sensor unit 200 and the feature point FP. Specifically, as shown in Fig. 10A, in the case where the distance between the sensor unit 200 and the feature point FP is short, the emission region determination unit 150 makes the size of an emission region ERn with the feature point FP as the center smaller. Meanwhile, as shown in Fig. 10B, in the case where the distance between the sensor unit 200 and the feature point FP is long, the emission region determination unit 150 makes the size of an emission region ERf with the feature point FP as the center larger. In accordance with this, the information processing apparatus 100 is capable of further increasing the possibility that the feature point FP is included in the determined emission region.

[0062] A flow of operations of the information processing apparatus 100 and the sensor unit 200 according to the second modified example will be described with reference to Fig. 11.

[0063] As shown in Fig. 11, first, the feature point extraction unit 130 extracts a feature point from point cloud data of an environment acquired by the point cloud acquisition unit 120 and an image of the environment acquired by the image acquisition unit 110 (S301).

[0064] Next, the self-position estimation unit 140 determines whether or not depth data of the extracted feature point has been obtained (S303). That is, the self-position estimation unit 140 determines whether or not sensing light has been emitted to the extracted feature point and the distance to the feature point has been measured.

[0065] In the case where depth data of the feature point has been obtained (S303 / YES), the self-position estimation unit 140 estimates the self-position of the sensor unit 200 on the basis of the feature point (S305). Specifically, the self-position estimation unit 140 estimates the self-position of the sensor unit 200 by tracking the position of the corresponding feature point and the change in the depth between the respective frames of the image.

[0066] Subsequently, the emission region determination unit 150 estimates the position of the feature point in the next frame on the basis of the feature point in the current frame of the image and determines, as an emission region, a peripheral region with the estimated position of the feature point as the center (S307). At this time, the emission region determination unit 150 changes the size of the emission region on the basis of the size of the depth of the estimated feature point. For example, the emission region determination unit 150 may makes the size of the emission region larger as the depth of the estimated feature point is larger (i.e., the distance between the estimated feature point and the sensor unit 200 is longer).

[0067] Meanwhile, in the case where depth data of the feature point has not been obtained (S303 / NO), the self-position estimation unit 140 estimates the position of the feature point in the next frame on the basis of the feature point in the current frame of the image and determines, as an emission region, a peripheral region with the estimated position of the feature point as the center (S308). At this time, the emission region determination unit 150 may set the size of the emission region to a predetermined size.

[0068] After that, the light-emission control unit 240 controls the light-emitting unit 230 to emit sensing light only to the determined emission region (S309).

[0069] The information processing apparatus 100 according to the second modified example is capable of making the size of the emission region where sensing light is emitted larger, considering that the observation accuracy of the feature point is lower in the case where the distance between the sensor unit 200 and the feature point is long. In accordance with this, the information processing apparatus 100 according to the second modified example is capable of causing the sensor unit 200 to emit light to the feature point more reliably and thus acquiring depth data of the feature point more reliably.

[0070] (4.3. Third modified example) An information processing apparatus 101 according to a third modified example will be described with reference to Fig. 12 to Fig. 14. Fig. 12 is a block diagram showing a functional configuration of the information processing apparatus 101 according to the third modified example. Fig. 13 is an explanatory diagram describing a method of estimating the position of the feature point FP from the position of the sensor unit 200. Fig. 14 is a flowchart showing a flow of operations of the information processing apparatus 101 and the sensor unit 200 according to the third modified example.

[0071] As shown in Fig. 12, the information processing apparatus 101 includes the image acquisition unit 110, the point cloud acquisition unit 120, the feature point extraction unit 130, the self-position estimation unit 140, the emission region determination unit 150, and a position acquisition unit 160. The information processing apparatus 101 according to the third modified example further includes the position acquisition unit 160 as compared with the information processing apparatus 100 shown in Fig. 5. The information processing apparatus 101 and the sensor unit 200 may be included in a single apparatus or may be included in a system in which the information processing apparatus 101 and the sensor unit 200 are included in different apparatuses.

[0072] The position acquisition unit 160 detects information regarding the position and posture of the sensor unit 200. For example, the position acquisition unit 160 may be an inertial measurement unit (IMU) provided in the device that includes the sensor unit 200 or in the sensor unit 200 itself.

[0073] The information regarding the position and posture of the sensor unit 200 detected by the position acquisition unit 160 is used to estimate the self-position of the sensor unit 200 by the self-position estimation unit 140. The self-position estimation unit 140 is capable of estimating the self-position of the sensor unit 200 with higher accuracy by further using the information regarding the position and posture of the sensor unit 200 detected by the position acquisition unit 160.

[0074] As shown in Fig. 13, first, the emission region determination unit 150 projects the feature point FP onto a three-dimensional space on the basis of the information regarding the position and posture of the sensor unit 200 in the current frame and the position and depth of the feature point FP in the current frame of the image. Next, the emission region determination unit 150 estimates the self-position of the sensor unit 200 in the next frame on the basis of the information regarding the position and posture of the sensor unit 200. Subsequently, the emission region determination unit 150 is capable of estimating the position of the feature point FP in the next frame of the image on the basis of the position of the feature point FP projected onto the three-dimensional space in the current frame and the self-position of the sensor unit 200 in the next frame.

[0075] In accordance with this, the emission region determination unit 150 is capable of estimating the position of the feature point FP in the next frame of the image with higher accuracy. Therefore, the emission region determination unit 150 is capable of determining a smaller emission region and thus reducing power consumption of the sensor unit 200.

[0076] Further, the emission region determination unit 150 may change the size of the emission region on the basis of the depth of the estimated feature point FP, similarly to the second modified example. That is, the emission region determination unit 150 is capable of making the size of the emission region where sensing light is emitted smaller as the distance between the sensor unit 200 and the feature point FP is short.

[0077] A flow of operations of the information processing apparatus 101 and the sensor unit 200 according to the third modified example will be described with reference to Fig. 14.

[0078] As shown in Fig. 14, first, the emission region determination unit 150 projects the feature point in the current frame onto a three-dimensional space on the basis of the self-position of the sensor unit 200 in the current frame and the position and depth of the feature point in the current frame of the image (S401). Next, the self-position estimation unit 140 estimates the self-position of the sensor unit 200 in the next frame on the basis of the information regarding the position and posture of the sensor unit 200 acquired by the position acquisition unit 160 (S403).

[0079] Subsequently, the emission region determination unit 150 estimates the position of the feature point in the next frame of the image on the basis of the position of the feature point projected onto the three-dimensional space in the current frame and the self-position of the sensor unit 200 in the next frame (S405). Further, the emission region determination unit 150 determines, as an emission region, a peripheral region with the estimated position of the feature point as the center (S407). At this time, the emission region determination unit 150 may change the size of the emission region on the basis of the size of the depth of the estimated feature point. For example, the emission region determination unit 150 may make the size of the emission region larger as the depth of the estimated feature point is larger (i.e., the distance between the estimated feature point and the sensor unit 200 is longer).

[0080] After that, the light-emission control unit 240 controls the light-emitting unit 230 to emit sensing light only to the determined emission region (S409).

[0081] The information processing apparatus 101 according to the third modified example is capable of estimating the position of the feature point with higher accuracy by further using the information regarding the position and posture of the sensor unit 200. In accordance with, the information processing apparatus 101 according to the third modified example is capable of making the emission region smaller and thus further reducing power consumption of the sensor unit 200.

[0082] (4.4. Fourth modified example) An information processing apparatus 102 according to a fourth modified example will be described with reference to Fig. 15 to Fig. 17. Fig. 15 is a block diagram showing a functional configuration of the information processing apparatus 102 according to the fourth modified example. Fig. 16 is a graph diagram showing an example of a photon count histogram in the case where the sensor unit 200 is SPAD LiDAR. Fig. 17 is a flowchart showing a flow of operations of the information processing apparatus 102 and the sensor unit 200 according to the fourth modified example.

[0083] As shown in Fig. 15, the information processing apparatus 102 includes the image acquisition unit 110, the point cloud acquisition unit 120, the feature point extraction unit 130, the self-position estimation unit 140, the emission region determination unit 150, and a covariance estimation unit 170. The information processing apparatus 102 according to the fourth modified example further includes the covariance estimation unit 170 as compared with the information processing apparatus 100 shown in Fig. 5. The information processing apparatus 102 and the sensor unit 200 may be included in a single apparatus or may be included in a system in which the information processing apparatus 102 and the sensor unit 200 are included in different apparatuses.

[0084] Further, in the fourth modified example, the sensor unit 200 acquires an image of an environment and ranging information by SPAD LiDAR. Specifically, the sensor unit 200 acquires an image of an environment and ranging information to an object present in the environment by arithmetic processing of the photon count histogram acquired by SPAD LiDAR.

[0085] The covariance estimation unit 170 estimates the covariance of the peak in the histogram on the basis of the photon count histogram acquired by SPAD LiDAR of the sensor unit 200. The covariance of the peak is a value corresponding to a region where the observation point corresponding to the peak is likely to be actually present.

[0086] Specifically, the photon count histogram of SPAD LiDAR is acquired as, for example, the graph diagram shown in Fig. 16. As shown in Fig. 4, in the photon count histogram of SPAD LiDAR, the ambient light Lb from the lighting 31 or the like is present as a count of the background, and the reflected light Lr of the sensing light Le is present as a peak P. The covariance estimation unit 170 is capable of estimating the covariance of the peak on the basis of the width of the peak P or the S / N ratio of the peak P and the background.

[0087] In the fourth modified example, the emission region determination unit 150 changes the size of the emission region on the basis of the estimated covariance of the peak (observation point). For example, the emission region determination unit 150 may make the emission region smaller as the estimated covariance of the peak (observation point) is smaller. In accordance with this, the information processing apparatus 102 is capable of causing the sensor unit 200 to more reliably emit sensing light to the feature point and thus acquiring depth data of the feature point more reliably.

[0088] Further, the variance of the peak (observation point) estimated by the covariance estimation unit 170 can also be used to estimate the covariance of the self-position of the sensor unit 200 by the self-position estimation unit 140. The covariance of the estimation result indicates the reliability of estimation. For this reason, the self-position estimation unit 140 is capable of using the covariance as an index of the reliability of estimation when combining estimation of a plurality of self-positions.

[0089] A flow of operations of the information processing apparatus 102 and the sensor unit 200 according to the fourth modified example will be described with reference to Fig. 17.

[0090] As shown in Fig. 17, first, the feature point extraction unit 130 extracts a feature point from the photon count histogram of SPAD LiDAR (S501). Next, the covariance estimation unit 170 estimates the covariance of the peak from the photon count histogram of SPAD LiDAR (S503).

[0091] Subsequently, the self-position estimation unit 140 estimates the self-position of the sensor unit 200 on the basis of the extracted feature point (S505). Further, the self-position estimation unit 140 may estimate the covariance of the self-position of the sensor unit 200 using the covariance of the peak (S507).

[0092] Next, the emission region determination unit 150 estimates the position of the feature point in the next frame on the basis of the feature point in the current frame of the image and determines, as an emission region, a peripheral region with the estimated position of the feature point as the center (S509). At this time, the emission region determination unit 150 changes the size of the emission region on the basis of the size of the estimated covariance of the peak. For example, the emission region determination unit 150 may make the size of the emission region larger as the covariance of the peak is larger (i.e., the region where the observation point corresponding to the peak is likely to be actually present is larger).

[0093] After that, the light-emission control unit 240 controls the light-emitting unit 230 to emit sensing light only to the determined emission region (S511).

[0094] The information processing apparatus 102 according to the fourth modified example is capable of changing the size of the emission region where sensing light is emitted, on the basis of the covariance of the peak in the photon count histogram. In accordance with this, the information processing apparatus 102 according to the fourth modified example is capable of causing the sensor unit 200 to more reliably emit sensing light to the feature point and thus acquiring depth data of the feature point more reliably.

[0095] <5. Hardware configuration example> A hardware configuration of a computer 900 that is hardware for realizing the information processing apparatuses 100, 101, and 102 according to this embodiment will be described with reference to Fig. 18. Fig. 18 is a block diagram showing a configuration example of the computer 900.

[0096] The function of the computer 900 can be realized by cooperation of software and hardware described below. The functions of the feature point extraction unit 130, the self-position estimation unit 140, the emission region determination unit 150, and the covariance estimation unit 170 may be executed by, for example, the CPU 901. The functions of the image acquisition unit 110 and the point cloud acquisition unit 120 may be executed by, for example, the connection port 910 or the communication device 911.

[0097] As shown in Fig. 18, the computer 900 includes a central processing unit (CPU) 901, a read only memory (ROM) 902, and a random access memory (RAM) 903.

[0098] The computer 900 may further include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 910, and a communication device 911. The computer 900 may include a processing circuit such as a digital signal processor (DSP) and an application specific integrated circuit (ASIC) instead of or in addition to the CPU 901.

[0099] The CPU 901 functions as an arithmetic processing unit or control device and controls the operation in the computer 900 in accordance with various programs recorded on the ROM 902, the RAM 903, the storage device 908, or a removable recording medium attached to the drive 909. The ROM 902 stores a program, an operation parameter, and the like to be used by the CPU 901. The RAM 903 temporarily stores a program to be used in execution of the CPU 901, a parameter to be used at the time of the execution, and the like.

[0100] The CPU 901, the ROM 902, and the RAM 903 are connected to each other via the host bus 904a that is capable of performing high-speed data transmission. The host bus 904a is connected to the external bus 904b such as a peripheral component interconnect / interface (PCI) bus via the bridge 904. The external bus 904b is connected to various components via the interface 905.

[0101] The input device 906 is, for example, a device that receives an input from a user, such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. Note that the input device 906 may be a microphone that detects voice of a user, or the like. The input device 906 may be, for example, a remote control device that uses infrared rays or other radio waves, and may be an external connection device that is compatible with the operation of the computer 900.

[0102] The input device 906 further includes an input control circuit that outputs the input signal generated on the basis of information input by a user to the CPU 901. The user can give instructions for inputting various types of data or processing operation to the computer 900 by operating the input device 906.

[0103] The output device 907 is a device that is capable of visually or audibly presenting the information acquired or generated by the computer 900 to a user. The output device 907 may be, for example, a display device such as a liquid crystal display (LCD), a plasma display panel (PDP), an organic light emitting diode (OLED) display, a hologram, and a projector, a sound output device such as a speaker and a headphone, or a print device such as a printer device. The output device 907 is capable of outputting, as video such as text and an image or a sound such as audio and acoustics, the information acquired by the processing of the computer 900.

[0104] The storage device 908 is a data storage device configured as an example of the storage unit of the computer 900. The storage device 908 may include, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 is capable of storing programs to be executed by the CPU 901, various types of data, various types of data acquired from the outside, and the like.

[0105] The drive 909 is a reading or writing device for a removable recording medium such as a magnetic disc, an optical disc, a magneto-optical disc, and a semiconductor memory, and is built into the computer 900 or attached externally. For example, the drive 909 is capable of reading the information recorded on the attached removable recording medium and outputting the read information to the RAM 903. Further, the drive 909 is capable of writing information to the attached removable recording medium.

[0106] The connection port 910 is a port for directly connecting an external connection device to the computer 900. The connection port 910 may be, for example, a universal serial bus (USB) port, an IEEE1394 port, a small computer system interface (SCSI) port, or the like. Further, the connection port 910 may be an RS-232C port, an optical audio terminal, a high-definition multimedia interface (HDMI) (registered trademark) port, or the like. The connection port 910 is capable of transmitting / receiving various types of data between the computer 900 and the external connection device by being connected to the external connection device.

[0107] The communication device 911 is, for example, a communication interface configured by a communication device for connecting to a communication network 920. The communication device 911 may be, for example, a wired or wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), or a communication card for wireless USB (WUSB). Further, the communication device 911 may be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), a modem for various types of communication, or the like.

[0108] The communication device 911 is capable of transmitting / receiving signals and the like to / from, for example, the Internet or another communication device using a predetermined protocol such as TCP / IP. Further, the communication network 920 connected to the communication device 911 is a wired or wirelessly connected network and may be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0109] Note that a program for performing functions equivalent to those of the information processing apparatuses 100, 101, and 102 may be created in hardware such as the CPU 901, the ROM 902, and the RAM 903 built in the computer 900. Further, a computer-readable recording medium recoding the program may be provided.

[0110] Although an embodiment of the present disclosure has been described above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such an example. It is clear that a person with ordinary knowledge in the technical field of the present disclosure can conceive various changes or modifications within the scope of the technical idea described in the claims, and it is understood that these also fall within the technical scope of the present disclosure.

[0111] Further, the effects described in the present specification are merely explanatory or illustrative, and are not limitative. That is, the technology according to the present disclosure can have other effects that are obvious to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

[0112] Note that the following configurations also belong to the technical scope of the present disclosure. (1) An information processing apparatus, including: a feature point extraction unit that extracts a feature point from an image of an environment using the image and point cloud data of the environment derived on the basis of reflected light of sensing light emitted to the environment; and an emission region determination unit that determines, on the basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame. (2) The information processing apparatus according to (1) above, further including a self-position estimation unit that estimates, on the basis of the feature point, a self-position of a device on which an imaging device and a ranging sensor are mounted, the imaging device acquiring the image, the ranging sensor acquiring the point cloud data. (3) The information processing apparatus according to (1) or (2) above, in which the position of the feature point in the next frame is estimated on the basis of the position of the feature point in the current frame and the self-position. (4) The information processing apparatus according to any one of (1) to (3) above, in which the self-position estimation unit further estimates a covariance of the self-position on the basis of a covariance of an observation result of the reflected light. (5) The information processing apparatus according to any one of (1) to (4) above, in which the self-position is estimated further on the basis of a sensing result of an inertial measurement device mounted on the device. (6) The information processing apparatus according to any one of (1) to (5) above, in which the self-position estimation unit estimates the self-position by SLAM using the feature point. (7) The information processing apparatus according to any one of (1) to (6) above, in which the ranging sensor is an RGBD coaxial sensor integrated with the imaging device. (8) The information processing apparatus according to any one of (1) to (7) above, in which the emission region determination unit determines, where the position of the feature point in the next frame of the image has not been estimated, an entire sensing region of the ranging sensor as the emission region. (9) The information processing apparatus according to any one of (1) to (8) above, in which the feature point extraction unit extracts, where the point cloud data has not been acquired, the feature point from the image using only the image. (10) The information processing apparatus according to any one of (1) to (9) above, in which the emission region determination unit determines, as the emission region, a peripheral region around the position of the feature point in the next frame. (11) The information processing apparatus according to any one of (1) to (10) above, in which the emission region determination unit controls a size of the peripheral region on the basis of a distance to the feature point in the current frame. (12) The information processing apparatus according to any one of (1) to (11) above, in which the emission region determination unit controls a size of the peripheral region on the basis of a covariance of an observation result of the reflected light. (13) An information processing method executed by a computer, including: extracting a feature point from an image of an environment using the image and point cloud data of the environment derived on the basis of reflected light of sensing light emitted to the environment; and determining, on the basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame. (14) A program that causes a computer to function as: a feature point extraction unit that extracts a feature point from an image of an environment using the image and point cloud data of the environment derived on the basis of reflected light of sensing light emitted to the environment; and an emission region determination unit that determines, on the basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame. (15) An information processing system, comprising: a feature point extraction unit configured to extract a feature point from an image of an environment using the image or point cloud data of the environment; and an emission region determination unit configured to determine, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame, wherein the feature point extraction unit and the emission region determination unit are each implemented via at least one processor. (16) The information processing system according to (15), further comprising a self-position estimation unit configured to estimate, on a basis of the feature point, a self-position of a device on which an imaging device and a ranging sensor are mounted, the imaging device acquiring the image, the ranging sensor acquiring the point cloud data, wherein the self-position estimation unit is implemented via at least one processor. (17) The information processing system according to any one of (15) or (16), wherein the position of the feature point in the next frame is estimated on a basis of the position of the feature point in the current frame and the self-position. (18) The information processing system according to any one of (15) to (17), wherein the self-position estimation unit is further configured to estimate a covariance of the self-position on a basis of a covariance of an observation result of the reflected light. (19) The information processing system according to any one of (15) to (18), wherein the self-position is estimated further on a basis of a sensing result of an inertial measurement device mounted on the device. (20) The information processing system according to any one of (15) to (19), wherein the self-position estimation unit is further configured to estimate the self-position by SLAM using the feature point. (21) The information processing system according to any one of (15) to (20), wherein the ranging sensor is an RGBD coaxial sensor integrated with the imaging device. (22) The information processing system according to any one of (15) to (21), wherein the emission region determination unit is further configured to determine, where the position of the feature point in the next frame of the image has not been estimated, an entire sensing region of the ranging sensor as the emission region. (23) The information processing system according to any of (15) to (22), wherein the feature point extraction unit is further configured to extract, where the point cloud data has not been acquired, the feature point from the image using only the image. (24) The information processing system according to any one of (15) to (23), wherein the emission region determination unit is further configured to determine, as the emission region, a peripheral region around the position of the feature point in the next frame. (25) The information processing system according to any one of (15) to (24), wherein the emission region determination unit is further configured to control a size of the peripheral region on a basis of a distance to the feature point in the current frame. (26) The information processing system according to any one of (15) to (25), wherein the emission region determination unit is further configured to control a size of the peripheral region on a basis of a covariance of an observation result of the reflected light. (27) An information processing method executed by a computer, comprising: extracting a feature point from an image of an environment using the image or point cloud data of the environment; and determining, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame. (28) A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method comprising: extracting a feature point from an image of an environment using the image or point cloud data of the environment; and determining, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame. (29) The information processing system according to (15) to (26), wherein the point cloud data of the environment is derived on a basis of reflected light of sensing light emitted to the environment. (30) The information processing system according to any one of (15) to (26) or (29), further comprising an image light-receiving unit configured to generate an image of the environment, wherein the image light receiving unit is implemented via at least one processor. (31) The information processing system according to any one of (15) to (26), (29), or (30), further comprising a ranging light-receiving unit configured to measure a distance to an object present in the environment, wherein the ranging light receiving unit is implemented via at least one processor. (32) The information processing system according to any one of (15) to (26) or (29) to (31), further comprising a light source configured to emit sensing light to an object present in the environment. (33) The information processing system according to any one of (15) to (26) or (29) to (32), further comprising a light-emission control unit configured to control the light source, wherein the light-emission control unit is implemented via at least one processor. (34) The information processing system according to any one of (15) to (26) or (29) to (33), further comprising an information processing apparatus comprising the feature point extraction unit and the emission region determination unit, and a sensor device configured to image the environment and measure a distance to an object included in the environment.

[0113] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0114] 100,101,102 information processing apparatus 110 image acquisition unit 120 point cloud acquisition unit 130 feature point extraction unit 140 self-position estimation unit 150 emission region determination unit 160 position acquisition unit 170 covariance estimation unit 200 sensor unit 210 image light-receiving unit 220 ranging light-receiving unit 230 light-emitting unit 240 light-emission control unit

Claims

1. An information processing system, comprising: a feature point extraction unit configured to extract a feature point from an image of an environment using the image or point cloud data of the environment; and an emission region determination unit configured to determine, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame, wherein the feature point extraction unit and the emission region determination unit are each implemented via at least one processor.

2. The information processing system according to claim 1, further comprising a self-position estimation unit configured to estimate, on a basis of the feature point, a self-position of a device on which an imaging device and a ranging sensor are mounted, the imaging device acquiring the image, the ranging sensor acquiring the point cloud data, wherein the self-position estimation unit is implemented via at least one processor.

3. The information processing system according to claim 2, wherein the position of the feature point in the next frame is estimated on a basis of the position of the feature point in the current frame and the self-position.

4. The information processing system according to claim 2, wherein the self-position estimation unit is further configured to estimate a covariance of the self-position on a basis of a covariance of an observation result of the reflected light.

5. The information processing system according to claim 2, wherein the self-position is estimated further on a basis of a sensing result of an inertial measurement device mounted on the device.

6. The information processing system according to claim 2, wherein the self-position estimation unit is further configured to estimate the self-position by SLAM using the feature point.

7. The information processing system according to claim 2, wherein the ranging sensor is an RGBD coaxial sensor integrated with the imaging device.

8. The information processing system according to claim 2, wherein the emission region determination unit is further configured to determine, where the position of the feature point in the next frame of the image has not been estimated, an entire sensing region of the ranging sensor as the emission region.

9. The information processing system according to claim 1, wherein the feature point extraction unit is further configured to extract, where the point cloud data has not been acquired, the feature point from the image using only the image.

10. The information processing system according to claim 1, wherein the emission region determination unit is further configured to determine, as the emission region, a peripheral region around the position of the feature point in the next frame.

11. The information processing system according to claim 10, wherein the emission region determination unit is further configured to control a size of the peripheral region on a basis of a distance to the feature point in the current frame.

12. The information processing system according to claim 10, wherein the emission region determination unit is further configured to control a size of the peripheral region on a basis of a covariance of an observation result of the reflected light.

13. An information processing method executed by a computer, comprising: extracting a feature point from an image of an environment using the image or point cloud data of the environment; and determining, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame.

14. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method comprising: extracting a feature point from an image of an environment using the image or point cloud data of the environment; and determining, on a basis of a position of the feature point in a next frame of the image estimated from a position of the feature point in a current frame of the image, an emission region in the environment to be irradiated with the sensing light during the next frame.

15. The information processing system according to claim 1, wherein the point cloud data of the environment is derived on a basis of reflected light of sensing light emitted to the environment.

16. The information processing system according to claim 1, further comprising an image light-receiving unit configured to generate an image of the environment, wherein the image light receiving unit is implemented via at least one processor.

17. The information processing system according to claim 1, further comprising a ranging light-receiving unit configured to measure a distance to an object present in the environment, wherein the ranging light receiving unit is implemented via at least one processor.

18. The information processing system according to claim 1, further comprising a light source configured to emit sensing light to an object present in the environment.

19. The information processing system according to claim 18, further comprising a light-emission control unit configured to control the light source, wherein the light-emission control unit is implemented via at least one processor.

20. The information processing system according to claim 1, further comprising an information processing apparatus comprising the feature point extraction unit and the emission region determination unit, and a sensor device configured to image the environment and measure a distance to an object included in the environment.

Images