Information processing device, information processing method, and program

Abstract

There is provided an information processing system including: a feature point extraction unit configured to extract feature points from an image of an environment using the image of the environment or point cloud data; and an emission region determination unit configured to determine an emission region in an environment to be irradiated with the sensing light during a next frame of the image based on a position of the feature point in the next frame of the image estimated from a position of the feature point in the current frame of the image.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Japanese priority patent application JP 2023-174573, filed on October 6, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to information processing devices, information processing methods, and procedures. Background Technology

[0004] In recent years, advancements have been made in the development of technologies that use camera devices to image the environment and sensors (e.g., light detection and ranging (LiDAR)) that can use sensing light to detect the location of objects in the environment.

[0005] For example, in Simultaneous Localization and Mapping (SLAM), a combination of visual-SLAM using images captured by a camera device and LiDAR-SLAM using point cloud data measured by LiDAR is being considered. In visual-SLAM, highly accurate estimation can be performed in environments with limited structural information due to the use of texture information. However, the estimation accuracy decreases in the presence of lighting changes. Similarly, in LiDAR-SLAM, highly accurate estimation can be performed even in environments with varying lighting conditions due to the use of sensor light. However, the estimation accuracy also decreases in environments with limited structural information. For this reason, the combination of visual-SLAM and LiDAR-SLAM allows for further improvement in the robustness of the estimation to different environments.

[0006] In addition, in recent years, as disclosed in the following Patent Document 1, an imaging device has been developed that can measure the distance to an object by detecting the reflected light of light applied to an object existing in the environment and generating an image of the environment.

[0007] Citation List

[0008] Patent documents

[0009] PTL 1: WO 2020 / 255999 Summary of the Invention

[0010] Technical issues

[0011] However, since the imaging device disclosed in Patent Document 1 emits sensing light toward an object in order to measure the distance to an object present in the environment, its power consumption is higher than that of an imaging device that only performs imaging. In this regard, it is desirable to reduce the power consumption of an imaging device that can image the environment and measure the distance to an object present in the environment.

[0012] Solution to the problem

[0013] According to this disclosure, an information processing system is provided, including: a feature point extraction unit configured to extract feature points from an image of the environment using an image or point cloud data of the environment; and an emission region determination unit configured to determine an emission region in the environment to be illuminated with sensing light during the next frame based on the position of feature points in the next frame of the image estimated from the position of feature points in the current frame of the image, wherein the feature point extraction unit and the emission region determination unit are each implemented via at least one processor.

[0014] Furthermore, according to this disclosure, a computer-executed information processing method is provided, comprising: extracting feature points from an image of the environment using an image or point cloud data of the environment; and determining an emission area in the environment to be illuminated with sensing light during the next frame based on the estimated position of feature points in the next frame of the image according to the position of feature points in the current frame of the image.

[0015] Furthermore, according to this disclosure, a non-transitory computer-readable medium is provided, which includes a program that, when executed by a computer, causes the computer to perform an information processing method, the method comprising: extracting feature points from an image of the environment using an image or point cloud data of the environment; and determining an emission area in the environment to be illuminated with sensing light during the next frame based on the estimated positions of feature points in the next frame of the image according to the positions of feature points in the current frame of the image. Attached Figure Description

[0016] [ Figure 1A ] Figure 1A This is a schematic diagram illustrating an imaging apparatus controlled by an information processing device according to an embodiment of the present disclosure.

[0017] [ Figure 1B ] Figure 1B This is an illustrative diagram describing coaxial measurements used for imaging and ranging.

[0018] [ Figure 2 ] Figure 2 It is shown schematically. Figure 1A A schematic diagram illustrating an example of the mechanism of the imaging device shown.

[0019] [ Figure 3 ] Figure 3 It is shown schematically. Figure 1A A schematic diagram illustrating an example of the mechanism of the imaging device shown.

[0020] [ Figure 4 ] Figure 4 It is shown schematically. Figure 1AA schematic diagram illustrating an example of the mechanism of the imaging device shown.

[0021] [ Figure 5 ] Figure 5 This is a block diagram illustrating the functional configuration of an information processing apparatus according to an embodiment.

[0022] [ Figure 6 ] Figure 6 This is a schematic diagram illustrating the method for determining the launch area.

[0023] [ Figure 7 ] Figure 7 This is a flowchart illustrating the operation flow of the information processing apparatus and sensor unit according to an embodiment.

[0024] [ Figure 8 ] Figure 8 This is a flowchart illustrating the operation flow of the information processing device and sensor unit according to the first modified example.

[0025] [ Figure 9A ] Figure 9A This is an explanatory diagram describing the observation accuracy of feature points relative to their distance from the sensor unit.

[0026] [ Figure 9B ] Figure 9B This is an explanatory diagram describing the observation accuracy of feature points relative to their distance from the sensor unit.

[0027] [ Figure 10A ] Figure 10A This is a schematic diagram illustrating the change in the size of the emission region in the second modified example.

[0028] [ Figure 10B ] Figure 10B This is a schematic diagram illustrating the change in the size of the emission region in the second modified example.

[0029] [ Figure 11 ] Figure 11 This is a flowchart illustrating the operation flow of the information processing device and sensor unit according to the second modified example.

[0030] [ Figure 12 ] Figure 12 This is a block diagram illustrating the functional configuration of the information processing apparatus according to the third modified example.

[0031] [ Figure 13 ] Figure 13 This is an illustrative diagram describing a method for estimating the position of feature points based on the position of sensor units.

[0032] [ Figure 14 ] Figure 14This is a flowchart illustrating the operation flow of the information processing device and sensor unit according to the third modified example.

[0033] [ Figure 15 ] Figure 15 This is a block diagram illustrating the functional configuration of the information processing apparatus according to the fourth modified example.

[0034] [ Figure 16 ] Figure 16 This is a graphical representation of an example photon counting histogram when the sensor unit is a SPAD LiDAR.

[0035] [ Figure 17 ] Figure 17 This is a flowchart illustrating the operation flow of the information processing device and sensor unit according to the fourth modified example.

[0036] [ Figure 18 ] Figure 18 This is a block diagram illustrating an example of a computer configuration. Detailed Implementation

[0037] Embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration will be indicated by the same reference numerals, and redundant descriptions will be omitted.

[0038] Note that the descriptions will be presented in the following order.

[0039] 1. Imaging equipment

[0040] 2. Configuration of information processing devices

[0041] 3. Operation Example

[0042] 4. Modify the example

[0043] 4.1. First Modification Example

[0044] 4.2. Second Modification Example

[0045] 4.3. Third Modification Example

[0046] 4.4. Fourth Modification Example

[0047] 5. Hardware Configuration Example

[0048] <1. Imaging equipment>

[0049] First, refer to Figures 1A to 4 An imaging device controlled by an information processing apparatus according to an embodiment of the present disclosure is described. Figure 1A This is a schematic diagram illustrating an imaging device CS controlled by an information processing apparatus according to this embodiment. Figure 1B This is an illustrative diagram describing coaxial measurements used for imaging and ranging. Figures 2 to 4 Each is shown schematically. Figure 1A A schematic diagram of an example of the mechanism of the imaging device CS shown.

[0050] The imaging device CS controlled by the information processing apparatus according to this embodiment is a sensor capable of imaging the environment and measuring the distance to objects present in the environment. Specifically, the imaging device CS has an imaging function and a ranging function. The imaging function is used to capture RGB color images or monochrome images, and the ranging function is used to measure the distance to objects present in the environment based on the time of flight until the sensing light emitted to the object returns.

[0051] like Figure 1A As shown, the imaging device CS can coaxially perform imaging of the environment and distance measurement of objects present in the environment. That is, the imaging device CS can perform imaging of the same area and distance measurement from the same location. In this case, since the imaging area IF for imaging the environment and the distance measurement area DF for measuring the distance to objects present in the environment are the same, the imaging device CS can acquire images and distance measurement information that are not misaligned.

[0052] At the same time, such as Figure 1B As shown, the imaging device IS and the distance measuring device DS are separate devices, and they perform observations from different positions. Therefore, misalignment occurs between the imaging area IF of the imaging device IS that images the environment and the distance measuring area DF of the distance measuring device DS that measures the distance to objects present in the environment. In this case, to integrate the image captured by the imaging device IS with the distance measurement information from the distance measuring device DS, it is necessary to correct the positional relationship between the imaging device IS and the distance measuring device DS, and to correct the coordinates of the image and the coordinates of the distance measurement information. Furthermore, since misalignment occurs between the imaging area IF and the distance measuring area DF, the image and distance measurement information can be integrated for the areas where the imaging area IF and the distance measuring area DF overlap.

[0053] Since the imaging device CS, controlled by the information processing apparatus according to this embodiment, can coaxially perform imaging of the environment and distance measurement of objects present in the environment, image and distance measurement information can be integrated without performing corrections. Furthermore, since the imaging device CS can synchronously perform imaging of the environment and distance measurement of objects present in the environment, it is possible to acquire images and distance measurement information that are completely synchronized in time. Moreover, compared to providing imaging device IS and distance measurement device DS separately, the imaging device CS makes the hardware and configuration of the entire system more compact.

[0054] For example, such as Figure 2 As shown, the imaging device CS can be a sensor that includes a ranging pixel 25d and an imaging pixel 25i in the pixel region 20. The ranging pixel 25d is a pixel that receives sensing light (e.g., infrared light) emitted onto an object present in the environment. The imaging device CS can calculate the distance to the object based on the time-of-flight of the sensing light received by the ranging pixel 25d. The imaging pixel 25i includes, for example, a red pixel 25r that receives red light, a green pixel 25g that receives green light, and a blue pixel 25b that receives blue light. The imaging device CS can generate an image of the environment based on the amount of light received by each of the red pixel 25r, green pixel 25g, and blue pixel 25b. Since both the ranging pixel 25d and the imaging pixel 25i are present in the pixel region 20, the imaging device CS can coaxially perform imaging of the environment and measurement of the distance to objects present in the environment.

[0055] In addition, such as Figure 3As shown, the imaging device CS can be a sensor obtained by stacking a first substrate 21 in which imaging pixels 25i are disposed and a second substrate 22 in which ranging pixels 25d are disposed. The first substrate 21 includes a red pixel 25r that receives red light, a green pixel 25g that receives green light, and a blue pixel 25b that receives blue light. The imaging device CS can generate an image of the environment based on the amount of light received by each of the red pixel 25r, green pixel 25g, and blue pixel 25b disposed in the first substrate 21. The second substrate 22 includes a ranging pixel 25d that receives sensing light (e.g., infrared light) emitted onto an object present in the environment. The imaging device CS can calculate the distance to the object based on the time of flight of the sensing light received by the ranging pixel 25d disposed in the first substrate 21. The second substrate 22 can be stacked on the surface of the first substrate 21 opposite to the light-receiving surface, and the ranging pixel 25d receives the sensing light (e.g., infrared light) that has been transmitted through the first substrate 21. By stacking a first substrate 21 including imaging pixels 25i and a second substrate 22 including ranging pixels 25d, the imaging device CS is able to coaxially perform imaging of the environment and measurement of distances to objects present in the environment.

[0056] In addition, such as Figure 4 As shown, the imaging device CS can be a SPAD LiDAR that emits sensing light Le towards an object 30 in the environment and receives reflected light Lr from the object 30 via a single-photon avalanche diode (SPAD) 23. The SPAD 23 is a sensor capable of detecting light reception on a photon-by-photon basis. The imaging device CS can acquire ranging information of the object 30 based on the time-of-flight of the sensing light Le by receiving the reflected light Lr from the SPAD 23. Furthermore, the SPAD 23 receives not only the reflected light Lr but also the ambient light Lb from the illumination 31. Therefore, the imaging device CS can also acquire an image of the object 30 by integrating the histogram of the image received by the SPAD 23 (i.e., reflected light Lr + ambient light Lb). Accordingly, the imaging device CS can coaxially perform imaging of the environment and measurement of the distance to objects in the environment by changing the time histogram calculation of the photons received by the SPAD 23.

[0057] However, in the imaging device CS, since the imaging area IF and the ranging area DF are the same, the power consumption increases when the sensing light is emitted to all areas of the ranging area DF.

[0058] In this respect, the information processing apparatus according to this embodiment controls the emission area of ​​the sensing light based on feature points extracted from an image captured by the imaging device CS. Accordingly, the information processing apparatus is able to emit the sensing light only to the region in which feature points are estimated to exist for processing such as SLAM, and thus, the power consumption caused by the emission of the sensing light is reduced.

[0059] <2. Configuration of Information Processing Device>

[0060] Next, we will refer to Figure 5 The configuration of the information processing apparatus according to this embodiment is described. Figure 5 This is a block diagram illustrating the functional configuration of the information processing apparatus 100 according to this embodiment.

[0061] like Figure 5 As shown, the information processing device 100 includes an image acquisition unit 110, a point cloud acquisition unit 120, a feature point extraction unit 130, a self-position estimation unit 140, and a emission area determination unit 150. Furthermore, the sensor unit 200 includes an image light receiving unit 210, a ranging light receiving unit 220, a light emission unit 230, and a light emission control unit 240. The sensor unit 200 is capable of imaging the environment and measuring the distance to objects present in the environment. The information processing device 100 and the sensor unit 200 can be included in a single device, or they can be included in a system in which the information processing device 100 and the sensor unit 200 are included in different devices.

[0062] (Sensor unit 200)

[0063] The image light receiving unit 210 generates an image of the environment by receiving incident light. Specifically, the image light receiving unit 210 can generate an image of the environment by receiving incident light by each of a plurality of pixels arranged in a two-dimensional matrix. The image light receiving unit 210 may be, for example, a complementary metal-oxide-semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, or a single-photon avalanche diode (SPAD) image sensor.

[0064] The ranging light receiving unit 220 measures the distance to an object in the environment by receiving reflected light from the sensing light emitted from the light emitting unit 230 and directed towards the object. Specifically, the ranging light receiving unit 220 can measure the distance to the object by measuring the time of flight until the sensing light emitted from the light emitting unit 230 is reflected by the object, returns, and is received by the pixel. Furthermore, the ranging light receiving unit 220 can generate point cloud data of the environment based on the measured distance information to the object.

[0065] Note that the ranging light receiving unit 220 and the image light receiving unit 210 coaxially measure the distance to objects present in the environment. As a result, the information processing device 100 can perform arithmetic processing on the image of the environment generated by the image light receiving unit 210 and the ranging information of objects present in the environment measured by the ranging light receiving unit 220 without correction.

[0066] The light-emitting unit 230 emits sensing light toward objects present in the environment. Specifically, the light-emitting unit 230 may be a laser light source that emits a laser beam based on control from the light-emitting control unit 240. The light-emitting unit 230 may be, for example, a laser light source that emits infrared light.

[0067] The light emission control unit 240 controls the light emission unit 230 so that the sensed light is emitted onto the emission area input from the information processing device 100. Specifically, the light emission control unit 240 can control the light emission unit 230 so that the sensed light is emitted onto the emission area in which the information processing device 100 estimates the presence of feature points. The details of the emission area generated by the information processing device 100 will be described below.

[0068] The aforementioned sensor unit 200 can be included in a moving object such as a cart, drone, or car. Accordingly, the moving object, such as a cart, drone, or car, can estimate its own position based on the images and ranging information obtained by the sensor unit 200, thereby performing automated movement or manipulation support with high accuracy.

[0069] Furthermore, the sensor unit 200 can be included in a device such as a smartphone or a head-mounted display (HMD). Accordingly, the device, such as a smartphone or HMD, can estimate its own position based on images and ranging information obtained by the sensor unit 200, thereby overlaying augmented reality (AR) virtual space onto real space with high accuracy. Therefore, these devices can provide users with high-quality augmented reality (AR).

[0070] (Information processing device 100)

[0071] The image acquisition unit 110 acquires an image of the environment from the sensor unit 200. Specifically, the image acquisition unit 110 may be a connection port for acquiring an image of the environment from the sensor unit 200. Furthermore, the image acquisition unit 110 may acquire a received light signal from the image light receiving unit 210 and generate an image of the environment based on the acquired received light signal.

[0072] The point cloud acquisition unit 120 acquires point cloud data of the environment from the sensor unit 200. Specifically, the point cloud acquisition unit 120 may be a connection port for acquiring point cloud data of the environment from the sensor unit 200. In addition, the point cloud acquisition unit 120 may acquire ranging information of objects existing in the environment from the ranging light receiving unit 220, and generate point cloud data of the environment based on the acquired ranging information.

[0073] The feature point extraction unit 130 extracts feature points from the image based on the image of the environment acquired by the image acquisition unit 110 and the point cloud data of the environment acquired by the point cloud acquisition unit 120. Specifically, first, the feature point extraction unit 130 integrates the captured image and the point cloud data to generate an RGB-D image including color data and depth data. Next, the feature point extraction unit 130 can extract feature points from the image by applying a known feature point extraction algorithm to the generated RGB-D image. Examples of feature point extraction algorithms include Scale Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Features from Speed-Up Segmentation Test (FAST), Maximum Stable Extremum Region (MSER), and the Harris operator.

[0074] The self-position estimation unit 140 uses feature points extracted by the feature point extraction unit 130 to estimate the self-position of the sensor unit 200. Specifically, the self-position estimation unit 140 can estimate the self-position of the sensor unit 200 by tracking the position and depth of feature points extracted from the image between frames. For example, the self-position estimation unit 140 can estimate the change in the self-position of the sensor unit 200 by identifying corresponding feature points between frames of the image and tracking the changes in the position and depth of corresponding feature points between frames. As a detailed method for estimating the change in self-position based on the changes in the position and depth of feature points, a known SLAM algorithm can be used.

[0075] The emission region determination unit 150 determines the emission region from the light-emitting unit 230 based on the positions of feature points in the image. (Refer to...) Figure 6 The method for determining the transmission area ER by the transmission area determination unit 150 is described in detail. Figure 6 This is a schematic diagram illustrating the method for determining the emission region (ER).

[0076] like Figure 6As shown, firstly, the emission region determination unit 150 acquires feature point FP1 in the current frame of the image extracted by the feature point extraction unit 130. Next, based on feature point FP1 in the current frame of the image and the self-position of the sensor unit 200 estimated by the self-position estimation unit 140, the emission region determination unit 150 estimates the position of feature point FP2 in the next frame FV2 of the image. Subsequently, the emission region determination unit 150 determines an emission region ER of a predetermined size in the peripheral region surrounding the position of feature point FP2 in the next frame FV2 of the image. Figure 6 As shown, the shape of the emission region ER can be, for example, rectangular, circular, or elliptical.

[0077] Accordingly, the light emission control unit 240 of the sensor unit 200 can control the light emission unit 230 so that the sensing light is emitted only to the emission region ER in the next frame where feature points are estimated to exist. Therefore, since the area from which the sensing light is emitted from the light emission unit 230 can be limited, the information processing device 100 can reduce the power consumption of the sensor unit 200. Simultaneously, since the sensor unit 200 can increase the intensity of the sensing light to be emitted to the emission region ER with reduced power consumption, the detection range of the feature point depth can be expanded.

[0078] For example, when the device including the sensor unit 200 is a trolley, it is desirable for the trolley to move automatically to estimate its own position with high accuracy during item delivery or site patrol. Furthermore, it is desirable for the trolley to estimate its own position with low power consumption so that it can move for extended periods. According to the information processing apparatus 100 of this embodiment, the trolley can estimate its own position with low power consumption and high accuracy while maintaining its movement time.

[0079] For example, when the device including the sensor unit 200 is a drone, the drone, which is expected to fly autonomously, estimates its own position with high accuracy when performing hovering flight or waypoint flight. Furthermore, the drone, which is expected to fly autonomously, estimates its own position with low power consumption to increase flight time. According to the information processing apparatus 100 of this embodiment, the aforementioned drone can achieve highly accurate autonomous flight while maintaining flight time.

[0080] For example, when the device including sensor unit 200 is a vehicle, a large number of sensors are installed on the vehicle performing autonomous driving or driver assistance to ensure safety. Therefore, it is desirable for each of the sensors installed on the vehicle to have low power consumption in order to suppress battery consumption and improve fuel efficiency. According to the information processing apparatus 100 according to this embodiment, the aforementioned vehicle can estimate its own position with low power consumption using SLAM or the like.

[0081] For example, when the device including the sensor unit 200 is a portable device such as a smartphone, it is desirable for the portable device implementing augmented reality (AR) to estimate its own position with high accuracy in order to suppress the overlap gap between virtual space and real space. Furthermore, it is desirable for the portable device implementing AR to estimate its own position with low power consumption in order to increase battery operating time and suppress battery heat generation. According to the information processing apparatus 100 according to this embodiment, the aforementioned portable device can estimate its own position with low power consumption and high accuracy, and thus suppress battery heat generation.

[0082] For example, when the device including the sensor unit 200 is a head-mounted display (HMD), it is desirable for the HMD to realize virtual reality (VR) to estimate its own position with high accuracy in order to suppress the deviation between the movement of the head wearing the HMD and the estimated own position. Furthermore, since the VR-enabled HMD is worn on the head, it is desirable for the HMD to suppress battery heat generation. According to the information processing apparatus 100 of this embodiment, the aforementioned HMD can estimate its own position with low power consumption and high accuracy, and thus suppress battery heat generation.

[0083] <3. Operation Example>

[0084] Next, we will refer to Figure 7 The operation of the information processing apparatus 100 and the sensor unit 200 according to this embodiment is described. Figure 7 This is a flowchart illustrating the operation flow of the information processing apparatus 100 and the sensor unit 200 according to this embodiment.

[0085] like Figure 7 As shown, firstly, the light emission control unit 240 determines whether the emission area has been determined by the emission area determination unit 150 (S101).

[0086] If the emission region determination unit 150 does not estimate the position of the feature point and does not determine the emission region (S101 / No), the light emission control unit 240 controls the light emission unit 230 to emit sensing light across the entire viewing angle of the image in the next frame (S102). Conversely, if the emission region determination unit 150 estimates the position of the feature point and determines the emission region with that feature point as the center (S101 / Yes), the light emission control unit 240 controls the light emission unit 230 to emit sensing light only towards the determined emission region (S103).

[0087] Next, the ranging light receiving unit 220 receives the reflected light of the emitted sensing light, and thus the point cloud data of the environment is acquired by the point cloud acquisition unit 120 (S105). Subsequently, the feature point extraction unit 130 extracts feature points based on the point cloud data of the environment acquired by the point cloud acquisition unit 120 and the image of the environment acquired by the image acquisition unit 110 (S107).

[0088] Subsequently, the self-position estimation unit 140 determines whether it has obtained the depth data of the extracted feature points (S109). That is, the self-position estimation unit 140 determines whether sensing light has been emitted toward the extracted feature points and whether the distance to the feature points has been measured.

[0089] If depth data of feature points has been obtained (S109 / Yes), the self-position estimation unit 140 estimates the self-position of the sensor unit 200 based on the feature points (S111). Specifically, the self-position estimation unit 140 estimates the self-position of the sensor unit 200 by tracking changes in the position and depth of corresponding feature points between frames of the image. Conversely, if depth data of feature points has not been obtained (S109 / No), the self-position estimation unit 140 skips the estimation of the self-position of the sensor unit 200.

[0090] Then, the emission region determination unit 150 estimates the position of the feature points in the next frame based on the feature points in the current frame of the image, and determines the peripheral region centered on the estimated feature point position as the emission region (S113).

[0091] Based on the above operation, the information processing device 100 can determine the emission region around the estimated feature point and cause the light-emitting unit 230 to emit sensing light only into the emission region where the estimated feature point is located. Therefore, the information processing device 100 enables a reduction in the power consumption of the sensor unit 200.

[0092] <4. Modification Example>

[0093] (4.1. First Modification Example)

[0094] Reference Figure 8 The information processing device 100 is described according to the first modified example. Figure 8 This is a flowchart illustrating the operation flow of the information processing apparatus 100 and the sensor unit 200 according to a first modified example. In the first modified example, instead of emitting sensing light across the entire viewing angle of the image, the emission of sensing light is skipped when the emission area is not determined.

[0095] For example, if the emission area is not determined by the emission area determination unit 150 due to reasons such as the initialization of the information processing device 100, the light emission control unit 240 controls the light emission unit 230 not to emit sensing light. In this case, such as Figure 8 As shown, since no point cloud data of the environment is obtained, the feature point extraction unit 130 extracts feature points only based on the image of the environment obtained by the image acquisition unit 110 (S201).

[0096] Subsequently, the emission region determination unit 150 estimates the position of the feature points in the next frame of the image based on the extracted feature points in the current frame of the image (S203). The emission region determination unit 150 determines whether the position of the feature points in the next frame has been estimated (S205). If the position of the feature points in the next frame has been estimated (S205 / Yes), the emission region determination unit 150 determines the peripheral region centered on the estimated feature point position as the emission region (S207).

[0097] Meanwhile, without estimating the position of feature points in the next frame of the image (S205 / No), the emission region determination unit 150 does not determine the emission region. Therefore, the light emission control unit 240 controls the light emission unit 230 not to emit sensing light in the next frame.

[0098] According to the first modified example, the information processing apparatus 100 can perform control such that, when the emission area determination unit 150 has not determined the emission area, sensing light is not emitted across the entire viewing angle of the image. Therefore, according to the first modified example, the information processing apparatus 100 can further reduce the power consumption of the sensor unit 200.

[0099] (4.2. Second Modification Example)

[0100] Reference Figures 9A to 11 The information processing apparatus 100 is described according to the second modified example. Figure 9A and Figure 9B Each of these is an explanatory diagram describing the observation accuracy of feature point FP relative to the distance from sensor unit 200. Figure 10A and Figure 10B Each is a schematic diagram illustrating the change in the size of the emission region in the second modified example. Figure 11 This is a flowchart illustrating the operation flow of the information processing device 100 and the sensor unit 200 according to the second modified example.

[0101] The accuracy of the sensor unit 200 in observing the feature point FP (especially the accuracy of depth observation) is affected by the distance between the sensor unit 200 and the feature point FP.

[0102] For example, such as Figure 9AAs shown, for feature point FP, which is close to sensor unit 200, the position of feature point FP in the next frame is estimated with high accuracy due to high observation accuracy. Therefore, in the next frame, the possible region OB where feature point FP is estimated to exist is relatively narrow. Meanwhile, as... Figure 9B As shown, for a feature point FP that is far from the sensor unit 200, the position of the feature point FP in the next frame is estimated with low accuracy due to low observation accuracy. Therefore, in the next frame, the possible region OB where the feature point FP is estimated to exist is relatively wide. That is, since the observation accuracy (especially the depth observation accuracy) of the feature point FP decreases as the distance from the sensor unit 200 increases, the estimation accuracy of the position of the feature point FP in the next frame also decreases.

[0103] In the second modified example, the emission region determination unit 150 changes the size of the emission region in which the sense light is emitted based on the distance between the sensor unit 200 and the feature point FP. Specifically, as Figure 10A As shown, when the distance between the sensor unit 200 and the feature point FP is short, the emission region determination unit 150 makes the size of the emission region ERn centered on the feature point FP smaller. Meanwhile, as... Figure 10B As shown, when the distance between the sensor unit 200 and the feature point FP is long, the emission region determination unit 150 makes the size of the emission region ERf centered on the feature point FP larger. Accordingly, the information processing device 100 is able to further increase the probability that the feature point FP is included in the determined emission region.

[0104] Reference Figure 11 The operation flow of the information processing device 100 and sensor unit 200 according to the second modified example is described.

[0105] like Figure 11 As shown, firstly, the feature point extraction unit 130 extracts feature points based on the point cloud data of the environment acquired by the point cloud acquisition unit 120 and the image of the environment acquired by the image acquisition unit 110 (S301).

[0106] Next, the self-position estimation unit 140 determines whether it has obtained the depth data of the extracted feature points (S303). That is, the self-position estimation unit 140 determines whether sensing light has been emitted toward the extracted feature points and whether the distance to the feature points has been measured.

[0107] Having obtained the depth data of the feature points (S303 / Yes), the self-position estimation unit 140 estimates the self-position of the sensor unit 200 based on the feature points (S305). Specifically, the self-position estimation unit 140 estimates the self-position of the sensor unit 200 by tracking the changes in the position and depth of the corresponding feature points between frames of the image.

[0108] Subsequently, the emission region determination unit 150 estimates the positions of feature points in the next frame based on feature points in the current frame of the image, and determines the peripheral region centered on the estimated feature point positions as the emission region (S307). At this time, the emission region determination unit 150 changes the size of the emission region based on the depth of the estimated feature points. For example, the emission region determination unit 150 can make the size of the emission region larger as the depth of the estimated feature points increases (i.e., the distance between the estimated feature points and the sensor unit 200 is longer).

[0109] Meanwhile, in the absence of depth data for the feature points (S303 / No), the self-position estimation unit 140 estimates the position of the feature points in the next frame based on the feature points in the current frame of the image, and determines the peripheral region centered on the estimated feature point position as the emission region (S308). At this time, the emission region determination unit 150 can set the size of the emission region to a predetermined size.

[0110] Then, the light emission control unit 240 controls the light emission unit 230 to emit sensing light only to the determined emission area (S309).

[0111] Considering that the accuracy of feature point observation is low when the distance between the sensor unit 200 and the feature point is long, the information processing apparatus 100 according to the second modified example can make the size of the emitting area for emitting sensing light larger. Accordingly, the information processing apparatus 100 according to the second modified example can make the sensor unit 200 emit light towards the feature point more reliably, and thus acquire the depth data of the feature point more reliably.

[0112] (4.3. Third Modification Example)

[0113] Reference Figures 12 to 14 The information processing device 101 is described according to the third modified example. Figure 12 This is a block diagram illustrating the functional configuration of the information processing apparatus 101 according to the third modified example. Figure 13 This is an explanatory diagram illustrating a method for estimating the position of feature point FP based on the position of sensor unit 200. Figure 14 This is a flowchart illustrating the operation flow of the information processing device 101 and the sensor unit 200 according to the third modified example.

[0114] like Figure 12 As shown, the information processing device 101 includes an image acquisition unit 110, a point cloud acquisition unit 120, a feature point extraction unit 130, a self-position estimation unit 140, a transmission area determination unit 150, and a position acquisition unit 160. Figure 5 Compared to the information processing device 100 shown, the information processing device 101 according to the third modified example further includes a location acquisition unit 160. The information processing device 101 and the sensor unit 200 can be included in a single device, or they can be included in a system in which the information processing device 101 and the sensor unit 200 are included in different devices.

[0115] The position acquisition unit 160 detects information related to the position and orientation of the sensor unit 200. For example, the position acquisition unit 160 may be an inertial measurement unit (IMU) disposed in a device including the sensor unit 200 or disposed in the sensor unit 200 itself.

[0116] The information related to the position and orientation of the sensor unit 200 detected by the position acquisition unit 160 is used by the self-position estimation unit 140 to estimate the self-position of the sensor unit 200. The self-position estimation unit 140 can estimate the self-position of the sensor unit 200 with higher accuracy by further using the information related to the position and orientation of the sensor unit 200 detected by the position acquisition unit 160.

[0117] like Figure 13 As shown, firstly, the emission region determination unit 150 projects the feature point FP onto a three-dimensional space based on information related to the position and pose of the sensor unit 200 in the current frame and the position and depth of the feature point FP in the current frame of the image. Next, the emission region determination unit 150 estimates the position of the sensor unit 200 itself in the next frame based on information related to the position and pose of the sensor unit 200. Subsequently, the emission region determination unit 150 is able to estimate the position of the feature point FP in the next frame of the image based on the position of the feature point FP projected onto the three-dimensional space in the current frame and the position of the sensor unit 200 itself in the next frame.

[0118] Accordingly, the emission region determination unit 150 is able to estimate the position of feature point FP in the next frame of the image with high accuracy. Therefore, the emission region determination unit 150 is able to determine a smaller emission region, and thus reduce the power consumption of the sensor unit 200.

[0119] Furthermore, similar to the second modified example, the emission region determination unit 150 can change the size of the emission region based on the estimated depth of the feature point FP. That is, the emission region determination unit 150 can make the size of the emission region in which the sensing light is emitted smaller as the distance between the sensor unit 200 and the feature point FP decreases.

[0120] Reference Figure 14 The operation flow of the information processing device 101 and sensor unit 200 according to the third modified example is described.

[0121] like Figure 14 As shown, firstly, the emission region determination unit 150 projects the feature points in the current frame onto a three-dimensional space based on the sensor unit 200's own position in the current frame and the position and depth of the feature points in the current frame of the image (S401). Next, the self-position estimation unit 140 estimates the sensor unit 200's own position in the next frame based on the information related to the position and pose of the sensor unit 200 obtained by the position acquisition unit 160 (S403).

[0122] Subsequently, the emission region determination unit 150 estimates the position of the feature points in the next frame of the image based on the positions of the feature points projected onto the three-dimensional space in the current frame and the position of the sensor unit 200 in the next frame (S405). Furthermore, the emission region determination unit 150 determines the peripheral region centered on the estimated feature point position as the emission region (S407). At this time, the emission region determination unit 150 can change the size of the emission region based on the depth of the estimated feature points. For example, the emission region determination unit 150 can make the size of the emission region larger as the depth of the estimated feature points increases (i.e., the distance between the estimated feature points and the sensor unit 200 is longer).

[0123] Then, the light emission control unit 240 controls the light emission unit 230 to emit sensing light only to the determined emission area (S409).

[0124] According to the third modified example, the information processing apparatus 101 can estimate the position of feature points with higher accuracy by further utilizing information related to the position and orientation of the sensor unit 200. Accordingly, the information processing apparatus 101 according to the third modified example can reduce the emission area and thus further reduce the power consumption of the sensor unit 200.

[0125] (4.4. Fourth Modification Example)

[0126] Reference Figures 15 to 17 The information processing device 102 is described according to the fourth modified example. Figure 15This is a block diagram illustrating the functional configuration of the information processing apparatus 102 according to the fourth modified example. Figure 16 This is a graphical representation of an example photon counting histogram when the sensor unit 200 is a SPADLiDAR. Figure 17 This is a flowchart illustrating the operation flow of the information processing device 102 and the sensor unit 200 according to the fourth modified example.

[0127] like Figure 15 As shown, the information processing device 102 includes an image acquisition unit 110, a point cloud acquisition unit 120, a feature point extraction unit 130, a self-position estimation unit 140, a transmission area determination unit 150, and a covariance estimation unit 170. Figure 5 Compared to the information processing apparatus 100 shown, the information processing apparatus 102 according to the fourth modified example further includes a covariance estimation unit 170. The information processing apparatus 102 and the sensor unit 200 can be included in a single apparatus, or they can be included in a system in which the information processing apparatus 102 and the sensor unit 200 are included in different apparatuses.

[0128] Furthermore, in the fourth modified example, sensor unit 200 acquires images and ranging information of the environment via SPAD LiDAR. Specifically, sensor unit 200 acquires images of the environment and ranging information of objects present in the environment by performing arithmetic processing on the photon count histogram acquired by SPAD LiDAR.

[0129] The covariance estimation unit 170 estimates the covariance of peak values ​​in the histogram based on the photon count histogram acquired by the SPAD LiDAR of the sensor unit 200. The covariance of peak values ​​is a value corresponding to the region in which observation points may actually exist, where the observation points correspond to peak values.

[0130] Specifically, for example, obtaining the photon count histogram of SPAD LiDAR as... Figure 16 The diagram shown is as follows. Figure 4 As shown, in the photon counting histogram of the SPAD LiDAR, ambient light Lb from illumination 31 exists as a background count, and reflected light Lr from the sensed light Le exists as a peak P. The covariance estimation unit 170 can estimate the covariance of the peak based on the width of the peak P or the S / N ratio of the peak P to the background.

[0131] In the fourth modified example, the emission region determination unit 150 changes the size of the emission region based on the estimated covariance of the peak (observation point). For example, the emission region determination unit 150 can make the emission region smaller as the estimated covariance of the peak (observation point) becomes smaller. Accordingly, the information processing device 102 enables the sensor unit 200 to emit sensing light towards the feature point more reliably, and thus acquire depth data of the feature point more reliably.

[0132] Furthermore, the variance of the peak value (observation point) estimated by the covariance estimation unit 170 can also be used by the self-position estimation unit 140 to estimate the covariance of the self-position of the sensor unit 200. The covariance of the estimation result indicates the reliability of the estimation. Therefore, when combining multiple self-position estimates, the self-position estimation unit 140 can use the covariance as an indicator of the reliability of the estimation.

[0133] Reference Figure 17 The operation flow of the information processing device 102 and sensor unit 200 according to the fourth modified example is described.

[0134] like Figure 17 As shown, firstly, the feature point extraction unit 130 extracts feature points based on the photon count histogram of the SPAD LiDAR (S501). Next, the covariance estimation unit 170 estimates the covariance of the peak value based on the photon count histogram of the SPAD LiDAR (S503).

[0135] Subsequently, the self-position estimation unit 140 estimates the self-position of the sensor unit 200 based on the extracted feature points (S505). In addition, the self-position estimation unit 140 can use the covariance of the peak value to estimate the covariance of the self-position of the sensor unit 200 (S507).

[0136] Next, the emission region determination unit 150 estimates the positions of feature points in the next frame based on feature points in the current frame of the image, and determines the peripheral region centered on the estimated feature point positions as the emission region (S509). At this time, the emission region determination unit 150 changes the size of the emission region based on the magnitude of the covariance of the estimated peak. For example, the emission region determination unit 150 may make the size of the emission region larger as the covariance of the peak increases (i.e., the region where the observation point corresponding to the peak may actually exist is larger).

[0137] Then, the light emission control unit 240 controls the light emission unit 230 to emit sensing light only to the determined emission area (S511).

[0138] According to the fourth modified example, the information processing apparatus 102 can change the size of the emission region in which the sensing light is emitted based on the covariance of the peak values ​​in the photon counting histogram. Accordingly, the information processing apparatus 102 according to the fourth modified example can enable the sensor unit 200 to emit sensing light toward the feature point more reliably, and thus acquire depth data of the feature point more reliably.

[0139] <5. Hardware Configuration Example>

[0140] Reference Figure 18 The hardware configuration of computer 900 is described. Computer 900 is hardware used to implement information processing devices 100, 101 and 102 according to this embodiment. Figure 18 This is a block diagram illustrating a configuration example of computer 900.

[0141] The functions of computer 900 can be achieved through the collaboration of software and hardware as described below. The functions of feature point extraction unit 130, self-position estimation unit 140, emission region determination unit 150, and covariance estimation unit 170 can be executed by, for example, CPU 901. The functions of image acquisition unit 110 and point cloud acquisition unit 120 can be executed by, for example, connection port 910 or communication device 911.

[0142] like Figure 18 As shown, the computer 900 includes a central processing unit (CPU) 901, a read-only memory (ROM) 902, and a random access memory (RAM) 903.

[0143] Computer 900 may also include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a driver 909, a connection port 910, and a communication device 911. Instead of CPU 901 or in addition to CPU 901, computer 900 may include processing circuitry such as digital signal processors (DSPs) and application-specific integrated circuits (ASICs).

[0144] The CPU 901 functions as an arithmetic processing unit or control device, and controls the operation of the computer 900 according to various programs recorded in the ROM 902, RAM 903, storage device 908, or a removable recording medium attached to the drive 909. The ROM 902 stores programs, operating parameters, etc., to be used by the CPU 901. The RAM 903 temporarily stores programs to be used during the execution of the CPU 901, parameters to be used during execution, etc.

[0145] CPU 901, ROM 902, and RAM 903 are interconnected via a host bus 904a capable of performing high-speed data transfer. The host bus 904a is connected via a bridge 904 to an external bus 904b, such as a Peripheral Component Interconnect / Interface (PCI) bus. The external bus 904b is connected to various components via an interface 905.

[0146] Input device 906 is, for example, a device that receives input from a user, such as a mouse, keyboard, touch panel, button, switch, and joystick. Note that input device 906 may be a microphone or the like that that detects the user's voice. Input device 906 may be, for example, a remote control device that uses infrared or other radio waves, and may be an external connection device compatible with the operation of computer 900.

[0147] Input device 906 also includes input control circuitry, which outputs input signals generated based on information input by the user to CPU 901. The user can operate input device 906 to give instructions for inputting various types of data or performing processing operations on computer 900.

[0148] Output device 907 is a device capable of visually or audibly presenting information acquired or generated by computer 900 to a user. For example, output device 907 may be a display device such as a liquid crystal display (LCD), a plasma display panel (PDP), an organic light-emitting diode (OLED) display, a hologram, and a projector; a sound output device such as a speaker and headphones; or a printing device such as a printer. Output device 907 is capable of outputting information acquired through processing by computer 900 as video such as text and images, or sound such as audio and acoustics.

[0149] Storage device 908 is an example data storage device configured as a storage unit of computer 900. Storage device 908 may include, for example, magnetic storage devices (e.g., hard disk drives (HDDs)), semiconductor storage devices, optical storage devices, or magneto-optical storage devices. Storage device 908 is capable of storing programs to be executed by CPU 901, various types of data, various types of data acquired from external sources, etc.

[0150] Drive 909 is a read / write device for removable recording media such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memories, and is either integrated into computer 900 or externally connected. For example, drive 909 can read information recorded on the attached removable recording medium and output the read information to RAM 903. Furthermore, drive 909 can write information to the attached removable recording medium.

[0151] Connection port 910 is used to directly connect external devices to computer 900. Connection port 910 can be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, or a Small Computer System Interface (SCSI) port. Alternatively, connection port 910 can be an RS-232C port, an optical audio terminal, or a High Definition Multimedia Interface (HDMI) (registered trademark) port. Connection port 910 enables the sending and receiving of various types of data between computer 900 and external devices by connecting to them.

[0152] Communication device 911 is, for example, a communication interface configured for connecting to communication network 920. For example, communication device 911 may be a wired or wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), or a communication card for wireless USB (WUSB). Furthermore, communication device 911 may be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), a modem for various types of communication, etc.

[0153] The communication device 911 is capable of sending / receiving signals to / from, for example, the Internet or another communication device using predetermined protocols such as TCP / IP. Furthermore, the communication network 920 connected to the communication device 911 is a wired or wireless network, and can be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0154] Note that a program for performing functions equivalent to those of the information processing devices 100, 101, and 102 can be created in hardware such as the CPU 901, ROM 902, and RAM 903 built into the computer 900. Furthermore, a computer-readable recording medium for recording the program can be provided.

[0155] Although embodiments of the present disclosure have been described above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications will arise within the scope of the technical concept described in the claims, and it will be understood that such changes and modifications also fall within the technical scope of the present disclosure.

[0156] Furthermore, the effects described in this specification are merely illustrative or explanatory and are therefore not limiting. That is, in addition to or in lieu of the effects described above, the technology according to this disclosure may have other effects that would be apparent to those skilled in the art based on the description herein.

[0157] Note that the following configurations also fall within the technical scope of this disclosure. (1)

[0159] An information processing device, comprising:

[0160] A feature point extraction unit extracts feature points from an image of the environment using an image and point cloud data of the environment derived from reflected light from sensing light emitted into the environment; and

[0161] The emission region determination unit determines the emission region in the environment to be illuminated by the sensing light during the next frame based on the estimated positions of feature points in the next frame of the image according to the positions of feature points in the current frame of the image. (2)

[0163] The information processing apparatus according to (1) above further includes:

[0164] A self-position estimation unit estimates the self-position of a device equipped with an imaging device and a ranging sensor based on the feature points. The imaging device acquires the image, and the ranging sensor acquires the point cloud data. (3)

[0166] According to the information processing apparatus described in (1) or (2) above, wherein,

[0167] The position of the feature point in the next frame is estimated based on the position of the feature point in the current frame and the position of itself. (4)

[0169] The information processing apparatus according to any one of (1) to (3) above, wherein,

[0170] The self-position estimation unit also estimates the covariance of its own position based on the covariance of the observed results of the reflected light. (5)

[0172] The information processing apparatus according to any one of (1) to (4) above, wherein,

[0173] The device also estimates its own position based on the sensing results of an inertial measurement unit installed on the device. (6)

[0175] The information processing apparatus according to any one of (1) to (5) above, wherein,

[0176] The self-position estimation unit uses the feature points to estimate its own position via SLAM. (7)

[0178] The information processing apparatus according to any one of (1) to (6) above, wherein,

[0179] The ranging sensor is an RGBD coaxial sensor integrated with the imaging device. (8)

[0181] The information processing apparatus according to any one of (1) to (7) above, wherein,

[0182] The emission region determination unit determines the entire sensing area of ​​the ranging sensor as the emission region without estimating the position of the feature points in the next frame of the image. (9)

[0184] The information processing apparatus according to any one of (1) to (8) above, wherein,

[0185] The feature point extraction unit extracts the feature points from the image using only the image without acquiring the point cloud data. (10)

[0187] The information processing apparatus according to any one of (1) to (9) above, wherein,

[0188] The emission region determination unit determines the outer region surrounding the location of the feature point in the next frame as the emission region. (11)

[0190] The information processing apparatus according to any one of (1) to (10) above, wherein,

[0191] The emission region determination unit controls the size of the peripheral region based on the distance to feature points in the current frame. (12)

[0193] The information processing apparatus according to any one of (1) to (11) above, wherein,

[0194] The emission region determination unit controls the size of the peripheral region based on the covariance of the observed results of the reflected light. (13)

[0196] An information processing method executed by a computer, comprising:

[0197] Feature points are extracted from the image of the environment using image and point cloud data of the environment derived from reflected light from sensed light emitted into the environment; and

[0198] The emission area in the environment to be illuminated by the sensing light during the next frame is determined based on the estimated positions of feature points in the current frame of the image. (14)

[0200] A program that causes a computer to function as:

[0201] A feature point extraction unit extracts feature points from an image of the environment using an image and point cloud data of the environment derived from reflected light from sensing light emitted into the environment; and

[0202] The emission region determination unit determines the emission region in the environment to be illuminated by the sensing light during the next frame based on the estimated positions of feature points in the next frame of the image according to the positions of feature points in the current frame of the image. (15)

[0204] An information processing system, comprising:

[0205] A feature point extraction unit, configured to extract feature points from an image of the environment using an image or point cloud data of the environment; and

[0206] A emission region determination unit is configured to determine the emission region in the environment to be illuminated with sensing light during the next frame based on the estimated positions of feature points in the next frame of the image according to the positions of feature points in the current frame of the image.

[0207] The feature point extraction unit and the emission region determination unit are each implemented via at least one processor. (16)

[0209] The information processing system according to (15) further includes:

[0210] A self-position estimation unit is configured to estimate the self-position of a device equipped with an imaging device and a ranging sensor based on the feature points, wherein the imaging device acquires the image and the ranging sensor acquires the point cloud data.

[0211] The self-position estimation unit is implemented via at least one processor. (17)

[0213] The information processing system according to any one of (15) or (16), wherein,

[0214] The position of the feature point in the next frame is estimated based on the position of the feature point in the current frame and the position of itself. (18)

[0216] The information processing system according to any one of (15) to (17), wherein,

[0217] The self-position estimation unit is also configured to estimate the covariance of its own position based on the covariance of the observed results of the reflected light. (19)

[0219] The information processing system according to any one of (15) to (18), wherein,

[0220] The device also estimates its own position based on the sensing results of an inertial measurement unit installed on the device. (20)

[0222] The information processing system according to any one of (15) to (19), wherein,

[0223] The self-position estimation unit is also configured to use the feature points to estimate its own position via SLAM. (twenty one)

[0225] The information processing system according to any one of (15) to (20), wherein,

[0226] The ranging sensor is an RGBD coaxial sensor integrated with the imaging device. (twenty two)

[0228] The information processing system according to any one of (15) to (21), wherein,

[0229] The emission region determination unit is further configured to determine the entire sensing area of ​​the ranging sensor as the emission region without estimating the position of feature points in the next frame of the image. (twenty three)

[0231] The information processing system according to any one of (15) to (22), wherein,

[0232] The feature point extraction unit is also configured to extract the feature points from the image using only the image without acquiring the point cloud data. (twenty four)

[0234] The information processing system according to any one of (15) to (23), wherein,

[0235] The emission region determination unit is further configured to determine the peripheral region surrounding the location of the feature point in the next frame as the emission region. (25)

[0237] The information processing system according to any one of (15) to (24), wherein,

[0238] The emission region determination unit is also configured to control the size of the peripheral region based on the distance to feature points in the current frame. (26)

[0240] The information processing system according to any one of (15) to (25), wherein,

[0241] The emission region determination unit is also configured to control the size of the peripheral region based on the covariance of the observed results of the reflected light. (27)

[0243] An information processing method executed by a computer, comprising:

[0244] Extract feature points from an image of the environment using image or point cloud data of the environment; and

[0245] The emission area in the environment to be illuminated with sensing light during the next frame is determined based on the estimated positions of feature points in the current frame of the image. (28)

[0247] A non-transitory computer-readable medium includes a program thereon, which, when executed by a computer, causes the computer to perform an information processing method, the method comprising:

[0248] Extract feature points from an image of the environment using image or point cloud data of the environment; and

[0249] The emission area in the environment to be illuminated with sensing light during the next frame is determined based on the estimated positions of feature points in the current frame of the image. (29)

[0251] According to the information processing system described in (15) to (26), wherein,

[0252] The point cloud data of the environment is derived from the reflected light of the sensing light emitted into the environment. (30)

[0254] The information processing system according to any one of (15) to (26) or (29) further includes:

[0255] An image light receiving unit, the image light receiving unit being configured to generate an image of the environment;

[0256] The image light receiving unit is implemented via at least one processor. (31)

[0258] The information processing system according to any one of (15) to (26), (29) or (30) further includes:

[0259] A ranging optical receiving unit, configured to measure the distance to an object present in the environment.

[0260] The ranging optical receiving unit is implemented via at least one processor. (32)

[0262] The information processing system according to any one of (15) to (26) or (29) to (31) further includes:

[0263] A light source configured to emit sensing light toward an object present in the environment. (33)

[0265] The information processing system according to any one of (15) to (26) or (29) to (32) further includes:

[0266] A light-emitting control unit, configured to control the light source.

[0267] The light-emitting control unit is implemented via at least one processor. (34)

[0269] The information processing system according to any one of (15) to (26) or (29) to (33) further includes:

[0270] An information processing device, comprising the feature point extraction unit and the emission region determination unit, and

[0271] A sensor device configured to image the environment and measure distances to objects included in the environment.

[0272] Those skilled in the art will understand that various modifications, combinations, sub-combinations and alterations can be made according to design requirements and other factors, as long as such modifications, combinations, sub-combinations and alterations are within the scope of the appended claims or their equivalents.

[0273] List of reference numerals

[0274] 100, 101, 102 Information processing devices

[0275] 110 Image Acquisition Unit

[0276] 120 point cloud acquisition units

[0277] 130 Feature Point Extraction Units

[0278] 140 self-position estimation units

[0279] 150 Launch Area Determination Units

[0280] 160 Position Acquisition Unit

[0281] 170 Covariance Estimation Units

[0282] 200 sensor units

[0283] 210 Image light receiving unit

[0284] 220 ranging optical receiver unit

[0285] 230 light-emitting units

[0286] 240 Light-emitting control unit

Claims

1. An information processing system, comprising: A feature point extraction unit is configured to extract feature points from an image of the environment using an image or point cloud data of the environment. as well as A emission region determination unit is configured to determine the emission region in the environment to be illuminated by the sensing light during the next frame based on the estimated positions of feature points in the next frame of the image according to the positions of feature points in the current frame of the image. The feature point extraction unit and the emission region determination unit are each implemented via at least one processor.

2. The information processing system according to claim 1, further comprising: A self-position estimation unit is configured to estimate the self-position of a device equipped with an imaging device and a ranging sensor based on the feature points, wherein the imaging device acquires the image and the ranging sensor acquires the point cloud data. The self-position estimation unit is implemented via at least one processor.

3. The information processing system according to claim 2, wherein, The position of the feature point in the next frame is estimated based on the position of the feature point in the current frame and the position of itself.

4. The information processing system according to claim 2, wherein, The self-position estimation unit is also configured to estimate the covariance of its own position based on the covariance of the observation of the reflected light.

5. The information processing system according to claim 2, wherein, The device also estimates its own position based on the sensing results of an inertial measurement unit installed on the device.

6. The information processing system according to claim 2, wherein, The self-position estimation unit is also configured to use the feature points to estimate its own position via SLAM.

7. The information processing system according to claim 2, wherein, The ranging sensor is an RGBD coaxial sensor integrated with the imaging device.

8. The information processing system according to claim 2, wherein, The emission region determination unit is further configured to determine the entire sensing area of ​​the ranging sensor as the emission region without estimating the position of feature points in the next frame of the image.

9. The information processing system according to claim 1, wherein, The feature point extraction unit is also configured to extract the feature points from the image using only the image without acquiring the point cloud data.

10. The information processing system according to claim 1, wherein, The emission region determination unit is further configured to determine the peripheral region surrounding the location of the feature point in the next frame as the emission region.

11. The information processing system according to claim 10, wherein, The emission region determination unit is also configured to control the size of the peripheral region based on the distance to feature points in the current frame.

12. The information processing system according to claim 10, wherein, The emission region determination unit is also configured to control the size of the peripheral region based on the covariance of the observed results of the reflected light.

13. An information processing method executed by a computer, comprising: Feature points are extracted from the image of the environment using image or point cloud data of the environment; as well as The emission area in the environment to be illuminated by the sensing light during the next frame is determined based on the estimated positions of feature points in the current frame of the image.

14. A non-transitory computer-readable medium having a program thereon, the program causing the computer to perform an information processing method when executed by a computer, the method comprising: Feature points are extracted from the image of the environment using image or point cloud data of the environment; as well as The emission area in the environment to be illuminated by the sensing light during the next frame is determined based on the estimated positions of feature points in the current frame of the image.

15. The information processing system according to claim 1, wherein, The point cloud data of the environment is derived from the reflected light of the sensing light emitted into the environment.

16. The information processing system according to claim 1, further comprising: An image light receiving unit, configured to generate an image of the environment. The image light receiving unit is implemented via at least one processor.

17. The information processing system according to claim 1, further comprising: A ranging optical receiving unit, configured to measure the distance to an object present in the environment. The ranging optical receiving unit is implemented via at least one processor.

18. The information processing system according to claim 1, further comprising: A light source configured to emit sensing light toward an object present in the environment.

19. The information processing system according to claim 18, further comprising: A light-emitting control unit, configured to control the light source. The light-emitting control unit is implemented via at least one processor.

20. The information processing system according to claim 1, further comprising: An information processing device, comprising the feature point extraction unit and the emission region determination unit, and A sensor device configured to image the environment and measure distances to objects included in the environment.

Images