Self-position estimation device, self-position estimation system, and self-position estimation program

Abstract

This self-position estimation device (2A) comprises: an inertial measurement unit (5) that detects three-dimensional inertial motion; a vision sensor (6) that detects a change in brightness of light captured in each pixel; a processor (8) that identifies an extraction region for extracting a feature amount for self-position estimation in a detection region detected by the vision sensor (6) on the basis of a detection result from the inertial measurement unit (5), and estimates a self-position on the basis of the feature amount in the identified extraction region; and a housing (3A) that accommodates the inertial measurement unit (5), the vision sensor (6), and the processor (8).

Description

Self-location estimation device, self-location estimation system, and self-location estimation program

[0001] The present invention relates to a self-location estimation device, a self-location estimation system, and a self-location estimation program.

[0002] Conventionally, an information processing system that estimates the position and orientation of an input device has been known (see, for example, Patent Document 1). The information processing system described in Patent Document 1 includes an information processing device, a recording device, a head-mounted display, an input device, and an output device such as a television. Of these, the input device is held by a user and operated with the user's fingers. The input device includes an inertial measurement unit and transmits sensor data to the information processing device at a predetermined cycle. The input device also includes multiple markers for tracking the position and orientation of the input device, which are captured by multiple imaging devices provided in the head-mounted display. The imaging devices include visible light sensors such as a charge-coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor. Each of the imaging devices captures an image of an area in front of the user at a predetermined cycle, synchronized with the image data, and transmits the captured image of the real space to the information processing device. The information processing device has a first estimation function that analyzes images captured by the input device to estimate the position and orientation of the input device in real space. The information processing device also has a second estimation function that analyzes sensor data transmitted from the input device and estimates the position and orientation of the input device in real space, and derives the position and orientation of the input device using the estimation results of the first and second estimation functions.

[0003] JP 2023-120843 A

[0004] However, the information processing system described in Patent Document 1 processes images captured by a visible light sensor such as a CCD sensor or a CMOS sensor, which places a heavy processing load on the information processing device. This requires an expensive, high-performance processor in the information processing device, which increases costs. Furthermore, incorporating a function for estimating the position and orientation of the input device into the input device increases costs and makes it difficult to miniaturize the input device.

[0005] The present invention aims to solve at least part of the above problems, and one of its objectives is to provide a self-location estimation device, a self-location estimation system, and a self-location estimation program that can simplify processing and reduce costs.

[0006] A self-location estimation device according to a first aspect of the present invention comprises an inertial measurement unit that detects three-dimensional inertial motion, a vision sensor that detects changes in the brightness of light captured at each pixel, a processor that identifies an extraction area in the detection area detected by the vision sensor based on the detection results of the inertial measurement unit, from which feature values ​​for self-location estimation are extracted, and estimates the self-location based on the feature values ​​in the identified extraction area, and a housing that accommodates the inertial measurement unit, the vision sensor, and the processor.

[0007] A self-location estimation system according to a second aspect of the present invention comprises a self-location estimation device according to the first aspect and a light emitting device that emits reference light, and the processor sets the position of the self-location estimation device when the reference light is detected as its initial position.

[0008] A self-location estimation program according to a third aspect of the present invention is a self-location estimation program that causes a processor to execute a self-location estimation process, and causes the processor to function as an area identification means that identifies an extraction area from which features for self-location estimation are extracted in a detection area detected by a vision sensor based on detection results from an inertial measurement unit that detects three-dimensional inertial motion, a feature extraction means that extracts features in the identified extraction area based on the detection results from the vision sensor, a matching means that compares the extracted features with features of the surrounding environment that have been acquired in advance, and a self-location estimation means that estimates the self-location based on the comparison results from the matching means.

[0009] A block diagram showing the configuration of a self-location estimation system according to a first embodiment. A perspective view showing a self-location estimation device provided in a self-location estimation system according to a second embodiment. A plan view showing a self-location estimation device according to the second embodiment. A diagram showing an example of wearing the self-location estimation device according to the second embodiment. A plan view showing a self-location estimation device provided in a self-location estimation system according to a third embodiment.

[0010] [First Embodiment] A first embodiment of the present invention will now be described with reference to the drawings. [Schematic Configuration of Self-Location Estimation System] Fig. 1 is a block diagram showing the configuration of a self-location estimation system 1 according to this embodiment. The self-location estimation system 1 according to this embodiment includes a self-location estimation device 2A that is attached to an operating device or a human body, and the self-location estimation device 2A estimates its own position and transmits the estimated position. As shown in Fig. 1, the self-location estimation system 1 includes an information processing device 11 and a light emitting device 12 in addition to the self-location estimation device 2A. Each component of the self-location estimation system 1 will now be described.

[0011] The information processing device 11 receives the position of the self-location estimation device 2A estimated by the self-location estimation device 2A and executes a predetermined process. For example, the information processing device 11 executes a game application and progresses a game according to the game application in accordance with the received position of the self-location estimation device 2A. The light emitting device 12 emits reference light when defining an initial position in the self-location estimation device 2A. As will be described in detail later, the self-location estimation device 2A defines the position at which it receives the reference light from the light emitting device 12 as its initial position. The light emitting device 12 may be an independent light emitting device or may be provided in another device. For example, the light emitting device 12 may be provided in a head-mounted display worn by a user, a stationary display, a wearable device worn by a user, or an operation device held by a user.

[0012] [Configuration of Self-Location Estimation Device] As described above, the self-location estimation device 2A estimates its own position and transmits the estimated position to an external device. For example, the self-location estimation device 2A estimates its own position within a space such as a living room and transmits the estimated position to an information processing device 11. The self-location estimation device 2A includes a housing 3A, a grayscale camera 4, an inertial measurement unit 5, a vision sensor 6, a memory 7, a processor 8, and a communication unit 9.

[0013] [Configuration of the housing] The housing 3A is a housing that houses the grayscale camera 4, the inertial measurement unit 5, the vision sensor 6, the memory 7, the processor 8, and the communication unit 9. The housing 3A is configured appropriately according to the type, shape, etc. of the attachment part to which the self-location estimation device 2A is attached. The self-location estimation device 2A is unitized by the housing 3A.

[0014] [Configuration of the Grayscale Camera] The grayscale camera 4 is a global shutter type grayscale camera. That is, the grayscale camera 4 is a grayscale camera that captures an entire frame at once. The grayscale camera 4 is provided in the housing 3A, and outputs the image of the outside of the housing 3A to the processor 8. That is, the grayscale camera 4 captures an image of the outside of the self-location estimation device 2A, and outputs the captured grayscale image to the processor 8. Such a grayscale camera 4 operates at a frame rate that is significantly lower than the processing cycle of the vision sensor 6 in the self-location estimation device 2A, for example, a frame rate of 1 to 5 fps.

[0015] [Configuration of the Inertial Measurement Unit and Vision Sensor] The inertial measurement unit 5 has an orthogonal triaxial acceleration sensor and an orthogonal triaxial angular velocity sensor, and measures the triaxial acceleration and triaxial angular velocity acting on the self-localization estimation device 2A. The inertial measurement unit 5 outputs the acceleration and angular velocity detection results to the processor 8. The vision sensor 6 is a vision sensor called an EVS (Event-Based Vision Sensor) and has multiple pixels that detect light. The vision sensor 6 outputs the luminance change detection results of the multiple pixels to the processor 8. More specifically, the vision sensor 6 detects an event when the luminance change of light captured by each pixel exceeds a predetermined threshold, and outputs the coordinates and brightness polarity of the pixel when the event occurred, as well as the time when the event occurred, to the processor 8. Optical flow can be generated based on the information output from the vision sensor 6. The information output from the vision sensor 6 can also be referred to as event data.

[0016] [Memory Configuration] The memory 7 has a volatile memory that functions as a work memory for the processor 8 and a non-volatile memory that stores programs and data. For example, the memory 7 stores a self-location estimation program for executing the self-location estimation process, as well as environmental information used in the self-location estimation process. The environmental map can be, for example, an image captured by the grayscale camera 4 or the vision sensor 6 of the omnidirectional surrounding environment of the self-location estimation device 2A.

[0017] [Processor Configuration] The processor 8 controls the operation of the self-location estimation device 2A. In addition, the processor 8 reads a self-location estimation program stored in the memory 7 and executes a self-location estimation process. By executing the self-location estimation process, the processor 8 identifies a comparison area for self-location estimation based on the detection results from the inertial measurement unit 5 and the detection results from the vision sensor 6, and estimates the self-location based on feature amounts in the identified comparison area. The processor 8 includes a motion determination unit 81, a feature amount extraction unit 82, a matching unit 83, a self-location estimation unit 84, and a correction unit 85 that function by executing the self-location estimation program. The motion determination unit 81, the feature amount extraction unit 82, the matching unit 83, and the self-location estimation unit 84 correspond to a motion determination means, a feature amount extraction means, a matching means, and a self-location estimation means.

[0018] The motion determination unit 81 corresponds to a region identification unit that identifies a comparison region. Based on the detection results of the inertial measurement unit 5 and the vision sensor 6, the motion determination unit 81 determines whether the cause of the luminance change is due to a change in movement and posture of the self-location estimation device 2A or a change in the surrounding environment, and identifies a comparison region in which to compare feature quantities for self-location estimation within the region in which luminance change is detected by the vision sensor 6. That is, based on the detection results of the inertial measurement unit 5 and the vision sensor 6, the motion determination unit 81 identifies, as an extraction region, a target change region that changes in response to a change in movement and posture of the self-location estimation device 2A within the region in which luminance change is detected by the vision sensor 6.

[0019] Specifically, the motion determination unit 81 generates an optical flow from the detection results of the vision sensor 6. Then, based on the angular velocity information and acceleration information detected by the inertial measurement unit 5, the motion determination unit 81 identifies, within the generated optical flow region, a detection region due to factors other than the movement and posture change of the self-location estimation device 2A as a non-target change region. For example, within the generated optical flow region, the motion determination unit 81 identifies, as a non-target change region, a brightness change region in which an image of a moving object located in the surrounding environment of the self-location estimation device 2A and whose brightness change is detected by the vision sensor 6. Then, the motion determination unit 81 removes the identified non-target change region from the generated optical flow region to identify a target change region. The target change region is an extraction region from which feature values ​​are extracted and a comparison region that is compared with the environmental map by the matching unit 83.

[0020] The feature extraction unit 82 extracts features within the target change region identified by the motion determination unit 81 from at least one of the optical flow generated by the motion determination unit and the light detection results input from the vision sensor 6. Examples of the features include local features such as SIFT (Scale-Invariant Feature Transform) and PIRF (Position-Invariant Robust Features). Furthermore, when grayscale images are input from the grayscale camera 4 at a predetermined frequency, such as 5 fps, the feature extraction unit 82 extracts features from the input grayscale images. Specifically, when grayscale images are input from the grayscale camera 4, the feature extraction unit 82 combines the features extracted from at least one of the generated optical flow and the event data from the vision sensor 6 with the features extracted from the input grayscale images to extract more accurate features.

[0021] As described above, when extracting features based only on grayscale images from the grayscale camera 4, the frame rate of the grayscale camera 4 needs to be increased. In this case, since grayscale images with large data sizes need to be processed at high speed, the processor 8 needs to be a high-performance processor. In contrast, in this embodiment, the feature extraction unit 82 extracts features from at least one of the optical flow and event data. When grayscale images are input from the grayscale camera 4 at a frame rate significantly lower than the processing cycle of the vision sensor 6, the feature extraction unit 82 combines the feature extracted from at least one of the optical flow and event data with the feature extracted from the grayscale image, thereby reducing the amount of processing by the processor 8 and improving the accuracy of the extracted feature.

[0022] The matching unit 83 compares the feature extracted by the feature extraction unit 82 with the environmental information stored in the memory 7. In this embodiment, the matching unit 83 specifies the direction in which the feature extracted from the grayscale image has moved, based on the optical flow based on the event data of the vision sensor 6, thereby speeding up the matching process.

[0023] The self-position estimation unit 84 estimates the self-position of the self-position estimation device 2A based on the comparison result by the matching unit 83. For example, the self-position estimation unit 84 identifies a position, such as an orientation and an altitude, at which the feature amounts extracted by the feature amount extraction unit 82 match, based on the environmental information stored in the memory 7, and estimates the identified position as the position of the self-position estimation device 2A. The self-position estimation unit 84 transmits the estimated position and attitude of the self-position estimation device 2A to the outside via the communication unit 9. Specifically, the self-position estimation unit 84 transmits the estimated position and attitude of the self-position estimation device 2A to the information processing device 11 via the communication unit 9. Note that the self-position estimation unit 84 defines the position and attitude of the self-position estimation device 2A when at least one of the grayscale camera 4 and the vision sensor 6 receives reference light from the light emission device 12 as the initial position and initial attitude.

[0024] The correction unit 85 corrects the output value of the inertial measurement unit 5 when the position of the self-position estimation device 2A is estimated by the self-position estimation unit 84 based on the feature amount extracted by the feature amount extraction unit 82 from the grayscale image input from the grayscale camera 4. Here, the position of the self-position estimation device 2A estimated based on the grayscale image input from the grayscale camera 4 has a relatively high degree of accuracy. Therefore, by correcting the output value of the inertial measurement unit 5 in accordance with changes in the estimated position of the self-position estimation device 2A based on the grayscale image, if an error occurs in the inertial measurement unit 5 due to a drift phenomenon or the like, the error can be eliminated.

[0025] [Configuration of Communication Unit] The communication unit 9 constitutes a transmitting section. The communication unit 9 communicates with an external device under the control of the processor 8. For example, the communication unit 9 communicates with an information processing device 11 and transmits information indicating the estimated position and attitude of the self-position estimation device 2A to the information processing device 11.

[0026] Effects of First Embodiment The self-localization system 1 according to the present embodiment described above provides the following effects. The self-localization device 2A includes a housing 3A, an inertial measurement unit 5, a vision sensor 6, and a processor 8. The housing 3A accommodates the inertial measurement unit 5, the vision sensor 6, and the processor 8. The inertial measurement unit 5 detects three-dimensional inertial motion of the self-localization estimation device 2A. The vision sensor 6 detects changes in the brightness of light captured for each pixel. Based on the detection results by the inertial measurement unit 5, the processor 8 identifies an extraction area in the detection area of ​​the vision sensor 6 from which feature amounts for self-localization estimation are extracted, and estimates the self-localization based on the feature amounts in the identified extraction area.

[0027] Here, when estimating self-position by processing captured images sequentially input from a grayscale camera at a high frame rate, high-speed processing is required. Furthermore, because each captured image has a large image size, a high-performance, expensive processor is required. In contrast, the vision sensor 6 detects and outputs changes in light brightness within the detection range pixel by pixel, so the processor 8 does not require image processing for self-position estimation. This simplifies the processing of the processor 8, allowing the processor 8 to be realized with an inexpensive, low-power processor such as an MCU (microcontroller unit). This reduces the cost of the self-position estimation device 2A.

[0028] Based on the detection results from the inertial measurement unit 5, the processor 8 can identify, as an extracted region, a region in the detection region of the vision sensor 6 where brightness changes based on changes in the movement and attitude of the self-localization estimation device 2A. In other words, based on the detection results from the inertial measurement unit, the processor 8 can exclude, from the detection region of the vision sensor 6, a region where brightness changes are detected by the vision sensor 6 due to factors other than the movement and attitude changes of the self-localization estimation device 2A. This makes it possible to exclude, for example, a region where brightness changes are detected by the vision sensor 6 due to the movement of an object other than the self in its surrounding environment from the detection region of the vision sensor 6. This allows the self-localization to be accurately estimated based on the detection results from the inertial measurement unit 5 and the vision sensor 6. Therefore, a self-localization estimation device 2A capable of accurately estimating its self-localization can be configured at low cost.

[0029] In the self-location estimation device 2A, the processor 8 includes a motion determination unit 81 as an area identification unit, a feature extraction unit 82, a matching unit 83, and a self-location estimation unit 84. The motion determination unit 81 identifies the above-mentioned comparison area. The feature extraction unit 82 extracts feature amounts in the identified comparison area based on the detection results of the vision sensor 6. The matching unit 83 compares the extracted feature amounts with feature amounts of the surrounding environment that have been acquired in advance and stored in the memory 7. The self-location estimation unit 84 estimates the self-location of the self-location estimation device 2A based on the comparison results of the matching unit 83. With this configuration, the feature amounts extracted based on the detection results of the vision sensor 6 are compared with feature amounts of the surrounding environment to estimate the self-location, thereby enabling the self-location estimation device 2A to accurately estimate its self-location.

[0030] In the self-location estimation device 2A, the motion determination unit 81 as an area identification unit removes non-target changed areas, which are detection areas of moving objects, from the detection area detected by the vision sensor, based on the results of comparing the acceleration and angular velocity detection results from the inertial measurement unit 5 with the optical flow generated based on the detection results from the vision sensor, and identifies a target changed area, which is an extracted area. With this configuration, by removing non-target changed areas from the detection area detected by the vision sensor 6, it is possible to accurately identify a target changed area where brightness changes occur due to movement and posture changes of the self-location estimation device 2A. Therefore, the position of the self-location estimation device 2A can be accurately estimated.

[0031] The self-location estimation device 2A includes a communication unit 9 that transmits information indicating the position and attitude estimated by the processor 8. The communication unit 9 corresponds to a transmitter. With this configuration, the information can be transmitted to, for example, an information processing device 11 provided outside the self-location estimation device 2A. This increases the versatility of the self-location estimation device 2A.

[0032] In the self-location estimation device 2A, the vision sensor 6 outputs the coordinates, time, and brightness polarity of the pixel where a change in brightness has occurred. With this configuration, the size of the data output from the vision sensor 6 and the size of the data processed by the processor 8 can be reduced, thereby reducing the processing load on the processor 8. Therefore, a less expensive processor can be used as the processor 8, and the self-location estimation device 2A that can accurately estimate its own location can be configured at low cost.

[0033] The self-location estimation device 2A includes a grayscale camera 4 mounted on the housing 3A. The grayscale camera 4 captures an image of the exterior of the housing 3A and outputs the captured image to the processor 8. The feature extraction unit 82 of the processor 8 extracts features from the target change region, which is the extraction region, based on the captured image from the grayscale camera 4. The gyro sensor may accumulate errors known as a drift phenomenon. If the detection results from a gyro sensor with such accumulated errors are used for self-location estimation, the estimated self-position may contain errors. In response to this, the detection results from the gyro sensor of the inertial measurement unit 5 can be corrected based on the captured image from the grayscale camera 4, and the accuracy of the position and attitude of the self-location estimation device 2A estimated by the processor 8 can be improved. Furthermore, because the grayscale camera 4 used in this manner does not provide images primarily used for self-location estimation, it can be operated at a frame rate lower than the frame rate when the grayscale camera 4 is used instead of the vision sensor 6. Therefore, even when a grayscale camera 4 is used, it is possible to prevent a large increase in the size of the data processed by the processor 8, and it is also possible to prevent a large increase in the power consumption of the self-position estimation device 2A.

[0034] The self-location estimation system 1 includes the self-location estimation device 2A described above and a light emitting device 12 that emits reference light. The self-location estimation unit 84 of the processor 8 sets the position of the self-location estimation device 2A when the reference light is detected as the initial position. This configuration makes it easier to determine the initial position of the self-location estimation device.

[0035] The self-location estimation program recorded in the memory 7 as a recording medium causes the processor 8 to execute a self-location estimation process. By executing the self-location estimation program, the processor functions as a motion determination unit 81, a feature extraction unit 82, a matching unit 83, and a self-location estimation unit 84, which serve as area identification units. The motion determination unit 81 identifies a target change area in the detection area detected by the vision sensor 6 based on the detection results of the inertial measurement unit 5, which detects three-dimensional inertial motion, and the detection results of the vision sensor 6, which detects changes in the brightness of light captured for each pixel. The target change area is an extraction area from which feature amounts for self-location estimation are extracted. The feature extraction unit 82 extracts feature amounts in the identified target change area based on the detection results of the vision sensor 6. The matching unit 83 compares the extracted feature amounts with feature amounts of the surrounding environment previously acquired and recorded in the memory 7. The self-location estimation unit 84 estimates the self-location of the self-location estimation device 2A based on the comparison results of the matching unit 83. By having the processor 8 execute such a self-location estimation program, the effects of the self-location estimation device 2A described above can be achieved.

[0036] [Second Embodiment] Next, a second embodiment of the present invention will be described. The self-location estimation system according to this embodiment has a configuration similar to that of the self-location estimation system 1 according to the first embodiment, and the housing of the self-location estimation device is configured in a ring shape. In the following description, parts that are the same or approximately the same as parts already described will be assigned the same reference numerals and descriptions thereof will be omitted.

[0037] FIG. 2 is a perspective view showing a self-location estimation device 2B included in the self-location estimation system according to this embodiment, FIG. 3 is a plan view showing the self-location estimation device 2B, and FIG. 4 is a diagram showing an example in which the self-location estimation device 2B is worn on the index finger of a user's right hand. The self-location estimation system according to this embodiment has the same configuration and functions as the self-location estimation system 1 according to the first embodiment, except that it includes the self-location estimation device 2B shown in FIGS. 2 to 4 instead of the self-location estimation device 2A. The self-location estimation device 2B has the same configuration and functions as the self-location estimation device 2A, except that it includes a housing 3B instead of the housing 3A. That is, the self-location estimation device 2B includes a housing 3B, a grayscale camera 4, an inertial measurement unit 5, a vision sensor 6, a memory 7, a processor 8, and a communication unit 9.

[0038] Similar to the housing 3A, the housing 3B accommodates the housing 3B, a grayscale camera 4, an inertial measurement unit 5, a vision sensor 6, a memory 7, a processor 8, and a communication unit 9. The housing 3B is ring-shaped and configured to be wearable on a user's finger, as shown in FIG. 4, for example. The housing 3B may also be configured to be wearable on the user's wrist, for example. Multiple vision sensors 6 are provided on opposite sides of the outer periphery of the housing 3B. That is, the housing 3B is provided with multiple vision sensors 6 with different imaging directions. In the example shown in FIGS. 2 to 4, the housing 3B is provided with two vision sensors 6 with imaging directions opposite to each other. Although not shown, a grayscale camera 4 is provided on the outer periphery of the housing 3B.

[0039] At least one button 3B1 is provided on the outer peripheral surface of the housing 3B at a position sandwiched between the multiple vision sensors 6. As shown in FIG. 4 , when the self-position estimation device 2B is worn on the index finger F1 of the right hand RH, the button 3B1 is provided at a position on the housing 3B that allows input with, for example, the thumb F2. The button 3B1 is, for example, a push button whose button head protrudes and retracts into the housing 3B. When the user inputs an operation, the button 3B1 outputs an operation signal to the processor 8.

[0040] As described above, a plurality of vision sensors 6 are provided in the housing 3B. Therefore, the processor 8 may include a switching unit that switches between the vision sensors 6 that output detection results from which an optical flow can be generated and the detection results from which an optical flow can be generated, among the detection results input from the plurality of vision sensors 6. The self-localization system according to the present embodiment described above can achieve the same effects as the self-localization system 1 according to the first embodiment.

[0041] [Third Embodiment] Next, a third embodiment of the present invention will be described. The self-location estimation system according to this embodiment has a configuration similar to that of the self-location estimation system 1 according to the first embodiment, and the housing of the self-location estimation device is configured to be attachable to an object including a human body. In the following description, parts that are the same or substantially the same as parts already described will be assigned the same reference numerals and description thereof will be omitted.

[0042] FIG. 5 is a plan view showing a self-location estimation device 2C of the self-location estimation system according to this embodiment. More specifically, FIG. 5 is a view showing the surface of the housing 3C opposite to the surface on which the vision sensor 6 is exposed. The self-location estimation system according to this embodiment has the same configuration and functions as the self-location estimation system 1 according to the first embodiment, except that it includes the self-location estimation device 2C shown in FIG. 5 instead of the self-location estimation device 2A. The self-location estimation device 2C has the same configuration and functions as the self-location estimation device 2A according to the first embodiment, except that it includes a housing 3C instead of the housing 3A. That is, the self-location estimation device 2C includes a grayscale camera 4, an inertial measurement unit 5, a vision sensor 6, a memory 7, a processor 8, and a communication unit 9, which are not shown in FIG. 5, in addition to the housing 3C.

[0043] The housing 3C accommodates the grayscale camera 4, the inertial measurement unit 5, the vision sensor 6, the memory 7, the processor 8, and the communication unit 9. The housing 3C has an attachment portion 3C1 provided on a surface different from the surface on which the grayscale camera 4 and the vision sensor 6 are provided. The attachment portion 3C1 is a portion that can be attached to an object including a human body, and in this embodiment, it is a screw hole to which a screw portion provided on a tripod can be fixed. Note that the attachment portion 3C1 may be a clip that can be attached to clothing, etc., and the configuration of the attachment portion 3C1 can be changed as appropriate. The self-localization system according to this embodiment described above can achieve the same effects as the self-localization system 1 according to the first embodiment.

[0044] [Modifications of the Embodiments] The present invention is not limited to the above-described embodiments, and modifications and improvements within the scope of achieving the object of the present invention are included in the present invention. In the above-described embodiments, the self-location estimation devices 2A, 2B, and 2C are equipped with a grayscale camera 4. However, this is not limiting, and the self-location estimation devices 2A, 2B, and 2C do not have to be equipped with a grayscale camera 4. On the other hand, instead of the grayscale camera 4 and the vision sensor 6, a hybrid sensor integrating these may be employed in the self-location estimation devices 2A, 2B, and 2C, or a SPAD (Single Photon Avalanche Diode) imaging sensor may be employed.

[0045] In each of the above embodiments, the processor 8 includes a motion determination unit 81 as a region identification unit, a feature extraction unit 82, a matching unit 83, and a self-position estimation unit 84. However, the configuration of the processor 8 is not limited to this, and as long as it can estimate the position and orientation of the self-position estimation device based on the detection results from the inertial measurement unit 5 and the vision sensor 6, the configuration of the processor 8 is not limited to the above.

[0046] In the above embodiments, the self-location estimation devices 2A, 2B, and 2C are each provided with a communication unit 9 that transmits information indicating the estimated position and attitude. However, this is not limiting, and the self-location estimation devices do not necessarily need to transmit information indicating the estimated position and attitude to the outside. For example, if the processor 8 processes the estimated position and attitude internally in the self-location estimation device, such as by recording it in the memory 7, the communication unit 9 may be omitted.

[0047] In the above embodiments, the vision sensor 6 outputs the coordinates, time, and brightness polarity of the pixel where the luminance change occurred. However, this is not limiting, and the vision sensor 6 may output at least the coordinates of the pixel where the luminance change occurred, and may not output other information.

[0048] In each of the above embodiments, the processor 8 sets the position and orientation at the time when the reference light emitted from the outside is detected as the initial position and orientation. However, this is not limited to this, and the initial position and orientation may be defined by, for example, the user. In other words, the method of defining the initial position and initial orientation in the self-position estimation device is not limited to the above.

[0049] In each of the above embodiments, the self-location estimation program read and executed by the processor 8 is recorded in the memory 7. However, this is not limiting, and the self-location estimation program may be recorded in a computer-readable recording medium other than the memory 7, for example, a disk-type recording medium, or other recording medium. Furthermore, the self-location estimation program may be provided via a network, or may be recorded in a memory within the arithmetic processing circuit.

[0050] [Summary of the Invention] The summary of the invention is provided below: [1] A self-location estimation device comprising: an inertial measurement unit that detects three-dimensional inertial motion; a vision sensor that detects changes in brightness of light captured for each pixel; a processor that identifies an extraction area in a detection area detected by the vision sensor based on a detection result by the inertial measurement unit, from which feature amounts for self-location estimation are extracted, and estimates a self-location based on the feature amounts in the identified extraction area; and a housing that accommodates the inertial measurement unit, the vision sensor, and the processor.

[0051] Here, when estimating self-location by processing captured images sequentially input at a high frame rate from a grayscale camera, not only is processing required at high speed, but each captured image is large in size, requiring an expensive processor with high processing power. In contrast, a vision sensor detects and outputs changes in light brightness within its detection range for each pixel, so the processor does not require image processing for self-location estimation. This simplifies the processor processing, making it possible to realize a processor with the above configuration even when using an inexpensive, low-power processor, thereby reducing the cost of the self-location estimation device.

[0052] The processor can then identify, based on the detection results from the inertial measurement unit, an area in the detection area of ​​the vision sensor where brightness changes based on changes in the movement and attitude of the self-localization device as an extracted area. In other words, based on the detection results from the inertial measurement unit, the processor can exclude, from the detection area of ​​the vision sensor, an area where the vision sensor detects brightness changes due to factors other than the movement and attitude changes of the self-localization device. This allows, for example, an area where the vision sensor detects brightness changes due to the movement of an object other than the self in its surrounding environment to be excluded from the detection area of ​​the vision sensor. This allows the self-localization to be accurately estimated based on the detection results from the inertial measurement unit and the vision sensor. Therefore, a self-localization device capable of accurately estimating its self-localization can be configured at low cost.

[0053] [2] The self-location estimation device according to [1], wherein the processor comprises: an area specification unit that specifies the extraction area; a feature extraction unit that extracts feature amounts in the specified extraction area based on the detection results of the vision sensor; a matching unit that compares the extracted feature amounts with previously acquired feature amounts of the surrounding environment; and a self-location estimation unit that estimates the self-location based on the comparison results by the matching unit. With this configuration, the feature amounts extracted based on the detection results of the vision sensor are compared with the feature amounts of the surrounding environment to estimate the self-location, thereby enabling the self-location estimation device to estimate the self-location with high accuracy.

[0054] [3] The self-localization device according to [2], wherein the region identification unit identifies the extracted region by removing a detection region of a moving object from the detection region of the vision sensor based on a comparison result between the acceleration and angular velocity detection results of the inertial measurement unit and an optical flow generated based on the detection results of the vision sensor. With this configuration, by removing the detection region of the moving object, it is possible to accurately identify a comparison region in which a brightness change occurs due to a movement and attitude change of the self-localization device including the inertial measurement unit and the vision sensor. Therefore, it is possible to accurately estimate the position of the self-localization device.

[0055] [4] The self-location estimation device according to any one of [1] to [3], further comprising a transmitter that transmits information indicating the location estimated by the processor. With this configuration, the information can be transmitted to, for example, an information processing device provided outside the self-location estimation device. This increases the versatility of the self-location estimation device.

[0056] [5] The self-location estimation device according to any one of [1] to [4], wherein the vision sensor outputs the coordinates, time, and brightness polarity of the pixel where the luminance change occurs. This configuration reduces the size of the data output from the vision sensor and the size of the data processed by the processor, thereby reducing the processing load on the processor. Therefore, a less expensive processor can be used as the processor, allowing for the inexpensive construction of a self-location estimation device capable of accurately estimating the self-location.

[0057] [6] The self-localization device according to any one of [1] to [5], further comprising a grayscale camera mounted on the housing and configured to output an image of the exterior of the housing to the processor, wherein the processor extracts the feature from the extraction area based on the image captured by the grayscale camera. The gyro sensor may accumulate errors known as a drift phenomenon. If the detection results of a gyro sensor with such accumulated errors are used for self-localization, the estimated self-localization may contain errors. In response to this, the detection results of the gyro sensor of the inertial measurement unit can be corrected based on the image captured by the grayscale camera, and the accuracy of the position and attitude of the self-localization device estimated by the processor can be improved. Furthermore, since the grayscale camera used in this manner does not provide images primarily used for self-localization, it can be operated at a frame rate lower than that when a grayscale camera is used instead of a vision sensor. Therefore, even when a grayscale camera is used, a significant increase in the size of data processed by the processor and a significant increase in power consumption of the self-localization device can be suppressed.

[0058] [7] A self-location estimation system comprising: the self-location estimation device according to any one of [1] to [6]; and a light emitting device that emits reference light, wherein the processor sets the position of the self-location estimation device when the reference light is detected as an initial position. With this configuration, it is possible to easily define the initial position of the self-location estimation device.

[0059] [8] A self-location estimation program that causes a processor to execute a self-location estimation process, the self-location estimation program causing the processor to function as: area specifying means that specifies an extraction area for extracting feature quantities for self-location estimation in a detection area detected by a vision sensor based on detection results from an inertial measurement unit that detects three-dimensional inertial motion, feature extraction means that extracts feature quantities in the specified extraction area based on the detection results from the vision sensor, matching means that compares the extracted feature quantities with feature quantities of the surrounding environment that have been acquired in advance, and self-location estimation means that estimates the self-location based on the comparison results from the matching means. By having a processor execute such a self-location estimation program, it is possible to achieve the same effects as the self-location estimation device described above.

[0060] 1...self-position estimation system, 11...information processing device, 12...light emitting device, 2A, 2B, 2C...self-position estimation device, 3A, 3B, 3C...housing, 4...grayscale camera, 5...inertial measurement unit, 6...vision sensor, 7...memory, 8...processor, 81...motion determination unit (area identification unit, area identification means), 82...feature extraction unit (feature extraction means), 83...matching unit (matching means), 84...self-position estimation unit (self-position estimation means), 85...correction unit, 9...communication unit (transmitter).

Claims

1. A self-position estimation device comprising: an inertial measurement unit that detects three-dimensional inertial movement; a vision sensor that detects changes in the brightness of light captured for each pixel; a processor that identifies an extraction area in the detection area detected by the vision sensor based on the detection results of the inertial measurement unit, from which feature values ​​for self-position estimation are extracted, and estimates self-position based on the feature values ​​in the identified extraction area; and a housing that contains the inertial measurement unit, the vision sensor, and the processor.

2. A self-location estimation device according to claim 1, characterized in that the processor comprises: an area identification unit that identifies the extraction area; a feature extraction unit that extracts features in the identified extraction area based on the detection results of the vision sensor; a matching unit that compares the extracted features with features of the surrounding environment acquired in advance; and a self-location estimation unit that estimates the self-location based on the comparison results by the matching unit.

3. A self-location estimation device according to claim 2, characterized in that the area identification unit identifies the extracted area by removing a detection area of ​​a moving object from the detection area of ​​the vision sensor based on the result of comparing the acceleration and angular velocity detection results of the inertial measurement unit with an optical flow generated based on the detection results of the vision sensor.

4. A self-location estimation device according to any one of claims 1 to 3, further comprising a transmitting unit that transmits information indicating the location estimated by the processor.

5. A self-location estimation device according to any one of claims 1 to 3, characterized in that the vision sensor outputs the coordinates, time and brightness polarity of the pixel where the luminance change has occurred.

6. A self-location estimation device according to any one of claims 1 to 3, further comprising a grayscale camera provided in the housing for capturing an image of the outside of the housing and outputting the captured image to the processor, wherein the processor extracts the feature amount from the extraction area based on the captured image taken by the grayscale camera.

7. A self-location estimation system comprising: a self-location estimation device according to any one of claims 1 to 3; and a light emitting device that emits reference light, wherein the processor sets the position of the self-location estimation device when the reference light is detected as its initial position.

8. A self-location estimation program that causes a processor to execute a self-location estimation process, comprising: an area identification means that identifies an extraction area for extracting features for self-location estimation in a detection area detected by a vision sensor based on the detection results of an inertial measurement unit that detects three-dimensional inertial motion; a feature extraction means that extracts features in the identified extraction area based on the detection results of the vision sensor; a matching means that compares the extracted features with features of the surrounding environment that have been acquired in advance; and a self-location estimation program that estimates the self-location based on the comparison results of the matching means.

Images