Information processing device and localization method

Abstract

An information processing device for localization of a train equipped with sensing devices that detect target objects in a railway environment surrounding the train comprises: a data obtaining unit that obtains multimodal sensor data including image data of the target objects and point cloud data of distances from the train to the target objects detected by the sensing devices; a selector that selects a relative localization method for localizing the train based on the image data or an absolute localization method for localizing the train based on the point cloud data; and a localization unit that localizes the train using the relative localization method or the absolute localization method selected by the selector.

Description

INFORMATION PROCESSING DEVICE AND LOCALIZATION METHOD

[0001] The present invention relates to an information processing device and a method for localization of train system.

[0002] In railway operations, localization of train systems, also known as train positioning or train tracking, is critical. For achieving safe operation of trains, it needs to accurately determine the position and speed of each train on the tracks. In recent years, positioning technologies using wireless signals indicative of positional information transmitted from satellites or the like are available for detecting positions and speeds of moving objects accurately. For instance, Global Navigation Satellite System (GNSS) such as Global Positioning System (GPS) is widely used in general.

[0003] In case that GNSS is used for localization of train systems, since railway tracks may pass through tunnels, urban areas, or areas with dense foliage, signal blockages may be occurred to affect the accuracy of GNSS. For improving the accuracy of localization by GNSS when signal blockages are occurred, the system described in the following PTL 1 is presented. PTL 1 discloses that various sensors including cameras, RADAR and LiDAR systems mounted on a train detect moving or still objects on or near the railway track along which the train is running, and the detection results of them are used for compensation of localization by GNSS. The objects of detection target include, for instance, a platform edge, a building, an installation near the railway track, a tree that has fallen, an animal on the track, and so on.

[0004] When various sensor data is acquired by a plurality of sensors mounted on a train, a high-performance computer performs sensor fusion processing that fuses the sensor data to provide a reliable image / point cloud of the train's surroundings even after dark or in fog. It can determine the course of the railway track and the train's exact position relative to landmarks such as buildings or electricity pylons. The computer compares the real-time sensor fusion data with the reference landmarks and HD map. By analyzing this information and factoring in the motion data of the train (speed and direction), it determines the train's exact position relative to the landmarks, thereby ensuring precise train localization. This technique disclosed by PTL 1 is also known as Simultaneous Localization and Mapping (SLAM).

[0005] [PTL 1] US 2016 / 0121912 A1

[0006] SLAM relies on detecting and tracking distinctive features or landmarks in the environment to estimate the exact position. However, in railway environments, there may be limited diversity in the landmarks, particularly in long stretches of railway tracks or in areas with repetitive structures. In addition, because operating frequencies of LiDARs are mostly about 20Hz or low, it is challenging for LiDARs mounted on a train to detect dense point cloud enough to matching for localization in case of high-speed railway systems. Due to these reasons, the SLAM-based localization approach cannot sufficiently work in railway environment.

[0007] The present invention has been conceived in consideration of problems such as those described above, and its principal object is to improve localization of trains in railway environment.SOLUTION TO TECHNICAL PROBLEM

[0008] An information processing device according to the present invention, for localization of a train equipped with sensing devices that detect target objects in a railway environment surrounding the train, comprises: a data obtaining unit that obtains multimodal sensor data including image data of the target objects and point cloud data of distances from the train to the target objects detected by the sensing devices; a selector that selects a relative localization method for localizing the train based on the image data or an absolute localization method for localizing the train based on the point cloud data; and a localization unit that localizes the train using the relative localization method or the absolute localization method selected by the selector. A localization method according to the present invention, which is used for localization of a train equipped with sensing devices that detect target objects surrounding the train, comprises: obtaining multimodal sensor data including image data of the target objects and point cloud data of distances from the train to the target objects detected by the sensing devices; selecting a relative localization method for localizing the train based on the image data or an absolute localization method for localizing the train based on the point cloud data; and localizing the train using the relative localization method or the absolute localization method which is selected.

[0009] According to the present invention, it is possible to improve localization of trains in railway environment.

[0010] These and other objects, advantages, purposes and features of the present invention will become apparent upon review of the following specification in conjunction with the drawings.

[0011] [Fig. 1] Figure 1 is a schematic drawing of a train mounted thereon a plurality of sensing devices and an information processing device according to an embodiment of the present invention. [Fig. 2] Figure 2 represents an example of dynamic scene with a railway environment surrounding a train running on a railway track. [Fig. 3] Figure 3 is an exemplary illustration of a static scene observed from a train standing at a station. [Fig. 4] Figure 4 represents a functional block diagram of the information processing device according to an embodiment of the present invention. [Fig. 5] Figure 5 is a flowchart illustrating the processing executed by the information processing device. [Fig. 6] Figure 6 is an example of overall railway environment in which a train is running between railway stations.

[0012] The following detailed description delineates various features and functions of the proposed invention with reference to the included figures. The illustrative system, functions and method embodiment described herein are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated here.

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0014] Figure 1 is a schematic drawing of a train mounted thereon a plurality of sensing devices and an information processing device according to an embodiment of the present invention. As shown in figure 1, a train 100 equipped with sensing devices 101a, 101b and 102, a GNSS sensor 103, an inertial measurement unit (IMU) 104, and an odometry sensor 105, runs on a railway track 10 toward a direction 20.

[0015] The sensing devices 101a, 101b and 102 each detect target objects surrounding the train 100. In particular, the sensing devices 101a and 101b are each capable of capturing images of the target objects seen from the train 100 in the direction 20. Cameras may be used as the sensing devices 101a and 101b. The sensing device 102 is capable of scanning the target objects located in front of the train 100 in the direction 20 and detecting the distances from the train 100 to the scanned target objects. 2D or 3D LiDARs may be used as the sensing device 102. Note that figure 1 shows a pair of sensing devices 101a and 101b provided in parallel toward the direction 20 is mounted in close proximity of the sensing device 102, but arrangements and the number of sensing devices mounted on the train 100 are not limited to this example.

[0016] The GNSS sensor 103 receives GNSS signals transmitted from the GNSS satellites orbiting the Earth and, based on the received GNSS signals, calculates the position of the train 100. The IMU 104 detects the current orientation of the train 100. The odometry sensor 105 detects the rotation speed of the wheel of the train 100 and, based on the detection result, calculates the distance traveled by the train 100. Based on these values obtained by the GNSS sensor 103, the IMU 104 and the odometry sensor 105, the current position of the train 100 can be estimated. Hereafter, the GNSS sensor 103, the IMU 104 and the odometry sensor 105 are collectively called as "positioning sensors".

[0017] Note that it is not necessary for the train 100 to be equipped with all of the GNSS sensor 103, the IMU 104 and the odometry sensor 105. It is acceptable that at least any one of the GNSS sensor 103, the IMU 104 and the odometry sensor 105 is mounted on the train 100. In addition, the train 100 may be equipped with neither of them in case that the position of the train 100 can be estimated based on the detection result of the target objects by the sensing devices mentioned above. In other words, the train 100 is optionally equipped with the positioning sensors.

[0018] The train 100 also has an information processing device 110. The information processing device 110 is in communication, via a wireless or wired communications interface, with the sensing devices 101a, 101b and 102, and the positioning sensors including the GNSS sensor 103, the IMU 104 and / or the odometry sensor 105. The information processing device 110 is capable of storing and processing the data acquired from these devices and sensors.

[0019] A coordinate system 120 is defined for the train 100. As shown in figure 1, the coordinate system 120 has x, y and z dimensional axes where x is longitudinal forward, y is latitudinal and z is upward direction, and the origin of these axes is at the center bottom of the train 100 body.

[0020] Figure 2 represents an example of dynamic scene with a railway environment surrounding the train 100 running on the railway track 10. In the scene 200 exemplary illustrated by Figure 2, the railway environment around the train 100 may change due to seasonal variations (e.g., leaves falling in autumn), weather conditions (e.g., heavy rain causing landslides), or even human activities (e.g., maintenance or logging). In many areas in which the train 100 is operated, railway environment surrounding the train 100 corresponds to dynamic scenes like the scene 200.

[0021] While the railway environment in the scene 200 will change dynamically, key point features of the railway environment such as the railway track 10, electric poles 202, sleepers 203 and a trackside signaling system 204 will not change. For instance, the railway track 10 can provide a continuous reference for the path of the train 100. The position of the railway track 10 is known and fixed, so it can serve as a baseline for relative positioning of the train 100. The trackside signaling system 204, such as a signal light or a sensor, communicates with the train 100 to provide information about track occupancy and upcoming conditions. The sleepers 203 are spaced at regular intervals to support the railway track 10. Their positions are known and fixed relative to the railway track 10. The electric poles 202 are often spaced regularly along the railway track 10, and their positions are known.

[0022] As explained above, the key point features including the railway track 10, the electric poles 202, the sleepers 203 and the signaling system 204 can serve as valuable reference points for calculating and tracking the relative position of the train 100 within the dynamic scene 200. By using these fixed elements as reference points, an operator and the information processing device 110 of the train 100 can monitor and manage the position and movement of the train 100 running along the railway track 10 more effectively, thereby contributing to safe and efficient railway operations. In other words, by leveraging the key point features of the railway environment including the railway track 10, the electric poles 202, the sleepers 203 and the signaling system 204, and by integrating data from various sensors output by detecting the key point features, the operator and the information processing device 110 can maintain an accurate and up-to-date estimate of a relative position of the train 100. This information is crucial for ensuring safety, efficiency, and effective management of railway operations, particularly in dynamic and changing environments.

[0023] Figure 3 is an exemplary illustration of a static scene observed from the train 100 standing at a station. In the scene 300 illustrated by Figure 3, landmarks of the station in the railway environment around the train 100 such as a platform edge 302, which is an edge of a platform 301 placed along the railway track 10 in the station, and a building 303 may not change frequently. In other words, since the physical arrangements of the platform edge 302, the building 303 and other facilities of the station are carefully planned and constructed to accommodate the anticipated needs of the railway system for many years, changes to these facilities in their positions and shapes due to layout change of the station are infrequent.

[0024] As explained above, when the train 100 is standing at a station, the landmarks of the station including the platform edge 302 and the building 303 can serve as permanent features in the static scene 300 to provide consistent reference points for train positioning. By using information of positions and shapes of these landmarks that can be observed from the train 100, localization of the train 100 in the static scene 300 may be performed precisely.

[0025] Figure 4 represents a functional block diagram of the information processing device 110 according to an embodiment of the present invention. The information processing device 110 functionally has a data obtaining unit 111, a pre-processing unit 112, a selector 113, and a localization unit 114. The information processing device 110 may realize these functional blocks by executing predetermined programs with a CPU. In addition, the information processing device 110 has databases of landmark pose data 115, key point topological map 116 and point cloud map 117. The information processing device 110 may realize the databases with non-volatile recording medium such as an HDD or an SSD.

[0026] The data obtaining unit 111 obtains the multimodal sensor data from the sensing devices 101a, 101b and 102. When the train 100 is moving in the dynamic scene 200, the multimodal sensor data including the image data of the key point features such as the railway track 10, the electric poles 202, the sleepers 203 or the signaling system 204 is transmitted from the sensing devices to the information processing device 110. On the other hand, when the train 100 is standing in the static scene 300, the multimodal sensor data including the point cloud data of distances from the train 100 to the landmarks such as the platform edge 302 or the building 303 is transmitted from the sensing devices to the information processing device 110. The data obtaining unit 111 can obtain these data from the sensing devices according to the state of motion of the train 100.

[0027] In addition, the data obtaining unit 111 can obtain the sensed data by the positioning sensors, i.e., the GNSS sensor 103, the IMU 104 and / or the odometry sensor 105. These data may be used for localization of the train 100 together with the multimodal sensor data.

[0028] The pre-processing unit 112 performs pre-processing on the multimodal sensor data obtained by the data obtaining unit 111. The pre-processing may comprise multiple processing such as filtration, noise removal, object feature retrieval, object clustering, data transformation or the like, but not limited to them. As far as necessary for use of the multimodal sensor data in the subsequent processing by the selector 113 or the localization unit 114, any processing may be performed by the pre-processing unit 112. Hereafter, the multimodal sensor data after performing the pre-processing by the pre-processing unit 112, which includes the image data of the key point features and the point cloud data of the landmarks, is also referred as the multimodal sensor data.

[0029] The selector 113 selects either one of localization methods as described above in Figures 2 and 3 based on the state of motion of the train 100 and the landmark pose data 115. The localization method explained with Figure 2, which will be called as "relative localization method" hereafter, is a method for localizing the train 100 in the dynamic scene 200 based on the image data of the key point features detected while the train 100 is moving on the railway track 10. The localization method explained with Figure 3, which will be called as "absolute localization method" hereafter, is a method for localizing the train 100 in the static scene 300 based on the point cloud data of the landmarks detected while the train 100 is standing at a station. The specific processing way to select these localization methods by the selector 113 will be described later.

[0030] The localization unit 114 localizes the train 100 using the relative localization method or the absolute localization method selected by the selector 113 based on the multimodal sensor data after pre-processing by the pre-processing unit 112. Specifically, if the relative localization method is selected by the selector 113, the localization unit 114 localizes the train 100 based on the image data of the multimodal sensor data and the key point topological map 116. The key point topological map 116 is map data indicating positional relationships of the key point features in the dynamic scene 200 along the railway track 10. On the other hand, if the absolute localization method is selected by the selector 113, the localization unit 114 localizes the train 100 based on the point cloud data of the multimodal sensor data and the point cloud map 117. The point cloud map 117 is map data indicating point cloud of the landmarks in the static scene 300 at a station. Note that the point cloud map 117 may be updated with the point cloud data obtained as the multimodal sensor data.

[0031] Figure 5 is a flowchart illustrating the processing executed by the information processing device 110. The steps of the processing of the flowchart shown in figure 5 is executed repeatedly by a predetermined processing cycle. The processing can be implemented using any suitable programming language and data structures.

[0032] In step S10, the data obtaining unit 111 obtains the multimodal sensor data from the sensing devices 101a, 101b and 102. In this processing, the sensed data by the positioning sensors (the GNSS sensor 103, the IMU 104 and / or the odometry sensor 105) may be obtained additionally to the multimodal sensor data.

[0033] In step S20, the pre-processing unit 112 performs pre-processing on the multimodal sensor data obtained at step S10. The data generated by performing the pre-processing on the multimodal sensor data is used in the subsequent processing.

[0034] In step S30, the localization unit 114 estimates the current pose of the train 100. In this processing, the localization unit 114 roughly estimates the current position and current orientation of the train 100 as the current pose of it based on, e.g., the currently-detected multimodal sensor data obtained in the current cycle of processing and the previous pose of the train 100 obtained in the previous cycle of processing. Alternatively or additionally, the sensed data by the positioning sensors may be used for estimating the current position.

[0035] In step S40, the selector 113 judges whether or not the train 100 is moving based on the current pose estimated at step S30. If it is judged that the train 100 is moving, the processing proceeds to step S80. If it is judged that the train 100 is not moving, the selector 113 decides that the train 100 is standing at a station, and then the processing proceeds to step S50. Note that if the train 100 is slowly moving at a speed lower than a predetermined speed, it may be judged in step S40 as the train 100 is not moving.

[0036] In step S50, the selector 113 judges whether or not a landmark is present at a detectable location by the sensing device 102. In this processing, the selector 113 can make the judgement based on the landmark pose data 115 and the current pose of the train 100. If there is at least one or more landmarks that can be detected by the sensing device 102 mounted on the train 100 at the current pose, and therefore the point cloud data of distances from the train 100 to the landmarks was obtained at step S10 as the multimodal sensor data, the processing proceeds to step S60. If there is no landmark that can be detected by the sensing device 102 mounted on the train 100 at the current pose, the processing proceeds to step S80.

[0037] In step S60, the selector 113 selects the absolute localization method as the localization method to be used by the localization unit 114.

[0038] In step S70, the localization unit 114 performs localization of the train 100 using the absolute localization method that has been selected at step S60. In this processing, the localization unit 114 can localize the train 100 accurately based on the point cloud data obtained at step S10 and the point cloud map 117.

[0039] In step S80, the selector 113 selects the relative localization method as the localization method to be used by the localization unit 114.

[0040] In step S90, the localization unit 114 performs localization of the train 100 using the relative localization method that has been selected at step S80. In this processing, the localization unit 114 can localize the train 100 without point cloud data based on the image data obtained at step S10 and the key point topological map 116.

[0041] After performing localization of the train 100 in step S70 or S90 and thus the current position of the train 100 has been specified, the processing of figure 5 is finished.

[0042] Figure 6 is an example of overall railway environment in which the train 100 is running between railway stations. In Figure 6, an example of situation where the train 100 departures a railway station 30a, runs along the railway track 10 and arrives at a railway station 30b.

[0043] When the train 100 is standing at the railway station 30a before departure, the absolute localization method is selected and performed by the information processing device 110. In this processing, for instance, the platform 301a of the railway station 30a is detected and the point cloud data thereof is obtained by the sensing device 102. By comparing the obtained point cloud data with the point cloud map 117, the information processing device 110 calculates the current position of the train 100.

[0044] When the train 100 is moving toward the railway station 30b, the relative localization method is selected and performed by the information processing device 110. In this processing, for instance, the railway track 10, the electric poles 202, the sleepers 203 and the trackside signaling system 204 are detected and the image data thereof is obtained by the sensing devices 101a and 101b. By comparing the obtained image data with the key point topological map 116, the information processing device 110 calculates the current position of the train 100.

[0045] When the train 100 arrived and is standing at the railway station 30b, the absolute localization method is selected and performed by the information processing device 110. In this processing, for instance, the platform 301b of the railway station 30b is detected and the point cloud data thereof is obtained by the sensing device 102. By comparing the obtained point cloud data with the point cloud map 117, the information processing device 110 calculates the current position of the train 100.

[0046] According to an embodiment of the present invention explained above, the following operations and effects are obtained.

[0047] (1) The information processing device 110 is for localization of the train 100 equipped with the sensing devices 101a, 101b and 102 that detect target objects in a railway environment surrounding the train 100. The information processing device 110 comprises the data obtaining unit 111 that obtains multimodal sensor data including image data of the target objects and point cloud data of distances from the train 100 to the target objects detected by the sensing devices 101a, 101b and 102, the selector 113 that selects the relative localization method for localizing the train 100 based on the image data or the absolute localization method for localizing the train 100 based on the point cloud data, and the localization unit 114 that localizes the train 100 using the relative localization method or the absolute localization method selected by the selector 113. By employing this configuration, it is possible to improve localization of trains in railway environment.

[0048] (2) The selector 113 selects the relative localization method or the absolute localization method based on a state of motion of the train 100. Specifically, if the train 100 is moving (step S40: YES), the selector 113 selects the relative localization method (step S80), and if the train 100 is standing (step S40: NO), the selector 113 selects the absolute localization method (step S60). By this processing, the relative localization method or the absolute localization method can be selected properly according to the state of motion of the train 100.

[0049] (3) The information processing device 110 further comprises the landmark pose data 115 indicative of positions and orientations of landmarks included in the target objects. If the train 100 is standing (step S40: NO), the selector 113 decides whether or not at least any one of the landmarks is present at a location where the sensing device 102 can detect it base on the landmark pose data 115 (step S50), and if it is decided that any of the landmarks is not present at the location (step S50: NO), the selector 113 selects the relative localization method (step S80) even though the train 100 is standing. By employing this configuration, it is possible to prevent from the absolute localization method being inappropriately selected when the sensing device 102 cannot detect any landmark.

[0050] (4) The target objects may include the key point features of the railway environment such as the railway track 10, the electric poles 202, the sleepers 203 and the trackside signaling system 204. The localization unit 114 can localize the train 100 using the relative localization method (step S90) based on the image data of the key point features and the predetermined positional relationships of the key point features indicated by the key point topological map 116. By this processing, it is possible to localize the train 100 using the relative localization method in the dynamic scene 200 without point cloud data.

[0051] (5) The target objects may include the landmark such as the platform edge 302 and the building 303. The localization unit 114 can localize the train 100 using the absolute localization method (step S70) based on the point cloud map 117 of the landmark recorded in advance and the point cloud data of the landmark currently obtained at step S10 by the data obtaining unit 111. By this processing, it is possible to localize the train 100 using the absolute localization method accurately in the static scene 300.

[0052] The embodiments and variants explained above are only examples; the present invention is not to be considered as being limited by the details thereof. Provided that the essential characteristics of the present invention are retained, other implementations are also included within the scope of the present invention.

[0053] 100: train 101a, 101b, 102: sensing device 103: GNSS sensor 104: IMU 105: odometry sensor 110: information processing device 111: data obtaining unit 112: pre-processing unit 113: selector 114: localization unit 115: landmark pose data 116: key point topological map 117: point cloud map 120: coordinate system

Claims

1. An information processing device for localization of a train equipped with sensing devices that detect target objects in a railway environment surrounding the train, comprising: a data obtaining unit that obtains multimodal sensor data including image data of the target objects and point cloud data of distances from the train to the target objects detected by the sensing devices; a selector that selects a relative localization method for localizing the train based on the image data or an absolute localization method for localizing the train based on the point cloud data; and a localization unit that localizes the train using the relative localization method or the absolute localization method selected by the selector.

2. An information processing device according to Claim 1, wherein the selector selects the relative localization method or the absolute localization method based on a state of motion of the train.

3. An information processing device according to Claim 2, wherein: if the train is moving, the selector selects the relative localization method; and if the train is standing, the selector selects the absolute localization method.

4. An information processing device according to Claim 3, further comprising landmark pose data indicative of positions and orientations of landmarks included in the target objects; wherein: if the train is standing, the selector decides whether or not at least any one of the landmarks is present at a location where the sensing devices can detect it base on the landmark pose data; and if it is decided that any of the landmarks is not present at the location, the selector selects the relative localization method even though the train is standing.

5. An information processing device according to Claim 1, wherein: the target objects include key point features of the railway environment; and the localization unit localizes the train using the relative localization method based on the image data of the key point features and predetermined positional relationships of the key point features.

6. An information processing device according to Claim 1, wherein: the target objects include a landmark; and the localization unit localizes the train using the absolute localization method based on a point cloud map of the landmark recorded in advance and the point cloud data of the landmark currently obtained by the data obtaining unit.

7. A localization method, which is used for localization of a train equipped with sensing devices that detect target objects surrounding the train, comprising: obtaining multimodal sensor data including image data of the target objects and point cloud data of distances from the train to the target objects detected by the sensing devices; selecting a relative localization method for localizing the train based on the image data or an absolute localization method for localizing the train based on the point cloud data; and localizing the train using the relative localization method or the absolute localization method which is selected.

Images