Method and device for measuring railroad space

Abstract

To allow a method and device for measuring a track space to obtain, with high accuracy, continuous three-dimensional shape information of the track space on which a railway vehicle travels.SOLUTION: Provided is a method for measuring a track space in which a shape of an object in the track space along which a railway vehicle travels is measured by a computer, The method includes the steps of: acquiring a plurality of images captured at the same timing by a plurality of imaging devices (left camera 11, middle camera 12, and right camera 13) installed on the railway vehicle; acquiring, based on the plurality of images captured at the same timing, three-dimensional shape information of a partial space (from "distance r" to "distance r+speed V×imaging interval S") in a traveling direction of the railway vehicle out of the track space (range sr) that falls within the angles of view of the plurality of imaging devices; and overlaying the three-dimensional shape information for each captured timing of the plurality of imaging devices.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a method and device for measuring the shape of an object in a track space through which a railway vehicle travels. [Background technology]

[0002] For railway operators, intrusion (obstruction) of the construction gauge due to features around the tracks is a serious issue related to train safety. Many of the features that intrude into the construction gauge are vegetation around the tracks, and if on-site inspections can be carried out frequently, advance measures can often be taken. However, it is difficult to enter on foot into the track space where commercial vehicles are running and carry out on-site inspections frequently. For this reason, the track space has been checked from a moving train. Conventionally, the following methods have been used to check the track space from a train:

[0003] (1) Visual Inspection The visual inspection method involves having a worker get on the front of the train and visually inspect the area around the train. This method depends on the worker's level of skill, can overlook areas other than those being monitored, and cannot record objective numerical values.

[0004] (2) Laser method The laser method is a method of measuring with a device called a vehicle-mounted photo laser surveying system. This device scans a laser spot light with a mirror or the like to measure an object, and measures the distance to the object as well as its position and orientation at the time of measurement using a Global Navigation Satellite System (GNSS) and an Inertial Measurement Unit (IMU), and measures the measurement position with the laser spot light as a 3D point cloud with x, y, and z coordinates. Measurements with this vehicle-mounted photo laser surveying system can continuously measure the track space, making it suitable for measuring the entire long-distance commercial line, and can measure with high and constant accuracy. However, the measurement range is wide, and point cloud data with low measurement accuracy outside the track premises is also acquired as noise, resulting in a huge amount of data. In addition, since the vehicle-mounted photo laser surveying system is large and laser light does not pass through window glass, it can only be installed on limited vehicles such as test cars and measurement cars, and cannot be installed on general commercial vehicles.

[0005] (3) Image-based method An example of an image-based method is to mount two cameras on the left and right of the driver's seat of a frontal vehicle, obtain stereo images, obtain parallax by stereo matching, and calculate the distance from the cameras (see, for example, Patent Document 1).Other methods that use two cameras while a vehicle is traveling include a method for measuring raised parts of the trackbed ballast (see, for example, Patent Document 2) and a method for detecting obstructions to the construction gauge of a railway vehicle (see, for example, Non-Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2016-192108 A [Patent Document 2] JP 2013-257258 A [Non-patent literature]

[0007] [Non-Patent Document 1] Masahiro Hayashi, Yukitaka Ariyama, Noboru Nakajima, Kyohei Kawasaki, Atsushi Shimizu, Masashi Miwa, "Train Patrol Support System Using Image Analysis to Support Railway Wayside Inspection Work," NEC Technical Report, Vol. 74, No. 1, August 2021, pp. 81-85 Summary of the Invention [Problem to be solved by the invention]

[0008] Here, we will explain a reference technique for performing 3D reconstruction using a stereo camera while a vehicle is traveling. The method of this reference technique uses images from multiple viewpoints taken by stereo cameras cl, cr installed at the front of the vehicle, as shown in Fig. 22, and reconstructs 3D using images cl(t0), cr(t0), cl(t1), and cr(t1) from a total of four viewpoints at the stereo camera positions at times t0 and t1.

[0009] When performing 3D reconstruction using images captured at different times like this, the difference between the distance to the image capture target A at time t0 and the distance to the image capture target A at time t1 (distance at t0 > distance at t1) of the stereo cameras cl, cr installed at the front of the vehicle is not taken into consideration. Therefore, the error that increases as the distance becomes longer is not reflected, and the image capture target A cannot be detected with high accuracy.

[0010] An object of the present invention is to provide a method and device for measuring a track space that can acquire, with high accuracy, continuous three-dimensional shape information of the track space through which a railway vehicle travels. [Means for solving the problem]

[0011] In one aspect, a method for measuring rail space is a method for measuring rail space using a computer to measure the shape of an object in the rail space along which a rail vehicle travels, and includes the steps of: acquiring multiple images captured at the same time by multiple imaging devices installed on the rail vehicle; acquiring three-dimensional shape information of a portion of the rail space in the direction of travel of the rail vehicle that falls within the angle of view of the multiple imaging devices based on the multiple images captured at the same time; and overlaying the three-dimensional shape information for each imaging timing of the multiple imaging devices.

[0012] In another aspect, a railway space measurement device is a railway space measurement device that measures the shape of an object in the railway space along which a railway vehicle travels, and includes an image acquisition unit that acquires multiple images captured at the same timing by multiple imaging devices installed on the railway vehicle, a three-dimensional shape information acquisition unit that acquires three-dimensional shape information of a portion of the railway space in the direction of travel of the railway vehicle, which falls within the angle of view of the multiple imaging devices, based on the multiple images captured at the same timing, and an overlay unit that overlays the three-dimensional shape information for each imaging timing of the multiple imaging devices. Effect of the Invention

[0013] According to the above aspect, continuous three-dimensional shape information of the track space on which the railway vehicle travels can be acquired with high accuracy. [Brief description of the drawings]

[0014] [Figure 1] 1 is a functional block diagram for explaining a rail space measurement device according to an embodiment; [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a rail space measurement device according to an embodiment. [Diagram 3] FIG. 2 is an explanatory diagram for explaining the size of an imaging range of one imaging element in one embodiment. [Figure 4] FIG. 2 is an explanatory diagram for explaining positional accuracy corresponding to one imaging element according to an embodiment. [Diagram 5] FIG. 13 is a diagram showing a simulation image captured by a middle camera in one embodiment. [Figure 6] FIG. 4 is an explanatory diagram for explaining a method for calculating the angle of view of each camera in an embodiment. [Figure 7] FIG. 2 is an explanatory diagram for explaining a method for calculating distances from three cameras to nearest objects that fit within the angle of view in one embodiment. [Figure 8] FIG. 2 is an explanatory diagram for explaining a method for calculating a distance to a range required for 3D reconstruction according to an embodiment. [Figure 9] FIG. 1 is an explanatory diagram (part 1) for explaining a 3D reconstruction target area after distance restriction in one embodiment. [Figure 10] FIG. 2 is an explanatory diagram (part 2) for explaining a 3D reconstruction target area after distance restriction in one embodiment. [Figure 11] FIG. 2 is an explanatory diagram for explaining coordinate transformation in one embodiment. [Figure 12] FIG. 2 is an explanatory diagram for explaining an example of a rail space according to an embodiment. [Figure 13] 1 is a flowchart for explaining a measurement method for a rail space according to an embodiment. [Figure 14] FIG. 2 is a diagram illustrating a vehicle-mounted device and a measuring device for a rail space in a first modified example of an embodiment. [Figure 15] 13 is a flowchart for explaining a measurement method for a rail space in a first modified example of an embodiment. [Figure 16] 13 is a flowchart for explaining a measurement method of a rail space in a second modified example of an embodiment. [Figure 17] FIG. 13 is an explanatory diagram for explaining a velocity calculation in a second modified example of an embodiment. [Figure 18] FIG. 1 is an explanatory diagram (part 1) for explaining a measurement result of a rail space in one embodiment. [Figure 19] FIG. 2 is an explanatory diagram (part 2) for explaining the measurement results of the rail space in one embodiment. [Figure 20] FIG. 11 is an explanatory diagram (part 3) for explaining the measurement results of the rail space in one embodiment. [Figure 21] FIG. 11 is an explanatory diagram (part 4) for explaining the measurement results of the rail space in one embodiment. [Figure 22] FIG. 11 is an explanatory diagram for explaining three-dimensional reconstruction in a reference technique. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A method and device for measuring a rail space according to an embodiment of the present invention will now be described with reference to the drawings.

[0016] FIG. 1 is a functional block diagram for explaining a rail space measurement device 1 according to the present embodiment.

[0017] The rail space measurement device 1 shown in Fig. 1 includes an image acquisition unit 2, a three-dimensional shape information acquisition unit 3, a superposition unit 4, and a position and orientation information acquisition unit 5. The rail space measurement device 1 may be installed on a railway vehicle, but is installed independently of the railway vehicle. The vehicle-mounted device 10 shown in Fig. 1 includes a left camera 11, a middle camera 12, a right camera 13, a GNSS receiver 14, an IMU 15, and a synchronization device 16, and is installed, for example, at the front of the railway vehicle. A system including the rail space measurement device 1 and the vehicle-mounted device 10 can be called a rail space measurement system.

[0018] The image acquisition unit 2 of the railway space measurement device 1 acquires a plurality of images captured at the same timing by the left camera 11, the middle camera 12, and the right camera 13 of the vehicle-mounted device 10. These images may be all or part of a still image, or may be an image extracted from a video.

[0019] The left camera 11, the middle camera 12, and the right camera 13 are examples of a plurality of imaging devices installed on a railway vehicle. The left camera 11, the middle camera 12, and the right camera 13 may be located at the same position in the traveling direction of the railway vehicle, and are arranged at equal intervals at positions having the same vertical height, for example.

[0020] According to epipolar geometry, a point in one of two images always exists on an epipolar line in the other image. Stereo matching is a method for estimating a point on an epipolar line based on the features surrounding a point in an image, but using two imaging devices results in large errors in position calculation. Therefore, from the viewpoint of reducing errors in position calculation, it is desirable to use three or more imaging devices.

[0021] It should be noted that the multiple imaging devices do not need to be at the same height in the vertical direction, and for example, multiple imaging devices may be arranged vertically, or three imaging devices may be arranged to form a triangle, and there is no limitation. Also, it is desirable for the imaging device to capture images at the front of the railcar in the direction of travel, but it may also capture images at the rear of the railcar in the direction of travel.

[0022] The three-dimensional shape information acquisition unit 3 acquires three-dimensional shape information of a part of space (a part of the track space that fits within the viewing angles of the left camera 11, the middle camera 12, and the right camera 13, with a distance restriction in the traveling direction of the railway vehicle, which will be described later, based on three images captured at the same time by the left camera 11, the middle camera 12, and the right camera 13. For example, the three-dimensional shape information acquisition unit 3 acquires point cloud data (an example of overall three-dimensional shape information) of the entire area that can be acquired as point cloud data from the three images captured at the same time by the left camera 11, the middle camera 12, and the right camera 13, using any three-dimensional restoration software of a multi-view camera method (multi-eye camera method) using SfM (Structure from Motion)-MVS (Multi-View Stereo) technology. Then, the three-dimensional shape information acquisition unit 3 extracts point cloud data (an example of three-dimensional shape information) of the part of the space from this entire point cloud data. Note that point cloud data can be called a collection of point data having three-dimensional coordinates.

[0023] The overlapping unit 4 overlaps the point cloud data of the part of the space acquired at each imaging timing of the left camera 11, the middle camera 12, and the right camera 13. The processing of the overlapping unit 4 will be described later.

[0024] The position and attitude information acquisition unit 5 acquires measured position information of the traveling railway vehicle from the GNSS receiver 14 of the positioning satellite system. The position and attitude information acquisition unit 5 also acquires attitude information of the railway vehicle, for example, from a gyro sensor of the IMU 15. The GNSS receiver 14 is an example of a position information acquisition device that acquires position information of the imaging device (railroad vehicle), and the IMU 15 is an example of an attitude information acquisition device that acquires attitude information of the imaging device (railroad vehicle). The position information and attitude information of each of the cameras 11, 12, 13 (railroad vehicle) may be acquired by any device.

[0025] The synchronization device 16 of the vehicle-mounted device 10 sends a synchronization signal to each of the cameras 11, 12, and 13, the GNSS receiver 14, and the IMU 15. For example, the synchronization device 16 issues an image capture instruction to each of the cameras 11, 12, and 13 at a predetermined timing, and causes the GNSS receiver 14 to receive position information and the IMU 15 to acquire attitude information at that image capture timing, or causes the position and attitude information acquisition unit 5 to acquire position information and attitude information linked to the image capture timing (synchronization signal).

[0026] FIG. 2 is a diagram showing an example of the hardware configuration of the rail space measurement device 1. As shown in FIG.

[0027] 2 includes a processor 21, a memory 22, an auxiliary storage device 23, an input device 24, an output device 25, a portable recording medium drive device 26 that drives a portable recording medium 28, a bus 27, and a communication device 29. The auxiliary storage device 23 and the portable recording medium 28 are each an example of a non-transitory computer-readable recording medium on which a program is recorded.

[0028] The processor 21 is, for example, one or more arbitrary processing circuits including a CPU (Central Processing Unit). The processor 21 may function as some or all of the functional components (for example, the image acquisition unit 2, the three-dimensional shape information acquisition unit 3, and the superposition unit 4) of the rail space measurement device 1 shown in FIG. 1 by expanding a program stored in the auxiliary storage device 23 or the portable recording medium 28 into the memory 22 and executing the program.

[0029] The memory 22 is, for example, any semiconductor memory such as a RAM (Random Access Memory). The memory 22 functions as a work memory for storing programs or data stored in the auxiliary storage device 23 or the portable recording medium 28 when a program is executed. The auxiliary storage device 23 is, for example, a non-volatile memory such as a hard disk or a flash memory. The auxiliary storage device 23 is mainly used for storing various data and programs.

[0030] The portable recording medium drive device 26 accommodates a portable recording medium 28. The portable recording medium drive device 26 can output data stored in the memory 22 or the auxiliary storage device 23 to the portable recording medium 28, and can also read out programs, data, and the like from the portable recording medium 28. The portable recording medium 28 is any recording medium that can be carried around. Examples of the portable recording medium 28 include an SD card, a Universal Serial Bus (USB) flash memory, a Compact Disc (CD), and a Digital Versatile Disc (DVD).

[0031] The input device 24 is a keyboard, a mouse, etc. The output device 25 is a display device, a printer, etc.

[0032] The communication device 29 is a wireless or wired communication module that communicates with some or all of the vehicle-mounted device 10 shown in Fig. 1. The communication device 29 may function as some or all of the functional components (e.g., the image acquisition unit 2, the three-dimensional shape information acquisition unit 3, and the superposition unit 4) of the rail space measurement device 1 shown in Fig. 1.

[0033] The bus 27 connects the processor 21, the memory 22, the auxiliary storage device 23, etc. so that they can transmit and receive data to one another.

[0034] The configuration of the computer 20 shown in FIG. 2 is an example of the hardware configuration of the rail space measurement device 1 shown in FIG. 1. The rail space measurement device 1 is not limited to this configuration. The rail space measurement device 1 may be a general-purpose device or a dedicated device. The rail space measurement device 1 may include, for example, a specially designed electric circuit such as an ASIC (Application Specific Integrated Circuit). Furthermore, the rail space measurement device 1 may be configured using an FPGA (Field-Programmable Gate Array).

[0035] 3, when the left camera 11 (middle camera 12 and right camera 13 as well) has a lattice-shaped imaging element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), the size of the imaging range of one imaging element in the imaging region of the left camera 11 becomes narrower as the distance from the left camera 11 to the imaging target becomes shorter. Therefore, the closer the distance to the imaging target becomes, the smaller the position error calculated from the image becomes, and the higher the position accuracy becomes.

[0036] As shown in Fig. 4, if the size (width) of one imaging element is is and the focal length of the lens is fl, the positional accuracy pa(r) at a distance r to the imaging target can be expressed as "pa(r) = r / fl × is". It can be said that the error of this positional accuracy pa(r) decreases as the distance r becomes shorter. Note that in this embodiment, the dimensions of the horizontal plane are mainly described as an example, but the same can be said for the vertical plane as well as the horizontal plane.

[0037] As shown in FIG. 5, in a simulation image captured by the middle camera 12, the angle of view (image contour) of the middle camera 12 includes the vicinity (closest vicinity) of the railroad car to the distant area including the vanishing point.

[0038] As shown in FIG. 6, the angle of view 2α of a camera (for example, the left camera 11, the middle camera 12, and the right camera 13) is calculated by the following equation: “α=tan -1 It can be calculated as ((is×c) / (2×fl)).

[0039] The area in which 3D reconstruction can be performed using the three cameras, left camera 11, middle camera 12, and right camera 13 shown in Fig. 7, is the area (the area shaded in Fig. 7) that falls within the angle of view (2α) of the three cameras 11, 12, and 13. The distance r0 from the three cameras 11, 12, and 13 to this area can be calculated by "r0=bl / (2×tan(α))" using the installation distance (distance between the cameras at both ends, installation width) bl of the three cameras 11, 12, and 13, and α of the angle of view 2α.

[0040] As shown in Figure 8, the distance r from the cameras 11, 12, 13 to the area (the area filled in in Figure 8) that falls within the angle of view (2α) of the three cameras 11, 12, 13 and includes the range sr required for 3D reconstruction (the track space described below, which is the range to be measured) can be calculated as "r = (bl + sr) / (2 × tan(α))" using the installation distance bl of the three cameras 11, 12, 13, the required range sr, and α of the angle of view 2α.

[0041] As shown in Fig. 9, consider a case where a vehicle with three cameras 11, 12, and 13 installed at the front that can take pictures synchronously at a shooting interval (imaging interval) S is moving on a track at a speed V, and images are taken continuously at times t1, t2 (=t1+S), and so on. In this case, in order to perform 3D reconstruction using multi-viewpoint images with a certain degree of accuracy, for example, when using three images taken at time t1, it is necessary to perform 3D reconstruction using multi-viewpoint images for an area surrounded by distance r(t1) and distance r(t2)=r(t1)+V×S (area filled in Fig. 9). This area can be said to be a space limited by a predetermined distance from the end of the space on the side of the cameras 11, 12, and 13 in the traveling direction of the railway vehicle, out of the track space that falls within the angle of view of the cameras 11, 12, and 13.

[0042] In order to perform 3D reconstruction of an area as close as possible to the cameras 11, 12, and 13 in order to reduce the difference in the position accuracy pa(r) shown in FIG. 4 during continuous recording at times t0 to tn, it is necessary to reduce the distance from the distance r to the distance r+V×S. Since the speed V of the railroad car is not constant, V×S is also not constant. In order to keep V×S constant, the shooting interval S may be varied according to the speed V. In addition, in FIG. 9, the horizontal cross section of the 3D reconstruction area at time t1 is represented by a trapezoid, but the horizontal cross section of the 3D reconstruction area may be a rectangular parallelepiped shape so that the width perpendicular to the traveling direction of the railroad car falls within the range sr. In this specification, shooting is used in the same sense as imaging, that is, to take an image.

[0043] As also shown in Figure 10, in order to perform point cloud measurement (acquisition of three-dimensional shape information) using three-dimensional reconstruction calculations with a multi-view camera method using three cameras 11, 12, and 13 installed in front of the vehicle for the track space (range sr) on which a railway vehicle travels, it is necessary to determine the installation distance bl of the three cameras 11, 12, and 13, the running speed V, the shooting interval S, the camera angle of view 2α, and the image sensor size is so that the difference in position accuracy pa(r) due to the distance r from the camera at the time of shooting is small.

[0044] In addition, to perform point cloud measurement on the track space (range sr) on which the railway vehicle runs, as shown in FIG. 11, it is necessary to repeatedly perform point cloud measurement including coordinate conversion to convert a point cloud obtained in camera coordinates cx, cy, cz (one example of coordinate values ​​on a first coordinate system based on the inside of the railway vehicle) based on one of three cameras 11, 12, 13 installed in front of the vehicle (for example, the middle camera 12) into world coordinates x, y, z (one example of coordinate values ​​of a second coordinate system corresponding to the geodetic system) of the world geodetic system that can represent the linearity of the track, and to superimpose the obtained point cloud with the coordinate values ​​of the world coordinates. The conversion from the camera coordinates to the world coordinates can be performed based on the position and attitude information of the railway vehicle obtained from the GNSS receiver 14 and the IMU 15. Note that the coordinate values ​​of the second coordinate system corresponding to the geodetic system may be other coordinate values ​​having a reference that does not change due to the movement of the railway vehicle, such as the coordinate values ​​of the Japanese Geodetic System.

[0045] Here, a method for measuring a track space, which measures the shape of the track space including the construction gauge, will be described, as shown in Fig. 12. Note that in this embodiment, the entire specified track space including the construction gauge and the margin (the entire surface perpendicular to the traveling direction of the railway vehicle) is measured, but it is also possible to measure only one of the lower, left, right, upper, and center parts (construction gauge) in the track space. In other words, the range sr required for the above-mentioned three-dimensional reconstruction is not limited to the entire track space, and may be, for example, one of the lower, left, right, upper, and center parts of the track space.

[0046] As shown in FIG. 13, first, the synchronizer 16 of the vehicle-mounted device 10 shown in FIG. 1 transmits synchronization signals to the cameras 11, 12, 13, the GNSS receiver 14, and the IMU 15 at the above-mentioned shooting interval S (step S11).

[0047] The cameras 11, 12, and 13 perform synchronous shooting at the same timing based on the synchronization signal (step S12). The image acquisition unit 2 acquires the images captured by the cameras 11, 12, and 13 (left camera image, center camera image, and right camera image).

[0048] The GNSS receiver 14 measures the position information of the railcar (camera) by receiving it based on the synchronization signal, and the IMU 15 measures the attitude information of the railcar (camera) based on the synchronization signal (step S13). Then, the position and attitude information acquisition unit 5 acquires the position and attitude information of the camera.

[0049] The 3D shape information acquisition unit 3 performs 3D reconstruction from multi-viewpoint images using the left camera image, middle camera image, and right camera image taken at the same timing (step S14), and acquires point cloud data (an example of overall 3D shape information) of the entire area that can be acquired as point cloud data in the camera coordinates shown in Figure 11 above.

[0050] Next, the three-dimensional shape information acquisition unit 3 extracts three-dimensional point cloud data of an area from the distance r to the area including the range sr (railway space) required for three-dimensional reconstruction shown in Figs. 8 to 10 as a distance restriction from the reference cameras 11, 12, and 13 up to a distance r+V×S (step S15). This allows a three-dimensional point cloud (an example of three-dimensional shape information of a part of space) in the camera coordinates after the distance restriction to be extracted. Note that instead of acquiring point cloud data of the entire acquireable area as described above (step S14) and then extracting point cloud data of the area subject to distance restriction (step S15), point cloud data of only the area subject to distance restriction may be acquired in advance.

[0051] Next, the superposition unit 4 performs coordinate conversion of the extracted 3D point cloud in the camera coordinates after the distance restriction into world coordinates capable of expressing the line route configuration (step S16), using the "camera position and orientation" of the reference camera obtained by the above-mentioned position and orientation measurement synchronized with the shooting (step S13).

[0052] The overlapping unit 4 adds the "3D point cloud in world coordinates" to the previously calculated 3D point cloud and overlaps them (step S17). Note that, in order to prevent the measurement area from being lost due to the occurrence of the above-mentioned V×S error, the 3D restoration area after the distance limit may be made larger in the traveling direction of the railway vehicle than the area from the distance r to the distance r+V×S, and a process may be performed to remove the overlapping parts when overlapping the 3D point clouds.

[0053] Then, the above-mentioned procedure from transmission of the synchronization signal (step S11) to superimposition (step S17) is repeated in the track space of the measurement range along with the movement of the railcar, thereby calculating a point cloud of the track space in world coordinates. After this, if it is determined that the shape of a feature (object) is included based on the difference between the obtained point cloud and a specified point cloud (for example, a point cloud that does not include features) or based on the existence of a point cloud, the user may be notified of the intrusion of the feature along with the position information (coordinates, etc.) of the feature. Note that examples of causes of the intrusion of a feature include deformation of utility poles, traffic lights, platform limits, tracks, roadbeds, etc., and growth of vegetation around the track.

[0054] In the above description, the process of acquiring images and position / posture information (steps S11-13) and the subsequent processes of 3D restoration, distance restriction, coordinate conversion, and overlay (steps S14-S17) are performed consecutively, but the process of 3D restoration, distance restriction, coordinate conversion, and overlay may be performed after the image and position / posture information are acquired within the travel range of the railcar. This example will be described below as a first modified example with reference to Figs. 14 and 15.

[0055] <First Modification> FIG. 14 is a diagram showing a vehicle-mounted device 30 and a rail space measuring device 1 in a first modified example of the present embodiment.

[0056] FIG. 15 is a flowchart for explaining the method for measuring the rail space in the first modified example.

[0057] The track space measurement device 1 shown in Fig. 14 can be similar to the above-described track space measurement device 1 shown in Fig. 1. The vehicle-mounted device 30 shown in Fig. 14 includes a temporary memory device 31 in addition to the components of the vehicle-mounted device 10 shown in Fig. 1 (the left camera 11, the middle camera 12, the right camera 13, the GNSS receiver 14, the IMU 15, and the synchronizer 16).

[0058] The temporary storage device 31 stores images captured by the left camera 11, the middle camera 12, and the right camera 13, position information of the railway vehicle (camera) received by the GNSS receiver 14, and attitude information of the railway vehicle (camera) acquired by the IMU 15. These images, position information, and attitude information are linked to a synchronization signal (e.g., a transmission time) transmitted by the synchronization device 16, and are stored so that images of the same timing can be identified.

[0059] In addition, the temporary memory device 31 repeatedly takes images of the railway vehicle and acquires its position and attitude information while it is traveling, and then transmits the images and position and attitude information to the image acquisition unit 2 and position and attitude information acquisition unit 5 of the railway space measurement device 1 in response to a request from the railway space measurement device 1 or at any arbitrary timing.

[0060] The temporary storage device 31 may be configured, for example, by a hard disk, a non-volatile memory such as a flash memory, a processor such as a CPU, a semiconductor memory such as a RAM, a communication module, and the like.

[0061] When measuring the track space, as shown in FIG. 15, first, the synchronization device 16 of the vehicle-mounted device 30 shown in FIG. 14 sends synchronization signals to each of the cameras 11, 12, 13, the GNSS receiver 14, and the IMU 15 at the above-mentioned shooting interval S (step S21).

[0062] The cameras 11, 12, and 13 perform synchronous shooting at the same timing based on the synchronization signal (step S22).

[0063] Furthermore, based on the synchronization signal, the GNSS receiver 14 receives and measures the position information of the railway vehicle (camera), and the IMU 15 measures the attitude information of the railway vehicle (camera) (step S23).

[0064] Then, the temporary storage device 31 repeatedly stores each image and the position and orientation information in association with the synchronization signal along with the movement of the railway vehicle in the track space within the measurement range (step S24).

[0065] After the railway vehicle has started running, the image acquisition unit 2 acquires from the temporary memory device 31 images captured by each camera 11, 12, 13 and linked to the synchronization signal, and the position and orientation information acquisition unit 5 acquires from the temporary memory device 31 the position and orientation information of the cameras linked to the synchronization signal (step S25).

[0066] The subsequent processes from the 3D reconstruction process (step S26) to the overlapping process (step S29) in the multi-viewpoint images can be similar to the processes in steps S14 to S17 shown in FIG.

[0067] <Second Modification> In this second modified example, an example will be described in which camera position and attitude information required for coordinate conversion from camera coordinates to world coordinates cannot be acquired from the GNSS receiver 14 and the IMU 15. Examples of cases in which position information and attitude information cannot be acquired include a case in which the GNSS receiver 14 cannot receive position information while the railway vehicle is traveling in a tunnel, and a case in which an information acquisition error occurs from the GNSS receiver 14 and the IMU 15 of the position and attitude information acquisition unit 5.

[0068] As shown in FIG. 16, first, the synchronizer 16 sends a synchronization signal to each of the cameras 11, 12, and 13 at the above-mentioned shooting interval S (step S31).

[0069] The cameras 11, 12, and 13 perform synchronous shooting at the same timing based on the synchronization signal (step S32). The image acquisition unit 2 acquires the images captured by the cameras 11, 12, and 13 (left camera image, center camera image, and right camera image).

[0070] The 3D shape information acquisition unit 3 performs 3D reconstruction using multi-viewpoint images using the left camera image, middle camera image, and right camera image captured at the same time (step S33), and acquires point cloud data (an example of overall 3D shape information) of the entire area that can be acquired as point cloud data in the camera coordinates shown in Figure 11 above.

[0071] Next, the three-dimensional shape information acquisition unit 3 extracts three-dimensional point cloud data of an area from the distance r to the distance r+V×S from the reference cameras 11, 12, 13, which includes the range sr (railway space) required for three-dimensional reconstruction shown in Figures 8 to 10, as a distance restriction (step S34). This allows a three-dimensional point cloud (an example of three-dimensional shape information of a part of the space) in the camera coordinates after the distance restriction to be extracted.

[0072] In the second modified example, since position and attitude information cannot be acquired from the GNSS receiver 14 and the IMU 15, the superposition unit 4 calculates the speed of the railway vehicle (step S35). As shown in Fig. 17, this speed calculation is carried out by performing 3D reconstruction from each of the multiple camera images at time t1 and the multiple camera images taken immediately before at time t0, and calculating the reference camera position (xt0, yt0, zt0) at time t0 based on the coordinates of the camera that is the reference at time t1 from the obtained 3D point cloud, thereby calculating the camera position difference between time t1 and time t0, and dividing the calculated position difference by the interval (t1-t0) between the shooting times to obtain the speed V at time t1.

[0073] The superimposing unit 4 also acquires position and orientation information based on the calculated speed, the position in world coordinates according to the line configuration, and the photographing orientation according to cant, gradient, etc. Here, the alignment of the track is designed and constructed as a linear form consisting of a total distance (kilometers), straight lines, transition curves, circular curves, etc., based on the ministerial ordinance that prescribes technical standards for railways. Therefore, by having the position and orientation information acquiring unit 5 acquire line configuration information, it is possible to acquire position and orientation information based on changes in position and orientation since the previous time position information was acquired.

[0074] Next, the superimposing unit 4 performs coordinate conversion of the extracted three-dimensional point group in the camera coordinates after the distance restriction into world coordinates using the above-mentioned position and orientation information (step S37).

[0075] Furthermore, the overlapping unit 4 adds the "three-dimensional point cloud in world coordinates" to the three-dimensional point cloud calculated previously and overlaps them (step S38).

[0076] Moreover, the overlapping unit 4 stores the captured image so as to use it for speed calculation at the next imaging timing (step S39).

[0077] Then, the above-mentioned procedure from the synchronization signal transmission process (step S31) to the image storage process (step S39) is repeated together with the movement of the railway vehicle in the track space of the measurement range, whereby a point cloud of the track space in world coordinates is calculated.

[0078] In the flowchart shown in Figure 16, position and attitude information at all imaging timings is obtained based on the speed and line shape, but for imaging timings at which position and attitude information can be obtained from the GNSS receiver 14 and IMU 15, acquisition based on the speed and line shape can be omitted.

[0079] Also, in this second variant, similarly to the first variant described above, images may be acquired (discontinuously) within the range in which the railway vehicle is traveling, and then the following processes may be performed: acquisition of position and attitude information, 3D restoration, distance limitation, coordinate transformation, overlay, and image storage.

[0080] <Measurement results of track space> Hereinafter, the measurement results of the line space in this embodiment will be described with reference to FIGS.

[0081] Assuming operation on conventional railway lines, if the shooting interval during continuous shooting is 1 / 120 seconds and the speed V of the railway vehicle is 130 km / h as shown in FIG. 18, the above-mentioned V×S is 0.3 m.

[0082] Similarly, assuming operation on conventional railways, the necessary point cloud acquisition range sr shown in Fig. 10 is sr = 5.8m, which is the railway construction limit of 3.8m + 1m range on the left and right, the installation distance bl of the three cameras installed in front of the vehicle is bl = 1m (0.5 x 2), the size of one pixel of the camera used is 0.0015mm, and the angle of view 2α is 55°, the nearby distance r to acquire the range sr, which is the necessary shooting range, is 6.55m. Furthermore, the 3D reconstruction area (r to r + V x S) after the distance restriction considering the above-mentioned shooting interval V x S = 0.3m is 6.55m to 6.85m, and the position accuracy pa (r) in the 3D reconstruction range is 3.14mm to 3.63mm.

[0083] The images captured under the above conditions are the simulated images captured by the front camera shown in Fig. 19(a) to (c). Fig. 20 shows a point cloud obtained by limiting the distance from the camera to the subject during 3D reconstruction of the multi-viewpoint images using the simulated images. Fig. 21 shows a point cloud obtained by superimposing the point cloud on the world coordinates.

[0084] In the present embodiment described above, the method for measuring the track space is a method for measuring the shape of an object in the track space along which a railway vehicle runs by using a computer (e.g., computer 20), and includes a step of acquiring multiple images captured at the same timing by multiple imaging devices (e.g., left camera 11, middle camera 12, and right camera 13) installed on the railway vehicle (acquiring images by step S12 in Figure 11), a step of acquiring three-dimensional shape information (e.g., point cloud data) of a portion of the track space (e.g., range sr) in the direction of travel of the railway vehicle that falls within the angle of view of the multiple imaging devices based on the multiple images captured at the same timing (step 15), and a step of overlaying the 3D shape information for each imaging timing of the multiple imaging devices (step S17).

[0085] From another perspective, the railway space measurement device 1 is a railway space measurement device 1 that measures the shape of an object in the railway space along which a railway vehicle runs, and is equipped with an image acquisition unit 2 that acquires multiple images captured at the same timing by multiple imaging devices (e.g., left camera 11, middle camera 12, and right camera 13) installed on the railway vehicle, a three-dimensional shape information acquisition unit 3 that acquires three-dimensional shape information (e.g., point cloud data) of a portion of the space in the direction of travel of the railway vehicle (e.g., the area from distance r to distance r+V×S) within the angle of view of the multiple imaging devices based on the multiple images captured at the same timing, and an overlay unit 4 that overlays the 3D shape information for each imaging timing of the multiple imaging devices.

[0086] From another perspective, the program is a program for measuring the shape of an object in the track space along which a railway vehicle travels, and causes a computer (e.g., computer 20) to realize the following functions: acquiring multiple images captured at the same timing by multiple imaging devices (e.g., left camera 11, middle camera 12, and right camera 13) installed on the railway vehicle; acquiring three-dimensional shape information (e.g., point cloud data) of a portion of the track space (e.g., range sr) that falls within the angle of view of the multiple imaging devices in the direction of travel of the railway vehicle (e.g., the area from distance r to distance r+V×S) based on the multiple images captured at the same timing; and overlaying the three-dimensional shape information for each imaging timing of the multiple imaging devices.

[0087] In these track space measurement method, track space measurement device 1, and program, a part of the space is 3D restored using multiple images captured at the same time, and this part of the space is limited to an area with a fixed distance from the imaging device, and the obtained 3D shape information is superimposed, so that continuous 3D shape information of the track space can be obtained with high accuracy compared to a case where 3D restoration is performed using images with different distances. In addition, since the measurement can be performed using captured images rather than visual inspection or laser, the track space can be easily confirmed using 3D shape information such as a point cloud, compared to a case where an operator visually inspects the track space or a case where a vehicle-mounted photo laser surveying system that is difficult to install is used. Therefore, according to this embodiment, it is easy to predict the intrusion of a feature into the track space where a railroad vehicle is running.

[0088] In addition, in this embodiment, the above-mentioned portion of space before superimposition from which three-dimensional shape information is acquired is a space limited by a predetermined distance from the end of the space on the side of the multiple imaging devices in the direction of travel, out of the track space that falls within the angle of view of the multiple imaging devices.

[0089] This makes it possible to obtain 3D shape information of an area close to the imaging device with little error, and to overlay the obtained 3D shape information of multiple point clouds, etc. This makes it possible to predict the intrusion of features into the railroad space with higher accuracy.

[0090] Furthermore, in this embodiment, in the process of acquiring three-dimensional shape information of a portion of the space (three-dimensional shape information acquisition unit 3), the three-dimensional shape information is acquired in coordinate values ​​on a first coordinate system based on the interior of the railway vehicle (e.g., coordinate values ​​in camera coordinates), and in the process of overlaying the three-dimensional shape information (overlay unit 4), the coordinate values ​​of the first coordinate system are converted to coordinate values ​​of a second coordinate system corresponding to the geodetic system (e.g., coordinate values ​​in world coordinates, which are coordinate values ​​of the World Geodetic System), and the three-dimensional shape information of the portion of the space is overlaid based on the coordinate values ​​of the second coordinate system.

[0091] This allows the 3D shape information of the above-mentioned part of the space to be accurately overlaid with coordinate values ​​in a coordinate system corresponding to the geodetic system, making it possible to continuously display features in the railway space in a coordinate system corresponding to the geodetic system, and to accurately grasp the positions of features on site. [Explanation of symbols]

[0092] 1. Track space measurement device 2. Image acquisition section 3 3D shape information acquisition section 4. Overlapped section 5 Position and orientation information acquisition unit 10 Vehicle-mounted equipment 11 Left Camera 12. Medium Camera 13 Right Camera 14 GNSS receiver 15 IMU 16 Synchronous Device 20. Computers 30 Vehicle-mounted equipment 31 Temporary storage

Claims

1. A method for measuring a rail space by using a computer to measure the shape of an object in the rail space in which a rail vehicle runs, comprising the steps of: acquiring a plurality of images captured at the same time by a plurality of imaging devices installed in the railway vehicle; acquiring three-dimensional shape information of a portion of the track space that falls within angles of view of the imaging devices in a traveling direction of the railway vehicle based on the plurality of images captured at the same timing; superimposing the three-dimensional shape information for each imaging timing of the plurality of imaging devices; A method for measuring a rail space, comprising:

2. The part of the space is a space limited by a predetermined distance from an end of the space on the side of the imaging devices, out of the line space that falls within the angle of view of the imaging devices.

2. The method for measuring a rail space according to claim 1.

3. In the step of acquiring the three-dimensional shape information, the three-dimensional shape information is acquired as coordinate values ​​in a first coordinate system based on an interior of the railway vehicle; In the step of superimposing the three-dimensional shape information, coordinate values ​​of the first coordinate system are converted into coordinate values ​​of a second coordinate system corresponding to a geodetic system, and the three-dimensional shape information is superimposed based on the coordinate values ​​of the second coordinate system.

2. The method for measuring a rail space according to claim 1.

4. A measurement device for a railroad space that measures the shape of an object in a railroad space in which a railroad vehicle runs, comprising: an image acquisition unit that acquires a plurality of images captured at the same time by a plurality of imaging devices installed in the railway vehicle; a three-dimensional shape information acquisition unit that acquires three-dimensional shape information of a portion of the track space that falls within angles of view of the imaging devices in a traveling direction of the railway vehicle based on the plurality of images captured at the same timing; an overlapping unit that overlaps the three-dimensional shape information for each imaging timing of the plurality of imaging devices; A measuring device for a rail space comprising:

Images