Track information generating device, track information generating method and program

Abstract

To provide a rail track information generation device for generating rail track information specifying a search range of obstacle detection without any manpower.SOLUTION: A rail track information generation device of an embodiment includes: an acquisition section for acquiring positioning information showing a position of a railway vehicle for each point of a rail track based on a positioning signal received by a positioning device installed in the railway vehicle from a satellite through traveling of the railway vehicle on a pair of right / left rails, and image information obtained by capturing images in front of the railway vehicle for each point of the rail track imaged by an imaging device installed on the railway vehicle; a first generation section for generating first position information showing a center position of the right / left rails for each point of the rail track based on the positioning information; a second generation section for generating second position information showing positions of the right / left rails for each point of the rail track based on the first position information; a third generation section for generating third position information showing a position in the image information from the second position information; and a construction section for registering the generated second position information in the rail track information regarding the rail track stored in a storage device.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a track information generating device, a track information generating method, and a program. [Background technology]

[0002] In recent years, there has been a demand for automated driving of railway vehicles. To realize automated driving of railway vehicles, technology for monitoring the road ahead of the moving train is important, and it is required to detect obstacles, for example, 250 to 350 meters away. It is also necessary to detect obstacles within a range of 40 cm from the rails. For this reason, a technology is known that detects obstacles by setting an obstacle detection area based on the rails in an image captured by a forward monitoring camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-164013 [Patent Document 2] Patent Publication No. 2021-018395 [Patent Document 3] International Publication No. 2021 / 044564 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in outdoor environments, the visibility of rails changes depending on the time of day and the environment, such as the weather and season, and the illuminance fluctuates greatly.In addition, the monitoring distance is wide, ranging from 250 to 350 m, making it difficult to consistently detect rails in captured images. [Means for solving the problem]

[0005] A track information generating device according to an embodiment is a track information generating device mounted on a railway vehicle, and includes: an acquisition unit that acquires positioning information indicating the position of the railway vehicle at each point on a route based on positioning signals from artificial satellites received by a positioning device installed on the railway vehicle as the railway vehicle travels on a pair of left and right rails; and image information of images of the area ahead of the railway vehicle at each point on the route captured by an imaging device installed on the railway vehicle; a first generation unit that generates first position information indicating a center position of the left and right rails at each point on the route based on the positioning information; a second generation unit that generates second position information indicating the positions of the left and right rails at each point on the route based on the first position information; a third generation unit that generates third position information indicating a position in the image information from the second position information; and a construction unit that registers the generated second position information in track information related to the track stored in a storage device. the third generation unit comprises a first estimation unit that estimates an attitude of the imaging device, and a coordinate transformation processing unit that performs coordinate transformation on the second position information of a three-dimensional coordinate system based on the estimated attitude of the imaging device and parameters of the imaging device to generate fourth position information of the three-dimensional coordinate system of the imaging device, and performs coordinate transformation on the fourth position information to generate the third position information of a two-dimensional coordinate system; the construction unit associates the third position information with the fourth position information and registers them in the railway information; the acquisition unit further acquires acceleration information for each of the points detected by an inertial sensor that detects acceleration of the railway vehicle; the railway information generation device further comprises a second estimation unit that estimates land gradient information indicating a gradient for each of the points based on the acceleration information; and the coordinate transformation processing unit performs coordinate transformation on the fourth position information based on the land gradient information for each of the points to generate the third position information. . [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing the outline of the configuration of a track information generating system according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram showing an example of a route on which a railway vehicle according to an embodiment runs. [Figure 3] FIG. 3 is a schematic diagram illustrating an example of the structure of the map DB according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the structure of the line DB according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of two-dimensional point sequence data according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of land gradient information. [Figure 7] FIG. 7 is a diagram illustrating an example of functional blocks of the track information generating device according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of image information in which a sequence of points representing left and right rail position information and a sequence of points representing position information of the centers of the left and right rails are projected in the embodiment. [Figure 9]FIG. 9 is a flowchart illustrating an example of a procedure of a line information generation process according to the embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of a procedure for basic data collection processing according to the embodiment. [Figure 11] FIG. 11 is a flowchart illustrating an example of a procedure of the intermediate data generation process according to the embodiment. [Figure 12] FIG. 12 is a flowchart illustrating an example of a procedure of the configuration data generation process according to the embodiment. [Figure 13] FIG. 13 is a schematic diagram for explaining a process for generating position information of the left and right rails from position information of the centers of the left and right rails in the embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of attitude information of the camera according to the embodiment. [Figure 15] FIG. 15 is a diagram showing the flow of coordinate transformation in the embodiment. [Figure 16] FIG. 16 is a diagram showing an example of coordinate transformation by perspective projection from three-dimensional left and right rail position information in the camera coordinate system to two-dimensional left and right rail position information in the image coordinate system when there is an uphill gradient in the embodiment. [Figure 17] FIG. 17 is a diagram showing an example of coordinate transformation by perspective projection from three-dimensional left and right rail position information in the camera coordinate system to two-dimensional left and right rail position information in the image coordinate system when there is a downhill gradient in the embodiment. [Figure 18] FIG. 18 is a flowchart illustrating an example of a procedure for the configured data correction process according to the embodiment. [Figure 19] FIG. 19 is a diagram showing an example of a screen of an adjustment application displayed on a display device in the embodiment. [Figure 20] FIG. 20 is a diagram showing an example of a screen of the adjustment application on which a sequence of points P2 of corrected left and right rail position information is superimposed in the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, a track information generating device, a track information generating method, and a program according to an embodiment will be described in detail with reference to the accompanying drawings.

[0008] (Embodiment) FIG. 1 is a block diagram showing the outline of the configuration of a track information generating system according to an embodiment. The track information generating system 10 according to this embodiment is mounted on a railway vehicle 1, and mainly includes an antenna 300, a positioning device 310, a camera 400, a track information generating device 100, an inertial sensor 600, a display device 700, a storage device 500, and an on-board device 200. Here, a railway vehicle may be referred to as a vehicle. In this embodiment, the vehicle is a one-car train, but is not limited to this and may be a two-or-more-car train. The display device 700 is an example of an output device.

[0009] Antenna 300 is installed near the center of the width of railcar 1 and receives radio signals transmitted from positioning satellites. The radio signals are signals that include information for measuring the position of railcar 1.

[0010] The positioning device 310 is mounted on the railway vehicle 1 and is capable of measuring the position of the railway vehicle 1 based on radio wave signals received by the antenna 300 from artificial satellites. The position of the railway vehicle 1 is determined as position information near the center of the left and right rails of the track. The positioning device 310, for example, constitutes a GNSS (Global Navigation Satellite System), performs positioning (satellite positioning) of the railway vehicle 1 based on radio waves received by the antenna 300, and outputs positioning information as the measurement result to the track information generating device 100 and the on-board device 200.

[0011] Here, GNSS is a general term for satellite positioning systems such as the US GPS, Japan's Quasi-Zenith Satellite System (QZSS), Russia's GLONASS (GLO), the European Union's Galileo (GAL), and China's Beidou (BDS). Therefore, the receiver that constitutes the antenna 300 and the positioning device 310 is configured to be capable of performing positioning using any of these satellite positioning systems.

[0012] The inertial sensor 600 is a self-contained sensor such as a three-axis acceleration sensor, a three-axis gyro sensor, a three-axis geomagnetic sensor, etc. The inertial sensor 600 outputs output data of each sensor to the rail information generation device 100.

[0013] The camera 400 is provided at the front of the railway vehicle 1 and captures an image ahead of the railway vehicle 1. The camera 400 sends captured image information to the track information generating device 100. The camera 400 is an example of an imaging device.

[0014] The display device 700 displays various data from the track information generating device 100. The display device 700 is, for example, a monitor, etc. The display device 700 is an example of an output device.

[0015] The storage device 500 is, for example, a storage medium such as an HDD (Hard Disc Drive) or an SSD (Solited State Drive). A map database 510 (referred to as "map DB 510") is stored in the storage device 500. A railroad database 511 (referred to as "railroad DB 511") is registered in the map DB 510. Details of the map DB 510 and the railroad DB 511 will be described later.

[0016] The track information generating device 100 generates a track database 511, which is part of a map DB 510 in the storage device 500, based on the position of the railway vehicle 1 and image data from the camera 400. The track information generating device 100 will be described in detail later.

[0017] The on-board device 200 is a device that controls the running of the railway vehicle 1 based on the position of the railway vehicle 1 measured by the positioning device 310 and the map DB 510. Examples of the on-board device 200 include a train control management system (TCMS), a driver assistance control device for automatic driving, and an on-board monitor.

[0018] FIG. 2 is a schematic diagram showing an example of a route on which a railway vehicle 1 according to an embodiment travels. As shown in FIG. 2, the railway vehicle 1 travels on a rail R from a departure station to a destination station. Landmarks L1, such as signs, are installed around the rail R. A detection target is determined for each set point P1 on the rail R. The set points P1 are set at predetermined intervals on the rail R. The intervals between the set points P1 are not limited to being constant and may vary depending on the location.

[0019] The map DB 510 contains information on a map of an area including the route of the railway vehicle 1 . Fig. 3 is a schematic diagram showing an example of the structure of a map DB 510 according to an embodiment. As shown in Fig. 3, the map DB 510 is composed of multiple layers. In the example of Fig. 3, the map DB 510 is composed of a layer in which basic map data at a 1 / 2500 scale from the Geospatial Information Authority of Japan is registered, a layer in which detection area sections and detection area ranges are registered, and a layer in which information on civil engineering structures such as tunnels and bridges is registered. A rail DB 511 is registered in one of these multiple layers.

[0020] It should be noted that these pieces of information are merely examples, and the map DB 510 may not have all of these pieces of information, or may have other pieces of information.

[0021] The track DB 511 is a database in which information about tracks is set for each point. The information about tracks includes point sequence data about track centerlines at the track unit level and section level.

[0022] FIG. 4 is a diagram illustrating an example of the structure of the line DB 511 according to the embodiment. As shown in FIG. 4, the track DB 511 associates set point identification information, set point position information, location information, station information, rail information, landmark information, land elevation information, and railroad crossing information for each set point P1.

[0023] The set point identification information is a number for identifying each set point P1. The set point position information is information indicating the position of the point where the set point P1 is set. For example, the set point position information is information such as the latitude, longitude, and altitude of the target set point P1.

[0024] The location information is information relating to the location of the corresponding set point P1. The location information includes information such as the location, rail turnout, terrain topography, presence or absence of landmarks, and presence or absence of railroad crossings. The location is information indicating whether the location is near a station or between stations. The rail turnout is information indicating whether a turnout is installed on the rail R. The terrain topography is information indicating whether the land at the point where the set point P1 is set is flat, uphill, or downhill. The presence or absence of landmarks is information indicating whether there are landmarks L1. The presence or absence of railroad crossings is information indicating whether there are railroad crossings.

[0025] The station information is information that is set when the corresponding set point P1 is a station. The station information includes a number for identifying the station, a direction indicating the traveling direction of the railway vehicle 1 at the station, and the like.

[0026] The rail information is information relating to the rail R. The rail information includes detailed information, two-dimensional data, and three-dimensional data. The detailed information includes the type of rail R, the number of branches of the rail R, the direction of travel of the railway vehicle 1 at multiple branches of the rail R, and the shape of the rail R, such as a straight line, a right curve, or a left curve. The two-dimensional data is information indicating the shape of the rail R in the captured image information. The two-dimensional data includes the number of nodes and coordinates that represent the position of each node N1 in two dimensions. The three-dimensional data is information indicating the shape of the rail R in the range image. The two-dimensional data includes the number of nodes and coordinates that represent the position of each node N1 in three dimensions.

[0027] Here, the two-dimensional data is left and right rail position information consisting of two-dimensional point sequence data in an image coordinate system (described later). Also, the three-dimensional data is left and right rail position information consisting of three-dimensional point sequence data in a camera coordinate system (described later). The two-dimensional data is an example of third position information. The three-dimensional data is an example of fourth position information.

[0028] Fig. 5 is a diagram showing an example of two-dimensional point sequence data in an embodiment. Fig. 5 shows point sequence data at one point P1. As shown in Fig. 5, nodes N1 are points that indicate the shape of the rail R. Coordinates are information that indicates the position of each node N1. By representing the shape of the rail R as points, two-dimensional data and three-dimensional data can be managed more easily than when the shape of the rail R1 is represented by lines.

[0029] The landmark information is information relating to the landmark L1 to be detected at the corresponding set point P1. The landmark information includes the number, type, size, and coordinates. The number is information indicating the number of landmarks L1 to be detected at the corresponding set point P1. The type is information indicating the type of landmark L1 to be detected, such as a left curve, a right curve, an uphill slope, a downhill slope, or the distance to the station. The size is information indicating the size of the landmark L1 in the captured image. The coordinates are information such as coordinates indicating the position where the landmark L1 exists in the captured image. For example, when the position where the landmark L1 exists is represented by a rectangle circumscribing the landmark L1 in the captured image, the coordinates are coordinates indicating the upper right and lower left, etc.

[0030] The land gradient information is information that indicates the gradient of the land at the corresponding set point P1. The land gradient information includes a first gradient, a second gradient, and a gradient change point.

[0031] FIG. 6 is a diagram showing an example of land gradient information. The first gradient is information indicating the gradient of a first area A1 from the railway vehicle 1 to the gradient change point in the captured image. For example, the first gradient is information indicating the elevation change per 1000 meters of horizontal distance. The second gradient is information indicating the gradient of a second area A2 that is farther away from the gradient change point in the captured image. For example, the second gradient is information indicating the elevation change per 1000 meters of horizontal distance. The gradient change point is a change point C1 where the gradient changes in the direction of travel of the railway vehicle 1.

[0032] The railroad crossing information is information indicating the type of railroad crossing if there is a railroad crossing at the corresponding set point P1. The type of railroad crossing is information indicating whether or not there is a crossing gate.

[0033] Next, the details of the track information generating device 100 according to this embodiment will be described. 7 is a diagram illustrating an example of functional blocks of the track information generating device 100 according to the embodiment. As shown in FIG. 7, the track information generating device 100 according to the embodiment mainly includes a basic data collecting unit 110, an intermediate data generating unit 120, and a configuration data generating unit 130.

[0034] The basic data collection unit 110 collects basic data that is the basis for generating the track DB 511. The basic data includes, for example, image information, position information, acceleration information, and camera parameters. The basic data collection unit 110 includes a location-specific image information acquisition unit 111, a location-specific position information acquisition unit 112, a location-specific acceleration information acquisition unit 113, and a camera parameter setting unit 114.

[0035] The location-specific image information acquisition unit 111 acquires image information of the area ahead of the railway vehicle 1 captured by the camera 400 at each location on the route as the railway vehicle 1 travels on the rails.

[0036] The location-specific position information acquisition unit 112 acquires position information as positioning information measured by the position measurement device 310 at each point on the route as the railway vehicle 1 travels on the rails.

[0037] The point-specific acceleration information acquisition unit 113 acquires acceleration information detected by the three-axis acceleration sensor of the inertial sensor 600 at each point on the route as the railway vehicle 1 travels on the rails. The camera parameter setting unit 114 sets external parameters such as the mounting height of the camera 400 on the railway vehicle 1.

[0038] Here, the location-specific image information acquisition unit 111, the location-specific position information acquisition unit 112, the location-specific acceleration information acquisition unit 113, and the camera parameter setting unit 114 are examples of an acquisition unit.

[0039] The intermediate data generation unit 120 generates intermediate data from the basic data for generating the track DB 511. The intermediate data corresponds to position information of the left and right rail centers, land gradient information, etc., which will be described later. The intermediate data generation unit 120 includes a left and right rail center position information generation unit 121 and a land gradient information estimation unit 123.

[0040] The left and right rail center position information generator 121 generates position information for the pair of rails, i.e., the center of the left and right rails, based on the location-specific position information acquired multiple times by the location-specific position information acquirer 112. The left and right rail center position information generator 121 is an example of a first generator. The position information for the center of the left and right rails is an example of first position information.

[0041] The land gradient information estimation unit 123 generates land gradient information, which is relative land gradient information within the monitoring range for each location, based on the acceleration information for each location acquired by the location-specific acceleration information acquisition unit 113. The land gradient information estimation unit 123 is an example of a second estimation unit. When generating the land gradient information, not only the acceleration information but also a topographical map of the vicinity of the railway line may be referenced.

[0042] The configuration data generation unit 130 generates configuration data that constitutes the track DB 151. The configuration data generation unit 130 includes a left / right rail position information generation unit 139, a coordinate conversion processing unit 131, a camera posture estimation unit 132, a track area / rail extraction unit 133, a left / right rail matching processing unit 134, a left / right rail position information generation unit 135, a track DB construction unit 136, and a track DB correction unit 137. The configuration data corresponds to left / right rail position information and the like.

[0043] The left and right rail position information generator 139 generates left and right rail position information from the left and right rail center position information generated by the left and right rail center position information generator. The left and right rail position information generator 122 is an example of a second generator. The left and right rail position information is an example of second position information.

[0044] The coordinate conversion processing unit 131 executes coordinate conversion processing to express, in three coordinate systems, positional information of the left and right rails in the traveling direction of the railway vehicle 1. More specifically, the coordinate conversion processing unit 131 performs coordinate conversion from a three-dimensional world coordinate system to a three-dimensional camera coordinate system, and coordinate conversion from the three-dimensional camera coordinate system to a two-dimensional image coordinate system.

[0045] The camera attitude estimation unit 132 estimates the attitude of the camera 400 installed on the railway vehicle 1, that is, the rotation angles around three axes (roll angle, pitch angle, yaw angle).

[0046] The track area / rail extraction unit 133 extracts a track area and left and right rails from the image information acquired by the location-specific image information acquisition unit 111 based on image features.

[0047] The left and right rail matching processing unit 134 calculates a score for the degree of overlap between the left and right rails extracted from the image information by the track area / rail extraction unit 133 and the left and right rail position information projected onto the image information by the track area / rail extraction unit 133.

[0048] The left and right rail position information generation unit 135 generates left and right rail position information in the camera coordinate system from the left and right rail position information in the world coordinate system when the score of the left and right rail matching processing unit 134 is maximum, using the coordinate transformation by the coordinate transformation processing unit 131. Furthermore, the left and right rail position information generation unit 135 generates left and right rail position information in the image coordinate system from the left and right rail position information in the camera coordinate system, using the coordinate transformation by the coordinate transformation processing unit 131. The track area / rail extraction unit 133, the left and right rail matching processing unit 134, and the left and right rail position information generation unit 135 are an example of a third generation unit.

[0049] Here, the left and right rail position information in the world coordinate system and the left and right rail position information in the camera coordinate system are three-dimensional point sequence data. The left and right rail position information in the image coordinate system is two-dimensional point sequence data. The left and right rail position information in the camera coordinate system is an example of fourth position information. The left and right rail position information in the image coordinate system is an example of third position information. Details of the coordinate conversion process will be described later.

[0050] The track DB construction unit 136 constructs the track DB 511 by registering data such as left and right rail position information in the camera coordinate system, which is three-dimensional point sequence data generated by the left and right rail position information generation unit 135, and left and right rail position information in the image coordinate system, which is two-dimensional point sequence data, in the track DB 511 in the map DB 510.

[0051] The track DB correction unit 137 projects the sequence of points of the left and right rail position information and the sequence of points of the center position information of the left and right rails in the track DB 511 constructed by the track DB construction unit 136 onto the image information captured by the camera 400 and displays it on the display device 700.

[0052] 8 is a diagram showing an example of image information in which a sequence of points representing left and right rail position information and a sequence of points representing left and right rail center position information are projected in an embodiment. As shown in Fig. 8, the sequence of points P2 representing left and right rail position information is projected onto rail R in the image information, and further, the sequence of points CL representing left and right rail center position information is projected.

[0053] If a user who sees the projection display on the display device 700 determines that there is a positional deviation from the actual left and right rails, the camera posture estimation unit 132 corrects the rotation angles in the three axial directions calculated and issues an instruction to regenerate the left and right rail information. Based on the instruction, the track DB correction unit 137 performs a correction process to regenerate the left and right rail information and re-register it in the track DB 511.

[0054] Next, a description will be given of a process for generating track information by the track information generating device 100 according to the present embodiment configured as described above. 9 is a flowchart illustrating an example of a procedure for a line information generation process according to the embodiment. First, the basic data collection unit 110 collects basic data (S101).

[0055] FIG. 10 is a flowchart illustrating an example of a procedure of the basic data collection process according to the embodiment. First, an operator installs the camera 400 for forward monitoring, which is capable of synchronously recording data, the positioning device 310, the antenna 300, and the inertial sensor 600 on the railway vehicle 1 (S201). Then, the camera parameter setting unit 114 measures the mounting position information of each sensor and sets the camera parameters (S202).

[0056] Next, the railway vehicle 1 travels multiple times on the route for which the track DB 511 is to be constructed, and the location-specific image information acquisition unit 111 acquires image information, the location-specific position information acquisition unit 112 acquires position information of the railway vehicle 1, and the location-specific acceleration information acquisition unit 113 acquires and records acceleration information (S203). Then, the basic data collection unit 110 generates a data set of the acquired position information, image information, and acceleration information that represent each point on the route (S204). The data set of position information, image information, and acceleration information becomes basic data.

[0057] Returning to FIG. 9, the intermediate data generation unit 120 generates intermediate data from the basic data (S102).

[0058] 11 is a flowchart illustrating an example of the procedure of the intermediate data generation process according to the embodiment. First, the left and right rail center position information generator 121 performs noise removal on the position information to generate position information (point sequence data) of the left and right rail centers (S301).

[0059] Next, the land gradient information estimation unit 123 generates land gradient information at each point on the route from the data set of the position information and acceleration information (S302).

[0060] Returning to FIG. 9, the configuration data generating unit 130 generates configuration data for the line DB 511 from the generated intermediate data (S103).

[0061] FIG. 12 is a flowchart illustrating an example of a procedure of the configuration data generation process according to the embodiment. First, the left and right rail position information generating unit 139 generates left and right rail position information based on the left and right rail center position information generated by the left and right rail center position information generating unit 121 (S401).

[0062] FIG. 13 is a schematic diagram for explaining a process for generating position information of the left and right rails from position information of the centers of the left and right rails in the embodiment. 13, for each of the multiple points in the sequence of position information for the centers of the left and right rails, the left and right rail position information generator 139 of this embodiment rotates the vector from one point to the point one position further in the direction of travel of the railway vehicle 1 by 90° to the left and right. Then, the left and right rail position information generator 139 generates a point that is extended to a length that is half the left and right width, and generates the sequence of points consisting of the multiple points that have been generated as left and right rail position information.

[0063] One method generates left and right rail position information by simply rotating a vector to a point several points ahead of any point in the sequence of points for the center position information for the left and right rails by 90 degrees to the left and right. However, with this method, the actual rail position and the left and right rail position information tend to deviate from each other in places with large rail curvature, such as curves and branching sections. In contrast, the method of this embodiment not only rotates the rails by 90 degrees but also performs an extension process, making it possible to generate left and right rail position information for a sequence of points that more accurately fits the rails, even in places with large rail curvature, such as curves and branching sections.

[0064] Returning to FIG. 12, next, the camera posture estimation unit 132 estimates posture information of the camera 400 for each point (S402). Fig. 14 is a diagram showing an example of attitude information of camera 400 in an embodiment. As shown in Fig. 14, consider a three-axis camera coordinate system in which the width direction of railway vehicle 1, the traveling direction of railway vehicle 1, and the height direction of railway vehicle 1 are the x-axis, y-axis, and z-axis, respectively. In this case, attitude information of camera 400 includes a pitch angle θ, which is a rotation angle around the x-axis, a roll angle φ, which is a rotation angle around the y-axis, and a yaw angle ψ, which is a rotation angle around the z-axis.

[0065] Returning to Figure 12, next, the coordinate transformation processing unit 131 performs coordinate transformation based on the estimated posture information of the camera 400, and the left and right rail position information generation unit 135 generates coordinate-transformed position information of the left and right rails (S403).

[0066] FIG. 15 is a diagram showing the flow of coordinate transformation in the embodiment. As shown in FIG. 15, there are three coordinate systems: a three-dimensional world coordinate system, a three-dimensional camera coordinate system, and a two-dimensional image coordinate system. In this embodiment, the coordinate conversion processing unit 131 converts the three-dimensional left and right rail position information (point sequence P) in the world coordinate system generated by the left and right rail matching processing unit 134 (1. Generation of three-dimensional point sequence data). W (X,Y,Z)) to obtain the three-dimensional left and right rail position information (point sequence P C (x,y,z) (2. Rotation and translation of the coordinate system).

[0067] The coordinate conversion processing unit 131 further converts three-dimensional left and right rail position information (point sequence P C (x,y,z)) to obtain the two-dimensional left and right rail position information (point sequence P I_i (u,v)) (3. Perspective projection).

[0068] Then, the track DB correction unit 137, which will be described later, determines whether there is a misalignment between the edges of the left and right rails and the left and right rail position information (point sequence data) (4. Matching of the edges of the left and right rails with the point sequence), and if there is a misalignment, the process returns to 2. Rotation and translation of the coordinate system, and the coordinate transformation processing unit 131 performs coordinate transformation again.

[0069] Here, in the conversion from the world coordinate system to the camera coordinate system by the coordinate conversion processing unit 131, the rotation angles around three axes (roll angle / pitch angle / yaw angle) as the attitude information of the camera 400 are determined as follows.

[0070] <Initial rotation angle setting> The coordinate conversion processing unit 131 selects point sequence data (a collection of past position measurement results) which is position information of the center of the left and right rails at a position close to the position measurement result (current position) near the antenna 300 in world coordinates, and calculates a vector representing the direction of travel of the railway vehicle 1.

[0071] Next, the coordinate conversion processing unit 131 determines an initial value so that the Y axis of the camera coordinate system overlaps with the vector of the traveling direction.

[0072] The coordinate conversion processing unit 131 determines an initial value of the pitch angle, which is the angle of rotation around the X axis, based on the gradient information of the track. The coordinate conversion processing unit 131 sets the initial value of the roll angle, which is the angle of rotation around the Y axis, to 0 (°).

[0073] <Searching for and determining the rotation angle> As a first process, the coordinate transformation processing unit 131 transforms from the world coordinate system to the camera coordinate system based on the initial values ​​of the roll angle, pitch angle, and yaw angle (transformation from three-dimensional coordinates to three-dimensional coordinates), and performs a perspective projection transformation based on the values ​​of the external and internal parameters of the camera 400 (transformation from three-dimensional coordinates to two-dimensional coordinates).

[0074] As a result, the position information (point sequence data) of the left and right rails in the world coordinate system is converted into position information (point sequence data) of the left and right rails in the image coordinate system (conversion from three-dimensional coordinates to three-dimensional coordinates to two-dimensional coordinates).

[0075] As a second process, the coordinate conversion processing unit 131 applies image recognition techniques such as edge extraction processing and deep learning techniques to the image information captured by the camera 400, and extracts areas that appear to be the left and right rails.

[0076] The coordinate conversion processing unit 131 digitizes the degree of overlap between the left and right rail position information (point sequence data) generated in the first process and the left and right rail areas generated in the second process. At this time, the coordinate conversion processing unit 131 performs calculations giving priority to overlap between left and right rails that are located in close to medium distances, because in the second process, it is easy to extract track areas that are close to the camera 400, but extraction is difficult in areas that are far away.

[0077] The coordinate conversion processing unit 131 calculates the numerical value of the degree of overlap by changing mainly the values ​​of the pitch angle and yaw angle, and searches for the values ​​of the rotation angles (roll angle, pitch angle, yaw angle) that maximize the value.The coordinate conversion processing unit 131 then adopts the values ​​of the rotation angles (roll angle, pitch angle, yaw angle) that maximize the numerical value of the degree of overlap.

[0078] Here, the three-dimensional left and right rail position information in the camera coordinate system (point sequence P C (x,y,z)) from the left and right rail position information in the image coordinate system (point sequence P I_i The coordinate transformation to (u,v) uses the perspective projection method. C Since (x,y,z) is three-dimensional, a sequence of points PC (x,y,z) may have a gradient.

[0079] FIG. 16 is a diagram showing an example of coordinate transformation by perspective projection from three-dimensional left and right rail position information in the camera coordinate system to two-dimensional left and right rail position information in the image coordinate system when there is an uphill gradient in the embodiment.

[0080] FIG. 17 is a diagram showing an example of coordinate transformation by perspective projection from three-dimensional left and right rail position information in the camera coordinate system to two-dimensional left and right rail position information in the image coordinate system when there is a downhill gradient in the embodiment.

[0081] In either case, the coordinate transformation processing unit 131 performs coordinate transformation from the position of the camera 400 to two-dimensional left and right rail position information in the image coordinate system by perspectively projecting the three-dimensional left and right rail position information in the camera coordinate system onto a two-dimensional virtual plane based on the external and internal parameter values ​​of the camera 400 and the land gradient information for each point.

[0082] 12, the track DB construction unit 136 registers the left and right rail position information that has been coordinate converted by the coordinate conversion processing unit 131 and generated by the right rail position information generation unit 135 in the track DB 511 (S404). In this way, the track DB 511 is constructed.

[0083] Next, returning to FIG. 9, the line DB corrector 137 corrects the configuration data of the line DB 511 (S104).

[0084] FIG. 18 is a flowchart illustrating an example of a procedure for the configured data correction process according to the embodiment. First, the track DB correction unit 137 superimposes the generated left and right rail position information on the image information and displays it on the display device 700 using the adjustment application. Then, the user visually checks this image information to see if there is any discrepancy between the left and right rails in the image and the superimposed left and right rail position information (S501).

[0085] If a deviation has occurred, the user takes measures such as issuing an instruction to the adjustment application to recalculate the attitude information of the camera 400, thereby correcting the deviation (S502).

[0086] 19 is a diagram showing an example of an adjustment app screen displayed on the display device 700 in the embodiment. As shown in Fig. 19, the adjustment app screen displays an image display area 1901 and a rotation angle editing area 1902. In the image display area 1901, a sequence of points P2 of left and right rail position information is displayed superimposed on image information captured by the camera 400.

[0087] However, in the example of Figure 19, it can be seen that the sequence of points P2 of the left and right rail position information is misaligned with the actual positions of the left and right rails R on the image information. For this reason, the user resets the attitude information of the camera 400, such as the roll angle, pitch angle, and yaw angle, in the rotation angle editing area, and presses or touches the recalculate / draw button 1903. This causes each section of the configuration data generation unit 130 to recalculate and generate the left and right rail position information using the newly input roll angle, pitch angle, and yaw angle, thereby correcting the track DB 511.

[0088] Then, when the correction is completed, the track DB correction unit 137 displays the recalculated left and right rail position information in the image display area 1901 of the adjustment application, superimposed on the image information captured by the camera 400 for the point sequence P2.

[0089] Fig. 20 is a diagram showing an example of an adjustment app screen in which a sequence of points P2 of corrected left and right rail position information is superimposed in the embodiment. In the example of Fig. 20, it can be seen that, as a result of the correction, the sequence of points P2 of the left and right rail position information matches the actual positions of the left and right rails R on the image information. This completes the correction process for the track DB 511.

[0090] The method in the comparative example combines two main pieces of information to obtain track information from images. The first method uses edge extraction processing and deep learning techniques to extract the track area and left and right rails from images obtained by a forward-looking monitoring camera mounted on the train. The second method involves constructing a map of the route the train will travel on in advance, estimating the train's own position based on positioning information obtained from GNSS and other sources, and then comparing it with the map (map matching) to obtain information such as the rail shape in the direction the train is traveling.

[0091] The first method has difficulty consistently extracting rail tracks over long distances. In outdoor environments, lighting conditions and rail visibility change significantly depending on the weather, season, and time of day. For example, when it snows, the track area and rails are covered in snow, making them difficult for humans to see. Furthermore, the second method uses map matching to obtain information about the track (e.g., position, shape, etc.) along which the train travels. However, this process compensates for the reduced accuracy of self-location estimation due to GNSS positioning errors, and it is difficult to directly link the track information on the map to the extraction of track areas and rails on the image. Thus, simply combining the methods in the comparative example makes it difficult to reliably obtain track information under various conditions.

[0092] In contrast, in this embodiment, as the railway vehicle 1 runs on a pair of left and right rails, the track information generating device 100 acquires position information indicating the position of the railway vehicle 1 at each point on the route based on positioning signals from artificial satellites received by a positioning device installed on the railway vehicle 1, and image information of the area in front of the railway vehicle 1 at each point on the route captured by a camera 400 installed on the railway vehicle 1, and generates position information for the centers of the left and right rails indicating the center position of the left and right rails for each point on the route based on the position information, generates left and right rail position information indicating the positions of the left and right rails for each point on the route based on the position information for the centers of the left and right rails, generates left and right rail position information from the left and right rail position information indicating the position in the image information, and registers the generated left and right rail position information in the image information in the railway DB 151 stored in the memory device 500.

[0093] Therefore, according to this embodiment, the track DB511 can be constructed semi-automatically, so that the track DB511 that specifies the search range for obstacle detection can be generated more accurately without manual intervention, and the on-board device 200 can use the track DB511 to accurately monitor the road ahead regardless of the environment or time of day.

[0094] That is, in this embodiment, by referring to the track DB 551 constructed in advance by the on-board device 200 while the railway vehicle 1 is running, accurate track information (track area, left and right rail positions) can be acquired from the image information of the forward monitoring camera 400 without being affected by weather, season, or time of day, and stable, highly accurate obstacle detection can be achieved. Also, in this embodiment, high accuracy in obstacle detection can be achieved in areas where it is difficult to accurately determine the positions of the left and right rails using image recognition technology alone, such as in the distance or ahead of a curve.

[0095] In addition, in this embodiment, the track information generating device 100 estimates the attitude of the camera 400, and based on the estimated attitude of the camera 400 and the parameters of the camera 400, performs coordinate transformation on the three-dimensional left and right rail position information in the world coordinate system to generate three-dimensional left and right rail position information in the camera coordinate system, and performs coordinate transformation on the three-dimensional left and right rail position information in the camera coordinate system to generate two-dimensional left and right rail position information in the image coordinate system, and associates the three-dimensional left and right rail position information in the camera coordinate system with the two-dimensional left and right rail position information in the image coordinate system and registers them in the track DB 511.

[0096] Therefore, in this embodiment, the left and right rail position information can be calculated more accurately by converting the left and right rail position information into a coordinate system different from the coordinate system used during positioning, the coordinate system of the camera 400, and the image system of the image information. Therefore, according to this embodiment, the track DB 511 can be generated more accurately.

[0097] In addition, in this embodiment, the track information generating device 100 further acquires acceleration information for each point detected by an inertial sensor 600 that detects the acceleration of the railway vehicle 1, estimates land gradient information indicating the gradient for each point based on the acceleration information, and performs coordinate transformation on the three-dimensional left and right rail position information in the camera coordinate system based on the land gradient information for each point to generate two-dimensional left and right rail position information in the image coordinate system.

[0098] Therefore, according to this embodiment, even when the railway vehicle 1 runs on inclined rails, the left and right rail position information can be calculated more accurately by performing coordinate conversion on the left and right rail position information. Therefore, according to this embodiment, the track DB 511 can be generated more accurately.

[0099] Furthermore, in this embodiment, the track information generating device 100 superimposes two-dimensional left and right rail position information in the image coordinate system on the image information for each location and outputs it to the display device 700, and when an instruction to correct the two-dimensional left and right rail position information in the image coordinate system is received from the user, the track information generating device 100 corrects the track DB 511 with the corrected two-dimensional left and right rail position information in the image coordinate system.

[0100] Therefore, according to this embodiment, even if the left and right rail position information in the generated track DB 511 is misaligned with the actual rail positions, it can be easily corrected. Therefore, according to this embodiment, the track DB 511 can be generated more accurately using a simple method.

[0101] Furthermore, in this embodiment, for each of the multiple points in the sequence of position information for the centers of the left and right rails, the track information generating device 100 rotates the vector from one point to the point one position further in the direction of travel of the railway vehicle 1 by 90° left and right to generate a point extended to a length that is half the left and right width, and generates the sequence of points consisting of the multiple points generated as three-dimensional left and right rail position information in the world coordinate system.

[0102] Therefore, according to this embodiment, it is possible to generate left and right rail position information of a sequence of points that more accurately fits the rails, even in places where the rail curvature is large, such as curves and branching sections. Therefore, according to this embodiment, it is possible to generate the track DB 511 more accurately.

[0103] The track information generating device 100 of the above-described embodiment is equipped with a control device such as a CPU, a storage device such as a ROM (Read Only Memory) or RAM, an external storage device such as an HDD, SSD or CD drive, a display device such as a display device, and input devices such as a keyboard and a mouse, and has a hardware configuration that utilizes a normal computer.

[0104] The track information generating program executed by the track information generating device 100 of the above-described embodiment is provided in a state that it is pre-installed in a ROM or the like. The track information generation program executed by the track information generating device 100 of the above-described embodiment may be configured to be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk). Furthermore, the track information generating program executed by the track information generating device 100 of the above-described embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the track information generating program executed by the track information generating device 100 of the above-described embodiment may be configured to be provided or distributed via a network such as the Internet.

[0105] The track information generation program executed by the track information generation device 100 of the above-mentioned embodiment has a modular structure including each of the functional units described above, and in actual hardware, a CPU (processor) reads out the track information generation program from the above-mentioned ROM and executes it, thereby loading each of the functional units (location-specific image information acquisition unit 111, location-specific position information acquisition unit 112, location-specific acceleration information acquisition unit 113, camera parameter setting unit 114, left and right rail center position information generation unit 121, land gradient information estimation unit 123, left and right rail position information generation unit 139, coordinate conversion processing unit 131, camera attitude estimation unit 132, track area / rail extraction unit 133, left and right rail matching processing unit 134, left and right rail position information generation unit 135, track DB construction unit 136, track DB correction unit 137) onto a main memory device, and each functional unit is generated on the main memory device.

[0106] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0107] 1. Railway vehicles 10. Track information generation system 100 Track information generation device 110 Basic Data Collection Department 111 Location-specific image information acquisition unit 112 Location information acquisition unit by point 113 Point-specific acceleration information acquisition unit 114 Camera parameter setting section 120 Intermediate Data Generation Unit 121 Left and right rail center position information generation unit 123 Land gradient information estimation unit 130 Configuration data generation unit 131 Coordinate transformation processing unit 132 Camera Pose Estimation Unit 133 Track Area / Rail Extraction 134 Left and right rail matching processing unit 135 Left and right rail position information generation unit 136 Track DB Construction Department 137 Track DB Correction Unit 139 Left and right rail position information generation unit 200 Onboard equipment 300 Antenna 310 Positioning Device 400 cameras 500 storage device 600 Inertial Sensor 700 Display device

Claims

1. A track information generating device mounted on a railway vehicle, an acquisition unit that acquires positioning information indicating the position of the railway vehicle at each point on the route based on positioning signals from artificial satellites received by a positioning device installed on the railway vehicle as the railway vehicle travels on a pair of left and right rails, and image information of images of the area ahead of the railway vehicle at each point on the route captured by an imaging device installed on the railway vehicle; a first generating unit that generates first position information indicating a center position between the left and right rails for each point on the route based on the positioning information; a second generation unit that generates second position information indicating the positions of the left and right rails for each point on the route based on the first position information; a third generation unit that generates third position information indicating a position in the image information from the second position information; a construction unit that registers the generated third position information in track information related to the track stored in a storage device, The third generation unit a first estimation unit that estimates the attitude of the imaging device; a coordinate transformation processing unit that performs coordinate transformation on the second position information of the three-dimensional coordinate system based on the estimated attitude of the imaging device and parameters of the imaging device to generate fourth position information of the three-dimensional coordinate system of the imaging device, and performs coordinate transformation on the fourth position information to generate third position information of the two-dimensional coordinate system, the construction unit associates the third location information with the fourth location information and registers them in the railway information; The acquisition unit further acquires acceleration information for each of the points detected by an inertial sensor that detects acceleration of the railway vehicle, The track information generating device includes: a second estimation unit that estimates land gradient information indicating a gradient for each of the points based on the acceleration information, the coordinate transformation processing unit performs coordinate transformation on the fourth position information based on land gradient information for each of the points to generate the third position information. Track information generation device.

2. a correction unit that superimposes the third position information on the image information for each of the locations and outputs the superimposed information to an output device, and corrects the track information with the corrected third position information when receiving an instruction to correct the third position information from a user; The track information generating device according to claim 1 , further comprising:

3. the first position information is a point sequence composed of a plurality of points, the second generation unit rotates a vector from one point to a point one step further in the direction of travel of the railway vehicle by 90° left and right for each of the plurality of points in the sequence of points of the first position information, generates a point extended to a length half the width of the left and right, and generates the sequence of points consisting of the generated plurality of points as the second position information. The track information generating device according to claim 1 .

4. A track information generating method executed by a track information generating device mounted on a railway vehicle, comprising: an acquisition step of acquiring positioning information indicating the position of the railway vehicle at each point on the route based on positioning signals from artificial satellites received by a positioning device installed on the railway vehicle as the railway vehicle travels on a pair of left and right rails, and image information of an area ahead of the railway vehicle at each point on the route captured by an imaging device installed on the railway vehicle; a first generation step of generating first position information indicating a center position between the left and right rails for each point on the route based on the positioning information; a second generation step of generating second position information indicating positions of the left and right rails for each point on the route based on the first position information; a third generation step of generating third position information indicating a position in the image information from the second position information; a construction step of registering the generated second position information in track information relating to the track stored in a storage device, The third generation step includes: a first estimation step of estimating an attitude of the imaging device; a coordinate transformation processing step of performing coordinate transformation on the second position information of the three-dimensional coordinate system based on the estimated attitude of the imaging device and parameters of the imaging device to generate fourth position information of the three-dimensional coordinate system of the imaging device, and performing coordinate transformation on the fourth position information to generate third position information of the two-dimensional coordinate system, the constructing step includes registering the third location information and the fourth location information in the railway information in association with each other; The acquiring step further includes acquiring acceleration information for each of the points detected by an inertial sensor that detects acceleration of the railway vehicle; The line information generating method includes: a second estimation step of estimating land gradient information indicating a gradient for each of the points based on the acceleration information, the coordinate transformation processing step performs coordinate transformation on the fourth position information based on land gradient information for each of the points to generate the third position information. Track information generation method.

5. A program to be executed by a computer of a track information generating device mounted on a railway vehicle, an acquisition step of acquiring positioning information indicating the position of the railway vehicle at each point on the route based on positioning signals from artificial satellites received by a positioning device installed on the railway vehicle as the railway vehicle travels on a pair of left and right rails, and image information of an area ahead of the railway vehicle at each point on the route captured by an imaging device installed on the railway vehicle; a first generation step of generating first position information indicating a center position between the left and right rails for each point on the route based on the positioning information; a second generation step of generating second position information indicating positions of the left and right rails for each point on the route based on the first position information; a third generation step of generating third position information indicating a position in the image information from the second position information; a construction step of registering the generated second position information in track information relating to the track stored in a storage device; The third generation step includes: a first estimation step of estimating an attitude of the imaging device; a coordinate transformation processing step of performing coordinate transformation on the second position information of the three-dimensional coordinate system based on the estimated attitude of the imaging device and parameters of the imaging device to generate fourth position information of the three-dimensional coordinate system of the imaging device, and performing coordinate transformation on the fourth position information to generate third position information of the two-dimensional coordinate system, the constructing step includes registering the third location information and the fourth location information in the railway information in association with each other; The acquiring step further includes acquiring acceleration information for each of the points detected by an inertial sensor that detects acceleration of the railway vehicle; The program a second estimation step of estimating land gradient information indicating a gradient for each of the points based on the acceleration information; the coordinate transformation processing step performs coordinate transformation on the fourth position information based on land gradient information for each of the points to generate the third position information. program.

Images