Correction device and correction method

Abstract

Provided is a correction device including: a rail detection unit 102 which detects a first rail point cloud 30an indicating positions of rails 20Lt and 20Rt based on first sensor information acquired by a first sensor 101a; a calibration feasibility determination unit 103 which determines feasibility of calibration of the first sensor 101a based on a first linearity indicating a degree of a straight line with respect to the first rail point cloud 30an; and a sensor information correction unit 105 which corrects the first sensor information based on the first rail point cloud 30an and a point cloud 30at which is a reference rail point cloud that indicates three-dimensional positions of rails and is different from the first rail point cloud in a case where the calibration feasibility determination unit 103 determines that calibration is possible. Accordingly, a correction device and a correction method capable of performing more accurate calibration without providing calibration equipment are provided.

Description

Technical Field

[0001] The present invention relates to a correction device and a correction method. In particular, the present invention relates to a correction device and a correction method suitable for correcting sensor information output from a sensor mounted on a train.Background Art

[0002] For example, the train front monitoring system is expected to expand the front monitoring area and improve the object detection rate by mounting a plurality of sensors having different monitoring areas and frequency characteristics. In such a system, a plurality of pieces of sensor information is integrated by using an external parameter (attachment position and angle information) of each sensor. The external parameter of each sensor has an individual difference depending on the vehicle, and changes over time due to vehicle body vibration or the like, so that the actual external parameter has an error with respect to the design value. When this error increases, a position error of an object included in each sensor increases, and it becomes difficult to integrate sensor information. Therefore, in order to prevent failure caused by a sensor external parameter error when integrating a plurality of pieces of sensor information, it is necessary to calculate and correct external parameters such as the position and posture of each sensor. That is, a correction device or the like that performs calibration and corrects sensor information is required.

[0003] PTL 1 describes a camera calibration device including a measurement area including at least two reference lines that include a line intersecting two points corresponding to a shape of an own vehicle and are arranged within an imaging range of a camera, and a calibration pattern having a predetermined pattern shape between the reference lines, and an image processor. Then, in this device, the image processor includes: imaging processing of generating a measurement image by imaging a measurement area in which the calibration pattern and the reference line are arranged with the camera; stop error calculation processing of calculating a coordinate relationship between the stop position of the own vehicle and the pattern shape as a stop error based on the two reference lines; and external parameter calculation processing of calculating an external parameter corresponding to the attachment posture of the camera based on the pattern shape in a state where the stop error is canceled.

[0004] PTL 2 discloses a vehicle body inclination estimation device in which a rail detection unit that obtains a position of each rail from an image captured by a camera, a three-dimensional shape calculation unit that calculates a three-dimensional shape of the rail based on a position of each rail obtained from an image captured by the camera, and an inclination estimation unit that suppresses an influence of noise by using M estimation or RANSAC when aligning the three-dimensional shape of the rail with a rail shape model prepared in advance by an ICP algorithm are provided in an inclination estimation device based on an image obtained by capturing a front rail or a rear rail of a vehicle by the camera disposed along a transverse direction on a roof of the vehicle.Citation ListPatent Literature

[0005] PTL 1: JP 2013-2820 A PTL 2: JP 2019-23017 A Summary of InventionTechnical Problem

[0006] However, providing the measurement area is expected to require maintenance for preventing deterioration such as contamination and loss in addition to the provision cost, and thus it is desirable not to provide such a measurement area.

[0007] An object of the present invention is to provide a correction device and a correction method capable of performing more accurate calibration without providing calibration equipment.Solution to Problem

[0008] In order to solve the above problem, the present invention is a correction device including: a rail detection unit which detects a first rail point cloud indicating a position of a rail based on first sensor information acquired by a first sensor; a calibration feasibility determination unit which determines feasibility of calibration of the first sensor based on a first linearity indicating a degree of a straight line with respect to the first rail point cloud; and a sensor information correction unit which corrects the first sensor information based on the first rail point cloud and a reference rail point cloud that indicates a three-dimensional position of a rail and is different from the first rail point cloud in a case where the calibration feasibility determination unit determines that calibration is possible. In this case, it is possible to provide a correction device capable of performing more accurate calibration without providing calibration equipment.

[0009] Here, the first linearity can indicate the degree of variation with respect to a first rail point cloud in addition to the degree of the straight line. In this case, the accuracy of the evaluation of the linearity can be improved.

[0010] In addition, the calibration feasibility determination unit can obtain the first linearity based on an average value of distances between a rail straight line model estimated by performing straight line extraction from the first rail point cloud and points included in the first rail point cloud. In this case, the linearity can be more easily obtained.

[0011] Furthermore, the calibration feasibility determination unit can obtain a first distance average value and a second distance average value as the average value for each of the pair of rails, and sets a reciprocal of an average value of the first distance average value and the second distance average value as the first linearity. In this case, a more intuitive value can be obtained as the linearity.

[0012] Furthermore, the calibration feasibility determination unit can determine presence or absence of a branch from the first rail point cloud, and determines that calibration is not possible in a case where there is a branch. In this case, the estimation of the external parameter is more accurate.

[0013] Then, the calibration feasibility determination unit can determine presence or absence of a branch by comparing a distance in a width direction of a vehicle with a gauge of the rail for the first rail point cloud. In this case, the branch determination can be performed more easily.

[0014] Furthermore, a reference rail point cloud can be a second rail point cloud that is detected based on second sensor information acquired by a second sensor and indicates a position of a rail. In this case, the sensor information of the first sensor and the second sensor can be integrated.

[0015] Furthermore, the calibration feasibility determination unit can determine feasibility of calibration based on a difference between the first linearity and a second linearity indicating a degree of a straight line with respect to the second rail point cloud. In this case, it is possible to more accurately determine whether calibration is performed.

[0016] Furthermore, a calibration unit which obtains triaxial rotation and translation of the first sensor by using the first rail point cloud and the reference rail point cloud in a case where the calibration feasibility determination unit determines that calibration is possible is further included, and the sensor information correction unit corrects the first sensor information using the triaxial rotation and the translation obtained by the calibration unit. In this case, the accuracy of correction of the first sensor information is improved.

[0017] Furthermore, the calibration unit can record a plurality of times of the triaxial rotation and the translation of the first sensor and outputs a median value of the triaxial rotation and the translation of the first sensor at a plurality of times, and the sensor information correction unit corrects the first sensor information using the output median value. In this case, it is possible to more accurately estimate the external parameter.

[0018] Furthermore, the first rail point cloud indicates a two-dimensional position of a rail, and the calibration unit can convert the reference rail point cloud into an image coordinate system including the first rail point cloud, acquires the reference rail point cloud represented in two dimensions, and calculates the triaxial rotation and the translation for matching the first rail point cloud with the reference rail point cloud represented in two dimensions. In this case, an angle error or the like of the rail point cloud, which occurs when the point cloud on the two-dimensional image is reconstructed into a three-dimensional model, does not occur, and it is easy to obtain a more accurate estimation result of the external parameter.

[0019] Furthermore, it is possible to further include a sensor attachment abnormality detection unit which determines that the sensor attachment angle is abnormal in a case where the triaxial rotation and the translation obtained by the calibration unit exceed a predetermined threshold. In this case, sensor attachment abnormality can be detected.

[0020] Then, the sensor attachment abnormality detection unit can determine that the sensor attachment angle is abnormal in a case where both the triaxial rotation and the translation obtained from the first rail point cloud and a reference rail point cloud as a second rail point cloud detected based on second sensor information acquired by a second sensor and indicating the position of the rail, and the triaxial rotation and the translation obtained from the first rail point cloud and a third rail point cloud detected based on third sensor information acquired by a third sensor and indicating a position of a rail exceed a predetermined threshold. In this case, the sensor attachment abnormality can be determined using the first sensor and the second sensor.

[0021] Furthermore, the calibration feasibility determination unit can determine that calibration is possible in a case where the first rail point cloud is included in a calibration-capable section based on a map including a relationship between an on-track position of a vehicle and a calibration-capable section. In this case, it is possible to grasp in advance the region including a curve section where it is difficult to estimate the external parameter, and to more reliably estimate the external parameter.

[0022] Then, the calibration feasibility determination unit can issue a sensor abnormality signal in a case where the first linearity is a value indicating that calibration is not possible and the first rail point cloud is included in a calibration-capable section. In this case, the sensor attachment abnormality can be detected using the map.

[0023] Furthermore, the rail detection unit may detect a point cloud indicating positions of objects parallel to each other instead of the first rail point cloud. For example, the rail detection unit detects a first overhead line column point cloud instead of the first rail point cloud. In this case, a point cloud necessary for calibration can be acquired using objects parallel to each other exemplified by the overhead line columns.

[0024] In addition, the present invention is a correction method including, by a processor executing software recorded in the memory, detecting a first rail point cloud indicating a position of a rail based on first sensor information acquired by a first sensor; determining feasibility of calibration of the first sensor based on a first linearity indicating a degree of a straight line with respect to the first rail point cloud; and correcting the first sensor information based on the first rail point cloud and a reference rail point cloud that indicates a three-dimensional position of a rail and is different from the first rail point cloud in a case where it is determined that the calibration is possible. In this case, it is possible to provide a correction method capable of performing more accurate calibration without providing calibration equipment.Advantageous Effects of Invention

[0025] According to the present invention, it is possible to provide a correction device and a correction method capable of performing more accurate calibration without providing calibration equipment.Brief Description of Drawings

[0026] [FIG. 1] FIG. 1 is a diagram illustrating an overall configuration of a sensor correction system according to a first embodiment. [FIG. 2] FIG. 2 is a flowchart illustrating an operation from acquisition of first sensor information to correction of information of the first sensor in the first embodiment. [FIG. 3] FIGS. 3(a) to 3(b) are diagrams illustrating a method of calculating linearity. FIGS. 3(c) to 3(e) are diagrams illustrating specific examples when calculating the linearity. [FIG. 4] FIGS. 4(a) to 4(b) are diagrams illustrating specific examples of a method of performing branch determination in a case where a branch determination function is added to the calibration feasibility determination unit. [FIG. 5] FIGS. 5(a) to 5(d) are diagrams illustrating a coordinate system used in the first embodiment and a method of estimating an external parameter using a first rail point cloud. [FIG. 6] FIG. 6 is a diagram in which a horizontal axis indicates the number of accumulated frames of an external parameter estimation result and a vertical axis indicates an external parameter estimation result. [FIG. 7] FIG. 7 is a diagram illustrating an overall configuration of a sensor correction system according to a second embodiment. [FIG. 8] FIGS. 8(a) to 8(e) are diagrams illustrating an operation of an automatic calibration unit in the second embodiment. [FIG. 9] FIG. 9 is a flowchart illustrating an operation of the automatic calibration device in a case where a monocular camera information is used as the first sensor information. [FIG. 10] FIGS. 10(a) to 10(c) are diagrams illustrating details of the operation in S208 of FIG. 9 and a coordinate system definition to be used. [FIG. 11] FIG. 11 is a diagram illustrating an overall configuration of a sensor correction system according to a fourth embodiment. [FIG. 12] FIGS. 12(a) to 12(b) are diagrams illustrating a railway digital map and a method of determining whether or not an external parameter can be estimated using a railway digital map. [FIG. 13] FIG. 13 is a diagram illustrating an overall configuration of a sensor correction system according to a fifth embodiment. Description of Embodiments

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Here, the present invention will be described with reference to the first to fifth embodiments.[First embodiment]<Description of overall configuration of sensor correction system 1a>

[0028] FIG. 1 is a diagram illustrating an overall configuration of a sensor correction system 1a according to a first embodiment.

[0029] As illustrated, the sensor correction system 1a is mounted on a vehicle 10. The vehicle 10 is a vehicle housing that travels on an iron track, and examples thereof include a railway and a tram. The vehicle 10 travels on the pair of rails 20Lt and 20Rt. Here, the rail on the left side in the traveling direction of the vehicle 10 is a rail 20Lt, and the rail on the right side in the traveling direction of the vehicle 10 is a 20Rt.

[0030] The sensor correction system 1a includes a first sensor 101a and an automatic calibration device 11a.

[0031] The automatic calibration device 11a is an example of a correction device, and includes a rail detection unit 102, a calibration feasibility determination unit 103, an automatic calibration unit 104, and a sensor information correction unit 105. Information acquired by the first sensor 101a is used as an input. Note that, in the present embodiment, the automatic calibration unit 104 is mounted on the vehicle 10, but may be mounted on a facility provided on the ground in a case where a means for transmitting information of the sensor 101a to the ground is provided.

[0032] In the automatic calibration device 11a, using the data acquired by the first sensor 101a, the rail detection unit 102 estimates the positions of the left and right rails 20Lt and 20Rt in front of the vehicle 10 and acquires the estimated positions as a first rail point cloud 30an. Further, the calibration feasibility determination unit 103 determines the feasibility of calibration based on the linearity of the first rail point cloud 30an. Then, in a case where it is determined that calibration is possible, the automatic calibration unit 104 estimates the external parameter. Furthermore, the sensor information correction unit 105 corrects the data acquired by the first sensor 101a using the estimation result of the external parameter.

[0033] The first sensor 101a is a sensor that outputs first sensor information to be a source for estimating the first rail point cloud 30an by the rail detection unit 102. Examples of the first sensor 101a include a laser radar, a stereo camera, and a monocular camera. The first sensor information is a three-dimensional point cloud in a case where the first sensor 101a is a laser radar. In a case where the first sensor 101a is a stereo camera, the first sensor is a parallax image and left and right camera images. Furthermore, in a case where the first sensor 101a is a monocular camera, the first sensor is a two-dimensional image.

[0034] The rail detection unit 102 has a function of estimating a three-dimensional point cloud using the first sensor information. It can also be said that the rail detection unit 102 detects the first rail point cloud 30an indicating the positions of the rails 20Lt and 20Rt based on the first sensor information acquired by the first sensor 101a.

[0035] In a case where the first sensor information is a three-dimensional point cloud output from the laser radar, the rail detection unit 102 extracts a point cloud corresponding to the left and right rails 20Lt and 20Rt from the three-dimensional point cloud, and sets the extracted point cloud as a first rail point cloud 30an. In a case where the first sensor information is the parallax image of the stereo camera and the left and right camera images, the rail detection unit 102 estimates the first rail point cloud 30an by using two-dimensional rail extraction using pattern matching or the like on an image, depth estimation results by the parallax image, and the like. In a case where the input image is a monocular camera image, the rail detection unit 102 performs two-dimensional rail extraction using, for example, pattern matching on an image or the like, and estimates the first rail point cloud 30an using a depth estimation result based on a comparison between actual widths (=gauges) of the left and right rails 20Lt and 20Rt and the observed gauge.

[0036] The first rail point cloud 30an is a point cloud corresponding to the three-dimensional positions of the left and right rails 20Lt and 20Rt, and can be, for example, a point cloud corresponding to the inside of the uppermost surfaces of the left and right rails 20Lt and 20Rt. In the present embodiment, it is assumed that the first rail point cloud 30an is expressed by a vehicle fixed coordinate system' 40n in which the first rail point cloud 30an is subjected to coordinate conversion using a rotation design value and a translation design value for conversion from the first sensor coordinate system to a vehicle fixed coordinate system 40t fixed to the vehicle 10.

[0037] The vehicle fixed coordinate system 40t is a coordinate system in which the origin and the direction are fixed with respect to the vehicle 10. In the present embodiment, an x-direction of the vehicle fixed coordinate system 40t is defined as a straight direction along the rails 20Lt and 20Rt, a y-direction is defined as a direction along the ties, a z-direction is defined as a direction perpendicular to x and y, an x-origin is defined as a front surface of the vehicle body, a y-origin is defined as a middle line of the left and right rails 20Lt and 20Rt, and a z-origin is defined as a plane defined by the upper surfaces of the left and right rails 20Lt and 20Rt, but other methods may be used.

[0038] The vehicle fixed coordinate system' 40n is a coordinate system that transitions as a result of coordinate-transforming the sensor information in the first sensor coordinate system using the rotation design value and the translation design value. Since the rotation design value and the translation design value are different from the actual rotation and translation, the origin and the direction of the axis of the vehicle fixed coordinate system' 40n are different from those of the actual vehicle fixed coordinate system 40t. Therefore, the first rail point cloud 30an exists at a position different from a point cloud 30at corresponding to the positions of the actual rails 20Lt and 20Rt in the vehicle fixed coordinate system 40t.

[0039] The calibration feasibility determination unit 103 evaluates the first rail point cloud 30an based on the linearity. Then, the calibration feasibility determination unit 103 sets a threshold value for the linearity and determines the feasibility of calibration based on the threshold value. For example, the calibration feasibility determination unit 103 determines that calibration is possible in a case where the linearity is greater than or equal to a predetermined threshold value, and determines that calibration is impossible in a case where the linearity is less than the predetermined threshold value. Therefore, it can also be said that the calibration feasibility determination unit 103 determines the feasibility of calibration of the first sensor 101a based on the linearity indicating the degree of the straight line with respect to the first rail point cloud 30an. Note that this linearity can also be said to be a first linearity in the sense of distinguishing it from a second linearity to be described later.

[0040] The calibration feasibility determination unit 103 has a function of outputting a calibration feasibility signal for determining whether calibration is possible or not and the first rail point cloud 30an to the automatic calibration unit 104.

[0041] The linearity is an index for evaluating whether or not the estimated first rail point cloud 30an is a straight line and a degree of variation in data of the first rail point cloud 30an, and details thereof will be described later. The linearity threshold is a value that is adjusted in advance so that the estimation accuracy of the external parameter in the automatic calibration unit 104 can satisfy the requirement.

[0042] The calibration feasibility signal is information for determining whether the automatic calibration unit 104 may execute the external parameter estimation processing using the first rail point cloud 30an. The calibration feasibility signal is, for example, a signal of 0 or 1, and 0 means that calibration is impossible, and 1 means that calibration is possible.

[0043] The automatic calibration unit 104 has a function of estimating an external parameter of the first sensor 101a using the first rail point cloud 30an and the point cloud 30at corresponding to the positions of the actual rails 20Lt and 20Rt in a case where it is determined that calibration is possible by the calibration feasibility signal. Therefore, in a case where the calibration feasibility determination unit 103 determines that calibration is possible, the automatic calibration unit 104 functions as a calibration unit that obtains triaxial rotation and translation of the first sensor 101a by using the first rail point cloud 30an and the reference rail point cloud (in the first embodiment, the point cloud 30at corresponding to the positions of the actual rails 20Lt and 20Rt). Details of a method of estimating the external parameter using the first rail point cloud 30an will be described later.

[0044] The sensor information correction unit 105 has a function of converting data of the first sensor 101a into the vehicle fixed coordinate system 40t using the estimation result of the external parameter of the first sensor 101a. It can also be said that the sensor information correction unit 105 corrects the first sensor information based on the first rail point cloud 30an and a reference rail point cloud (in the first embodiment, the point cloud 30at corresponding to the actual positions of the rails 20Lt and 20Rt) indicating the three-dimensional positions of the rails 20Lt and 20Rt and different from the first rail point cloud an in a case where the calibration feasibility determination unit 103 determines that calibration is possible. In this case, the sensor information correction unit 105 corrects the first sensor information using the triaxial rotation and translation obtained by the automatic calibration unit 104.

[0045] FIG. 2 is a flowchart illustrating an operation from acquisition of the first sensor information to correction of the information of the first sensor 101a in the first embodiment.

[0046] First, the rail detection unit 102 acquires first sensor information at a certain time (S101).

[0047] Next, the rail detection unit 102 determines whether the data dimension of the first sensor information is three-dimensional (S102).

[0048] As a result, in a case where the data dimension is three-dimensional (Yes in S102), the rail detection unit 102 detects the rails 20Lt and 20Rt for the three-dimensional point cloud (S103). A case where the data dimension is three-dimensional is, for example, a case of a point cloud acquired by a laser radar.

[0049] On the other hand, in a case where the data dimension is two-dimensional (No in S102), the rail detection unit 102 first detects the rails 20Lt and 20Rt on the two-dimensional image (S104), performs three-dimensional reconstruction of the rail detection result using the parallax image and the gauge (S105), and outputs the rail detection result to the calibration feasibility determination unit 103. A case where the data dimension is two-dimensional is, for example, a case of a stereo camera image or a monocular camera image.

[0050] Next, the calibration feasibility determination unit 103 calculates the linearity by the first rail point cloud 30an (S106).

[0051] Then, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 1 in a case where the linearity is greater than or equal to a predetermined threshold value, and sets the calibration feasibility signal to 0 in a case where the linearity is less than the predetermined threshold value (S107).

[0052] Next, the automatic calibration unit 104 determines whether the calibration feasibility signal is 1 (S108).

[0053] As a result, in a case where the calibration feasibility signal is 1 (Yes in S108), the automatic calibration unit 104 retrieves the point cloud 30at corresponding to the actual positions of the rails 20Lt and 20Rt as the second rail point cloud (the reference rail point cloud) (S109).

[0054] Then, the automatic calibration unit 104 estimates the external parameter of the first sensor 101a by calculating the rotation and translation between the first rail point cloud 30an and the second rail point cloud (S110).

[0055] Furthermore, the sensor information correction unit 105 corrects the first sensor information (S111).

[0056] On the other hand, when the calibration feasibility signal is 0 (No in S108), the correction of the first sensor information is not performed, and the series of processing ends.<Explanation of linearity>

[0057] Next, the linearity will be described in detail.

[0058] FIGS. 3(a) to 3(b) are diagrams illustrating a method of calculating the linearity. FIGS. 3(c) to 3(e) are diagrams illustrating specific examples when calculating the linearity.

[0059] First, as illustrated in FIG. 3(a), the calibration feasibility determination unit 103 estimates a left rail straight line model 201L along the point cloud corresponding to the left rail 20Lt with respect to the first rail point cloud 30an obtained by the rail detection unit 102. In addition, the calibration feasibility determination unit 103 estimates a right rail straight line model 201R along a point cloud corresponding to the right rail 20Rt. As a method of estimating these straight line models, for example, straight line estimation by RANSAC (RANdom SAmple Consensus) is performed on the first rail point cloud 30an, and estimate one of the left and right straight line models as the first straight line model. Then, straight line estimation by RANSAC is performed again by removing a point cloud within a predetermined distance range with respect to the first straight line model, and estimate the other straight line model as the second straight line model. Furthermore, among the first straight line model and the second straight line model, the left rail straight line model 201L can be set to have a larger y-intercept, and the right rail straight line model 201R can be set to have a smaller y-intercept.

[0060] Next, the calibration feasibility determination unit 103 extracts a point cloud 204L existing in a range 203L within a predetermined distance 202 with respect to the left rail straight line model 201L and a point cloud 204R existing in a range 203R within the predetermined distance 202 with respect to the right rail straight line model 201R. The distance 202 is a value that can include a curved rail point cloud in the ranges 203L and 203R.

[0061] Furthermore, as illustrated in FIG. 3(b), the calibration feasibility determination unit 103 calculates a distance 205Li between a point 204Li included in the point cloud 204L and the left rail straight line model 201L over all points of the point cloud 204L, and calculates a first distance average value calculated by averaging. In addition, the calibration feasibility determination unit 103 calculates a distance 205Ri between a point 204Ri of the point cloud 204R and the right rail straight line model 201R over all points of the point cloud 204R, and calculates a second distance average value calculated by averaging. Next, the calibration feasibility determination unit 103 calculates a third distance average value by averaging the first distance average value and the second distance average value. Then, the calibration feasibility determination unit 103 defines the reciprocal of the third distance average value as the linearity. In this case, the larger the value of the linearity, the higher the degree of linearity of the rails 20Lt and 20Rt.

[0062] As described above, the calibration feasibility determination unit 103 obtains the linearity (the first linearity) based on the average value of the distances between the rail straight line models 201L and 201R estimated by performing straight line extraction from the first rail point cloud 30an and the points included in the first rail point cloud 30an. In addition, the calibration feasibility determination unit 103 obtains the first distance average value and the second distance average value as the average value for each of the pair of rails 20Lt and 20Rt, and sets the reciprocal of the average value of the first distance average value and the second distance average value as the linearity (the first linearity).

[0063] Next, a difference in linearity according to the aspect of the first rail point cloud 30an will be described.

[0064] FIG. 3(c) is a specific example in a case where the first rail point cloud 30an with less variation can be acquired with respect to the straight rails 20Lt and 20Rt. In such a case, since the third distance average value becomes small, the linearity, which is the reciprocal of the third distance average value, becomes a large value.

[0065] On the other hand, FIG. 3(d) is a specific example in a case where the first rail point cloud 30an having a large variation is acquired with respect to the straight rails 20Lt and 20Rt. In a case where such a rail detection result is used in the automatic calibration unit 104, the external parameters are estimated based on an inaccurate straight line model, which may lead to an increase in an estimation error of the external parameters. Therefore, it should be determined that calibration is not possible. In the case of FIG. 3(d), for example, since there is a point cloud 206 separated from the left rail straight line model 201L, the third distance average value becomes large, and the linearity, which is the reciprocal of the third distance average value, becomes a smaller value in comparison with the case in FIG. 3(c).

[0066] FIG. 3(e) is a specific example in a case where the first rail point cloud 30an is acquired for the curved rails 20Lt and 20Rt. In the present embodiment, it is assumed that the first rail point cloud 30an input to the automatic calibration unit 104 is a straight rail, and it is necessary to prevent the point cloud corresponding to the curved rails 20Lt and 20Rt from being input to the automatic calibration unit 104. In a case of FIG. 3(e), for example, since there is a point cloud 208 whose distance from the left rail straight line model 201L is long due to the rails 20Lt and 20Rt being curved, the third distance average value becomes large, and the linearity, which is the reciprocal of the third distance average value, becomes a larger value in comparison with the case in FIG. 3(c).

[0067] Therefore, in a case where the linearity in the cases of FIGS. 3(d) and 3(e) is obtained, the calibration feasibility determination unit 103 sets a threshold value for setting the calibration feasibility signal to 0. As a result, in a case where the variation of the first rail point cloud 30an is large and in a case where the rail is a curved rail, calibration is not performed, and it is possible to calculate more accurate external parameters.

[0068] The definition and the property of the linearity calculated by the calibration feasibility determination unit 103 have been described above.

[0069] In the embodiment described above, the calibration feasibility determination unit 103 basically determines the value of the calibration feasibility signal only based on the linearity, but the calibration feasibility signal may be set to 0 in a case where the rails 20Lt and 20Rt branch. Specifically, in a case where the rail detection unit 102 cannot stably extract a straight rail due to branching of the rails 20Lt and 20Rt, the calibration feasibility determination unit 103 may further add a function of determining that the rails 20Lt and 20Rt branch and setting the calibration feasibility signal to 0.

[0070] FIGS. 4(a) to 4(b) are diagrams illustrating specific examples of a method of performing branch determination in a case where a branch determination function is added to the calibration feasibility determination unit 103.

[0071] As illustrated in FIG. 4(a), the calibration feasibility determination unit 103 calculates a distance 212 (Δy) between a point 211L having the largest y and a point 211R having the smallest y among point clouds existing in a section equally divided by a width 210 in the x-direction of the vehicle fixed coordinate system 209. Then, as illustrated in FIG. 4(b), the calibration feasibility determination unit 103 creates a diagram in which the distance 212 (Δy) is plotted for each x-direction section. The x-coordinate in each x-direction section is, for example, the midpoint between the maximum value and the minimum value of each x-direction section.

[0072] FIG. 4(b) is a diagram plotting the distance 212 (Δy) for each x-direction section. In FIG. 4(b), if the rails 20Lt and 20Rt are straight, the distance 212 substantially matches the gauge D of the rails 20Lt and 20Rt. However, in a case where a branch occurs, the distance between the point 211L and the point 211R increases, so that the distance 212 becomes a value larger than the gauge D. Therefore, for example, a value slightly larger than the gauge D is set as a branch threshold 214, and in a case where the distance 212 is larger than the branch threshold 214, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 0. That is, the calibration feasibility determination unit 103 determines the presence or absence of a branch from the first rail point cloud 30an, and determines that calibration is not possible in a case where there is a branch. At this time, the calibration feasibility determination unit 103 determines the presence or absence of branch by comparing the distance in the width direction of the vehicle with the gauge D of the rails 20Lt and 20Rt for the first rail point cloud 30an.

[0073] In FIG. 4(b), a case where the distance 212 is larger than the branch threshold 214 is indicated by a point 216, for example. In FIG. 4(b), a case where the distance 212 is smaller than the branch threshold 214 is indicated by a point 215, for example. As a result, in a case where there is a branch even in part in the acquired first rail point cloud 30an, it is possible to prevent the estimation of the external parameter from being inaccurate when the branch occurs, and to estimate the external parameter with higher accuracy.

[0074] In addition, the calibration feasibility determination unit 103 may further have a function of setting the calibration feasibility signal to 0 in a case where the degree of parallelism between the rail detection result and the ground is less than a predetermined threshold value. In the present exemplary embodiment, the degree of parallelism is an index indicating how parallel two planes are.

[0075] Specifically, first, a first normal vector corresponding to a plane defined by the first rail point cloud 30an is calculated. Next, a point cloud in the vicinity of the first rail point cloud 30an is extracted from the first sensor information, and calculate a second normal vector corresponding to a plane defined by the point cloud remaining by removing the first rail point cloud 30an from the extracted point cloud and indicating the ground. Finally, a result of performing absolute value processing on a reciprocal of an angle formed by the first normal vector and the second normal vector is defined as the degree of parallelism. In this case, the larger the value of the degree of parallelism is, the closer the degree of parallelism is. The point cloud in the vicinity of the first rail point cloud 30an is, for example, a point cloud existing inside the left and right rails 20Lt and 20Rt.

[0076] Since the plane defined by the first rail point cloud 30an and the plane of the ground near the rails 20Lt and 20Rt are substantially parallel, in a case where the degree of parallelism is small, there is a possibility that the rail detection result indicates that it is not parallel to the ground and the rail detection result is invalid. Then, if a threshold is set for the degree of parallelism and such an invalid rail detection result is not used by the automatic calibration unit 104 by the threshold determination of the degree of parallelism, more accurate external parameter calculation can be performed.

[0077] Next, the operation of the automatic calibration unit 104 will be described in detail.

[0078] FIGS. 5(a) to 5(d) are diagrams illustrating a coordinate system used in the first embodiment and a method of estimating an external parameter using the first rail point cloud 30an.

[0079] FIG. 5(a) is a diagram illustrating a first sensor coordinate system. The first sensor coordinate system is a coordinate system based on the first sensor 101a. The rail detection unit 102 converts the information expressed in the first sensor coordinate system into the vehicle fixed coordinate system' 40n (FIG. 5(b)) using the external parameter design value, and performs rail detection. The vehicle fixed coordinate system' 40n is a coordinate system that transitions from the first sensor coordinate system using the external parameter design value (rotation Rs2bn, translation ts2bn), and is a coordinate system slightly different from the vehicle fixed coordinate system 40t (FIG. 5(c)).

[0080] Since the external parameter for performing conversion between the first sensor coordinate system (FIG. 5(a)) and the vehicle fixed coordinate system 40t (FIG. 5(c)) is different from the external parameter design value, a mismatch occurs between the vehicle fixed coordinate system' 40n (FIG. 5(b)) and the vehicle fixed coordinate system (FIG. 5(c)). The estimation of the external parameter is to calculate rotation and translation necessary for converting the first sensor information into the vehicle fixed coordinate system 40t (FIG. 5(c)) instead of the vehicle fixed coordinate system' 40n (FIG. 5(b)), by calculating the rotation Rbn2b and the translation tbn2b for conversion from the vehicle fixed coordinate system' 40n (FIG. 5(b)) to the vehicle fixed coordinate system 40t (FIG. 5(c)).

[0081] Next, an aspect of a rail detection result in each coordinate system will be described. 301Ls and 301Rs indicate detection results of the left and right rails 20Lt and 20Rt in the first sensor coordinate system (FIG. 5(a)). A left rail model 301Lbn and the right rail model 301Rbn are rail detection results in the vehicle fixed coordinate system' 40n (FIG. 5(b)). Further, the left rail model 301Lbt and the right rail model 301Rbt are the actual positions of the rails 20Lt and 20Rt in the vehicle fixed coordinate system (FIG. 5(c)). As shown in FIG. 5(b), since the vehicle fixed coordinate system' 40n and the vehicle fixed coordinate system 40t are different, the left rail model 301Lbn and the left rail model 301Lbt, and the right rail model 301Rbn and the right rail model 301Rbt do not match.

[0082] Here, rotation and translation for converting the left rail model 301Lbn into the left rail model 301Lbt and the right rail model 301Rbn into the right rail model 301Rbt are rotation Rbn2b and translation tbn2b from the vehicle fixed coordinate system' 40n (FIG. 5(b)) to the vehicle fixed coordinate system 40t (FIG. 5(c)). Therefore, in the present embodiment, the rotation Rbn2b and the translation tbn2b for converting a first representative position pbn and a first triaxial vector (exbn, eybn, ezbn) of the first rail point cloud 30an in the vehicle fixed coordinate system' 40n (FIG. 5(b)) into a second representative position pb and a second representative triaxial vector (exb, eyb, ezb) corresponding to the first rail point cloud 30an in the vehicle fixed coordinate system (FIG. 5(c)) are calculated.

[0083] As a method of calculating the first representative position pbn, for example, an arbitrary x-direction position where the first rail point cloud 30an exists is set as an x-coordinate of the first representative position, the first rail point cloud 30an within a predetermined yz range is extracted based on the x-coordinate of the first representative position, and the first rail point cloud after extraction is set. The left rail model 301Lbn and the right rail model 301Rbn are estimated for the first rail point cloud after extraction, and a middle line of the left rail model 301Lbn and the right rail model 301Rb is set as the y-coordinate of the first representative position. Furthermore, the z-coordinate corresponding to the x-coordinate of the first representative position and the y-coordinate of the first representative position on the three-dimensional plane defined by the left rail model 301Lbn and the right rail model 301Rb is set as the z-coordinate of the first representative position.

[0084] The first representative position pbn is arbitrary in the x-direction. However, based on the finding that there is a possibility that the angle estimation accuracy is improved when the distant object detection information is used, the x-coordinate may be set so as to use the farther first rail point cloud 30an, thereby improving the estimation accuracy of the external parameter related to the angle.

[0085] As a method of calculating the first triaxial vector (exbn, eybn, ezbn), for example, a unit vector in the x-direction of the left rail model 301Lbn is set to exbn, a normal vector of a plane defined by the left rail model 301Lbn and the right rail model 301Rb is set to ezbn, and a unit vector eybn perpendicular to exbn and ezbn can be obtained by an outer product of exbn and ezbn.

[0086] As for the method of calculating the second representative position pb, first, in the case of a straight rail having no vehicle body inclination, since the positions where the rails 20Lt and 20Rt exist in the vehicle fixed coordinate system are known, a rail point cloud in the vehicle fixed coordinate system corresponding to the actual positions of the rails 20Lt and 20Rt in the vehicle fixed coordinate system 40t is set.

[0087] Next, the x-coordinate of the second representative position pb is defined as the x-coordinate of the first representative position pbn on the assumption that an x-direction error between the vehicle fixed coordinate system' 40n (FIG. 5(b)) and the vehicle fixed coordinate system 40t (FIG. 5(c)) is small. Further, a first rail point cloud 30an within a predetermined range based on the x-coordinate of the first representative position pbn is extracted and defined as an ideal rail point cloud. The left rail model 301Lbt and the right rail model 301Rbt are estimated with respect to the ideal rail point cloud, and the middle line between the left rail model 301Lbt and the right rail model 301Rt is set as the y-coordinate of the second representative position pb. Furthermore, the z-coordinate corresponding to the x-coordinate of the first representative position pbn and the y-coordinate of the first representative position pbn on the three-dimensional plane defined by the left rail model 301Lbt and the right rail model 301Rt is defined as the z-coordinate of the second representative position pb.

[0088] Regarding the method of calculating the second triaxial vector (exb, eyb, ezb), for example, the unit vector in the x-direction of the left rail model 301Lbt is set to exb, the normal vector of the plane defined by the left rail model 301Lbt and the right rail model 301Rt is set to ezb, and the unit vector eyb perpendicular to exb and ezb can be obtained by the outer product of exb and ezb.

[0089] By solving the equation illustrated in FIG. 5(d) using the first representative position pbn, the first triaxial vector (exbn, eybn, ezbn), the second representative position pb, and the second triaxial vector (exb, eyb, ezb) obtained by the above method, the rotation Rbn2b and the translation tbn2b for conversion from the vehicle fixed coordinate system' 40n (FIG. 5(b)) to the vehicle fixed coordinate system 40t (FIG. 5(c)) can be obtained.

[0090] The above is an overall processing in the automatic calibration device 11a.

[0091] By forming such an automatic calibration device 11a, a new target is unnecessary for detection of the rails 20Lt and 20Rt, and automatic calibration before traveling and during traveling can be performed. In addition, since it is not premised that the vehicle body is reflected as the first sensor information, for example, there is no problem even in a case where it is difficult for the vehicle body to be reflected by using a sensor that monitors a distance at a narrow angle as the first sensor 101a.

[0092] In addition, since the braking distance of the railway is long, it is necessary to detect an object over a long distance in order to stop without colliding with the object found by the front monitoring system. With respect to the object position estimation error caused by the error of the external parameter, in a case where the vehicle traveling direction is set to x, the transverse direction is set to y, and a direction perpendicular to these directions is set to z, the influence of the yaw and the pitch estimation error increases as the distance increases. Since a distant object cannot be correctly detected when these errors become large, it is particularly important to accurately estimate yaw and pitch angles in a railway.

[0093] Since the rails 20Lt and 20Rt are objects extending long in the x-direction and have a large amount of sensor information in the x-axis direction, rail detection accuracy in the x-axis direction is expected to be good. Therefore, in the automatic calibration method utilizing rail detection, it is expected that the estimation accuracy of the external parameter in the x-axis direction is good. The external parameters related to the x-axis direction are a yaw angle and a pitch angle to be estimated with high accuracy in railway front monitoring. By estimating the yaw angle and the pitch angle with high accuracy, the performance of the first sensor 101a for detecting a distant object is improved, and safer traveling is enabled.

[0094] In addition, in the method of estimating the inclination of the related art, the inclination calculation result fluctuates due to a fluctuation factor such as a gradient, cant, vibration during traveling, or vibration of passenger movement even when the vehicle is stationary. Even if the conventional technique is utilized by replacing the inclination estimation with the estimation of the external parameter, it is difficult for the conventional technique to estimate the external parameter with high accuracy in the presence of such a variation factor. In order to estimate the external parameters with high accuracy, measures for increasing the accuracy of the result are required in addition to the external parameter estimation by rail detection. In addition, when the external parameter is defined as a displacement with respect to the design value, it is considered to estimate the external parameter based on a difference between the ideal position of the rail observed by the sensor and the position actually observed, but the ideal position of rail detection is different depending on whether the own vehicle is traveling on a curve or is still traveling on a straight line, and thus it is difficult to calculate the ideal position.

[0095] On the other hand, in the present embodiment, by removing an inappropriate rail point cloud at the time of estimating the external parameter using the calibration feasibility determination unit 103, a variation factor is eliminated, and highly accurate calibration can be performed. In the present embodiment, rail detection is performed on the rails 20Lt and 20Rt in the straight section, so that an ideal position can be easily specified at the time of rail detection. Further, by estimating the external parameter based on the triaxial rotation and translation of the first sensor 101a, a more accurate external parameter can be calculated.

[0096] Therefore, the automatic calibration device 11a in the present embodiment can contribute to cost reduction of the front monitoring system and improvement of safety at the time of traveling.

[0097] In the present embodiment, the method of estimating the external parameter in a case where the calibration feasibility signal is 1 and calibrating the first sensor 101a each time has been described. On the other hand, the external parameter in a case where the calibration feasibility signal is 1 may be accumulated for a certain number of frames, and the representative value of the estimated value of the external parameter between the certain frames may be used as the external parameter used for calibration, thereby improving the estimation accuracy of the external parameter.

[0098] FIG. 6 is a diagram in which the horizontal axis indicates the number of accumulated frames of the external parameter estimation results and the vertical axis indicates the external parameter estimation results. Then, in FIG. 6, the estimation result of the external parameter for each time is indicated by a result 302, and a result 303 of calculating the median value of the accumulated estimation results of the external parameter is schematically illustrated. As a specific example, FIG. 6 illustrates a case where the vertical axis is set to the y-direction translation ty, but the same applies to the cases of the x-direction translation tx and the z-direction translation tz.

[0099] In the above-described example, the calibration feasibility determination unit 103 can exclude a certain amount of curved rails and rail detection results with a large amount of noise. However, although the linearity is high, a phenomenon in which a straight line different from a rail is erroneously detected as a rail occurs at a low frequency. It is difficult to exclude such a result of erroneous detection by linearity determination. At this time, for example, as indicated by 304 in FIG. 6, fluctuation occurs in the calculated external parameter, which leads to deterioration of calibration accuracy.

[0100] In order to prevent a decrease in calibration accuracy due to such an event, by accumulating data for a certain number of frames as in the result 303 and using the representative value as the external parameter used for calibration, a low-frequency event appearing as an outlier can be removed, and a more accurate external parameter can be estimated. As the representative value, for example, there is a method of adopting the median value which is hardly affected by an outlier. In this case, the automatic calibration unit 104 records the triaxial rotation and translation of the first sensor 101a at a plurality of times, and outputs the median value of the triaxial rotation and translation of the first sensor 101a at the plurality of times, and the sensor information correction unit 105 corrects the first sensor information using the output median value.

[0101] Further, there is a method in which the fixed number of frames is set to the value fs when the variation of the representative value becomes small, and the fixed number of frames is determined by trial and error when the automatic calibration device 11a is introduced, or the fixed number of frames is determined based on the number of accumulated frames when the increase or decrease of the representative value converges within a certain range when the number of accumulated frames increases by a predetermined value.

[0102] In the present embodiment, the method of calculating the coordinate transformation between the first sensor coordinate system and the vehicle fixed coordinate system 40t for one first sensor 101a has been described. However, by applying the method of the present embodiment to a plurality of sensors capable of detecting the rails 20Lt and 20Rt and calculating the rotation and translation from each sensor to the vehicle fixed coordinate system, the plurality of sensors can be more accurately integrated into the vehicle fixed coordinate system 40t. In addition, in this method, since the external parameter can be estimated independently for each sensor, there is also an advantage that the external parameter can be estimated even when the visible ranges of the sensors do not overlap. Therefore, according to this method, more accurate automatic calibration can be performed without installing a new target in the system using a plurality of sensors, and thus, it is possible to reduce calibration cost in the system using a front monitoring device and to travel a vehicle more safely.

[0103] In the present embodiment, the calibration method by rail detection has been described, but objects parallel to each other may be detected instead of the rails 20Lt and 20Rt. For example, two parallel upright overhead line columns whose positions with respect to the vehicle 10 are known can be detected by the first sensor 101a and used instead of the first rail point cloud 30an. In this case, the rail detection unit 102 can perform calibration by detecting a first overhead line column point cloud, which is a point cloud corresponding to two parallel upright overhead line columns.[Second embodiment]

[0104] In the first embodiment, the method of estimating the external parameter of the first sensor 101a by comparing the rail detection result of the straight line with the rail straight line in the ideal state in the vehicle fixed coordinate system 40t has been described. On the other hand, in a case where it is not necessary to calculate the attachment position and angle of each sensor with respect to the sensor attachment angle design value, a method of integrating the sensor information in a second vehicle fixed coordinate system' using the first rail detection result detected by the first sensor 101a and the second rail detection result by the second sensor 101b different from the first sensor 101a may be used. Hereinafter, the details will be described as a second embodiment.

[0105] FIG. 7 is a diagram illustrating an overall configuration of a sensor correction system 1b according to the second embodiment.

[0106] The sensor correction system 1b to be illustrated is different from the sensor correction system 1a of the first embodiment illustrated in FIG. 1 in that a second sensor 101b is added to the automatic calibration device 11b, but the other points are similar.

[0107] In addition, in FIG. 7, the first rail point cloud 30an is illustrated as first sensor information obtained by detecting the left and right rails 20Lt and 20Rt by the first sensor 101a. Furthermore, in FIG. 7, a second rail point cloud 30bn is illustrated as second sensor information obtained by detecting the left and right rails 20Lt and 20Rt by the second sensor 101b. It can also be said that FIG. 7 illustrates the first rail point cloud 30an represented by the first vehicle fixed coordinate system' 40an and the second rail point cloud 30bn represented by the second vehicle fixed coordinate system' 40bn.

[0108] Since the external parameter of each of the first sensor 101a and the second sensor 101b has an error with respect to the design value, the first rail point cloud 30an and the second rail point cloud 30bn do not match as illustrated in FIG. 7. In the present embodiment, external parameter for integrating the first sensor information expressed in the first vehicle fixed coordinate system' 40an into the second vehicle fixed coordinate system' 40bn is estimated, and the first sensor information is corrected.

[0109] In the present embodiment, the rail detection unit 102 estimates the first rail point cloud 30an using the information of the first sensor 101a, and further estimates the second rail point cloud 30bn using the information of the second sensor 101b.

[0110] The calibration feasibility determination unit 103 calculates the first linearity based on the first rail point cloud 30an and the second linearity based on the second rail point cloud 30bn according to the method described in the first embodiment. Then, in a case where both the first linearity and the second linearity are equal to or greater than a predetermined threshold value, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 1, and the automatic calibration unit 104 estimates the external parameter. On the other hand, in a case where at least one of the first linearity and the second linearity is less than a predetermined threshold, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 0, and the automatic calibration unit 104 does not perform the estimation of the external parameter.

[0111] In the second embodiment, the reference rail point cloud is the second rail point cloud 30bn that is detected based on the second sensor information acquired by the second sensor 101b and indicates the positions of the rails 20Lt and 20Rt. Then, the calibration feasibility determination unit 103 determines whether calibration is possible based on a difference between the first linearity and the second linearity indicating the degree of linearity with respect to the second rail point cloud 30bn.

[0112] FIGS. 8(a) to 8(e) are diagrams illustrating the operation of the automatic calibration unit 104 in the second embodiment.

[0113] FIG. 8 illustrates a coordinate system used in the present embodiment and a method for estimating the external parameter by the automatic calibration unit 104.

[0114] First, a coordinate system will be described.

[0115] The first sensor coordinate system (FIG. 8(a)) is a coordinate system based on the first sensor 101a. The rail detection unit 102 converts the information expressed in the first sensor coordinate system into the first vehicle fixed coordinate system' (FIG. 8(b)) using the external parameter design value, and performs rail detection. The first vehicle fixed coordinate system' is a coordinate system that transforms and transitions from the first sensor coordinate system to the first vehicle fixed coordinate system' (FIG. 8(b)) using the external parameter design value (rotation Rs12bn1, translation ts12bn1).

[0116] The second sensor coordinate system (FIG. 8(c)) is a coordinate system based on the second sensor 101b. The rail detection unit 102 converts the information expressed in the second sensor coordinate system into the second vehicle fixed coordinate system' (FIG. 8(d)) using the external parameter design value, and performs rail detection. The second vehicle fixed coordinate system' is a coordinate system that transforms and transitions from the second sensor coordinate system to the second vehicle fixed coordinate system' using the external parameter design value (rotation Rs22bn2, translation ts22bn2).

[0117] Next, an aspect of a rail detection result in each coordinate system will be described. 401Ls and 401Rs indicate detection results of the left and right rails 20Lt and 20Rt in the first sensor coordinate system (FIG. 8(a)). A left rail model 401Lbn and a right rail model 401Rbn are rail detection results in the first vehicle fixed coordinate system' (FIG. 8(b)). Further,402Ls and 402Rs indicate detection results of the left and right rails 20Lt and 20Rt in the second sensor coordinate system (FIG. 8(c)). Furthermore, a left rail model 402Lbn and a right rail model 402Rbn are rail detection results in the second vehicle fixed coordinate system' (FIG. 8(d)). Since the first vehicle fixed coordinate system' and the second vehicle fixed coordinate system' are different, 401Lbn and 402Lbn, and 401Rbn and 402Rbn do not match.

[0118] Here, the rotation and translation converting 401Lbn to 402Lbn and 401Rbn to 402Rbn are rotation Rbn12bn2 and translation tbn12bn2 from the first vehicle fixed coordinate system' to the second vehicle fixed coordinate system'. Therefore, in the present embodiment, the rotation Rbn12bn2 and the translation tbn12bn2 for converting a first representative position pbn1 and the first triaxial vector (exbn1, eybn1, ezbn1) of the first rail point cloud 30an in the first vehicle fixed coordinate system' into a second representative position pbn2 and the second triaxial vector (exbn2, eybn2, ezbn2) corresponding to the second rail point cloud 30bn in the second vehicle fixed coordinate system' are calculated.

[0119] As a method of calculating the first representative position pbn1, for example, an arbitrary x-direction position where the first rail point cloud 30an exists is set as an x-coordinate of the first representative position, and the first rail point cloud 30an within a predetermined range is extracted based on the x-coordinate of the first representative position, and the extracted first rail point cloud is set as the first rail point cloud after extraction. The left rail model 401Lbn and the right rail model 401Rbn are estimated for the first rail point cloud after extraction, and a middle line of the left rail model 401Lbn and the right rail model 401Rb is set as the y-coordinate of the first representative position. Furthermore, the z-coordinate corresponding to the x-coordinate of the first representative position and the y-coordinate of the first representative position on the three-dimensional plane defined by the left rail model 401Lbn and the right rail model 401Rb is set as the z-coordinate of the first representative position.

[0120] As a method of calculating the first triaxial vector (exbn1, eybn1, ezbn1), for example, a unit vector in the direction of the left rail model 401Lbn is set to exbn1, a normal vector of a plane defined by the left rail model 401Lbn and the right rail model 401Rb is set to ezbn1, and a unit vector eybn1 perpendicular to exbn1 and ezbn1 can be obtained by an outer product of exbn1 and ezbn1.

[0121] As a method of calculating the second representative position pbn2, the x-coordinate of the first representative position is defined as the x-coordinate of the second representative position on the assumption that the x-direction error between the first vehicle fixed coordinate system and the second vehicle fixed coordinate system is small. Further, the second rail point cloud 30bn within a predetermined range is extracted based on the x-coordinate of the second representative position, and the extracted second rail point cloud is set as the second rail point cloud after extraction. The left rail model 402Lbn and the right rail model 402Rbn are estimated for the second rail point cloud after extraction, and the middle line of the left rail model 402Lbn and the right rail model 402Rb is set as the y-coordinate of the second representative position. Furthermore, the z-coordinate corresponding to the x-coordinate of the second representative position and the y-coordinate of the second representative position on the three-dimensional plane defined by the left rail model 402Lbn and the right rail model 402Rb is set as the z-coordinate of the second representative position.

[0122] As a method of calculating the second triaxial vector (exbn2, eybn2, ezbn2), for example, a unit vector in the direction of the left rail model 402Lbn is set to exbn2, a normal vector of a plane defined by the left rail model 402Lbn and the right rail model 402Rb is set to ezbn2, and a unit vector eybn2 perpendicular to exbn2 and ezbn2 can be obtained by an outer product of exbn2 and ezbn2.

[0123] By solving the equation described in FIG. 8(e) using the first representative position pbn1, the first triaxial vector (exbn1, eybn1, ezbn1), the second representative position pbn2, and the second triaxial vector (exbn2, eybn2, ezbn2) obtained by the above method, the rotation Rbn12bn2 and the translation tbn12bn2 for conversion from the first vehicle fixed coordinate system' to the second vehicle fixed coordinate system can be obtained.

[0124] The method for calculating the external parameter using the first rail point cloud 30an and the second rail point cloud 30bn has been described above. According to the above method, since the external parameter of the first sensor 101a can be calculated by rail detection, highly accurate automatic calibration can be performed without newly installing a target. Therefore, the calibration cost in the system using the front monitoring device can be reduced, and the vehicle can travel more safely.

[0125] In the present embodiment, the calibration based on the detection result of the straight rail is the basis, but in the case of the second embodiment using the first rail point cloud 30an and the second rail point cloud 30bn, the first rail point cloud 30an and the second rail point cloud 30bn may be curved rails. In a case where the actual rails 20Lt and 20Rt are curved, there is a method of calculating the rotation and the translation by, for example, iterative closest point (ICP) matching between the first rail point cloud 30an and the second rail point cloud 30bn.[Third embodiment]

[0126] In the first embodiment and the second embodiment, the example in that the two-dimensional rail detection result is converted into a three-dimensional rail point cloud in the rail detection unit 102, and the external parameter is estimated in the automatic calibration unit 104 in a case where the first sensor 101a is a two-dimensional image. In a case where the two-dimensional rail point cloud detected by the monocular camera is converted into a three-dimensional model, it is difficult to convert the two-dimensional rail point cloud into a three-dimensional model unless an assumption that, for example, there is no constant gradient or no roll rotation around the x-axis is made in order to estimate the depth direction, and it is not easy to acquire an accurate three-dimensional point cloud. In a case where the target performance of the external parameter estimation result cannot be satisfied due to an error in reconstructing a three-dimensional model from the two-dimensional rail detection result by the monocular camera, it is necessary to consider a method of not performing three-dimensional reconstruction.

[0127] The third embodiment has been made in view of the above problems, and the external parameter is estimated on the two-dimensional image for monocular camera information in that three-dimensional reconstruction is difficult to be accurately performed.

[0128] FIG. 9 is a flowchart illustrating the operation of the automatic calibration device 11a in a case where the monocular camera information is used as the first sensor information.

[0129] First, the rail detection unit 102 acquires monocular camera information as first sensor information (S201).

[0130] Next, the rail detection unit 102 performs rail detection on the two-dimensional image (S202).

[0131] Next, the rail detection unit 102 calculates a three-dimensional rail point cloud by using the rail detection result on the two-dimensional image, acquires a two-dimensional rail point cloud by using the rail detection result on the two-dimensional image, and holds the two-dimensional rail point cloud for use in the automatic calibration unit 104 (S203).

[0132] Next, the calibration feasibility determination unit 103 calculates the linearity based on the rail point cloud information after the three-dimensional reconstruction (S204), and calculates the calibration feasibility signal (S205). The operation of the calibration feasibility determination unit 103 is similar to that of the first embodiment.

[0133] Although it has been described that inaccuracy may occur in a case where the two-dimensional rail point cloud is reconstructed into a three-dimensional model, a phenomenon occurs in which the three-dimensional point cloud reconstructed from the two-dimensional point cloud is uniformly different in angle such as a roll angle with respect to the correct three-dimensional point cloud accurately reconstructed into a three-dimensional model by some means, and thus, it can be considered that the relative positional relationship between the points in the point cloud is maintained. Therefore, in a case where the three-dimensional point cloud converted from the two-dimensional point cloud draws a curve, the correct three-dimensional point cloud is also a curve, and in a case where the three-dimensional point cloud converted from the two-dimensional point cloud has a high noise level, the correct three-dimensional point cloud also has a high noise level, and the linearity is considered to have the same value. Therefore, there is no problem even if the threshold determination using the linearity is performed by the method described in the first embodiment.

[0134] Next, the automatic calibration unit 104 determines whether the calibration feasibility signal is 1 (S206).

[0135] As a result, in a case where the calibration feasibility signal is 1 (Yes in S206), the automatic calibration unit 104 retrieves the point cloud 30at corresponding to the actual positions of the rails 20Lt and 20Rt as the second rail point cloud 30bn (S207).

[0136] Then, the automatic calibration unit 104 estimates the external parameter by performing matching between the first rail point cloud 30an and the second rail point cloud 30bn on the two-dimensional image (S208).

[0137] Furthermore, the sensor information correction unit 105 corrects the first sensor information (S209).

[0138] On the other hand, when the calibration feasibility signal is 0 (No in S206), the correction of the first sensor information is not performed, and the series of processing ends.

[0139] The second rail point cloud 30bn may be a three-dimensional rail point cloud in an ideal state as in the first embodiment or may be estimated by the second sensor 101b as in the second embodiment. In the third embodiment, a specific example in a case where the ideal rail point cloud is adopted as the second rail point cloud 30bn in the present embodiment will be described.

[0140] FIGS. 10(a) to 10(c) are diagrams illustrating details of the operation of S208 of FIG. 9 and a coordinate system definition to be used.

[0141] The image coordinate system (FIG. 10(c)) is a two-dimensional orthogonal coordinate system with, for example, the upper left of the image as an origin, and points on the image coordinate system are converted into a vehicle fixed coordinate system' (FIG. 10(b)) slightly different from the vehicle fixed coordinate system by using a camera attachment position, a design value of a posture, a camera matrix, and the like. The estimation target of the external parameter in the present embodiment is rotation Rcbn2b and translation tcbn2b from the vehicle fixed coordinate system' (FIG. 10(b)) to the vehicle fixed coordinate system (FIG. 10(a)).

[0142] In the present embodiment, in order to estimate rotation and translation by matching in the two-dimensional image coordinate system, the automatic calibration unit 104 first sets ideal rail point clouds 601Lb and 601Rb in the vehicle fixed coordinate system as in the first embodiment. Then, the automatic calibration unit 104 calculates rail point clouds 601Lbn and 601Rbn by performing conversion into the vehicle fixed coordinate system' using rotation Rb2cbn and translation tb2cbn that are inverse conversions of the rotation Rcbn2b and the translation tcbn2b. Further, the automatic calibration unit 104 converts the rail point clouds 601Lbn and 601Rbn into the image coordinate system using the external parameter design information of the first sensor 101a, the camera matrix, and the like, and obtains 601Lc and 601Rc. Since the rotation Rb2cbn and the translation tb2cbn are undefined, 601Lc and 601Rc are functions of the rotation Rb2cbn and the translation tb2cbn.

[0143] The automatic calibration unit 104 calculates, by numerical calculation, rotation Rb2cbn and translation tb2cbn that match the two-dimensional rail point clouds 602Lc and 602Rc detected by the 601Lc and 601Rc in the image coordinate system. The numerical calculation method may be a generally used optimization algorithm such as a steepest descent method or an annealing method.

[0144] In the third embodiment, it can also be said that the first rail point cloud 30an indicates two-dimensional positions of the rails 20Lt and 20Rt, and the automatic calibration unit 104 converts the reference rail point cloud (in this case, the second rail point cloud 30bn) into an image coordinate system including the first rail point cloud 30an to acquire a reference rail point cloud represented in two dimensions (the second rail point cloud 30bn), and calculates triaxial rotation and translation for matching the first rail point cloud 30an with the reference rail point cloud represented in two dimensions.

[0145] In the present embodiment, an angle error or the like of the rail point cloud, which occurs when the point cloud on the two-dimensional image is reconstructed into a three-dimensional model, does not occur. Therefore, it is expected that a more accurate estimation result of the external parameter can be obtained, and safer vehicle travel becomes possible.[Fourth embodiment]

[0146] In the first to third embodiments, the linearity of the rail point cloud is calculated to avoid the rail point cloud including the curve section from being used by the automatic calibration unit 104. In addition to the above, the fourth embodiment illustrates a method of grasping in advance a region where a rail point cloud includes a curve section and it is difficult to estimate an external parameter by using a railway digital map describing information associated with an on-track position of railroad, and more reliably estimating the external parameter.

[0147] FIG. 11 is a diagram illustrating an overall configuration of a sensor correction system 1c according to the fourth embodiment.

[0148] The sensor correction system 1c to be illustrated is different from the sensor correction system 1a of the first embodiment illustrated in FIG. 1 in that a railway digital map 106, an on-track position estimation sensor 107, and a presence position estimation unit 108 are added to the automatic calibration device 11c, but the other points are similar.

[0149] The railway digital map 106 is a database including at least information for determining whether or not to estimate an external parameter at a point, such as curvature or presence or absence of a branch at a point corresponding to an on-track position in a section where the vehicle 10 travels.

[0150] The on-track position estimation sensor 107 is a sensor necessary for estimating an on-track position of the vehicle 10, and examples thereof include a receiver of a railway wayside coil, an encoder for measuring the rotation speed of wheels, and a GNSS (Global Navigation Satellite System) receiver.

[0151] The presence position estimation unit 108 has a function of calculating the position of the vehicle on the track in accordance with the information acquired by the on-track position estimation sensor 107.

[0152] FIGS. 12(a) to 12(b) are diagrams illustrating a method of determining whether or not the external parameter can be estimated using the railway digital map 106 and the railway digital map 106.

[0153] FIG. 12(a) is a diagram illustrating a specific example of the railway digital map 106.

[0154] In the railway digital map 106, the curvature of the rail and the presence or absence of a branch are recorded for each on-track position (kilometrage [km]) along a middle line 701 of the left and right rails 20Lt and 20Rt. For example, the position 703a is a straight track having a kilometrage of 1.0 km, a curvature of 0, and no branch (0 means no branch, and 1 means presence of a branch). A hatched section of a region 702a indicates that the curvature and the branch information are the same as those at a position 703a. The position 703b is a point where a branch occurs and a curve is formed, and stores information that the curvature is 0.0016 and there is a branch. Furthermore, a position 703c is a straight track having a kilometrage of 2.5 km, a curvature of 0, and no branch. A hatched section of the region 702b indicates that the curvature and the branch information are the same as those at the position 703c.

[0155] FIG. 12(b) is a diagram illustrating a method of determining whether or not the external parameter can be estimated using the railway digital map 106.

[0156] For example, in a case where the vehicle 10 is present at a position 705a and the section of a region 706 where rail detection is possible is included in the region 702a, it is ensured that all the regions where rail detection is possible are on a straight track. This means that the calibration feasibility determination unit 103 determines that calibration is possible in a case where the first rail point cloud 30an is included in the calibration-capable section based on the map (In this case, the railway digital map 106) including the relationship between the on-track position of the vehicle 10 and the calibration-capable section.

[0157] On the other hand, in a case where a section of the region 706 in which the rail can be detected is not included in the region 702a, and includes a section in which the curvature exceeds the predetermined threshold or includes a section in which a branch exists, for example, as in the position 703b, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 0. As a result, it is possible to prevent the rail point cloud including curves and branches from being used by the automatic calibration unit 104. Therefore, it is expected that a more accurate estimation result of the external parameter can be obtained, and safer vehicle travel becomes possible.

[0158] In addition, in the calibration feasibility determination unit 103, in a case where the linearity falls below a predetermined threshold even though the front of the vehicle 10 is a straight line on which calibration can be performed, such as a straight section and a section not including a branch, the calibration feasibility determination unit 103 may issue an alert indicating that there is a sensor abnormality while setting the calibration feasibility signal to 0. In this case, it can also be said that the calibration feasibility determination unit 103 issues a sensor abnormality signal in a case where the first linearity is a value for which calibration cannot be performed and the first rail point cloud 30an is included in the calibration-capable section.

[0159] By adding a function of issuing an alert to the calibration feasibility determination unit 103, it is possible to detect a sensor abnormality and reflect the sensor abnormality in the operation of the vehicle 10, thereby enabling safer vehicle traveling.[Fifth embodiment]

[0160] If the rotation and translation of each sensor with respect to the vehicle fixed coordinate system can be calculated by the method described in the first embodiment, the sensor attachment abnormality can be detected by comparing the design value of the rotation and translation of each sensor with the current value. The fifth embodiment has been made in view of this, and the sensor attachment abnormality is detected using the design value and the calibration result of the first embodiment.

[0161] FIG. 13 is a diagram illustrating an overall configuration of a sensor correction system 1d according to the fifth embodiment.

[0162] The sensor correction system 1d to be illustrated is different from the sensor correction system 1a of the first embodiment illustrated in FIG. 1 in that a sensor attachment abnormality detection unit 109 is added to the automatic calibration device 11d, but the other points are similar.

[0163] The sensor attachment abnormality detection unit 109 determines that the sensor attachment angle is abnormal in a case where the triaxial rotation and translation obtained by the automatic calibration unit 104 exceed a predetermined threshold value.

[0164] The external parameter of the first sensor 101a output from the automatic calibration unit 104 is input to the sensor attachment abnormality detection unit 109. The sensor attachment abnormality detection unit 109 has a function of calculating a difference between the external parameter of the first sensor 101a output from the automatic calibration unit 104 and an external parameter design value of the first sensor 101a as an abnormality degree on each coordinate axis (biaxial or triaxial for translation and triaxial for rotation), and determining that the sensor attachment abnormality is occurred in a case where the abnormality degree exceeds a predetermined threshold.

[0165] As described in the second embodiment, it is also possible to determine the sensor attachment abnormality by using the first sensor 101a and the second sensor 101b and by using the estimation result of the first-to-second external parameters between the first sensor 101a and the second sensor 101b.

[0166] In this case, a third sensor different from the first sensor 101a and the second sensor 101b is used, and an estimation result of the first-to-fourth external parameters between the first sensor 101a and the third sensor is further acquired by the automatic calibration unit 104. Next, in a case where the element of the result of the estimation of the first-to-second external parameters exceeds the predetermined threshold, and the element of the result of the estimation of the first-to-fourth external parameters exceed the predetermined threshold, it is possible to estimate that the attachment abnormality of the first sensor 101a occurs.

[0167] In this case, it can also be said that the sensor attachment abnormality detection unit 109 determines that the sensor attachment angle is abnormal in a case where both the triaxial rotation and translation obtained from the first rail point cloud 30an and the reference rail point cloud as the second rail point cloud 30bn detected based on the second sensor information acquired by the second sensor 101b and indicating the positions of the rails 20Lt and 20Rt, and the triaxial rotation and translation obtained from the first rail point cloud 30an and a third rail point cloud detected based on third sensor information acquired by a third sensor and indicating the positions of the rails 20Lt and 20Rt exceed a predetermined threshold.

[0168] According to the fifth embodiment, it is possible to detect the attachment angle abnormality of each sensor in a section where the rail can be detected with a predetermined linearity or more. Therefore, it is possible to prevent a failure in detection of an object ahead due to an attachment angle abnormality and to travel more safely.

[0169] In the embodiments described in detail above, the case where the automatic calibration is performed and the automatic calibration is performed has been described, but the present invention is not limited thereto, and the calibration may be manually performed.

[0170] In the embodiments described in detail above, the linearity is an index for evaluating whether or not the rail point cloud is a straight line and the degree of variation in data of the rail point cloud. However, the degree of variation in data of the rail point cloud does not necessarily need to be taken into consideration as long as sufficient accuracy for determining a straight rail can be ensured only by determining whether or not the rail point cloud is a straight line.<Description of correction method>

[0171] The processing performed by the automatic calibration devices 11a to 11d described above is realized by cooperation of software and hardware resources. That is, a processor in a computer provided in each of the automatic calibration devices 11a to 11d loads software for realizing each function described above into a memory and executes the software to realize each function.

[0172] Therefore, the processing performed by the automatic calibration devices 11a to 11d can be considered as a correction method in which the processor executes the software recorded in the memory to detect the first rail point cloud 30an indicating the positions of the rails 20Lt and 20Rt based on the first sensor information acquired by the first sensor 101a, determine whether calibration of the first sensor 101a is possible based on the first linearity indicating the degree of the straight line with respect to the first rail point cloud 30an, and correct the first sensor information based on the first rail point cloud 30an and the reference rail point cloud indicating the three-dimensional positions of the rails 20Lt and 20Rt and different from the first rail point cloud 30an in a case where it is determined that calibration is possible.

[0173] Although the present embodiment has been described above, the technical scope of the present invention is not limited to the scope described in the above embodiment. It is apparent from the description of the claims that various modifications or improvements are added to the above embodiments within the technical scope of the present invention.Reference Signs List

[0174] 1a to 1dsensor correction system 11a to 11cautomatic calibration device 20Lt, 20Rtrail 30anfirst rail point cloud 30atpoint cloud 101afirst sensor 101bsecond sensor 102rail detection unit 103calibration feasibility determination unit 104automatic calibration unit 105sensor information correction unit 106railway digital map 107on-track position estimation sensor 108on-track position estimation unit 109sensor attachment abnormality detection unit

Examples

first embodiment

[First embodiment]

[0028]FIG. 1 is a diagram illustrating an overall configuration of a sensor correction system 1a according to a first embodiment.

[0029]As illustrated, the sensor correction system 1a is mounted on a vehicle 10. The vehicle 10 is a vehicle housing that travels on an iron track, and examples thereof include a railway and a tram. The vehicle 10 travels on the pair of rails 20Lt and 20Rt. Here, the rail on the left side in the traveling direction of the vehicle 10 is a rail 20Lt, and the rail on the right side in the traveling direction of the vehicle 10 is a 20Rt.

[0030]The sensor correction system 1a includes a first sensor 101a and an automatic calibration device 11a.

[0031]The automatic calibration device 11a is an example of a correction device, and includes a rail detection unit 102, a calibration feasibility determination unit 103, an automatic calibration unit 104, and a sensor information correction unit 105. Information acquired by the first sensor 101a is used...

second embodiment

[Second embodiment]

[0104]In the first embodiment, the method of estimating the external parameter of the first sensor 101a by comparing the rail detection result of the straight line with the rail straight line in the ideal state in the vehicle fixed coordinate system 40t has been described. On the other hand, in a case where it is not necessary to calculate the attachment position and angle of each sensor with respect to the sensor attachment angle design value, a method of integrating the sensor information in a second vehicle fixed coordinate system' using the first rail detection result detected by the first sensor 101a and the second rail detection result by the second sensor 101b different from the first sensor 101a may be used. Hereinafter, the details will be described as a second embodiment.

[0105]FIG. 7 is a diagram illustrating an overall configuration of a sensor correction system 1b according to the second embodiment.

[0106]The sensor correction system 1b to be illustrate...

third embodiment

[Third embodiment]

[0126]In the first embodiment and the second embodiment, the example in that the two-dimensional rail detection result is converted into a three-dimensional rail point cloud in the rail detection unit 102, and the external parameter is estimated in the automatic calibration unit 104 in a case where the first sensor 101a is a two-dimensional image. In a case where the two-dimensional rail point cloud detected by the monocular camera is converted into a three-dimensional model, it is difficult to convert the two-dimensional rail point cloud into a three-dimensional model unless an assumption that, for example, there is no constant gradient or no roll rotation around the x-axis is made in order to estimate the depth direction, and it is not easy to acquire an accurate three-dimensional point cloud. In a case where the target performance of the external parameter estimation result cannot be satisfied due to an error in reconstructing a three-dimensional model from the ...

Technical Field

[0001] The present invention relates to a correction device and a correction method. In particular, the present invention relates to a correction device and a correction method suitable for correcting sensor information output from a sensor mounted on a train.Background Art

[0002] For example, the train front monitoring system is expected to expand the front monitoring area and improve the object detection rate by mounting a plurality of sensors having different monitoring areas and frequency characteristics. In such a system, a plurality of pieces of sensor information is integrated by using an external parameter (attachment position and angle information) of each sensor. The external parameter of each sensor has an individual difference depending on the vehicle, and changes over time due to vehicle body vibration or the like, so that the actual external parameter has an error with respect to the design value. When this error increases, a position error of an object included in each sensor increases, and it becomes difficult to integrate sensor information. Therefore, in order to prevent failure caused by a sensor external parameter error when integrating a plurality of pieces of sensor information, it is necessary to calculate and correct external parameters such as the position and posture of each sensor. That is, a correction device or the like that performs calibration and corrects sensor information is required.

[0003] PTL 1 describes a camera calibration device including a measurement area including at least two reference lines that include a line intersecting two points corresponding to a shape of an own vehicle and are arranged within an imaging range of a camera, and a calibration pattern having a predetermined pattern shape between the reference lines, and an image processor. Then, in this device, the image processor includes: imaging processing of generating a measurement image by imaging a measurement area in which the calibration pattern and the reference line are arranged with the camera; stop error calculation processing of calculating a coordinate relationship between the stop position of the own vehicle and the pattern shape as a stop error based on the two reference lines; and external parameter calculation processing of calculating an external parameter corresponding to the attachment posture of the camera based on the pattern shape in a state where the stop error is canceled.

[0004] PTL 2 discloses a vehicle body inclination estimation device in which a rail detection unit that obtains a position of each rail from an image captured by a camera, a three-dimensional shape calculation unit that calculates a three-dimensional shape of the rail based on a position of each rail obtained from an image captured by the camera, and an inclination estimation unit that suppresses an influence of noise by using M estimation or RANSAC when aligning the three-dimensional shape of the rail with a rail shape model prepared in advance by an ICP algorithm are provided in an inclination estimation device based on an image obtained by capturing a front rail or a rear rail of a vehicle by the camera disposed along a transverse direction on a roof of the vehicle.Citation ListPatent Literature

[0005] PTL 1: JP 2013-2820 A PTL 2: JP 2019-23017 A Summary of InventionTechnical Problem

[0006] However, providing the measurement area is expected to require maintenance for preventing deterioration such as contamination and loss in addition to the provision cost, and thus it is desirable not to provide such a measurement area.

[0007] An object of the present invention is to provide a correction device and a correction method capable of performing more accurate calibration without providing calibration equipment.Solution to Problem

[0008] In order to solve the above problem, the present invention is a correction device including: a rail detection unit which detects a first rail point cloud indicating a position of a rail based on first sensor information acquired by a first sensor; a calibration feasibility determination unit which determines feasibility of calibration of the first sensor based on a first linearity indicating a degree of a straight line with respect to the first rail point cloud; and a sensor information correction unit which corrects the first sensor information based on the first rail point cloud and a reference rail point cloud that indicates a three-dimensional position of a rail and is different from the first rail point cloud in a case where the calibration feasibility determination unit determines that calibration is possible. In this case, it is possible to provide a correction device capable of performing more accurate calibration without providing calibration equipment.

[0009] Here, the first linearity can indicate the degree of variation with respect to a first rail point cloud in addition to the degree of the straight line. In this case, the accuracy of the evaluation of the linearity can be improved.

[0010] In addition, the calibration feasibility determination unit can obtain the first linearity based on an average value of distances between a rail straight line model estimated by performing straight line extraction from the first rail point cloud and points included in the first rail point cloud. In this case, the linearity can be more easily obtained.

[0011] Furthermore, the calibration feasibility determination unit can obtain a first distance average value and a second distance average value as the average value for each of the pair of rails, and sets a reciprocal of an average value of the first distance average value and the second distance average value as the first linearity. In this case, a more intuitive value can be obtained as the linearity.

[0012] Furthermore, the calibration feasibility determination unit can determine presence or absence of a branch from the first rail point cloud, and determines that calibration is not possible in a case where there is a branch. In this case, the estimation of the external parameter is more accurate.

[0013] Then, the calibration feasibility determination unit can determine presence or absence of a branch by comparing a distance in a width direction of a vehicle with a gauge of the rail for the first rail point cloud. In this case, the branch determination can be performed more easily.

[0014] Furthermore, a reference rail point cloud can be a second rail point cloud that is detected based on second sensor information acquired by a second sensor and indicates a position of a rail. In this case, the sensor information of the first sensor and the second sensor can be integrated.

[0015] Furthermore, the calibration feasibility determination unit can determine feasibility of calibration based on a difference between the first linearity and a second linearity indicating a degree of a straight line with respect to the second rail point cloud. In this case, it is possible to more accurately determine whether calibration is performed.

[0016] Furthermore, a calibration unit which obtains triaxial rotation and translation of the first sensor by using the first rail point cloud and the reference rail point cloud in a case where the calibration feasibility determination unit determines that calibration is possible is further included, and the sensor information correction unit corrects the first sensor information using the triaxial rotation and the translation obtained by the calibration unit. In this case, the accuracy of correction of the first sensor information is improved.

[0017] Furthermore, the calibration unit can record a plurality of times of the triaxial rotation and the translation of the first sensor and outputs a median value of the triaxial rotation and the translation of the first sensor at a plurality of times, and the sensor information correction unit corrects the first sensor information using the output median value. In this case, it is possible to more accurately estimate the external parameter.

[0018] Furthermore, the first rail point cloud indicates a two-dimensional position of a rail, and the calibration unit can convert the reference rail point cloud into an image coordinate system including the first rail point cloud, acquires the reference rail point cloud represented in two dimensions, and calculates the triaxial rotation and the translation for matching the first rail point cloud with the reference rail point cloud represented in two dimensions. In this case, an angle error or the like of the rail point cloud, which occurs when the point cloud on the two-dimensional image is reconstructed into a three-dimensional model, does not occur, and it is easy to obtain a more accurate estimation result of the external parameter.

[0019] Furthermore, it is possible to further include a sensor attachment abnormality detection unit which determines that the sensor attachment angle is abnormal in a case where the triaxial rotation and the translation obtained by the calibration unit exceed a predetermined threshold. In this case, sensor attachment abnormality can be detected.

[0020] Then, the sensor attachment abnormality detection unit can determine that the sensor attachment angle is abnormal in a case where both the triaxial rotation and the translation obtained from the first rail point cloud and a reference rail point cloud as a second rail point cloud detected based on second sensor information acquired by a second sensor and indicating the position of the rail, and the triaxial rotation and the translation obtained from the first rail point cloud and a third rail point cloud detected based on third sensor information acquired by a third sensor and indicating a position of a rail exceed a predetermined threshold. In this case, the sensor attachment abnormality can be determined using the first sensor and the second sensor.

[0021] Furthermore, the calibration feasibility determination unit can determine that calibration is possible in a case where the first rail point cloud is included in a calibration-capable section based on a map including a relationship between an on-track position of a vehicle and a calibration-capable section. In this case, it is possible to grasp in advance the region including a curve section where it is difficult to estimate the external parameter, and to more reliably estimate the external parameter.

[0022] Then, the calibration feasibility determination unit can issue a sensor abnormality signal in a case where the first linearity is a value indicating that calibration is not possible and the first rail point cloud is included in a calibration-capable section. In this case, the sensor attachment abnormality can be detected using the map.

[0023] Furthermore, the rail detection unit may detect a point cloud indicating positions of objects parallel to each other instead of the first rail point cloud. For example, the rail detection unit detects a first overhead line column point cloud instead of the first rail point cloud. In this case, a point cloud necessary for calibration can be acquired using objects parallel to each other exemplified by the overhead line columns.

[0024] In addition, the present invention is a correction method including, by a processor executing software recorded in the memory, detecting a first rail point cloud indicating a position of a rail based on first sensor information acquired by a first sensor; determining feasibility of calibration of the first sensor based on a first linearity indicating a degree of a straight line with respect to the first rail point cloud; and correcting the first sensor information based on the first rail point cloud and a reference rail point cloud that indicates a three-dimensional position of a rail and is different from the first rail point cloud in a case where it is determined that the calibration is possible. In this case, it is possible to provide a correction method capable of performing more accurate calibration without providing calibration equipment.Advantageous Effects of Invention

[0025] According to the present invention, it is possible to provide a correction device and a correction method capable of performing more accurate calibration without providing calibration equipment.Brief Description of Drawings

[0026] [FIG. 1] FIG. 1 is a diagram illustrating an overall configuration of a sensor correction system according to a first embodiment. [FIG. 2] FIG. 2 is a flowchart illustrating an operation from acquisition of first sensor information to correction of information of the first sensor in the first embodiment. [FIG. 3] FIGS. 3(a) to 3(b) are diagrams illustrating a method of calculating linearity. FIGS. 3(c) to 3(e) are diagrams illustrating specific examples when calculating the linearity. [FIG. 4] FIGS. 4(a) to 4(b) are diagrams illustrating specific examples of a method of performing branch determination in a case where a branch determination function is added to the calibration feasibility determination unit. [FIG. 5] FIGS. 5(a) to 5(d) are diagrams illustrating a coordinate system used in the first embodiment and a method of estimating an external parameter using a first rail point cloud. [FIG. 6] FIG. 6 is a diagram in which a horizontal axis indicates the number of accumulated frames of an external parameter estimation result and a vertical axis indicates an external parameter estimation result. [FIG. 7] FIG. 7 is a diagram illustrating an overall configuration of a sensor correction system according to a second embodiment. [FIG. 8] FIGS. 8(a) to 8(e) are diagrams illustrating an operation of an automatic calibration unit in the second embodiment. [FIG. 9] FIG. 9 is a flowchart illustrating an operation of the automatic calibration device in a case where a monocular camera information is used as the first sensor information. [FIG. 10] FIGS. 10(a) to 10(c) are diagrams illustrating details of the operation in S208 of FIG. 9 and a coordinate system definition to be used. [FIG. 11] FIG. 11 is a diagram illustrating an overall configuration of a sensor correction system according to a fourth embodiment. [FIG. 12] FIGS. 12(a) to 12(b) are diagrams illustrating a railway digital map and a method of determining whether or not an external parameter can be estimated using a railway digital map. [FIG. 13] FIG. 13 is a diagram illustrating an overall configuration of a sensor correction system according to a fifth embodiment. Description of Embodiments

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Here, the present invention will be described with reference to the first to fifth embodiments.[First embodiment]<Description of overall configuration of sensor correction system 1a>

[0028] FIG. 1 is a diagram illustrating an overall configuration of a sensor correction system 1a according to a first embodiment.

[0029] As illustrated, the sensor correction system 1a is mounted on a vehicle 10. The vehicle 10 is a vehicle housing that travels on an iron track, and examples thereof include a railway and a tram. The vehicle 10 travels on the pair of rails 20Lt and 20Rt. Here, the rail on the left side in the traveling direction of the vehicle 10 is a rail 20Lt, and the rail on the right side in the traveling direction of the vehicle 10 is a 20Rt.

[0030] The sensor correction system 1a includes a first sensor 101a and an automatic calibration device 11a.

[0031] The automatic calibration device 11a is an example of a correction device, and includes a rail detection unit 102, a calibration feasibility determination unit 103, an automatic calibration unit 104, and a sensor information correction unit 105. Information acquired by the first sensor 101a is used as an input. Note that, in the present embodiment, the automatic calibration unit 104 is mounted on the vehicle 10, but may be mounted on a facility provided on the ground in a case where a means for transmitting information of the sensor 101a to the ground is provided.

[0032] In the automatic calibration device 11a, using the data acquired by the first sensor 101a, the rail detection unit 102 estimates the positions of the left and right rails 20Lt and 20Rt in front of the vehicle 10 and acquires the estimated positions as a first rail point cloud 30an. Further, the calibration feasibility determination unit 103 determines the feasibility of calibration based on the linearity of the first rail point cloud 30an. Then, in a case where it is determined that calibration is possible, the automatic calibration unit 104 estimates the external parameter. Furthermore, the sensor information correction unit 105 corrects the data acquired by the first sensor 101a using the estimation result of the external parameter.

[0033] The first sensor 101a is a sensor that outputs first sensor information to be a source for estimating the first rail point cloud 30an by the rail detection unit 102. Examples of the first sensor 101a include a laser radar, a stereo camera, and a monocular camera. The first sensor information is a three-dimensional point cloud in a case where the first sensor 101a is a laser radar. In a case where the first sensor 101a is a stereo camera, the first sensor is a parallax image and left and right camera images. Furthermore, in a case where the first sensor 101a is a monocular camera, the first sensor is a two-dimensional image.

[0034] The rail detection unit 102 has a function of estimating a three-dimensional point cloud using the first sensor information. It can also be said that the rail detection unit 102 detects the first rail point cloud 30an indicating the positions of the rails 20Lt and 20Rt based on the first sensor information acquired by the first sensor 101a.

[0035] In a case where the first sensor information is a three-dimensional point cloud output from the laser radar, the rail detection unit 102 extracts a point cloud corresponding to the left and right rails 20Lt and 20Rt from the three-dimensional point cloud, and sets the extracted point cloud as a first rail point cloud 30an. In a case where the first sensor information is the parallax image of the stereo camera and the left and right camera images, the rail detection unit 102 estimates the first rail point cloud 30an by using two-dimensional rail extraction using pattern matching or the like on an image, depth estimation results by the parallax image, and the like. In a case where the input image is a monocular camera image, the rail detection unit 102 performs two-dimensional rail extraction using, for example, pattern matching on an image or the like, and estimates the first rail point cloud 30an using a depth estimation result based on a comparison between actual widths (=gauges) of the left and right rails 20Lt and 20Rt and the observed gauge.

[0036] The first rail point cloud 30an is a point cloud corresponding to the three-dimensional positions of the left and right rails 20Lt and 20Rt, and can be, for example, a point cloud corresponding to the inside of the uppermost surfaces of the left and right rails 20Lt and 20Rt. In the present embodiment, it is assumed that the first rail point cloud 30an is expressed by a vehicle fixed coordinate system' 40n in which the first rail point cloud 30an is subjected to coordinate conversion using a rotation design value and a translation design value for conversion from the first sensor coordinate system to a vehicle fixed coordinate system 40t fixed to the vehicle 10.

[0037] The vehicle fixed coordinate system 40t is a coordinate system in which the origin and the direction are fixed with respect to the vehicle 10. In the present embodiment, an x-direction of the vehicle fixed coordinate system 40t is defined as a straight direction along the rails 20Lt and 20Rt, a y-direction is defined as a direction along the ties, a z-direction is defined as a direction perpendicular to x and y, an x-origin is defined as a front surface of the vehicle body, a y-origin is defined as a middle line of the left and right rails 20Lt and 20Rt, and a z-origin is defined as a plane defined by the upper surfaces of the left and right rails 20Lt and 20Rt, but other methods may be used.

[0038] The vehicle fixed coordinate system' 40n is a coordinate system that transitions as a result of coordinate-transforming the sensor information in the first sensor coordinate system using the rotation design value and the translation design value. Since the rotation design value and the translation design value are different from the actual rotation and translation, the origin and the direction of the axis of the vehicle fixed coordinate system' 40n are different from those of the actual vehicle fixed coordinate system 40t. Therefore, the first rail point cloud 30an exists at a position different from a point cloud 30at corresponding to the positions of the actual rails 20Lt and 20Rt in the vehicle fixed coordinate system 40t.

[0039] The calibration feasibility determination unit 103 evaluates the first rail point cloud 30an based on the linearity. Then, the calibration feasibility determination unit 103 sets a threshold value for the linearity and determines the feasibility of calibration based on the threshold value. For example, the calibration feasibility determination unit 103 determines that calibration is possible in a case where the linearity is greater than or equal to a predetermined threshold value, and determines that calibration is impossible in a case where the linearity is less than the predetermined threshold value. Therefore, it can also be said that the calibration feasibility determination unit 103 determines the feasibility of calibration of the first sensor 101a based on the linearity indicating the degree of the straight line with respect to the first rail point cloud 30an. Note that this linearity can also be said to be a first linearity in the sense of distinguishing it from a second linearity to be described later.

[0040] The calibration feasibility determination unit 103 has a function of outputting a calibration feasibility signal for determining whether calibration is possible or not and the first rail point cloud 30an to the automatic calibration unit 104.

[0041] The linearity is an index for evaluating whether or not the estimated first rail point cloud 30an is a straight line and a degree of variation in data of the first rail point cloud 30an, and details thereof will be described later. The linearity threshold is a value that is adjusted in advance so that the estimation accuracy of the external parameter in the automatic calibration unit 104 can satisfy the requirement.

[0042] The calibration feasibility signal is information for determining whether the automatic calibration unit 104 may execute the external parameter estimation processing using the first rail point cloud 30an. The calibration feasibility signal is, for example, a signal of 0 or 1, and 0 means that calibration is impossible, and 1 means that calibration is possible.

[0043] The automatic calibration unit 104 has a function of estimating an external parameter of the first sensor 101a using the first rail point cloud 30an and the point cloud 30at corresponding to the positions of the actual rails 20Lt and 20Rt in a case where it is determined that calibration is possible by the calibration feasibility signal. Therefore, in a case where the calibration feasibility determination unit 103 determines that calibration is possible, the automatic calibration unit 104 functions as a calibration unit that obtains triaxial rotation and translation of the first sensor 101a by using the first rail point cloud 30an and the reference rail point cloud (in the first embodiment, the point cloud 30at corresponding to the positions of the actual rails 20Lt and 20Rt). Details of a method of estimating the external parameter using the first rail point cloud 30an will be described later.

[0044] The sensor information correction unit 105 has a function of converting data of the first sensor 101a into the vehicle fixed coordinate system 40t using the estimation result of the external parameter of the first sensor 101a. It can also be said that the sensor information correction unit 105 corrects the first sensor information based on the first rail point cloud 30an and a reference rail point cloud (in the first embodiment, the point cloud 30at corresponding to the actual positions of the rails 20Lt and 20Rt) indicating the three-dimensional positions of the rails 20Lt and 20Rt and different from the first rail point cloud an in a case where the calibration feasibility determination unit 103 determines that calibration is possible. In this case, the sensor information correction unit 105 corrects the first sensor information using the triaxial rotation and translation obtained by the automatic calibration unit 104.

[0045] FIG. 2 is a flowchart illustrating an operation from acquisition of the first sensor information to correction of the information of the first sensor 101a in the first embodiment.

[0046] First, the rail detection unit 102 acquires first sensor information at a certain time (S101).

[0047] Next, the rail detection unit 102 determines whether the data dimension of the first sensor information is three-dimensional (S102).

[0048] As a result, in a case where the data dimension is three-dimensional (Yes in S102), the rail detection unit 102 detects the rails 20Lt and 20Rt for the three-dimensional point cloud (S103). A case where the data dimension is three-dimensional is, for example, a case of a point cloud acquired by a laser radar.

[0049] On the other hand, in a case where the data dimension is two-dimensional (No in S102), the rail detection unit 102 first detects the rails 20Lt and 20Rt on the two-dimensional image (S104), performs three-dimensional reconstruction of the rail detection result using the parallax image and the gauge (S105), and outputs the rail detection result to the calibration feasibility determination unit 103. A case where the data dimension is two-dimensional is, for example, a case of a stereo camera image or a monocular camera image.

[0050] Next, the calibration feasibility determination unit 103 calculates the linearity by the first rail point cloud 30an (S106).

[0051] Then, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 1 in a case where the linearity is greater than or equal to a predetermined threshold value, and sets the calibration feasibility signal to 0 in a case where the linearity is less than the predetermined threshold value (S107).

[0052] Next, the automatic calibration unit 104 determines whether the calibration feasibility signal is 1 (S108).

[0053] As a result, in a case where the calibration feasibility signal is 1 (Yes in S108), the automatic calibration unit 104 retrieves the point cloud 30at corresponding to the actual positions of the rails 20Lt and 20Rt as the second rail point cloud (the reference rail point cloud) (S109).

[0054] Then, the automatic calibration unit 104 estimates the external parameter of the first sensor 101a by calculating the rotation and translation between the first rail point cloud 30an and the second rail point cloud (S110).

[0055] Furthermore, the sensor information correction unit 105 corrects the first sensor information (S111).

[0056] On the other hand, when the calibration feasibility signal is 0 (No in S108), the correction of the first sensor information is not performed, and the series of processing ends.<Explanation of linearity>

[0057] Next, the linearity will be described in detail.

[0058] FIGS. 3(a) to 3(b) are diagrams illustrating a method of calculating the linearity. FIGS. 3(c) to 3(e) are diagrams illustrating specific examples when calculating the linearity.

[0059] First, as illustrated in FIG. 3(a), the calibration feasibility determination unit 103 estimates a left rail straight line model 201L along the point cloud corresponding to the left rail 20Lt with respect to the first rail point cloud 30an obtained by the rail detection unit 102. In addition, the calibration feasibility determination unit 103 estimates a right rail straight line model 201R along a point cloud corresponding to the right rail 20Rt. As a method of estimating these straight line models, for example, straight line estimation by RANSAC (RANdom SAmple Consensus) is performed on the first rail point cloud 30an, and estimate one of the left and right straight line models as the first straight line model. Then, straight line estimation by RANSAC is performed again by removing a point cloud within a predetermined distance range with respect to the first straight line model, and estimate the other straight line model as the second straight line model. Furthermore, among the first straight line model and the second straight line model, the left rail straight line model 201L can be set to have a larger y-intercept, and the right rail straight line model 201R can be set to have a smaller y-intercept.

[0060] Next, the calibration feasibility determination unit 103 extracts a point cloud 204L existing in a range 203L within a predetermined distance 202 with respect to the left rail straight line model 201L and a point cloud 204R existing in a range 203R within the predetermined distance 202 with respect to the right rail straight line model 201R. The distance 202 is a value that can include a curved rail point cloud in the ranges 203L and 203R.

[0061] Furthermore, as illustrated in FIG. 3(b), the calibration feasibility determination unit 103 calculates a distance 205Li between a point 204Li included in the point cloud 204L and the left rail straight line model 201L over all points of the point cloud 204L, and calculates a first distance average value calculated by averaging. In addition, the calibration feasibility determination unit 103 calculates a distance 205Ri between a point 204Ri of the point cloud 204R and the right rail straight line model 201R over all points of the point cloud 204R, and calculates a second distance average value calculated by averaging. Next, the calibration feasibility determination unit 103 calculates a third distance average value by averaging the first distance average value and the second distance average value. Then, the calibration feasibility determination unit 103 defines the reciprocal of the third distance average value as the linearity. In this case, the larger the value of the linearity, the higher the degree of linearity of the rails 20Lt and 20Rt.

[0062] As described above, the calibration feasibility determination unit 103 obtains the linearity (the first linearity) based on the average value of the distances between the rail straight line models 201L and 201R estimated by performing straight line extraction from the first rail point cloud 30an and the points included in the first rail point cloud 30an. In addition, the calibration feasibility determination unit 103 obtains the first distance average value and the second distance average value as the average value for each of the pair of rails 20Lt and 20Rt, and sets the reciprocal of the average value of the first distance average value and the second distance average value as the linearity (the first linearity).

[0063] Next, a difference in linearity according to the aspect of the first rail point cloud 30an will be described.

[0064] FIG. 3(c) is a specific example in a case where the first rail point cloud 30an with less variation can be acquired with respect to the straight rails 20Lt and 20Rt. In such a case, since the third distance average value becomes small, the linearity, which is the reciprocal of the third distance average value, becomes a large value.

[0065] On the other hand, FIG. 3(d) is a specific example in a case where the first rail point cloud 30an having a large variation is acquired with respect to the straight rails 20Lt and 20Rt. In a case where such a rail detection result is used in the automatic calibration unit 104, the external parameters are estimated based on an inaccurate straight line model, which may lead to an increase in an estimation error of the external parameters. Therefore, it should be determined that calibration is not possible. In the case of FIG. 3(d), for example, since there is a point cloud 206 separated from the left rail straight line model 201L, the third distance average value becomes large, and the linearity, which is the reciprocal of the third distance average value, becomes a smaller value in comparison with the case in FIG. 3(c).

[0066] FIG. 3(e) is a specific example in a case where the first rail point cloud 30an is acquired for the curved rails 20Lt and 20Rt. In the present embodiment, it is assumed that the first rail point cloud 30an input to the automatic calibration unit 104 is a straight rail, and it is necessary to prevent the point cloud corresponding to the curved rails 20Lt and 20Rt from being input to the automatic calibration unit 104. In a case of FIG. 3(e), for example, since there is a point cloud 208 whose distance from the left rail straight line model 201L is long due to the rails 20Lt and 20Rt being curved, the third distance average value becomes large, and the linearity, which is the reciprocal of the third distance average value, becomes a larger value in comparison with the case in FIG. 3(c).

[0067] Therefore, in a case where the linearity in the cases of FIGS. 3(d) and 3(e) is obtained, the calibration feasibility determination unit 103 sets a threshold value for setting the calibration feasibility signal to 0. As a result, in a case where the variation of the first rail point cloud 30an is large and in a case where the rail is a curved rail, calibration is not performed, and it is possible to calculate more accurate external parameters.

[0068] The definition and the property of the linearity calculated by the calibration feasibility determination unit 103 have been described above.

[0069] In the embodiment described above, the calibration feasibility determination unit 103 basically determines the value of the calibration feasibility signal only based on the linearity, but the calibration feasibility signal may be set to 0 in a case where the rails 20Lt and 20Rt branch. Specifically, in a case where the rail detection unit 102 cannot stably extract a straight rail due to branching of the rails 20Lt and 20Rt, the calibration feasibility determination unit 103 may further add a function of determining that the rails 20Lt and 20Rt branch and setting the calibration feasibility signal to 0.

[0070] FIGS. 4(a) to 4(b) are diagrams illustrating specific examples of a method of performing branch determination in a case where a branch determination function is added to the calibration feasibility determination unit 103.

[0071] As illustrated in FIG. 4(a), the calibration feasibility determination unit 103 calculates a distance 212 (Δy) between a point 211L having the largest y and a point 211R having the smallest y among point clouds existing in a section equally divided by a width 210 in the x-direction of the vehicle fixed coordinate system 209. Then, as illustrated in FIG. 4(b), the calibration feasibility determination unit 103 creates a diagram in which the distance 212 (Δy) is plotted for each x-direction section. The x-coordinate in each x-direction section is, for example, the midpoint between the maximum value and the minimum value of each x-direction section.

[0072] FIG. 4(b) is a diagram plotting the distance 212 (Δy) for each x-direction section. In FIG. 4(b), if the rails 20Lt and 20Rt are straight, the distance 212 substantially matches the gauge D of the rails 20Lt and 20Rt. However, in a case where a branch occurs, the distance between the point 211L and the point 211R increases, so that the distance 212 becomes a value larger than the gauge D. Therefore, for example, a value slightly larger than the gauge D is set as a branch threshold 214, and in a case where the distance 212 is larger than the branch threshold 214, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 0. That is, the calibration feasibility determination unit 103 determines the presence or absence of a branch from the first rail point cloud 30an, and determines that calibration is not possible in a case where there is a branch. At this time, the calibration feasibility determination unit 103 determines the presence or absence of branch by comparing the distance in the width direction of the vehicle with the gauge D of the rails 20Lt and 20Rt for the first rail point cloud 30an.

[0073] In FIG. 4(b), a case where the distance 212 is larger than the branch threshold 214 is indicated by a point 216, for example. In FIG. 4(b), a case where the distance 212 is smaller than the branch threshold 214 is indicated by a point 215, for example. As a result, in a case where there is a branch even in part in the acquired first rail point cloud 30an, it is possible to prevent the estimation of the external parameter from being inaccurate when the branch occurs, and to estimate the external parameter with higher accuracy.

[0074] In addition, the calibration feasibility determination unit 103 may further have a function of setting the calibration feasibility signal to 0 in a case where the degree of parallelism between the rail detection result and the ground is less than a predetermined threshold value. In the present exemplary embodiment, the degree of parallelism is an index indicating how parallel two planes are.

[0075] Specifically, first, a first normal vector corresponding to a plane defined by the first rail point cloud 30an is calculated. Next, a point cloud in the vicinity of the first rail point cloud 30an is extracted from the first sensor information, and calculate a second normal vector corresponding to a plane defined by the point cloud remaining by removing the first rail point cloud 30an from the extracted point cloud and indicating the ground. Finally, a result of performing absolute value processing on a reciprocal of an angle formed by the first normal vector and the second normal vector is defined as the degree of parallelism. In this case, the larger the value of the degree of parallelism is, the closer the degree of parallelism is. The point cloud in the vicinity of the first rail point cloud 30an is, for example, a point cloud existing inside the left and right rails 20Lt and 20Rt.

[0076] Since the plane defined by the first rail point cloud 30an and the plane of the ground near the rails 20Lt and 20Rt are substantially parallel, in a case where the degree of parallelism is small, there is a possibility that the rail detection result indicates that it is not parallel to the ground and the rail detection result is invalid. Then, if a threshold is set for the degree of parallelism and such an invalid rail detection result is not used by the automatic calibration unit 104 by the threshold determination of the degree of parallelism, more accurate external parameter calculation can be performed.

[0077] Next, the operation of the automatic calibration unit 104 will be described in detail.

[0078] FIGS. 5(a) to 5(d) are diagrams illustrating a coordinate system used in the first embodiment and a method of estimating an external parameter using the first rail point cloud 30an.

[0079] FIG. 5(a) is a diagram illustrating a first sensor coordinate system. The first sensor coordinate system is a coordinate system based on the first sensor 101a. The rail detection unit 102 converts the information expressed in the first sensor coordinate system into the vehicle fixed coordinate system' 40n (FIG. 5(b)) using the external parameter design value, and performs rail detection. The vehicle fixed coordinate system' 40n is a coordinate system that transitions from the first sensor coordinate system using the external parameter design value (rotation Rs2bn, translation ts2bn), and is a coordinate system slightly different from the vehicle fixed coordinate system 40t (FIG. 5(c)).

[0080] Since the external parameter for performing conversion between the first sensor coordinate system (FIG. 5(a)) and the vehicle fixed coordinate system 40t (FIG. 5(c)) is different from the external parameter design value, a mismatch occurs between the vehicle fixed coordinate system' 40n (FIG. 5(b)) and the vehicle fixed coordinate system (FIG. 5(c)). The estimation of the external parameter is to calculate rotation and translation necessary for converting the first sensor information into the vehicle fixed coordinate system 40t (FIG. 5(c)) instead of the vehicle fixed coordinate system' 40n (FIG. 5(b)), by calculating the rotation Rbn2b and the translation tbn2b for conversion from the vehicle fixed coordinate system' 40n (FIG. 5(b)) to the vehicle fixed coordinate system 40t (FIG. 5(c)).

[0081] Next, an aspect of a rail detection result in each coordinate system will be described. 301Ls and 301Rs indicate detection results of the left and right rails 20Lt and 20Rt in the first sensor coordinate system (FIG. 5(a)). A left rail model 301Lbn and the right rail model 301Rbn are rail detection results in the vehicle fixed coordinate system' 40n (FIG. 5(b)). Further, the left rail model 301Lbt and the right rail model 301Rbt are the actual positions of the rails 20Lt and 20Rt in the vehicle fixed coordinate system (FIG. 5(c)). As shown in FIG. 5(b), since the vehicle fixed coordinate system' 40n and the vehicle fixed coordinate system 40t are different, the left rail model 301Lbn and the left rail model 301Lbt, and the right rail model 301Rbn and the right rail model 301Rbt do not match.

[0082] Here, rotation and translation for converting the left rail model 301Lbn into the left rail model 301Lbt and the right rail model 301Rbn into the right rail model 301Rbt are rotation Rbn2b and translation tbn2b from the vehicle fixed coordinate system' 40n (FIG. 5(b)) to the vehicle fixed coordinate system 40t (FIG. 5(c)). Therefore, in the present embodiment, the rotation Rbn2b and the translation tbn2b for converting a first representative position pbn and a first triaxial vector (exbn, eybn, ezbn) of the first rail point cloud 30an in the vehicle fixed coordinate system' 40n (FIG. 5(b)) into a second representative position pb and a second representative triaxial vector (exb, eyb, ezb) corresponding to the first rail point cloud 30an in the vehicle fixed coordinate system (FIG. 5(c)) are calculated.

[0083] As a method of calculating the first representative position pbn, for example, an arbitrary x-direction position where the first rail point cloud 30an exists is set as an x-coordinate of the first representative position, the first rail point cloud 30an within a predetermined yz range is extracted based on the x-coordinate of the first representative position, and the first rail point cloud after extraction is set. The left rail model 301Lbn and the right rail model 301Rbn are estimated for the first rail point cloud after extraction, and a middle line of the left rail model 301Lbn and the right rail model 301Rb is set as the y-coordinate of the first representative position. Furthermore, the z-coordinate corresponding to the x-coordinate of the first representative position and the y-coordinate of the first representative position on the three-dimensional plane defined by the left rail model 301Lbn and the right rail model 301Rb is set as the z-coordinate of the first representative position.

[0084] The first representative position pbn is arbitrary in the x-direction. However, based on the finding that there is a possibility that the angle estimation accuracy is improved when the distant object detection information is used, the x-coordinate may be set so as to use the farther first rail point cloud 30an, thereby improving the estimation accuracy of the external parameter related to the angle.

[0085] As a method of calculating the first triaxial vector (exbn, eybn, ezbn), for example, a unit vector in the x-direction of the left rail model 301Lbn is set to exbn, a normal vector of a plane defined by the left rail model 301Lbn and the right rail model 301Rb is set to ezbn, and a unit vector eybn perpendicular to exbn and ezbn can be obtained by an outer product of exbn and ezbn.

[0086] As for the method of calculating the second representative position pb, first, in the case of a straight rail having no vehicle body inclination, since the positions where the rails 20Lt and 20Rt exist in the vehicle fixed coordinate system are known, a rail point cloud in the vehicle fixed coordinate system corresponding to the actual positions of the rails 20Lt and 20Rt in the vehicle fixed coordinate system 40t is set.

[0087] Next, the x-coordinate of the second representative position pb is defined as the x-coordinate of the first representative position pbn on the assumption that an x-direction error between the vehicle fixed coordinate system' 40n (FIG. 5(b)) and the vehicle fixed coordinate system 40t (FIG. 5(c)) is small. Further, a first rail point cloud 30an within a predetermined range based on the x-coordinate of the first representative position pbn is extracted and defined as an ideal rail point cloud. The left rail model 301Lbt and the right rail model 301Rbt are estimated with respect to the ideal rail point cloud, and the middle line between the left rail model 301Lbt and the right rail model 301Rt is set as the y-coordinate of the second representative position pb. Furthermore, the z-coordinate corresponding to the x-coordinate of the first representative position pbn and the y-coordinate of the first representative position pbn on the three-dimensional plane defined by the left rail model 301Lbt and the right rail model 301Rt is defined as the z-coordinate of the second representative position pb.

[0088] Regarding the method of calculating the second triaxial vector (exb, eyb, ezb), for example, the unit vector in the x-direction of the left rail model 301Lbt is set to exb, the normal vector of the plane defined by the left rail model 301Lbt and the right rail model 301Rt is set to ezb, and the unit vector eyb perpendicular to exb and ezb can be obtained by the outer product of exb and ezb.

[0089] By solving the equation illustrated in FIG. 5(d) using the first representative position pbn, the first triaxial vector (exbn, eybn, ezbn), the second representative position pb, and the second triaxial vector (exb, eyb, ezb) obtained by the above method, the rotation Rbn2b and the translation tbn2b for conversion from the vehicle fixed coordinate system' 40n (FIG. 5(b)) to the vehicle fixed coordinate system 40t (FIG. 5(c)) can be obtained.

[0090] The above is an overall processing in the automatic calibration device 11a.

[0091] By forming such an automatic calibration device 11a, a new target is unnecessary for detection of the rails 20Lt and 20Rt, and automatic calibration before traveling and during traveling can be performed. In addition, since it is not premised that the vehicle body is reflected as the first sensor information, for example, there is no problem even in a case where it is difficult for the vehicle body to be reflected by using a sensor that monitors a distance at a narrow angle as the first sensor 101a.

[0092] In addition, since the braking distance of the railway is long, it is necessary to detect an object over a long distance in order to stop without colliding with the object found by the front monitoring system. With respect to the object position estimation error caused by the error of the external parameter, in a case where the vehicle traveling direction is set to x, the transverse direction is set to y, and a direction perpendicular to these directions is set to z, the influence of the yaw and the pitch estimation error increases as the distance increases. Since a distant object cannot be correctly detected when these errors become large, it is particularly important to accurately estimate yaw and pitch angles in a railway.

[0093] Since the rails 20Lt and 20Rt are objects extending long in the x-direction and have a large amount of sensor information in the x-axis direction, rail detection accuracy in the x-axis direction is expected to be good. Therefore, in the automatic calibration method utilizing rail detection, it is expected that the estimation accuracy of the external parameter in the x-axis direction is good. The external parameters related to the x-axis direction are a yaw angle and a pitch angle to be estimated with high accuracy in railway front monitoring. By estimating the yaw angle and the pitch angle with high accuracy, the performance of the first sensor 101a for detecting a distant object is improved, and safer traveling is enabled.

[0094] In addition, in the method of estimating the inclination of the related art, the inclination calculation result fluctuates due to a fluctuation factor such as a gradient, cant, vibration during traveling, or vibration of passenger movement even when the vehicle is stationary. Even if the conventional technique is utilized by replacing the inclination estimation with the estimation of the external parameter, it is difficult for the conventional technique to estimate the external parameter with high accuracy in the presence of such a variation factor. In order to estimate the external parameters with high accuracy, measures for increasing the accuracy of the result are required in addition to the external parameter estimation by rail detection. In addition, when the external parameter is defined as a displacement with respect to the design value, it is considered to estimate the external parameter based on a difference between the ideal position of the rail observed by the sensor and the position actually observed, but the ideal position of rail detection is different depending on whether the own vehicle is traveling on a curve or is still traveling on a straight line, and thus it is difficult to calculate the ideal position.

[0095] On the other hand, in the present embodiment, by removing an inappropriate rail point cloud at the time of estimating the external parameter using the calibration feasibility determination unit 103, a variation factor is eliminated, and highly accurate calibration can be performed. In the present embodiment, rail detection is performed on the rails 20Lt and 20Rt in the straight section, so that an ideal position can be easily specified at the time of rail detection. Further, by estimating the external parameter based on the triaxial rotation and translation of the first sensor 101a, a more accurate external parameter can be calculated.

[0096] Therefore, the automatic calibration device 11a in the present embodiment can contribute to cost reduction of the front monitoring system and improvement of safety at the time of traveling.

[0097] In the present embodiment, the method of estimating the external parameter in a case where the calibration feasibility signal is 1 and calibrating the first sensor 101a each time has been described. On the other hand, the external parameter in a case where the calibration feasibility signal is 1 may be accumulated for a certain number of frames, and the representative value of the estimated value of the external parameter between the certain frames may be used as the external parameter used for calibration, thereby improving the estimation accuracy of the external parameter.

[0098] FIG. 6 is a diagram in which the horizontal axis indicates the number of accumulated frames of the external parameter estimation results and the vertical axis indicates the external parameter estimation results. Then, in FIG. 6, the estimation result of the external parameter for each time is indicated by a result 302, and a result 303 of calculating the median value of the accumulated estimation results of the external parameter is schematically illustrated. As a specific example, FIG. 6 illustrates a case where the vertical axis is set to the y-direction translation ty, but the same applies to the cases of the x-direction translation tx and the z-direction translation tz.

[0099] In the above-described example, the calibration feasibility determination unit 103 can exclude a certain amount of curved rails and rail detection results with a large amount of noise. However, although the linearity is high, a phenomenon in which a straight line different from a rail is erroneously detected as a rail occurs at a low frequency. It is difficult to exclude such a result of erroneous detection by linearity determination. At this time, for example, as indicated by 304 in FIG. 6, fluctuation occurs in the calculated external parameter, which leads to deterioration of calibration accuracy.

[0100] In order to prevent a decrease in calibration accuracy due to such an event, by accumulating data for a certain number of frames as in the result 303 and using the representative value as the external parameter used for calibration, a low-frequency event appearing as an outlier can be removed, and a more accurate external parameter can be estimated. As the representative value, for example, there is a method of adopting the median value which is hardly affected by an outlier. In this case, the automatic calibration unit 104 records the triaxial rotation and translation of the first sensor 101a at a plurality of times, and outputs the median value of the triaxial rotation and translation of the first sensor 101a at the plurality of times, and the sensor information correction unit 105 corrects the first sensor information using the output median value.

[0101] Further, there is a method in which the fixed number of frames is set to the value fs when the variation of the representative value becomes small, and the fixed number of frames is determined by trial and error when the automatic calibration device 11a is introduced, or the fixed number of frames is determined based on the number of accumulated frames when the increase or decrease of the representative value converges within a certain range when the number of accumulated frames increases by a predetermined value.

[0102] In the present embodiment, the method of calculating the coordinate transformation between the first sensor coordinate system and the vehicle fixed coordinate system 40t for one first sensor 101a has been described. However, by applying the method of the present embodiment to a plurality of sensors capable of detecting the rails 20Lt and 20Rt and calculating the rotation and translation from each sensor to the vehicle fixed coordinate system, the plurality of sensors can be more accurately integrated into the vehicle fixed coordinate system 40t. In addition, in this method, since the external parameter can be estimated independently for each sensor, there is also an advantage that the external parameter can be estimated even when the visible ranges of the sensors do not overlap. Therefore, according to this method, more accurate automatic calibration can be performed without installing a new target in the system using a plurality of sensors, and thus, it is possible to reduce calibration cost in the system using a front monitoring device and to travel a vehicle more safely.

[0103] In the present embodiment, the calibration method by rail detection has been described, but objects parallel to each other may be detected instead of the rails 20Lt and 20Rt. For example, two parallel upright overhead line columns whose positions with respect to the vehicle 10 are known can be detected by the first sensor 101a and used instead of the first rail point cloud 30an. In this case, the rail detection unit 102 can perform calibration by detecting a first overhead line column point cloud, which is a point cloud corresponding to two parallel upright overhead line columns.[Second embodiment]

[0104] In the first embodiment, the method of estimating the external parameter of the first sensor 101a by comparing the rail detection result of the straight line with the rail straight line in the ideal state in the vehicle fixed coordinate system 40t has been described. On the other hand, in a case where it is not necessary to calculate the attachment position and angle of each sensor with respect to the sensor attachment angle design value, a method of integrating the sensor information in a second vehicle fixed coordinate system' using the first rail detection result detected by the first sensor 101a and the second rail detection result by the second sensor 101b different from the first sensor 101a may be used. Hereinafter, the details will be described as a second embodiment.

[0105] FIG. 7 is a diagram illustrating an overall configuration of a sensor correction system 1b according to the second embodiment.

[0106] The sensor correction system 1b to be illustrated is different from the sensor correction system 1a of the first embodiment illustrated in FIG. 1 in that a second sensor 101b is added to the automatic calibration device 11b, but the other points are similar.

[0107] In addition, in FIG. 7, the first rail point cloud 30an is illustrated as first sensor information obtained by detecting the left and right rails 20Lt and 20Rt by the first sensor 101a. Furthermore, in FIG. 7, a second rail point cloud 30bn is illustrated as second sensor information obtained by detecting the left and right rails 20Lt and 20Rt by the second sensor 101b. It can also be said that FIG. 7 illustrates the first rail point cloud 30an represented by the first vehicle fixed coordinate system' 40an and the second rail point cloud 30bn represented by the second vehicle fixed coordinate system' 40bn.

[0108] Since the external parameter of each of the first sensor 101a and the second sensor 101b has an error with respect to the design value, the first rail point cloud 30an and the second rail point cloud 30bn do not match as illustrated in FIG. 7. In the present embodiment, external parameter for integrating the first sensor information expressed in the first vehicle fixed coordinate system' 40an into the second vehicle fixed coordinate system' 40bn is estimated, and the first sensor information is corrected.

[0109] In the present embodiment, the rail detection unit 102 estimates the first rail point cloud 30an using the information of the first sensor 101a, and further estimates the second rail point cloud 30bn using the information of the second sensor 101b.

[0110] The calibration feasibility determination unit 103 calculates the first linearity based on the first rail point cloud 30an and the second linearity based on the second rail point cloud 30bn according to the method described in the first embodiment. Then, in a case where both the first linearity and the second linearity are equal to or greater than a predetermined threshold value, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 1, and the automatic calibration unit 104 estimates the external parameter. On the other hand, in a case where at least one of the first linearity and the second linearity is less than a predetermined threshold, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 0, and the automatic calibration unit 104 does not perform the estimation of the external parameter.

[0111] In the second embodiment, the reference rail point cloud is the second rail point cloud 30bn that is detected based on the second sensor information acquired by the second sensor 101b and indicates the positions of the rails 20Lt and 20Rt. Then, the calibration feasibility determination unit 103 determines whether calibration is possible based on a difference between the first linearity and the second linearity indicating the degree of linearity with respect to the second rail point cloud 30bn.

[0112] FIGS. 8(a) to 8(e) are diagrams illustrating the operation of the automatic calibration unit 104 in the second embodiment.

[0113] FIG. 8 illustrates a coordinate system used in the present embodiment and a method for estimating the external parameter by the automatic calibration unit 104.

[0114] First, a coordinate system will be described.

[0115] The first sensor coordinate system (FIG. 8(a)) is a coordinate system based on the first sensor 101a. The rail detection unit 102 converts the information expressed in the first sensor coordinate system into the first vehicle fixed coordinate system' (FIG. 8(b)) using the external parameter design value, and performs rail detection. The first vehicle fixed coordinate system' is a coordinate system that transforms and transitions from the first sensor coordinate system to the first vehicle fixed coordinate system' (FIG. 8(b)) using the external parameter design value (rotation Rs12bn1, translation ts12bn1).

[0116] The second sensor coordinate system (FIG. 8(c)) is a coordinate system based on the second sensor 101b. The rail detection unit 102 converts the information expressed in the second sensor coordinate system into the second vehicle fixed coordinate system' (FIG. 8(d)) using the external parameter design value, and performs rail detection. The second vehicle fixed coordinate system' is a coordinate system that transforms and transitions from the second sensor coordinate system to the second vehicle fixed coordinate system' using the external parameter design value (rotation Rs22bn2, translation ts22bn2).

[0117] Next, an aspect of a rail detection result in each coordinate system will be described. 401Ls and 401Rs indicate detection results of the left and right rails 20Lt and 20Rt in the first sensor coordinate system (FIG. 8(a)). A left rail model 401Lbn and a right rail model 401Rbn are rail detection results in the first vehicle fixed coordinate system' (FIG. 8(b)). Further,402Ls and 402Rs indicate detection results of the left and right rails 20Lt and 20Rt in the second sensor coordinate system (FIG. 8(c)). Furthermore, a left rail model 402Lbn and a right rail model 402Rbn are rail detection results in the second vehicle fixed coordinate system' (FIG. 8(d)). Since the first vehicle fixed coordinate system' and the second vehicle fixed coordinate system' are different, 401Lbn and 402Lbn, and 401Rbn and 402Rbn do not match.

[0118] Here, the rotation and translation converting 401Lbn to 402Lbn and 401Rbn to 402Rbn are rotation Rbn12bn2 and translation tbn12bn2 from the first vehicle fixed coordinate system' to the second vehicle fixed coordinate system'. Therefore, in the present embodiment, the rotation Rbn12bn2 and the translation tbn12bn2 for converting a first representative position pbn1 and the first triaxial vector (exbn1, eybn1, ezbn1) of the first rail point cloud 30an in the first vehicle fixed coordinate system' into a second representative position pbn2 and the second triaxial vector (exbn2, eybn2, ezbn2) corresponding to the second rail point cloud 30bn in the second vehicle fixed coordinate system' are calculated.

[0119] As a method of calculating the first representative position pbn1, for example, an arbitrary x-direction position where the first rail point cloud 30an exists is set as an x-coordinate of the first representative position, and the first rail point cloud 30an within a predetermined range is extracted based on the x-coordinate of the first representative position, and the extracted first rail point cloud is set as the first rail point cloud after extraction. The left rail model 401Lbn and the right rail model 401Rbn are estimated for the first rail point cloud after extraction, and a middle line of the left rail model 401Lbn and the right rail model 401Rb is set as the y-coordinate of the first representative position. Furthermore, the z-coordinate corresponding to the x-coordinate of the first representative position and the y-coordinate of the first representative position on the three-dimensional plane defined by the left rail model 401Lbn and the right rail model 401Rb is set as the z-coordinate of the first representative position.

[0120] As a method of calculating the first triaxial vector (exbn1, eybn1, ezbn1), for example, a unit vector in the direction of the left rail model 401Lbn is set to exbn1, a normal vector of a plane defined by the left rail model 401Lbn and the right rail model 401Rb is set to ezbn1, and a unit vector eybn1 perpendicular to exbn1 and ezbn1 can be obtained by an outer product of exbn1 and ezbn1.

[0121] As a method of calculating the second representative position pbn2, the x-coordinate of the first representative position is defined as the x-coordinate of the second representative position on the assumption that the x-direction error between the first vehicle fixed coordinate system and the second vehicle fixed coordinate system is small. Further, the second rail point cloud 30bn within a predetermined range is extracted based on the x-coordinate of the second representative position, and the extracted second rail point cloud is set as the second rail point cloud after extraction. The left rail model 402Lbn and the right rail model 402Rbn are estimated for the second rail point cloud after extraction, and the middle line of the left rail model 402Lbn and the right rail model 402Rb is set as the y-coordinate of the second representative position. Furthermore, the z-coordinate corresponding to the x-coordinate of the second representative position and the y-coordinate of the second representative position on the three-dimensional plane defined by the left rail model 402Lbn and the right rail model 402Rb is set as the z-coordinate of the second representative position.

[0122] As a method of calculating the second triaxial vector (exbn2, eybn2, ezbn2), for example, a unit vector in the direction of the left rail model 402Lbn is set to exbn2, a normal vector of a plane defined by the left rail model 402Lbn and the right rail model 402Rb is set to ezbn2, and a unit vector eybn2 perpendicular to exbn2 and ezbn2 can be obtained by an outer product of exbn2 and ezbn2.

[0123] By solving the equation described in FIG. 8(e) using the first representative position pbn1, the first triaxial vector (exbn1, eybn1, ezbn1), the second representative position pbn2, and the second triaxial vector (exbn2, eybn2, ezbn2) obtained by the above method, the rotation Rbn12bn2 and the translation tbn12bn2 for conversion from the first vehicle fixed coordinate system' to the second vehicle fixed coordinate system can be obtained.

[0124] The method for calculating the external parameter using the first rail point cloud 30an and the second rail point cloud 30bn has been described above. According to the above method, since the external parameter of the first sensor 101a can be calculated by rail detection, highly accurate automatic calibration can be performed without newly installing a target. Therefore, the calibration cost in the system using the front monitoring device can be reduced, and the vehicle can travel more safely.

[0125] In the present embodiment, the calibration based on the detection result of the straight rail is the basis, but in the case of the second embodiment using the first rail point cloud 30an and the second rail point cloud 30bn, the first rail point cloud 30an and the second rail point cloud 30bn may be curved rails. In a case where the actual rails 20Lt and 20Rt are curved, there is a method of calculating the rotation and the translation by, for example, iterative closest point (ICP) matching between the first rail point cloud 30an and the second rail point cloud 30bn.[Third embodiment]

[0126] In the first embodiment and the second embodiment, the example in that the two-dimensional rail detection result is converted into a three-dimensional rail point cloud in the rail detection unit 102, and the external parameter is estimated in the automatic calibration unit 104 in a case where the first sensor 101a is a two-dimensional image. In a case where the two-dimensional rail point cloud detected by the monocular camera is converted into a three-dimensional model, it is difficult to convert the two-dimensional rail point cloud into a three-dimensional model unless an assumption that, for example, there is no constant gradient or no roll rotation around the x-axis is made in order to estimate the depth direction, and it is not easy to acquire an accurate three-dimensional point cloud. In a case where the target performance of the external parameter estimation result cannot be satisfied due to an error in reconstructing a three-dimensional model from the two-dimensional rail detection result by the monocular camera, it is necessary to consider a method of not performing three-dimensional reconstruction.

[0127] The third embodiment has been made in view of the above problems, and the external parameter is estimated on the two-dimensional image for monocular camera information in that three-dimensional reconstruction is difficult to be accurately performed.

[0128] FIG. 9 is a flowchart illustrating the operation of the automatic calibration device 11a in a case where the monocular camera information is used as the first sensor information.

[0129] First, the rail detection unit 102 acquires monocular camera information as first sensor information (S201).

[0130] Next, the rail detection unit 102 performs rail detection on the two-dimensional image (S202).

[0131] Next, the rail detection unit 102 calculates a three-dimensional rail point cloud by using the rail detection result on the two-dimensional image, acquires a two-dimensional rail point cloud by using the rail detection result on the two-dimensional image, and holds the two-dimensional rail point cloud for use in the automatic calibration unit 104 (S203).

[0132] Next, the calibration feasibility determination unit 103 calculates the linearity based on the rail point cloud information after the three-dimensional reconstruction (S204), and calculates the calibration feasibility signal (S205). The operation of the calibration feasibility determination unit 103 is similar to that of the first embodiment.

[0133] Although it has been described that inaccuracy may occur in a case where the two-dimensional rail point cloud is reconstructed into a three-dimensional model, a phenomenon occurs in which the three-dimensional point cloud reconstructed from the two-dimensional point cloud is uniformly different in angle such as a roll angle with respect to the correct three-dimensional point cloud accurately reconstructed into a three-dimensional model by some means, and thus, it can be considered that the relative positional relationship between the points in the point cloud is maintained. Therefore, in a case where the three-dimensional point cloud converted from the two-dimensional point cloud draws a curve, the correct three-dimensional point cloud is also a curve, and in a case where the three-dimensional point cloud converted from the two-dimensional point cloud has a high noise level, the correct three-dimensional point cloud also has a high noise level, and the linearity is considered to have the same value. Therefore, there is no problem even if the threshold determination using the linearity is performed by the method described in the first embodiment.

[0134] Next, the automatic calibration unit 104 determines whether the calibration feasibility signal is 1 (S206).

[0135] As a result, in a case where the calibration feasibility signal is 1 (Yes in S206), the automatic calibration unit 104 retrieves the point cloud 30at corresponding to the actual positions of the rails 20Lt and 20Rt as the second rail point cloud 30bn (S207).

[0136] Then, the automatic calibration unit 104 estimates the external parameter by performing matching between the first rail point cloud 30an and the second rail point cloud 30bn on the two-dimensional image (S208).

[0137] Furthermore, the sensor information correction unit 105 corrects the first sensor information (S209).

[0138] On the other hand, when the calibration feasibility signal is 0 (No in S206), the correction of the first sensor information is not performed, and the series of processing ends.

[0139] The second rail point cloud 30bn may be a three-dimensional rail point cloud in an ideal state as in the first embodiment or may be estimated by the second sensor 101b as in the second embodiment. In the third embodiment, a specific example in a case where the ideal rail point cloud is adopted as the second rail point cloud 30bn in the present embodiment will be described.

[0140] FIGS. 10(a) to 10(c) are diagrams illustrating details of the operation of S208 of FIG. 9 and a coordinate system definition to be used.

[0141] The image coordinate system (FIG. 10(c)) is a two-dimensional orthogonal coordinate system with, for example, the upper left of the image as an origin, and points on the image coordinate system are converted into a vehicle fixed coordinate system' (FIG. 10(b)) slightly different from the vehicle fixed coordinate system by using a camera attachment position, a design value of a posture, a camera matrix, and the like. The estimation target of the external parameter in the present embodiment is rotation Rcbn2b and translation tcbn2b from the vehicle fixed coordinate system' (FIG. 10(b)) to the vehicle fixed coordinate system (FIG. 10(a)).

[0142] In the present embodiment, in order to estimate rotation and translation by matching in the two-dimensional image coordinate system, the automatic calibration unit 104 first sets ideal rail point clouds 601Lb and 601Rb in the vehicle fixed coordinate system as in the first embodiment. Then, the automatic calibration unit 104 calculates rail point clouds 601Lbn and 601Rbn by performing conversion into the vehicle fixed coordinate system' using rotation Rb2cbn and translation tb2cbn that are inverse conversions of the rotation Rcbn2b and the translation tcbn2b. Further, the automatic calibration unit 104 converts the rail point clouds 601Lbn and 601Rbn into the image coordinate system using the external parameter design information of the first sensor 101a, the camera matrix, and the like, and obtains 601Lc and 601Rc. Since the rotation Rb2cbn and the translation tb2cbn are undefined, 601Lc and 601Rc are functions of the rotation Rb2cbn and the translation tb2cbn.

[0143] The automatic calibration unit 104 calculates, by numerical calculation, rotation Rb2cbn and translation tb2cbn that match the two-dimensional rail point clouds 602Lc and 602Rc detected by the 601Lc and 601Rc in the image coordinate system. The numerical calculation method may be a generally used optimization algorithm such as a steepest descent method or an annealing method.

[0144] In the third embodiment, it can also be said that the first rail point cloud 30an indicates two-dimensional positions of the rails 20Lt and 20Rt, and the automatic calibration unit 104 converts the reference rail point cloud (in this case, the second rail point cloud 30bn) into an image coordinate system including the first rail point cloud 30an to acquire a reference rail point cloud represented in two dimensions (the second rail point cloud 30bn), and calculates triaxial rotation and translation for matching the first rail point cloud 30an with the reference rail point cloud represented in two dimensions.

[0145] In the present embodiment, an angle error or the like of the rail point cloud, which occurs when the point cloud on the two-dimensional image is reconstructed into a three-dimensional model, does not occur. Therefore, it is expected that a more accurate estimation result of the external parameter can be obtained, and safer vehicle travel becomes possible.[Fourth embodiment]

[0146] In the first to third embodiments, the linearity of the rail point cloud is calculated to avoid the rail point cloud including the curve section from being used by the automatic calibration unit 104. In addition to the above, the fourth embodiment illustrates a method of grasping in advance a region where a rail point cloud includes a curve section and it is difficult to estimate an external parameter by using a railway digital map describing information associated with an on-track position of railroad, and more reliably estimating the external parameter.

[0147] FIG. 11 is a diagram illustrating an overall configuration of a sensor correction system 1c according to the fourth embodiment.

[0148] The sensor correction system 1c to be illustrated is different from the sensor correction system 1a of the first embodiment illustrated in FIG. 1 in that a railway digital map 106, an on-track position estimation sensor 107, and a presence position estimation unit 108 are added to the automatic calibration device 11c, but the other points are similar.

[0149] The railway digital map 106 is a database including at least information for determining whether or not to estimate an external parameter at a point, such as curvature or presence or absence of a branch at a point corresponding to an on-track position in a section where the vehicle 10 travels.

[0150] The on-track position estimation sensor 107 is a sensor necessary for estimating an on-track position of the vehicle 10, and examples thereof include a receiver of a railway wayside coil, an encoder for measuring the rotation speed of wheels, and a GNSS (Global Navigation Satellite System) receiver.

[0151] The presence position estimation unit 108 has a function of calculating the position of the vehicle on the track in accordance with the information acquired by the on-track position estimation sensor 107.

[0152] FIGS. 12(a) to 12(b) are diagrams illustrating a method of determining whether or not the external parameter can be estimated using the railway digital map 106 and the railway digital map 106.

[0153] FIG. 12(a) is a diagram illustrating a specific example of the railway digital map 106.

[0154] In the railway digital map 106, the curvature of the rail and the presence or absence of a branch are recorded for each on-track position (kilometrage [km]) along a middle line 701 of the left and right rails 20Lt and 20Rt. For example, the position 703a is a straight track having a kilometrage of 1.0 km, a curvature of 0, and no branch (0 means no branch, and 1 means presence of a branch). A hatched section of a region 702a indicates that the curvature and the branch information are the same as those at a position 703a. The position 703b is a point where a branch occurs and a curve is formed, and stores information that the curvature is 0.0016 and there is a branch. Furthermore, a position 703c is a straight track having a kilometrage of 2.5 km, a curvature of 0, and no branch. A hatched section of the region 702b indicates that the curvature and the branch information are the same as those at the position 703c.

[0155] FIG. 12(b) is a diagram illustrating a method of determining whether or not the external parameter can be estimated using the railway digital map 106.

[0156] For example, in a case where the vehicle 10 is present at a position 705a and the section of a region 706 where rail detection is possible is included in the region 702a, it is ensured that all the regions where rail detection is possible are on a straight track. This means that the calibration feasibility determination unit 103 determines that calibration is possible in a case where the first rail point cloud 30an is included in the calibration-capable section based on the map (In this case, the railway digital map 106) including the relationship between the on-track position of the vehicle 10 and the calibration-capable section.

[0157] On the other hand, in a case where a section of the region 706 in which the rail can be detected is not included in the region 702a, and includes a section in which the curvature exceeds the predetermined threshold or includes a section in which a branch exists, for example, as in the position 703b, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 0. As a result, it is possible to prevent the rail point cloud including curves and branches from being used by the automatic calibration unit 104. Therefore, it is expected that a more accurate estimation result of the external parameter can be obtained, and safer vehicle travel becomes possible.

[0158] In addition, in the calibration feasibility determination unit 103, in a case where the linearity falls below a predetermined threshold even though the front of the vehicle 10 is a straight line on which calibration can be performed, such as a straight section and a section not including a branch, the calibration feasibility determination unit 103 may issue an alert indicating that there is a sensor abnormality while setting the calibration feasibility signal to 0. In this case, it can also be said that the calibration feasibility determination unit 103 issues a sensor abnormality signal in a case where the first linearity is a value for which calibration cannot be performed and the first rail point cloud 30an is included in the calibration-capable section.

[0159] By adding a function of issuing an alert to the calibration feasibility determination unit 103, it is possible to detect a sensor abnormality and reflect the sensor abnormality in the operation of the vehicle 10, thereby enabling safer vehicle traveling.[Fifth embodiment]

[0160] If the rotation and translation of each sensor with respect to the vehicle fixed coordinate system can be calculated by the method described in the first embodiment, the sensor attachment abnormality can be detected by comparing the design value of the rotation and translation of each sensor with the current value. The fifth embodiment has been made in view of this, and the sensor attachment abnormality is detected using the design value and the calibration result of the first embodiment.

[0161] FIG. 13 is a diagram illustrating an overall configuration of a sensor correction system 1d according to the fifth embodiment.

[0162] The sensor correction system 1d to be illustrated is different from the sensor correction system 1a of the first embodiment illustrated in FIG. 1 in that a sensor attachment abnormality detection unit 109 is added to the automatic calibration device 11d, but the other points are similar.

[0163] The sensor attachment abnormality detection unit 109 determines that the sensor attachment angle is abnormal in a case where the triaxial rotation and translation obtained by the automatic calibration unit 104 exceed a predetermined threshold value.

[0164] The external parameter of the first sensor 101a output from the automatic calibration unit 104 is input to the sensor attachment abnormality detection unit 109. The sensor attachment abnormality detection unit 109 has a function of calculating a difference between the external parameter of the first sensor 101a output from the automatic calibration unit 104 and an external parameter design value of the first sensor 101a as an abnormality degree on each coordinate axis (biaxial or triaxial for translation and triaxial for rotation), and determining that the sensor attachment abnormality is occurred in a case where the abnormality degree exceeds a predetermined threshold.

[0165] As described in the second embodiment, it is also possible to determine the sensor attachment abnormality by using the first sensor 101a and the second sensor 101b and by using the estimation result of the first-to-second external parameters between the first sensor 101a and the second sensor 101b.

[0166] In this case, a third sensor different from the first sensor 101a and the second sensor 101b is used, and an estimation result of the first-to-fourth external parameters between the first sensor 101a and the third sensor is further acquired by the automatic calibration unit 104. Next, in a case where the element of the result of the estimation of the first-to-second external parameters exceeds the predetermined threshold, and the element of the result of the estimation of the first-to-fourth external parameters exceed the predetermined threshold, it is possible to estimate that the attachment abnormality of the first sensor 101a occurs.

[0167] In this case, it can also be said that the sensor attachment abnormality detection unit 109 determines that the sensor attachment angle is abnormal in a case where both the triaxial rotation and translation obtained from the first rail point cloud 30an and the reference rail point cloud as the second rail point cloud 30bn detected based on the second sensor information acquired by the second sensor 101b and indicating the positions of the rails 20Lt and 20Rt, and the triaxial rotation and translation obtained from the first rail point cloud 30an and a third rail point cloud detected based on third sensor information acquired by a third sensor and indicating the positions of the rails 20Lt and 20Rt exceed a predetermined threshold.

[0168] According to the fifth embodiment, it is possible to detect the attachment angle abnormality of each sensor in a section where the rail can be detected with a predetermined linearity or more. Therefore, it is possible to prevent a failure in detection of an object ahead due to an attachment angle abnormality and to travel more safely.

[0169] In the embodiments described in detail above, the case where the automatic calibration is performed and the automatic calibration is performed has been described, but the present invention is not limited thereto, and the calibration may be manually performed.

[0170] In the embodiments described in detail above, the linearity is an index for evaluating whether or not the rail point cloud is a straight line and the degree of variation in data of the rail point cloud. However, the degree of variation in data of the rail point cloud does not necessarily need to be taken into consideration as long as sufficient accuracy for determining a straight rail can be ensured only by determining whether or not the rail point cloud is a straight line.<Description of correction method>

[0171] The processing performed by the automatic calibration devices 11a to 11d described above is realized by cooperation of software and hardware resources. That is, a processor in a computer provided in each of the automatic calibration devices 11a to 11d loads software for realizing each function described above into a memory and executes the software to realize each function.

[0172] Therefore, the processing performed by the automatic calibration devices 11a to 11d can be considered as a correction method in which the processor executes the software recorded in the memory to detect the first rail point cloud 30an indicating the positions of the rails 20Lt and 20Rt based on the first sensor information acquired by the first sensor 101a, determine whether calibration of the first sensor 101a is possible based on the first linearity indicating the degree of the straight line with respect to the first rail point cloud 30an, and correct the first sensor information based on the first rail point cloud 30an and the reference rail point cloud indicating the three-dimensional positions of the rails 20Lt and 20Rt and different from the first rail point cloud 30an in a case where it is determined that calibration is possible.

[0173] Although the present embodiment has been described above, the technical scope of the present invention is not limited to the scope described in the above embodiment. It is apparent from the description of the claims that various modifications or improvements are added to the above embodiments within the technical scope of the present invention.Reference Signs List

[0174] 1a to 1dsensor correction system 11a to 11cautomatic calibration device 20Lt, 20Rtrail 30anfirst rail point cloud 30atpoint cloud 101afirst sensor 101bsecond sensor 102rail detection unit 103calibration feasibility determination unit 104automatic calibration unit 105sensor information correction unit 106railway digital map 107on-track position estimation sensor 108on-track position estimation unit 109sensor attachment abnormality detection unit

Examples

first embodiment

[First embodiment]

[0028]FIG. 1 is a diagram illustrating an overall configuration of a sensor correction system 1a according to a first embodiment.

[0029]As illustrated, the sensor correction system 1a is mounted on a vehicle 10. The vehicle 10 is a vehicle housing that travels on an iron track, and examples thereof include a railway and a tram. The vehicle 10 travels on the pair of rails 20Lt and 20Rt. Here, the rail on the left side in the traveling direction of the vehicle 10 is a rail 20Lt, and the rail on the right side in the traveling direction of the vehicle 10 is a 20Rt.

[0030]The sensor correction system 1a includes a first sensor 101a and an automatic calibration device 11a.

[0031]The automatic calibration device 11a is an example of a correction device, and includes a rail detection unit 102, a calibration feasibility determination unit 103, an automatic calibration unit 104, and a sensor information correction unit 105. Information acquired by the first sensor 101a is used...

second embodiment

[Second embodiment]

[0104]In the first embodiment, the method of estimating the external parameter of the first sensor 101a by comparing the rail detection result of the straight line with the rail straight line in the ideal state in the vehicle fixed coordinate system 40t has been described. On the other hand, in a case where it is not necessary to calculate the attachment position and angle of each sensor with respect to the sensor attachment angle design value, a method of integrating the sensor information in a second vehicle fixed coordinate system' using the first rail detection result detected by the first sensor 101a and the second rail detection result by the second sensor 101b different from the first sensor 101a may be used. Hereinafter, the details will be described as a second embodiment.

[0105]FIG. 7 is a diagram illustrating an overall configuration of a sensor correction system 1b according to the second embodiment.

[0106]The sensor correction system 1b to be illustrate...

third embodiment

[Third embodiment]

[0126]In the first embodiment and the second embodiment, the example in that the two-dimensional rail detection result is converted into a three-dimensional rail point cloud in the rail detection unit 102, and the external parameter is estimated in the automatic calibration unit 104 in a case where the first sensor 101a is a two-dimensional image. In a case where the two-dimensional rail point cloud detected by the monocular camera is converted into a three-dimensional model, it is difficult to convert the two-dimensional rail point cloud into a three-dimensional model unless an assumption that, for example, there is no constant gradient or no roll rotation around the x-axis is made in order to estimate the depth direction, and it is not easy to acquire an accurate three-dimensional point cloud. In a case where the target performance of the external parameter estimation result cannot be satisfied due to an error in reconstructing a three-dimensional model from the ...

Claims

1. A correction device comprising: a rail detection unit which detects a first rail point cloud indicating a position of a rail based on first sensor information acquired by a first sensor; a calibration feasibility determination unit which determines feasibility of calibration of the first sensor based on a first linearity indicating a degree of a straight line with respect to the first rail point cloud; and a sensor information correction unit which corrects the first sensor information based on the first rail point cloud and a reference rail point cloud that indicates a three-dimensional position of a rail and is different from the first rail point cloud in a case where the calibration feasibility determination unit determines that calibration is possible.

2. The correction device according to claim 1, wherein the first linearity indicates a degree of variation with respect to the first rail point cloud in addition to the degree of the straight line.

3. The correction device according to claim 2, wherein the calibration feasibility determination unit obtains the first linearity based on an average value of distances between a rail straight line model estimated by performing straight line extraction from the first rail point cloud and points included in the first rail point cloud.

4. The correction device according to claim 3, wherein the calibration feasibility determination unit obtains a first distance average value and a second distance average value as the average value for each of the pair of rails, and sets a reciprocal of an average value of the first distance average value and the second distance average value as the first linearity.

5. The correction device according to claim 1, wherein the calibration feasibility determination unit determines presence or absence of a branch from the first rail point cloud, and determines that calibration is not possible in a case where there is a branch.

6. The correction device according to claim 5, wherein the calibration feasibility determination unit determines presence or absence of a branch by comparing a distance in a width direction of a vehicle with a gauge for the first rail point cloud.

7. The correction device according to claim 1, wherein the reference rail point cloud is a second rail point cloud that is detected based on second sensor information acquired by a second sensor and indicates a position of a rail.

8. The correction device according to claim 7, wherein the calibration feasibility determination unit determines feasibility of calibration based on a difference between the first linearity and a second linearity indicating a degree of a straight line with respect to the second rail point cloud.

9. The correction device according to claim 1, further comprising a calibration unit which obtains triaxial rotation and translation of the first sensor by using the first rail point cloud and the reference rail point cloud in a case where the calibration feasibility determination unit determines that calibration is possible, wherein the sensor information correction unit corrects the first sensor information using the triaxial rotation and the translation obtained by the calibration unit.

10. The correction device according to claim 9, wherein the calibration unit records a plurality of times of the triaxial rotation and the translation of the first sensor and outputs a median value of the triaxial rotation and the translation of the first sensor at a plurality of times, and the sensor information correction unit corrects the first sensor information using the output median value.

11. The correction device according to claim 9, wherein the first rail point cloud indicates a two-dimensional position of a rail, and the calibration unit converts the reference rail point cloud into an image coordinate system including the first rail point cloud, acquires the reference rail point cloud represented in two dimensions, and calculates the triaxial rotation and the translation for matching the first rail point cloud with the reference rail point cloud represented in two dimensions.

12. The correction device according to claim 9, further comprising a sensor attachment abnormality detection unit which determines that the sensor attachment angle is abnormal in a case where the triaxial rotation and the translation obtained by the calibration unit exceed a predetermined threshold.

13. The correction device according to claim 12, wherein the sensor attachment abnormality detection unit determines that the sensor attachment angle is abnormal in a case where both the triaxial rotation and the translation obtained from the first rail point cloud and a reference rail point cloud as a second rail point cloud detected based on second sensor information acquired by a second sensor and indicating the position of the rail, and the triaxial rotation and the translation obtained from the first rail point cloud and a third rail point cloud detected based on third sensor information acquired by a third sensor and indicating a position of a rail exceed a predetermined threshold.

14. The correction device according to claim 1, wherein the calibration feasibility determination unit determines that calibration is possible in a case where the first rail point cloud is included in a calibration-capable section based on a map including a relationship between an on-track position of a vehicle and a calibration-capable section.

15. The correction device according to claim 14, wherein the calibration feasibility determination unit issues a sensor abnormality signal in a case where the first linearity is a value indicating that calibration is not possible and the first rail point cloud is included in a calibration-capable section.

16. The correction device according to claim 1, wherein the rail detection unit detects a first overhead line column point cloud instead of the first rail point cloud.

17. A correction method comprising, by a processor executing software recorded in the memory, detecting a first rail point cloud indicating a position of a rail based on first sensor information acquired by a first sensor; determining feasibility of calibration of the first sensor based on a first linearity indicating a degree of a straight line with respect to the first rail point cloud; and correcting the first sensor information based on the first rail point cloud and a reference rail point cloud that indicates a three-dimensional position of a rail and is different from the first rail point cloud in a case where it is determined that the calibration is possible.

Images