Correction device and correction method

Abstract

To provide a correction device and a correction method capable of performing more accurate calibration even without providing a calibration facility.SOLUTION: This correction device comprises: a rail detection unit 102 that detects a first rail point group 30an representing the positions of rails 20Lt and 20Rt on the basis of first sensor information acquired by a first sensor 101a; a calibration permission determination unit 103 that determines calibration permission of the first sensor 101a on the basis of first straightness representing the degree of straightness with respect to the first rail point group 30an; and a sensor information correction unit 105 that corrects the first sensor information on the basis of the first rail point group 30an and a point group 30at which is a reference rail point group representing the three-dimensional positions of rails and being different from the first rail point group, when the calibration permission determination unit 103 determines that calibration is permitted.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a correction device and a correction method, and more particularly to a correction device and a correction method suitable for correcting sensor information output from a sensor mounted on a train. [Background technology]

[0002] For example, a train forward monitoring system is expected to expand the forward monitoring area and improve the object detection rate by equipping multiple sensors with different monitoring areas and frequency characteristics. In such a system, multiple sensor information is integrated by using the external parameters (mounting position, angle information) of each sensor. Since the external parameters of each sensor vary depending on the vehicle and change over time due to vehicle body vibration, etc., the actual external parameters have an error with respect to the design value. If this error increases, the position error of the object contained in each sensor increases, making it difficult to integrate the sensor information. Therefore, in order to prevent failure due to sensor external parameter errors when integrating multiple sensor information, it is necessary to calculate and correct the external parameters such as the position and attitude of each sensor. In other words, a correction device or the like is required to perform calibration and correct the sensor information.

[0003] Patent Document 1 describes a camera calibration device that includes a measurement area having at least two reference lines arranged within the imaging range of a camera, including lines intersecting with two points corresponding to the shape of the vehicle, and a calibration pattern having a predetermined pattern shape between the reference lines, and an image processing unit. This device includes an imaging process in which the image processing unit generates a measurement image by imaging the measurement area in which the calibration pattern and the reference lines are arranged with the camera, a stopping error calculation process that calculates the coordinate relationship between the stopping position of the vehicle and the pattern shape as a stopping error based on the two reference lines, and an external parameter calculation process that calculates an external parameter corresponding to the mounting posture of the camera based on the pattern shape with the stopping error canceled.

[0004] Patent Document 2 describes a vehicle body inclination estimation device that is provided with a rail detection unit that determines the position of the rail from the images taken by the cameras, a three-dimensional shape calculation unit that calculates the three-dimensional shape of the rail based on the rail positions determined from the images taken by the cameras, and an inclination estimation unit that uses M-estimation or RANSAC to suppress the effects of noise when aligning the three-dimensional shape of the rail with a previously prepared rail shape model using an ICP algorithm. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2013-2820 A [Patent Document 2] JP 2019-23017 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, installing a measurement area is expected to require maintenance to prevent deterioration such as dirt and damage, in addition to installation costs, so it is preferable not to install such a measurement area. An object of the present invention is to provide a correction device and a correction method that are capable of performing more accurate calibration without providing any calibration equipment. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention provides a correction device having a rail detection unit that detects a first rail point cloud representing a rail position based on first sensor information acquired by a first sensor, a calibration feasibility determination unit that determines whether or not the first sensor can be calibrated based on a first straightness that represents the degree of straightness with respect to the first rail point cloud, and a sensor information correction unit that corrects the first sensor information based on the first rail point cloud and a reference rail point cloud that represents the three-dimensional position of the rail and is different from the first rail point cloud when the calibration feasibility determination unit determines that calibration is possible. In this case, it is possible to provide a correction device that can perform more accurate calibration without providing any equipment for calibration.

[0008] Here, the first straightness can represent the degree of variation with respect to the first rail point cloud in addition to the degree of straightness, which can improve the accuracy of the evaluation of the straightness. The calibration feasibility determination unit may determine the first straightness based on an average value of distances between a rail straightness model estimated by extracting straight lines from the first rail point cloud and points included in the first rail point cloud. In this case, the straightness can be determined more easily. Furthermore, the calibration feasibility determination unit can obtain a first distance average value and a second distance average value as average values ​​for each of the pair of rails, and determine the first straightness as the reciprocal of the average value of the first distance average value and the second distance average value. In this case, a more intuitive value can be obtained as the straightness.

[0009] Furthermore, the calibration feasibility determination unit can determine the presence or absence of a branch from the first rail point cloud, and determine that calibration is impossible if a branch is present. In this case, the estimation of the external parameters becomes more accurate. The calibration feasibility determination unit can then determine the presence or absence of a branch by comparing the distance in the vehicle width direction and the gauge of the rail for the first rail point cloud. In this case, the branch determination can be made more easily.

[0010] The reference rail point cloud may be a second rail point cloud that is detected based on second sensor information acquired by a second sensor and represents the position of the rail. In this case, the sensor information of the first sensor and the second sensor may be integrated. Furthermore, the calibration feasibility determination unit can determine whether the calibration can be performed based on the difference between the first straightness and the second straightness representing the degree of straightness with respect to the second rail point cloud. In this case, the determination of whether the calibration can be performed can be made more accurately.

[0011] The present invention further includes a calibration unit that, when the calibration feasibility determination unit determines that calibration is possible, determines the three-axis rotation and translation of the first sensor using the first rail point cloud and the reference rail point cloud, and the sensor information correction unit can correct the first sensor information using the three-axis rotation and translation determined by the calibration unit. In this case, the accuracy of the correction of the first sensor information is improved. Furthermore, the calibration unit records the three-axis rotation and translation of the first sensor at multiple times and outputs a median value of the three-axis rotation and translation of the first sensor at the multiple times, and the sensor information correction unit corrects the first sensor information using the output median value. In this case, more accurate estimation of the external parameters is possible. Furthermore, the first rail point cloud represents a two-dimensional position of the rail, and the calibration unit converts the reference rail point cloud into an image coordinate system including the first rail point cloud to obtain a two-dimensional reference rail point cloud, and calculates three-axis rotation and translation to match the first rail point cloud with the two-dimensional reference rail point cloud. In this case, angle errors in the rail point cloud that occur when converting a point cloud on a two-dimensional image into a three-dimensional image do not occur, making it easier to obtain more accurate estimation results for the external parameters. Furthermore, the present invention may further include a sensor mounting abnormality detection unit that determines that the sensor mounting angle is abnormal when the three-axis rotation and translation obtained in the calibration unit exceed a predetermined threshold value. In this case, the sensor mounting abnormality can be detected. The sensor installation abnormality detection unit can determine that a sensor installation angle is abnormal when the three-axis rotation and translation calculated from the first rail point cloud and the reference rail point cloud as the second rail point cloud detected based on the second sensor information acquired by the second sensor and representing the position of the rail, and the three-axis rotation and translation calculated from the first rail point cloud and the third rail point cloud detected based on the third sensor information acquired by the third sensor and representing the position of the rail, all exceed predetermined thresholds. In this case, the first sensor and the second sensor can be used to determine the sensor installation abnormality.

[0012] Furthermore, the calibration feasibility determination unit can determine that calibration is possible when the first rail point cloud is included in a calibration possible section based on a map including the relationship between the vehicle's location on the track and the calibration possible section. In this case, it is possible to grasp in advance areas including curve sections where it is difficult to estimate the external parameters, and to more reliably estimate the external parameters. The calibration feasibility determination unit can be configured to issue a sensor abnormality signal when the first straightness is a value that does not allow calibration and the first rail point cloud is included in a calibration possible section. In this case, the map can be used to detect sensor installation abnormality. Furthermore, the rail detection unit may detect a point cloud representing the positions of parallel objects instead of the first rail point cloud. For example, the rail detection unit detects a first pole point cloud instead of the first rail point cloud. In this case, the point cloud required for calibration can be obtained by using parallel objects such as poles.

[0013] The present invention also provides a correction method in which a processor executes software recorded in a memory to detect a first rail point cloud representing the position of a rail based on first sensor information acquired by a first sensor, judge whether calibration of the first sensor is possible based on a first straightness representing the degree of straightness with respect to the first rail point cloud, and if it is determined that calibration is possible, correct the first sensor information based on the first rail point cloud and a reference rail point cloud that represents the three-dimensional position of the rail and is different from the first rail point cloud. In this case, a correction method that allows more accurate calibration to be performed without providing any calibration equipment can be provided. Effect of the Invention

[0014] According to the present invention, it is possible to provide a correction device and a correction method that can perform more accurate calibration without providing any calibration equipment. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram showing an overall configuration of a sensor correction system according to a first embodiment. [Diagram 2] FIG. 4 is a flowchart illustrating an operation from acquisition of first sensor information to correction of the information of the first sensor in the first embodiment. [Diagram 3] 1A to 1B are diagrams showing a method for calculating straightness, and FIG. 1C to 1E are diagrams showing specific examples of calculating straightness. [Figure 4] 13(a) and 13(b) are diagrams showing a specific example of a method for making a branch judgment when a branch judgment function is added to a calibration feasibility judgment section. [Diagram 5] 5(a) to 5(d) are diagrams showing a coordinate system used in the first embodiment and a method of estimating external parameters using a first rail point cloud. [Figure 6] 13 is a graph showing the number of accumulated frames of the estimation results of the external parameters on the horizontal axis and the estimation results of the external parameters on the vertical axis. [Figure 7] FIG. 13 is a diagram showing an overall configuration of a sensor correction system according to a second embodiment. [Figure 8] 13(a) to 13(e) are diagrams illustrating the operation of an automatic calibration unit in the second embodiment. [Figure 9] FIG. 11 is a flowchart showing the operation of the automatic calibration device when monocular camera information is used as the first sensor information. [Figure 10] 10(a) to 10(c) are diagrams showing details of the operation of S208 in FIG. 9 and the definition of the coordinate system to be used. [Figure 11] FIG. 13 is a diagram showing an overall configuration of a sensor correction system according to a fourth embodiment. [Figure 12] 1A and 1B are diagrams illustrating a digital railway map and a method for determining whether or not external parameters can be estimated using the digital railway map. [Figure 13] FIG. 13 is a diagram showing an overall configuration of a sensor correction system according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. The present invention will be described with reference to first to fifth embodiments. [First embodiment] <Explanation of Overall Configuration of Sensor Correction System 1a> FIG. 1 is a diagram showing the overall configuration of a sensor correction system 1a according to the first embodiment. As shown in the figure, the sensor correction system 1a is mounted on a vehicle 10. The vehicle 10 is a vehicle housing that runs on a railroad track, such as a train or a tram. The vehicle 10 runs on a pair of rails 20Lt and 20Rt. Note that here, the rail on the left side of the traveling direction of the vehicle 10 is referred to as the rail 20Lt, and the rail on the right side is referred to as the rail 20Rt.

[0017] The sensor correction system 1a also includes a first sensor 101a and an automatic calibration device 11a. The automatic calibration device 11a is an example of a correction device, and includes a rail detection unit 102, a calibration feasibility determination unit 103, an automatic calibration unit 104, and a sensor information correction unit 105, and receives information acquired by a first sensor 101a as an input. Note that, in this embodiment, the automatic calibration unit 104 is mounted on the vehicle 10, but if the automatic calibration unit 104 has a means for transmitting information from the sensor 101a to the ground, it may be mounted on equipment installed on the ground.

[0018] In the automatic calibration device 11a, the rail detection unit 102 uses data acquired by the first sensor 101a to estimate the positions of the left and right rails 20Lt, 20Rt in front of the vehicle 10 and acquires them as a first rail point cloud 30an. Furthermore, the calibration feasibility determination unit 103 determines whether calibration is possible based on the straightness of the first rail point cloud 30an. Then, when it is determined that calibration is possible, the automatic calibration unit 104 estimates external parameters. Furthermore, the sensor information correction unit 105 corrects the data acquired by the first sensor 101a using the estimation results of the external parameters.

[0019] The first sensor 101a is a sensor that outputs first sensor information that is the basis for estimating the first rail point cloud 30an by the rail detection unit 102. Examples of the first sensor 101a include a laser radar, a stereo camera, and a monocular camera. The first sensor information is a three-dimensional point cloud when the first sensor 101a is a laser radar. Also, when the first sensor 101a is a stereo camera, the first sensor information is a parallax image and left and right camera images. Furthermore, when the first sensor 101a is a monocular camera, the first sensor information is a two-dimensional image.

[0020] The rail detection unit 102 has a function of estimating a three-dimensional point cloud using the first sensor information. It can also be said that the rail detection unit 102 detects a first rail point cloud 30an representing the positions of the rails 20Lt and 20Rt based on the first sensor information acquired by the first sensor 101a. When the first sensor information is a three-dimensional point cloud output by a laser radar, the rail detection unit 102 extracts a point cloud corresponding to the left and right rails 20Lt, 20Rt from the three-dimensional point cloud, and sets it as the first rail point cloud 30an. When the first sensor information is a parallax image of a stereo camera and left and right camera images, the rail detection unit 102 estimates the first rail point cloud 30an using two-dimensional rail extraction using pattern matching on the image, etc., and a depth estimation result based on the parallax image, etc. When the input image is a monocular camera image, the rail detection unit 102 performs two-dimensional rail extraction using, for example, pattern matching on the image, and estimates the first rail point cloud 30an using a depth estimation result based on a comparison between the actual width (= gauge) of the left and right rails 20Lt, 20Rt and the observed gauge.

[0021] The first rail point cloud 30an is a point cloud corresponding to the three-dimensional positions of the left and right rails 20Lt, 20Rt, and may be, for example, a point cloud corresponding to the inside of the top surface of the left and right rails 20Lt, 20Rt. In this embodiment, the first rail point cloud 30an is expressed in a vehicle-fixed coordinate system '40n by performing coordinate conversion using a rotation design value and a translation design value for converting from the first sensor coordinate system to a vehicle-fixed coordinate system 40t fixed to the vehicle 10.

[0022] The vehicle-fixed coordinate system 40t is a coordinate system whose origin and orientation are fixed with respect to the vehicle 10. In this embodiment, the x direction of the vehicle-fixed coordinate system 40t is the straight-line direction along the rails 20Lt, 20Rt, the y direction is the direction along the sleepers, and the z direction is the direction perpendicular to x and y, and the x origin is defined as the front of the car body, the y origin is on the midline of the left and right rails 20Lt, 20Rt, and the z origin is on the plane spanned by the upper surfaces of the left and right rails 20Lt, 20Rt, but other methods may be used.

[0023] The vehicle-fixed coordinate system '40n is a coordinate system that transitions as a result of coordinate conversion of the sensor information of the first sensor coordinate system using the rotation design value and translation design value. Since the rotation design value and translation design value differ from the actual rotation and translation, the origin and axis direction of the vehicle-fixed coordinate system '40n differ from those of the actual vehicle-fixed coordinate system 40t. Therefore, the first rail point cloud 30an exists in a position different from the point cloud 30at corresponding to the actual positions of the rails 20Lt, 20Rt in the vehicle-fixed coordinate system 40t.

[0024] The calibration feasibility determination unit 103 evaluates the first rail point group 30an based on the straightness. The calibration feasibility determination unit 103 sets a threshold value for the straightness and determines whether calibration can be performed based on this threshold value. For example, the calibration feasibility determination unit 103 determines that calibration is possible when the straightness is equal to or greater than a predetermined threshold value, and determines that calibration is not possible when the straightness is less than the predetermined threshold value. Therefore, it can be said that the calibration feasibility determination unit 103 determines whether calibration of the first sensor 101a can be performed based on the straightness that represents the degree of straightness with respect to the first rail point group 30an. Note that this straightness can also be said to be a first straightness in the sense of being distinguished from a second straightness described later. The calibration feasibility determination unit 103 has a function of outputting, to the automatic calibration unit 104, a calibration feasibility signal for determining whether or not calibration can be performed, and the first rail point group 30an.

[0025] The straightness is an index for evaluating whether the estimated first rail point group 30an is straight or not, and the degree of data variation of the first rail point group 30an, and will be described in detail later. The straightness threshold is a value that is adjusted in advance so that the estimation accuracy of the external parameters in the automatic calibration unit 104 can satisfy the requirements.

[0026] The calibration feasibility signal is information for determining whether the automatic calibration unit 104 may execute the estimation process of the external parameters using the first rail point cloud 30an. The calibration feasibility signal is, for example, a signal of 0 or 1, where 0 means that calibration is not possible and 1 means that calibration is possible.

[0027] When the calibration feasibility signal determines that calibration is possible, the automatic calibration unit 104 has a function of estimating the external parameters of the first sensor 101a using the first rail point cloud 30an and the point cloud 30at corresponding to the actual positions of the rails 20Lt, 20Rt. Therefore, when the calibration feasibility determination unit 103 determines that calibration is possible, the automatic calibration unit 104 functions as a calibration unit that determines the three-axis rotation and translation of the first sensor 101a using the first rail point cloud 30an and the reference rail point cloud (in the first embodiment, the point cloud 30at corresponding to the actual positions of the rails 20Lt, 20Rt). Details of the method of estimating the external parameters using the first rail point cloud 30an will be described later.

[0028] The sensor information correction unit 105 has a function of converting data of the first sensor 101a into the vehicle-fixed coordinate system 40t using the estimation result of the external parameters of the first sensor 101a. When the calibration feasibility determination unit 103 determines that calibration is possible, the sensor information correction unit 105 can also be said to correct the first sensor information based on the first rail point cloud 30an and a reference rail point cloud (in the first embodiment, a point cloud 30at corresponding to the actual positions of the rails 20Lt, 20Rt) that represents the three-dimensional positions of the rails 20Lt, 20Rt and is different from the first rail point cloud an. In this case, the sensor information correction unit 105 corrects the first sensor information using the three-axis rotation and translation obtained by the automatic calibration unit 104.

[0029] FIG. 2 is a flow chart illustrating an operation from acquisition of first sensor information to correction of information from the first sensor 101a in the first embodiment. First, the rail detection unit 102 acquires first sensor information at a certain time (S101). Next, the rail detection unit 102 determines whether the data dimension of the first sensor information is three-dimensional (S102). As a result, if the data dimension is three-dimensional (Yes in S102), the rail detection unit 102 detects the rails 20Lt and 20Rt from the three-dimensional point cloud (S103). The data dimension is three-dimensional, for example, in the case of a point cloud acquired by a laser radar. On the other hand, if the data dimension is two-dimensional (No in S102), the rail detection unit 102 first detects the rails 20Lt, 20Rt on the two-dimensional image (S104), and converts the rail detection result into three-dimensional data using a parallax image or a gauge (S105), and outputs the result to the calibration feasibility determination unit 103. The data dimension is two-dimensional, for example, in the case of a stereo camera image or a monocular camera image.

[0030] Next, the calibration feasibility determination unit 103 calculates the straightness based on the first rail point cloud 30an (S106). Then, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 1 if the linearity is equal to or greater than the predetermined threshold, and sets the calibration feasibility signal to 0 if the linearity is less than the predetermined threshold (S107).

[0031] Next, the automatic calibration unit 104 determines whether the calibration enable / disable signal is 1 or not (S108). As a result, when the calibration enable / disable signal is 1 (Yes in S108), the automatic calibration unit 104 captures the point cloud 30at corresponding to the actual positions of the rails 20Lt, 20Rt as a second rail point cloud (reference rail point cloud) (S109). Then, the automatic calibration unit 104 estimates the external parameters of the first sensor 101a by calculating the rotation and translation between the first rail point cloud 30an and the second rail point cloud (S110). Furthermore, the sensor information correcting unit 105 corrects the first sensor information (S111). On the other hand, if the calibration availability signal is 0 (No in S108), the first sensor information is not corrected and the series of processes ends.

[0032] <Description of Straightness> Next, the straightness will be described in detail. Figures 3(a)-(b) are diagrams showing a method for calculating the straightness, and Figures 3(c)-(e) are diagrams showing specific examples of calculating the straightness. First, as shown in FIG. 3(a), the calibration feasibility determination unit 103 estimates a left rail straight model 201L along a point cloud corresponding to the left rail 20Lt for the first rail point cloud 30an obtained by the rail detection unit 102. The calibration feasibility determination unit 103 also estimates a right rail straight model 201R along a point cloud corresponding to the right rail 20Rt. As a method for estimating these straight models, for example, a straight line estimation by RANSAC (RANdom SAmple Consensus) is performed on the first rail point cloud 30an to estimate one of the left and right straight models as the first straight line model, and then a point cloud within a predetermined distance range from the first straight line model is removed and straight line estimation by RANSAC is performed again to estimate the other straight line model as the second straight line model. Furthermore, of the first straight line model and the second straight line model, the one with the larger y-intercept can be the left rail straight line model 201L, and the one with the smaller y-intercept can be the right rail straight line model 201R.

[0033] Next, the calibration feasibility determination unit 103 extracts a point cloud 204L that exists in a range 203L that is within a predetermined distance 202 from the left rail straight model 201L, and a point cloud 204R that exists in a range 203R that is within a predetermined distance 202 from the right rail straight model 201R. The distance 202 is set to a value that allows the curved rail point cloud to be included in the ranges 203L and 203R.

[0034] Furthermore, as shown in FIG. 3(b), the calibration feasibility determination unit 103 calculates the distance 205Li between the point 204Li included in the point cloud 204L and the left rail straight model 201L over all points of the point cloud 204L, and calculates a first distance average value by taking the average. The calibration feasibility determination unit 103 also calculates the distance 205Ri between the point 204Ri of the point cloud 204R and the right rail straight model 201R over all points of the point cloud 204R, and calculates a second distance average value by taking the average. Next, the calibration feasibility determination unit 103 calculates a third distance average value by taking the average of the first distance average value and the second distance average value. Then, the calibration feasibility determination unit 103 defines the reciprocal of the third distance average value as the straightness. In this case, the larger the straightness value, the higher the degree of straightness of the rails 20Lt and 20Rt.

[0035] In this way, the calibration feasibility determination unit 103 determines straightness (first straightness) based on the average value of the distances between the rail straight line models 201L, 201R estimated by extracting straight lines from the first rail point cloud 30an and the points included in the first rail point cloud 30an. In addition, the calibration feasibility determination unit 103 determines a first distance average value and a second distance average value as average values ​​for each of the pair of rails 20Lt, 20Rt, and determines the reciprocal of the average value of the first distance average value and the second distance average value as the straightness (first straightness).

[0036] Next, a difference in linearity according to the shape of the first rail point group 30an will be described. 3(c) shows a specific example of a case where the first rail point cloud 30an with little variation can be obtained for the straight rails 20Lt and 20Rt. In such a case, the third distance average value becomes small, and the straightness, which is the reciprocal of the third distance average value, becomes a large value.

[0037] On the other hand, FIG. 3(d) is a specific example in which the first rail point cloud 30an with large variation is acquired for the straight rails 20Lt, 20Rt. If such rail detection results are used in the automatic calibration unit 104, the external parameters are estimated based on an inaccurate straight model, which may lead to an increase in the estimation error of the external parameters, so it should be determined that calibration is not possible. In the case of FIG. 3(d), for example, since there is a point cloud 206 that is far from the left rail straight model 201L, the third distance average value is large, and the straightness, which is the reciprocal of the third distance average value, is a smaller value than that of FIG. 3(c).

[0038] Also, Fig. 3(e) is a specific example of a case where the first rail point cloud 30an is acquired for the curved rails 20Lt, 20Rt. In this embodiment, the first rail point cloud 30an input to the automatic calibration unit 104 is assumed to be a straight rail, and it is necessary to prevent the point cloud corresponding to the curved rails 20Lt, 20Rt from being input to the automatic calibration unit 104. In the case of Fig. 3(e), for example, because the rails 20Lt, 20Rt are curved, there exists a point cloud 208 that is far from the left rail straight model 201L, so the third distance average value is large, and the straightness, which is the reciprocal of the third distance average value, is a larger value than that of Fig. 3(c).

[0039] 3(d) and (e), the calibration feasibility determination unit 103 sets a threshold value for setting the calibration feasibility signal to 0. This prevents calibration from being performed when the first rail point group 30an varies greatly or when the rail is curved, making it possible to calculate the external parameters more accurately. The above is an explanation of the definition and properties of the straightness calculated by the calibration feasibility determination unit 103.

[0040] In the embodiment described above, the calibration feasibility determination unit 103 basically determines the value of the calibration feasibility signal based only on the straightness, but when the rails 20Lt, 20Rt branch off, the calibration feasibility signal may be set to 0. Specifically, when the rail detection unit 102 cannot stably extract a straight rail due to the branching of the rails 20Lt, 20Rt, the calibration feasibility determination unit 103 may further be provided with a function of determining that the rails 20Lt, 20Rt branch off and setting the calibration feasibility signal to 0.

[0041] 4(a) and 4(b) are diagrams showing a specific example of a method for performing branch judgment when a branch judgment function is added to the calibration feasibility judgment unit 103. In FIG. As shown in Fig. 4(a), the calibration feasibility determination unit 103 calculates a distance 212 (Δy) between a point 211L whose y is maximum and a point 211R whose y is minimum among a group of points existing in a section equally divided by a width 210 in the x direction of the vehicle-fixed coordinate system 209. Then, as shown in Fig. 4(b), the calibration feasibility determination unit 103 creates a diagram in which the distance 212 (Δy) is plotted for each x direction section. The x coordinate in each x direction section is, for example, the midpoint between the maximum value and the minimum value of each x direction section.

[0042] FIG. 4(b) is a diagram in which the distance 212 (Δy) is plotted for each x-direction section. In FIG. 4(b), if the rails 20Lt and 20Rt are straight, the distance 212 roughly coincides with the gauge D of the rails 20Lt and 20Rt. However, if a branch occurs, the distance between the point 211L and the point 211R increases, and the distance 212 becomes a value larger than the gauge D. Therefore, for example, a value slightly larger than the gauge D is set as the branch threshold 214, and when the distance 212 is larger than the branch threshold 214, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 0. That is, the calibration feasibility determination unit 103 determines the presence or absence of a branch from the first rail point group 30an, and determines that calibration is impossible if a branch exists. At this time, the calibration feasibility determination unit 103 determines the presence or absence of a branch by comparing the distance in the width direction of the vehicle with the gauge D of the rails 20Lt and 20Rt for the first rail point group 30an.

[0043] In Fig. 4(b), a case where the distance 212 is greater than the branch threshold 214 is indicated by, for example, point 216. Also in Fig. 4(b), a case where the distance 212 is smaller than the branch threshold 214 is indicated by, for example, point 215. This prevents the estimation of the external parameters from becoming inaccurate when a branch occurs, even if only partially, in the acquired first rail point cloud 30an, and enables estimation of the external parameters with higher accuracy.

[0044] Furthermore, the calibration feasibility determination unit 103 may further have a function of setting the calibration feasibility signal to 0 when the parallelism between the rail detection result and the ground surface falls below a predetermined threshold. In this embodiment, the parallelism is an index that indicates the degree to which two planes are parallel. Specifically, the method of calculating the parallelism is as follows: first, a first normal vector corresponding to the plane on which the first rail point group 30an is laid out is calculated. Next, a point group near the first rail point group 30an is extracted from the first sensor information, and the first rail point group 30an is removed from the extracted point group to calculate a second normal vector corresponding to the plane on which the point group representing the ground is laid out. Finally, the parallelism is determined by performing absolute value processing on the reciprocal of the angle between the first normal vector and the second normal vector. In this case, the larger the parallelism value, the closer to parallelism the two are. The point group near the first rail point group 30an is, for example, a point group present inside the left and right rails 20Lt and 20Rt.

[0045] Since the plane on which the first rail point group 30an is located and the plane on which the ground is located near the rails 20Lt, 20Rt are located are generally parallel, if the parallelism is small, the rail detection result may not be parallel to the ground, and the rail detection result may be incorrect. If a threshold is set for the parallelism and such incorrect rail detection results are prevented from being used by the automatic calibration unit 104 by judging the parallelism based on the threshold, more accurate calculation of external parameters becomes possible.

[0046] Next, the operation of the automatic calibration unit 104 will be described in detail. 5(a) to (d) are diagrams showing a coordinate system used in the first embodiment and a method of estimating external parameters using the first rail point cloud 30an. FIG. 5(a) is a diagram showing the first sensor coordinate system. The first sensor coordinate system is a coordinate system based on the first sensor 101a. The rail detection unit 102 converts information expressed in the first sensor coordinate system into a vehicle-fixed coordinate system '40n (FIG. 5(b)) using external parameter design values, and performs rail detection. The vehicle-fixed coordinate system '40n is a coordinate system that transitions from the first sensor coordinate system using external parameter design values ​​(rotation Rs2bn, translation ts2bn), and is a coordinate system that is slightly different from the vehicle-fixed coordinate system 40t (FIG. 5(c)).

[0047] Since the external parameters for converting between the first sensor coordinate system (FIG. 5(a)) and the vehicle-fixed coordinate system 40t (FIG. 5(c)) differ from the external parameter design values, a mismatch occurs between the vehicle-fixed coordinate system'40n (FIG. 5(b)) and the vehicle-fixed coordinate system (FIG. 5(c)). Estimation of the external parameters means calculating the rotation Rbn2b and translation tbn2b required for converting from the vehicle-fixed coordinate system'40n (FIG. 5(b)) to the vehicle-fixed coordinate system 40t (FIG. 5(c)), thereby calculating the rotation and translation required for converting the first sensor information to the vehicle-fixed coordinate system 40t (FIG. 5(c)) rather than the vehicle-fixed coordinate system'40n (FIG. 5(b)).

[0048] Next, the form of the rail detection result in each coordinate system will be described. 301Ls, 301Rs are the detection results of the left and right rails 20Lt, 20Rt in the first sensor coordinate system (Fig. 5(a)). Furthermore, the left rail model 301Lbn and the right rail model 301Rbn are the rail detection results in the vehicle-fixed coordinate system '40n (Fig. 5(b)). Furthermore, the left rail model 301Lbt and the right rail model 301Rbt are the actual positions of the rails 20Lt, 20Rt in the vehicle-fixed coordinate system (Fig. 5(c)). As shown in Fig. 5(b), since the vehicle-fixed coordinate system '40n and the vehicle-fixed coordinate system 40t are different, the left rail model 301Lbn and the left rail model 301Lbt, and the right rail model 301Rbn and the right rail model 301Rbt do not match.

[0049] Here, the rotation and translation for converting the left rail model 301Lbn to the left rail model 301Lbt and the right rail model 301Rbn to the right rail model 301Rbt are the rotation Rbn2b and translation tbn2b from the vehicle-fixed coordinate system '40n (Fig. 5(b)) to the vehicle-fixed coordinate system 40t (Fig. 5(c)). Therefore, in this embodiment, the rotation Rbn2b and translation tbn2b for converting the first representative position pbn and the first three-axis direction vector (exbn, eybn, ezbn) of the first rail point cloud 30an in the vehicle-fixed coordinate system '40n (Fig. 5(b)) to the second representative position pb and the second representative three-axis direction vector (exb, eyb, ezb) corresponding to the first rail point cloud 30an in the vehicle-fixed coordinate system (Fig. 5(c)) are calculated.

[0050] As a method of calculating the first representative position pbn, for example, any x-direction position where the first rail point group 30an exists is set as the x-coordinate of the first representative position, and the first rail point group 30an within a predetermined yz range is extracted based on the x-coordinate of the first representative position, and the extracted first rail point group is set. The left rail model 301Lbn and the right rail model 301Rbn are estimated for the extracted first rail point group, and the y-coordinate of the first representative position is set on the midline between the left rail model 301Lbn and the right rail model 301Rb. Furthermore, the z-coordinate corresponding to the x-coordinate of the first representative position and the y-coordinate of the first representative position on the three-dimensional plane spanned by the left rail model 301Lbn and the right rail model 301Rb is set as the z-coordinate of the first representative position.

[0051] The first representative position pbn is arbitrary in the x direction. However, based on the knowledge that the angle estimation accuracy may be improved by using distant object detection information, the x coordinate may be set to use the more distant first rail point group 30an, thereby improving the estimation accuracy of the external parameters related to the angle.

[0052] As a method for calculating the first three-axis vector (exbn, eybn, ezbn), for example, the unit vector in the x-direction of the left rail model 301Lbn is defined as exbn, the normal vector of the plane spanned by the left rail model 301Lbn and the right rail model 301Rb is defined as ezbn, and the unit vector eybn perpendicular to exbn and ezbn can be obtained by taking the cross product of exbn and ezbn.

[0053] Regarding the method of calculating the second representative position pb, first, in the case of a straight rail with no vehicle body inclination, since the positions of the rails 20Lt, 20Rt in the vehicle-fixed coordinate system are known, a vehicle-fixed coordinate system rail point cloud corresponding to the actual positions of the rails 20Lt, 20Rt in the vehicle-fixed coordinate system 40t is set. Next, the x-coordinate of the second representative position pb is defined as the x-coordinate of the first representative position pbn, assuming that the x-direction error between the vehicle-fixed coordinate system '40n (FIG. 5(b)) and the vehicle-fixed coordinate system 40t (FIG. 5(c)) is small. Furthermore, the first rail point cloud 30an within a predetermined range is extracted based on the x-coordinate of the first representative position pbn, and defined as an ideal rail point cloud. The left rail model 301Lbt and the right rail model 301Rbt are estimated with respect to the ideal rail point cloud, and the y-coordinate of the second representative position pb is set as the midline between the left rail model 301Lbt and the right rail model 301Rt. Furthermore, the z-coordinate corresponding to the x-coordinate of the first representative position pbn and the y-coordinate of the first representative position pbn on the three-dimensional plane spanned by the left rail model 301Lbt and the right rail model 301Rt is set as the z-coordinate of the second representative position pb.

[0054] Regarding the method of calculating the second three-axis vector (exb, eyb, ezb), for example, the unit vector in the x-direction of the left rail model 301Lbt is defined as exb, and the normal vector of the plane spanned by the left rail model 301Lbt and the right rail model 301Rt is defined as ezb, and the unit vector eyb perpendicular to exb and ezb can be obtained by taking the cross product of exb and ezb.

[0055] By solving the equation shown in Figure 5(d) using the first representative position pbn, first three-axis direction vector (exbn, eybn, ezbn), second representative position pb, and second three-axis direction vector (exb, eyb, ezb) obtained by the above method, it is possible to obtain the rotation Rbn2b and translation tbn2b for transforming from the vehicle-fixed coordinate system '40n (Figure 5(b)) to the vehicle-fixed coordinate system 40t (Figure 5(c)).

[0056] The above is an overview of the processing in the automatic calibration device 11a. By constructing such an automatic calibration device 11a, a new target is not required for detecting the rails 20Lt and 20Rt, and automatic calibration before and during traveling becomes possible. In addition, since it is not a prerequisite that the vehicle body is reflected as the first sensor information, there is no problem even if the vehicle body is difficult to reflect by using, for example, a sensor that monitors a distant object at a narrow angle as the first sensor 101a.

[0057] In addition, because the braking distance of trains is long, long-distance object detection is necessary to stop the train without colliding with objects detected by the forward monitoring system. Regarding the object position estimation error caused by errors in external parameters, if the vehicle's traveling direction is x, the direction of the sleepers is y, and the direction perpendicular to these is z, the farther away the object is, the greater the influence of yaw and pitch estimation errors. If these errors become large, distant objects cannot be detected correctly, so for trains, it is particularly important to accurately estimate the yaw and pitch angles. The rails 20Lt and 20Rt are objects that extend long in the x-direction, and since there is a large amount of sensor information in the x-axis direction, it is expected that the rail detection accuracy in the x-axis direction is good. Therefore, an automatic calibration method using rail detection is expected to have good estimation accuracy of external parameters in the x-axis direction. The external parameters related to the x-axis direction are the yaw angle and pitch angle that should be estimated with high accuracy in railway forward monitoring. By estimating the yaw angle and pitch angle with high accuracy, the performance of the first sensor 101a in detecting distant objects is improved, enabling safer driving.

[0058] In addition, in the conventional inclination estimation method, the inclination calculation result varies due to variable factors such as gradient, cant, and vibration during running, or vibration due to passenger movement even when stationary. Even if the conventional method is used by replacing the inclination estimation with the estimation of external parameters, it is difficult to estimate the external parameters with high accuracy in the presence of such variable factors. In order to estimate the external parameters with high accuracy, measures to improve the accuracy of the results are required in addition to external parameter estimation by rail detection. In addition, if the external parameters are defined as displacements relative to the design values, it is possible to estimate the external parameters based on the difference between the ideal position of the rail observed by the sensor and the actually observed position, but the ideal position of the rail detection differs depending on whether the vehicle is running on a curve or still on a straight line, making it difficult to calculate the ideal position.

[0059] In contrast, in this embodiment, by removing inappropriate rail point clouds when estimating the external parameters using the calibration feasibility determination unit 103, it is possible to eliminate the cause of variation and perform highly accurate calibration. In addition, in this embodiment, rail detection is performed on the rails 20Lt and 20Rt in a straight section, so that it is easy to identify the ideal position when detecting the rails. Furthermore, by estimating the external parameters based on the three-axis rotation and translation of the first sensor 101a, it is possible to calculate more accurate external parameters. Therefore, the automatic calibration device 11a in this embodiment can contribute to reducing the cost of forward monitoring systems and improving safety during driving.

[0060] In the present embodiment, a method has been described in which the external parameters are estimated when the calibration feasibility signal is 1, and the first sensor 101a is calibrated each time. On the other hand, the estimation accuracy of the external parameters may be improved by accumulating the external parameters when the calibration feasibility signal is 1 for a certain number of frames, and setting a representative value of the estimated values ​​of the external parameters for the certain number of frames as the external parameters to be used for calibration.

[0061] In Fig. 6, the horizontal axis indicates the number of accumulated frames of the estimation results of the external parameters, and the vertical axis indicates the estimation results of the external parameters. In Fig. 6, the estimation results of the external parameters for each iteration are shown as results 302, and the result 303 of calculating the median of the estimation results of the accumulated external parameters is also shown. In Fig. 6, as a specific example, the vertical axis indicates the y-direction translation ty, but the same applies to the x-direction translation tx and the z-direction translation tz. In the above example, the calibration feasibility determination unit 103 can eliminate a certain amount of curved rails and rail detection results with a lot of noise. However, a phenomenon occurs, although it is low in frequency, where a straight line that has a high degree of straightness but is different from a rail is erroneously detected as a rail. It is difficult to eliminate such erroneous detection results by determining the straightness. In this case, for example, as shown by 304 in FIG. 6, fluctuations occur in the calculated external parameters, leading to a decrease in calibration accuracy.

[0062] In order to prevent deterioration of calibration accuracy due to such events, data for a certain number of frames is accumulated as in the result 303, and a representative value is used as the external parameter for calibration, thereby removing low-frequency events that appear as outliers and enabling more accurate estimation of the external parameters. For the representative value, for example, a method of adopting a median value that is less susceptible to the influence of outliers can be used. In this case, the automatic calibration unit 104 records the three-axis rotation and translation of the first sensor 101a at multiple times and outputs the median value of the three-axis rotation and translation of the first sensor 101a at the multiple times, and the sensor information correction unit 105 corrects the first sensor information using the output median value.

[0063] Alternatively, a method may be used in which a fixed number of frames is set as the value fs when the fluctuation in the representative value becomes small, and this is determined by trial and error when the automatic calibration device 11a is introduced, or the number of accumulated frames may be increased by a predetermined value, and then the value fs may be determined based on the number of accumulated frames when the increase or decrease in the representative value converges within a certain range.

[0064] In this embodiment, a method for calculating the coordinate conversion between the first sensor coordinate system for one first sensor 101a and the vehicle-fixed coordinate system 40t has been described, but by applying the method of this embodiment to a plurality of sensors capable of detecting the rails 20Lt and 20Rt and calculating the rotation and translation from each sensor to the vehicle-fixed coordinate system, it is possible to more accurately integrate the plurality of sensors into the vehicle-fixed coordinate system 40t. In addition, since this method can estimate external parameters independently for each sensor, it is also advantageous in that the external parameters can be estimated even when the visible ranges of the sensors do not overlap. Therefore, according to this method, in a system using a plurality of sensors, more accurate automatic calibration is possible without installing a new target, so that the calibration cost in a system using a forward monitoring device can be reduced and the vehicle can be driven more safely. In this embodiment, the calibration method by rail detection has been described, but objects parallel to each other may be detected instead of the rails 20Lt and 20Rt. For example, two parallel upright overhead poles whose positions relative to the vehicle 10 are known can be detected by the first sensor 101a and used instead of the first rail point cloud 30an. In this case, calibration can be performed by the rail detection unit 102 detecting the first overhead pole point cloud, which is a point cloud corresponding to the two parallel upright overhead poles.

[0065] Second Embodiment In the first embodiment, a method for estimating the external parameters of the first sensor 101a has been described by comparing the straight rail detection result with the rail straight line in an ideal state in the vehicle-fixed coordinate system 40t. On the other hand, when it is not necessary to calculate the mounting position and angle of each sensor relative to the sensor mounting angle design value, a method may be used in which the first rail detection result detected by the first sensor 101a and the second rail detection result by a second sensor 101b different from the first sensor 101a are used to integrate each sensor information in a second vehicle-fixed coordinate system'. This content will be described below as the second embodiment.

[0066] FIG. 7 is a diagram showing the overall configuration of a sensor correction system 1b according to the second embodiment. The illustrated sensor correction system 1b differs from the sensor correction system 1a of the first embodiment shown in FIG. 1 in that a second sensor 101b is added to the automatic calibration device 11b, but otherwise is similar. 7 illustrates a first rail point cloud 30an as first sensor information obtained by detecting the left and right rails 20Lt, 20Rt using the first sensor 101a. Furthermore, in FIG. 7, a second rail point cloud 30bn is illustrated as second sensor information obtained by detecting the left and right rails 20Lt, 20Rt using the second sensor 101b. This can also be said to illustrate the first rail point cloud 30an expressed in the first vehicle-fixed coordinate system '40an and the second rail point cloud 30bn expressed in the second vehicle-fixed coordinate system '40bn. Since the external parameters of the first sensor 101a and the second sensor 101b each have an error with respect to the design value, the first rail point cloud 30an and the second rail point cloud 30bn do not match as shown in Fig. 7. In this embodiment, the external parameters are estimated and the first sensor information is corrected in order to integrate the first sensor information expressed in the first vehicle-fixed coordinate system '40an into the second vehicle-fixed coordinate system '40bn.

[0067] In this embodiment, the rail detection unit 102 estimates a first rail point cloud 30an using information from a first sensor 101a, and further estimates a second rail point cloud 30bn using information from a second sensor 101b. The calibration feasibility determination unit 103 calculates a first straightness based on the first rail point group 30an and a second straightness based on the second rail point group 30bn according to the method described in the first embodiment. Then, if both the first straightness and the second straightness are equal to or greater than a predetermined threshold, the calibration feasibility determination unit 103 sets a calibration feasibility signal to 1 and performs estimation of external parameters by the automatic calibration unit 104. On the other hand, if at least one of the first straightness and the second straightness is less than the predetermined threshold, the calibration feasibility determination unit 103 sets a calibration feasibility signal to 0 and does not perform estimation of external parameters by the automatic calibration unit 104. In the second embodiment, the reference rail point cloud is a second rail point cloud 30bn that is detected based on second sensor information acquired by a second sensor 101b and represents the positions of the rails 20Lt, 20Rt. The calibration feasibility determination unit 103 then determines whether calibration is possible based on the difference between the first straightness and a second straightness that represents the degree of straightness with respect to the second rail point cloud 30bn.

[0068] 8(a) to (e) are diagrams showing the operation of the automatic calibration unit 104 in the second embodiment. FIG. 8 shows a coordinate system used in this embodiment and a method for estimating external parameters by the automatic calibration unit 104. In FIG. First, the coordinate system will be described. The first sensor coordinate system (FIG. 8(a)) is a coordinate system based on the first sensor 101a. The rail detection unit 102 converts information expressed in the first sensor coordinate system into a first vehicle-fixed coordinate system' (FIG. 8(b)) using external parameter design values, and performs rail detection. The first vehicle-fixed coordinate system' is a coordinate system that is converted and transitioned from the first sensor coordinate system to the first vehicle-fixed coordinate system' (FIG. 8(b)) using external parameter design values ​​(rotation Rs12bn1, translation ts12bn1). The second sensor coordinate system (FIG. 8(c)) is a coordinate system based on the second sensor 101b. The rail detection unit 102 converts the information expressed in the second sensor coordinate system into a second vehicle-fixed coordinate system' (FIG. 8(d)) using the external parameter design values, and performs rail detection. The second vehicle-fixed coordinate system' is a coordinate system that is converted and transitioned from the second sensor coordinate system to the second vehicle-fixed coordinate system' using the external parameter design values ​​(rotation Rs22bn2, translation ts22bn2).

[0069] Next, the form of the rail detection result in each coordinate system will be described. 401Ls, 401Rs are the detection results of the left and right rails 20Lt, 20Rt in the first sensor coordinate system (Fig. 8(a)). Furthermore, the left rail model 401Lbn and the right rail model 401Rbn are the rail detection results in the first vehicle-fixed coordinate system' (Fig. 8(b)). Furthermore, 402Ls, 402Rs are the detection results of the left and right rails 20Lt, 20Rt in the second sensor coordinate system (Fig. 8(c)). Furthermore, the left rail model 402Lbn and the right rail model 402Rbn are the rail detection results in the second vehicle-fixed coordinate system' (Fig. 8(d)). Since the first vehicle-fixed coordinate system' and the second vehicle-fixed coordinate system' are different, 401Lbn and 402Lbn, and 401Rbn and 402Rbn do not match.

[0070] Here, the rotation and translation that transform 401Lbn to 402Lbn and 401Rbn to 402Rbn are rotation Rbn12bn2 and translation tbn12bn2 from the first vehicle-fixed coordinate system' to the second vehicle-fixed coordinate system'. Therefore, in this embodiment, the rotation Rbn12bn2 and translation tbn12bn2 are calculated to transform the first representative position pbn1 and the first three-axis direction vector (exbn1, eybn1, ezbn1) of the first rail point cloud 30an in the first vehicle-fixed coordinate system' into the second representative position pbn2 and the second three-axis direction vector (exbn2, eybn2, ezbn2) corresponding to the second rail point cloud 30bn in the second vehicle-fixed coordinate system'.

[0071] As a method of calculating the first representative position pbn1, for example, any x-direction position where the first rail point group 30an exists is set as the x-coordinate of the first representative position, and the first rail point group 30an within a predetermined range is extracted based on the x-coordinate of the first representative position, and the extracted first rail point group is set. The left rail model 401Lbn and the right rail model 401Rbn are estimated for the extracted first rail point group, and the y-coordinate of the first representative position is set on the midline between the left rail model 401Lbn and the right rail model 401Rb. Furthermore, the z-coordinate corresponding to the x-coordinate of the first representative position and the y-coordinate of the first representative position on the three-dimensional plane spanned by the left rail model 401Lbn and the right rail model 401Rb is set as the z-coordinate of the first representative position.

[0072] A method for calculating the first triaxial vector (exbn1, eybn1, ezbn1) is, for example, to define the unit vector in the direction of the left rail model 401Lbn as exbn1, the normal vector of the plane spanned by the left rail model 401Lbn and the right rail model 401Rb as ezbn1, and the unit vector eybn1 perpendicular to exbn1 and ezbn1 can be obtained by taking the cross product of exbn1 and ezbn1.

[0073] As a method of calculating the second representative position pbn2, the x coordinate of the first representative position is defined as the x coordinate of the second representative position, assuming that the x-direction error between the first vehicle-fixed coordinate system and the second vehicle-fixed coordinate system is small. Furthermore, the second rail point cloud 30bn within a predetermined range is extracted based on the x coordinate of the second representative position, and is defined as the second rail point cloud after extraction. The left rail model 402Lbn and the right rail model 402Rbn are estimated for the second rail point cloud after extraction, and the y coordinate of the second representative position is defined as the midline between the left rail model 402Lbn and the right rail model 402Rb. Furthermore, the z coordinate corresponding to the x coordinate of the second representative position and the y coordinate of the second representative position on the three-dimensional plane spanned by the left rail model 402Lbn and the right rail model 402Rb is defined as the z coordinate of the second representative position.

[0074] As a method for calculating the second triaxial vector (exbn2, eybn2, ezbn2), for example, the unit vector in the direction of the left rail model 402Lbn is defined as exbn2, the normal vector of the plane spanned by the left rail model 402Lbn and the right rail model 402Rb is defined as ezbn2, and the unit vector eybn2 perpendicular to exbn2 and ezbn2 can be obtained by taking the cross product of exbn2 and ezbn2.

[0075] By solving the equation shown in Figure 8(e) using the first representative position pbn1, first three-axis direction vector (exbn1, eybn1, ezbn1), second representative position pbn2, and second three-axis direction vector (exbn2, eybn2, ezbn2) obtained by the above method, it is possible to obtain the rotation Rbn12bn2 and translation tbn12bn2 for transforming from the first vehicle-fixed coordinate system' to the second vehicle-fixed coordinate system.

[0076] The above is the method for calculating external parameters using the first rail point cloud 30an and the second rail point cloud 30bn. According to the above method, the external parameters of the first sensor 101a can be calculated by rail detection, so that highly accurate automatic calibration can be performed without installing a new target. This reduces the calibration cost in a system using a forward monitoring device, and enables safer vehicle travel.

[0077] In this embodiment, the calibration is based on the detection result of a straight rail, but in the second embodiment using the first rail point group 30an and the second rail point group 30bn, the first rail point group 30an and the second rail point group 30bn may be curved rails. If the actual rails 20Lt, 20Rt are curved, for example, there is a method of calculating the rotation and translation by ICP (Iterative Closest Point) matching between the first rail point group 30an and the second rail point group 30bn.

[0078] Third embodiment In the first and second embodiments, when the first sensor 101a in the rail detection unit 102 produces a two-dimensional image, the two-dimensional rail detection result is converted into a three-dimensional rail point cloud, and the automatic calibration unit 104 estimates the external parameters. When converting a two-dimensional rail point cloud detected by a monocular camera into a three-dimensional image, it is difficult to convert the two-dimensional rail point cloud into a three-dimensional image unless assumptions are made, such as a constant gradient or no roll rotation around the x-axis, in order to estimate the depth direction, and it is not easy to obtain an accurate three-dimensional point cloud. If the target performance of the external parameter estimation result cannot be met due to errors when converting the two-dimensional rail detection result by the monocular camera into a three-dimensional image, it is necessary to consider a method that does not convert the two-dimensional rail point cloud into a three-dimensional image. The third embodiment has been made in consideration of the above problems, and for monocular camera information that is difficult to convert accurately into three-dimensional data, external parameters are estimated on a two-dimensional image.

[0079] FIG. 9 is a flow chart showing the operation of the automatic calibration device 11a when monocular camera information is used as the first sensor information. First, the rail detection unit 102 acquires monocular camera information as first sensor information (S201). Next, the rail detection unit 102 performs rail detection on the two-dimensional image (S202). Next, the rail detection unit 102 calculates a three-dimensional rail point cloud using the rail detection results on the two-dimensional image, while also obtaining a two-dimensional rail point cloud using the rail detection results on the two-dimensional image and storing it for use by the automatic calibration unit 104 (S203).

[0080] Next, the calibration feasibility determination unit 103 calculates the straightness based on the three-dimensional rail point cloud information (S204), and calculates a calibration feasibility signal (S205). The operation of the calibration feasibility determination unit 103 is similar to that of the first embodiment.

[0081] As mentioned above, when a two-dimensional rail point cloud is three-dimensionalized, it becomes inaccurate. However, since a phenomenon occurs in which the three-dimensional point cloud converted from the two-dimensional point cloud has a uniform difference in angles such as roll angle from a correct three-dimensional point cloud that has been accurately three-dimensionalized by some means, it can be considered that the relative positional relationship between the points in the point cloud is maintained. Therefore, if the three-dimensional point cloud converted from the two-dimensional point cloud draws a curve, the correct three-dimensional point cloud is also a curve, and if the three-dimensional point cloud converted from the two-dimensional point cloud is high noise, the correct three-dimensional point cloud is also high noise and the straightness has the same value. Therefore, there is no problem in performing threshold judgment using straightness by the method shown in the first embodiment.

[0082] Next, the automatic calibration unit 104 determines whether the calibration enable / disable signal is 1 (S206). As a result, when the calibration enable / disable signal is 1 (Yes in S206), the automatic calibration unit 104 captures the point cloud 30at corresponding to the actual positions of the rails 20Lt, 20Rt as the second rail point cloud 30bn (S207). Then, the automatic calibration unit 104 estimates the external parameters by matching the first rail point cloud 30an with the second rail point cloud 30bn on the two-dimensional image (S208). Furthermore, the sensor information correcting unit 105 corrects the first sensor information (S209). On the other hand, if the calibration availability signal is 0 (No in S206), the first sensor information is not corrected and the series of processes ends.

[0083] The second rail points cloud 30bn may be a three-dimensional rail points cloud in an ideal state as in the first embodiment, or may be estimated by the second sensor 101b as in the second embodiment. In the third embodiment, a specific example will be shown in which the ideal rail points cloud is adopted as the second rail points cloud 30bn in this embodiment.

[0084] 10(a) to 10(c) are diagrams showing details of the operation of S208 in FIG. 9 and the definition of the coordinate system to be used. The image coordinate system (Fig. 10(c)) is a two-dimensional orthogonal coordinate system with the origin at, for example, the upper left of the image, and by using the camera mounting position, design values ​​of the attitude, the camera matrix, etc., points on the image coordinate system are transformed into a vehicle-fixed coordinate system' (Fig. 10(b)), which is slightly different from the vehicle-fixed coordinate system. The external parameters to be estimated in this embodiment are the rotation Rcbn2b and translation tcbn2b from the vehicle-fixed coordinate system' (Fig. 10(b)) to the vehicle-fixed coordinate system (Fig. 10(a)).

[0085] In this embodiment, in order to estimate rotation and translation by matching in a two-dimensional image coordinate system, the automatic calibration unit 104 first sets ideal rail point groups 601Lb and 601Rb in a vehicle-fixed coordinate system, as in the first embodiment. Then, the automatic calibration unit 104 calculates rail point groups 601Lbn and 601Rbn by transforming to a vehicle-fixed coordinate system' using rotation Rb2cbn and translation tb2cbn, which are inverse transforms of rotation Rcbn2b and translation tcbn2b. Furthermore, the automatic calibration unit 104 transforms the rail point groups 601Lbn and 601Rbn to an image coordinate system using external parameter design information of the first sensor 101a, a camera matrix, and the like, to obtain 601Lc and 601Rc. Since the rotation Rb2cbn and the translation tb2cbn are indefinite, 601Lc and 601Rc are functions of the rotation Rb2cbn and the translation tb2cbn.

[0086] The automatic calibration unit 104 calculates the rotation Rb2cbn and the translation tb2cbn by numerical calculation so that the above 601Lc and 601Rc coincide with the two-dimensional rail point clouds 602Lc and 602Rc detected in the image coordinate system. The numerical calculation method may be a commonly used optimization algorithm such as the steepest descent method or simulated annealing method.

[0087] In the third embodiment, the first rail point cloud 30an represents the two-dimensional positions of the rails 20Lt, 20Rt, and the automatic calibration unit 104 converts the reference rail point cloud (in this case, the second rail point cloud 30bn) into an image coordinate system that includes the first rail point cloud 30an to obtain a two-dimensional reference rail point cloud (second rail point cloud 30bn), and calculates the three-axis rotation and translation to match the first rail point cloud 30an with the two-dimensional reference rail point cloud.

[0088] In this embodiment, there is no error in the angle of the rail point cloud, which occurs when converting a point cloud on a two-dimensional image into a three-dimensional image. Therefore, it is expected that more accurate estimation results of the external parameters can be obtained, enabling safer vehicle travel.

[0089] [Fourth embodiment] In the first to third embodiments, the straightness of the rail point cloud is calculated to prevent the rail point cloud including curved sections from being used by the automatic calibration unit 104. In addition to the above, the fourth embodiment shows a method of using a digital railway map that describes information associated with the railway's location to grasp in advance areas where the rail point cloud includes curved sections and where estimating the external parameters is difficult, thereby more reliably estimating the external parameters.

[0090] FIG. 11 is a diagram showing the overall configuration of a sensor correction system 1c according to the fourth embodiment. The sensor correction system 1c shown in the figure differs from the sensor correction system 1a of the first embodiment shown in Figure 1 in that a railway digital map 106, a sensor for estimating track position 107, and a track position estimation unit 108 have been added to the automatic calibration device 11c, but is otherwise similar.

[0091] The digital railway map 106 is a database that includes at least information for determining whether external parameters at a point, such as the curvature or the presence or absence of a branch at a point corresponding to the track position in the section in which the vehicle 10 travels, can be estimated. The on-rail position estimation sensor 107 is a sensor required for estimating the on-rail position of the vehicle 10, and examples thereof include a railway ground coil receiver, an encoder that measures the number of wheel rotations, and a GNSS (Global Navigation Satellite System) receiver. The on-rail position estimating unit 108 has a function of calculating the on-rail position of the vehicle in accordance with the information acquired by the on-rail position estimating sensor 107 .

[0092] 12(a) and 12(b) are diagrams showing the digital railway map 106 and a method for determining whether or not the external parameters can be estimated using the digital railway map 106. In FIG. FIG. 12( a ) is a diagram showing a specific example of the digital railway map 106 . In the digital railway map 106, the curvature of the rails and the presence or absence of a branch are recorded for each on-track position (kilometers [km]) along the median line 701 of the left and right rails 20Lt and 20Rt. For example, position 703a is a straight track with a kilometer distance of 1.0 km, a curvature of 0, and no branch (0 means no branch, 1 means there is a branch). The section indicated by diagonal lines in area 702a indicates that the curvature and branch information are the same as those of position 703a. Position 703b is a point where a branch occurs and the track is curved, and information is stored that the curvature is 0.0016 and there is a branch. Furthermore, position 703c is a straight track with a kilometer distance of 2.5 km, a curvature of 0, and no branch. The section indicated by diagonal lines in area 702b indicates that the curvature and branch information are the same as those of position 703c.

[0093] FIG. 12(b) is a diagram showing a method for determining whether or not the external parameters can be estimated using the digital railway map 106. For example, when the vehicle 10 is at position 705a and the section of area 706 where rail detection is possible is included in area 702a, it is guaranteed that the entire area where rail detection is possible is a straight track. This can be said that the calibration feasibility determination unit 103 determines that calibration is possible when the first rail point cloud 30an is included in the calibration possible section, based on a map (in this case, the railway digital map 106) including the relationship between the on-rail position of the vehicle 10 and the calibration possible section.

[0094] On the other hand, if the section of area 706 where rail detection is possible is not included in area 702a and includes a section where the curvature exceeds a predetermined threshold value, such as position 703b, or includes a section where a branch exists, the calibration feasibility determination unit 103 sets the calibration feasibility signal to 0. This makes it possible to prevent a rail point cloud including a curve or a branch from being used by the automatic calibration unit 104. This is expected to result in more accurate estimation results of external parameters, enabling safer vehicle travel.

[0095] Furthermore, when the calibration feasibility determination unit 103 finds that the straightness falls below a predetermined threshold even though the area ahead of the vehicle 10 is a straight line where calibration can be performed, such as a straight section that is not a branch, the calibration feasibility determination unit 103 may set the calibration feasibility signal to 0 and issue an alert indicating that there is a sensor abnormality. In this case, it can also be said that the calibration feasibility determination unit 103 issues a sensor abnormality signal when the first straightness is a value that does not allow calibration and the first rail point cloud 30an is included in a section where calibration is possible. By adding a function of issuing an alert to the calibration feasibility determination unit 103, a sensor abnormality can be detected and reflected in the operation of the vehicle 10, enabling safer vehicle driving.

[0096] Fifth embodiment If the rotation and translation of each sensor relative to a vehicle-fixed coordinate system can be calculated by the method described in the first embodiment, it is possible to compare the design values ​​and current values ​​of the rotation and translation of each sensor and detect sensor installation abnormalities. In the fifth embodiment, which takes this into consideration, sensor installation abnormalities are detected using the design values ​​and the calibration results of the first embodiment.

[0097] FIG. 13 is a diagram showing the overall configuration of a sensor correction system 1d according to the fifth embodiment. The sensor correction system 1d shown in the figure differs from the sensor correction system 1a of the first embodiment shown in FIG. 1 in that a sensor installation error detection unit 109 is added to the automatic calibration device 11d, but is otherwise similar.

[0098] The sensor mounting abnormality detection unit 109 determines that the sensor mounting angle is abnormal when the three-axis rotation and translation determined by the automatic calibration unit 104 exceed predetermined threshold values. The external parameters of the first sensor 101a output by the automatic calibration unit 104 are input to a sensor installation abnormality detection unit 109. The sensor installation abnormality detection unit 109 has a function of calculating the difference between the external parameters of the first sensor 101a output by the automatic calibration unit 104 and the external parameter design values ​​of the first sensor 101a as an abnormality degree for each coordinate axis (two or three translational axes and three rotational axes), and determining that the sensor installation is abnormal when the abnormality degree exceeds a predetermined threshold value.

[0099] Furthermore, as described in the second embodiment, it is also possible to use the first sensor 101a and the second sensor 101b and determine sensor installation abnormality using the estimated results of the first inter-sensor external parameters between the first sensor 101a and the second sensor 101b. In this case, a third sensor different from the first sensor 101a and the second sensor 101b is used, and an estimation result of a first four-way external parameter between the first sensor 101a and the third sensor is further obtained by the automatic calibration unit 104. Next, if an element of the estimation result of the first four-way external parameter exceeds a predetermined threshold and an element of the estimation result of the first four-way external parameter exceeds a predetermined threshold, it can be estimated that the first sensor 101a is abnormally attached.

[0100] In this case, the sensor mounting abnormality detection unit 109 can be said to determine that a sensor mounting angle abnormality exists when the three-axis rotation and translation obtained from the first rail point cloud 30an and the reference rail point cloud serving as the second rail point cloud 30bn, which is detected based on the second sensor information acquired by the second sensor 101b and represents the positions of the rails 20Lt, 20Rt, and the three-axis rotation and translation obtained from the first rail point cloud 30an and the third rail point cloud, which is detected based on the third sensor information acquired by the third sensor and represents the positions of the rails 20Lt, 20Rt, both exceed predetermined threshold values.

[0101] According to the fifth embodiment, it is possible to detect an abnormality in the mounting angle of each sensor in a section where rail detection is possible with a certain straightness or more. Therefore, it is possible to prevent failure to detect a forward object due to an abnormality in the mounting angle, and to enable safer driving.

[0102] In the above detailed embodiment, the automatic calibration is performed automatically, but the present invention is not limited to this, and the calibration may be performed manually. Furthermore, in the embodiment described above in detail, straightness is an index for evaluating whether the rail point cloud is a straight line or not, and the degree of variation in the rail point cloud data, but if sufficient accuracy can be ensured to determine a straight rail simply by determining whether the rail point cloud is a straight line, the degree of variation in the rail point cloud data does not necessarily have to be taken into consideration.

[0103] <Explanation of correction method> The above-described processes performed by the automatic calibration devices 11a to 11d are realized by the cooperation of software and hardware resources. That is, a processor in a computer provided in each of the automatic calibration devices 11a to 11d loads software for realizing each of the above-described functions into memory, executes the software, and realizes each of these functions.

[0104] Therefore, the processing performed by the automatic calibration devices 11a to 11d can be considered to be a correction method in which the processor executes software recorded in the memory to detect a first rail point cloud 30an representing the positions of the rails 20Lt, 20Rt based on the first sensor information acquired by the first sensor 101a, determines whether or not the first sensor 101a can be calibrated based on a first straightness representing the degree of straightness relative to the first rail point cloud 30an, and if it is determined that calibration is possible, corrects the first sensor information based on the first rail point cloud 30an and a reference rail point cloud that represents the three-dimensional positions of the rails 20Lt, 20Rt and is different from the first rail point cloud 30an.

[0105] Although the present embodiment has been described above, the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear from the claims that various modifications and improvements to the above embodiment are also included in the technical scope of the present invention. [Explanation of symbols]

[0106] 1a to 1d: sensor correction system, 11a to 11c: automatic calibration device, 20Lt, 20Rt: rail, 30an: first rail point cloud, 30at: point cloud, 101a: first sensor, 101b: second sensor, 102: rail detection unit, 103: calibration feasibility determination unit, 104: automatic calibration unit, 105: sensor information correction unit, 106: railway digital map, 107: sensor for estimating on-track position, 108: on-track position estimation unit, 109: sensor installation abnormality detection unit, rail: 20Lt, 20Rt

Claims

1. A rail detection unit that detects a first rail point group representing the position of a rail based on first sensor information acquired by a first sensor; A calibration feasibility determination unit that determines whether calibration of the first sensor is possible based on a first linearity representing the degree of a straight line with respect to the first rail point group; When it is determined in the calibration feasibility determination unit that calibration is possible, a sensor information correction unit that corrects the first sensor information based on the first rail point group and a reference rail point group representing the three-dimensional position of the rail and different from the first rail point group; A correction device having the above.

2. The correction device according to claim 1, wherein the first linearity represents, in addition to the degree of the straight line, the degree of variation with respect to the first rail point group.

3. The calibration feasibility determination unit obtains the first linearity based on an average value of distances between a rail straight line model estimated by performing straight line extraction from the first rail point group and points included in the first rail point group. The correction device according to claim 2.

4. The calibration feasibility determination unit obtains a first distance average value and a second distance average value as the average values for each of a pair of rails, and takes the reciprocal of the average value of the first distance average value and the second distance average value as the first linearity. The correction device according to claim 3.

5. The calibration feasibility determination unit determines the presence or absence of a branch from the first rail point group, and determines that calibration is not possible when there is a branch. The correction device according to claim 1.

6. The calibration feasibility determination unit according to claim 5, wherein the calibration feasibility determination unit determines the presence or absence of a branch by comparing the distance in the vehicle width direction and the gauge with respect to the first rail point group.

7. The correction device according to claim 1, wherein the reference rail point cloud is detected based on second sensor information acquired by a second sensor, and is a second rail point cloud representing the position of the rail.

8. The correction device according to claim 7, wherein the calibration feasibility determination unit determines the feasibility of calibration based on the difference between the first straightness and a second straightness representing the degree of straightness with respect to the second rail point cloud.

9. When it is determined that calibration is possible in the calibration feasibility determination unit, the correction device further includes a calibration unit that obtains the three-axis rotation and translation of the first sensor using the first rail point cloud and the reference rail point cloud. The correction device according to claim 1, wherein the sensor information correction unit corrects the first sensor information using the three-axis rotation and translation obtained by the calibration unit.

10. The calibration unit records the three-axis rotation and translation of the first sensor at a plurality of times, and outputs the median value of the three-axis rotation and translation of the first sensor at the plurality of times. The correction device according to claim 9, wherein the sensor information correction unit corrects the first sensor information using the output median value.

11. The first rail point cloud represents the two-dimensional position of the rail. The calibration unit converts the reference rail point cloud into an image coordinate system including the first rail point cloud to obtain a two-dimensional reference rail point cloud, and calculates the three-axis rotation and translation for matching the first rail point cloud with the two-dimensional reference rail point cloud. The correction device according to claim 9.

12. The correction device according to claim 9, further comprising a sensor mounting abnormality detection unit that determines that the sensor mounting angle is abnormal when the three-axis rotation and translation obtained by the calibration unit exceed a predetermined threshold.

13. The sensor mounting abnormality detection unit determines that there is an abnormal sensor mounting angle when both the three-axis rotation and the translation obtained from the first rail point group and a reference rail point group as a second rail point group that is detected based on second sensor information acquired by a second sensor and represents the position of the rail, and the three-axis rotation and the translation obtained from the first rail point group and a third rail point group that is detected based on third sensor information acquired by a third sensor and represents the position of the rail both exceed a predetermined threshold value. The correction device according to claim 12.

14. The calibration feasibility determination unit determines that calibration is feasible when the first rail point group is included in a calibration feasible section based on a map including the relationship between the in-track position of the vehicle and the calibration feasible section. The correction device according to claim 1.

15. The calibration feasibility determination unit issues a sensor abnormality signal when the first straightness is a value for which calibration is not possible and the first rail point group is included in a calibration feasible section. The correction device according to claim 14.

16. The correction device according to claim 1, wherein a first overhead line pole point group is used as a point group of two parallel and upright overhead line poles whose positions with respect to the vehicle are known, instead of the first rail point group.

17. By a processor executing software recorded in a memory, detecting a first rail point group representing the position of a rail based on first sensor information acquired by a first sensor, determining the feasibility of calibration of the first sensor based on a first straightness representing the degree of straightness of a straight line with respect to the first rail point group, when it is determined that calibration is feasible, correcting the first sensor information based on the first rail point group and a reference rail point group that represents the three-dimensional position of the rail and is different from the first rail point group, correction method.

Images