Sensing correction device

Abstract

To build a technique that calculates all necessary sensor state information including sensor state information in which a calculation is difficult because of other performance drop factors about changing sensor state information due to various performance drop factors in a sensing correction device having a sensor data correction unit calculating a parameter required for correcting factors of a performance drop from a result of comparison between normal time sensor data and operation time sensor data.SOLUTION: In a sensing correction device having a sensor data correction unit calculating post-corrected sensor data having a plurality of performance drop factors corrected from a result of comparison between normal time sensor data and operation time sensor data both including information on a characteristic object, the sensor data correction unit is configured to make a correction of the operation time sensor data, using sensor state information corresponding to each performance drop factor in order from the performance drop factor higher in independency of the plurality of performance drop factors.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a sensing correction device.

Background Art

[0002] In a railway driverless system and a railway operation support system, a system for avoiding a collision between a train and an object by monitoring the front of the train and detecting an object that obstructs the train's travel is essential. The means for monitoring the front of the train is realized by passing sensor data including information in front of the train acquired by a front monitoring sensor mounted in front of the train to a front monitoring logic unit that executes a program for estimating the position of an object in front of the vehicle. Examples of the front monitoring sensor include a camera and a LiDAR (Light Detection And Ranging) that scans a wide area with a laser and detects an object by performing laser ranging.

[0003] The performance of the above front monitoring sensor deteriorates due to various factors such as adhesion of an object to the sensor, sensor mounting angle, and displacement of the mounting position. The deterioration of sensor performance refers to a phenomenon in which sensor data acquired during operation becomes different from the original due to various factors. If the front monitoring logic unit uses sensor data affected by the deterioration of sensor performance, performance degradation of the front monitoring logic unit such as an increase in the error of estimating the position of an object in front of the vehicle and a decrease in the object detection rate occurs.

[0004] For example, when the mounting angle of a sensor mounted on a vehicle fluctuates due to vibration, the position of the object included in the sensor data moves relatively, so that the position of the object calculated by the front monitoring logic becomes a value different from the actual one. In such a case, there is a possibility of failing to detect an object that obstructs the vehicle's travel. If the detection of an object that obstructs the vehicle's travel fails, vehicle control based on the object detection cannot be performed, and there is a risk that a collision with the object cannot be avoided, which is dangerous.

[0005] Therefore, in order to achieve safe driving in a system equipped with a forward monitoring logic unit, it is necessary to correct the sensor data to remove the influence of various factors from the sensor data. If such correction can be performed in real time during vehicle operation, forward monitoring performance can be maintained even when sensor performance deteriorates, and safer driving can be achieved.

[0006] To correct the sensor data as described above, one method is to derive sensor state information by comparing sensor data under normal conditions and under operational conditions, and then correct the operational sensor data. This could potentially be realized by applying the inventions described in Patent Documents 1 and 2.

[0007] However, sensor status information refers to information that affects sensor data, defined in relation to each factor causing sensor performance degradation, and includes things like the sensor mounting angle and mounting position.

[0008] Patent Document 1 describes a method for calculating the frequency response characteristics from the air-fuel ratio adjustment means to the air-fuel ratio sensor, and for isolating and quantifying the deterioration mode of the air-fuel ratio sensor by using the gain characteristics and phase characteristics of the frequency response characteristics. Specifically, it proposes a method for comparing the gain characteristics, with frequency on the horizontal axis and gain on the vertical axis, and the phase characteristics, with frequency on the horizontal axis and phase on the vertical axis, with those of the normal state, and diagnosing that the frequency response characteristics have changed if the gain characteristics or phase characteristics differ by more than a predetermined value compared to the normal state.

[0009] Patent Document 2 proposes a method for maintaining the accuracy of hydrogen concentration detection in a hydrogen sensor system by adjusting the ratio of hydrogen concentration to output by using a coefficient calculated from parameters such as the battery operating time and power generation time, which are related to the degree of degradation of the hydrogen detection element. The formula for calculating the coefficient involves weighting the parameters and taking a weighted average of the coefficients obtained for each parameter. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2008-180225 [Patent Document 2] Japanese Patent Publication No. 2012-177634 [Overview of the project] [Problems that the invention aims to solve]

[0011] Actual sensor performance can degrade due to various factors such as lens distortion, sensor mounting position, angle deviation, and dirt accumulation on the sensor surface. Since sensor status information is defined according to the factors causing the performance degradation, multiple pieces of sensor status information need to be calculated. Therefore, to correct operational sensor data, it is necessary to individually calculate these multiple pieces of sensor status information by comparing sensor data under normal conditions with operational data.

[0012] Furthermore, sensor data during operation differs from normal sensor data due to multiple performance degradation factors. When calculating sensor state information corresponding to a certain factor, there is sensor state information that is difficult to calculate by comparing operational sensor data with normal sensor data due to the influence of sensor data changes caused by other factors.

[0013] Therefore, in order to accurately correct sensor data, it is necessary to solve the above problems. In other words, when correcting sensor data, it is necessary to calculate sensor state information corresponding to multiple existing performance degradation factors individually, while taking into account the influence of other performance degradation factors, and to correct the sensor data during operation. However, even if the contents of Patent Documents 1 and 2 are applied to the correction of sensor data, these problems cannot be solved.

[0014] Therefore, the object of the present invention is to construct a method for calculating all necessary sensor state information, including sensor state information that is difficult to calculate due to other performance degradation factors, in a sensing correction device having a sensor data correction unit that calculates parameters necessary to correct the factors of performance degradation from the results of comparing normal sensor data and operational sensor data. [Means for solving the problem]

[0015] To solve these problems, the present invention provides a sensing correction device having a sensor data correction unit that calculates corrected sensor data in which the factors causing performance degradation have been corrected based on the results of comparing normal sensor data containing information on characteristic objects with operational sensor data, wherein the sensor data correction unit corrects the operational sensor data using sensor state information corresponding to each performance degradation factor, in order from the performance degradation factors with the highest degree of independence among a plurality of performance degradation factors. [Effects of the Invention]

[0016] This invention provides a safer forward monitoring system that maintains forward monitoring performance by calculating all necessary sensor state information, including sensor state information that is difficult to calculate due to other performance degradation factors, and correcting the sensor data during operation, in relation to sensor state information that changes due to various performance degradation factors. [Brief explanation of the drawing]

[0017] [Figure 1] Configuration diagram of a vehicle forward monitoring system including a sensing and correction device. [Figure 2] Flowchart for calculating sensor state information according to Embodiment 1 of the present invention [Figure 3] A flowchart illustrating an example of a method for determining the order in which factor corrections are performed according to Embodiment 1 of the present invention. [Figure 4] List of performance degradation factors and their names related to Embodiment 1 of the present invention [Figure 5] Table showing specific examples of differences in sensor status information for verification corresponding to each factor [Figure 6] Flowchart for calculating the status information of camera data during operation according to Example 1 of the present invention [Figure 7] Explanatory diagram of the method for calculating sensor status information in the case of a camera according to Example 1 of the present invention [Figure 8] Configuration diagram of a forward monitoring system including a sensing correction device according to Example 2 of the present invention [Figure 9] Flowchart for obtaining normal sensor data according to Example 2 of the present invention [Figure 10] Configuration diagram of a forward monitoring system including a sensing correction device according to Example 3 of the present invention [Figure 11] Flowchart of the sensor deterioration diagnosis unit according to Example 3 of the present invention

Mode for Carrying Out the Invention

[0018] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings for the purpose of understanding the present invention. Note that the following embodiments are an example of embodying the present invention and do not limit the technical scope of the present invention. In particular, the examples described below will be described by taking the case of application to a forward monitoring sensor, but the present invention can be similarly applied to other monitoring sensors such as side monitoring sensors and rear monitoring sensors.

[0019] Hereinafter, embodiments will be described with reference to the drawings.

Example

[0020] In this example, an example of a sensing correction device that calculates all the sensor status information necessary for correcting the sensor data during operation after the forward monitoring sensor mounted on the vehicle acquires information on features in front of the vehicle as the sensor data during operation is shown.

[0021] Figure 1 shows a vehicle forward monitoring system 1 that includes a sensing correction device 10 that receives normal sensor data 101 and operational sensor data 202 as inputs and outputs corrected sensor data 103.

[0022] Using Figure 1, we will outline the method for calculating corrected sensor data 103 in the vehicle forward monitoring system 1.

[0023] First, the forward monitoring sensor 203 mounted on the vehicle 20 acquires information containing the characteristic object 30 in front of the vehicle 20 as operational sensor data 202, and inputs the operational sensor data 202 to the sensor data correction unit 102 in the sensing correction device 10.

[0024] Furthermore, normal sensor data 101, which includes information about characteristic objects 30 in front of the vehicle 20 when the performance of the forward monitoring sensor 203 has not deteriorated, is input to the sensor data correction unit 102. The sensor data correction unit 102 uses the operational sensor data 202 and the normal sensor data 101 to calculate corrected sensor data 103, which removes the effects of sensor performance deterioration during operation.

[0025] Vehicle 20 refers to any device capable of performing movement control based on information from the forward-facing sensor 203. Examples include trains, automobiles, construction machinery, and robots, but Vehicle 20 is not limited to these.

[0026] The forward-looking sensor 203 can include laser sensors such as LiDAR, millimeter-wave radar, cameras, infrared sensors, ultrasonic sensors, etc., but the forward-looking sensor 203 is not limited to these; it can be any sensor that can acquire information to calculate the position of the feature object 30.

[0027] Feature object 30 is an object that exists within the range that can be observed using the forward-looking sensor 203 and has certain characteristics that make it easy to find in the operational sensor data 202. Examples of feature objects 30 include, in the case of a camera, a calibration board with alternating black and white patterns, and in the case of LiDAR, a reflector made of metal with high reflectivity.

[0028] The operational sensor data 202 is sensor data that includes information about the feature object 30 obtained by the forward monitoring sensor 203 during the operation of the vehicle 20. In the case of a camera, for example, it is a two-dimensional image including the feature object 30, and in the case of LiDAR, for example, it is a three-dimensional point cloud including the feature object 30.

[0029] Operational operation refers to the time when the power to the forward monitoring sensor 203 is turned on. In the case of a car, this would be, for example, during driving or after engine ignition, and in the case of a train, it would be, for example, during train operation, when stopped at a station, or after power is supplied to the train in the depot.

[0030] The sensing correction device 10 is a device that has a sensor data correction unit 102 that holds normal sensor data 101 and inputs the normal sensor data 101 and operational sensor data 202 to calculate corrected sensor data 103 in which the effects of sensor performance degradation during operation have been removed. Alternatively, it may be implemented by a program executed on a CPU. The sensing correction device 10 may be mounted inside the vehicle 20 or mounted on equipment outside the vehicle 20.

[0031] The sensor data correction unit 102, upon receiving normal sensor data 101 and operational sensor data 202, calculates sensor state information corresponding to the performance degradation factors in the operational sensor data 202, starting with the most independent performance degradation factors. By using this sensor state information to remove the influence of the performance degradation factors, it calculates corrected sensor data 103. Details of the operation of the sensor data correction unit 102 will be described later.

[0032] A highly independent performance degradation factor is defined as a factor that, despite other performance degradation factors causing differences in the position and shape of feature objects 30 in the operational sensor data 202 and the normal sensor data 101, can calculate sensor state information with high accuracy for a particular performance degradation factor, and when the result of correcting the sensor data using the sensor state information closely matches the normal sensor data, that factor is considered to be a highly independent performance degradation factor.

[0033] Furthermore, accuracy is an indicator that represents the precision of the sensor state information calculation. The higher the accuracy, the more the sensor data corrected by the sensor state information is free from the effects of performance degradation factors corresponding to the sensor state information.

[0034] Sensor status information exists for each performance degradation factor. By using this sensor status information and a calculation formula that explains the sensor performance degradation, it is possible to obtain sensor data from which the performance degradation factors have been removed from the operational sensor data.

[0035] For example, in the case of a LiDAR sensor that acquires vehicle forward information as three-dimensional point cloud information, if the performance degradation is due to a shift in the sensor mounting position, the sensor state information is represented by a vector on a three-dimensional orthogonal xyz coordinate system. The formula for explaining the shift in the sensor mounting position is that the current sensor mounting position is obtained by adding the vector to the xyz coordinates of the sensor mounting position relative to the normal sensor data 101. Therefore, to obtain sensor data with the shift in the sensor mounting position removed, one simply needs to subtract the vector from the three-dimensional point cloud information acquired by the current sensor.

[0036] Normal sensor data 101 is sensor data containing information about the feature object 30 when there is no performance degradation of the forward monitoring sensor 203. This data is acquired before the vehicle 20 acquires the operational sensor data 202 containing information about the feature object 30, and is stored in the sensing correction device 10. One method for acquiring normal sensor data 101 is to use a new forward monitoring sensor to acquire sensor data in advance for various positional relationships between the vehicle and the feature object 30, and store this data in the sensing correction device 10. When inputting normal sensor data 101 to the sensor data correction unit 102, the normal sensor data 101 that is closest to the positional relationship between the vehicle 20 and the feature object 30 at the time the operational sensor data 202 was acquired should be selected and input to the sensor data correction unit 102.

[0037] Corrected sensor data 103 is sensor data obtained after the sensor data correction unit 102 has performed all factor corrections on the operating sensor data 202.

[0038] Next, the method for calculating sensor state information in the sensor data correction unit 102 will be specifically explained. Figure 2 is a flowchart for calculating sensor state information corresponding to performance degradation factors of the forward monitoring sensor 203 in the sensor data correction unit 102.

[0039] In Figure 2, the performance degradation factors are referred to in order from most independent to least independent, such as the first factor, second factor, etc., and the corrections are performed on the operating sensor data 202 in the order of first factor correction S10, second factor correction S20, etc.

[0040] In the first factor correction S10, first, the first sensor state information S101 is calculated using the normal sensor data 101 and the operational sensor data 202, and the first sensor state information S102 corresponding to the first factor is calculated.

[0041] Next, the first sensor state information application S103 is performed on the operational sensor data 202 using the first sensor state information S102 to correct for the first factor, and the first sensor data S104 is obtained from the operational sensor data 202 with the influence of the first factor removed.

[0042] In the second factor correction S20, first, the second sensor state information calculation S201 is performed using the normal sensor data 101 and the first sensor data S104, and the second sensor state information S202 corresponding to the second factor is calculated.

[0043] Next, the second sensor state information application S203 is performed on the first sensor data S104 using the second sensor state information S202 to correct for the second factor, thereby obtaining the second sensor data S204 from which the influence of the second factor has been removed from the first sensor data S104.

[0044] Furthermore, Figure 2 shows the flowchart up to the second factor correction S20. However, if there are more than two sensor degradation factors that require correction, the factor correction process is repeated from the third factor correction onward until all factor corrections are performed. For example, if there are three performance degradation factors, after acquiring the second sensor data S204, the third factor correction is performed to acquire the third state information and the third sensor data.

[0045] Finally, the sensor data calculated after all factor corrections are performed is output as corrected sensor data 103. For example, if there are three factors causing performance degradation, the first, second, and third factor corrections are performed, and the third sensor data calculated by the third factor correction is output as corrected sensor data 103.

[0046] By using the sensor data correction unit 102 shown in Example 1, it becomes possible to calculate sensor state information and correct sensor data, which are difficult to calculate due to other performance degradation factors.

[0047] For example, when calculating the second sensor state information S202, the case where it is difficult to calculate using the operational sensor data 202 and the normal sensor data 101 due to the influence of the first factor is described. In the present invention, the first factor correction S10 is performed before calculating the second sensor state information S202, and the second sensor state information S202 is calculated using the first sensor data S104 and the normal sensor data 101. Since the first sensor data S104 is not affected by the first factor, it is possible to calculate the second sensor state information S202 and obtain the second sensor data S204 from which the influence of the second factor has been removed.

[0048] Therefore, by using the sensor data correction unit 102 shown in this embodiment, all necessary sensor state information, including sensor state information that is difficult to calculate due to other performance degradation factors, can be calculated, and corrected sensor data 103 can be obtained from which all performance degradation factors have been removed. Since the performance of the forward monitoring logic unit 201 can be maintained by using the corrected sensor data 103, a safer vehicle forward monitoring system 1 can be provided.

[0049] Next, we will explain in more detail how to determine the order in which to apply factor corrections by identifying highly independent factors that cause performance degradation.

[0050] Figure 3 is a flowchart illustrating an example of a method for determining the order in which factor corrections are performed in the sensor data correction unit 102. A camera is used as the forward monitoring sensor 203 in this example. The order in which factor corrections are performed is determined before the vehicle is put into operation, and during operation, the factor corrections are performed according to that determined order.

[0051] First, we list the possible factors that could be causing the camera performance to degrade. Based on this list, we determine that the factors causing the camera performance degradation are lens distortion (factor A), field of view defects (factor B), and sensor misalignment (factor C) (Figure 4).

[0052] Next, we focus on factor A and examine whether the sensor state information corresponding to factor A can be correctly calculated even when the sensor performance is degraded by factors B and C. If the same value is calculated as the sensor state information corresponding to factor A in both cases—when the sensor performance is degraded by factors B and C and when it is not—then it can be said that the sensor state information corresponding to factor A can be calculated independently of factors B and C. Therefore, in such cases, factor A has a high degree of independence. The specific procedure for calculating the degree of independence of factor A is shown in the flowchart in Figure 3.

[0053] Figure 3 is a flowchart for calculating the degree of independence of factor A, using normal sensor data 101, first pre-degradation sensor state information S511 corresponding to factor A, second pre-degradation sensor state information S512 corresponding to factor B, and third pre-degradation sensor state information S513 corresponding to factor C.

[0054] The first pre-degradation sensor state information S511, the second pre-degradation sensor state information S512, and the third pre-degradation sensor state information S513 are sensor state information corresponding to the sensor performance degradation factors, and are information that is set in advance before the calculation in the flowchart of Figure 3.

[0055] First, a first performance degradation conversion S501 is performed on the normal sensor data 101 using the first pre-degradation sensor state information S511 for verification, thereby generating the first post-degradation sensor data S502.

[0056] The first performance degradation conversion S501 is a process that uses the first pre-degradation sensor state information S511 for verification and takes the normal sensor data 101 as input to generate the first post-degradation sensor data S502, which reproduces the sensor performance degradation corresponding to the sensor performance degradation factor set in the first pre-degradation sensor state information S511 for verification. For example, if lens distortion factor A is set as the first pre-degradation sensor state information S511 for verification, the first post-degradation sensor data S502 is generated with the lens distortion reproduced on the normal sensor data 101.

[0057] Next, a second performance degradation conversion S503 is performed on the first post-degradation sensor data S502 using the second pre-degradation sensor state information for verification, thereby generating the second post-degradation sensor data S504.

[0058] Furthermore, a third performance degradation conversion S505 is performed on the second post-degradation sensor data S504 using the second pre-degradation sensor state information for verification, thereby generating the third post-degradation sensor data S506.

[0059] The third degraded sensor data S506 is data in which performance degradation has occurred due to factors A, B, and C compared to the normal sensor data 101. Therefore, if the sensor state information corresponding to factor A, obtained using the third degraded sensor data S506 as input, is the same as the first pre-degradation sensor state information S511 used for verification, then factor A can be said to be a highly independent factor that allows for the correct calculation of sensor state information despite performance degradation caused by factors B and C.

[0060] Therefore, by performing the first sensor state information calculation S507 on the third post-decrease sensor data S506, the first verification post-decrease sensor state information S508 is obtained, and the difference with the first verification pre-decrease sensor state information S511 is calculated (S509) and obtained as the verification sensor state information difference S510. In this embodiment, the verification sensor state information difference corresponding to factor A is denoted as Δa, and the verification sensor state information differences corresponding to factors B and C are denoted as Δb and Δc.

[0061] For example, if the sensor state information corresponding to factor A is a matrix, the verification sensor state information difference S510 can be calculated by treating each element of the matrix as a one-dimensional vector, and then taking the absolute value of the difference between the one-dimensional vector of the first verification pre-degradation sensor state information S511 and the vector of the first verification post-degradation sensor state information S508.

[0062] The calculation of the sensor state information difference S510 for verification, as shown in the flowchart of Figure 3, is also performed for factors B and C. For example, if the sensor state information difference S510 corresponds to factor B, the first performance degradation conversion S501 and the first pre-degradation sensor state information for verification are set to correspond to factor B. Furthermore, the second performance degradation conversion S503 and the second pre-degradation sensor state information for verification S512 are set to correspond to factor C, and the third performance degradation conversion S505 and the third pre-degradation sensor state information for verification S513 are set to correspond to factor A.

[0063] Figure 3 shows the calculation of the difference in verification sensor state information S510 when affected by two factors, but the difference in verification sensor state information S510 when affected by one factor, for factors A, B, and C, is also calculated. Specifically, from the flowchart in Figure 4, the third performance degradation conversion S505, the third pre-degradation verification sensor state information S513, and the third post-degradation sensor data S506 are removed, and the first sensor state information calculation S507 is performed on the second post-degradation sensor data S504.

[0064] Finally, the verification sensor state information differences S510 obtained by the above method are compiled into a table, and the factors are arranged in order of their degree of independence. Figure 5 is a table showing a specific example in which the verification sensor state information differences S510 corresponding to factors A, B, and C are arranged in relation to the sensor performance degradation factors applied to normal sensor data.

[0065] Figure 5 shows, for example, that in the verification sensor state information difference S510, Δa based on factor A is calculated as 0.01 when sensor performance degradation due to factors B and C is applied in addition to factor A to the normal sensor data 101 (when affected by two factors), and Δa based on factor A is calculated as 0.02 when sensor performance degradation due to factor B is applied in addition to factor A to the normal sensor data 101 (when affected by one factor).

[0066] For each of the Δa, Δb, and Δc values ​​of the verification sensor state information difference S510, a pre-set threshold is set. If a value is smaller than the threshold, it is considered unaffected by the corresponding factor, and the number of values ​​smaller than the threshold is defined as independence. For example, if the pre-set thresholds for Δa, Δb, and Δc are 1, 1.5, and 2 respectively, then in the case of Δa, it falls below the threshold for all factors, and factor A is unaffected by factors B and C, so its independence is the highest, and the independence is 3. Factor B is affected by factor A because Δb is 5.0, but is unaffected by factor C because Δc is 0.02, which is below the threshold. Since there is only one Δb below the threshold, the independence is 1. Δc does not fall below the threshold for any factor, so its independence is 0.

[0067] Regarding the independence values ​​described above, if we rearrange the corresponding factors in descending order of their independence, we can arrange them from most to least independent. In this embodiment, the independence value for factor A is 3, the independence value for factor B is 1, and the independence value for factor C is 0. Therefore, arranging the factors from most to least independent results in factor A, factor B, and factor C.

[0068] Furthermore, the method of sorting by independence is not limited to using a threshold; for example, by normalizing the absolute values ​​of Δa, Δb, and Δc |Δa|, |Δb|, and |Δc| so that the maximum possible value is 1, then taking the average of |Δa|, |Δb|, and |Δc|, and sorting them in ascending order of the average, the order of independence can be determined.

[0069] By using the method for determining the order of factor correction as shown in Example 1, it is possible to determine the appropriate order of factor correction using a quantitative evaluation scale, thereby enabling more accurate calculation of the sensor state information of the forward monitoring logic unit 201. Since the performance of the forward monitoring logic unit 201 can be maintained by using the corrected sensor data 103, a safer forward monitoring system 1 can be provided.

[0070] Next, as a specific example of this embodiment, we will show an example of how the sensor data correction unit 102 calculates sensor state information when the forward monitoring sensor 201 is a camera.

[0071] First, we list the factors that degrade camera performance. In this embodiment, the factors that degrade camera performance are classified into, for example, steady-state noise such as lens distortion, misalignment of mounting position, misalignment of mounting angle, and dirt adhering to the lens, and transient noise such as rain and fog, which change position relative to the camera moment by moment.

[0072] The optimal order for performing factor corrections is determined using the method described above, and it is assumed that the corrected factors, starting with the first factor correction, are lens distortion, sensor mounting angle misalignment, sensor position misalignment, steady-state noise, and transient noise.

[0073] Based on the above analysis, a flowchart for factor correction is obtained. Figure 6 is a flowchart for calculating the state information of camera data during operation in this embodiment, and a detailed explanation is provided below.

[0074] First, in the initial factor correction, lens distortion correction S601 is performed using normal sensor data 101 and operational sensor data 202, which include information about the feature object 30, to obtain lens distortion information S611 and distortion-corrected sensor data S602.

[0075] Lens distortion information S611 is, for example, a coordinate transformation matrix that transforms a camera image without lens distortion into a camera image with lens distortion. It can be calculated by performing camera calibration, a technique that estimates lens distortion using a calibration board with a grid of black and white areas in front of the camera.

[0076] In lens distortion correction S601, distortion-corrected sensor data S602 is obtained by multiplying the camera image data during operation by the inverse of the coordinate transformation matrix to remove distortion.

[0077] Next, in the second factor correction, the sensor mounting angle correction S603 is performed using the distortion-corrected sensor data S602, which includes information about the feature object 30, and the normal sensor data 101, to obtain sensor mounting angle deviation information S613 and sensor data S604 after mounting angle correction.

[0078] Sensor mounting angle deviation information S613 is angular information that expresses the rotation from the sensor mounting angle corresponding to the normal sensor data 101 to the sensor mounting angle corresponding to the operational sensor data 202 in terms of roll, pitch, and yaw angles. For example, the roll, pitch, and yaw angles required to rotate the feature object 30 from the mounting angle calculated using the normal sensor data 101 to the mounting angle of the feature object 30 calculated using the distortion-corrected sensor data S602 can be used as the sensor mounting angle deviation information S613.

[0079] In sensor mounting angle correction S603, the distortion-corrected sensor data S602 is rotated by an angle obtained by multiplying the sensor mounting angle deviation information S613 by -1, thereby obtaining sensor data S604 after mounting angle correction, in which the deviation in the sensor mounting angle has been eliminated.

[0080] Next, in the third factor correction, sensor position correction S605 is performed using the mounting angle corrected sensor data S604, which includes information about the feature object 30, and the normal sensor data 101, to obtain sensor position deviation information S615 and position corrected sensor data S606.

[0081] Sensor position deviation information S615 is vector information that expresses the parallel movement from the sensor mounting angle corresponding to the normal sensor data 101 to the sensor mounting angle corresponding to the operational sensor data 202 in xyz coordinates. For example, the vector information necessary to parallel move the feature object 30 from the position calculated using the normal sensor data 101 to the position of the feature object 30 calculated using the sensor data S604 after mounting angle correction can be used as sensor position deviation information S615.

[0082] In sensor position correction S605, the sensor data S604 after mounting angle correction is converted to a vector on a pixel-by-pixel basis by multiplying the sensor position deviation information S615 by -1, and then added to obtain sensor data S606 after position correction in which the deviation in the sensor mounting angle has been eliminated.

[0083] In the fourth factor correction, steady-state noise correction S607 is performed using the normal sensor data 101 and the position-corrected sensor data S606 to obtain steady-state noise information S617 and steady-state noise-corrected sensor data S608.

[0084] Steady-state noise information S617 refers to information that constantly obstructs a certain range of the sensor's field of view in the position-corrected sensor data S606, such as mud or water stains on the lens, or scratches on the lens. Furthermore, steady-state noise information S617 can be defined, for example, as the range in the image data of an object if, when the difference between the normal sensor data 101 and the position-corrected sensor data S606 is extracted, an object that is not present in the normal sensor data 101 is present in the position-corrected sensor data S606, and the object does not change for a certain period of time.

[0085] In steady-state noise correction S607, for example, the same range as the steady-state noise information S617 of the normal sensor data 101 is copied to the steady-state noise information S617 present in the position-corrected sensor data S606, and pasted onto the steady-state noise information S617 of the position-corrected sensor data S606, thereby obtaining steady-state noise-corrected sensor data S608 from which the steady-state noise has been removed.

[0086] In the fifth factor correction, transient noise correction S609 is performed using the normal sensor data 101 and the steady-state noise-corrected sensor data S608 to obtain transient noise information S619 and transient noise-corrected sensor data S60A. Since the fifth factor correction is the final factor correction, the corrected sensor data S60A is output as corrected sensor data 103.

[0087] Non-stationary noise information S619 refers to information present in the sensor data S608 after stationary noise correction, which describes the range that obstructs the sensor's field of view, changing moment by moment. Examples include raindrops, snow, and fog during rainfall.

[0088] The importance of performing the above corrections from highly independent factors will be specifically explained using Figure 7. Figure 7 shows the transition from the operational sensor data image G101 to the distortion-corrected sensor data image G103, the mounting angle-corrected sensor data image G105, the position-corrected sensor data G107, and the steady-state noise-corrected sensor data image G109 when the forward monitoring sensor 203 is a camera and factor corrections are performed in the following order: lens distortion correction G102, sensor mounting angle correction G104, sensor position correction G106, and steady-state noise correction G108.

[0089] However, the object G201, indicated by a dotted line in the normal sensor data image G111, is a superimposed object from the operational sensor data image G101 and does not actually exist in the normal sensor data image G111. Also, G30 is a feature object 30 included in the normal sensor data image, G50 is a feature object 30 included in the operational sensor data image, and G40 is steady-state noise included in the operational sensor data. G50 and G40 are deformed, moved, and rotated due to camera lens distortion, sensor mounting angle, and mounting position deviations.

[0090] When distortion correction G102 is applied to the sensor data image G101 during operation in Figure 7, the distortion of feature object G50 is corrected, and the shapes of feature objects G51 and G30 in the distortion-corrected sensor data G103 and the normal sensor data image G113 match. At this point, for example, it becomes possible to compare the angles of line segments G501 and G511 present in feature objects G30 and G51 for the first time, making it easier to estimate the deviation of the rotation angle.

[0091] Next, when the sensor mounting angle misalignment correction G104 is applied to the distortion-corrected sensor data image G103 in Figure 7, the mounting angle misalignment of feature object G51 is corrected, and the mounting angles of feature objects G52 and G30 in the angle-corrected sensor data image G105 and the normal sensor data image G115 match. At this point, for example, since the line segments G501 and G521 present in feature objects G30 and G52 are parallel to each other, it becomes possible to compare the positions of the line segments for the first time, making it easier to estimate the position misalignment.

[0092] Furthermore, by applying sensor position shift correction G106 to the sensor data image G105 during operation in Figure 7, the positional shifts of the surrounding environment of feature objects G30 and G52 are corrected, and the feature objects G30, G53 and the surrounding environment in the position-corrected sensor data image G107 and the normal sensor data image G117 can be made to match. At this time, since the positions of feature objects G30, G53 and objects in the surrounding environment in the position-corrected sensor data image G107 and the normal sensor data image G117 match, only the steady-state noise correction value G127 can be extracted by taking the difference between the position-corrected sensor data G107 and the normal sensor data G117.

[0093] As described above, all corrections to camera data during operation can be performed in response to a decrease in camera performance, and the performance of the forward monitoring logic can be maintained even when camera performance deteriorates, thus providing a safer forward monitoring system.

[0094] Furthermore, as a specific example in this embodiment, an example of a method for calculating sensor state information in the sensor data correction unit 102 when the forward monitoring sensor 201 is a LiDAR is shown.

[0095] First, we list the factors that degrade LiDAR performance. In this embodiment, the factors that degrade LiDAR performance are classified into steady-state noise such as misalignment of mounting position, misalignment of mounting angle, and dirt adhering to the sensor, and transient noise such as rain and fog, which change position relative to the camera moment by moment.

[0096] Next, the optimal order for factor correction is determined by using the degree of agreement. Assuming that the first factor correction is performed in the order of sensor mounting angle misalignment, sensor position misalignment, steady-state noise, and transient noise, then the operational LiDAR sensor data can be corrected and state information calculated using the same method as when using a camera as an example.

[0097] As described above, the LiDAR sensor data can be fully corrected in response to a decrease in LiDAR performance, and the performance of the forward-looking logic can be maintained even when LiDAR performance deteriorates, thus providing a safer forward-looking system.

[0098] Furthermore, while this embodiment describes cameras and LiDAR as specific examples of forward-looking sensors 203, the forward-looking sensor 203 is not limited to these, and the same method can be applied regardless of the type of forward-looking sensor 203. Moreover, since it does not depend on the mounting position of the sensor, sensors used for side-looking or rear-looking monitoring are also applicable to forward-looking sensors, as they acquire data (camera images or LiDAR point clouds) related to objects in a certain direction, and therefore the same method as in this embodiment can be applied.

[0099] Furthermore, while this embodiment shows an example of passing corrected sensor data to the forward monitoring logic unit, it is also possible to pass corrected sensor data to the vehicle's self-position estimation logic.

[0100] For example, Simultaneous Localization and Mapping (SLAM), a technique that simultaneously performs self-localization and mapping using LiDAR sensor data for forward, side, or rearward monitoring, can be used. Therefore, if the sensor is LiDAR, it is possible to pass the corrected sensor data to the vehicle's self-localization logic. Similarly, with cameras, the vehicle's self-localization can be estimated using a technique called Visual SLAM, which estimates three-dimensional environmental information and the camera's position and mounting angle from the camera image. Thus, if the sensor is a camera, it is also possible to pass the corrected sensor data to the self-localization logic.

[0101] Therefore, by using this embodiment, it is possible to maintain the accuracy of the vehicle's self-position estimation and provide a safer forward-facing monitoring system.

[0102] In this embodiment, data correction for a forward-looking sensor was used as an example, but this embodiment can be applied to sensors other than forward-looking sensors as long as normal sensor data and operational sensor data for a certain feature can be obtained. For example, it can be applied to self-position estimation sensors such as GNSS (Global Navigation Satellite System) sensors.

[0103] For example, in the case of GNSS, when the vehicle is stopped at a point with known latitude and longitude, using the vehicle as a feature object, the vehicle's latitude and longitude are acquired as operational sensor data, and the latitude and longitude of the known point are acquired as normal sensor data, allowing correction by the sensor data correction unit. Furthermore, by passing the corrected sensor data to the self-position estimation logic, the accuracy of the vehicle's self-position estimation can be maintained, providing a safer forward monitoring system. [Examples]

[0104] Example 2 shows a specific example of a method for obtaining corrected sensor data 103 more accurately by selecting appropriate normal sensor data 101 according to the vehicle's self-position estimation result and then inputting the normal sensor data 101 to the sensor data correction unit 102.

[0105] Figure 8 shows the system configuration of the forward monitoring system 1 in Example 2. As shown in Figure 8, in this example, in addition to the system configuration of Example 1, there is also a self-position estimation logic unit 204 and a normal sensor data selection unit 105.

[0106] The self-position estimation logic unit 204 estimates the vehicle's own position and then passes the self-position and mounting angle information 104 to the normal sensor data selection unit 105. The self-position estimation logic unit 204 estimates the vehicle's current position using sensors, and the sensors used for self-position estimation include, for example, GNSS sensors and LiDAR mounted on the vehicle that measure the current position on the ground using artificial satellites, IMUs (Inertial Measurement Units) that detect translational and rotational motion in three orthogonal axes, magnetic sensors, and encoders installed near the wheels. The self-position estimation logic unit 204 may also receive position information from transponders installed on the ground.

[0107] Furthermore, the self-position and mounting angle information 104 represents the vehicle's self-position and mounting angle in some coordinate system. For example, the self-position may be expressed in three-dimensional Cartesian coordinates or latitude and longitude, and the mounting angle may be expressed in roll, pitch, yaw angles or quaternions relative to a certain coordinate system.

[0108] The normal sensor data selection unit 105 estimates the position and mounting angle of the feature object 30 as seen from the vehicle 20, based on its own position and mounting angle information 104 and the position information of the feature object 30 which it has stored in advance, and outputs the normal sensor data 101 that is closest to its own position and mounting angle.

[0109] The normal sensor data selection unit 105 will be explained in more detail using Figure 9.

[0110] Figure 9 is a flowchart illustrating the procedure for selecting the most suitable normal sensor data 101 from the self-position and mounting angle information 104 in the normal sensor data selection unit 105.

[0111] First, the self-position and mounting angle information 104 is input into the feature name and position database S701, and the feature name information S702 and feature position information S712 are obtained.

[0112] The feature name and location database S701 selects a feature 30 from among the feature objects that can be observed by the forward monitoring sensor 203 in front of the vehicle 20, based on the vehicle's own position and mounting angle information 104, which is the feature closest to the vehicle 20 or the feature closest to a certain distance, and outputs the feature name information S702 and feature location information S712 for the feature 30.

[0113] The feature name information S702 is information representing the name of the feature 30 that was assigned in advance before the vehicle 20 was put into operation. The feature location information S712 is the location information of the feature, described in the same coordinate system as the self-position information in the self-position and mounting angle information 104.

[0114] Next, using the feature object position information S712 and the self-position and mounting angle information 104, the relative position and mounting angle calculation S713 of the feature object is performed to calculate the relative position and mounting angle information S714 of the feature object.

[0115] The feature object relative position and mounting angle information S714 refers to the position and mounting angle information of the feature object 30 as viewed from the coordinate system fixed to the vehicle 20.

[0116] Furthermore, the feature object name information S702 and the relative position and mounting angle information S714 of the feature object are input to the normal sensor database S703, and the normal sensor data 101 is obtained.

[0117] The normal sensor database S703 stores a set of previously acquired normal sensor data, feature name information S702 included in the normal sensor data, and feature relative position and mounting angle information S714 as seen from the vehicle 20. When the feature name information S702 and the feature relative position and mounting angle information S714 are input to the normal sensor database S703, the normal sensor database S703 outputs the normal sensor data 101 that is closest to the feature relative position and mounting angle information S714 among the feature corresponding to the feature name information S702 stored in the normal sensor database S703.

[0118] By using the normal sensor data selection unit described in Example 2, when operational sensor data 202 for the feature object 30 is acquired, the most appropriate normal sensor data 101 for the position and mounting angle of the vehicle 20 and the feature object 30 can be used. Therefore, the sensor data correction unit 102, which takes the normal sensor data 101 as input, can acquire more accurate corrected sensor data 103, further contributing to preventing performance degradation of the forward monitoring logic unit 201. [Examples]

[0119] Example 3 shows a specific case in which a system is further added that diagnoses the degree of sensor degradation and determines the operational policy for vehicle operation by using the operational sensor status information 106 calculated by the sensor data correction unit 102.

[0120] Figure 10 is a system block diagram of this embodiment, and the sensing correction device 10 has an operation policy determination unit 108 that determines an operation policy based on the deterioration of the sensor after passing the operating sensor status information 106 to the sensor deterioration diagnosis unit 107.

[0121] The operational sensor status information 106 is information that includes all status information corresponding to the degradation factors of the forward monitoring sensor 203. For example, if there are three status information corresponding to the degradation factors of the forward monitoring sensor 203, namely sensor status information A, sensor status information B, and sensor status information C, then the operational sensor status information 106 includes all of the sensor status information A, sensor status information B, and sensor status information C.

[0122] The sensor degradation diagnosis unit 107 is a program that calculates a quantitative evaluation result of sensor performance degradation by using the operating sensor status information 106 to calculate how much the performance of the forward monitoring logic unit 201 has deteriorated. In this embodiment, based on the idea that the quantitative evaluation result of the forward monitoring logic performance, such as the range in which the forward monitoring logic can be performed, is similar to the quantitative evaluation result of the sensor performance, such as the range in which the sensor can acquire data, the sensor degradation diagnosis unit 107 executes a simulation of a sensor that has deteriorated in performance and outputs the quantitative evaluation result of the performance of the forward monitoring logic unit 201, which can be used to determine the operational policy, as the quantitative evaluation result of sensor performance degradation.

[0123] A specific example of the flowchart for the sensor degradation diagnosis unit in this embodiment is shown in Figure 11.

[0124] First, based on the operating sensor status information 106, a simulation model of the degraded sensor is created S801.

[0125] A sensor simulation model, for example, in the case of a camera sensor, takes as input the surrounding environment and the positions of distinctive objects placed in a virtual space recreated on a PC, as well as specifications such as the camera's field of view and mounting position on the vehicle, and obtains simulated camera image data that reproduces actual camera data. At this time, if the sensor status information 106 during operation is input to the sensor simulation model, it is possible to reproduce the degradation of sensor performance corresponding to the sensor status information.

[0126] Next, simulated sensor data S803 is obtained by performing a sensor simulation S802 according to a test scenario S811.

[0127] Test scenario S811 is a compilation of events related to forward monitoring that may occur during vehicle operation, prepared to evaluate the performance of the forward monitoring logic unit 201. Examples of such events include the appearance of an object at a certain distance in front of the vehicle.

[0128] Sensor simulation involves recreating the vehicle's surrounding environment in a virtual space based on a test scenario, driving the vehicle, and acquiring simulated sensor data S803 using a sensor simulation model.

[0129] Furthermore, the forward monitoring logic unit 201 is applied to the simulated sensor data S803 (S804), the performance evaluation of the forward monitoring logic unit 201 S805 is performed, and logic performance information S806 based on the current sensor degradation status is calculated.

[0130] Specifically, the evaluation of the forward monitoring logic unit 201 involves determining whether the forward monitoring logic unit 201 was able to detect an object placed at a certain position in front of the vehicle based on the test scenario S811, and calculating the range in which the forward monitoring logic unit 201 can detect objects during operation as logic performance information S806 based on the current sensor degradation status.

[0131] Finally, the logic performance information S806 is input to the operation policy determination unit 108, and the vehicle operation policy information S807 is obtained.

[0132] The operational policy determination unit 108 sets restrictions on the vehicle's driving method based on the sensor performance during operation. For example, focusing on the distance information included in the logic performance information S806 that allows for forward monitoring, it sets a limit on the vehicle's maximum speed so that the vehicle can stop at a certain distance, acquires the speed limit as vehicle operational policy information, and reflects it in the vehicle's driving.

[0133] By utilizing sensor simulation, the process from reproducing the degradation of sensor performance during operation to evaluating the performance of the forward monitoring logic unit 201 can be automatically executed on a PC. This allows for the rapid calculation of logic performance information S806 and the determination of operational policies.

[0134] Another method for evaluating the forward monitoring logic section due to sensor performance degradation is to reproduce the performance degradation using a real sensor and acquire sensor data by placing an object in front of the sensor. However, reproducing sensor performance degradation with a real sensor different from the forward monitoring sensor 203 is expected to require more effort compared to simulation methods, such as reproducing the sensor mounting position, positional deviation, and mounting angle deviation, as well as acquiring data by placing characteristic objects in various positions.

[0135] The method shown in Example 3 allows for restricting the vehicle's driving method according to the sensor's degradation status, preventing situations such as driving when forward monitoring is difficult due to sensor degradation. Furthermore, by utilizing simulation, driving method restrictions can be implemented quickly in response to sensor performance degradation, providing a safer forward monitoring system 1.

[0136] Furthermore, by limiting the speed according to the distance at which forward monitoring is possible, it is possible to avoid situations where a collision with an object ahead is unavoidable because the forward monitoring distance is shorter than the braking distance of the vehicle 20, thereby providing a safer forward monitoring system 1.

[0137] Furthermore, the logic performance information S806, based on the degradation status of the forward monitoring sensor 203, has the characteristic that the range in which forward monitoring can be performed narrows when the sensor performance deteriorates, and therefore can be used as a quantitative indicator of the sensor degradation status.

[0138] For example, if the evaluation index is calculated using the volume of the area to be detected by the forward monitoring logic unit 201 as the denominator and the volume of the area currently capable of forward monitoring, based on the degradation status of the forward monitoring sensor 203, as the numerator, then the index will decrease if the sensor performance deteriorates. Therefore, for example, if the index falls below a first threshold, a warning is issued indicating that the sensor performance has deteriorated, and if the index falls below a second threshold, information recommending sensor replacement due to a sensor failure is output. This allows for prompt action depending on the sensor failure status, providing a safer forward monitoring system 1.

[0139] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention. Furthermore, each of the embodiments described above may be used individually or in combination, and the configuration of one embodiment may be added to the configuration of another embodiment. [Explanation of Symbols]

[0140] 1. Forward-looking system 10. Sensing Correction Device 20...vehicles 30. Characteristic items 101...Normal sensor data 102...Sensor data correction unit 103...Corrected sensor data 104...Self-position and mounting angle information 105...Normal Sensor Data Selection Section 106...Sensor status information during operation 107...Sensor Degradation Diagnosis Department 108...Operation Policy Determination Department 201...Forward Monitoring Logic Unit 202...Sensor data during operation 203...Forward-looking sensor 204...Self-localization logic section

Claims

1. In a sensing correction device that takes sensor data acquired by a forward-monitoring sensor mounted on a vehicle as input and outputs corrected sensor data from which the effects of performance degradation of the forward-monitoring sensor have been removed, The sensor data includes normal sensor data that contains information about characteristic objects in front of the vehicle when the performance of the forward monitoring sensor has not deteriorated, The sensor data correction unit takes the normal sensor data and operational sensor data, which includes information on the characteristic object acquired by the forward monitoring sensor during vehicle operation, as inputs, and calculates the corrected sensor data using the normal sensor data and the operational sensor data. The aforementioned sensor data correction unit is Regarding the three performance degradation factors of the forward-monitoring sensor, the order is determined based on whether sensor state information can be calculated independently of all or any of the other performance degradation factors, using the sensor data when the performance of the forward-monitoring sensor is degraded due to all performance degradation factors and the normal sensor data, and using the sensor data when the performance of the forward-monitoring sensor is degraded due to any other performance degradation factor and the normal sensor data. In this order, the first sensor state information corresponding to the first performance degradation factor is calculated using the normal sensor data and the operational sensor data, the operational sensor data is corrected using the first sensor state information, and the first sensor data is calculated. The second sensor state information corresponding to the second performance degradation factor is calculated using the normal sensor data and the first sensor data, the first sensor data is corrected using the second sensor state information, and the second sensor data is calculated. The third sensor state information corresponding to the third performance degradation factor is calculated using the normal sensor data and the second sensor data, the second sensor data is corrected using the third sensor state information, and the corrected sensor data is calculated as the third sensor data. A sensing correction device characterized by the following.

2. A sensing correction device according to claim 1, The first, second, and third performance degradation factors of the forward-facing sensor are those that were listed as performance degradation factors of the forward-facing sensor before the vehicle was put into operation. A sensing correction device characterized by the following.

3. In a forward monitoring system comprising a vehicle equipped with a forward monitoring sensor and a sensing correction device that calculates corrected sensor data from sensor data acquired by the forward monitoring sensor, in which the effects of performance degradation of the forward monitoring sensor have been removed, The sensing correction device, The sensor data includes normal sensor data that contains information about characteristic objects in front of the vehicle when the performance of the forward monitoring sensor has not deteriorated, The sensor data correction unit takes the normal sensor data and operational sensor data, which includes information on the characteristic object acquired by the forward monitoring sensor during vehicle operation, as inputs, and calculates the corrected sensor data using the normal sensor data and the operational sensor data. The aforementioned sensor data correction unit is Regarding the three performance degradation factors of the forward-monitoring sensor, the order is determined based on whether sensor state information can be calculated independently of all or any of the other performance degradation factors, using the sensor data when the performance of the forward-monitoring sensor is degraded due to all performance degradation factors and the normal sensor data, and using the sensor data when the performance of the forward-monitoring sensor is degraded due to any other performance degradation factor and the normal sensor data. In this order, first sensor state information corresponding to the first performance degradation factor is calculated using the normal sensor data and the operational sensor data, the operational sensor data is corrected using the first sensor state information, and first sensor data is calculated. Second sensor state information corresponding to the second performance degradation factor is calculated using the normal sensor data and the first sensor data, the first sensor data is corrected using the second sensor state information, and second sensor data is calculated. Third sensor state information corresponding to the third performance degradation factor is calculated using the normal sensor data and the second sensor data, the second sensor data is corrected using the third sensor state information, and the corrected sensor data is calculated as the third sensor data. A forward-looking system characterized by the following features.

4. In a sensing correction method that calculates corrected sensor data from sensor data acquired by a forward-monitoring sensor mounted on a vehicle, in which the effects of performance degradation of the forward-monitoring sensor have been removed, Using the normal sensor data, which includes information on characteristic objects in front of the vehicle when the performance of the forward monitoring sensor has not deteriorated, and the operational sensor data, which includes information on the characteristic objects acquired by the forward monitoring sensor during the operation of the vehicle, the corrected sensor data is calculated. The calculation of the corrected sensor data is performed as follows: Regarding the three performance degradation factors of the forward-monitoring sensor, the order is determined based on whether sensor state information can be calculated independently of all or any of the other performance degradation factors, using the sensor data when the performance of the forward-monitoring sensor is degraded due to all performance degradation factors and the normal sensor data, and using the sensor data when the performance of the forward-monitoring sensor is degraded due to any other performance degradation factor and the normal sensor data. In this order, first sensor state information corresponding to the first performance degradation factor is calculated using the normal sensor data and the operational sensor data, the operational sensor data is corrected using the first sensor state information, and first sensor data is calculated. Second sensor state information corresponding to the second performance degradation factor is calculated using the normal sensor data and the first sensor data, the first sensor data is corrected using the second sensor state information, and second sensor data is calculated. Third sensor state information corresponding to the third performance degradation factor is calculated using the normal sensor data and the second sensor data, the second sensor data is corrected using the third sensor state information, and the corrected sensor data is calculated as the third sensor data. A sensing correction method characterized by the following.

Images