Information processing device and calibration method

Abstract

In the information processing device 103, the data obtaining unit 1031 obtains multimodal sensor data including image data of natural scene containing various objects and point cloud data of distances from the vehicle to the objects in a direction where the vehicle is moving. The object labeling unit 1033 labels each of the objects in the image data using semantic segmentation. The feature point data extraction unit 1035 extracts feature point data of labeled objects. The distribution calculation unit 1036 calculates distributions of eigen values of the feature point data of the labeled objects in each direction. The extrinsic calibration parameter determination unit 1037 determines the extrinsic calibration parameter by selecting an object among the labeled objects for calibration in at least one rotational alignment of the sensing devices based on the distributions calculated by the distribution calculation unit 1036.

Description

INFORMATION PROCESSING DEVICE AND CALIBRATION METHOD

[0001] The present invention relates to an information processing device and a calibration method for calibrating sensing devices mounted on a transportation apparatus.

[0002] To realize driverless operation of transportation apparatus such as trains and the like, highly reliable forward monitoring sensors are required. To ensure high reliability for the monitoring result, a sensor fusion that integrates the detection results of multiple sensing devices such as a camera, a Light Detection And Ranging (LiDAR), and an Inertial Measurement Unit (IMU) may be used. Each sensing device has its own coordinate system, to integrate properly, strict calibration of each sensor position and posture is required.

[0003] For example, a LiDAR may use one coordinate system and a camera may use another coordinate system. If the coordinate systems used by two different sensors are not calibrated together, any sensor fusion processing that combines data from the two sensors is likely to be inaccurate. Furthermore, the calibration parameters of various sensors of autonomous transportation apparatus drifted over time. It is necessary to compute the relative pose between onboard sensors to process the data in a common frame. Thus, extrinsic calibration computes the sensor’s relative pose improving data consistency between them.

[0004] Conventional techniques of calibration require manual processing by experts, thereby requiring the autonomous transportation apparatus mounted a plurality of sensing devices thereon to be provided to the experts for calibration of these sensing devices.

[0005] As the technology improves, automatic calibration methods have been introduced recently. Most common method involves displaying an object with predefined geometry at a known position, which is known as target-based extrinsic sensor calibration method. But this target-based calibration method introducing an artificial calibration of target objects in the existing infrastructure is cumbersome, time consuming and expensive due to additional infrastructure required. In addition, robust long-term autonomy requires a continuous assessment of calibration accuracy, which makes the use of artificial target object impractical.

[0006] Due to the above defects of target-based calibration, there are a lot of methods proposed which achieve automatic and targetless extrinsic calibration. For example, extracting geometric features from the natural scene and utilizing it as a calibration object can solve this issue.

[0007] In extrinsic sensor calibration method or system, the main challenge is to achieve calibration accuracy, more specifically rotational calibration accuracy. Translational calibration accuracy is generally easier to achieve than rotational calibration accuracy due to the simpler nature of linear movement, the ease of using single-axis references, the reduced effects of external factors, and the simpler identification and correction of errors. Hence, the main challenge is to calculate rotational calibration parameters accurately. The accuracy of extracted feature in natural scene depends on the regularity of the environment and the performance of the feature extraction algorithms.

[0008] The following PTL 1 is known as a prior art of the present invention. PTL 1 discloses a method and system for determining extrinsic calibration parameters for at least one pair of sensing devices mounted on transportable apparatus. The method obtains image data captured by an image generating sensing device and 3D point cloud data by a LiDAR, selects an image at a particular pose, generates a laser reflectance image based on a portion of the point cloud corresponding to the pose, and computes a metric measuring alignment between the selected image and the laser reflectance image.

[0009] [PTL 1] US 9,875,557 B2

[0010] In PTL 1, the laser reflectance image is generated based on a portion of the point cloud corresponding to the pose. The main drawback of this method is that it may not capture the entire scene accurately. Additionally, using only a portion of the point cloud can result in a loss of information, as some objects or details may be omitted. As a result, the sensing devices may be calibrated with less accuracy. This can be particularly problematic in applications where high levels of accuracy are required, such as in autonomous vehicles.

[0011] The present invention has been conceived in consideration of problems such as those described above, and its principal object is to improve rotational calibration accuracy of sensing devices.SOLUTION TO TECHNICAL PROBLEM

[0012] An information processing device according to the present invention, for determining extrinsic calibration parameter of sensing devices mounted on a transportation apparatus, comprises: a data obtaining unit that obtains multimodal sensor data including image data of natural scene containing various objects and point cloud data of distances from the transportation apparatus to the objects detected by the sensing devices in a direction where the transportation apparatus is moving; an object labeling unit that labels each of the objects in the image data using semantic segmentation; a feature point data extraction unit that extracts feature point data of labeled objects, which are labeled by the object labeling unit, from the point cloud data; a distribution calculation unit that calculates distributions of eigen values of the feature point data of the labeled objects in each direction of a coordinate system defined for the transportation apparatus; and an extrinsic calibration parameter determination unit that determines the extrinsic calibration parameter by selecting an object among the labeled objects for calibration in at least one rotational alignment of the sensing devices based on the distributions calculated by the distribution calculation unit. A calibration method according to the present invention comprises: obtaining multimodal sensor data including image data of natural scene containing various objects and point cloud data of distances from a transportation apparatus to the objects detected by the sensing devices mounted on the transportation apparatus in a direction where the transportation apparatus is moving; labeling each of the objects in the image data using semantic segmentation; extracting feature point data of the labeled objects from the point cloud data; calculating distributions of eigen values of the feature point data of the labeled objects in each direction of a coordinate system defined for the transportation apparatus; selecting an object among the labeled objects for calibration in at least one rotational alignment of the sensing devices based on the calculated distributions; and performing calibration of the sensing devices in the rotational alignment using the selected object.

[0013] According to the present invention, it is possible to improve rotational calibration accuracy of sensing devices.

[0014] These and other objects, advantages, purposes and features of the present invention will become apparent upon review of the following specification in conjunction with the drawings.

[0015] [Fig.1]Fig.1 is a schematic drawing of transportable apparatus mounted thereon a plurality of sensing devices and an information processing device according to an embodiment of the present invention. [Fig.2]Fig.2 represents an example of natural scene observed from the vehicle. [Fig.3]Fig.3 is an example of image data generated by capturing the natural scene. [Fig.4]Fig.4 is a figure for explanation of processing executed on the objects labeled as “electric pole” in the image data. [Fig.5]Fig.5 is a figure for explanation of processing executed on the object labeled as “railway track” in the image data. [Fig.6]Fig.6 is a figure for explanation of processing executed on the object labeled as “ground” in the image data. [Fig.7]Fig.7 represents a functional block diagram of the information processing device according to an embodiment of the present invention. [Fig.8]Fig.8 is the first section of the flowchart illustrating the processing executed by the information processing device. [Fig.9]Fig.9 is the second section of the flowchart illustrating the processing executed by the information processing device.

[0016] The following detailed description delineates various features and functions of the proposed invention with reference to the included figures. The illustrative system, functions and method embodiment described herein are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated here.

[0017] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0018] Figure 1 is a schematic drawing of a transportation apparatus mounted thereon a plurality of sensing devices and an information processing device according to an embodiment of the present invention. As shown in figure 1, a vehicle 100, which is a transportation apparatus runs on a railway track 10 toward a direction 20, is fitted with sensing devices 101a, 101b, 102a and 102b.

[0019] The sensing devices 101a and 101b are each capable of capturing image seen from the vehicle 100 in the direction 20. Cameras may be used as the sensing devices 101a and 101b. The sensing devices 102a and 102b are each capable of scanning objects located in front of the vehicle 100 in the direction 20 and detecting the distances from the vehicle 100 to the scanned objects. 2D or 3D LiDARs may be used as the sensing devices 102a and 102b. Note that figure 1 shows a pair of sensing devices 101a and 101b provided in parallel toward the direction 20 is mounted in close proximity of another pair of sensing devices 102a and 102b provided in parallel toward the direction 20, but arrangements and the number of sensing devices mounted on the vehicle 100 are not limited to this example.

[0020] The vehicle 100 also has an information processing device 103, which is in communication with the sensing devices 101a, 101b, 102a and 102b (via a wireless or wired communications interface) and is capable of storing and processing the data of all the sensing devices. A coordinate system 110 is defined for the vehicle 100. As shown in figure 1, the coordinate system 110 has x, y and z dimensional axes where x is longitudinal forward, y is latitudinal and z is upward direction, and the origin of these axes is at the center bottom of the vehicle 100 body.

[0021] Figure 2 represents an example of natural scene observed from the vehicle 100 shown in figure 1. When the vehicle 100 is moving along the railway track 10 in the direction 20, the natural scene 30 is observed from the vehicle 100 with the sensing devices 101a, 101b, 102a and 102b shown in figure 1. Image data of the natural scene 30 containing various objects is captured by the sensing devices 101a and 101b. The point cloud data comprising a plurality of scanned points indicative of distances from the vehicle 100 to the objects in the natural scene 30 is detected by the sensing devices 102a and 102b. These data are transmitted from each sensing devices to the information processing device 103 as multimodal sensor data.

[0022] Figure 3 is an example of image data generated by capturing the natural scene 30 shown in figure 2. The image data 40 representing the natural scene 30 observed by the sensing device 101a or 101b contains various objects. For instance, the object 33 is the railway track 10 along which the vehicle 100 is moving, the objects 31a, 31b and 31c are electric poles located along the railway track 10, the object 32 is bushes located aside the railway track 10, and the object 34 is grounds. These objects are each labeled by the information processing device 103 using semantic segmentation for the natural scene 30. The information processing device 103 may perform semantic segmentation by, for instance, machine learning using pre-trained model.

[0023] Sematic segmentation is a well-known technique of linking each pixel of an image with a class name. The labeling of the pixels in the image data 40 may be executed by assigning a class label to each pixel based on the learned features. This allows the model to identify the objects in the natural scene 30 and differentiate them from the background. By executing this processing, the objects 31a, 31b and 31c are labeled as “electric pole”, the object 32 is labeled as “bushes”, the object 33 is labeled as “railway track”, and the object 34 is labeled as “ground”. These labeled objects are potential candidates for calibration objects.

[0024] There are several existing algorithms for labeling sematic segmentation. In this embodiment, any kind of labeling method of semantic segmentation can be utilized for e.g., Synthetic labeling or interactive labeling with pre-trained model. The set of instructions in the form of sematic segmentation algorithm is processed in the information processing device 103.

[0025] Figure 4 is a figure for explanation of processing executed on the objects 31a, 31b and 31c labeled as “electric pole” in the image data 40. In figure 4(a), the binary image 41 shows an example of binary image of the objects 31a-31c extracted from the image data 40. This binary image 41 exhibits to visualize a complex topological behavior and smooth boundaries of the objects 31a-31c. It can be visualized with the help of binary image 41 that every shape of the objects 31a-31c labeled as “electric pole” is like a thin cylinder with long z distribution as per the defined coordinated system 110. The object 31a is the longest and the object 31c is the shortest among all, as the object 31a is nearest and the object 31c is at farthest distance as seen from the vehicle 100.

[0026] In figure 4(b), the graph 51 delineates the normal distribution curves of eigen values of feature point data of the object 31a in each direction of the coordinates system 110. The feature point data of the object 31a, which indicates the distances from the vehicle 100 to each point on the object 31a, is generated by extracting the data set corresponding to the region of the object 31a from the point cloud data obtained from the sensing device 102a or 102b. The graph 51 includes distribution curves 511a, 511b and 511c indicating the normal distribution of eigen values of the feature point data in the x-axis direction, y-axis direction and z-axis direction of the coordinated system 110 respectively. The shape of eigen vector 512 indicates the eigen values of the object 31a in x, y and z directions.

[0027] As the shape of the object 31a is such that it has relatively long distribution in z direction with respect to x and y directions, the distribution curve 511c has higher values than the distribution curves 511a and 511b. In other words, since the object 31a has higher degree of scattering in z direction, it has low rotational alignment confidence level along z direction (in pitch rotation). Thus, by utilizing the object 31a for extrinsic calibration parameters along x and y directions (in roll and yaw rotations), the sensing devices can be calibrated with high accuracy.

[0028] Figure 5 is a figure for explanation of processing executed on the object 33 labeled as “railway track” in the image data 40. In figure 5(a), the binary image 42 shows an example of binary image of the object 33 extracted from the image data 40. This binary image 42 exhibits to visualize a complex topological behavior and smooth boundaries of the object 33. It can be visualized with the help of binary image 42 that the shape of object 33 labeled as “railway track” is long x distribution as per the defined coordinated system 110.

[0029] In figure 5(b), the graph 52 delineates the normal distribution curves of eigen values of feature point data of the object 33 in each direction of the coordinates system 110. The feature point data of the object 33, which indicates the distances from the vehicle 100 to each point on the object 33, is generated by extracting the data set corresponding to the region of the object 33 from the point cloud data obtained from the sensing device 102a or 102b. The graph 52 includes distribution curves 521a, 521b and 521c indicating the normal distribution of eigen values of the feature point data in the x-axis direction, y-axis direction and z-axis direction of the coordinated system 110 respectively. The shape of eigen vector 522 indicates the eigen values of the object 33 in x, y and z directions.

[0030] As the shape of the object 33 is such that it has relatively long distribution in x direction with respect to y and z directions, the distribution curve 521a has higher values than the distribution curves 521b and 521c. In other words, since the object 33 has higher degree of scattering in x direction, it has low rotational alignment confidence level around x direction (in roll rotation). Thus, by utilizing the object 33 for extrinsic calibration parameters along y and z directions (in yaw and pitch rotations), the sensing devices can be calibrated with high accuracy.

[0031] Figure 6 is a figure for explanation of processing executed on the object 34 labeled as “ground” in the image data 40. In figure 6(a), the binary image 43 shows an example of binary image of the object 34 extracted from the image data 40. This binary image 43 exhibits to visualize a complex topological behavior and smooth boundaries of the object 34. It can be visualized with the help of binary image 43 that the shape of object 34 labeled as “ground” is long x and y distributions as per the defined coordinated system 110.

[0032] In figure 6(b), the graph 53 delineates the normal distribution curves of eigen values of feature point data of the object 34 in each direction of the coordinates system 110. The feature point data of the object 34, which indicates the distances from the vehicle 100 to each point on the object 34, is generated by extracting the data set corresponding to the region of the object 34 from the point cloud data obtained from the sensing device 102a or 102b. The graph 53 includes distribution curves 531a, 531b and 531c indicating the normal distribution of eigen values of the feature point data in the x-axis direction, y-axis direction and z-axis direction of the coordinated system 110 respectively. The shape of eigen vector 532 indicates the eigen values of the object 34 in x, y and z directions.

[0033] As the shape of the object 34 is such that it has relatively long distribution in x and y directions with respect to z direction, the distribution curves 531a and 531b have higher values than the distribution curve 531c. In other words, since the object 34 has higher degree of scattering in x and y directions, low rotational alignment confidence level around x direction and along y direction (in roll and yaw rotations). Thus, by utilizing the object 34 for extrinsic calibration parameters along z direction (in pitch rotation), the sensing devices can be calibrated with high accuracy.

[0034] In the processing explained with referring to figures 4-6, a principal component analysis (PCA) algorithm may be utilized to extract the feature point data from the point cloud data corresponding to the objects 31a, 33 and 34. PCA works by identifying the principal components of the point cloud data. These principal components are linear combinations of the original variables that account for the largest amount of variation in the dataset. By projecting the data onto these principal components, we can reduce the dimensionality of the feature point data while still retaining most of the original variation.

[0035] Figure 7 represents a functional block diagram of the information processing device 103 according to an embodiment of the present invention. The information processing device 103 functionally has a data obtaining unit 1031, a calibration judgement unit 1032, an object labeling unit 1033, an object selection unit 1034, a feature point data extraction unit 1035, a distribution calculation unit 1036, an extrinsic calibration parameter determination unit 1037, and a calibration unit 1038. The information processing device 103 may realize these functional blocks by executing predetermined programs with a CPU.

[0036] The data obtaining unit 1031 obtains the multimodal sensor data from the sensing devices 101a, 101b, 102a and 102b. The multimodal sensor data including image data of the natural scene 30 containing objects 31a, 31b, 31c, 32, 33 and 34, and point cloud data of distances from the vehicle 100 to these objects, is transmitted from the sensing devices to the information processing device 103, thereby the data obtaining unit 1031 can obtain these data.

[0037] The calibration judgement unit 1032 makes a decision whether or not calibration for the sensing devices is necessary. If it is judged as calibration is necessary, the following processing is performed by the other units.

[0038] The object labeling unit 1033 takes the image data of the multimodal sensor data obtained by the data obtaining unit 1031 and labels each of the objects in the image data using semantic segmentation. By this processing, the objects are labeled as “electric poles”, railway track”, “bushes” and “ground” respectively.

[0039] The object selection unit 1034 selects, if a plurality of objects are labeled in the image data with an identical label by the object labeling unit 1033, the nearest one from these objects. In this processing, the distances of the objects can be determined based on the point cloud data included in the multimodal sensor data obtained by the data obtaining unit 1031. By this processing, the object 31a is selected in the objects 31a-31c each labeled as “electric poles”. The object 31a thus selected and the objects 32, 33 and 34 labeled as “bushes”, “railway track” and “ground” respectively are designated as targets of the following process.

[0040] The feature point data extraction unit 1035 extracts feature point data for each of the objects 31a, 32, 33 and 34. In this processing, the data set corresponding to the region of each object is extracted from the point cloud data included in the multimodal sensor data obtained by the data obtaining unit 1031, and the extracted data set is designated as the feature point data of each object. As explained earlier, PCA algorithm may be used to identify the principal components of the point cloud data.

[0041] The distribution calculation unit 1036 calculates distribution of eigen values of the feature point data of each object extracted by the feature point data extraction unit 1035 in each direction of the coordinate system 110. By this processing, as shown in figure 4(b), the distribution curves 511a, 511b and 511c and the eigen vector 512 illustrated in the graph 51 are calculated for the object 31a in x, y and z directions of the coordinate system 110. Similarly, as shown in figure 5(b), the distribution curves 521a, 521b and 521c and the eigen vector 522 illustrated in the graph 52 are calculated for the object 33 in x, y and z directions of the coordinate system 110. Moreover, as shown in figure 6(b), the distribution curves 531a, 531b and 531c and the eigen vector 532 illustrated in the graph 53 are calculated for the object 34 in x, y and z directions of the coordinate system 110. Note that illustration of distribution curves and eigen vector for the object 32 are omitted, they are likely to have some extent value (size) in each direction.

[0042] The extrinsic calibration parameter determination unit 1037 determines extrinsic calibration parameters based on the distribution of eigen values of the feature point data calculated by the distribution calculation unit 1036 for each object. In this processing, at first, calibration confidence level is calculated for each object in each direction from the distribution of eigen values. Specifically, higher the distribution of eigen values is in one direction, lower the confidence level is calculated in that direction. Next, the calculated calibration confidence level of each object in each direction is compared with a predetermined level and, if the calibration confidence level of any object in any direction is higher than the predetermined level, the object is selected as an object for calibration in the rotational alignment along the direction. By this processing, calibration confidence levels of the distribution curves 511a and 511b with low eigen values as shown in figure 4(b) are judged to be higher than the predetermined level, thereby the object 31a is selected as an object for calibration in roll and yaw rotations. Similarly, calibration confidence levels of the distribution curves 521b and 521c with low eigen values as shown in figure 5(b) are judged to be higher than the predetermined level, thereby the object 33 is selected as an object for calibration in yaw and pitch rotations. Moreover, calibration confidence level of the distribution curve 531c as shown in figure 6(b) with low eigen values is judged to be higher than the predetermined level, thereby the object 34 is selected as an object for calibration in pitch rotation.

[0043] To summarize, better calibration accuracy for roll and raw rotational calibration can be achieved if the object 31a is utilized as an extrinsic calibration object. In addition, better calibration accuracy for yaw and pitch rotational calibration can be achieved if the object 33 is utilized as an extrinsic calibration object. Moreover, better calibration accuracy for pitch rotational calibration can be achieved if the object 34 is utilized as an extrinsic calibration object. Lastly comparison and selection of suitable object for calibration for particular rotational alignment calibration where its accuracy is maximum is determined by the extrinsic calibration parameter determination unit 1037, thereby the extrinsic calibration parameters for the sensing devices 101a, 101b, 102a and 102b are determined.

[0044] The calibration unit 1038 performs calibration of the sensing devices 101a, 101b, 102a and 102b utilizing the extrinsic calibration parameters determined by the extrinsic calibration parameter determination unit 1037.In this processing, for instance, any one of the sensing devices 101a, 101b, 102a and 102b is selected as a base sensor and the other sensing devices are calibrated to be matched with the base sensor utilizing the selected object for calibration in each rotational direction.

[0045] Figures 8 and 9 are the flowchart illustrating the processing executed by the information processing device 103. The steps of the processing of the flowchart shown in figures 8 and 9 are exemplary, some of them may be omitted and / or re-ordered. Further, the processing can be implemented using any suitable programming language and data structures.

[0046] The whole flowchart is explained in two sections. The first section including steps S10-S70 shown in figure 8 explains the processing to check if the extrinsic calibration is required or not by a simple used algorithms in the existing literatures. The second section including steps S100-S180 shown in figure 9 explains the processing to determine the extrinsic calibration parameters and execute calibration. The overall flowchart combining the first and second sections is the efficient scenario for the proposed algorithm.

[0047] In step S10, the data obtaining unit 1031 obtains the multimodal sensor data from the sensing devices 101a, 101b, 102a and 102b.

[0048] In step S20, the calibration judgement unit 1032 selects any one of the sensing devices 101a, 101b, 102a and 102b, such as the sensing device 101a, as a base sensor for extrinsic calibration.

[0049] In step S30, the calibration judgement unit 1032 selects region of interest (ROI) to derive directional feature point from each sensing device and derives directional feature point in all sensing devices. By performing this processing, the directional feature in each direction is determined for each of the sensing devices 101a, 101b, 102a and 102b.

[0050] In step S40, the calibration judgement unit 1032 projects the directional feature points of the sensing devices 101b, 102a and 102b, which are not selected in the step S20, to that of the sensing device 101a selected as the base sensor, using geometric coordinate relation between the frames of these sensing devices. Each sensing device has its own frame of reference, therefore it is necessary to compare the same detected feature points in one frame of reference i.e., base sensor. Once all the feature points are transformed into the same frame of reference, the processing proceed to next step S50.

[0051] In step S50, the calibration judgement unit 1032 finds the error among all projected feature points and base sensor feature points.

[0052] In step S60, the error obtains in the step S50 is checked whether the error is greater than a given threshold value. If the error is greater than the threshold value, it is judged to be necessary with calibration, then the processing proceeds to step S100 of the second section in figure 9. If the error is not greater than the threshold value, the processing proceeds to step S70.

[0053] In step S70, the decision of no calibration required is made. Subsequently the processing returns to step S20, and further steps will be executed again.

[0054] In step S100, the object labeling unit 1033 labels the objects in the image data obtained in step S10 by using an algorithm for sematic segmentation.

[0055] In step S110, the object selection unit 1034 selects the nearest sematic labeled object for a potential calibration object among the objects to which an identical semantic label is attached. If there are a plurality of object groups with respective identical labels, the nearest object in each group is selected in step S110. If there is only one object with a specific label, the processing of step S110 is omitted for the object. By this processing, one or more objects each have different labels are selected for the following process.

[0056] In step S120, the object selection unit 1034 selects any one of the objects selected in step S110. The currently selected object in the step S120 is designated as a target object for the following steps S130-S170.

[0057] In step S130, the feature point data extraction unit 1035 extracts feature point data from the point cloud data obtained in step S10 for the target object selected in step S120. In this processing, PCA algorithm may be used to identify the principal components of the point cloud data.

[0058] In step S140, the distribution calculation unit 1036 calculates the normal distribution of eigen values of the feature point data in each direction extracted in step S130 and, based on the calculated distribution, the extrinsic calibration parameter determination unit 1037 calculates confidence level of calibration in each direction for the current target object selected in step S120. This processing is executed as explained with reference to figures 4, 5 and 6.

[0059] In step S150, the extrinsic calibration parameter determination unit 1037 makes a decision, based on the confidence levels of calibration calculated in step S140, if there is any direction with maximum confidence level for the current target object among all objects selected in step S110. In case that the confidence level of calibration for the current target object is higher than a predetermined level and maximum among all objects in at least one direction, the extrinsic calibration parameter determination unit 1037 judges “yes” in step S150 and then proceeds to step S160. On the other hand, in case that the confidence level of calibration for the current target object is equal to or lower than the predetermined level or not maximum in any direction, the extrinsic calibration parameter determination unit 1037 judges “no” in step S150 and then proceeds to step S180.

[0060] In step S160, the extrinsic calibration parameter determination unit 1037 selects the current target object for extrinsic calibration parameter. In this processing, the feature point data of the current target object is selected as the extrinsic calibration parameter for the direction judged as the maximum in step S150.

[0061] In step S170, the calibration unit 1038 performs calibration of the sensing devices in the direction for which the extrinsic calibration parameter is selected in step S160 using the extrinsic calibration parameter, i.e., the feature point data of the target object. In this processing, the sensing devices can be calibrated to minimize the error found by the processing like steps S40-S50 in figure 8.

[0062] In step S180, the decision is made if all objects selected in step S110 have been selected as target objects in step S120. In case that at least one object has not been selected in step S120, then the process returns to step S120 and, after selecting any remaining object in step S120 as a target object, the following steps will be executed. On the other hand, in case that all objects have been selected in step S120, the processing of figures 8 and 9 is finished.

[0063] According to an embodiment of the present invention explained above, the following operations and effects are obtained.

[0064] (1) The information processing device 103 determines extrinsic calibration parameter of sensing devices 101a, 101b, 102a and 102b mounted on the vehicle 100 which is a transportation apparatus. The information processing device 103 comprises a data obtaining unit 1031, an object labeling unit 1033, a feature point data extraction unit 1035, a distribution calculation unit 1036, and an extrinsic calibration parameter determination unit 1037. The data obtaining unit 1031 obtains multimodal sensor data (step S10) including image data 40 of natural scene 30 containing various objects and point cloud data of distances from the vehicle 100 to the objects 31a, 32, 33 and 34 detected by the sensing devices in a direction where the vehicle 100 is moving. The object labeling unit 1033 labels each of the objects 31a, 32, 33 and 34 in the image data 40 (step S100) using semantic segmentation. The feature point data extraction unit 1035 extracts feature point data of labeled objects 31a, 32, 33 and 34 (step S130), which are labeled by the object labeling unit 1033, from the point cloud data. The distribution calculation unit 1036 calculates distributions of eigen values of the feature point data of the labeled objects 31a, 32, 33 and 34 (step S140) in each direction of a coordinate system 110 defined for the vehicle 100. The extrinsic calibration parameter determination unit 1037 determines the extrinsic calibration parameter (step S160) by selecting an object among the labeled objects 31a, 32, 33 and 34 for calibration in at least one rotational alignment of the sensing devices based on the distributions calculated by the distribution calculation unit 1036. By employing this configuration, it is possible to improve rotational calibration accuracy of sensing devices.

[0065] (2) The extrinsic calibration parameter determination unit 1037 calculates calibration confidence level for each of the labeled objects 31a, 32, 33 and 34 in each direction of the coordinate system 110 based on the distribution for the given labeled object in the given direction (step S140) and, if the calibration confidence level of any of the labeled objects in any of the directions is higher than a predetermined level (step S150: yes), selects the labeled object as an object for calibration in the rotational alignment along the direction (step S160). By this processing, the object for calibration can be selected properly in each rotational alignment.

[0066] (3) The information processing device 103 further comprises an object selection unit 1034 that selects the nearest one from the labeled objects 31a, 31b and 31c in the image data 40 for each kind of labels (step S110). The feature point data extraction unit 1035 extracts the feature point data for each of the labeled objects selected by the object selection unit 1034. By employing this configuration, it is possible to select the most suitable object for calibration among a plurality of objects with an identical label.

[0067] (4) The object labeling unit 1033 labels each of the objects 31a, 31b, 31c, 32, 33 and 34 in the image data 40 as any one of predetermined objects including at least an electric pole and a railway track. By this processing, the objects observed from the vehicle 100 moving on the railway track 10 can be labeled correctly.

[0068] (5) The object labeling unit 1033 may utilize the semantic segmentation by machine learning using pre-trained model. If so, it is possible to label the objects with more accuracy.

[0069] (6) The feature point data extraction unit 1035 can perform principal component analysis (PCA) on the point cloud data corresponding to each of the labeled objects and, based on the result of principal component analysis, extract the feature point data. By doing this, the principal components of the point cloud data can be identified more precisely to extract the feature point data.

[0070] The embodiment of the present invention described herein can provide extrinsic sensor calibration using the natural scene 30. In adverse weather conditions, the multimodal sensor data obtained by the data obtaining unit 1031 from the sensing devices can be improved or compensated through a combination of multi-sensor fusion, robust feature detection and tracking, synthetic data generation, data augmentation, robust calibration algorithms, and integration with environmental models trained with machine learning. These techniques can help to compensate for the effects of adverse weather conditions and improve the accuracy and reliability of extrinsic sensor calibration in challenging environments.

[0071] The embodiments and variants explained above are only examples; the present invention is not to be considered as being limited by the details thereof. Provided that the essential characteristics of the present invention are retained, other implementations are also included within the scope of the present invention.

[0072] 100: vehicle 101a, 101b, 102a, 102b: sensing device 103: information processing device 110: coordinate system 1031: data obtaining unit 1032: calibration judgement unit 1033: object labeling unit 1034: object selection unit 1035: feature point data extraction unit 1036: distribution calculation unit 1037: extrinsic calibration parameter determination unit 1038: calibration unit

Claims

1. An information processing device for determining extrinsic calibration parameter of sensing devices mounted on a transportation apparatus, comprising: a data obtaining unit that obtains multimodal sensor data including image data of natural scene containing various objects and point cloud data of distances from the transportation apparatus to the objects detected by the sensing devices in a direction where the transportation apparatus is moving; an object labeling unit that labels each of the objects in the image data using semantic segmentation; a feature point data extraction unit that extracts feature point data of labeled objects, which are labeled by the object labeling unit, from the point cloud data; a distribution calculation unit that calculates distributions of eigen values of the feature point data of the labeled objects in each direction of a coordinate system defined for the transportation apparatus; and an extrinsic calibration parameter determination unit that determines the extrinsic calibration parameter by selecting an object among the labeled objects for calibration in at least one rotational alignment of the sensing devices based on the distributions calculated by the distribution calculation unit.

2. An information processing device according to Claim 1, wherein the extrinsic calibration parameter determination unit calculates calibration confidence level for each of the labeled objects in each direction of the coordinate system based on the distribution for the given labeled object in the given direction and, if the calibration confidence level of any of the labeled objects in any of the directions is higher than a predetermined level, selects the labeled object as an object for calibration in the rotational alignment along the direction.

3. An information processing device according to Claim 1, further comprising an object selection unit that selects the nearest one from the labeled objects in the image data for each kind of labels, wherein the feature point data extraction unit extracts the feature point data for each of the labeled objects selected by the object selection unit.

4. An information processing device according to Claim 1, wherein the object labeling unit labels each of the objects in the image data as any one of predetermined objects including at least an electric pole and a railway track.

5. An information processing device according to Claim 1, wherein the object labeling unit utilizes the semantic segmentation by machine learning using pre-trained model.

6. An information processing device according to Claim 1, wherein the feature point data extraction unit performs principal component analysis on the point cloud data corresponding to each of the labeled objects and, based on the result of principal component analysis, extracts the feature point data.

7. An information processing device according to Claim 1, wherein the data obtaining unit compensates the multimodal sensor data obtained from the sensing devices.

8. A calibration method comprising: obtaining multimodal sensor data including image data of natural scene containing various objects and point cloud data of distances from a transportation apparatus to the objects detected by the sensing devices mounted on the transportation apparatus in a direction where the transportation apparatus is moving; labeling each of the objects in the image data using semantic segmentation; extracting feature point data of the labeled objects from the point cloud data; calculating distributions of eigen values of the feature point data of the labeled objects in each direction of a coordinate system defined for the transportation apparatus; selecting an object among the labeled objects for calibration in at least one rotational alignment of the sensing devices based on the calculated distributions; and performing calibration of the sensing devices in the rotational alignment using the selected object.

Images