Information processing device and calibration method

Abstract

In the information processing device 103, the data acquisition unit 1031 obtains multimodal sensor data including image data of a natural landscape containing various objects and point cloud data of the distance from the vehicle to the objects in the direction the vehicle is moving. The object labeling unit 1033 labels each of the objects in the image data using semantic segmentation. The feature point data extraction unit 1035 extracts feature point data of the labeled objects. The distribution calculation unit 1036 calculates the distribution of eigenvalues ​​of the feature point data of the labeled objects in each direction. The external calibration parameter determination unit 1037 determines the external calibration parameters by selecting objects from the labeled objects for calibration in at least one rotation alignment of the detection device, based on the distribution calculated by the distribution calculation unit 1036.

Description

Technical Field

[0001] The present invention relates to an information processing device and a calibration method for calibrating a detection device mounted on a transport device.

Background Art

[0002] In order to realize the unmanned operation of a transport device such as a train, a highly reliable forward monitoring sensor is required. In order to ensure the high reliability of the monitoring result, sensor fusion that integrates the detection results of a plurality of detection devices such as a camera, light detection and ranging (LiDAR), and an inertial measurement unit (IMU) may be used. Each detection device has its own coordinate system, and accurate calibration of each sensor position and attitude is required for proper integration.

[0003] For example, LiDAR may use one coordinate system, and a camera may use another coordinate system. If the coordinate systems used by two different sensors are not calibrated together, any sensor fusion process that combines data from the two sensors is likely to be inaccurate. Furthermore, the calibration parameters of various sensors of an autonomous transport device drift over time. It is necessary to calculate the relative pose between the mounted sensors and process the data in a common frame. Therefore, external calibration is used to calculate the relative pose of the sensors that improves the data consistency between them.

[0004] Conventional techniques for calibration require manual processing by experts, so it is necessary to provide an autonomous transport device equipped with a plurality of detection devices to an expert for calibration of these detection devices.

[0005] As technology advances, automated calibration methods have been introduced in recent years. The most common method involves displaying an object with a predefined shape at a known location, and this method is known as target-based external sensor calibration. However, this target-based calibration method, which introduces the manual calibration of objects within existing infrastructure, is cumbersome, time-consuming, and expensive due to the additional infrastructure required. In addition, robust long-term autonomy necessitates continuous evaluation of calibration accuracy, which makes the use of human-made objects impractical.

[0006] Due to the aforementioned shortcomings of target-based calibration, numerous methods have been proposed to achieve automated, targetless external calibration. For example, this problem can be solved by extracting geometric features from natural landscapes and using them as the calibration target.

[0007] In external sensor calibration methods or systems, the main challenge is achieving calibration accuracy, more specifically, rotational calibration accuracy. Translational calibration accuracy is generally easier to achieve than rotational calibration accuracy due to the simpler nature of linear motion, the ease of using a single-axis reference, the reduced influence of external factors, and the simpler identification and correction of errors. Therefore, the main challenge is accurately calculating the rotational calibration parameters. The accuracy of features extracted from natural landscapes depends on the regularity of the environment and the performance of the feature extraction algorithm.

[0008] The following Patent Document 1 is known as prior art of the present invention. Patent Document 1 discloses a method and system for determining external calibration parameters for at least one pair of detection devices mounted on a transport device. The method obtains image data acquired by an image generation detection device and 3D point cloud data from LiDAR, an image at a specific pose is selected, a laser reflection image is generated based on a portion of the point cloud corresponding to the pose, and a metric measuring alignment between the selected image and the laser reflection image is calculated. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] U.S. Patent No. 9,875,557 [Overview of the project] [Problems that the invention aims to solve]

[0010] In Patent Document 1, the laser reflection image is generated based on a portion of the point cloud corresponding to the pose. The main drawback of this method is that it may not be possible to accurately capture the entire landscape. In addition, some objects or details may be omitted, and using only a portion of the point cloud results in a loss of information. As a result, the detection device may be calibrated with lower accuracy. This can be particularly problematic in applications requiring a high level of accuracy, such as autonomous vehicles.

[0011] The present invention was devised in consideration of the problems described above, and its main objective is to improve the rotational calibration accuracy of the sensing device. [Means for solving the problem]

[0012] The information processing device according to the present invention, which determines external calibration parameters for a detection device implemented on a transport device, comprises: a data acquisition unit that obtains multimodal sensor data including image data of a natural landscape including various objects and point cloud data of the distance from the transport device to an object detected by the detection device in the direction in which the transport device is moving; an object labeling unit that labels each of the objects in the image data using semantic segmentation; a feature point data extraction unit that extracts feature point data of the labeled objects labeled by the object labeling unit from the point cloud data; a distribution calculation unit that calculates the distribution of eigenvalues ​​of the feature point data of the labeled objects in each direction of the coordinate system defined for the transport device; and an external calibration parameter determination unit that determines external calibration parameters by selecting an object from the labeled objects for calibration in at least one rotational alignment of the detection device based on the distribution calculated by the distribution calculation unit. The calibration method according to the present invention includes the steps of: obtaining multimodal sensor data including image data of a natural landscape including various objects and point cloud data of the distance from the transport device to an object detected by a detection device mounted on the transport device in the direction in which the transport device is moving; labeling each of the objects in the image data using semantic segmentation; extracting feature point data of the labeled objects from the point cloud data; calculating the distribution of eigenvalues ​​of the feature point data of the labeled objects in each direction of the coordinate system defined for the transport device; selecting an object from the labeled objects for calibration in at least one rotational alignment of the detection device based on the calculated distribution; and performing calibration of the detection device in rotational alignment using the selected object. [Effects of the Invention]

[0013] According to the present invention, it is possible to improve the rotational calibration accuracy of the detection device.

[0014] These objectives, effects, goals, and features of the present invention, as well as other objectives, effects, goals, and features, will become clear from the following discussion of the specification in conjunction with the drawings. [Brief explanation of the drawing]

[0015] [Figure 1] This is a schematic diagram of a transport device in which a plurality of detection devices and information processing devices are implemented according to an embodiment of the present invention. [Figure 2] Examples of natural landscapes observed from a vehicle are shown. [Figure 3] This is an example of image data generated by acquiring natural landscapes. [Figure 4] This diagram illustrates the process performed on objects labeled as "utility poles" in image data. [Figure 5] This diagram illustrates the process performed on objects labeled as "railway tracks" in image data. [Figure 6] This diagram illustrates the process performed on objects labeled as "ground" in image data. [Figure 7] This shows a functional block diagram of an information processing device according to an embodiment of the present invention. [Figure 8] This is the first section of a flowchart illustrating the processes performed by an information processing device. [Figure 9] This is the second section of the flowchart illustrating the processes performed by the information processing device. [Modes for carrying out the invention]

[0016] In the following detailed description, various features and functions of the proposed invention will be described in detail with reference to the attached figures. The exemplary systems, functions, and methods described herein are not limiting. It will be readily apparent that several aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are considered herein.

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0018] FIG. 1 is a schematic view of a transport device in which a plurality of detection devices and an information processing device are implemented according to an embodiment of the present invention. As shown in FIG. 1, on a vehicle 100 which is a transport device traveling on a track 10 toward direction 20, detection devices 101a, 101b, 102a, and 102b are mounted.

[0019] The detection devices 101a and 101b can each acquire an image of the direction 20 as viewed from the vehicle 100. A camera may be used as the detection devices 101a and 101b. The detection devices 102a and 102b can each scan an object located in front of the vehicle 100 in the direction 20 and detect the distance from the vehicle 100 to the scanned object. A 2D or 3D LiDAR may be used as the detection devices 102a and 102b. Note that in FIG. 1, a pair of detection devices 101a and 101b provided in parallel toward the direction 20 are mounted extremely close to another pair of detection devices 102a and 102b provided in parallel toward the direction 20, but the arrangement and number of the detection devices mounted on the vehicle 100 are not limited to this example.

[0020] The vehicle 100 also has an information processing device 103, which can communicate with the detection devices 101a, 101b, 102a, and 102b (via a wireless or wired communication interface) and can store and process the data of all the detection devices. A coordinate system 110 is defined for the vehicle 100. As shown in FIG. 1, the coordinate system 110 has x, y, and z dimensional axes, where x is the longitudinal forward direction, y is the lateral direction, z is the upward direction, and the origin of these axes is at the center lower part of the vehicle 100 body.

[0021] Figure 2 shows an example of a natural landscape observed from the vehicle 100 shown in Figure 1. When the vehicle 100 is moving along the railway track 10 in direction 20, the natural landscape 30 is observed from the vehicle 100 by the detection devices 101a, 101b, 102a, and 102b shown in Figure 1. Image data of the natural landscape 30, including various objects, is acquired by the detection devices 101a and 101b. Point cloud data, including multiple scanned points indicating the distance from the vehicle 100 to the objects in the natural landscape 30, is detected by the detection devices 102a and 102b. This data is transmitted as multimodal sensor data from each detection device to the information processing device 103.

[0022] Figure 3 is an example of image data generated by acquiring the natural landscape 30 shown in Figure 2. The image data 40 showing the natural landscape 30 observed by the detection device 101a or 101b includes various objects. For example, object 33 is the railway track 10 on which the vehicle 100 is traveling, objects 31a, 31b, and 31c are utility poles located along the railway track 10, object 32 is a bush located beside the railway track 10, and object 34 is the ground. Each of these objects is labeled by the information processing device 103 using semantic segmentation with respect to the natural landscape 30. The information processing device 103 may perform semantic segmentation, for example, by learning using a pre-trained model.

[0023] Semantic segmentation is a well-known technique that links each pixel in an image to a classification name. Pixel labeling in image data 40 may be performed by assigning a classification label to each pixel based on learned features. This allows the model to identify objects in the natural landscape 30 and distinguish them from the background. By performing this process, objects 31a, 31b, and 31c are labeled "utility poles," object 32 is labeled "bushes," object 33 is labeled "railway tracks," and object 34 is labeled "ground." These labeled objects become potential candidates for calibration.

[0024] There are several existing algorithms for labeling in semantic segmentation. In this embodiment, any type of labeling method in semantic segmentation can be used, for example, for synthetic labeling or interactive labeling using a pre-trained model. A set of instructions in the form of a semantic segmentation algorithm is processed in the information processing device 103.

[0025] Figure 4 is a diagram illustrating the processing performed on objects 31a, 31b, and 31c labeled as "utility poles" in image data 40. In Figure 4(a), the binary image 41 shows an example of a binary image of objects 31a to 31c extracted from image data 40. This binary image 41 visualizes the complex topological behavior and smooth boundaries of objects 31a to 31c. By utilizing the binary image 41 for visualization, it is possible to see that all the shapes of objects 31a to 31c labeled as "utility poles" are thin-walled cylindrical shapes with a long z-distribution, as defined in the coordinate system 110. Among them, object 31a is the longest and object 31c is the shortest, because the distance from the vehicle 100 is closest for object 31a and furthest for object 31c.

[0026] In Figure 4(b), Graph 51 depicts the normal distribution curves of the eigenvalues ​​of the feature point data of the target 31a in each direction of the coordinate system 110. The feature point data of the target 31a, which indicates the distance from the vehicle 100 to each point on the target 31a, is generated by extracting a dataset corresponding to the region of the target 31a from the point cloud data obtained from the detection device 102a or 102b. Graph 51 includes distribution curves 511a, 511b, and 511c, respectively, which show the normal distribution of the eigenvalues ​​of the feature point data in the x-axis, y-axis, and z-axis directions of the coordinate system 110. The shape of the eigenvectors 512 indicates the eigenvalues ​​of the target 31a in the x, y, and z directions.

[0027] Because the shape of object 31a has a relatively long distribution in the z direction compared to the x and y directions, the distribution curve 511c has a higher value than the distribution curves 511a and 511b. In other words, because object 31a has a higher degree of dispersion in the z direction, the rotational alignment reliability along the z direction (in pitch rotation) is low. Therefore, by using object 31a as the external calibration parameter along the x and y directions (in roll and yaw rotation), the sensing device can be calibrated with high accuracy.

[0028] Figure 5 is a diagram illustrating the processing performed on the object 33 labeled as "railway track" in the image data 40. In Figure 5(a), the binary image 42 shows an example of a binary image of the object 33 extracted from the image data 40. This binary image 42 visualizes the complex and topological behavior and smooth boundaries of the object 33. By utilizing the binary image 42 for visualization, the shape of the object 33 labeled as "railway track" has a long x-distribution, as defined in the coordinate system 110.

[0029] In Figure 5(b), Graph 52 depicts the normal distribution curves of the eigenvalues ​​of the feature point data of the target 33 in each direction of the coordinate system 110. The feature point data of the target 33, which indicates the distance from the vehicle 100 to each point on the target 33, is generated by extracting a dataset corresponding to the region of the target 33 from the point cloud data obtained from the detection device 102a or 102b. Graph 52 includes distribution curves 521a, 521b, and 521c, respectively, which show the normal distribution of the eigenvalues ​​of the feature point data in the x, y, and z directions of the coordinate system 110. The shape of the eigenvectors 522 indicates the eigenvalues ​​of the target 33 in the x, y, and z directions.

[0030] Because the shape of object 33 has a relatively long distribution in the x-direction compared to the y and z-directions, distribution curve 521a has a higher value than distribution curves 521b and 521c. In other words, because object 33 has a higher degree of dispersion in the x-direction, the rotational alignment reliability around the x-direction (in roll rotation) is low. Therefore, by using object 33 as an external calibration parameter along the y and z-directions (in yaw and pitch rotation), the sensing device can be calibrated with high accuracy.

[0031] Figure 6 is a diagram illustrating the processing performed on the object 34 labeled as "ground" in the image data 40. In Figure 6(a), the binary image 43 shows an example of a binary image of the object 34 extracted from the image data 40. This binary image 43 visualizes the complex and topological behavior and smooth boundaries of the object 34. By utilizing the binary image 43, it can be visualized that the shape of the object 34 labeled as "ground" has long x and y distributions as defined in the coordinate system 110.

[0032] In Figure 6(b), Graph 53 depicts the normal distribution curves of the eigenvalues ​​of the feature point data of the target 34 in each direction of the coordinate system 110. The feature point data of the target 34, which indicates the distance from the vehicle 100 to each point on the target 34, is generated by extracting a dataset corresponding to the region of the target 34 from the point cloud data obtained from the detection device 102a or 102b. Graph 53 includes distribution curves 531a, 531b, and 531c, respectively, which show the normal distribution of the eigenvalues ​​of the feature point data in the x, y, and z directions of the coordinate system 110. The shape of the eigenvectors 532 indicates the eigenvalues ​​of the target 34 in the x, y, and z directions.

[0033] Because the shape of object 34 has a relatively long distribution in the x and y directions relative to the z direction, distribution curves 531a and 531b have higher values ​​than distribution curve 531c. In other words, because object 34 has a higher degree of dispersion in the x and y directions, the rotational alignment reliability is low (in roll and yaw rotation) around the x direction and along the y direction. Therefore, by using object 34 as an external calibration parameter along the z direction (in pitch rotation), the sensing device can be calibrated with high accuracy.

[0034] In the processes described with reference to Figures 4 to 6, the Principal Component Analysis (PCA) algorithm may be used to extract feature point data from the point cloud data corresponding to subjects 31a, 33, and 34. PCA works by identifying the principal components of the point cloud data. These principal components are linear combinations of the original variables that account for the largest variation in the dataset. By projecting the data onto these principal components, it is possible to reduce the dimensionality of the feature point data while retaining most of the original variables.

[0035] Figure 7 shows a functional block diagram of an information processing device 103 according to an embodiment of the present invention. Functionally, the information processing device 103 includes a data acquisition unit 1031, a calibration judgment unit 1032, a target labeling unit 1033, a target selection unit 1034, a feature point data extraction unit 1035, a distribution calculation unit 1036, an external calibration parameter determination unit 1037, and a calibration unit 1038. The information processing device 103 may realize these functional blocks by executing a predetermined program on a CPU.

[0036] The data acquisition unit 1031 obtains multimodal sensor data from the detection devices 101a, 101b, 102a, and 102b. The multimodal sensor data, which includes image data of the natural landscape 30 including the objects 31a, 31b, 31c, 32, 33, and 34, and point cloud data of the distance from the vehicle 100 to these objects, is transmitted from the detection devices to the information processing device 103, allowing the data acquisition unit 1031 to obtain this data.

[0037] The calibration determination unit 1032 determines whether or not calibration is required for the detection device. If calibration is deemed necessary, the following processes are performed by other units.

[0038] The object labeling unit 1033 retrieves image data from the multimodal sensor data obtained by the data acquisition unit 1031 and labels each object in the image data using semantic segmentation. Through this process, the objects are labeled as "utility pole," "railway tracks," "bushes," and "ground," respectively.

[0039] The target selection unit 1034 selects the closest target from among multiple targets that have been labeled with the same label in the image data by the target labeling unit 1033. In this process, the distance to the target can be determined based on the point cloud data included in the multimodal sensor data obtained by the data acquisition unit 1031. Through this process, target 31a is selected from targets 31a to 31c that are labeled as "utility pole". The selected target 31a, as well as targets 32, 33, and 34 that are labeled as "bush", "railway tracks", and "ground", respectively, are designated as targets for the following process.

[0040] The feature point data extraction unit 1035 extracts feature point data for each of the targets 31a, 32, 33, and 34. In this process, the dataset corresponding to the region of each target is extracted from the point cloud data included in the multimodal sensor data obtained by the data acquisition unit 1031, and the extracted dataset is designated as the feature point data for each target. As mentioned above, the PCA algorithm may be used to identify the principal components of the point cloud data.

[0041] The distribution calculation unit 1036 calculates the distribution of eigenvalues ​​of the feature point data extracted by the feature point data extraction unit 1035 for each object in each direction of the coordinate system 110. Through this process, as shown in Figure 4(b), the distribution curves 511a, 511b, and 511c and eigenvectors 512 shown in graph 51 are calculated for object 31a in the x, y, and z directions of the coordinate system 110. Similarly, as shown in Figure 5(b), the distribution curves 521a, 521b, and 521c and eigenvectors 522 shown in graph 52 are calculated for object 33 in the x, y, and z directions of the coordinate system 110. Furthermore, as shown in Figure 6(b), the distribution curves 531a, 531b, and 531c and eigenvectors 532 shown in graph 53 are calculated for object 34 in the x, y, and z directions of the coordinate system 110. Furthermore, the distribution curves and eigenvectors for subject 32 are omitted from the illustration, and they may have a certain value (size) in each direction.

[0042] The external calibration parameter determination unit 1037 determines the external calibration parameters for each object based on the distribution of eigenvalues ​​of the feature point data calculated by the distribution calculation unit 1036. In this process, first, the calibration confidence is calculated for each object in each direction from the distribution of eigenvalues. Specifically, the higher the distribution of eigenvalues ​​in a given direction, the lower the confidence calculated in that direction. Next, the calculated calibration confidence for each object in each direction is compared with a predetermined level, and if the calibration confidence of any object in any direction is higher than the predetermined level, the object is selected as the object to be calibrated in rotational alignment along that direction. Through this process, as shown in Figure 4(b), the calibration confidence of the distribution curves 511a and 511b, which have low eigenvalues, is determined to be higher than the predetermined level, and therefore object 31a is selected as the object to be calibrated in roll and yaw rotation. Similarly, as shown in Figure 5(b), when the calibration confidence of distribution curves 521b and 521c, which have low eigenvalues, is determined to be higher than a predetermined level, object 33 is selected as the target for calibration in yaw and pitch rotation. Also, as shown in Figure 6(b), when the calibration confidence of distribution curve 531c, which has low eigenvalues, is determined to be higher than a predetermined level, object 34 is selected as the target for calibration in pitch rotation.

[0043] In summary, when object 31a is used as the object to be externally calibrated, better calibration accuracy for roll and low rotational calibration can be achieved. In addition, when object 33 is used as the object to be externally calibrated, better calibration accuracy for yaw and pitch rotational calibration can be achieved. Furthermore, when object 34 is used as the object to be externally calibrated, better calibration accuracy for pitch rotational calibration can be achieved. Finally, the external calibration parameters for the sensing devices 101a, 101b, 102a, and 102b are determined by the external calibration parameter determination unit 1037, which compares and selects the object best suited for a specific rotational alignment calibration that achieves maximum accuracy.

[0044] The calibration unit 1038 calibrates the detection devices 101a, 101b, 102a, and 102b using the external calibration parameters determined by the external calibration parameter determination unit 1037. In this process, for example, one of the detection devices 101a, 101b, 102a, and 102b is selected as the base sensor, and the other detection devices are calibrated to match the base sensor using the objects selected for calibration in each rotational direction.

[0045] Figures 8 and 9 are flowcharts showing the processes performed by the information processing device 103. The steps in the flowcharts shown in Figures 8 and 9 are illustrative, and some of them may be omitted and / or rearranged. Furthermore, the processes can be implemented using any preferred programming language and data structure.

[0046] The entire flowchart is explained in two sections. The first section, including steps S10-S70 shown in Figure 8, describes the process of determining whether external calibration is necessary or if it can be done using a simple algorithm used in existing literature. The second section, including steps S100-S180 shown in Figure 9, describes the process of determining the external calibration parameters and performing the calibration. The entire flowchart, combining the first and second sections, represents an efficient scenario for the proposed algorithm.

[0047] In step S10, the data acquisition unit 1031 obtains multimodal sensor data from the detection devices 101a, 101b, 102a, and 102b.

[0048] In step S20, the calibration decision unit 1032 selects one of the detection devices 101a, 101b, 102a, and 102b, such as detection device 101a, as the base sensor for external calibration.

[0049] In step S30, the calibration judgment unit 1032 selects a target region (ROI), derives directional feature points from each detection device, and derives directional feature points for all detection devices. By performing this process, the directional features in each direction are determined for each of the detection devices 101a, 101b, 102a, and 102b.

[0050] In step S40, the calibration decision unit 1032 uses the geometric coordinate relationships between the frames of these detection devices to project the directional feature points of detection devices 101b, 102a, and 102b that were not selected in step S20 onto the detection device 101a selected as the base sensor. Since each detection device has its own reference frame, it is necessary to compare identical feature points detected in a certain reference frame, i.e., the base sensor. Once all feature points have been converted to the same reference frame, the process proceeds to the next step S50.

[0051] In step S50, the calibration judgment unit 1032 finds the error between all projected feature points and the feature points of the base sensor.

[0052] In step S60, it is checked whether the error obtained in step S50 is greater than a given threshold. If the error is greater than the threshold, it is determined that calibration is necessary, and the process then proceeds to step S100 in the second section of Figure 9. If the error is not greater than the threshold, the process proceeds to step S70.

[0053] In step S70, it is determined that calibration is not necessary. The process then returns to step S20, and further steps are executed again.

[0054] In step S100, the target labeling unit 1033 labels the target in the image data obtained in step S10 by using a semantic segmentation algorithm.

[0055] In step S110, the object selection unit 1034 selects the closest semantically labeled object from among objects with the same semantic label as the potential calibration object. If there are multiple object groups, each with the same label, the closest object in each group is selected in step S110. If there is only one object with a particular label, the processing in step S110 is omitted for that object. This process results in the selection of one or more objects, each with a different label, for subsequent processes.

[0056] In process S120, the target selection unit 1034 selects one of the targets selected in process S110. The target currently selected in process S120 is designated as the target for the subsequent processes S130 to S170.

[0057] In step S130, the feature point data extraction unit 1035 extracts feature point data from the point cloud data obtained in step S10 for the object selected in step S120. In this process, the PCA algorithm may be used to identify the principal components of the point cloud data.

[0058] In step S140, the distribution calculation unit 1036 calculates the normal distribution of the eigenvalues ​​of the feature point data extracted in step S130 in each direction. Based on the calculated distribution, the external calibration parameter determination unit 1037 calculates the confidence level of the calibration in each direction for the current object selected in step S120. This process is carried out as described with reference to Figures 4, 5, and 6.

[0059] In step S150, the external calibration parameter determination unit 1037 determines, based on the calibration confidence calculated in step S140, whether there is a direction among all the objects selected in step S110 that has the highest confidence for the current object. If the calibration confidence for the current object is higher than a predetermined level and is the highest among all objects in at least one direction, the external calibration parameter determination unit 1037 determines "yes" in step S150 and proceeds to step S160. On the other hand, if the calibration confidence for the current object is below a predetermined level and is not the highest in any direction, the external calibration parameter determination unit 1037 determines "no" in step S150 and proceeds to step S180.

[0060] In step S160, the external calibration parameter determination unit 1037 selects the current object as the external calibration parameter. In this process, the feature point data of the current object is selected as the external calibration parameter for the direction determined to be maximum in step S150.

[0061] In step S170, the calibration unit 1038 uses external calibration parameters, i.e., feature point data of the object, to calibrate the detection device in the direction in which the external calibration parameters were selected in step S160. In this process, the detection device can be calibrated to minimize the errors found through processes such as those shown in steps S40-S50 in Figure 8.

[0062] In step S180, it is determined whether all the objects selected in step S110 have been selected as objects in step S120. If at least one object has not been selected in step S120, the process returns to step S120, where one of the remaining objects is selected as an object, and then the subsequent steps are executed. On the other hand, if all objects have been selected in step S120, the processes shown in Figures 8 and 9 are completed.

[0063] According to the embodiments of the present invention described above, the following actions and effects can be obtained.

[0064] (1) The information processing device 103 determines the external calibration parameters for the detection devices 101a, 101b, 102a, and 102b that are mounted on the vehicle 100, which is a transport device. The information processing device 103 comprises a data acquisition unit 1031, an object labeling unit 1033, a feature point data extraction unit 1035, a distribution calculation unit 1036, and an external calibration parameter determination unit 1037. The data acquisition unit 1031 obtains multimodal sensor data including image data 40 of a natural landscape 30 including various objects and point cloud data of the distance from the vehicle 100 to objects 31a, 32, 33, and 34 detected by the detection devices in the direction in which the vehicle 100 is moving (step S10). The object labeling unit 1033 labels each of the objects 31a, 32, 33, and 34 in the image data 40 using semantic segmentation (step S100). The feature point data extraction unit 1035 extracts feature point data of labeled targets 31a, 32, 33, and 34, which have been labeled by the target labeling unit 1033, from the point cloud data (step S130). The distribution calculation unit 1036 calculates the distribution of eigenvalues ​​of the feature point data of the labeled targets 31a, 32, 33, and 34 in each direction of the coordinate system 110 defined for the vehicle 100 (step S140). The external calibration parameter determination unit 1037 determines the external calibration parameters by selecting targets from the labeled targets 31a, 32, 33, and 34 for calibration in at least one rotation alignment of the detection device, based on the distribution calculated by the distribution calculation unit 1036 (step S160). By adopting this configuration, the rotation calibration accuracy of the detection device can be improved.

[0065] (2) The external calibration parameter determination unit 1037 calculates the calibration confidence level for each of the labeled objects 31a, 32, 33, and 34 in each direction of the coordinate system 110 based on the distribution of a given labeled object in a given direction (step S140). If the calibration confidence level of any of the labeled objects in any of the directions is higher than a predetermined level (step S150: yes), the unit selects the labeled object as the object to be calibrated in the rotational alignment along the direction (step S160). This process makes it possible to appropriately select the object to be calibrated in each rotational alignment.

[0066] (3) The information processing device 103 further includes an object selection unit 1034 that selects the closest labeled object 31a, 31b, and 31c in the image data 40 for each type of label (step S110). The feature point data extraction unit 1035 extracts feature point data for each of the labeled objects selected by the object selection unit 1034. By adopting this configuration, it is possible to select the most suitable object for calibration from multiple objects having the same label.

[0067] (4) The target labeling unit 1033 labels each of the targets 31a, 31b, 31c, 32, 33, and 34 in the image data 40 as one of the predetermined targets, which include at least a utility pole and the railway tracks. This process makes it possible to accurately label targets observed from a vehicle 100 moving along the railway tracks 10.

[0068] (5) The object labeling unit 1033 may utilize semantic segmentation by machine learning using a pre-trained model. If so, it is possible to label objects with higher accuracy.

[0069] (6) The feature point data extraction unit 1035 performs principal component analysis (PCA) on the point cloud data corresponding to each of the labeled objects, and can extract feature point data based on the results of the principal component analysis. By doing so, the principal components of the point cloud data can be identified more accurately, and feature point data can be extracted.

[0070] Embodiments of the present invention described herein provide external sensor calibration using a natural landscape 30. Under adverse weather conditions, multimodal sensor data obtained from the sensing device by the data acquisition unit 1031 can be enhanced or compensated for through a combination of multiplex sensor fusion, robust characteristic detection and tracking, synthetic data generation, data augmentation, robust calibration algorithms, and integration with a machine learning-trained environmental model. These techniques help compensate for the effects of adverse weather conditions and can improve the accuracy and reliability of external sensor calibration in challenging environments.

[0071] The embodiments and modifications described above are merely illustrative, and the present invention is not to be considered as being limited by its details. However, other implementations are also included within the scope of the invention, provided that the essential features of the invention are retained. [Explanation of symbols]

[0072] 100 vehicles 101a, 101b, 102a, 102b detection devices 103 Information Processing Devices 110 Coordinate System 1031 Data Acquisition Unit 1032 Calibration Judgment Unit 1033 Target labeling unit 1034 Target Selection Unit 1035 Feature Point Data Extraction Unit 1036 Distribution Calculation Unit 1037 External Calibration Parameter Determination Unit 1038 Calibration Unit

Claims

1. An information processing device that determines the external calibration parameters of a detection device implemented on a transport device, A data acquisition unit that obtains multimodal sensor data including image data of a natural landscape including various objects and point cloud data of the distance from the transport device to the object detected by the detection device in the direction in which the transport device is moving, A target labeling unit that labels each of the targets in the image data using semantic segmentation, A feature point data extraction unit extracts feature point data of a labeled object labeled by the target labeling unit from the point cloud data, A distribution calculation unit that calculates the distribution of eigenvalues ​​of the labeled object's feature point data in each direction of the coordinate system defined for the transport device, An information processing device comprising: an external calibration parameter determination unit that determines the external calibration parameter by selecting an object from the labeled object for calibration of at least one rotational alignment of the detection device based on the distribution calculated by the distribution calculation unit.

2. The information processing device according to claim 1, wherein the external calibration parameter determination unit calculates a calibration confidence level for each of the labeled objects in each direction of the coordinate system based on the distribution of the given labeled objects in the given direction, and if the calibration confidence level of any of the labeled objects in any of the directions is higher than a predetermined level, the labeled object is selected as the object to be calibrated in the rotational alignment along the direction.

3. For each type of label, the system further includes a target selection unit that selects the closest one from the labeled targets in the image data. The information processing device according to claim 1, wherein the feature point data extraction unit extracts the feature point data for each of the labeled targets selected by the target selection unit.

4. The information processing device according to claim 1, wherein the target labeling unit labels each of the targets in the image data as at least one of a predetermined set of targets including utility poles and railway tracks.

5. The information processing device according to claim 1, wherein the target labeling unit utilizes semantic segmentation by machine learning using a pre-trained model.

6. The information processing device according to claim 1, wherein the feature point data extraction unit performs principal component analysis on the point cloud data corresponding to each of the labeled targets, and extracts the feature point data based on the results of the principal component analysis.

7. The data acquisition unit compensates for the multimodal sensor data obtained from the detection device, as per claim 1.

8. A step of obtaining multimodal sensor data including image data of a natural landscape including various objects and point cloud data of the distance from the transport device to the object detected by a detection device mounted on the transport device in the direction in which the transport device is moving, A step of labeling each of the objects in the image data using semantic segmentation, A step of extracting feature point data of the labeled object from the point cloud data, A step of calculating the distribution of eigenvalues ​​of the feature point data of the labeled object in each direction of the coordinate system defined for the transport device, Based on the calculated distribution, the process of selecting an object from the labeled objects for calibration in at least one rotational alignment of the detection device, A calibration method comprising the step of calibrating the detection device in the rotational alignment using the selected object.

Images