Information processing program, information processing device, and information processing method

Abstract

To support providing learning data suitable for highly accurate learning.SOLUTION: An information processing program causes a computer to execute: an image acquisition step of acquiring image data; a point cloud acquisition step of acquiring point cloud data with depth information defined for each of multiple points, the point cloud data including unknown calibration parameter which is used for calibrating temporal-spatial error of at least one of temporal factor and spatial factor for the image data and uncalibrated temporal-spatial error; a calibration parameter calculation step of calculating calibration parameter for each learning data including the image data and the point cloud data; and a determination step of determining validity of the learning data used for calculating the calibration parameter, based on the calibration parameter.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing program, an information processing apparatus, and an information processing method.

Background Art

[0002] There has been proposed a technique for learning the relationship from a learning data set composed of a plurality of learning data including image data photographed from one viewpoint and depth information of each pixel, and estimating the depth information of each pixel from the image data of one viewpoint. For learning, it is necessary to use learning data calibrated so that the correspondence relationship of the temporal and spatial elements between the image data photographed by the imaging sensor and the point cloud data defining the depth information for each pixel measured by the distance measuring sensor is accurate. Therefore, a technique for calibrating learning data using calibration parameters has been disclosed.

[0003] However, in the prior art, even for learning data having temporal and spatial errors that cannot be completely calibrated using calibration parameters, calibration is performed using the calibration parameters and used for learning, and the learning accuracy may decrease. That is, in the prior art, it has been difficult to support the provision of learning data suitable for high-precision learning.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The problem to be solved by the present invention is to provide an information processing program, an information processing apparatus, and an information processing method that can provide learning data suitable for high-precision learning.

Means for Solving the Problems

[0007] The information processing program according to the embodiment causes a computer to execute an image acquisition step of acquiring image data, a point cloud acquisition step of acquiring point cloud data in which depth information is defined for each of a plurality of points where calibration parameters used to calibrate at least one of a temporal element and a spatial element with respect to the image data are unknown and the temporal-spatial error is uncalibrated, a calibration parameter calculation step of calculating the calibration parameters for each learning data including the image data and the point cloud data, and a determination step of determining the validity of the learning data used for the calculation of the calibration parameters based on the calibration parameters.

Brief Description of the Drawings

[0008] [Figure 1] Schematic diagram of an information processing system. [Figure 2A] Schematic diagram of a waveform before application of a time offset. [Figure 2B] Schematic diagram of a waveform after application of a time offset. [Figure 3] Image diagram of a reprojection error. [Figure 4A]An illustrative diagram of the position trajectory before applying calibration parameters. [Figure 4B] An illustrative diagram of the positional trajectory after applying calibration parameters. [Figure 5] A flowchart illustrating the process for determining the effectiveness of training data. [Figure 6] A flowchart illustrating the learning process. [Figure 7] Hardware configuration diagram. [Modes for carrying out the invention]

[0009] The information processing program, information processing device, and information processing method will be described in detail below with reference to the attached drawings.

[0010] Figure 1 is a schematic diagram of an example of the information processing system 1 of this embodiment.

[0011] The information processing system 1 comprises an information processing device 10, a UI (user interface) unit 12, a shooting sensor 14, and a distance measuring sensor 16. The information processing device 10, the UI unit 12, the shooting sensor 14, and the distance measuring sensor 16 are communicated together via a bus 18 or the like.

[0012] The UI unit 12 has a display function and an input function. The display function displays various types of information. The display function is, for example, a display, a projection device, etc. The input function accepts user input. The input function is, for example, a pointing device such as a mouse and touchpad, a keyboard, etc. The display function and input function may be integrated into a touch panel.

[0013] The imaging sensor 14 is a fixed-focus camera that obtains image data by capturing images from a single viewpoint. The imaging sensor 14 sequentially outputs the image data captured in chronological order to the information processing device 10.

[0014] The distance measuring sensor 16 obtains point cloud data in which depth information is defined for each of the multiple points within the distance measuring range in real space. The distance measuring sensor 16 is, for example, a laser sensor, radar sensor, millimeter-wave sensor, etc. The distance measuring sensor 16 sequentially outputs the point cloud data obtained by distance measurement in a time series to the information processing device 10.

[0015] The shooting range of the image sensor 14 and the distance measurement range of the distance measurement sensor 16 are pre-adjusted so that at least a portion of them overlap. Preferably, the shooting range of the image sensor 14 and the distance measurement range of the distance measurement sensor 16 match.

[0016] The information processing device 10 performs processes such as determining the validity of training data including image data and point cloud data, and training a learning model.

[0017] The information processing device 10 comprises a storage unit 20, a communication unit 22, and a processing unit 24. The storage unit 20, the communication unit 22, the UI unit 12, the imaging sensor 14, and the distance measuring sensor 16 are communicated with the processing unit 24.

[0018] The memory unit 20 stores various types of data. The communication unit 22 is a communication interface for communicating with external information processing devices, etc., via a network or the like.

[0019] The memory unit 20, UI unit 12, image sensor 14, and distance measuring sensor 16 may be configured to communicate with the processing unit 24 via wired or wireless means. For example, at least one of the memory unit 20, UI unit 12, image sensor 14, and distance measuring sensor 16 may be connected to the processing unit 24 via a network or the like.

[0020] Furthermore, the storage unit 20 may be located outside the information processing device 10. Alternatively, the storage unit 20 and at least one of the one or more functional units included in the processing unit 24 (described later) may be mounted on an external information processing device that is connected to the processing unit 24 via a network or the like in a way that allows communication.

[0021] The processing unit 24 performs information processing in the information processing device 10. The processing unit 24 includes an image acquisition unit 24A, a point cloud acquisition unit 24B, a calibration parameter calculation unit 24C, a determination unit 24D, an output control unit 24E, a calibration unit 24F, and a learning unit 24G.

[0022] The image acquisition unit 24A, point cloud acquisition unit 24B, calibration parameter calculation unit 24C, determination unit 24D, output control unit 24E, calibration unit 24F, and learning unit 24G are implemented by, for example, one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be implemented by a processor such as a dedicated IC, i.e., by hardware. Each of the above units may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

[0023] The image acquisition unit 24A acquires image data. The image acquisition unit 24A sequentially acquires image data that is output sequentially from the imaging sensor 14.

[0024] The point cloud acquisition unit 24B acquires point cloud data. The point cloud acquisition unit 24B sequentially acquires point cloud data output sequentially from the distance measuring sensor 16.

[0025] As described above, point cloud data is data in which depth information is defined for each of multiple points. Depth information is information that represents the distance from the distance measuring sensor 16 to a point, which is an element in real space. Each of the multiple points that make up the point cloud data corresponds to one of the pixels that make up the image data.

[0026] The point cloud data acquired by the point cloud acquisition unit 24B is point cloud data with unknown calibration parameters and uncalibrated temporal and spatial errors. In other words, the point cloud acquisition unit 24B acquires point cloud data with unknown calibration parameters and uncalibrated temporal and spatial errors.

[0027] Calibration parameters are parameters used to calibrate at least one of the temporal and spatial errors in image data. In this embodiment, calibration parameters are described as representing temporal and spatial errors.

[0028] The temporal and spatial error of the temporal element refers to the time offset, which represents the difference in acquisition timing between the image data and the point cloud data. The image data acquisition timing refers to the timing when the image acquisition unit 24A acquires the image data from the imaging sensor 14. The point cloud data acquisition timing refers to the timing when the point cloud acquisition unit 24B acquires the point cloud data from the distance measuring sensor 16. Note that the image data acquisition timing may also be the timing when the imaging sensor 14 captures the image data. In this case, the point cloud data acquisition timing may also be the timing when the distance measuring sensor 16 measures the distance of the point cloud data.

[0029] The time offset is represented by the difference in the acquisition start time and acquisition update speed of the image acquisition unit 24A and the point cloud acquisition unit 24B, respectively. The acquisition start time refers to the time when the image acquisition unit 24A and the point cloud acquisition unit 24B each started acquiring image data and point cloud data, respectively. The update speed refers to the speed at which the image acquisition unit 24A and the point cloud acquisition unit 24B each acquire image data and point cloud data, respectively.

[0030] In other words, the fact that the calibration parameters used to calibrate the temporal and spatial errors of the temporal elements are unknown means that the discrepancies in the acquisition start times and acquisition update rates of the image acquisition unit 24A and the point cloud acquisition unit 24B are unknown. Furthermore, the fact that the temporal and spatial errors of the temporal elements are uncalibrated means that the discrepancies in the acquisition start times and acquisition update rates of the image acquisition unit 24A and the point cloud acquisition unit 24B for at least one of the image data and point cloud data are uncalibrated.

[0031] The temporal spatial error of spatial elements refers to at least one of the following: the spatial correspondence between pixels constituting image data and points constituting point cloud data is unknown, or there is a discrepancy in the correspondence. The spatial correspondence is represented by internal parameters that describe the physical structure of the imaging sensor 14, and external parameters that describe the spatial and relative positional relationship between the imaging sensor 14 and the distance measuring sensor 16.

[0032] In other words, the fact that the calibration parameters used to calibrate the temporal and spatial errors of spatial elements are unknown means that at least one of the intrinsic parameters and the extrinsic parameters is unknown. Furthermore, the fact that the temporal and spatial errors of spatial elements are uncalibrated means that there is no accurate correspondence between each of the multiple points constituting the point cloud data and each of the multiple pixels constituting the image data. When an accurate correspondence is made, each of the multiple pixels constituting the image data and each of the multiple points constituting the point cloud data will represent, for each pair of pixels and points, an element of the same object at the same location and time in real space.

[0033] Here, if the imaging sensor 14 and the distance measuring sensor 16 are hardware-synchronized, the image acquisition unit 24A and the point cloud acquisition unit 24B can acquire synchronized image data and point cloud data from the imaging sensor 14 and the distance measuring sensor 16, respectively. Furthermore, if it is difficult to hardware-synchronize these sensors due to design reasons or other reasons, the image acquisition unit 24A and the point cloud acquisition unit 24B can acquire synchronized image data and point cloud data based on the timestamps obtained from the imaging sensor 14 and the distance measuring sensor 16, respectively. However, errors may occur in the timestamps of the imaging sensor 14 and the distance measuring sensor 16 due to factors such as temperature and GPS (Global Positioning System) reception status. Therefore, the point cloud data may contain temporal and spatial errors represented by such temporal elements, or the temporal and spatial errors may be unknown.

[0034] Furthermore, in order to associate each of the multiple pixels constituting the image data with each of the multiple points constituting the point cloud data in a way that they represent the same elements in real space, it is necessary to use internal parameters that represent the physical structure of the imaging sensor 14 and external parameters that represent the spatial and relative positional relationship between the imaging sensor 14 and the distance measuring sensor 16. However, there are cases where at least one of these internal and external parameters is unknown. That is, point cloud data may contain temporal and spatial errors of the spatial elements represented by such internal and external parameters, or the temporal and intervalal errors may be unknown.

[0035] The point cloud acquisition unit 24B acquires point cloud data with uncalibrated temporal and spatial errors because the calibration parameters used to correct for such temporal and spatial errors are unknown.

[0036] The calibration parameter calculation unit 24C calculates calibration parameters for each set of training data, which includes image data and point cloud data.

[0037] In this embodiment, the calibration parameter calculation unit 24C acquires a group consisting of multiple image data and multiple point cloud data for one sequence in a time series, acquired by the image acquisition unit 24A and the point cloud acquisition unit 24B respectively, as training data.

[0038] A sequence is a predetermined period of time. This predetermined time may be, for example, 10 seconds or 20 seconds, but is not limited to this time. In other words, the calibration parameter calculation unit 24C acquires multiple image data and multiple point cloud data that are consecutive in time series, acquired by the image acquisition unit 24A and the point cloud acquisition unit 24B, respectively, during one sequence, as training data. The calibration parameter calculation unit 24C may also use a pair of one image data and one point cloud data as training data, and is not limited to acquiring image data and point cloud data for one sequence as training data.

[0039] In this embodiment, the calibration parameter calculation unit 24C calculates a calibration parameter and an error evaluation value representing the magnitude of the temporal and spatial errors represented by the calibration parameter for each training data.

[0040] For example, the calibration parameter calculation unit 24C calculates the calibration parameters and error evaluation values ​​using one of the following three calculation methods: the first to the third calculation method.

[0041] First, let me explain the first calculation method.

[0042] In the first calculation method, the calibration parameter calculation unit 24C calculates a time offset as a calibration parameter, which indicates the timing difference between the acquisition of image data and point cloud data included in the training data, based on at least one of the following: position and orientation of the imaging sensor 14 and the distance measuring sensor 16, relative angle, relative distance, angular velocity, and velocity. The calibration parameter calculation unit 24C also calculates an error evaluation value for the calculated calibration parameter.

[0043] In detail, the calibration parameter calculation unit 24C calculates a time offset by estimating the acquisition start time and acquisition update rate of the image acquisition unit 24A and the point cloud acquisition unit 24B, respectively.

[0044] Specifically, the calibration parameter calculation unit 24C determines the correspondence between multiple time-series image data and multiple point cloud data that constitute the training data by aligning the trajectories acquired from, for example, VO (Visual Odometry) and LO (LiDAR Odometry), and calculates a rough time offset. Then, the calibration parameter calculation unit 24C determines the initial values ​​for the data acquisition start time and acquisition update speed of the image acquisition unit 24A and the point cloud acquisition unit 24B from the calculated rough time offset. The calibration parameter calculation unit 24C generates the initial values ​​for the data acquisition start time and acquisition update speed from the rough time offset, and uses these generated initial values ​​to calculate the calibration parameters described later.

[0045] Next, the calibration parameter calculation unit 24C determines at least one of the following from VO and LO: position and orientation of the imaging sensor 14 and the distance measuring sensor 16, relative angle, relative distance, angular velocity, and speed.

[0046] The calibration parameter calculation unit 24C determines at least one of the following for the distance measuring sensor 16 that acquired the point cloud data: position and orientation, relative angle, relative distance, angular velocity, and velocity. The relative angle and relative distance of the distance measuring sensor 16 refer to the relative angle and relative distance of the distance measuring sensor 16 at the time of distance measurement of the point cloud data acquired at the previous timing, for each distance measurement of the multiple point cloud data in the time series that constitute the learning data.

[0047] The calibration parameter calculation unit 24C only needs to determine at least one of the following parameters for the distance measuring sensor 16 that acquired point cloud data: position, orientation, relative angle, relative distance, angular velocity, and speed, based on the time-series position and orientation information obtained from the LO.

[0048] Similarly, the calibration parameter calculation unit 24C determines at least one of the following for the imaging sensor 14 that captured the image data: position and orientation, relative angle, relative distance, angular velocity, and velocity. The relative angle and relative distance of the imaging sensor 14 refer to the relative angle and relative distance of the imaging sensor 14 at the time of capturing each of the multiple time-series image data that constitute the training data, relative to the imaging sensor 14 at the time of capturing the image data acquired at the previous timing.

[0049] The calibration parameter calculation unit 24C only needs to determine at least one of the following parameters of the imaging sensor 14 that acquired the image data: position and orientation, relative angle, relative distance, angular velocity, and velocity, based on the time-series position and orientation information obtained from VO.

[0050] The calibration parameter calculation unit 24C then interpolates at least one of the time-series position and orientation, relative angle, relative distance, angular velocity, and velocity of the imaging sensor 14 or distance measuring sensor 16 so that the update speeds of the image acquisition unit 24A and the point cloud acquisition unit 24B are the same. Interpolation means adjusting at least one of the time-series position and orientation, relative angle, relative distance, angular velocity, and velocity of the imaging sensor 14 or distance measuring sensor 16 so that the update speeds of the image acquisition unit 24A and the point cloud acquisition unit 24B are the same. The calibration parameter calculation unit 24C then calculates a time offset, which is a difference in acquisition timing, by determining the acquisition start time or update speed at which a high correlation can be obtained from the cross-correlation function for waveforms a and waveform b obtained by normalizing at least one of the interpolated position and orientation, relative angle, relative distance, angular velocity, and velocity.

[0051] Waveform a represents the transition of at least one of the following time series: position, orientation, relative angle, relative distance, angular velocity, and velocity, at the time of acquisition (i.e., at the time of shooting) for each of the multiple image data in the training data along a time series. Waveform b represents the transition of at least one of the following time series: position, orientation, relative angle, relative distance, angular velocity, and velocity, at the time of acquisition (i.e., at the time of ranging) for each of the multiple point cloud data in the training data along a time series.

[0052] The calibration parameter calculation unit 24C then determines the number of image data points for each acquisition timing, i, for at least one of the above-mentioned interpolated position and orientation, relative angle, relative distance, angular velocity, and velocity obtained from the image data and point cloud data included in the training data. The calibration parameter calculation unit 24C also determines N to be the value of the larger of the image data and point cloud data points, k to be the sum of the number values ​​of the image data and point cloud data points, and conj() to be the function that converts each element of the array to its complex conjugate. Then, at a specific update rate, the array c showing the correlation between k combinations of acquisition start times for the image data and point cloud data is expressed by the following equation (1) from the cross-correlation function.

[0053]

number

[0054] The calibration parameter calculation unit 24C obtains the above sequence c for multiple update speeds and searches for at least one of the data acquisition start time and update speed that shows a high correlation using a grid search. The calibration parameter calculation unit 24C calculates the time offset, which is represented by the difference between at least one of the data acquisition start time and update speed obtained through this search, as the calibration parameter.

[0055] Figure 2A is a schematic diagram of an example of waveforms a and b before applying the calculated time offset. Figure 2B is a schematic diagram of an example of waveforms a and b after applying the calculated time offset.

[0056] Returning to Figure 1, we continue the explanation.

[0057] Furthermore, the calibration parameter calculation unit 24C calculates an error evaluation value that represents the magnitude of the temporal and spatial errors expressed by the calculated calibration parameters. Specifically, the calibration parameter calculation unit 24C calculates a higher error evaluation value the lower the correlation between the image data and point cloud data included in the training data. Correlation represents the degree of agreement and relationship between the temporal and spatial elements expressed by the temporal and spatial errors. In the first calculation method, the calibration parameter calculation unit 24C calculates a higher error evaluation value the lower the correlation represented by the above-mentioned array c. Methods for calculating the error evaluation value include using the overall correlation of multiple image data and multiple point cloud data included in the searched training data as the error evaluation value, or dividing multiple image data and multiple point cloud data included in the training data along a time series into multiple time intervals and using the correlation of each time interval as the error evaluation value.

[0058] Next, I will explain the second calculation method.

[0059] In the second calculation method, the calibration parameter calculation unit 24C reprojects the pixels constituting the image data onto the image data included in the training data or other image data, using the attitude information of the distance measuring sensor 16 at the time of acquisition of the point cloud data, which is presumed to have been acquired at the same timing as the image data. The calibration parameter calculation unit 24C then calculates calibration parameters based on a reprojection error that represents at least one of the distance between the reprojected pixels and the pixels before reprojection, the difference in pixel values, and the difference in the feature quantities of the pixels. In the second calculation method, the calibration parameter calculation unit 24C calculates calibration parameters that include at least one of the following: a time offset indicating the timing difference in acquisition between the image data included in the training data and the point cloud data; internal parameters indicating the structure of the imaging sensor 14 that captured the image data; and external parameters indicating the spatial relationship between the imaging sensor 14 and the distance measuring sensor 16.

[0060] In detail, the calibration parameter calculation unit 24C determines the correspondence between multiple time-series image data and multiple point cloud data included in the training data by aligning the trajectories obtained from, for example, VO and LO, and calculates and applies a rough time offset. The calibration parameter calculation unit 24C generates initial values ​​for the acquisition start time and acquisition update rate from the rough time offset, and uses these generated initial values ​​to calculate the calibration parameters described later.

[0061] Next, the calibration parameter calculation unit 24C defines the world coordinates of the time-series point cloud data included in the learning data based on the orientation of the distance measuring sensor 16 at the initial time t0, and then calculates the orientation T of the distance measuring sensor 16 at the time of acquisition (distance measurement) for each of the multiple time-series point cloud data. i L The calibration parameter calculation unit 24C obtains the image I of the image data acquired at the same time as the point cloud data. i Image I of other image data besides the said image data, obtained from feature points or optical flow, etc. j The corresponding pixel P between ij IFind it. Then, the calibration parameter calculation unit 24C uses the pixel correspondence relationship and the relative pose (T i L ) -1 T j L obtained from the distance measurement sensor 16 to determine the position p of the pixel P ij I in the world coordinates by triangulation. Furthermore, the calibration parameter calculation unit 24C projects the position p i w using the pose T k L at other times onto the image I of the image data, and calculates the distance on the image coordinates between the pixel P i w and the pixel P k ki , the difference in pixel values between the pixel P I ij and the pixel P I ki I , and the difference in feature amounts between the pixel P ki I and the pixel P ij I ki I ki to obtain the reprojection error pe ki .

[0062] Then, the calibration parameter calculation unit 24C converts the pose of the distance measurement sensor 16 at the time of distance measurement of each of the plurality of point cloud data in the time series included in the learning data into a continuous value by linear interpolation. The calibration parameter calculation unit 24C uses the obtained reprojection error pe ki to solve at least one of the external parameter T L C , the internal parameter π, and the time offset Δt k、 including the update speed and the difference from the initial time by a non-linear least squares method such as the LM (Levenberg Marquardt algorithm) method. By this process, the calibration parameter calculation unit 24C calculates at least one of the time offset Δt k , the internal parameter π, and the external parameter T L C as calibration parameters.

[0063] Figure 3 shows the reprojection error pe ki This is an illustrative diagram.

[0064] reprojection error pe ki This is expressed by equation (2) below.

[0065]

number

[0066] In formula (2), t k This is image I k This shows the initial value at the time the data was obtained.

[0067] The calibration parameter calculation unit 24C calculates all pixels on the image of other nearby image data with similar acquisition times for the image data included in the training data. i* For each feature point, the reprojection error pe ki The calibration parameter calculation unit 24C then calculates the reprojection error pe of each of the multiple image data included in the training data. ki The sum of these values ​​can be calculated as the reprojection error of the image data.

[0068] Furthermore, the calibration parameter calculation unit 24C calculates an error evaluation value that represents the magnitude of the temporal and spatial errors expressed by the calculated calibration parameters. Specifically, the calibration parameter calculation unit 24C uses at least one of the sum of the reprojection errors of each image data in the sequence included in the training data, or the sum of the reprojection errors of each pixel included in each image data, to calculate an error evaluation value that is higher the larger the sum of the reprojection errors.

[0069] Next, I will explain the third calculation method.

[0070] In the third calculation method, the calibration parameter calculation unit 24C calculates calibration parameters, which include at least one of a time offset, an internal parameter, and an external parameter, based on the difference between the position information of the object observed by the point cloud data and the position information of the object observed by the image data.

[0071] In detail, the calibration parameter calculation unit 24C determines the correspondence between multiple time-series image data and multiple point cloud data that constitute the training data by aligning the trajectories acquired from, for example, VO and LO, and calculates and applies a rough time offset. The calibration parameter calculation unit 24C generates initial values ​​for the acquisition start time and acquisition update rate from the rough time offset, and uses these generated initial values ​​to calculate the calibration parameters described later.

[0072] Next, the calibration parameter calculation unit 24C detects the same object observed in each of the multiple point cloud data included in the training data. The calibration parameter calculation unit 24C can detect the object, for example, by a machine learning-trained 3D object detector or by semantic segmentation of the point cloud data. The calibration parameter calculation unit 24C also determines the 3D position x of the detected object relative to the distance measuring sensor 16. i The calibration parameter calculation unit 24C calculates the size of the detected object using the point cloud data.

[0073] The calibration parameter calculation unit 24C then detects the object that appears in multiple image data included in the training data. The calibration parameter calculation unit 24C detects the object using machine learning-based object detectors, semantic segmentation, etc. Then, the calibration parameter calculation unit 24C calculates the size of the object from the point cloud data and the image data of the object. i Using the size and internal parameter π, the three-dimensional position x of the object relative to the imaging sensor 14 is determined. i The following is calculated. The internal parameter π can be determined by pre-calculating or pre-defined initial values.

[0074] The calibration parameter calculation unit 24C calculates the three-dimensional position x of the object from the point cloud data. i 'and the three-dimensional position x of the object calculated from the image data. i The difference in position te i The calibration parameter calculation unit 24C calculates the time offset Δt which includes the difference between the update rate and the initial time. i , internal parameter π, and external parameter T L C For at least one of these, the 3D position of the object calculated from the image data and point cloud data is converted to a continuous value by linear interpolation using a nonlinear least squares method such as the LM method, and then the difference te i Solve the problem. Through this calculation process, the calibration parameter calculation unit 24C determines the time offset Δt i , internal parameter π, and external parameter T L C At least one of these is calculated as a calibration parameter.

[0075] position difference te i The synchronization time before calibration is t i Therefore, it can be expressed by the following equation (3).

[0076]

number

[0077] In formula (3), t i This is image I i This shows the initial value at the time the data was obtained.

[0078] position difference te i For this, you can use the sum of all synchronized data.

[0079] Figure 4A shows the position x before applying the calibration parameters. i 'and position x i This is an illustrative diagram of an example of the trajectory. Figure 4B shows the position x after applying the calibration parameters. i 'and position x iThis is an illustrative diagram of an example of the trajectory. In Figures 4A and 4B, the vertical axis represents the vertical horizontal distance from the imaging sensor 14 and the distance measuring sensor 16 to the object, and the horizontal axis represents the horizontal horizontal distance.

[0080] Furthermore, the calibration parameter calculation unit 24C calculates an error evaluation value that represents the magnitude of the temporal and spatial errors expressed by the calculated calibration parameters. Specifically, the calibration parameter calculation unit 24C calculates an error evaluation value that is the sum of the differences in the positions of objects between all image data and all point cloud data in the sequence included in the training data, or a higher value the larger the position difference in each data.

[0081] Returning to Figure 1, we continue the explanation.

[0082] Furthermore, the calibration parameter calculation unit 24C may further calculate a confidence level for the training data used to calculate the error evaluation value, based on the calculated error evaluation value. The confidence level is lower the larger the temporal and spatial error represented by the error evaluation value. The confidence level is an evaluation value that represents the effectiveness of the training data. A higher confidence level means that the training data is more effective.

[0083] Furthermore, the calibration parameter calculation unit 24C may normalize at least one of the variances of the error evaluation values ​​within multiple image data and multiple point cloud data included in the training data, or within multiple training data and multiple point cloud data, and then calculate the confidence level for the training data.

[0084] The calibration parameter calculation unit 24C stores the learning data, the calibration parameters calculated from the learning data, the error evaluation value, and the confidence level in the storage unit 20, associating them with each other.

[0085] The determination unit 24D determines the validity of the training data used to calculate the calibration parameters based on the calibration parameters. Determining the validity of the training data means determining whether the calibrated training data, which has been calibrated using the calibration parameters calculated from the training data, is valid training data for training the learning model.

[0086] In detail, the determination unit 24D determines that the higher the error evaluation value of the temporal and spatial errors represented by the calibration parameters, the lower the effectiveness of the training data used to calculate the calibration parameters. Conversely, the lower the error evaluation value, the higher the effectiveness of the training data used to calculate the calibration parameters representing the temporal and spatial errors represented by the error evaluation value.

[0087] Specifically, for example, the determination unit 24D pre-sets a threshold for use in determining the effectiveness of the error evaluation value. If the error evaluation value is less than the threshold, the determination unit 24D determines that the training data used to calculate the calibration parameters representing the temporal and spatial errors expressed by the error evaluation value is highly effective training data. If the error evaluation value is greater than or equal to the threshold, the determination unit 24D determines that the training data used to calculate the calibration parameters representing the temporal and spatial errors expressed by the error evaluation value is less effective training data. The determination unit 24D may also determine that the training data it has determined to be less effective is invalid training data.

[0088] Furthermore, the determination unit 24D may recalculate the error evaluation value based on the variance within the training data, or within each of the multiple image data and multiple point cloud data included in the training data. If the recalculated error evaluation value is less than a predetermined threshold, the determination unit 24D determines that the training data used to calculate the calibration parameters representing the temporal and spatial errors expressed by the error evaluation value is highly effective training data. If the error evaluation value is greater than or equal to the threshold, the determination unit 24D determines that the training data used to calculate the calibration parameters representing the temporal and spatial errors expressed by the error evaluation value is less effective training data. The determination unit 24D may also determine that the training data determined to be less effective is invalid training data.

[0089] Furthermore, the determination unit 24D may determine that a lower value is used when the error evaluation value is high, as the value representing the result of the effectiveness judgment.

[0090] The determination unit 24D then stores the effectiveness determination result in the storage unit 20, associating it with the error evaluation value and learning data used in the determination.

[0091] Therefore, the memory unit 20 stores the effectiveness determination result, calibration parameters, error evaluation value, and confidence level in association with each piece of training data, which includes multiple image data and multiple point cloud data for one sequence.

[0092] The output control unit 24E outputs the determination result of the determination unit 24D. For example, the output control unit 24E outputs the training data and the determination result of the effectiveness of the training data in association with each other to the UI unit 12. By outputting the determination result to the UI unit 12, the determination result of the effectiveness of the training data can be provided to the user in a visually understandable manner. The user can then instruct the system to remove training data that has been determined to have low effectiveness and to use training data that has been determined to have high effectiveness for training the learning model, thereby enabling the use of training data suitable for high-precision learning.

[0093] Furthermore, the output control unit 24E outputs the training data and the results of the effectiveness determination of the training data to an external information processing device via the communication unit 22. The external information processing device removes the training data that has been determined to have low effectiveness according to the effectiveness determination results, and uses the training data that has been determined to have high effectiveness for training the learning model, thereby enabling the use of training data suitable for high-precision training.

[0094] Furthermore, the output control unit 24E may output the training data and the result of determining the validity of the training data to the calibration unit 24F.

[0095] In this case, the output control unit 24E outputs the training data whose effectiveness is determined to be above a predetermined value and the calibration parameters calculated from the training data to the calibration unit 24F. Alternatively, the output control unit 24E outputs the training data acquired by the image acquisition unit 24A and the point cloud acquisition unit 24B, the calibration parameters calculated from the training data, and at least one of the results of the effectiveness determination of the training data and the confidence level for the training data to the calibration unit 24F.

[0096] Effectiveness being above a predetermined value means that the error evaluation value, which represents the magnitude of the temporal and spatial errors expressed by the calibration parameters calculated from the training data, is above the threshold value mentioned above.

[0097] The calibration unit 24F calibrates at least one of the image data and point cloud data included in training data whose effectiveness is above a predetermined value, using calibration parameters calculated from the training data. That is, for training data that the determination unit 24D has determined to be valid because the error evaluation value is below a threshold, the calibration unit 24F calibrates at least one of the image data and point cloud data included in the training data, using calibration parameters calculated from the training data. The calibration unit 24F should calibrate at least one of the image data and point cloud data included in the training data using calibration parameters so that the temporal and spatial errors between the image data and point cloud data are canceled out. Known methods can be used for the calibration process of at least one of the image data and point cloud data using calibration parameters.

[0098] Furthermore, regardless of whether the effectiveness is above or below a predetermined value, the calibration unit 24F may calibrate at least one of the image data and point cloud data contained in each of the learning data stored in the storage unit 20 using calibration parameters calculated from the learning data.

[0099] The learning unit 24G learns a learning model that outputs estimated depth information for each pixel constituting the image data from the image data, based on the training data calibrated by the calibration unit 24F. The learning model is a machine learning model.

[0100] For example, consider a case where the calibration unit 24F calibrates at least one of the image data and point cloud data included in training data whose effectiveness is above a predetermined value, using calibration parameters calculated from the training data. In other words, consider a case where the calibration unit 24F calibrates training data that the determination unit 24D has determined to be valid because its error evaluation value is below a threshold, using calibration parameters.

[0101] In this case, the learning unit 24G inputs image data included in the learning data, which has been determined to be valid learning data and calibrated by the calibration parameters, into the learning model, and adjusts the parameters of the learning model so that the error between the estimated depth information for each pixel output from the learning model and the depth information for each point defined by the point cloud data included in the learning data is minimized. Through this parameter adjustment process, the learning unit 24G learns the learning model.

[0102] The learning unit 24G can learn a highly accurate learning model that outputs highly accurate estimated depth information from image data by training a learning model using learning data that has been determined to be valid learning data and calibrated by calibration parameters.

[0103] Furthermore, consider a case where, for example, the calibration unit 24F calibrates at least one of the image data and point cloud data contained in all the learning data received from the output control unit 24E, regardless of whether the effectiveness is above or below a predetermined value, using calibration parameters calculated from the learning data.

[0104] In this case, the learning unit 24G inputs the image data included in the calibrated training data into the learning model and calculates the error between the estimated depth information for each pixel output from the learning model and the depth information for each point defined by the point cloud data included in the training data. The learning unit 24G then adjusts the parameters of the learning model so that the smaller the temporal and spatial error represented by the calibration parameters calculated by the calibration parameter calculation unit 24C, the greater the confidence level, and the smaller the error, the greater the weighted error, which is weighted according to the confidence level. Specifically, the learning unit 24G calculates a weighted error that is weighted according to the error between the estimated depth information and the depth information, with the value increasing as the confidence level increases. Specifically, for example, the learning unit 24G calculates a weighted error that is weighted according to the confidence level, which is weighted according to the error between the estimated depth information and the depth information. The learning unit 24G then adjusts the parameters of the learning model so that the weighted error is minimized. Through this parameter adjustment process, the learning unit 24G learns the learning model.

[0105] The learning unit 24G can learn a highly accurate learning model that outputs highly accurate estimated depth information from image data by adjusting the parameters of the learning model so that the weighted error, which is weighted according to the confidence level, is minimized.

[0106] Alternatively, the learning unit 24G may generate a permutation of the training data according to the confidence level, and input the image data contained in the training data into the learning model according to the permutation for training. More specifically, the learning unit 24G may retrieve all the training data stored in the storage unit 20 and sort the training data in descending order of confidence level. Then, it may input the image data contained in the training data into the learning model in descending order of confidence level for training.

[0107] Alternatively, the learning unit 24G may sort the training data, which has been calibrated using calibration parameters, in descending order of confidence, and then input the image data contained in the training data into the learning model in descending order of confidence for training.

[0108] The learning unit 24G generates permutations of training data according to the confidence level, and inputs the image data contained in the training data into the learning model according to the permutations. This allows for faster convergence of learning, improved learning efficiency, reduced learning time, and highly accurate learning.

[0109] Next, an example of the information processing flow executed by the information processing device 10 of this embodiment will be described.

[0110] Figure 5 is a flowchart showing an example of the flow of the learning data effectiveness determination process performed by the information processing device 10 of this embodiment.

[0111] The calibration parameter calculation unit 24C acquires training data that includes image data acquired from the imaging sensor 14 by the image acquisition unit 24A and point cloud data acquired from the distance measuring sensor 16 by the point cloud acquisition unit 24B (step S100). For example, the calibration parameter calculation unit 24C acquires a group consisting of multiple image data and multiple point cloud data for one sequence in a time series as training data.

[0112] The calibration parameter calculation unit 24C calculates calibration parameters for the training data acquired in step 100 (step S102). The calibration parameter calculation unit 24C also calculates an error evaluation value that represents the magnitude of the temporal and spatial errors represented by the calibration parameters calculated in step S102 (step S104). Furthermore, based on the error evaluation value calculated in step S104, the calibration parameter calculation unit 24C calculates a confidence level for the training data used to calculate the error evaluation value, with the confidence level decreasing as the temporal and spatial errors represented by the error evaluation value increase (step S106).

[0113] The determination unit 24D determines whether the error evaluation value calculated in step S104 is less than the threshold (step S108). If it determines that the error evaluation value is less than the threshold (step S108: Yes), the determination unit 24D determines that the effectiveness of the training data acquired in step S100 is high (step S110). Then, the process proceeds to step S114.

[0114] On the other hand, if the error evaluation value is determined to be above a threshold (step S108: No), the determination unit 24D determines that the effectiveness of the training data acquired in step S100 is low (step S112). Then, the process proceeds to step S114.

[0115] In step S114, the determination unit 24D stores in the storage unit 20 the learning data acquired in step 100, the calibration parameters calculated in step S102, the effectiveness determination result from step S110 or step 112, and the error evaluation value and confidence level calculated in steps S104 and S106, respectively, in association with each other.

[0116] The output control unit 24E outputs the determination result of the determination unit 24D in step S110 or step S112 to the UI unit 12 and an external information processing device, etc. (step S116). Then, this routine ends.

[0117] The processing unit 24 executes steps S100 to S114 for each learning data set of a different sequence, so that for each of the multiple learning data sets, the calibration parameters, the effectiveness determination result, the error evaluation value, and the confidence level are stored in the storage unit 20 in association with each other.

[0118] Next, we will describe the learning process performed by the processing unit 24.

[0119] Figure 6 is a flowchart showing an example of the learning process flow executed by the processing unit 24.

[0120] The calibration unit 24F calibrates at least one of the image data and point cloud data contained in each of the training data stored in the storage unit 20 using calibration parameters calculated from the training data (step S200). As described above, the calibration unit 24F may also calibrate at least one of the image data and point cloud data contained in training data for which the error evaluation value is less than a threshold and the effectiveness is determined to be high, using calibration parameters calculated from the training data.

[0121] Then, the learning unit 24G learns a learning model that outputs estimated depth information for each pixel constituting the image data from the image data, based on the training data calibrated in step S200 (step S202). Then, this routine ends.

[0122] As described above, the information processing program executed by the information processing device of this embodiment includes an image acquisition step for acquiring image data, a point cloud acquisition step for acquiring point cloud data in which depth information is defined for each of a plurality of points, where the calibration parameters used to calibrate at least one of the temporal and spatial errors of the image data are unknown and the temporal and spatial errors are uncalibrated, a calibration parameter calculation step for calculating calibration parameters for each set of training data including image data and point cloud data, a determination step for determining the validity of the training data used to calculate the calibration parameters based on the calibration parameters, and an information processing program to be executed by a computer.

[0123] Conventional technologies disclose methods for calculating calibration parameters for uncalibrated training data where the calibration parameters are unknown, and using these parameters to calibrate the training data. However, training data calibrated using calibration parameters may contain training data that failed to be calibrated. In other words, training data with temporal and spatial errors that cannot be fully calibrated even with the use of calibration parameters cannot have their temporal and spatial errors corrected by applying the calibration parameters. If such calibrated training data is used to train a learning model, it means that training data that failed to be calibrated is being used, and high-precision learning may not be achievable. In short, conventional technologies did not provide training data suitable for high-precision learning.

[0124] On the other hand, the information processing program of this embodiment determines the effectiveness of the training data used to calculate the calibration parameters based on the calibration parameters calculated for each uncalibrated training data set, which includes image data and point cloud data. In other words, the information processing program of this embodiment determines the effectiveness of the training data when calibrated using the calibration parameters.

[0125] Therefore, the information processing program of this embodiment can determine the effectiveness of training data calibrated using calibration parameters, indicating whether or not it is suitable for training a learning model. Furthermore, the information processing program of this embodiment can provide the effectiveness of the training data as data to be used for training a learning model. Therefore, the information processing program of this embodiment can support the provision of training data suitable for high-precision learning.

[0126] Therefore, the information processing program of this embodiment can support the provision of training data suitable for high-precision learning.

[0127] Furthermore, since the information processing program of this embodiment learns the learning model according to the effectiveness of the training data, it can perform highly accurate learning.

[0128] Next, an example of the hardware configuration of the information processing device 10 of the above embodiment will be described.

[0129] Figure 7 is a hardware configuration diagram of an example of the information processing device 10 of the above embodiment.

[0130] The information processing device 10 of the above embodiment includes a control device such as a CPU (Central Processing Unit) 90D, a storage device such as a ROM (Read Only Memory) 90E, a RAM (Random Access Memory) 90F, and an HDD (Hard Disk Drive) 90G, an I / F unit 90B which is an interface to various devices, an output unit 90A which outputs various information such as output information, an input unit 90C which accepts user operations, and a bus 90H which connects each unit, and has a hardware configuration that uses a normal computer.

[0131] In the information processing device 10 of the above embodiment, each of the above components is realized on a computer by the CPU 90D reading an information processing program from ROM 90E onto RAM 90F and executing it.

[0132] The information processing program for executing each of the above processes performed by the information processing device 10 in the above embodiment may be stored in the HDD 90G. Alternatively, the information processing program for executing each of the above processes performed by the information processing device 10 in the above embodiment may be pre-installed and provided in the ROM 90E.

[0133] Furthermore, the information processing program for executing the above-described process performed by the information processing device 10 of the above embodiment may be provided as a computer program product by being stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disc), or flexible disk (FD). Alternatively, the program for executing the above-described process performed by the information processing device 10 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the program for executing the above-described process performed by the information processing device 10 of the above embodiment may be provided or distributed via a network such as the Internet.

[0134] Although embodiments of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0135] 10 Information Processing Devices 14. Image sensor 16 Distance measuring sensor 24A Image acquisition unit 24B Point cloud acquisition part 24C Calibration Parameter Calculation Unit 24D Judgment part 24E Output Control Unit 24F Calibration Department 24G Learning Department

Claims

1. Image acquisition step to obtain image data, A point cloud acquisition step in which depth information is defined for each of a plurality of points, for which the calibration parameters used to calibrate the temporal and spatial errors of at least one of the temporal and spatial elements of the aforementioned image data are unknown and the temporal and spatial errors are uncalibrated; A calibration parameter calculation step, which calculates the calibration parameter for each training data including the image data and the point cloud data, A determination step in which the effectiveness of the training data used to calculate the calibration parameters is determined based on the calibration parameters, An information processing program that causes a computer to execute something.

2. The calibration parameter calculation step is as follows: The calibration parameters and an error evaluation value representing the magnitude of the temporal and spatial errors expressed by the calibration parameters are calculated. The aforementioned determination step is, The higher the error evaluation value, the lower the effectiveness is judged. The information processing program according to claim 1.

3. The calibration parameter calculation step is as follows: Based on at least one of the position, orientation, relative angle, relative distance, angular velocity, and speed of the distance measuring sensor that acquired the point cloud data and the imaging sensor that captured the image data, a time offset indicating the timing difference between the acquisition of the image data and the point cloud data included in the training data is calculated as the calibration parameter. The information processing program according to claim 1.

4. The calibration parameter calculation step is as follows: Based on a reprojection error that represents at least one of the distance, the difference in pixel values, and the difference in pixel features between the reprojected pixels, which are reprojected onto the image data or other image data included in the training data, and the pixels constituting the image data, using the attitude information of the distance sensor at the time of acquisition of the point cloud data, which is presumed to have been acquired at the same time as the image data, and the pixels constituting the image data, The calibration parameter is calculated, which includes at least one of the following: a time offset indicating the timing difference between the acquisition of the image data and the point cloud data included in the learning data; an internal parameter indicating the structure of the imaging sensor that captured the image data; and an external parameter indicating the spatial relationship between the imaging sensor and the distance measuring sensor. The information processing program according to claim 1.

5. The calibration parameter calculation step is as follows: Based on the difference between the position information of an object observed by the point cloud data and the position information of the object observed by the image data, the calibration parameter is calculated, which includes at least one of the following: a time offset indicating the timing difference between the acquisition of the image data and the point cloud data included in the training data; an internal parameter indicating the structure of the imaging sensor that captured the image data; and an external parameter indicating the spatial relationship between the imaging sensor and the distance measuring sensor that acquired the point cloud data. The information processing program according to claim 1.

6. The calibration parameter calculation step is as follows: Based on the error evaluation value, a confidence level for the training data is further calculated, with the confidence level decreasing as the temporal and spatial errors represented by the error evaluation value increase. The information processing program according to claim 2.

7. An output control step that outputs the determination result of the determination step, The information processing program according to claim 1, which includes the following:

8. A calibration step in which at least one of the image data and point cloud data included in the training data whose effectiveness is greater than or equal to a predetermined value is calibrated by the calibration parameter calculated from the training data, A learning step of training a learning model that outputs estimated depth information for each pixel constituting the image data from the image data based on the calibrated training data, The information processing program according to claim 1, which includes the following:

9. The aforementioned learning steps are: The image data included in the calibrated training data, whose effectiveness is equal to or greater than a predetermined value, is input to the training model. The parameters of the learning model are adjusted so that the error between the estimated depth information for each pixel output from the learning model and the depth information for each point defined by the point cloud data included in the calibrated learning data is minimized. The information processing program according to claim 8.

10. The aforementioned learning steps are: The image data included in the calibrated training data is input to the training model, and the error between the estimated depth information for each pixel output from the training model and the depth information for each point defined by the point cloud data included in the calibrated training data is calculated. The parameters of the learning model are adjusted so that the weighted error, which is obtained by weighting the error according to the confidence level, is minimized as the temporal and spatial errors represented by the calibration parameters calculated from the learning data become smaller. The information processing program according to claim 9.

11. The aforementioned learning steps are: The system generates permutations of the training data according to the magnitude of the confidence level, where the temporal and spatial errors represented by the calibration parameters are smaller, and inputs the image data included in the training data into the training model according to the permutations. The information processing program according to claim 8.

12. Image acquisition unit that acquires image data, A point cloud acquisition unit acquires point cloud data in which depth information is defined for each of a plurality of points, for which the calibration parameters used to calibrate the temporal and spatial errors of at least one of the temporal and spatial elements of the aforementioned image data are unknown and the temporal and spatial errors are uncalibrated. A calibration parameter calculation unit calculates the calibration parameter for each training data including the image data and the point cloud data, A determination unit that determines the effectiveness of the learning data used to calculate the calibration parameters based on the calibration parameters, An information processing device equipped with the following features.

13. Image acquisition step to obtain image data, A point cloud acquisition step in which depth information is defined for each of a plurality of points, for which the calibration parameters used to calibrate the temporal and spatial errors of at least one of the temporal and spatial elements of the aforementioned image data are unknown and the temporal and spatial errors are uncalibrated; A calibration parameter calculation step, which calculates the calibration parameter for each training data including the image data and the point cloud data, A determination step in which the effectiveness of the training data used to calculate the calibration parameters is determined based on the calibration parameters, Information processing methods including

Images