Method and apparatus for calibration between heterogeneous sensors

Abstract

A method and an apparatus for calibration between heterogeneous sensors are disclosed. According to one aspect of the present disclosure, provided is a computer-implemented method for calibration between heterogeneous sensors, comprising: a process of generating a first depth image from data of a first sensor; a process of generating a second depth image from data of a second sensor having a modality different from that of the first sensor; and a process of estimating an external parameter between the first sensor and the second sensor from the first depth image and the second depth image by using a transformer-based network.

Description

Method and device for calibration between heterogeneous sensors

[0001] The present disclosure relates to a method and apparatus for calibration between heterogeneous sensors.

[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0003] Sensor fusing technology is necessary to integrate and utilize information from heterogeneous sensors, such as lidar and cameras, in autonomous mobility. Sensor fusing requires prerequisite calibration of the relative positions of the sensors.

[0004] Sensor calibration techniques can be broadly categorized into target-based and targetless methods. Target-based methods identify feature information on the target board before the vehicle is released and operated, and calibrate the positional information between sensors. However, this method has the disadvantage of not being able to correct for distortions that occur during vehicle operation (e.g., physical distortion of the sensor or deterioration due to temperature changes).

[0005] The present disclosure aims to provide a method and device capable of correcting the relative positions between sensors in real time or near real time.

[0006] The present disclosure aims to provide a deep learning model capable of correcting positional information between sensors in an end-to-end manner, a learning method thereof, and an inference method using the same.

[0007] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0008] According to one aspect of the present disclosure, a computer-implemented method for calibration between heterogeneous sensors is provided, the method comprising: generating a first depth image from data of a first sensor; generating a second depth image from data of a second sensor having a modality distinct from that of the first sensor; and estimating an extrinsic parameter between the first sensor and the second sensor from the first depth image and the second depth image using a transformer-based network.

[0009] According to another aspect of the present disclosure, there is provided a device comprising: at least one memory storing instructions; and at least one processor, wherein the at least one processor executes the instructions to generate a first depth image from data of a first sensor, generate a second depth image from data of a second sensor having a modality distinct from that of the first sensor, and estimate an external parameter between the first sensor and the second sensor from the first depth image and the second depth image using a transformer-based network.

[0010] According to another aspect of the present disclosure, there is provided a device comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to implement a network for estimating an external parameter between different coordinate systems, wherein the network comprises: a transformer encoder configured to extract features from a first depth image generated from data of a first sensor and a second depth image generated from data of a second sensor having a modality distinct from the first sensor; a transformer decoder configured to estimate, based on the features extracted by the transformer encoder, one or more corresponding points of the second depth image that match one or more query points of the first depth image; and one or more regressors configured to estimate, based on the query points and the corresponding points, the external parameter.

[0011] According to embodiments of the present disclosure, the relative positions between sensors can be corrected in real time or near-real time. This can improve the accuracy of perception tasks during autonomous vehicle operation.

[0012] According to an embodiment of the present disclosure, a separate infrastructure for sensor calibration is not required, and online auto-calibration is possible.

[0013] According to an embodiment of the present disclosure, by converting heterogeneous sensor data into depth images, the multi-modality (e.g., lidar and camera) of the system can be reduced to a single modality, thereby reducing the complexity of the system and increasing its accuracy.

[0014] According to an embodiment of the present disclosure, as the deep learning model learns the process of extracting corresponding points and calculating external parameters end-to-end, the module that processes the corresponding task is abstracted and / or simplified, thereby facilitating modular development.

[0015] According to an embodiment of the present disclosure, the robustness of the model can be increased by inferring correspondences between heterogeneous images (e.g., a depth image estimated from a camera image and a depth image projected from a lidar point cloud) using a transformer-based network.

[0016] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0017] Figure 1 is a block diagram schematically showing a correction device according to one embodiment of the present disclosure.

[0018] FIG. 2 is an exemplary diagram referenced to explain the operation of a preprocessing module and an estimation module according to one embodiment of the present disclosure.

[0019] FIG. 3 is a diagram schematically illustrating a network learning process according to one embodiment of the present disclosure.

[0020] FIG. 4 is a flowchart illustrating a sensor calibration method according to one embodiment of the present disclosure.

[0021] FIG. 5 is a block diagram schematically illustrating an exemplary computing device that can be used to implement a method or device according to the present disclosure.

[0022] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0023] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.

[0024] The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0025] FIG. 1 is a block diagram schematically illustrating a correction device according to one embodiment of the present disclosure. FIG. 2 is an exemplary diagram referenced to explain the operation of a preprocessing module and an estimation module according to one embodiment of the present disclosure.

[0026] As illustrated in FIG. 1, a calibration device (10) according to an embodiment of the present disclosure may include all or part of a preprocessing module (100), an estimation module (120), and a calibration module (140). Not all blocks illustrated in FIG. 1 are essential components, and some blocks included in other embodiments may be added, changed, or deleted. Meanwhile, the components illustrated in FIG. 1 represent functionally distinct elements, and at least one of the components may be implemented in a form that is integrated with each other in an actual physical environment.

[0027] The preprocessing module (100) preprocesses the data obtained from the camera (12) and the lidar (14), respectively, to produce depth images (D1 and D 2,miss ) is generated. The data acquired from the camera (12) may be a two-dimensional image (e.g., a two-dimensional color image or a two-dimensional grayscale image), and the data acquired from the lidar (14) may be a point cloud (PCL).

[0028] Referring to FIG. 2, the preprocessing module (100) can generate a first depth image (D1) by applying a depth estimation algorithm to a two-dimensional image. The preprocessing module (100) can utilize various learning-based and / or rule-based depth estimation algorithms to generate the first depth image (D1). For example, the preprocessing module (100) can generate the first depth image (D1) by using a monocular depth estimation model that is pre-trained to estimate depth based on images captured by a single camera.

[0029] The preprocessing module (100) projects the point cloud onto a two-dimensional plane to create a second depth image (D 2,miss ) can be generated. The preprocessing module (100) sets the preset initial external parameters (T init) can be used to project the point cloud. The initial external parameters can be external parameters determined in the previous time step (or iteration). The second depth image (D 2,miss ) may be a depth image that is miss-calibrated by initial external parameters.

[0030] In general, a depth image projected from a point cloud has a relatively sparse resolution compared to a depth image estimated from a 2D image. To compensate for this, a preprocessing module (100) performs interpolation on the projected depth image to generate a second depth image (D) that is relatively dense compared to the projected depth image. 2,miss ) can be generated. For example, the preprocessing module (100) can interpolate a depth image using a bi-lateral filter.

[0031] The estimation module (120) estimates pose information of the camera (12) and the lidar (14) using a pre-trained network (20). The estimation module (120) can estimate a 6-degree of freedom (6-DoF) extrinsic parameter indicating the relative rotation and movement between the pose of the camera (12) and the pose of the lidar (14).

[0032] The network (20) may include all or part of a transformer encoder (200), a transformer decoder (220), a rotation regressor (240), and a translation regressor (260).

[0033] The estimation module (120) is configured to generate a first depth image (D1) and a second depth image (D 2,miss) can be combined and input to the transformer encoder (200). The first depth image (D1) and the second depth image (D 2,miss ) can be combined in the width direction or the height direction. For example, the first depth image (D1) and the second depth image (D 2,miss ) each having a size of 256×256×3 (i.e., height×width×channel), an input image having a size of 256×512×3 can be input to the transformer encoder (200). The estimation module (120) can divide the input image into a plurality of tokens and input them to the transformer encoder (200). For example, the estimation module (120) can divide the input image into a plurality of patches (paths) and vectorize the patches to generate a sequence of tokens. As another example, the estimation module (120) can generate a sequence of tokens by flattening the spatial dimension of the input image or the feature map generated therefrom.

[0034] The transformer decoder (220) generates a first depth image (D1) and a second depth image (D) based on the features extracted by the transformer encoder (200). 2,miss ) performs matching between them. The estimation module (120) can select one or more points (hereinafter, query points) from the first depth image (D1) and inject them as query information into the transformer decoder (220). The transformer decoder (220) can select a second depth image (D1) corresponding to each of the query points. 2,miss) can estimate one or more points (hereinafter, corresponding points). For example, the transformer decoder (220) can receive 100 query points and generate 100 pairs of query points-corresponding points. The query points may be randomly extracted from all points of the first depth image (D1) or selected from filtered points. For example, the estimation module (120) can estimate one or more points (hereinafter, corresponding points) of the second depth image (D1) from among the points of the first depth image (D1). 2,miss ) points having a depth value similar to one or more of the points, and refining the filtered points to obtain a predetermined number of query points (e.g., 100). The KNN (K-Nearest Neighbor) algorithm may be used for filtering and / or refining the points, but is not limited thereto.

[0035] The estimation module (120) injects pairs of query points and corresponding points into the rotation regressor (240) and the translation regressor (260), thereby generating a rotation matrix (R pred ) and the moving matrix (t pred) can be obtained. The rotation regressor (240) and the translation regressor (260) are networks trained to perform a regression task, and can be implemented as, for example, a multi-layer perceptron (MLP). To augment the global spatial context, a calibration flow can be additionally input to the regressors (240 and 260). The calibration flow is a deviation between a query point and a corresponding point, and when the query point and the corresponding point are (x, y, z) and (x', y', z'), respectively, the calibration flow can be calculated as (x-x', y-y', z-z'). The estimation module (120) can concatenate information of pairs of query points and corresponding points {(x,y,z), (x',y',z')} and the correction flow (x-x', y-y', z-z') derived therefrom and input them to the regressors (240 and 260).

[0036] The estimation module (120) is a rotation matrix (R pred ) and the moving matrix (t pred ) based on external parameters (T pred ) can be calculated. The external parameter (T pred ) can be calculated as in mathematical expression 1.

[0037]

[0038] The external parameters derived from the output of the network (20) are the second depth image (D 2,miss ) (i.e., the coordinate system of the depth image miscorrected by the initial external parameters) and the coordinate system of the first depth image (D1) can be represented. The estimation module (120) can represent the transformation relationship between the coordinate system of the first depth image (D1) and the initial external parameters (T init ) and estimated external parameters (T pred) can be used to calculate the external parameters between the camera (12) and the lidar (14). For example, the final external parameter (T) between the camera (12) and the lidar (14) L→C ) can be calculated as in mathematical formula 2.

[0039]

[0040] The calibration module (140) can calibrate the data of the lidar (14) using the calculated external parameters. For example, the calibration module (140) can calibrate the data of the lidar (14) using the calculated final external parameters (T L→C ) by projecting the point cloud onto the image plane of the camera (12), a corrected second depth image (D 2,cal ) can be generated. As another example, the correction module (140) can generate an estimated external parameter (T pred ), based on the second depth image (D 2,miss ) corrected from the second depth image (D 2,cal ) can also be generated. Optionally, the correction module (140) can generate a corrected second depth image (D 2,cal ) is interpolated (e.g., by applying a bilateral filter) and / or a corrected second depth image (D 2,cal ) and 2D images can also be combined (e.g., channel-wise).

[0041] FIG. 3 is a diagram schematically illustrating a network learning process according to one embodiment of the present disclosure.

[0042] The learning process of the network (20) according to one embodiment of the present disclosure is executed by a learning device (30), and the learning device (30) can be executed on a computing device. The learning device may perform each function by one or more processors available to the computing device, and may include a computer-readable storage having instructions stored therein and connected to such processors, but is not limited thereto.

[0043] Training samples for learning the network (20) may include data from the camera (12) (e.g., 2D images), data from the lidar (14) (e.g., point clouds), and / or ground truth (G / T) external parameters between the camera (12) and the lidar (14).

[0044] The learning device (30) can perform preprocessing on the data of the camera (12) to generate a first depth image and perform preprocessing on the data of the lidar (14) to generate a second depth image. The preprocessing operation performed by the learning device (30) is the same as or corresponding to the operation of the preprocessing module (100) described above in FIGS. 1 and 2, and thus a detailed description thereof will be omitted. The learning device (30) can add arbitrary values ​​to the G / T external parameters and use them as initial external parameters, but is not limited thereto.

[0045] The learning device (30) can combine depth images in the width direction or the height direction to generate an input image of the network (20). The learning device (30) can input the input image to the transformer encoder (200) of the network (20) and input one or more query points selected from the first depth image as query information to the transformer decoder (220) of the network (20). The network (20) can generate corresponding points of the second depth image corresponding to the query points and a rotation matrix (R) based on the input image and the query points. pred ), and the moving matrix (t pred ) can be estimated.

[0046] The learning device (30) estimates the corresponding points and rotation matrix (R) by the network (20). pred ), and / or the moving matrix (t pred ) can be used to update the parameters of the network (20).

[0047] The learning device (30) uses rotation loss (L) as a loss for learning the network (20). R ), translation loss (L T ), point cloud distance loss (point cloud distance loss, L P ), correspondence matching point distance loss (L C ) and calibration flow loss (L CF ) can be used in whole or in part.

[0048] Rotational loss (L R ) and movement loss (L T ) represent the regression loss for the outputs of the rotation regressor (240) and the translation regressor (260), respectively. The rotation loss (L R ) is the rotation matrix (R) estimated by the rotation regressor (240). pred ) and the G / T rotation matrix (e.g., angular distance or mean square error, etc.), and the translation loss (L T ) is the moving matrix (t) estimated by the moving regressor (260). pred ) and the G / T translation matrix (e.g., L1 norm or mean square error, etc.). The G / T rotation matrix and the G / T translation matrix can be calculated based on, but are not limited to, the deviation between the initial external parameters and the G / T external parameters.

[0049] Point cloud distance loss (L P ) is the estimated external parameter (T pred) represents the point-by-point distance (e.g., L2 norm) between the reconstructed point cloud and the original point cloud. The learning device (30) uses the initial external parameters used in the generation of the second depth image, the estimated external parameters (T pred ), and G / T external parameters can be used to reconstruct the point cloud, but are not limited thereto.

[0050] Corresponding matching point distance loss (L C ) represents the distance (e.g., L2 norm) between the corresponding points estimated by the transformer decoder (220) and the G / T corresponding points. The learning device (30) may use the initial external parameters and / or the G / T external parameters to produce the G / T corresponding points, but is not limited thereto.

[0051] Correction flow loss (L CF ) may represent the difference (e.g., L1 norm) between the correction flow calculated using the query points and the corresponding points estimated by the transformer decoder (220) and the G / T correction flow. Here, the G / T correction flow may be the deviation between the query points and the G / T corresponding points. Depending on the implementation, instead of the query points, miscorrected corresponding points calculated based on the initial external parameters (e.g., points having the same pixel location as the query point and having a depth value extracted or interpolated from the point cloud) may be used.

[0052] The total loss (L) used for learning the network (20) can be expressed as in mathematical expression 3.

[0053]

[0054] Here, λ T , λ R , λ P , λ C , and λ CFrepresents the weight applied to each loss. Depending on the implementation, the corresponding point distance loss (L C ) or correction flow loss (L CF ) may be omitted (or have their weights set to 0).

[0055] FIG. 4 is a flowchart illustrating a sensor calibration method according to one embodiment of the present disclosure.

[0056] The method illustrated in FIG. 4 can be implemented by executing one or more functions of the aforementioned correction device (10) and / or learning device (30) by at least one computing device. The following description is given in terms of operations performed by the computing device.

[0057] In process S400, a computing device generates a first depth image from data from a first sensor. The first sensor may be a camera. The computing device may generate the first depth image using a pre-trained depth estimation model. The depth estimation model may be a model trained to estimate depth from images captured at multiple viewpoints using a single camera.

[0058] In step S420, the computing device generates a second depth image from data of a second sensor having a modality distinct from that of the first sensor. The second sensor may be a lidar. The computing device may acquire three-dimensional data (e.g., a point cloud) from the second sensor. The computing device may project the acquired three-dimensional data into a two-dimensional depth image using preset initial external parameters. The computing device may apply interpolation to the projected two-dimensional depth image to generate a second depth image that is relatively dense compared to the projected two-dimensional depth image. Here, applying interpolation may include applying a bilateral filter to the projected two-dimensional depth image.

[0059] In step S440, the computing device estimates extrinsic parameters between the first sensor and the second sensor from the first depth image and the second depth image using a transformer-based network. The network may include a transformer encoder, a transformer decoder, and one or more regressors.

[0060] A transformer encoder may be configured to extract features from an input image including a first depth image and a second depth image. The computing device may concatenate the first depth image and the second depth image in the width or height direction to generate the input image.

[0061] The transformer decoder may be configured to estimate one or more corresponding points of the second depth image, each of which matches one or more query points of the first depth image, based on features extracted from the transformer encoder. The query points may be a predetermined number of points selected from among the points of the first depth image. As an example, the computing device may input one or more points randomly selected from among the points of the first depth image as query points to the transformer decoder. As another example, the computing device may filter points in the first depth image that have a similar depth value to at least one point of the second depth image, and input a predetermined number of points selected from among the filtered points as query points to the transformer decoder. The filtering of points having similar depth values ​​and / or the selection of query points may use, but is not limited to, a KNN (K-Nearest Neighbor) algorithm.

[0062] The one or more regressors may include a rotation regressor configured to estimate a rotation matrix based on pairs of query points and corresponding points. The rotation matrix and translation matrix estimated by the one or more regressors may represent a rotation and a translation, respectively, between a coordinate system of the first depth image and a coordinate system of the second depth image. The one or more regressors may include a translation regressor configured to estimate a translation matrix based on pairs of query points and corresponding points. Each of the one or more regressors may include a multi-layer perceptron (MLP). According to an implementation, the one or more regressors may share one or more layers. The one or more regressors may estimate a rotation matrix and a translation matrix, respectively, based on the query points and the corresponding points, and the correction flows. To this end, the computing device may produce correction flows based on a deviation between the query points and the corresponding points estimated by the transformer decoder.

[0063] The network can be trained end-to-end using a loss calculated based on the estimated corresponding points, the estimated rotation matrix, and the estimated translation matrix for the training samples. The loss can include a first loss calculated based on the difference between the rotation matrix estimated by the first regressor and the ground truth (G / T) rotation matrix. Additionally or alternatively, the loss can include a second loss calculated based on the difference between the translation matrix estimated by the second regressor and the G / T translation matrix. Additionally or alternatively, the loss can include a third loss calculated based on the distance between the point cloud reconstructed based on the rotation matrix and translation matrix estimated by the regressors and the original point cloud acquired from the second sensor. Additionally or alternatively, the loss can include a fourth loss for a calibration flow representing a deviation between the query points calculated from the training samples and the corresponding points estimated by the transformer decoder. The fourth loss may be calculated based on, for example, the difference between the correction flow and the G / T correction flow, which represents the deviation between the query points and the G / T corresponding points. Additionally or alternatively, the loss may include a fifth loss calculated based on the distance between the corresponding points estimated by the transformer decoder and the G / T corresponding points.

[0064] In step S460, the computing device can generate a corrected second depth image based on the estimated external parameters.

[0065] FIG. 5 is a block diagram schematically illustrating an exemplary computing device that can be used to implement a method or device according to the present disclosure.

[0066] The computing device (50) may include some or all of a memory (500), a processor (520), storage (540), an input / output interface (560), and a communication interface (580). The computing device (50) may structurally and / or functionally include at least a portion of the correction device (10) and / or the learning device (30). The computing device (50) may be a stationary computing device such as a desktop computer, a server, etc., as well as a mobile computing device such as a laptop computer, a smart phone, etc. The computing device (50) may also be mounted on a mobility equipped with various sensors such as an autonomous vehicle or an Autonomous Mobile Robot (AMR). The computing device (50) may also be implemented as any specialized hardware accelerator capable of efficiently processing operations for an artificial intelligence model. For example, the computing device (50) may include a graphic processing unit (GPU), a tensor processing unit (TPU), or a neural processing unit (NPU).

[0067] The memory (500) may store a program that causes the processor (520) to perform a method or operation according to various embodiments of the present disclosure. For example, the program may include a plurality of instructions executable by the processor (520), and the method illustrated in FIG. 4 may be performed by executing the plurality of instructions by the processor (520). The memory (500) may be a single memory or a plurality of memories. In this case, information required to perform the method or operation according to various embodiments of the present disclosure may be stored in a single memory or may be divided and stored in a plurality of memories. When the memory (500) is composed of a plurality of memories, the plurality of memories may be physically separated. The memory (500) may include at least one of a volatile memory and a nonvolatile memory. The volatile memory includes a static random access memory (SRAM) or a dynamic random access memory (DRAM), and the nonvolatile memory includes a flash memory.

[0068] The processor (520) may include at least one core capable of executing at least one instruction. The processor (520) may execute instructions stored in the memory (500). The processor (520) may be a single processor or multiple processors.

[0069] Storage (540) maintains stored data even when power supplied to the computing device (50) is cut off. For example, storage (540) may include non-volatile memory, or may include storage media such as magnetic tape, optical disk, or magnetic disk. A program stored in storage (540) may be loaded into memory (500) before being executed by processor (520). Storage (540) may store a file written in a programming language, and a program generated from the file by a compiler or the like may be loaded into memory (500). Storage (540) may store data to be processed by processor (520) and / or data processed by processor (520).

[0070] The input / output interface (560) may include input devices such as a keyboard, mouse, etc., and may include output devices such as a display device, printer, etc. A user may trigger execution of a program by the processor (520) and / or check the processing results of the processor (520) through the input / output interface (560).

[0071] The communication interface (580) may provide access to an external network. The computing device (50) may communicate with other devices via the communication interface (580).

[0072] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0073] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0074] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0075] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0076] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0077]

[0078] Statement regarding sponsored research or development

[0079] This invention is the result of research conducted with the support of the Korea Institute of Industrial Technology Evaluation and Planning with funds from the Ministry of Trade, Industry and Energy (Project ID: 2001785100, Subproject ID: 20017826, Project Name: Development of 200TOPS-class Scalable AI Accelerator Processor for Real-Time Artificial Intelligence Inference, Contribution Ratio: 1 / 2), and with the support of the National IT Industry Promotion Agency with funds from the Ministry of Science and ICT (Project ID: 1711193602, Subproject ID: 2020-0-01305-004, Project Name: Development of 2,000 TFLOPS-class Server Artificial Intelligence Deep Learning Processor and Module, Contribution Ratio: 1 / 2).

[0080]

[0081] CROSS-REFERENCE TO RELATED APPLICATION

[0082] This patent application claims priority to Korean Patent Application No. 10-2023-0133588, filed in Korea on October 6, 2023, and Korean Patent Application No. 10-2023-0180430, filed in Korea on December 13, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. A computer-implemented method for calibration between heterogeneous sensors, A process of generating a first depth image from data of a first sensor; A process of generating a second depth image from data of a second sensor having a modality distinct from that of the first sensor; and A method comprising a process of estimating an extrinsic parameter between the first sensor and the second sensor from the first depth image and the second depth image using a transformer-based network.

2. In paragraph 1, The above network is, A transformer encoder configured to extract features from an input image including the first depth image and the second depth image; A transformer decoder configured to estimate one or more corresponding points of the second depth image that match one or more query points of the first depth image based on features extracted from the transformer encoder; A first regressor configured to estimate a rotation matrix based on the query points and the corresponding points; and A second regressor configured to estimate a translation matrix based on the above query points and the corresponding points. A method comprising:

3. In paragraph 2, The above input image is, A method wherein the first depth image and the second depth image are concatenated in the width or height direction.

4. In paragraph 2, The above network is, A method that is trained end-to-end using a loss calculated based on the estimated corresponding points, rotation matrix, and translation matrix for the training sample.

5. In paragraph 4, The above loss is, A first loss calculated based on the difference between the rotation matrix estimated by the first regressor and the G / T (ground truth) rotation matrix; and A second loss calculated based on the difference between the movement matrix estimated by the second regressor and the G / T movement matrix. A method comprising:

6. In paragraph 4, The above training sample includes a point cloud as data of the second sensor, A method wherein the loss includes a third loss calculated based on a distance between a point cloud reconstructed based on the estimated rotation matrix and the estimated translation matrix.

7. In paragraph 4, A method wherein the loss comprises a fourth loss calculated based on the difference between a calibration flow representing a deviation between the query points and corresponding points estimated by the transformer decoder and a G / T calibration flow representing a deviation between the query points and G / T corresponding points.

8. In paragraph 4, A method wherein the above loss includes a fifth loss calculated based on the distance between corresponding points estimated by the transformer decoder and G / T corresponding points.

9. In paragraph 2, The above estimation process is, A process of calculating correction flows based on the deviation between the query points and the corresponding points estimated by the transformer decoder, The above first regression period and the above second regression period, A method for estimating the rotation matrix and the translation matrix, respectively, based on the query points, the corresponding points, and the correction flows.

10. In paragraph 2, The above estimation process is, In the first depth image, a process of filtering points having a depth value similar to at least one point of the second depth image; and A process of inputting a predetermined number of points selected from the filtered points into the transformer decoder as query points. A method comprising:

11. In paragraph 2, The above estimation process is, A process of inputting a predetermined number of points randomly selected from among the points of the first depth image into the transformer decoder as the query points. A method comprising:

12. In paragraph 2, A method wherein the estimated rotation matrix and the estimated translation matrix represent rotation and translation, respectively, between the coordinate system of the first depth image and the coordinate system of the second depth image.

13. In paragraph 2, A method wherein the first regressor and the second regressor each include a multi-layer perceptron (MLP).

14. In paragraph 1, The process of generating the above first depth image is as follows: A method for generating the first depth image using a pre-trained depth estimation model to estimate depth from images captured at multiple viewpoints using a single camera.

15. In paragraph 1, The process of generating the above second depth image is as follows: A process of projecting 3D data acquired from the second sensor into a 2D depth image using preset initial external parameters; and A process of applying interpolation to the projected two-dimensional depth image to generate a second depth image that is relatively dense compared to the projected two-dimensional depth image. A method comprising:

16. In paragraph 15, A method wherein applying the interpolation comprises applying a bi-lateral filter to the projected two-dimensional depth image.

17. In paragraph 1, A method further comprising a process of generating a corrected second depth image based on the estimated external parameters.

18. In paragraph 1, A method wherein the first sensor comprises a camera and the second sensor comprises a lidar.

19. At least one memory for storing instructions; and Containing at least one processor, At least one processor executes the instructions, Generate a first depth image from data from the first sensor, Generating a second depth image from data of a second sensor having a modality distinct from that of the first sensor, A device for estimating external parameters between the first sensor and the second sensor from the first depth image and the second depth image using a transformer-based network.

20. At least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to implement a network for estimating extrinsic parameters between different coordinate systems, The above network is, A transformer encoder configured to extract features from a first depth image generated from data of a first sensor and a second depth image generated from data of a second sensor having a modality distinct from the first sensor; A transformer decoder configured to estimate one or more corresponding points of the second depth image that match one or more query points of the first depth image based on features extracted from the transformer encoder; and A device comprising one or more regressors configured to estimate the external parameters based on the query points and the corresponding points.

Images