Method and system for calibrating a plurality of sensors

Abstract

The present invention provides a method and system (600) to calibrate a plurality of sensors with overlapping and non-overlapping field of views. More specifically, the present invention provides a reference sensor (610), configured to perform a step of generating a three-dimensional point cloud data of its surrounding environment with a first field of view and a second field of view, a plurality of source sensors (620a – 620d), configured to perform the step of capturing at least one depth image of its surrounding environment, and an extrinsic calibration module (640), configured perform a step calibrating the acquired three-dimensional point cloud data and the depth image data.

Description

[0001]

[0002] METHOD AND SYSTEM FOR CALIBRATING A PLURALITY OF SENSORS

[0003] FIELD OF INVENTION

[0004] The present invention relates to calibration of a plurality of sensors. More specifically, a method and system for spatial calibration of point clouds from the plurality of sensors with overlapping and non-overlapping fields of view.

[0005] BACKGROUND OF THE INVENTION

[0006] Malaysia's oil palm plantation industry is labour-intensive. Currently, there is a shortage of labour in the plantation sector, specifically in upstream processes such as harvesting, collecting fresh fruit bunch (FFB), weeding works and other general works related to plantation field works. To carry out these tasks, mobile robots, or unmanned ground vehicles (UGV) are designed to cater for outdoor environment operation setup which is designed based on a multi-sensory system.

[0007] In the context of multi-sensory systems, achieving accurate spatial calibration is imperative for precise mapping and object detection. This system should be able to provide a reliable picture of the environment for the UGV to safely navigate and carry out a mission. The calibration of the multi-sensory system e.g., multi-camera-lidar systems faces a significant challenge when the horizontal fields of view of the relative sensor and the reference sensor do not overlap completely. This lack of overlap hinders the establishment of accurate correspondences between camera images and lidar point clouds, leading to inaccurate calibration results. Further, existing calibration systems and methods are usually directed towards a specific setup, or situations like forwardfacing setup. However, existing calibration systems and methods prove inadequate for systems or methods featuring a plurality of sensors with asymmetric sensor configurations, where references are directed towards the front, right, and left cameras.

[0008] There are a few disclosed technologies over the prior art that relate to sensor calibration. Among them is a PCT application WO2022142759A1, which discloses a method for joint calibration of Lidar and camera. The calibration process involves fixing a Lidar scanner and a camera on the same base, calibrating the camera's internal parameters, placing a checkerboard calibration board in an overlapping fields of view, collecting data from both the camera and Lidar scanner, screening and fine-tuning the collected data, normalizing lidar point cloud data, performing comer detection, and obtaining transformation relationships between camera image coordinates and lidar 3D point cloud coordinates. The prior art intends to derive rotation and displacement matrices for joint calibration to enable non-repetitive scanning of the Lidar and camera.

[0009] Yet another prior art includes the China patent application CN111505606B, which discloses a method for calibrating the relative pose of a multi-camera and laser radar system. The process involves establishing a world coordinate system in a calibration field, calibrating external parameters for each sensor, converting the camera coordinate system to a laser coordinate system, and generating a three-dimensional model using laser data. Multiple cameras capture images at a certain moment, mapping them to the three-dimensional model based on the relative pose relation between cameras and the laser coordinate system. This mapping process includes steps for generating a coloured point cloud. External camera parameters are then optimised based on the characteristic matching degree between coloured point cloud overlapping areas. This enables accurate external parameters for the multi-camera system under the laser coordinate system.

[0010] However, the aforementioned prior arts fail to address the situations where the fields of view of the relative sensors and the reference sensor do not overlap completely. Moreover, they are directed towards a specific setup as disclosed by them. Accordingly, it would be desirable to have a general method and system for calibrating a plurality of sensors that involves spatial calibration of point clouds from the sensors with overlapping and non-overlapping fields of view.

[0011] SUMMARY OF INVENTION

[0012] The main objective of the invention is to provide a method and system for calibration of a plurality of sensors that involves spatial calibration of point clouds from the sensors with overlapping and non-overlapping fields of view.

[0013] To achieve this objective, the invention provides a reference sensor that is configured to perform a step of generating a three-dimensional point cloud data of its surrounding environment with a first field of view and a second field of view, a plurality of source sensors that are configured to perform the step of capturing at least one depth image of its surrounding environment, and an extrinsic calibration module that is configured perform a step calibrating the acquired three-dimensional point cloud data and the depth image data.

[0014] Advantageously, the present invention enables an increase in overlapping regions or fields of view for calibration purposes. Advantageously as well, the present invention may enable savings in terms of hardware costs compared to conventional approaches. Advantageously as well, the present invention may be utilised in implementations that employ sensors with different facing setup configurations without causing errors or incorrect axis alignments.

[0015] The present invention intends to provide a method for calibrating a plurality of sensors, which may be computer-implemented. The method is characterised by the steps of generating, by a reference sensor a three-dimensional point cloud data of its surrounding environment with a first field of view and with a second field of view, wherein the reference sensor is arranged on a platform of a vehicle, capturing, by a plurality of source sensors at least one depth image of its surrounding environment, wherein each source sensor is arranged in vicinity of the reference sensor on the platform, and calibrating, by an extrinsic calibration module the three-dimensional point cloud data and the depth image data. The extrinsic calibration module calibrates the three-dimensional point cloud data and the depth image data by determination of a location of a calibration board using the three-dimensional point cloud data for aligning axes of the reference sensor thereto, so that features of the calibration board are extracted to determine at least one extrinsic parameter for the reference sensor and the source sensors that is in relation to the calibration board, for normalisation of the said extrinsic parameter to generate at least one transformed extrinsic parameter for calibrating the source sensors.

[0016] Preferably, the step of calibrating, by an extrinsic calibration module, the three- dimensional point cloud data and the depth image data, comprises the step of determining, by a calibration board location determination module the location of the calibration board with respect to the reference sensor using the three-dimensional point cloud data.

[0017] Preferably, the step of determining, by a calibration board location determination module, the location of the calibration board with respect to the reference sensor using the three-dimensional point cloud data comprises the steps of analysing the three- dimensional point cloud data with the first field of view, instructing the platform to rotate according to the analysed three-dimensional point cloud data by 360°, for generating a virtual 360° map, and converting the generated virtual 360° map into a projected two-dimensional depth image.

[0018] Preferably, the step of determining, by a calibration board location determination module, the location of the calibration board with respect to the reference sensor using the three-dimensional point cloud data comprises the steps of analysing the three- dimensional point cloud data with the second field of view from the reference sensor, and converting the three-dimensional point cloud data into a two-dimensional depth image.

[0019] Preferably, the step of determining, by a calibration board location determination module, the location of the calibration board with respect to the reference sensor using the three-dimensional point cloud data further comprises the steps of determining the location of the calibration board in the two-dimensional depth image, defining and labelling the location of the calibration board with respect to the reference sensor, and detecting the calibration board at its determined location for identifying the source sensors.

[0020] Preferably, the step of calibrating, by an extrinsic calibration module, the three- dimensional point cloud data and the depth image data, further comprises the step of aligning, by a frame axis virtual alignment module, an axial position of the reference sensor with respect to the location of the calibration board.

[0021] Preferably, the step of aligning, by a frame axis virtual alignment module, an axial position of the reference sensor with respect to the location of the calibration board, comprises the steps of receiving the location of the calibration board and the three- dimensional point cloud, and checking whether the reference sensor is facing towards the received three-dimensional point cloud. The result from the checking is either setting an axis of the reference sensor to 0, and performing a pattern setup for the source sensors, or setting an axis of the reference sensor to 1, and rotating the reference sensor along its z-axis by 90°.

[0022] Preferably, the step of calibrating, by an extrinsic calibration module, the three- dimensional point cloud data and the depth image data, further comprises the step of normalising, by an extrinsic parameter normalisation module, the extrinsic parameter for generating the transformed extrinsic parameter.

[0023] Preferably, the step of normalising, by an extrinsic parameter normalisation module, the extrinsic parameter for generating the transformed extrinsic parameter, further comprises the steps of receiving the location of the calibration board and the determined extrinsic parameters, and checking whether the source sensors is parallel to y-axis of the reference sensor. The result from the checking is either setting normalisation to 1, translating the source sensors along their x-axis by an addition of -1, and rotating the source sensors along their z-axis by 270° in a counterclockwise direction; or setting normalisation to , translating the source sensors along their y-axis by an addition of -1, and rotating the source sensors along their z-axis by 90° in a counterclockwise direction.

[0024] The present invention further intends to provide a system for calibrating a plurality of sensors, characterised by a reference sensor arranged on a platform of a vehicle that generates a three-dimensional point cloud data of its surrounding environment with a first field of view and with a second field of view, a plurality of source sensors arranged on the platform and in the vicinity of the reference sensor that captures at least one depth image of its surrounding environment, and an extrinsic calibration module that calibrates the three-dimensional point cloud data and the depth image data. The extrinsic calibration module calibrates the three-dimensional point cloud data and the depth image data by determination of a location of a calibration board using the three- dimensional point cloud data for aligning axes of the reference sensor thereto, so that features of the calibration board are extracted to determine at least one extrinsic parameter for the reference sensor and the source sensors that is in relation to the calibration board, for normalisation of the said extrinsic parameter to generate at least one transformed extrinsic parameter for calibrating the source sensors.

[0025] One skilled in the art will readily appreciate that the invention is well adapted to carry out the objects and obtain the ends and advantages mentioned, as well as those inherent therein. The embodiments described herein are not intended as limitations on the scope of the invention.

[0026] BRIEF DESCRIPTION OF DRAWINGS

[0027] To facilitate an understanding of the invention, there are illustrated in the accompanying drawings the preferred embodiments from an inspection of which when considered in connection with the following description, the invention, its construction, and operation and many of its advantages would be readily understood and appreciated.

[0028] FIG. 1 is a flowchart illustrating the process steps to calibrate multiple sensors with overlapping and non-overlapping fields of view according to an embodiment of the present invention.

[0029] FIG. 2 is a flowchart illustrating the process steps to calibrate the acquired data from a reference sensor and a source sensor according to an embodiment of the present invention.

[0030] FIG. 3 is a flowchart illustrating the process steps to determine the location of a calibration board for the reference sensor according to an embodiment of the present invention.

[0031] FIG. 4 is a flowchart illustrating the process steps to extract features of the calibration board at its determined location and to determine an extrinsic parameter for the reference sensor and the source sensor according to an embodiment of the present invention.

[0032] FIG. 5 is a flowchart illustrating the process steps to detect the calibration board for identification of the source sensor according to an embodiment of the present invention.

[0033] FIG. 6 is a flowchart illustrating the process steps to align the axial position of the reference sensor with respect to the location of the calibration board according to an embodiment of the present invention.

[0034] FIG. 7 is a flowchart illustrating the process steps to extract features of the calibration board at its determined location and to determine an extrinsic parameter for the reference sensor and the source sensor according to an embodiment of the present invention.

[0035] FIG. 8 is a flowchart illustrating the process steps to normalize the determined extrinsic parameters according to an embodiment of the present invention.

[0036] FIG. 9 is a block diagram of a system to calibrate multiple sensors with overlapping and non-overlapping fields of view according to an embodiment of the present invention.

[0037]

[0038] DETAILED DESCRIPTION OF THE INVENTION

[0039] The present invention relates to a method and system for calibration of the multiple sensors with overlapping and non-overlapping fields of view (FOVs). The invention may also be presented in a number of different embodiments with common elements.

[0040] According to the concept of the invention, there is included a reference sensor capable of generating a three-dimensional point cloud with a full 360° view within a limited field of vision, as well as additional source sensors capturing depth images of their surroundings. Further included is a data acquisition module that collects these various inputs before sending them to an extrinsic calibration module which aligns and fuses the information into calibrated data. Finally, further included is a visualisation module that displays the calibrated data in a virtual environment.

[0041] From hereon, it is to be noted that the term “spatial calibration” preferably refers to a process of correcting a source sensor offset effect.

[0042] From hereon as well, it is to be noted that the term “reference sensor” is preferably a sensor that serves as a primary or fixed point of reference in the calibration process.

[0043] From hereon as well, it is to be noted that the term “source sensor” is preferably a sensor whose extrinsic parameters are estimated relative to the reference sensor.

[0044] From hereon as well, it is to be noted that the term “calibration board” preferably refers to a reference board that is visible in the field of view of both sensors.

[0045] From hereon as well, it is to be noted that the term “extrinsic parameter” preferably refers to a transformation parameter that defines the rigid relationship that is a rotation matrix and a translation vector between two coordinate systems.

[0046] The invention will now be described in greater detail, by way of example, with reference to the drawings.

[0047] From hereon, one or more flowcharts pertaining to the method of the present invention for training a machine learning model for image classification are to be described. It is noted that the steps described in these flowcharts are not to be interpreted as nonlimiting, and minor modifications to the steps (e.g. additions, omissions, or swaps) are permissible by a skilled person without substantial deviation from as described.

[0048] FIG. 1 illustrates a flowchart describing the process steps, Steps 100 to 500, to calibrate multiple sensors with overlapping and non-overlapping fields of view (FOVs) according to an embodiment of the present invention.

[0049] The process steps of FIG. 1 may begin with Step 100, which involves generating a three-dimensional (3D) point cloud data of its surrounding environment with a first field of view and with a second field of view, by a reference sensor 610. In particular, the first field of view is a small or limited field of view with it being less than 360°, and the second field of view is a complete field of view with it being substantially equal to 360°. Preferably, the reference sensor 610 is arranged on a platform of a vehicle. Preferably, the vehicle is an unmanned ground vehicle.

[0050] Following Step 100 is Step 200, which involves capturing at least one depth image of its surrounding environment, by at least one or a plurality of source sensors 620a - 620d. Preferably, each of the source sensors 620a - 620d is arranged in the vicinity of the reference sensor 610 on the platform and is provided with an identification code.

[0051] Following Step 200 is Step 300, which involves acquiring the generated three- dimensional point cloud data from the reference sensor 610 and the captured depth image data from the source sensors 620a - 620d, by a data acquisition module 630. Preferably, the data acquisition module 630 is arranged in communication with the reference sensor 610 and the source sensors 620a - 620d on the platform.

[0052] Following Step 300 is Step 400, which involves the step of calibrating the acquired three-dimensional point cloud data and the depth image data, by an extrinsic calibration module 640. Preferably, the extrinsic calibration module 640 is arranged in communication with the data acquisition module 630 on the platform.

[0053] Finally, following Step 400 is Step 500, which involves visualising the calibrated data in a virtual space, by a visualisation module 650. Preferably, the visualisation module 650 is arranged on the platform.

[0054] FIG. 2 illustrates a flowchart describing the process steps, Steps 400a / 400b to 490, to calibrate the acquired data from a reference sensor and source sensors according to an embodiment of the present invention. More specifically, these steps pertain to calibration of the acquired three-dimensional point cloud data and the depth image data, by the extrinsic calibration module 640.

[0055] The process steps of FIG. 2 may begin with Step 400a and / or Step 400b. Step 400a and Step 400b may occur concurrently or non-concurrently.

[0056] In particular, Step 400a involves determining the location of a calibration board with respect to the reference sensor 610 using the acquired three-dimensional point cloud data with a first field of view (FOV < 360°). In particular, Step 400b involves determining the location of a calibration board with respect to the reference sensor 610 using the acquired three-dimensional point cloud data with a second field of view (FOV = 360°). Preferably, these steps may be performed by a calibration board location determination module 641. Preferably, the calibration board is a reference object arranged in the surrounding environment visible within the field of view of the reference sensor 610 and the source sensors 620a - 620d.

[0057] Following Step 400a and / or Step 400b is Step 430, which involves aligning an axial position of the reference sensor 610 with respect to the determined location of the calibration board, by a frame axis virtual alignment module 642.

[0058] Following Step 430 is Step 450, which involves detecting and extracting a plurality of visible features of the calibration board from the determined axial position, by a co- visible feature extraction module 643.

[0059] Following Step 450 is Step 470, which involves determining at least one extrinsic parameter for the reference sensor 610 and the source sensors 620a - 620d with respect to the calibration board based on the extracted features using a registration technique, by an extrinsic parameter estimation module 644. Preferably, the extrinsic parameter is a parameter related to a rotation matrix and a translation vector between two coordinate systems.

[0060] Finally, following Step 470 is Step 490, which involves normalising the determined extrinsic parameters for generating a transformed extrinsic parameter, by an extrinsic parameter normalisation module 645.

[0061] FIG. 3 illustrates a flowchart describing the process steps, Steps 401a / 401b to 425, to determine the location of a calibration board for the reference sensor according to an embodiment of the present invention. More specifically, these steps may be performed by the calibration board location determination module 641.

[0062] The process steps of FIG. 3 may begin with Step 401a and / or Step 401b. Step 401a and Step 401b may occur concurrently or non-concurrently.

[0063] In particular, Step 401a involves reading or analysing the acquired three-dimensional point cloud data with the first field of view (FOV < 360°). In particular, Step 401b involves reading or analysing the acquired three-dimensional point cloud data with the 360° or full range field of view from the reference sensor 610.

[0064] Following Step 401a is Step 407a, which involves instructing the platform to rotate according to the analysed three-dimensional point cloud data by 360° to generate a virtual 360° map based on a multiple scan matching process. Step 407a may further involve converting the generated virtual 360° map into a two-dimensional (2D) depth image.

[0065] Following Step 401a and / or Step 407a is Step 413, which involves estimating the location of the calibration board to generate a location label.

[0066] Further included is Step 419, which involves reading or analysing the depth image data acquired from the source sensors 620a - 620d.

[0067] Finally, following Step 413 and / or Step 419 is Step 425, which involves detecting the calibration board at the estimated location to identify the identification code of the source sensors 620a - 620d to trigger the source sensors 620a - 620d.

[0068] FIG. 4 illustrates a flowchart describing the process steps, Steps 414 to 418, to extract features of the calibration board at the determined location and to determine an extrinsic parameter for the reference sensor and the source sensors according to an embodiment of the present invention. More specifically, these steps may pertain to an estimation of the determined location of the calibration board, by the calibration board location determination module 641. The process steps of FIG. 4 may be regarded as sub-steps of Step 413 of FIG. 3.

[0069] The process steps of FIG. 4 may begin with Step 414, which involves converting the three-dimensional point cloud data received from the reference sensor 610 into a projected two-dimensional depth image.

[0070] Following Step 414 is Step 415, which involves developing and training a model in offline mode to detect the calibration board by receiving the three-dimensional point cloud data and converting it into the projected two-dimensional depth image. The output from these steps is a two-dimensional trained model.

[0071] Following Step 414 or Step 415 is Step 416, which involves testing the two-dimensional trained model in online mode to receive and convert the three-dimensional point cloud data into the projected two-dimensional depth image (which was from Step 414), and detecting the calibration board using the projected two-dimensional depth image.

[0072] Following Step 416 is Step 417, which involves locating the centre of the detected calibration board.

[0073] Finally, following Step 417 is Step 418, which involves extracting image coordinates for the located centre of the calibration board.

[0074] FIG. 5 illustrates a flowchart describing the process steps, Steps 426 to 429, to detect the calibration board for identification of the source sensors according to an embodiment of the present invention. More specifically, these steps may pertain to detection of the calibration board at the estimated location to identify the source sensors 620a - 620d, by the calibration board location determination module 641. The process steps of FIG. 5 may be regarded as sub-steps of Step 425 of FIG. 3.

[0075] The process steps of FIG. 5 may begin with Step 426, which involves setting one source sensor from the plurality of source sensors 620a - 620d as a selected source sensor.

[0076] Following Step 426 is Step 427, which involves reading or analysing the two- dimensional depth image from the selected source sensor to detect the calibration board based on the shape, edge, and / or feature detection techniques.

[0077] Following Step 427 is Step 428, which is a decision step whereby it is determined if the calibration board is found at the selected source sensor.

[0078] Should this be the case, the Step 428 proceeds to Step 429. Step 429 involves assigning the location label estimated or obtained from Step 413 to the selected source sensor.

[0079] Should this not be the case, Step 428 returns to Step 427 for a subsequent source sensor from the plurality of source sensors 620a - 620d for it to be set for it to be read or analysed as per the aforementioned steps.

[0080] FIG. 6 illustrates a flowchart describing the process steps, Steps 431 to 435, to align the axial position of the reference sensor with respect to the determined location of the calibration board according to an embodiment of the present invention. More specifically, these steps may pertain to alignment of the axial position of the reference sensor 610 with respect to the determined location of the calibration board, by the frame axis virtual alignment module 642.

[0081] The process steps of FIG. 6 may begin with a preliminary step which involves receiving the determined location details of the calibration board from Step 413, and the three- dimensional point cloud data from the reference sensor 610.

[0082] Following the preliminary step is Step 431. In particular, Step 431 is a decision step whereby it is determined if the x-axis of the reference sensor (610) is facing towards the coordinates of the calibration board.

[0083] Should this be the case, Step 431 proceeds to Step 432. Step 432 involves setting the axis of the reference sensor 610, more specifically its Lidar axis, to 0.

[0084] Should this not be the case, Step 431 proceeds to Step 433. Step 433 involves setting the axis of the reference sensor 610, more specifically its Lidar axis, to 1.

[0085] Following Step 433 is Step 434, which involves converting an angle 0 into a rotation matrix. Preferably, the angle 0 is a rotation angle between calibration board centre along its z-axis and the x-axis of the reference sensor 610.

[0086] Finally, following Step 434 is Step 435, which involves rotating the reference sensor 610 along its z-axis by 90°.

[0087] FIG. 7 illustrates a flowchart describing the process steps, Steps 451 to 471, to extract features of the calibration board at the determined location and to determine an extrinsic parameter for the reference sensor and the source sensors according to an embodiment of the present invention. More specifically, these steps may pertain to an extraction of features of the calibration board at the determined location, by the co-visible feature extraction module 643.

[0088] The process steps of FIG. 7 may begin with a first preliminary step which involves receiving an input from Step 426.

[0089] Following the first preliminary step is Step 451, which involves segmenting the plane using the three-dimensional point cloud data for the reference sensor 610.

[0090] Following Step 451 is Step 452, which involves extracting features of the calibration board based on edge and / or shape detection techniques for the reference sensor 610.

[0091] The process steps of FIG. 7 may also begin with a second preliminary step which involves receiving an input from the source sensors 620a - 620d that is in the form of point cloud data.

[0092] Following the second preliminary step is Step 453, which involves segmenting planes of the point cloud data using the depth image for each source sensor 620a - 620d.

[0093] Following Step 453 is Step 454, which involves extracting the features of the calibration board based on the edge and / or shape detection techniques for the source sensors 620a - 620d

[0094] Finally, following Step 452 and / or Step 453 is Step 471, which involves estimating extrinsic parameters based on the extracted features using a registration technique, by the extrinsic parameter estimation module 644.

[0095] FIG. 8 illustrates a flowchart describing the process steps, Steps 491 to 493, to normalise the determined extrinsic parameters according to an embodiment of the present invention More specifically, these steps may pertain to normalisation of the estimated extrinsic parameters for generating at least one transformed extrinsic parameter, by the extrinsic parameter normalisation module 645.

[0096] The process steps of FIG. 8 may begin with preliminary steps which involve receiving the determined location of the calibration board from Step 413, and receiving the determined extrinsic parameters from Step 471.

[0097] Following the first preliminary step is Step 491. Step 491 is a decision step whereby it is determined if the 0 is between 135° and 225°.

[0098] Should this be the case, Step 491 proceeds to Step 492, whereby no changes to translation vector(s) are made. Preferably, a check on the source sensors 620a - 620d and the reference sensor 610 is performed. More specifically, the source sensors 620a - 620d are checked to determine those that are parallel to a y-axis of the reference sensor 610. Should there be any source sensors 620a - 620d that are parallel to the y- axis of the reference sensor 610, then, (i) the normalisation is set to 1, (ii) the source sensors 620a - 620d are translated their x-axis by an addition of -1, and (iii) the source sensors 620a - 620d are rotated along their z-axis by 270° in a counterclockwise direction.

[0099] Should this not be the case, Step 491 proceeds to Step 493, whereby normalisation calculation is performed. More specifically, (i) the normalisation is set to 2, (ii) the source sensors 620a - 620d are translated their y-axis by an addition of -1, and (iii) the source sensors 620a - 620d are rotated along their z-axis by 90° in a counterclockwise direction.

[0100] FIG. 9 is a block diagram of a system 600 to calibrate multiple sensors with overlapping and non-overlapping fields of view, which comprises the reference sensor 610, at least one or the plurality of source sensors 620a - 620d, the data acquisition module 630, the extrinsic calibration module 640, and the visualisation module 650.

[0101] The reference sensor 610 is configured to generate the three-dimensional point cloud data of its surrounding environment with a first field of view (FOV < 360°) and with a second field of view (FOV = 360°). Preferably, the reference sensor 610 is arranged on a platform of a vehicle. Preferably, the reference sensor 610 is a Light Detection and Ranging (LIDAR) sensor. Preferably, the vehicle is an unmanned ground vehicle.

[0102] The source sensors 620a - 620d are configured to capture at least one depth image of its surrounding environment. Preferably, each source sensor 620a - 620d is arranged in the vicinity of the reference sensor 610 on the platform, and is provided with the identification code. Preferably, each source sensor 620a - 620d is a depth sensor.

[0103] The data acquisition module 630 is configured to acquire the generated three- dimensional point cloud data from the reference sensor 610 and the captured depth image data from the source sensors 620a - 620d. Preferably, the data acquisition module 630 is arranged in communication with the reference sensor 610 and the source sensors 620a - 620d on the platform.

[0104] The extrinsic calibration module 640 is configured to calibrate the acquired three- dimensional point cloud data and the depth image data. Preferably, the extrinsic calibration module 640 is arranged in communication with the data acquisition module 630 on the platform. The extrinsic calibration module 640 comprises the calibration board location determination module 641, the frame axis virtual alignment module 642, the co-visible feature extraction module 643, the extrinsic parameter estimation module 644, and the extrinsic parameter normalisation module 645.

[0105] The calibration board location determination module 641 is configured to determine the location of a calibration board with respect to the reference sensor 610 using the acquired three-dimensional point cloud data. Preferably, the calibration board is the reference object arranged in the surrounding environment visible within the fields of view of the reference sensor 610 and the source sensors 620a - 620d.

[0106] In particular, the calibration board location determination module 641 is configured to perform steps as described in FIG. 3. More specifically, the calibration board location determination module 641 shall (i) read or analyse the acquired three-dimensional point cloud data with the small field of view, (ii) instruct the platform to rotate according to the analysed three-dimensional point cloud data by 360°, and (iii) generate a virtual 360° map based on a multiple scan matching process.

[0107] In particular, the calibration board location determination module 641 is further configured to perform the further steps as described in FIG. 3. More specifically, the calibration board location determination module 641 shall (i) convert the generated virtual 360° map into the two-dimensional depth image, read or analyse the acquired three-dimensional point cloud data with the 360° view from the reference sensor 610, (ii) estimate the location of the calibration board, (iii) read or analyse the depth image data acquired from the source sensors 620a - 620d, (iv) detect the calibration board at the estimated location to identify identification code of the source sensors 620a - 620d to trigger the source sensors 620a - 620d.

[0108] In particular, the calibration board location determination module 641 is further configured perform the further steps as described in FIG. 4. More specifically, the calibration board location determination module 641 shall (i) convert the three- dimensional point cloud data received from the reference sensor 610 into the projected two-dimensional depth image, (ii) develop and train the model in offline mode to detect the calibration board by receiving the three-dimensional point cloud data and converting it into the projected two-dimensional depth image, (iii) test the two- dimensional trained model in online mode to receive and convert the three-dimensional point cloud data into the projected two-dimensional depth image, (iv) detect the calibration board using the proj ected two-dimensional depth image, (v) locate the centre of the detected calibration board, and (vi) extract the image coordinates for the located centre of the calibration board.

[0109] In particular, the calibration board location determination module 641 is further configured to perform steps as described in FIG. 5. More specifically, the calibration board location determination module 641 shall (i) detect the calibration board at the estimated location to identify the identification code of the source sensors 620a - 620d, (ii) set the one source sensor from the plurality of source sensors 620a - 620d as a selected source sensor, (iii) read or analyse the two-dimensional depth image from the selected source sensor to detect the calibration board based on the shape, edge, and / or feature detection techniques. Should the calibration board location determination module 641 determine that the calibration board is found at the selected source sensor, then it shall assign the location label to the selected source sensor. Else, the calibration board location determination module 641 moves on to a subsequent source sensor from the plurality of source sensors 620a - 620d and repeats the above thereupon.

[0110] The frame axis virtual alignment module 642 is to align an axial position of the reference sensor 610 with respect to the determined location of the calibration board. In particular, the frame axis virtual alignment module 642 is configured to perform steps as described in FIG. 6. More specifically, the frame axis virtual alignment module 642 shall (i) receive the determined location coordinates details of the calibration board and the three-dimensional point cloud data from the reference sensor 610, and (ii) check whether the x-axis of the reference sensor 610 is facing towards the coordinates of the calibration board. Should the frame axis virtual alignment module 642 determine that the x-axis of the reference sensor 610 is facing towards the coordinates of the calibration board, it shall set the axis of the reference sensor 610 to zero and proceed to perform a pattern setup for the source sensors 620a - 620d. Should the frame axis virtual alignment module 642 determine that the x-axis of the reference sensor 610 is facing towards the coordinates of the calibration board, it shall set the axis of the reference sensor 610 to one, convert an angle 0 into a rotation matrix, and rotate the reference sensor 610 along its z-axis by 90°. Preferably, the angle 0 is a rotation angle between the calibration board centre along its z-axis and the x-axis of the reference sensor 610.

[0111] The co-visible feature extraction module 643 is to detect and extract a plurality of visible features of the calibration board. In particular, the co-visible feature extraction module 643 is configured perform steps as described in FIG. 7. More specifically, the co-visible feature extraction module 643 shall (i) segment the plane using the three- dimensional point cloud data for the reference sensor 610, (ii) extract features of the calibration board based on edge and / or shape detection techniques for the reference sensor 610, (iii) segment the plane using the depth image for each source sensor 620a

[0112] - 620d, and (iv) extract the features of the calibration board based on the edge and / or shape detection techniques for the source sensors 620a - 620d.

[0113] The extrinsic parameter estimation module 644 is configured to determine at least one extrinsic parameter for the reference sensor 610 and the source sensors 620a - 620d with respect to the calibration board based on the extracted features using a registration technique.

[0114] The extrinsic parameter normalisation module 645 is configured to normalise the determined extrinsic parameters for generating the transformed extrinsic parameter. In particular, the extrinsic parameter normalisation module 645 is configured to perform steps as described in FIG. 8. More specifically, the extrinsic parameter normalisation module 645 shall (i) receive the determined location of the calibration board and the determined extrinsic parameters, (ii) check whether the angle 0 is between 135° and 225°, and (iii) check which of the source sensors 620a - 620d is parallel to a y-axis of the reference sensor 610. Should the angle 0 be between 135° and 225°, there is no change to the translation vector(s). Should there be source sensors 620a - 620d that are parallel to the y-axis of the reference sensor 610, the extrinsic parameter normalisation module 645 shall set normalisation to 1, translate the source sensors 620a - 620d along their x-axis by an addition of -1, and rotate the source sensors 620a - 620d along their z-axis by 270° in a counterclockwise direction. Should there be no source sensors 620a

[0115] - 620d that are parallel to the y-axis of the reference sensor 610, the extrinsic parameter normalisation module 645 shall calculate normalisation, and shall further set normalisation to 2, translate the source sensors 620a - 620d along their y-axis by addition of -1, and rotate the source sensors 620a - 620d along their z-axis by 90° in a counterclockwise direction.

[0116] The visualisation module 650 is configured to visualise the calibrated data in the virtual space. Preferably, the visualisation module 650 is arranged on the platform.

[0117] Whilst not shown, the system 600 may further include at least one processor that is interfaced with the reference sensor 610 and the source sensors 620a - 620d. The processor may further operate the data acquisition module 630, the extrinsic calibration module 640, and the visualisation module 650. It should be noted that while the aforementioned modules may be in a software embodiment, they may also be a hardware embodiment where they are directly connected to the processor. Alternatively, these modules may each be an independent computer system. Finally, the processor may be, but shall not be limited to, a conventional processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), or a combination thereof.

[0118] The present disclosure includes as contained in the appended claims, as well as that of the foregoing description. Although this invention has been described in its preferred form with a degree of particularity, it is understood that the present disclosure of the preferred form has been made only by way of example and that numerous changes in the details of construction and the combination and arrangements of parts may be resorted to without departing from the scope of the invention.

Claims

CLAIMS1. A method for calibrating a plurality of sensors, characterised in that, the method comprising the steps of: generating, by a reference sensor (610) a three-dimensional point cloud data of its surrounding environment with a first field of view and with a second field of view, wherein the reference sensor (610) is arranged on a platform of a vehicle; capturing, by a plurality of source sensors (620a- 620d) at least one depth image of its surrounding environment, wherein each source sensor (620a - 620d) is arranged in vicinity of the reference sensor (610) on the platform; and calibrating, by an extrinsic calibration module (640) the three-dimensional point cloud data and the depth image data, wherein the extrinsic calibration module (640) calibrates the three-dimensional point cloud data and the depth image data by determination of a location of a calibration board using the three-dimensional point cloud data for aligning axes of the reference sensor (610) thereto, whereby features of the calibration board are extracted to determine at least one extrinsic parameter for the reference sensor (610) and the source sensors (620a - 620d) that is in relation to the calibration board, for normalisation of the said extrinsic parameter to generate at least one transformed extrinsic parameter for calibrating the source sensors (620a - 620d).

2. The method according to claim 1, wherein the step of calibrating, by an extrinsic calibration module (640), the three-dimensional point cloud data and the depth image data, comprises the step of determining, by a calibration board location determination module (641), the location of the calibration board with respect to the reference sensor (610) using the three-dimensional point cloud data.

3. The method according to claim 2, wherein the step of determining, by a calibration board location determination module (641), the location of the calibration board with respect to the reference sensor (610) using the three-dimensional point clouddata comprises the steps of : analysing the three-dimensional point cloud data with the first field of view that is less than 360°; instructing the platform to rotate in accordance to the analysed three- dimensional point cloud data by 360°, for generating a virtual 360° map; and converting the generated virtual 360° map into a projected two-dimensional depth image.

4. The method according to claim 3, wherein the step of determining, by a calibration board location determination module (641), the location of the calibration board with respect to the reference sensor (610) using the three-dimensional point cloud data comprises the steps of : analysing the three-dimensional point cloud data with the second field of view that is equal to 360°, from the reference sensor (610); and converting the three-dimensional point cloud data into a two-dimensional depth image.

5. The method according to claim 4, wherein the step of determining, by a calibration board location determination module (641), the location of the calibration board with respect to the reference sensor (610) using the three-dimensional point cloud data further comprises the steps of : determining location of the calibration board in the two-dimensional depth image; defining and labelling the location of the calibration board with respect to the reference sensor (610); and detecting the calibration board at its determined location for identifying the source sensors (620a - 620d).

6. The method according to claim 2, wherein the step of calibrating, by an extrinsic calibration module (640), the three-dimensional point cloud data and the depth imagedata, further comprises the step of aligning, by a frame axis virtual alignment module (642), an axial position of the reference sensor (610) with respect to the location of the calibration board.

7. The method according to claim 6, wherein the step of aligning, by a frame axis virtual alignment module (642), an axial position of the reference sensor (610) with respect to the location of the calibration board, comprises the steps of : receiving the location of the calibration board and the three-dimensional point cloud; and checking whether the reference sensor (610) is facing towards the received three-dimensional point cloud, for either setting an axis of the reference sensor (610) to 0, and performing a pattern setup for the source sensors (620a - 620d); or setting an axis of the reference sensor (610) to 1, and rotating the reference sensor (610) along its z-axis by 90°.

8. The method according to claim 2 wherein the step of calibrating, by an extrinsic calibration module (640), the three-dimensional point cloud data and the depth image data, further comprises the step of normalising, by an extrinsic parameter normalisation module (645), the extrinsic parameter for generating the transformed extrinsic parameter.

9. The method according to claim 8, wherein the step of normalising, by an extrinsic parameter normalisation module (645), the extrinsic parameter for generating the transformed extrinsic parameter, further comprises the steps of : receiving the location of the calibration board and the determined extrinsic parameters; and, checking whether the source sensors (620a - 620d) is parallel to y-axis of the reference sensor (610), for eithersetting normalisation to 1, translating the source sensors (620a - 620d) along their x-axis by an addition of -1, and rotating the source sensors (620a - 620d) along their z-axis by 270° in a counterclockwise direction; or setting normalisation to 2, translating the source sensors (620a - 620d) along their y-axis by an addition of -1, and rotating the source sensors (620a - 620d) along their z-axis by 90° in a counterclockwise direction.

10. A system (600) for calibrating a plurality of sensors, characterised in that, the system comprising of : a reference sensor (610) arranged on a platform of a vehicle, which generates a three-dimensional point cloud data of its surrounding environment with a first field of view and with a second field of view; a plurality of source sensors (620a - 620d) arranged on the platform and in a vicinity of the reference sensor (610), for capturing at least one depth image of its surrounding environment; and an extrinsic calibration module (640) for calibrating the three-dimensional point cloud data and the depth image data; wherein the extrinsic calibration module (640) calibrates the three-dimensional point cloud data and the depth image data by determination of a location of a calibration board using the three-dimensional point cloud data for aligning axes of the reference sensor (610) thereto, whereby features of the calibration board are extracted to determine at least one extrinsic parameter for the reference sensor (610) and the source sensors (620a - 620d) that is in relation to the calibration board, for normalisation of the said extrinsic parameter to generate at least one transformed extrinsic parameter for calibrating the source sensors (620a - 620d).

Images