Information processing device, information processing method, and program

The information processing device and method improve calibration accuracy by controlling measurement light irradiation and detecting corresponding points to correct the positional relationship between LiDAR and camera, addressing the challenges of sensor shifts and fixed setups in applications like autonomous driving.

WO2024228330A1PCT designated stage expired Publication Date: 2024-11-07SONY GROUP CORP

Patent Information

Application Number
PCT/JP2024/014973
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-02
Filing Date
2024-04-15
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Current methods for calibrating the positional relationship between three-dimensional sensors like LiDAR and cameras are inaccurate and cumbersome, especially when sensors shift in applications like autonomous driving, and require the use of inspection jigs or rely on movement differences, which are impractical for fixed setups.

Method used

An information processing device and method that controls the irradiation timing of measurement light from a three-dimensional sensor, detects corresponding points in camera images, and corrects parameters indicating the positional relationship between LiDAR and camera based on observation points in a three-dimensional coordinate system and their corresponding points in a two-dimensional camera image system.

Benefits of technology

Enables precise and efficient calibration of the positional relationship between LiDAR and camera without the need for inspection jigs or sensor movement, maintaining high accuracy even in moving systems like vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024014973_07112024_PF_FP_ABST
    Figure JP2024014973_07112024_PF_FP_ABST
Patent Text Reader

Abstract

The present technology relates to an information processing device, an information processing method, and a program that make it possible to easily and highly accurately perform calibration of the positional relationship between a three-dimensional sensor and a camera. The information processing device of the present technology comprises: an irradiation control unit for controlling the irradiation timing of measurement light of a three-dimensional sensor; a corresponding-point detection unit for detecting, in a camera image acquired by a camera, a corresponding point corresponding to an observation point detected by the three-dimensional sensor by using the measurement light; and a correction unit for correcting a parameter, which indicates the positional relationship between the three-dimensional sensor and the camera, on the basis of the position of the observation point in a prescribed three-dimensional coordinate system and the position of the corresponding point in a two-dimensional coordinate system of the camera image. The present technology can be applied to, for example, a system in which LiDAR and a camera are used.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program suitable for use when calibrating the positional relationship between a three-dimensional sensor and a camera.

[0002] Currently, sensing technology using cameras and LiDAR (Light Detection and Ranging) is being developed and spread.

[0003] For example, in autonomous driving technology, sensor fusion technology that uses cameras and LiDAR simultaneously is used.

[0004] For example, sensing technology using cameras and LiDAR is used in background asset generation technology for the metaverse and movies. For example, in background asset generation technology, a three-dimensional spatial structure is generated with high precision using LiDAR, and a coloring process is performed using camera images, thereby creating a realistic space that differs from CG (Computer Graphics).

[0005] To improve the accuracy and quality of these techniques, the positional relationship between the camera and the LiDAR is calibrated, i.e., the parameters indicating the positional relationship between the camera and the LiDAR are corrected (adjusted).

[0006] For example, calibration is performed using a dedicated inspection jig such as a chart or board (see, for example, Patent Document 1).

[0007] For example, the amount of movement calculated by the camera and LiDAR is compared, and calibration is performed based on the difference in the amount of movement (see, for example, Patent Document 2).

[0008] JP 2021-38939 A International Publication No. 2022 / 190169

[0009] However, when using an inspection jig, for example, if a misalignment between sensors occurs after shipping a vehicle using sensor fusion technology, it is difficult to perform calibration. Also, for example, with background asset generation technology, it is necessary to capture a scene including the inspection jig and a scene without the inspection jig.

[0010] When calibration is performed based on the difference in the amount of movement between sensors, the accuracy of the calibration depends on the accuracy of detecting the amount of movement of the sensors. Also, when the sensors are fixed and do not move, as in background asset generation technology, it is difficult to perform calibration.

[0011] The present technology was developed in consideration of such circumstances, and enables calibration of the positional relationship between a three-dimensional sensor such as LiDAR and a camera to be performed easily and with high accuracy.

[0012] An information processing device according to one aspect of the present technology includes an irradiation control unit that controls the irradiation timing of the measurement light of a three-dimensional sensor, a corresponding point detection unit that detects a corresponding point in a camera image acquired by the camera that corresponds to an observation point detected by the three-dimensional sensor using the measurement light, and a correction unit that corrects parameters indicating the positional relationship between the LiDAR and the camera based on the position of the observation point in a predetermined three-dimensional coordinate system and the position of the corresponding point in a two-dimensional coordinate system of the camera image.

[0013] An information processing method of one aspect of the present technology controls the timing of irradiation of measurement light from a three-dimensional sensor, detects a corresponding point in a camera image acquired by the camera that corresponds to an observation point detected by the three-dimensional sensor using the measurement light, and corrects parameters indicating the positional relationship between the LiDAR and the camera based on the position of the observation point in a predetermined three-dimensional coordinate system and the position of the corresponding point in the two-dimensional coordinate system of the camera image.

[0014] A program of one aspect of the present technology causes a computer to control the timing of irradiation of measurement light from a three-dimensional sensor, detect a corresponding point in a camera image acquired by the camera that corresponds to an observation point detected by the three-dimensional sensor using the measurement light, and perform a process of correcting parameters indicating the positional relationship between the LiDAR and the camera based on the position of the observation point in a predetermined three-dimensional coordinate system and the position of the corresponding point in the two-dimensional coordinate system of the camera image.

[0015] In one aspect of the present technology, the timing of irradiation of the measurement light of a three-dimensional sensor is controlled, and in a camera image acquired by a camera, a corresponding point corresponding to an observation point detected by the three-dimensional sensor using the measurement light is detected, and parameters indicating the positional relationship between the LiDAR and the camera are corrected based on the position of the observation point in a predetermined three-dimensional coordinate system and the position of the corresponding point in the two-dimensional coordinate system of the camera image.

[0016] 7 is a block diagram showing a first embodiment of an information processing system to which the present technology is applied. FIG. 8 is a block diagram showing a first embodiment of the data processing device of FIG. 1. FIG. 9 is a block diagram showing an example configuration of functions implemented by the processor of FIG. 2. FIG. 10 is a flowchart for explaining a first embodiment of calibration processing. FIG. 11 is a flowchart for explaining a first embodiment of observation point / corresponding point detection processing. FIG. 12 is a flowchart for explaining a first embodiment of positional relationship information correction processing. FIG. 13 is a block diagram showing a second embodiment of the data processing device of FIG. 1. FIG. 14 is a block diagram showing an example configuration of functions implemented by the processor of FIG. 7. FIG. 15 is a flowchart for explaining a second embodiment of calibration processing. FIG. 16 is a block diagram showing a second embodiment of an information processing system to which the present technology is applied. FIG. 17 is a block diagram showing an example configuration of the data processing device of FIG. 10. FIG. 18 is a flowchart for explaining a second embodiment of observation point / corresponding point detection processing. FIG. 19 is a flowchart for explaining a second embodiment of positional relationship information correction processing. FIG. 19 is a block diagram showing a modified example of an information processing system to which the present technology is applied. FIG. 20 is a block diagram showing an example configuration of a computer.

[0017] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. First embodiment 2. Second embodiment (example of checking misalignment) 3. Third embodiment (example of using an EVS (Event-based Vision Sensor)) 4. Modification 5. Other

[0018] <<1. First Embodiment>> First, a first embodiment of the present technology will be described with reference to FIGS. 1 to 6 .

[0019] <Configuration Example of Vehicle Control System> FIG. 1 shows a configuration example of an information processing system 1 to which the present technology is applied.

[0020] The information processing system 1 is applied to a system or device that uses sensing technology using a LiDAR 11 and a camera 12, such as an autonomous driving technology or a background asset generation technology. The information processing system 1 includes the LiDAR 11, the camera 12, and a data processing device 13.

[0021] A LiDAR that scans measurement light consisting of pulsed laser light (laser pulses) within a predetermined field of view (FOV) (hereinafter referred to as LiDAR FOV) is used as the LiDAR 11. Specifically, for example, a mechanical rotation type LiDAR equipped with a drive motor or a solid-state type LiDAR such as a MEMS (Micro Electro Mechanical Systems) mirror type is used as the LiDAR 11.

[0022] Note that LiDARs that do not scan measurement light (for example, flash type) are not used in LiDAR 11.

[0023] The LiDAR 11 scans the LiDAR FOV with measurement light and detects the position (coordinates) in a predetermined three-dimensional coordinate system (hereinafter referred to as the LiDAR coordinate system) of each point (hereinafter referred to as an observation point) detected by the measurement light being reflected by an object in the LiDAR FOV. The LiDAR 11 generates a point cloud (point cloud data) indicating the distribution of the observation points in the LiDAR coordinate system and supplies it to the data processing device 13. The LiDAR 11 can also generate information including the individual detected positions of each observation point in the LiDAR coordinate system (hereinafter referred to as observation point information) and supply it to the data processing device 13.

[0024] The LiDAR coordinate system may be a global coordinate system or a coordinate system dedicated to the LiDAR 11.

[0025] The camera 12 captures an image within a predetermined FOV (hereinafter referred to as the camera FOV). The camera FOV at least partially overlaps with the LiDAR FOV. The camera 12 supplies data (hereinafter referred to as camera image data) representing an image obtained as a result of the capture (hereinafter referred to as the camera image) to the data processing device 13.

[0026] The data processing device 13 performs various types of data processing based on the point cloud and camera image data. For example, the data processing device 13 performs object detection processing, object recognition processing, object movement prediction processing, background asset generation processing, etc.

[0027] Furthermore, based on the observation point information and camera image data, the data processing device 13 calibrates the positional relationship between the LiDAR 11 and the camera 12. That is, the data processing device 13 executes a process of correcting (adjusting) parameters indicating the positional relationship (relative position and attitude) between the LiDAR 11 and the camera 12.

[0028] In the following, the calibration of the positional relationship between LiDAR 11 and camera 12 may be simply referred to as calibration of LiDAR 11 and camera 12.

[0029] <Configuration Example of Data Processing Device 13a> Next, a configuration example of a data processing device 13a which is a first embodiment of the data processing device 13 in Fig. 1 will be described with reference to Fig. 2 and Fig. 3. Fig. 2 shows a configuration example of the data processing device 13a. Fig. 3 shows a configuration example of functions implemented by the processor 51 of the data processing device 13a.

[0030] The data processing device 13 includes a processor 51, a main storage device 52, an auxiliary storage device 53, and a communication device 54. The processor 51, the main storage device 52, the auxiliary storage device 53, and the communication device 54 are connected to each other via a bus 55.

[0031] The processor 51 includes, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and the like.

[0032] The main storage device 52 includes, for example, a RAM (Random Access Memory).

[0033] The auxiliary storage device 53 includes, for example, a hard disk drive (HDD), a flash memory, a solid state drive (SSD), and the like.

[0034] The communication device 54 includes, for example, an electrical circuit having a communication function. The communication device 54 communicates with external devices such as the LiDAR 11 and the camera 12. The communication method used by the communication device 54 may be either wireless communication or wired communication.

[0035] The processor 51 loads a program stored in the auxiliary storage device 53 into the main storage device 52 and executes the program, thereby realizing functions including an irradiation control unit 61, a corresponding point detection unit 62, and a correction unit 63.

[0036] Note that only the functions that perform processing related to the calibration of LiDAR 11 and camera 12 are illustrated here, and functions that perform other processing are omitted from the illustration.

[0037] The irradiation control unit 61 controls the irradiation of the measurement light of the LiDAR 11 by supplying a LiDAR control signal to the LiDAR 11 via the bus 55 and the communication device 54. For example, the irradiation control unit 61 controls the irradiation direction and irradiation timing of the measurement light of the LiDAR 11. Furthermore, the irradiation control unit 61 notifies the corresponding point detection unit 62 of the irradiation direction and irradiation timing of the measurement light by supplying the LiDAR control signal to the corresponding point detection unit 62.

[0038] The corresponding point detection unit 62 executes a process of detecting corresponding points in the camera image that correspond to the observation points of the LiDAR 11, based on the observation point information from the LiDAR 11 and the camera image data from the camera 12. The corresponding point detection unit 62 supplies observation point / corresponding point information indicating the detection results of the observation points and corresponding points to the correction unit 63. Furthermore, when performing the corresponding point detection process, the corresponding point detection unit 62 instructs the irradiation control unit 61 to irradiate measurement light.

[0039] Based on the observation point / corresponding point information, the correction unit 63 corrects the parameters indicating the positional relationship between the LiDAR 11 and the camera 12. The correction unit 63 outputs positional relationship information including the corrected parameters.

[0040] In the following, the phrase "via the bus 55" will be omitted when each unit of the data processing device 13a exchanges data via the bus 55. For example, when the irradiation control unit 61 and the main storage device 52 exchange data via the bus 55, it will be simply stated that the irradiation control unit 61 and the main storage device 52 exchange data. In the following, the phrase "via the bus 55 and the communication device" will be omitted when each unit of the data processing device 13a communicates with the outside via the bus 55 and the communication device 54. For example, when the irradiation control unit 61 communicates with the LiDAR 11 via the bus 55 and the communication device 54, it will be simply stated that the irradiation control unit 61 communicates with the LiDAR 11. The same applies to the other embodiments.

[0041] <Calibration Process> Next, the calibration process executed by the information processing system 1 will be described with reference to the flowchart of FIG.

[0042] In step S1, the information processing system 1 executes an observation point / corresponding point detection process.

[0043] Here, the observation point / corresponding point detection process will be described in detail with reference to the flowchart of FIG.

[0044] In step S21, the LiDAR 11 irradiates measurement light under the control of the irradiation control unit 61. Specifically, the corresponding point detection unit 62 instructs the irradiation control unit 61 to irradiate measurement light. The irradiation control unit 61 supplies a LiDAR control signal to the LiDAR 11 and the corresponding point detection unit 62. The LiDAR 11 irradiates measurement light based on the LiDAR control signal.

[0045] Here, the irradiation direction of the measurement light is set to a different direction each time the process of step S21 is performed. For example, the irradiation direction of the measurement light may be moved according to the scanning order during normal operation (e.g., when creating a point cloud), or may be moved in a direction different from that during normal operation.

[0046] Furthermore, the timing of irradiating the measurement light is controlled so that the measurement light is irradiated a maximum of once within each exposure period of the camera 12. Therefore, the irradiation frequency of the measurement light of the LiDAR 11 is equal to or less than the frame rate of the camera 12. In other words, the irradiation period of the measurement light of the LiDAR 11 is equal to or greater than the frame interval (exposure period) of the camera 12.

[0047] In step S22, the LiDAR 11 detects an observation point. Specifically, the LiDAR 11 detects the position in the LiDAR coordinate system of the observation point detected by reflecting the measurement light irradiated in the processing of step S21. The LiDAR 11 generates observation point information including the detected position of the observation point and supplies it to the processor 51.

[0048] In step S23, the camera 12 acquires a camera image. Specifically, the camera 12 captures an image within the camera FOV. At this time, as described above, measurement light is irradiated from the LiDAR 11 within the exposure period of the camera 12. The camera 12 supplies the camera image data obtained by capturing an image within the camera FOV to the processor 51.

[0049] In step S24, the corresponding point detection unit 62 detects a corresponding point that corresponds to the observation point in the camera image.

[0050] Here, it is assumed that the area in the camera image corresponding to the observation point, i.e., the area where the measurement light is reflected, has high brightness. In response to this, for example, the corresponding point detection unit 62 detects the pixel in the camera image with the highest brightness as the corresponding point.

[0051] Note that, for example, due to factors such as a blurred image of a point that appears to be an observation point in a camera image, simply detecting the pixel with the highest brightness as the corresponding point may result in an erroneous detection of the corresponding point.

[0052] In response to this, for example, the corresponding point detection unit 62 may apply NMS (Non Maximum Suppression) to pixels in the camera image whose brightness is equal to or greater than a predetermined threshold, and detect the pixel with the maximum brightness from among those pixels as the corresponding point.

[0053] Furthermore, for example, the corresponding point detection unit 62 may detect pixels in the camera image whose brightness is equal to or greater than a predetermined threshold as corresponding points. In this case, multiple corresponding points may be detected from one camera image, or no corresponding points may be detected.

[0054] For example, the corresponding point detection unit 62 may apply NMS to pixels in the camera image whose brightness is equal to or greater than a predetermined threshold, and detect pixels whose brightness is equal to or greater than the predetermined threshold as corresponding points.

[0055] In step S25, the corresponding point detection unit 62 determines whether or not a predetermined number of corresponding points have been detected. If it is determined that the predetermined number of corresponding points has not yet been detected, the process returns to step S21.

[0056] Thereafter, in step S25, the processes of steps S21 to S25 are repeatedly executed until it is determined that a predetermined number or more of corresponding points have been detected. As a result, a plurality of corresponding points corresponding to a plurality of observation points detected using measurement light irradiated at different irradiation timings are detected in different camera images. In addition, a predetermined number or more pairs of observation points and corresponding points (hereinafter referred to as observation point-corresponding point pairs) are detected.

[0057] On the other hand, if it is determined in step S25 that the predetermined number or more of corresponding points have been detected, the process proceeds to step S26.

[0058] In step S26, the corresponding point detection unit 62 outputs information regarding the observation points and corresponding points. Specifically, the corresponding point detection unit 62 generates observation point / corresponding point information indicating the detection results of the observation points and corresponding points. The observation point / corresponding point information includes, for example, position information of each observation point / corresponding point pair. The position information of each observation point / corresponding point pair includes identification information for identifying each pair (e.g., the image ID of the camera image in which the corresponding point was detected), the position (coordinates) of the observation point in the LiDAR coordinate system, and the position (coordinates) of the corresponding point in the two-dimensional coordinate system of the camera image (hereinafter referred to as the camera image coordinate system). The corresponding point detection unit 62 supplies the observation point / corresponding point information to the correction unit 63.

[0059] Thereafter, the observation point / corresponding point detection process ends.

[0060] Returning to FIG. 4, in step S2, the data processing device 13 executes the positional relationship information correction process, and the calibration process ends.

[0061] Here, the positional relationship information correction process will be described in detail with reference to the flowchart of FIG.

[0062] In step S41, the correction unit 63 selects pairs of observation points and corresponding points.

[0063] Here, the corresponding points detected in the camera image may include erroneously detected pixels that do not actually correspond to the observation point.

[0064] In response to this, the correction unit 63 selects pairs of observation points and corresponding points by, for example, performing inlier estimation using RANSAC (RANdom Sample Consensus), and removes pairs that include corresponding points that are estimated to have been erroneously detected.

[0065] Specifically, for example, correction unit 63 selects three pairs of observation point and corresponding point pairs included in the observation point and corresponding point information. Here, if multiple corresponding points are detected in one camera image, three pairs of observation point and corresponding point pairs are selected so that pairs including corresponding points detected in the same camera image are not simultaneously included. In other words, three pairs of observation point and corresponding point pairs each including corresponding points detected in different camera images are selected.

[0066] The correction unit 63 estimates the position and orientation of the camera 12 in the LiDAR coordinate system by solving P3P using the three selected observation point / correspondence point pairs. The correction unit 63 projects each observation point included in the observation point / correspondence point information onto the camera image coordinate system based on the estimated position and orientation of the camera 12. The correction unit 63 calculates the position error (reprojection error) between the projection point obtained by projecting each observation point onto the camera image coordinate system and the corresponding point corresponding to each observation point (the corresponding point included in the same observation point / correspondence point pair). The correction unit 63 extracts observation point / correspondence point pairs for which the reprojection error is equal to or less than a threshold, and counts the number of extracted observation point / correspondence point pairs.

[0067] For example, the correction unit 63 performs the above-described processing for a predetermined number of three or more pairs of observation points and corresponding points, and detects the combination of observation points and corresponding points that maximizes the number of corresponding points for which the reprojection error is below a threshold.

[0068] It is also possible to perform the above-described processing for all three combinations of observation point / corresponding point pairs, excluding combinations that simultaneously include pairs containing corresponding points detected in the same camera image, and detect the combination of observation point / corresponding point pairs that maximizes the number of corresponding points for which the reprojection error is equal to or less than the threshold value.

[0069] Hereinafter, the combination of observation points and corresponding points pairs that maximizes the number of observation points and corresponding points pairs for which the reprojection error is equal to or less than the threshold value will be referred to as the “optimal combination of observation points and corresponding points pairs.” Furthermore, hereafter, the position and orientation of the camera 12 estimated based on the optimal combination of observation points and corresponding points pairs will be referred to as the “tentative position and tentative orientation of the camera 12.”

[0070] The correction unit 63 extracts observation point / corresponding point pairs for which the reprojection error is equal to or less than a threshold value at the tentative position and tentative attitude of the camera 12. The correction unit 63 also removes pairs other than the extracted observation point / corresponding point pairs from the observation point / corresponding point information. As a result, observation point / corresponding point pairs that include corresponding points that are likely to have been erroneously detected are removed from the observation point / corresponding point information.

[0071] In step S42, the correction unit 63 corrects parameters indicating the positional relationship between the LiDAR 11 and the camera 12. For example, the correction unit 63 optimizes parameters indicating the position and attitude of the camera 12 in the LiDAR coordinate system by solving PnP using the observation point / corresponding point pairs extracted in the processing of step S41.

[0072] Specifically, correction unit 63 projects each observation point of the observation point-corresponding point pair extracted in the processing of step S41 onto the camera image coordinate system based on the tentative position and tentative orientation of camera 12 that are based on the optimal combination of observation point-corresponding point pair. Correction unit 63 corrects parameters that indicate the position and orientation of camera 12 in the LiDAR coordinate system so that the sum or average value of the errors (reprojection errors) between the projection points obtained by projecting each observation point onto the camera image coordinate system and the corresponding points corresponding to each observation point is minimized.

[0073] The correction unit 63 corrects the parameters indicating the positional relationship between the LiDAR 11 and the camera 12 based on the position and attitude of the camera 12 in the corrected LiDAR coordinate system.

[0074] The type of parameters indicating the positional relationship between the LiDAR 11 and the camera 12 is not particularly limited. For example, the parameters indicating the positional relationship between the LiDAR 11 and the camera 12 may include parameters indicating the position and orientation of the camera 12 in the LiDAR coordinate system. For example, the parameters indicating the positional relationship between the LiDAR 11 and the camera 12 may include a translation matrix and a rotation matrix indicating the relative position and orientation of the LiDAR 11 and the camera 12. For example, the parameters indicating the positional relationship between the LiDAR 11 and the camera 12 may include a transformation matrix used to transform between the LiDAR coordinate system and the camera image coordinate system.

[0075] The correction unit 63 generates positional relationship information including parameters indicating the corrected positional relationship between the LiDAR 11 and the camera 12.

[0076] In step S43, the correction unit 63 outputs the positional relationship information between the LiDAR 11 and the camera 12. For example, the correction unit 63 outputs the positional relationship information and stores it in the main storage device 52 or the auxiliary storage device 53.

[0077] The positional relationship information stored in the main storage device 52 or the auxiliary storage device 53 is used as appropriate in subsequent processing.

[0078] Thereafter, the positional relationship information correction process ends.

[0079] In this way, it is possible to easily and highly accurately calibrate the positional relationship between the LiDAR 11 and the camera 12. In other words, it is possible to directly and highly accurately calibrate the positional relationship between the LiDAR 11 and the camera 12 without using an inspection jig or moving the LiDAR 11 or the camera 12.

[0080] <<2. Second Embodiment>> Next, a second embodiment of the present technology will be described with reference to FIGS. 7 to 9 .

[0081] <Configuration Example of Data Processing Device 13b> First, a configuration example of a data processing device 13b, which is a second embodiment of the data processing device 13 in Fig. 1, will be described with reference to Fig. 7 and Fig. 8. Fig. 7 shows a configuration example of the data processing device 13b. Fig. 8 shows a configuration example of functions implemented by the processor 51 of the data processing device 13b.

[0082] In the figure, parts corresponding to those in FIGS. 2 and 3 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0083] The hardware configuration of the data processing device 13b is similar to that of the data processing device 13a in FIG.

[0084] On the other hand, the functions realized by the processor 51 of the data processing device 13b differ from the functions realized by the processor 51 of the data processing device 13a in that a deviation detection unit 101 is added.

[0085] Note that only the functions that perform processing related to the calibration of LiDAR 11 and camera 12 are illustrated here, and functions that perform other processing are omitted from the illustration.

[0086] The corresponding point detection unit 62 supplies the deviation detection unit 101 with observation point / corresponding point information indicating the detection results of the observation points and corresponding points.

[0087] The displacement detection unit 101 checks for physical positional displacement between the LiDAR 11 and the camera 12 based on the observation point / correspondence point information. When the displacement detection unit 101 detects a positional displacement between the LiDAR 11 and the camera 12, it supplies the observation point / correspondence point information to the correction unit 63 and instructs the correction unit 63 to correct the positional relationship information between the LiDAR 11 and the camera 12. Furthermore, when checking for a positional displacement between the LiDAR 11 and the camera 12, the displacement detection unit 101 instructs the correspondence point detection unit 62 to detect corresponding points.

[0088] <Calibration Process> Next, the calibration process executed by the information processing system 1 including the data processing device 13b will be described with reference to the flowchart of FIG.

[0089] In step S101, the deviation detection unit 101 determines whether it is time to check for deviation between the LiDAR 11 and the camera 12. The determination process of step S101 is repeatedly executed until it is determined that it is time to check for deviation between the LiDAR 11 and the camera 12.

[0090] On the other hand, if the deviation detection unit 101 determines that it is time to check the positional deviation between the LiDAR 11 and the camera 12, it instructs the corresponding point detection unit 62 to detect corresponding points. Then, the processing proceeds to step S102.

[0091] The timing for checking the positional deviation between the LiDAR 11 and the camera 12 can be set arbitrarily. For example, the positional deviation check may be performed at a predetermined cycle or a predetermined time period. For example, the positional deviation check may be performed when a predetermined condition is satisfied. Specifically, the positional deviation check may be performed when the results of the subsequent processing deteriorate (for example, when the accuracy or certainty of the object detection processing, object recognition processing, or object movement prediction processing falls below a predetermined threshold). For example, when the information processing system 1 is installed in a vehicle, the positional deviation check may be performed when the vehicle starts, stops, etc.

[0092] In step S102, an observation point / corresponding point detection process is executed in the same manner as in step S1 in Fig. 4. In this process, the observation point / corresponding point information generated by the corresponding point detection unit 62 is supplied to the deviation detection unit 101.

[0093] In step S103, the displacement detection unit 101 projects each observation point onto the camera image coordinate system and calculates a reprojection error. Specifically, the displacement detection unit 101 projects each observation point onto the camera image coordinate system based on parameters that indicate the current positional relationship between the LiDAR 11 and the camera 12. The displacement detection unit 101 calculates the positional error (reprojection error) between the projection point obtained by projecting each observation point onto the camera image coordinate system and the corresponding point corresponding to each observation point.

[0094] In step S104, the displacement detection unit 101 determines whether or not a positional displacement has occurred between the LiDAR 11 and the camera 12. First, the displacement detection unit 101 selects pairs of projection points and corresponding points. Specifically, the displacement detection unit 101 removes pairs of projection points and corresponding points for which the reprojection error calculated in the processing of step S103 is equal to or greater than a predetermined threshold. This removes pairs of corresponding points and projection points that are likely to have been erroneously detected.

[0095] The displacement detection unit 101 calculates a statistical value (for example, an average value or a maximum value) of the reprojection error of the remaining projection point / corresponding point pairs. If the statistical value of the reprojection error is less than a predetermined threshold, the displacement detection unit 101 determines that no positional displacement has occurred between the LiDAR 11 and the camera 12, and the process returns to step S101.

[0096] Then, in step S104, the processing of steps S101 to S104 is repeatedly executed until it is determined that a positional deviation has occurred between the LiDAR 11 and the camera 12.

[0097] On the other hand, in step S104, if the statistical value of the reprojection error is greater than or equal to a predetermined threshold, the deviation detection unit 101 determines that a positional deviation has occurred between the LiDAR 11 and the camera 12, and the processing proceeds to step S105.

[0098] In step S105, the displacement detection unit 101 outputs information about the observation points and corresponding points. Specifically, the displacement detection unit 101 removes the observation point-corresponding point pairs removed in the processing of step S104 from the observation point-corresponding point information. In other words, observation point-corresponding point pairs that include corresponding points that are likely to have been erroneously detected are removed. The displacement detection unit 101 supplies the observation point-corresponding point information that includes the remaining observation point-corresponding point pairs to the correction unit 63.

[0099] In step S106, the positional relationship information correction process is executed in the same manner as in step S2 of FIG.

[0100] Thereafter, the process returns to step S101, and the processes from step S101 onwards are executed.

[0101] In this manner, it is possible to automatically detect positional deviation between the LiDAR 11 and the camera 12 and perform calibration of the positional relationship between the LiDAR 11 and the camera 12. As a result, even if, for example, the LiDAR 11 and the camera 12 are installed on a moving body such as a vehicle and positional deviation is likely to occur, it is possible to maintain the parameters indicating the positional relationship between the LiDAR 11 and the camera 12 with high accuracy.

[0102] <<3. Third Embodiment>> Next, a third embodiment of the present technology will be described with reference to FIGS. 10 to 13 .

[0103] 10 shows a configuration example of an information processing system 201 to which the present technology is applied. In the drawing, parts corresponding to those in the information processing system 1 in FIG. 1 are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0104] The information processing system 201 differs from the information processing system 1 in that an EVS 211 is added.

[0105] The EVS 211 supplies the data processing device 13 with the position of a pixel (hereinafter referred to as a reaction point) whose brightness has changed by more than a predetermined threshold, the detection time of the reaction point, and a signal (hereinafter referred to as a reaction point signal) indicating the polarity of the reaction point.

[0106] The position of the reaction point is indicated by coordinates in a two-dimensional coordinate system (hereinafter referred to as the EVS image coordinate system) of the image (hereinafter referred to as the EVS image) generated by the EVS 211. The polarity of the reaction point indicates whether the brightness of the reaction point has changed in a positive or negative direction.

[0107] In the following description, it is assumed that the calibration of the positional relationship between the camera 12 and the EVS 211 has been performed in advance, and parameters indicating the positional relationship between the camera 12 and the EVS 211 have been acquired in advance.

[0108] <Configuration example of data processing device 13c> Fig. 11 shows a third embodiment of the data processing device 13 in Fig. 1, and shows a configuration example of a data processing device 13c used in the information processing system 201 in Fig. 10. In the figure, parts corresponding to those in Fig. 2 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0109] The hardware configuration of the data processing device 13c is similar to that of the data processing device 13a in FIG.

[0110] The processor 51 loads a program stored in the auxiliary storage device 53 into the main storage device 52 and executes the program, thereby realizing functions including a corresponding point detection unit 251 and a correction unit 252 .

[0111] Note that only the functions that perform processing related to the calibration of LiDAR 11 and camera 12 are illustrated here, and functions that perform other processing are omitted from the illustration.

[0112] The corresponding point detection unit 251 executes a process of detecting corresponding points, which are reaction points corresponding to the observation points of the LiDAR 11, based on the observation point information from the LiDAR 11 and the reaction point signals from the EVS 211. The corresponding point detection unit 251 supplies the correction unit 252 with observation point / corresponding point information indicating the detection results of the observation points and corresponding points.

[0113] Based on the observation point / corresponding point information, the correction unit 252 corrects parameters indicating the positional relationship between the LiDAR 11 and the EVS 211. Furthermore, based on the parameters indicating the corrected positional relationship between the LiDAR 11 and the EVS 211 and the parameters indicating the known positional relationship between the camera 12 and the EVS 211, the correction unit 252 corrects parameters indicating the positional relationship between the LiDAR 11 and the camera 12. The correction unit 252 outputs positional relationship information including the corrected parameters.

[0114] <Calibration Processing> Next, the calibration processing executed by the information processing system 201 will be described with reference to the flowchart of FIG.

[0115] In step S1, the information processing system 201 executes an observation point / corresponding point detection process.

[0116] Here, the observation point / corresponding point detection process will be described in detail with reference to the flowchart of FIG.

[0117] In step S101, the LiDAR 11 starts the process of detecting an observation point. Specifically, the LiDAR 11 starts scanning the measurement light and detecting the position in the LiDAR coordinate system of the observation point detected by the reflected measurement light. The scanning of the measurement light at this time does not need to be synchronized with the camera 12 and the EVS 211, and can be the same as during normal operation (e.g., when creating a point cloud). The LiDAR 11 also starts the process of generating observation point information including the detected position and detection time of the observation point and supplying it to the processor 51.

[0118] In step S102, the EVS 211 starts the detection process of the reaction points. Specifically, the EVS 211 detects the reaction points, generates a reaction point detection signal indicating the position of the reaction point in the EVS image coordinate system, the detection time of the reaction point, and the polarity of the reaction point, and starts the process of supplying the signal to the processor 51.

[0119] In step S103, the corresponding point detection unit 251 detects a corresponding point corresponding to the observation point based on the detection time. Specifically, the corresponding point detection unit 251 detects a reaction point detected at approximately the same time as the observation point based on the observation point information and the reaction point signal as a corresponding point corresponding to the observation point.

[0120] In step S104, the corresponding point detection unit 251 determines whether or not a predetermined number of corresponding points have been detected. If it is determined that the predetermined number of corresponding points have not yet been detected, the process returns to step S103.

[0121] Thereafter, the processes of steps S103 and S104 are repeatedly executed until it is determined in step S104 that a predetermined number or more of corresponding points have been detected, thereby detecting pairs of observation points and corresponding points of a predetermined number or more.

[0122] On the other hand, if it is determined in step S104 that the predetermined number or more of corresponding points have been detected, the process proceeds to step S105.

[0123] In step S105, the corresponding point detection unit 251 outputs information regarding the observation points and corresponding points. Specifically, the corresponding point detection unit 251 generates observation point / corresponding point information that indicates the detection results of the observation points and corresponding points. The observation point / corresponding point information includes, for example, position information of each observation point / corresponding point pair. The position information of each observation point / corresponding point pair includes identification information for identifying each pair, the position (coordinates) of the observation point in the LiDAR coordinate system, and the position (coordinates) of the corresponding point in the EVS image coordinate system. The corresponding point detection unit 251 supplies the observation point / corresponding point information to the correction unit 252.

[0124] Thereafter, the observation point / corresponding point detection process ends.

[0125] Returning to FIG. 4, the data processing device 13c executes the positional relationship information correction process.

[0126] Here, the positional relationship information correction process will be described in detail with reference to the flowchart of FIG.

[0127] In step S121, pairs of observation points and corresponding points are selected by processing similar to that in step S41 of Fig. 6. Here, the processing in step S41 and the processing in step S121 differ in that the coordinate systems of the corresponding points are the camera image coordinate system and the EVS image coordinate system, but it is possible to select pairs of observation points and corresponding points by processing similar to that in step S41 and step S121.

[0128] In step S122, the parameters indicating the positional relationship between the LiDAR 12 and the EVS 211 are corrected by processing similar to that of step S42 in Fig. 6. Here, the processing of step S42 and the processing of step S122 differ in that the calibration targets are the camera 12 and the EVS 211, but the parameters indicating the positional relationship between the LiDAR 12 and the EVS 211 can be corrected by similar processing.

[0129] In step S123, the correction unit 252 corrects the parameter indicating the positional relationship between the LiDAR 11 and the camera 12. For example, the correction unit 252 corrects the parameter P le, and a parameter P indicating the known positional relationship between the EVS 211 and the camera 12 ec Based on this, a parameter P indicating the positional relationship between the LiDAR 11 and the camera 12 is lc Correct the following.

[0130] P lc =P le ×P ec ...(1)

[0131] The correction unit 252 generates positional relationship information including parameters indicating the corrected positional relationship between the LiDAR 11 and the camera 12.

[0132] In step S124, similar to the processing in step S43 of FIG. 6, positional relationship information between the LiDAR 11 and the camera 12 is output.

[0133] Thereafter, the positional relationship information correction process ends.

[0134] In this way, the positional relationship between the LiDAR 11 and the camera 12 is calibrated while the LiDAR 11 independently scans the measurement light in the same manner as in normal operation. This makes it possible to detect corresponding points at high speed, and shortens the processing time for calibrating the positional relationship between the LiDAR 11 and the camera 12.

[0135] <<4. Modifications>> Modifications of the above-described embodiments of the present technology will now be described.

[0136] For example, the corresponding point detection unit 62 may use a difference image to improve the detection accuracy of the corresponding points. For example, the corresponding point detection unit 62 may generate a difference image by calculating the difference between a camera image taken with the measurement light irradiated and a camera image taken with the measurement light not irradiated, and detect the corresponding points based on the difference image.

[0137] In this case, the corresponding point detection unit 62 may apply NMS to pixels in the difference image whose brightness is above a predetermined threshold, and detect the pixel with the highest brightness or the pixel with brightness above the threshold as the corresponding point.

[0138] 14, in order to improve the detection accuracy of the corresponding points, the lens of the camera 12 may be covered with an optical filter 301 that transmits only light having a wavelength close to that of the measurement light. This optical filter 301 is provided, for example, only during calibration.

[0139] For example, in the processing of step S41 in Figure 6, the correction unit 63 may select n sets of observation point / corresponding point pairs (four or more), solve the PnP problem, and estimate the position and orientation of the camera 12 in the LiDAR coordinate system, and calculate the reprojection error.

[0140] For example, when the information processing system 1 is installed in a moving body such as a vehicle and calibration of the positional relationship between the LiDAR 11 and the camera 12 is performed while the moving body is moving, it is not necessarily necessary to move the irradiation direction of the measurement light of the LiDAR 11.

[0141] This technology can be applied to, for example, not only LiDAR 11 but also a three-dimensional sensor that scans measurement light and detects the distribution (three-dimensional information) of the observation point in a three-dimensional coordinate system.

[0142] The type of camera to which the present technology can be applied is not particularly limited. For example, the present technology can be applied to a camera that can detect wavelength bands other than visible light and a camera that can detect depth information.

[0143] <<5. Others>> <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0144] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0145] In the computer 1000 , a CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0146] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0147] The input unit 1006 includes input switches, buttons, a microphone, an image sensor, etc. The output unit 1007 includes a display, a speaker, etc. The storage unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0148] In the computer 1000 configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program recorded in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0149] The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on a removable medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0150] In the computer 1000, the program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

[0151] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0152] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0153] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0154] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0155] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0156] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0157] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0158] (1) An information processing device comprising: an irradiation control unit that controls the irradiation timing of measurement light from a three-dimensional sensor; a corresponding point detection unit that detects, in a camera image acquired by a camera, a corresponding point detection unit that detects a corresponding point corresponding to an observation point detected by the three-dimensional sensor using the measurement light; and a correction unit that corrects a parameter indicating a positional relationship between the three-dimensional sensor and the camera based on the position of the observation point in a predetermined three-dimensional coordinate system and the position of the corresponding point in a two-dimensional coordinate system of the camera image. (2) The information processing device described in (1), wherein the irradiation control unit controls the irradiation timing so that a period for irradiating the measurement light is equal to or longer than a frame interval of the camera. (3) The information processing device described in (2), wherein the irradiation control unit controls the irradiation timing so that the measurement light is irradiated during an exposure period of the camera. (4) The information processing device described in any of (1) to (3), wherein the irradiation control unit further controls the irradiation direction of the measurement light. (5) The information processing device according to any one of (1) to (4), wherein the corresponding point detection unit detects, in different camera images, a plurality of corresponding points corresponding to a plurality of observation points detected using the measurement light irradiated at different irradiation timings, and the correction unit corrects the parameters based on a plurality of pairs of the observation points and the corresponding points. (6) The information processing device according to (5), wherein the correction unit selects the pairs based on a reprojection error, which is a positional error between the corresponding points and projection points obtained by projecting the observation points onto the two-dimensional coordinate system, and corrects the parameters based on the extracted pairs. (7) The information processing device according to (6), wherein the correction unit selects the pairs based on the reprojection error, which is based on the position and orientation of the camera estimated based on three or more pairs selected from the plurality of pairs. (8) The information processing device according to any one of (1) to (7), further comprising a positional deviation detection unit that checks for positional deviation between the three-dimensional sensor and the camera based on the detection results of the observation point and the corresponding point, and the correction unit corrects the parameters when the positional deviation is detected.(9) The information processing device according to (8), wherein the positional deviation detection unit checks the positional deviation based on a reprojection error, which is a positional error between the corresponding point and a projection point obtained by projecting the observation point onto the two-dimensional coordinate system based on the parameters. (10) The corresponding point detection unit detects, in different camera images, a plurality of corresponding points corresponding to a plurality of the observation points detected using the measurement light irradiated at different irradiation timings, and the positional deviation detection unit checks the positional deviation based on a plurality of pairs of the observation point and the corresponding point. (11) The information processing device according to (10), wherein the correction unit selects the pairs based on the reprojection error and corrects the positional deviation based on the reprojection error of the extracted pairs. (12) The information processing device according to any of (1) to (11), wherein the corresponding point detection unit detects the corresponding points based on luminance of the camera images. (13) The information processing device according to (12), wherein the corresponding point detection unit detects the corresponding points after applying NMS (Non Maximum Suppression) to pixels in the camera image whose brightness is equal to or greater than a predetermined threshold. (14) The information processing device according to (12) or (13), wherein the corresponding point detection unit detects the corresponding points based on a difference image between the camera image captured with the measurement light irradiated and the camera image captured without the measurement light irradiated. (15) The information processing device according to any of (1) to (14), wherein the three-dimensional sensor is a LiDAR (Light Detection and Ranging). (16) An information processing method, comprising: controlling irradiation timing of measurement light from a three-dimensional sensor; detecting, in a camera image acquired by a camera, corresponding points corresponding to observation points detected by the three-dimensional sensor using the measurement light; and correcting parameters indicating a positional relationship between the three-dimensional sensor and the camera based on the positions of the observation points in a predetermined three-dimensional coordinate system and the positions of the corresponding points in a two-dimensional coordinate system of the camera image.(17) A program for causing a computer to execute the following process: controlling the timing of irradiation of measurement light from a three-dimensional sensor; detecting a corresponding point in a camera image acquired by a camera that corresponds to an observation point detected by the three-dimensional sensor using the measurement light; and correcting parameters indicating the positional relationship between the three-dimensional sensor and the camera based on the position of the observation point in a predetermined three-dimensional coordinate system and the position of the corresponding point in the two-dimensional coordinate system of the camera image.

[0159] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0160] REFERENCE SIGNS LIST 1 Information processing system, 11 LiDAR, 12 Camera, 13, 13a, 13b, 13c Data processing device, 51 Processor, 61 Irradiation control unit, 62 Corresponding point detection unit, 63 Correction unit, 101 Misalignment detection unit, 201 Information processing system, 251 Corresponding point detection unit, 252 Correction unit

Claims

1. An information processing device comprising: an irradiation control unit that controls the irradiation timing of the measurement light of a three-dimensional sensor; a corresponding point detection unit that detects a corresponding point in a camera image acquired by the camera that corresponds to an observation point detected by the three-dimensional sensor using the measurement light; and a correction unit that corrects parameters indicating the positional relationship between the three-dimensional sensor and the camera based on the position of the observation point in a predetermined three-dimensional coordinate system and the position of the corresponding point in the two-dimensional coordinate system of the camera image.

2. The information processing device according to claim 1, wherein the irradiation control unit controls the irradiation timing so that the cycle of irradiating the measurement light is equal to or longer than the frame interval of the camera.

3. The information processing device according to claim 2, wherein the irradiation control unit controls irradiation timing so that the measurement light is irradiated during an exposure period of the camera.

4. The information processing device according to claim 1, wherein the irradiation control unit further controls an irradiation direction of the measurement light.

5. The information processing device according to claim 1, wherein the corresponding point detection unit detects, in different camera images, a plurality of corresponding points each corresponding to a plurality of observation points detected using the measurement light irradiated at different irradiation timings, and the correction unit corrects the parameters based on pairs of the plurality of observation points and the corresponding points.

6. The information processing device according to claim 5, wherein the correction unit selects the pairs based on a reprojection error, which is a positional error between the projection point obtained by projecting the observation point onto the two-dimensional coordinate system and the corresponding point, and corrects the parameters based on the extracted pairs.

7. The information processing device according to claim 6, wherein the correction unit selects the pair based on the reprojection error based on the position and orientation of the camera estimated based on three or more pairs selected from the plurality of pairs.

8. The information processing device according to claim 1, further comprising a position shift detection unit that checks for position shift between the three-dimensional sensor and the camera based on the detection results of the observation point and the corresponding point, and the correction unit corrects the parameters when the position shift is detected.

9. The information processing device according to claim 8, wherein the position shift detection unit checks the position shift based on a reprojection error, which is a position error between the corresponding point and a projection point obtained by projecting the observation point onto the two-dimensional coordinate system based on the parameters.

10. The information processing device according to claim 9, wherein the corresponding point detection unit detects, in different camera images, a plurality of corresponding points each corresponding to a plurality of observation points detected using the measurement light irradiated at different irradiation timings, and the position shift detection unit checks the position shift based on pairs of the plurality of observation points and the corresponding points.

11. The information processing device according to claim 10, wherein the correction unit selects the pairs based on the reprojection error, and corrects the positional deviation based on the reprojection error of the extracted pairs.

12. The information processing device according to claim 1, wherein the corresponding point detection unit detects the corresponding points based on the luminance of the camera image.

13. The information processing device according to claim 12, wherein the corresponding point detection section detects the corresponding points after applying NMS (Non Maximum Suppression) to pixels in the camera image whose luminance is equal to or greater than a predetermined threshold value.

14. The information processing device according to claim 12, wherein the corresponding point detection unit detects the corresponding points based on a difference image between the camera image taken with the measurement light irradiated and the camera image taken with the measurement light not irradiated.

15. The information processing device according to claim 1, wherein the three-dimensional sensor is a LiDAR (Light Detection And Ranging) sensor.

16. An information processing method comprising: controlling the timing of irradiation of a measurement light of a three-dimensional sensor; detecting a corresponding point in a camera image acquired by a camera that corresponds to an observation point detected by the three-dimensional sensor using the measurement light; and correcting parameters indicating the positional relationship between the three-dimensional sensor and the camera based on the position of the observation point in a predetermined three-dimensional coordinate system and the position of the corresponding point in the two-dimensional coordinate system of the camera image.

17. A program for causing a computer to execute a process of controlling the timing of irradiation of measurement light from a three-dimensional sensor, detecting a corresponding point in a camera image acquired by a camera that corresponds to an observation point detected by the three-dimensional sensor using the measurement light, and correcting parameters indicating the positional relationship between the three-dimensional sensor and the camera based on the position of the observation point in a predetermined three-dimensional coordinate system and the position of the corresponding point in the two-dimensional coordinate system of the camera image.

Citation Information

Patent Citations

  • Distance measuring device with distance correction function

    JP2020008531A

  • System and method for calibrating at least one camera and light detection and ranging sensor

    JP2021016151A

  • Method for calculating distance correction information, distance measuring device, moving body, and stereo camera device

    JP2021148643A

  • Distance measuring sensor, distance measuring device, and distance measuring method

    JP2022189184A

  • Semantic graph embedding lifted for all azimuth direction location recognition

    JP2023059794A

Cited By

  • Model learning device, model learning method, and program

    JP2026164040A