Information processing device, information processing method, and program

The information processing device uses 3D maps and sensor data to calibrate multiple sensors in autonomous driving systems, overcoming previous limitations by allowing flexible calibration without specialized equipment, enhancing accuracy and reducing costs.

WO2024241963A1PCT designated stage expired Publication Date: 2024-11-28SONY GROUP CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017784
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-05-14
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing autonomous driving technologies face limitations in calibrating multiple sensors simultaneously, requiring specific vehicle conditions and relying on time-consuming 3D measuring devices, which restricts calibration flexibility and accuracy.

Method used

An information processing device and method that uses a 3D map and sensing results from multiple sensors, such as cameras and LiDAR, to determine deviations and calibrate sensors, allowing for flexible calibration without specialized equipment, even during vehicle movement.

Benefits of technology

Enables efficient and accurate calibration of multiple sensors using real-world landmarks, reducing calibration time and cost, and improving autonomous driving accuracy in various environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017784_28112024_PF_FP_ABST
    Figure JP2024017784_28112024_PF_FP_ABST
Patent Text Reader

Abstract

This information processing device comprises: a deviation determination unit that determines a deviation between a plurality of sensors on the basis of a 3D map and sensing results of the plurality of sensors; and a calibration processing unit that performs calibration of the sensors on the basis of the determination result of the deviation determination unit.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program.

[0002] In recent years, in the field of surrounding environment recognition technology aimed at improving the accuracy of autonomous driving and driving assistance for moving objects such as automobiles, it is rare to use a single sensor to recognize the surrounding environment. In most cases, it is common to use a combination of multiple sensors of different types to recognize the surrounding environment.

[0003] To recognize the surrounding environment using multiple different sensors, it is necessary to integrate the image capture results of each sensor, which requires a calibration technique to determine the relative position and orientation between each sensor.

[0004] Therefore, a system has been disclosed that calibrates sensors while the vehicle is moving by using a map database that stores map information including calibration points and road information at those calibration points (Patent Document 1).

[0005] There is also a method of performing calibration using the measurement results of a 3D measuring device, which is a measuring device that can acquire the 3D structure and reflection intensity around the measuring device as 3D points and the reflection intensity of those points, and is also called a 3D laser scanner.

[0006] JP 2018-96715 A

[0007] However, the technology in Patent Document 1 has problems in that it has limitations on the movement and direction of the vehicle, and is a calibration method for a single sensor, not a calibration between multiple sensors, making it insufficient for application to recent surrounding environment recognition technologies that use multiple sensors.

[0008] In the technology of Patent Document 1, the vehicle must satisfy certain conditions in order to perform calibration. These conditions are that the vehicle is in a straight line relative to the road and is accelerating at a constant acceleration, or that the vehicle is traveling along a curve and is accelerating at a constant acceleration, and the conditions in which calibration can be performed are limited.

[0009] Furthermore, when calibration is performed using a 3D measuring device, the measurement takes time, and calibration can only be performed at a location where the 3D measuring device is available.

[0010] The present technology has been developed in consideration of such problems, and aims to provide an information processing device, an information processing method, and a program that can easily perform calibration of multiple sensors.

[0011] In order to solve the above-mentioned problems, the first technology is an information processing device that includes a 3D map and a deviation determination unit that determines deviations between multiple sensors based on the sensing results of the multiple sensors, and a calibration processing unit that calibrates the multiple sensors based on the determination results of the deviation determination unit.

[0012] The second technique is an information processing method that determines deviations between multiple sensors based on a 3D map and sensing results from the multiple sensors, and calibrates the sensors based on the determination results of the deviation determination unit.

[0013] Furthermore, the third technology is a program that causes a computer to execute an information processing method that determines deviations between multiple sensors based on a 3D map and sensing results of the multiple sensors, and calibrates the sensors based on the determination results of the deviation determination unit.

[0014] 1 is a block diagram showing the configuration of an information processing device 100. It is an explanatory diagram of a camera 200 and a LiDAR 300 provided on an automobile 10. It is a diagram showing the configuration of a 3D map. It is a flowchart showing processing in the information processing device 100. It is a flowchart showing processing in a deviation determination unit 105. It is an explanatory diagram of processing for detecting landmarks from a camera image. It is an explanatory diagram of how to determine the relative position of the camera 200 on a 3D map. It is an explanatory diagram of how to determine an error. It is a flowchart showing processing in the deviation determination unit 105. It is an explanatory diagram of how to determine the relative position of the LiDAR 300 on a 3D map. It is an explanatory diagram of nearest neighbor error. It is a flowchart showing processing in the deviation determination unit 105. It is an explanatory diagram of coordinate transformation to a LiDAR coordinate system. It is an explanatory diagram of coordinate transformation to a camera coordinate system. It is an explanatory diagram of projection onto a camera imaging surface coordinate system. It is an explanatory diagram of deviation determination between the camera 200 and the LiDAR 300. It is a block diagram showing the configuration of an information processing device 100 of a first modified example. It is a diagram showing the structure of a sensing result. It is a block diagram showing the configuration of an information processing device 100 of a second modified example. FIG. 10 is a block diagram showing a configuration of an information processing device 100 according to a third modified example.

[0015] Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order: <Embodiment> [Configuration of information processing device 100] [Processing in information processing device 100] <Modifications> [First modification] [Second modification] [Third modification] [Fourth modification] [Other modifications]

[0016] <Embodiment> [Configuration of Information Processing Device 100] The following describes the configuration of the information processing device 100 in this embodiment. As shown in Fig. 1 , the information processing device 100 includes a 3D map storage unit 101, a 3D map acquisition unit 102, an inter-sensor relative position storage unit 103, a sensing control unit 104, a deviation determination unit 105, and a calibration processing unit 106.

[0017] The information processing device 100 is connected to a camera 200 and a LiDAR 300 as multiple sensors to be calibrated. The number of sensors to be calibrated connected to the information processing device 100 does not necessarily have to be two, and any number greater than or equal to two may be used. Furthermore, not all of the multiple sensors connected to the information processing device 100 are always subject to calibration, and any multiple sensors may be subject to calibration. Furthermore, sensors other than the camera 200 and the LiDAR 300 may be connected to the information processing device.

[0018] The camera 200 is equipped with an imaging element, a signal processing circuit, etc., and is capable of capturing RGB (Red, Green, Blue) or monochrome color images and videos. The image output by the camera 200 as a sensing result is referred to as a camera image.

[0019] The LiDAR 300 is a distance measurement sensor that measures scattered light in response to irradiation with pulsed laser light and detects the distance to a target object or substance, as well as the properties and type of the object. The LiDAR 300 outputs the sensing results in the form of a 3D point cloud. Note that radar is also used as a distance measurement sensor in this technology, and since radar can also output a 3D point cloud like the LiDAR 300, calibration can be performed using the same processing as the LiDAR 300. In this embodiment, the distance measurement sensor is assumed to be the LiDAR 300.

[0020] As shown in Fig. 2, the camera 200 and the LiDAR 300 are mounted on the automobile 10 as on-board sensors for sensing the surroundings of the automobile 10 as a moving body. As shown in Fig. 2, the physical positional relationship between the camera 200 and the LiDAR 300 is referred to as the inter-sensor relative position. The inter-sensor relative position can also be said to include a translation component and a rotation component. In the example of Fig. 2, one camera 200 and one LiDAR 300 are provided to sense the area ahead of the automobile 10, but the number and installation positions of the cameras 200 and LiDAR 300 are not limited to this example.

[0021] The camera 200 and the LiDAR 300 are connected to the information processing device 100, and sense the surroundings of the automobile 10 under the control of the sensing control unit 104, and output the sensing results. The camera 200 and the LiDAR 300 may be connected to the information processing device 100 by wire or wirelessly. Examples of wired connection methods include HDMI (registered trademark) (High-Definition Multimedia Interface) and USB (Universal Serial Bus), and examples of wireless connection methods include Wi-Fi, Bluetooth (registered trademark), and NFC (Near Field Communication).

[0022] Furthermore, a position sensor 400 and a direction sensor 500 mounted on the automobile 10 are connected to the information processing device 100 .

[0023] The position sensor 400 is a sensor such as a GPS (Global Positioning System) or a GNSS (Global Navigation Satellite System) for acquiring the position of the automobile 10 equipped with the camera 200 and the LiDAR 300.

[0024] The orientation sensor 500 is a compass or the like that detects the orientation of the automobile 10 equipped with the camera 200 and the LiDAR 300.

[0025] The position sensor 400 and the direction sensor 500 supply the position information and direction information of the automobile 10, which are the detection results, to the information processing device 100 at any timing, regardless of the movement state of the automobile 10, such as whether the automobile 10 is running or stopped. The arbitrary timing may be, for example, a predetermined time once a day, or the point in time when the automobile 10 has been running continuously for a predetermined period of time (for example, one hour).

[0026] The position sensor 400 and the orientation sensor 500 may be connected to the information processing device 100 by wire or wirelessly. Wired connection methods include, for example, HDMI (registered trademark) and USB, and wireless connection methods include, for example, Wi-Fi, Bluetooth (registered trademark), and NFC. The position sensor 400 and the orientation sensor 500 may be functions that the automobile 10 is equipped with in advance, or may be configured as devices separate from the automobile 10 and installed in the automobile 10.

[0027] The 3D map storage unit 101 stores a plurality of pre-generated 3D maps. The 3D map storage unit 101 is configured with various storage media, such as a hard disk or flash memory. Examples of 3D maps include those published by the Geospatial Information Authority of Japan or the Ministry of Land, Infrastructure, Transport and Tourism, and those created by companies and the like using 3D measuring devices. However, the 3D map is not limited to these, and any data that represents a landscape including the earth's surface, topography, buildings, objects, etc. in three dimensions may be used.

[0028] The 3D map acquisition unit 102 checks, based on the position information and orientation information, whether or not a 3D map that is close to the position and direction of the automobile 10 exists in the 3D map storage unit 101. A 3D map that is close to the position and direction of the automobile 10 is a 3D map that includes the 3D map within a predetermined range based on the position of the automobile 10 and includes part or all of the 3D map in a direction at a predetermined angle based on the front of the automobile 10.

[0029] If a 3D map close to the position and direction of the automobile 10 exists, the 3D map acquisition unit 102 acquires the 3D map and supplies it to the deviation determination unit 105. The 3D map acquisition unit 102 also notifies the sensing control unit 104 that a 3D map close to the position and direction of the automobile 10 exists.

[0030] The inter-sensor relative position storage unit 103 stores information on the inter-sensor relative positions of the camera 200 and the LiDAR 300, which are sensors to be calibrated. The inter-sensor relative position storage unit 103 is configured with various storage media, for example, a hard disk, a flash memory, etc. The inter-sensor relative positions stored in the inter-sensor relative position storage unit 103 are updated by the calibration processing unit 106, but the initial values ​​of the inter-sensor relative positions are measured, for example, when the automobile 10 equipped with the camera 200 and the LiDAR 300 is delivered or shipped from the factory.

[0031] The sensing control unit 104 controls the camera 200 and the LiDAR 300 to sense the surroundings of the automobile 10. The sensing control unit 104 acquires the camera image output by the camera 200 as the sensing result and the 3D point cloud output by the LiDAR 300 as the sensing result, and supplies them to the deviation determination unit 105.

[0032] The deviation determination unit 105 determines whether or not there is a deviation between the positions of the camera 200 and the LiDAR 300 based on the sensing result and the 3D map. The deviation determination unit 105 supplies the current inter-sensor relative position calculated during deviation determination to the calibration processing unit 106.

[0033] The calibration processing unit 106 performs calibration by updating the current inter-sensor relative position supplied from the deviation determination unit 105 and storing it in the inter-sensor relative position storage unit 103 as a new inter-sensor relative position.

[0034] The information processing device 100 is configured as described above. The automobile 10 may have the functions of the information processing device 100 in advance, or a processor included in the automobile 10 may execute the functions of the information processing device 100, or the automobile 10, which has computer functions, may execute a program to realize the information processing device 100 and the information processing method. Furthermore, an electronic device (such as a personal computer, tablet terminal, smartphone, or server) separate from the automobile 10 may have the functions of the information processing device 100, or the information processing device 100 and the information processing method may be realized by an electronic device, which has computer functions, executing a program. The program may be installed in the automobile 10 or the electronic device in advance, or may be distributed by download or storage medium, etc., and installed by a user, etc. Furthermore, the information processing device 100 may be configured as a standalone device.

[0035] 3 shows the structure of the 3D map stored in the 3D map storage unit 101. The 3D map includes longitude, latitude, direction, 3D point group, and landmark information.

[0036] The latitude and longitude are information indicating the position of the 3D map, and the direction indicates the direction in which the 3D map is viewed from the position indicated by the latitude and longitude, as shown in Figure 3B.

[0037] It is desirable to store in advance a large number of 3D maps at different latitudes and longitudes, and 3D maps at the same latitude and longitude but in different directions, in the 3D map storage unit 101. This makes it possible to perform calibration at many positions.

[0038] A 3D point cloud is information acquired in advance using a 3D measuring device (such as a 3D laser scanner), LiDAR, or Radar. As shown in FIG. 3B , the 3D point cloud represents a landscape including the earth's surface, topography, buildings, objects, etc., that exist in a direction from a point specified by latitude and longitude. Because positions in the 3D point cloud are three-dimensional positions, they can be expressed using (x, y, z) coordinates. In this embodiment, the latitude and longitude point will be described as the origin.

[0039] If the 3D structure represented by the 3D point cloud is composed of flat surfaces, it must contain three or more flat surfaces that share a common point. Examples of three or more flat surfaces include two walls and one road. Furthermore, if the 3D structure represented by the 3D point cloud is composed of curved surfaces, each curved surface must be approximated as a flat surface, resulting in three or more flat surfaces that share a common point.

[0040] Landmark information is information that indicates landmarks that exist in the real world and are included in the 3D point cloud. Landmarks are objects that exist in the real world and are used in determining deviation. Examples of landmarks include signs, signs, and buildings, but any object whose position does not change can be used.

[0041] The structure of the landmark information is shown in Figure 3C. The landmark information is composed of an image showing a landmark (hereinafter referred to as a landmark image) and the position information of the landmark on the 3D point cloud. The landmark image may be an image of the shape of the landmark itself, or it may be a regular image of a rectangle with the landmark inside it. It is desirable that the landmark image be an image aligned with the camera 200 that is the target of calibration, but this is not limited as long as the landmark can be identified.

[0042] The position of each landmark on the 3D point cloud is a three-dimensional position and is represented by (xn, yn, zn). In the following description, the location (xn, yn, zn) of a landmark on the 3D point cloud of the 3D map may be simply referred to as the location (xn, yn, zn). n is a value that identifies each landmark and takes values ​​from 1 to n. In the following description, a 3D map A including four landmarks is used, and n is assumed to be 4. Note that the location (xn, yn, zn) of the landmark needs to be a coordinate system based on a certain point on the landmark (for example, the upper left corner or center of the landmark).

[0043] Each 3D map must contain four or more landmark images. As will be described later, this technology uses a method called PnP (Perspective-n-Point) to determine the relative positions of each sensor to be calibrated, and four or more landmarks are required to solve the equations in PnP. There is no limit to the number of landmarks; the more the better.

[0044] [1-2. Processing in Information Processing Device 100] Next, processing in the information processing device 100 will be described with reference to FIG.

[0045] First, in step S101 , the 3D map acquisition unit 102 acquires the current position information of the automobile 10 from the position sensor 400 and the current direction information from the direction sensor 500 .

[0046] Next, in step S102, the 3D map acquisition unit 102 checks whether a 3D map close to the position and direction of the automobile 10 exists in the 3D map storage unit 101 based on the position information and orientation information. If a 3D map exists, the process proceeds to step S103 (Yes in step S102). Note that if a 3D map close to the position and direction of the automobile 10 does not exist, the process ends (No in step S102). In this embodiment, the description will be given assuming that 3D map A shown in FIG. 3A exists as a 3D map close to the position and direction of the automobile 10.

[0047] Next, in step S103, the 3D map acquisition unit 102 acquires a 3D map that is close to the position and direction of the automobile 10 from the 3D map storage unit 101 and supplies the 3D map to the deviation determination unit 105. The 3D map acquisition unit 102 also notifies the sensing control unit 104 that a 3D map that is close to the position and direction of the automobile 10 exists.

[0048] Next, in step S104, upon receiving notification that a 3D map exists, the sensing control unit 104 controls the camera 200 and LiDAR 300, which are sensors to be calibrated, to sense the surroundings of the automobile 10. Then, the sensing control unit 104 supplies the camera image, which is the sensing result acquired by the camera 200, and the 3D point cloud, which is the sensing result acquired by the LiDAR 300, to the deviation determination unit 105.

[0049] Next, in step S105, the deviation determination unit 105 determines whether there is a deviation in the positions of the camera 200, which is the sensor to be calibrated, and the LiDAR 300. If there is a deviation, the process proceeds from step S106 to step S107 (Yes in step S106). If there is no deviation, the process ends (No in step S106). Details of the process of the deviation determination unit 105 will be described later.

[0050] Next, in step S107, the deviation determination unit 105 supplies the calibration processing unit 106 with information on the current inter-sensor relative position obtained in the deviation determination process.

[0051] Then, in step S108, the calibration processing unit 106 executes calibration by updating the inter-sensor relative position storage unit 103 with the current inter-sensor relative position supplied from the deviation determination unit 105 as the new inter-sensor relative position.

[0052] Next, the processing of the deviation determination unit 105 in step S105 will be described. The deviation determination unit 105 determines the positional deviation of the camera 200, the positional deviation of the LiDAR 300, and the positional deviation between the camera 200 and the LiDAR 300.

[0053] First, the process of determining the deviation of the camera 200 will be described with reference to FIG.

[0054] In step S201, the deviation determination unit 105 extracts landmark information from the 3D map supplied from the 3D map acquisition unit 102.

[0055] Next, in step S202, each of the multiple landmarks is detected from the camera image, as shown in FIG. 6. The detected position of each landmark is represented as a two-dimensional position (un, vn). In the following description, the detected position (un, vn) of a landmark in the camera image may be simply referred to as the detected position (un, vn). n is a value that identifies multiple landmarks and takes on values ​​from 1 to n. In this embodiment, there are four landmarks, so landmark 1, landmark 2, landmark 3, and landmark 4 are detected from the camera image.

[0056] Landmark detection can be performed using machine learning, artificial intelligence, or the like, such as a deep neural network (DNN), a convolutional neural network (CNN), or a random forest (RF). Examples of machine learning learning methods include neural networks and deep learning. Furthermore, the detection process may be performed using known object detection techniques, such as a template matching method or a matching method based on brightness distribution information of a subject.

[0057] Next, in step S203, as shown in FIG. 7, PnP (Perspective-n-Point) is performed between the position (xn, yn, zn) of each landmark on the 3D point cloud of the 3D map and the detected position (un, vn) of each landmark in the camera image, and the relative position on the 3D map of the camera 200 is calculated. C T 3 The relative position of the camera 200 on the 3D map is the relative position of the camera 200 in the 3D point cloud included in the 3D map.

[0058] In this embodiment, n is 4, so for landmarks 1 to 4, PnP is performed at the locations (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4) of each landmark on the 3D point cloud of the 3D map, and at the detected locations (u1, v1), (u2, v2), (u3, v3), and (u4, v4) of each landmark in the camera image.

[0059] Next, in S204, the relative position of the camera 200 on the 3D map is calculated as shown in FIG.C T 3 The position (xn, yn, zn) of the landmark on the 3D point cloud of the 3D map is projected onto the camera image using

[0060] The location of the landmark on the 3D point cloud of the 3D map is P 3 and the projection position of the landmark on the 3D point cloud of the 3D map onto the camera image is P duv Then, the projection function of the landmark onto the camera image is the relative position C T 3 Using this, the following equation 1 is obtained.

[0061] [Formula 1] P duv =lens_distortion (K. C T 3 ・P 3 )

[0062] In Equation 1, K is the camera matrix of the camera 200 (or the internal parameter matrix of the camera 200), and lens_distortion is a function that adds the lens distortion of the camera 200.

[0063] Projection position P duv is the position of each landmark projected onto the camera image, and is therefore a two-dimensional position expressed as (upn, vpn). In the following description, the projected position (upn, vpn) of each landmark onto the camera image may be simply referred to as the projected position (upn, vpn). The projected position (upn, vpn) can be expressed as in the following equation 2 using the existing position (xn, yn, zn) of the landmark on the 3D point cloud of the 3D map.

[0064] [Formula 2] (upn, vpn) = project (xn, yn, zn)

[0065] Then, the error between the projected position (upn, vpn) of each landmark on the 3D point cloud of the 3D map onto the camera image and the detected position (un, vn) of each landmark in the camera image is calculated.

[0066] For landmark 1, the error is the difference between the projected position (up1, vp1) and the detected position (u1, v1). For landmark 2, the error is the difference between the projected position (up2, vp2) and the detected position (u2, v2). For landmark 3, the error is the difference between the projected position (up3, vp3) and the detected position (u3, v3). For landmark 4, the error is the difference between the projected position (up4, vp4) and the detected position (u4, v4).

[0067] In addition, since the projection positions (upn, vpn) of landmarks on the 3D point cloud of the 3D map onto the camera image and the detected positions (un, vn) of the landmarks in the camera image are each the same number as the number of landmarks, they can be judged as a single value using, for example, a simple average or mean squared error, or they can be judged to be misaligned if the proportion of points exceeding a threshold value exceeds a certain threshold.

[0068] Then, in step S205, the error is compared with a predetermined threshold, and if the error exceeds the threshold, it is determined that the relative position on the 3D map of the camera 200 that captured the camera image is misaligned. The predetermined threshold is determined based on the accuracy required by the application that uses the sensor output, but for example, in the case of camera 200, it is set to 1 pixel because if the error is less than 1 pixel, it is difficult to visually check the misalignment.

[0069] Next, the process of determining the deviation of the LiDAR 300 will be described with reference to FIG. 9 .

[0070] In step S301, as shown in FIG. 10, ICP (Iterative Closest Point) is performed between the 3D point cloud of the 3D map and the 3D point cloud detected by the LiDAR 300, and the relative position on the 3D map of the LiDAR 300 is calculated. L T 3 The relative position of the LiDAR 300 on the 3D map is calculated. L T 3 is the relative position of the LiDAR 300 in the 3D point cloud contained in the 3D map.

[0071] The position in the 3D point cloud of the 3D map is expressed as (x, y, z), and the position in the 3D point cloud of the LiDAR 300 is expressed as (xL, yL, zL). Therefore, the position of each landmark in the 3D point cloud of the 3D map can be expressed as (xn, yn, zn), and the position of each landmark in the 3D point cloud of the LiDAR 300 can be expressed as (xLn, yLn, zLn).

[0072] Next, in step S302, the 3D point cloud of the 3D map is calculated based on the relative position L T 3 The coordinates are converted using the above method, and the nearest point error is calculated between the 3D point cloud detected by LiDAR 300.

[0073] Relative position of 3D point clouds on 3D maps L T 3 When the coordinate conversion is performed, the coordinate system of the 3D point cloud of the 3D map becomes the coordinate system of the LiDAR 300. Therefore, if there is no misalignment between the 3D point cloud of the 3D map and the 3D point cloud of the LiDAR 300, the 3D point cloud of the 3D map after the coordinate conversion and the 3D point cloud of the LiDAR 300 should match.

[0074] On the other hand, if there is a misalignment between the 3D point cloud of the 3D map and the 3D point cloud of LiDAR 300, the 3D point cloud of the 3D map after coordinate transformation and the 3D point cloud of LiDAR 300 will not match and will be misaligned, as shown in Figure 11.

[0075] Therefore, the difference between the nearest neighbors of each point in the overlapped 3D point clouds is calculated as the evaluation value of the deviation. This evaluation value is called the nearest neighbor error. The nearest neighbor error may be evaluated individually, but since the number of 3D points is enormous, unlike in the case of the camera 200, a single value is calculated using a simple average or mean square error, and this is called the nearest neighbor error.

[0076] Next, in step S303, the deviation determination unit 105 compares the nearest neighbor point error with a predetermined threshold, and if the nearest neighbor point error exceeds the threshold, it determines that the relative position on the 3D map of the LiDAR 300 that detected the 3D point cloud is deviated. The predetermined threshold is determined based on the accuracy required by the application that uses the sensor output, but for example, in the case of LiDAR 300, if it is below the ranging accuracy, it becomes difficult to determine whether it is deviation or ranging accuracy, so it is set to be below the ranging accuracy of LiDAR 300.

[0077] Next, the process of determining the positional deviation between the camera 200 and the LiDAR 300 will be described with reference to FIG. 12 .

[0078] First, in step S401, the relative position of the camera 200 on the 3D map is determined in the same manner as in the case of determining the deviation of the camera 200 described above. C T 3 In addition, the relative position of the LiDAR 300 on the 3D map is calculated in the same manner as in the case of determining the deviation of the LiDAR 300 described above. L T 3 Ask for.

[0079] Next, in step S402, the relative position C T 3 and relative position L T 3 is combined to obtain the current relative position between the camera 200 and the LiDAR 300, i.e., the current inter-sensor relative position C T L Ask for.

[0080] Here, the relative position between the sensors C T L Let us explain how to calculate the above. If the rotation from coordinate system A to coordinate system B is expressed as a 3x3 matrix and the translation as a 3x1 matrix, then B R A : A 3x3 matrix representing the rotation from coordinate system A to coordinate system B B P A : A 3x1 matrix that represents the translation from coordinate system A to coordinate system B.

[0081] At this time, the transformation from the A coordinate system to the B coordinate system is expressed by the following equation 3.

[0082] [Formula 3]

[0083] The relative position obtained by PnP and ICP is B T A is.

[0084] Since it is expressed as a rotation matrix and a translation vector between coordinate systems, the relative position C T 3 and relative position L T 3 The relative position between the sensors of the camera 200 and the LiDAR 300 is calculated using theC T L can be calculated using the following equation 4.

[0085] [Formula 4] C T L = C T 3 × 3 T L = C T 3 × ( L T 3 ) -1

[0086] Next, in step S403, the position (xn, yn, zn) of the landmark on the 3D point cloud of the 3D map is calculated as a relative position L T 3 This process converts the location (xn, yn, zn) of each landmark in the 3D point cloud coordinate system on the 3D map into a relative location (xn, yn, zn) as shown in FIG. L T 3 The position of each landmark after the coordinate conversion to the LiDAR coordinate system is defined as (xLn, yLn, zLn), and the result of the coordinate conversion of the positions of all landmarks is defined as the first coordinate conversion result PL.

[0087] Next, in step S404, the first coordinate transformation result PL is converted into the current inter-sensor relative position C T L 14, this process means that the location positions (xLn, yLn, zLn) of each landmark converted into the LiDAR coordinate system are converted into the camera coordinate system. The location positions of each landmark after coordinate conversion into the camera coordinate system are defined as (xLCn, yLCn, zLCn), and the result of the coordinate conversion of the locations of all landmarks is defined as the second coordinate conversion result PLC.

[0088] Next, in step S405, the second coordinate transformation result PLC is projected onto the camera image plane coordinate system. As shown in Fig. 15, this process means that the location positions (xLCn, yLCn, zLCn) of each landmark that have been transformed into the camera coordinate system are projected onto the camera image plane coordinate system. The projection function uses the above-mentioned Equation 1. The location positions of each landmark after the coordinate transformation into the camera image plane coordinate system are defined as (uLCn, vLCn), and the projection result of the locations of all landmarks is defined as the projection result PLCC.

[0089] 16, the difference between the projection result PLCC onto the camera image plane coordinate system and the detected position PC of the landmark in the camera image detected in the above-described deviation determination of the camera 200 is calculated. If the difference exceeds a predetermined threshold, the current inter-sensor relative position between the camera 200 and the LiDAR 300 is calculated. C T L and the inter-sensor relative position acquired from the inter-sensor relative position storage unit 103. The predetermined threshold is determined depending on the accuracy required by the application that uses the output of this sensor, but for example, in the case of camera 200, it is set to 1 pixel because it is difficult to visually check the deviation if it is 1 pixel or less.

[0090] The difference from the case of determining the deviation of the camera 200 described above is that the difference between the detected position PC in the camera image of the landmark is calculated by converting the position of the landmark in the 3D point cloud into a relative position on the 3D map of the LiDAR 300. L T 3 and the current relative position between the sensors of the camera 200 and the LiDAR 300. C T L The coordinates are converted by the coordinate system and projected onto the camera image plane coordinate system, and the projected result is calculated using the PLCC. C T L No matter how large the deviation is, as long as the projection result PLCC is not deviated, it is not a problem for the user of the automobile 10. C T LEven if the deviation is small, if the projection result PLCC deviates significantly, it will be a problem for the user. C T L The deviation is determined.

[0091] In this case, as with the above-mentioned determination of deviation of the camera 200, the determination may be made by using a single value, for example, a simple average or a mean squared error, or if the proportion of points exceeding the threshold value exceeds a certain threshold value, it may be determined that there is deviation.

[0092] The deviation determination unit 105 determines the current relative position between the sensors of the camera 200 and the LiDAR 300. C T L If it is determined that there is a deviation, the current relative position between the sensors calculated at the time of deviation determination is used. C T L The calibration processing unit 106 then supplies the current inter-sensor relative position C T L The inter-sensor relative position storage unit 103 is updated with the new relative position, and calibration is performed.

[0093] Evaluation between sensors of the same type, such as between cameras or between LiDARs, can be performed in the same manner as in the determination of the deviation between the camera 200 and the 3D map and the LiDAR 300 and the 3D map described above.

[0094] The deviation determination unit 105 determines the current relative position between the sensors. C T L The deviation between the camera 200 and the LiDAR 300 can also be determined by comparing the relative position between the sensors obtained from the relative position between the sensors storage unit 103 with the relative position between the sensors.

[0095] If the relative positions between the sensors of the camera 200 and the LiDAR 300 are misaligned, it is possible to check whether either one or both of the camera 200 and the LiDAR 300 are misaligned by referring to the positional misalignment of the camera 200 and the positional misalignment of the LiDAR 300.

[0096] The processing of the information processing device 100 is performed as described above. According to the present technology, it is possible to easily calibrate multiple sensors provided on a moving object by using a 3D map of the surroundings of the sensors and landmarks. Furthermore, it is possible to calibrate multiple sensors without using special equipment such as a 3D measuring device.

[0097] Since no time for measurement using a 3D measuring device is required, the time required for calibration can be reduced. Furthermore, since 3D measuring devices are expensive, not using a 3D measuring device can also reduce the cost of performing calibration. Furthermore, calibration can be performed even in places where a 3D measuring device is not available. Furthermore, calibration can be performed even when the moving object is moving.

[0098] Since landmarks that exist in the real world are used to determine the deviation for calibration, there is no need for markers that can be photographed by the camera 200, and calibration can be easily performed.

[0099] The ability to easily calibrate sensors mounted on the automobile 10 can improve the accuracy of the automatic driving and driving assistance of the automobile 10. This makes it possible to realize automatic driving and driving assistance that are robust in a variety of environments.

[0100] <Modifications> Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical ideas of the present technology are possible.

[0101] [First Modification] A first modification will be described. As shown in Fig. 17 , the information processing device 100 in the first modification differs from the embodiment in that it includes a sensing result storage unit 107. The other configurations are the same as those in the embodiment, and therefore description thereof will be omitted.

[0102] In the embodiment, it is assumed that a single sensing result from the sensor to be calibrated is used. However, the accuracy of the deviation determination and the calibration accuracy can be improved by using multiple sensing results. Therefore, in the first modified example, deviation determination and calibration are performed using multiple sensing results from the sensor to be calibrated.

[0103] The sensing result storage unit 107 stores a plurality of sensing results of the sensors to be calibrated, and is configured with various storage media such as a hard disk and a flash memory.

[0104] The sensing control unit 104 supplies and stores the sensing results in the sensing result storage unit 107. When a predetermined number of sensing results are stored in the sensing result storage unit 107, the deviation determination unit 105 performs deviation determination using the plurality of sensing results.

[0105] First, as in the embodiment, the deviation determination unit 105 determines whether the relative positions between the sensors are misaligned using a single sensing result. If it is determined that the relative positions between the sensors are misaligned, the deviation determination unit 105 supplies the sensing result to the sensing result storage unit 107 for storage.

[0106] When the sensing result storage unit 107 accumulates a predetermined number of sensing results, the deviation determination unit 105 performs deviation determination again using the multiple sensing results. Then, if the deviation determination unit 105 determines that the relative positions between the sensors are deviated, the deviation determination unit 105 C T L is supplied to the calibration processing unit 106, and the calibration processing unit 106 executes the calibration.

[0107] The deviation determination process in the deviation determination unit 105 is the same as in the embodiment, but when deviation determination is performed using multiple sensing results, multiple camera images as sensing results may be combined and processed as a single camera image, or deviation determination may be performed individually using each of the multiple camera images, and calibration may be performed using the median of multiple relative positions calculated as a result of the deviation determination. This also applies to the 3D point cloud as a sensing result acquired by the LiDAR 300.

[0108] 18 shows an example of the data structure of the sensing result stored in the sensing result storage unit 107. The sensing result has a configuration including the latitude of the sensing position, the longitude of the sensing position, the direction of the sensing position, and camera images or 3D point clouds continuing for the number of sensors to be calibrated.

[0109] [Second Modification] Next, a second modification will be described. As shown in Fig. 19 , in the second modification, the 3D map storage unit 101 is configured outside the information processing device 100, for example, on the cloud. In addition, a server 600 is provided that performs processing on the 3D map storage unit 101. The server 600 performs processing such as updating the 3D map in the 3D map storage unit 101, checking whether or not a 3D map is available in response to a request from the information processing device 100, and transmitting the 3D map to the information processing device 100.

[0110] The information processing device 100 includes a transmitting unit 108 and a receiving unit 109 for communicating with the server 600. The other configurations of the information processing device 100 are the same as those in the embodiment.

[0111] Based on the position information and direction information, the 3D map acquisition unit 102 inquires of the server 600 via the transmission unit 108 whether a 3D map close to the position and direction of the automobile 10 exists in the 3D map storage unit 101. The server 600 checks whether a 3D map close to the position and direction exists in the 3D map storage unit 101, and if so, transmits the 3D map to the information processing device 100. If a 3D map does not exist, the server 600 transmits a flag or the like to notify the information processing device 100. The 3D map acquisition unit 102 acquires the 3D map or flag via the reception unit 109. Other processing is the same as in the embodiment.

[0112] According to this second variant, it is possible to solve problems such as the difficulty of updating the 3D map because the information processing device 100 operating in a moving body such as an automobile 10 is equipped with a 3D map storage unit 101, and the need for the information processing device 100 to be equipped with a huge storage capacity.

[0113] Furthermore, by providing the 3D map storage unit 101 externally, the 3D map can be updated by accessing the 3D map storage unit 101 via a network, which makes it easier to update the 3D map. Also, it is possible to reduce the amount of storage for the database provided in the automobile 10 in which the information processing device 100 operates.

[0114] [Third Modification] Next, a third modification will be described. If calibration could be performed in a location where the automobile 10 is frequently parked, such as the owner's garage at home, the convenience of calibration is expected to improve. Therefore, when there is no misalignment between the sensors, multiple sensors sense locations where the automobile 10 is frequently parked, and the sensing results are accumulated. Then, the sensing results when there is no misalignment are used to determine the misalignment of the sensors and perform calibration.

[0115] The period when the sensor is not misaligned means, for example, within a certain period after manufacture or within a certain period after calibration. The period can be confirmed using a calendar / timekeeping function provided in the automobile 10.

[0116] 20 , the information processing device 100 of the third modified example includes a sensing result storage unit 110 instead of the 3D map storage unit 101. The information processing device 100 also includes a sensing result management unit 111 instead of the 3D map acquisition unit 102. Other than that, the configuration of the information processing device 100 is the same as that of the first embodiment.

[0117] When there is no deviation in the relative positions between the sensors, if the sensing result management unit 111 confirms based on the location information that the automobile 10 equipped with the sensor is located in a place where automobiles 10 are frequently parked, it links the location information and direction information of the automobile 10 with the sensing results of the sensor and outputs them to the sensing result storage unit 110 for accumulation.

[0118] Then, after the above-described certain period of time has elapsed, the sensing result management unit 111 acquires the sensing result from the sensing result storage unit 107 based on the position information and direction information, and supplies the sensing result to the sensing control unit 104, the deviation determination unit 105, and the calibration processing unit 106. Then, the information processing device 100 performs deviation determination and calibration using the normal sensing result instead of the 3D map in the embodiment.

[0119] In this way, by using the sensing result when there is no deviation in the relative positions between the sensors, the 3D map storage unit 101 becomes unnecessary, and it becomes possible to further increase the opportunities for performing calibration. Note that the 3D map storage unit 101 may be used in combination with this third modified example.

[0120] [Fourth Modification] Next, a fourth modification will be described. If calibration is not performed for a long period of time, the sensor may become misaligned, which may cause malfunctions in functions that use the sensor. Therefore, it is determined whether the distance traveled by the automobile 10 since the last calibration or the time elapsed since the last calibration exceeds a certain threshold. If the travel distance or the elapsed time exceeds the threshold, it is determined that calibration has not been performed for a long time, and a message recommending calibration and guidance information to a nearby calibration location are displayed on a display of a navigation system or the like provided in the automobile 10. This makes it possible to prevent calibration from not being performed for a long period of time.

[0121] A calibration-enabled point is a point where a 3D map exists and where a landmark is present. The guidance information may include a route from the current position on the map to the calibration-enabled point. Guidance may be provided by voice instead of or in addition to a display on the screen.

[0122] [Other Modifications] In the embodiment and modification examples, the mobile body is described as an automobile 10. However, the present technology is not limited to automobiles 10 and can be applied to mobile bodies such as hybrid electric vehicles, motorcycles, bicycles, personal mobility vehicles, airplanes, drones, ships, robots, construction machinery, and agricultural machinery (tractors). The present technology can improve the accuracy of automatic driving, automatic steering, autonomous movement, and the like of these mobile bodies. Furthermore, the information processing device 100 may operate on electronic devices such as personal computers, smartphones, tablet terminals, and wearable devices.

[0123] The present technology can also be configured as follows. (1) An information processing device comprising: a deviation determination unit that determines a deviation between a plurality of sensors based on a 3D map and sensing results of the plurality of sensors; and a calibration processing unit that calibrates the plurality of sensors based on the determination result of the deviation determination unit. (2) The information processing device according to (1), wherein the deviation determination unit determines a deviation between the plurality of sensors and a deviation of the sensors based on the 3D map. (3) The information processing device according to (1) or (2), wherein the plurality of sensors include a camera that outputs a camera image as the sensing result, and a ranging sensor that outputs 3D point cloud information as the sensing result. (4) The information processing device according to (3), wherein the deviation determination unit calculates an error between a projection result obtained by projecting a position of the landmark in the 3D map onto an imaging surface of the camera and a position of the landmark detected from the camera image, and compares the error with a predetermined threshold to determine the deviation between the camera and the ranging sensor. (5) The information processing device according to (4), wherein the position of the landmark in the 3D map is coordinate-transformed into a coordinate system of the distance measuring sensor to obtain a first coordinate transformation result, the first coordinate transformation result is coordinate-transformed into a coordinate system of the camera to obtain a second coordinate transformation result, and the second coordinate transformation result is projected onto an imaging surface of the camera to obtain the projection result. (6) The information processing device according to any of (1) to (5), wherein the deviation determination unit supplies the relative positions between the multiple sensors used in determining the deviation to the calibration processing unit, and the calibration processing unit performs calibration using the relative positions between the multiple sensors. (7) The information processing device according to (6), wherein the relative positions of each of the multiple sensors on the 3D map are combined to determine the relative positions between the multiple sensors. (8) The information processing device according to any of (1) to (7), wherein the sensor is a camera that outputs a camera image as the sensing result.(9) The information processing device according to any one of (1) to (10), wherein the 3D map includes information on landmarks that are objects present in the 3D map, and the deviation determination unit calculates an error between a position of the landmark in the 3D map projected onto the camera image and a position of the landmark detected from the camera image, and determines deviation of the camera by comparing the error with a predetermined threshold. (10) The information processing device according to (9), wherein the deviation determination unit calculates a relative position of the camera on the 3D map based on the position of the landmark detected from the camera image and the position of the landmark in the 3D map, and projects the position of the landmark in the 3D map onto the camera image based on the relative position. (11) The information processing device according to any one of (1) to (10), wherein the sensor is a ranging sensor that outputs a 3D point cloud as the sensing result. (12) The information processing device according to (11), wherein the 3D map includes a 3D point cloud, and the deviation determination unit calculates an error between the 3D point cloud of the 3D map and the 3D point cloud output from the ranging sensor, and determines a deviation of the ranging sensor by comparing the error with a predetermined threshold. (13) The information processing device according to (12), wherein the information processing device calculates a relative position of the ranging sensor on the 3D map based on the 3D point cloud of the 3D map and the 3D point cloud output from the ranging sensor, and performs coordinate transformation of the 3D point cloud of the 3D map based on the relative position, and calculates an error from the 3D point cloud output from the ranging sensor. (14) The information processing device according to any of (1) to (13), further comprising a 3D map acquisition unit that acquires the 3D maps corresponding to the multiple sensors based on positions and orientations of the multiple sensors from a 3D map storage unit that stores multiple 3D maps in advance. (15) The information processing device according to (14), wherein outputs of the plurality of sensors are supplied to the deviation determination unit when the 3D map acquisition unit acquires a 3D map corresponding to the positions of the plurality of sensors. (16) The information processing device according to any one of (1) to (15), wherein the plurality of sensors are provided on a moving body.(17) An information processing method for determining deviations between a plurality of sensors based on a 3D map and sensing results of the plurality of sensors, and calibrating the sensors based on the determination results of the deviation determination unit. (18) A program for causing a computer to execute an information processing method for determining deviations between the plurality of sensors based on a 3D map and sensing results of the plurality of sensors, and calibrating the sensors based on the determination results of the deviation determination unit.

[0124] 10... Automobile 100... Information processing device 101... 3D map storage unit 102... 3D map acquisition unit 105... Deviation determination unit 106... Calibration processing unit 200... Camera 300... LiDAR

Claims

1. An information processing device comprising: a deviation determination unit that determines deviation between multiple sensors based on a 3D map and sensing results of the multiple sensors; and a calibration processing unit that calibrates the multiple sensors based on the determination results of the deviation determination unit.

2. The information processing device according to claim 1, wherein the deviation determination unit determines deviations between the plurality of sensors and deviations of the sensors based on the 3D map.

3. The information processing device according to claim 1, wherein the plurality of sensors include a camera that outputs a camera image as the sensing result, and a distance measuring sensor that outputs 3D point cloud information as the sensing result.

4. The information processing device of claim 3, wherein the deviation determination unit calculates the error between the projection result of the position of the landmark on the 3D map onto the imaging surface of the camera and the position of the landmark detected from the camera image, and compares the error with a predetermined threshold to determine the deviation between the camera and the ranging sensor.

5. The information processing device of claim 4, further comprising: a coordinate transformation unit that transforms the position of the landmark in the 3D map into the coordinate system of the distance measuring sensor to obtain a first coordinate transformation result; a coordinate transformation unit that transforms the first coordinate transformation result into the coordinate system of the camera to obtain a second coordinate transformation result; and a projection unit that projects the second coordinate transformation result onto the imaging surface of the camera to obtain the projection result.

6. The information processing device according to claim 1, wherein the deviation determination unit supplies the relative positions between the plurality of sensors used in determining the deviation to the calibration processing unit, and the calibration processing unit performs calibration using the relative positions between the plurality of sensors.

7. The information processing device according to claim 6, wherein the relative positions of the plurality of sensors on the 3D map are synthesized to determine the relative positions between the plurality of sensors.

8. The information processing device according to claim 1, wherein the sensor is a camera that outputs a camera image as the sensing result.

9. The information processing device according to claim 8, wherein the 3D map includes information on landmarks, which are objects present within the 3D map, and the deviation determination unit determines an error between the position of the landmark in the 3D map projected onto the camera image and the position of the landmark detected from the camera image, and compares the error with a predetermined threshold to determine the deviation of the camera.

10. The information processing device according to claim 9, wherein the deviation determination unit determines a relative position of the camera on the 3D map based on the position of the landmark detected from the camera image and the position of the landmark on the 3D map, and projects the position of the landmark on the 3D map onto the camera image based on the relative position.

11. The information processing device according to claim 1, wherein the sensor is a distance measuring sensor that outputs a 3D point cloud as the sensing result.

12. The information processing device of claim 11, wherein the 3D map includes a 3D point cloud, and the deviation determination unit determines an error between the 3D point cloud of the 3D map and the 3D point cloud output from the ranging sensor, and compares the error with a predetermined threshold to determine the deviation of the ranging sensor.

13. The information processing device according to claim 12, which determines the relative position of the distance measuring sensor on the 3D map based on the 3D point cloud of the 3D map and the 3D point cloud output from the distance measuring sensor, performs coordinate transformation of the 3D point cloud of the 3D map based on the relative position, and determines the error with the 3D point cloud output from the distance measuring sensor.

14. The information processing device according to claim 1, further comprising a 3D map acquisition unit that acquires the 3D maps corresponding to the plurality of sensors based on the positions and directions of the plurality of sensors from a 3D map storage unit that stores a plurality of 3D maps in advance.

15. The information processing device according to claim 14, wherein when the 3D map acquisition unit acquires a 3D map corresponding to the positions of the plurality of sensors, outputs of the plurality of sensors are supplied to the deviation determination unit.

16. The information processing device according to claim 1, wherein the plurality of sensors are provided on a moving body.

17. An information processing method comprising: determining a deviation between a plurality of sensors based on a 3D map and sensing results of the plurality of sensors; and calibrating the sensors based on the determination results of the deviation determination unit.

18. A program that causes a computer to execute an information processing method that determines the deviation between multiple sensors based on a 3D map and the sensing results of the multiple sensors, and calibrates the sensors based on the determination result of the deviation determination unit.

Citation Information

Patent Citations

  • Elevator remote monitoring system

    JP2022048734A

  • Signal processing device and signal processing method

    WO2017159382A1

  • Information processing device, information processing method, and information processing program

    WO2020203657A1