Information processing device, information processing method, and program

Abstract

An information processing device according to one embodiment of the present technology comprises a map acquiring unit and an executing unit. The map acquiring unit acquires map information including a first feature point for performing calibration, on the basis of position information of a vehicle. The executing unit performs calibration of a sensor installed in the vehicle on the basis of the first feature point and a second feature point acquired by the sensor. This make is easier to perform the calibration.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program that can be applied to calibration.

[0002] Patent Document 1 describes an in-vehicle calibration system that calibrates sensors for recognizing the vehicle's driving state. This in-vehicle calibration system determines whether the vehicle has reached a preset calibration point, and calibrates the internal sensors based on road information such as the position, gradient, and curvature of the road at the calibration point and the vehicle's driving state. This allows for appropriate calibration of the vehicle while it is moving (see, for example, paragraphs

[0037] to

[0056] and Figures 1 to 3 of the specification of Patent Document 1).

[0003] JP 2018-96715 A

[0004] There is a demand for an information processing device, an information processing method, and a program that can more easily perform calibration without using a dedicated space for such calibration.

[0005] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a program that enable calibration to be performed more easily.

[0006] In order to achieve the above object, an information processing device according to an embodiment of the present technology includes a map acquisition unit and an execution unit. The map acquisition unit acquires map information including first feature points for performing calibration based on vehicle position information. The execution unit calibrates the sensor based on the first feature points and second feature points acquired by a sensor mounted on the vehicle.

[0007] In this information processing device, map information including first feature points for performing calibration is acquired based on vehicle position information. Furthermore, sensor calibration is performed based on the first feature points and second feature points acquired by a sensor mounted on the vehicle. This makes it possible to perform calibration more easily.

[0008] If the map information does not exist, the map acquisition unit may create map information indicating a map of the area around the vehicle.

[0009] The map information may include map information created by a vehicle other than the vehicle.

[0010] The information processing device may further include a feature point recognition unit that recognizes the second feature point based on sensing information acquired from the sensor.

[0011] The information processing device may further include a class information generation unit that generates class information for objects present around the vehicle based on sensing information acquired from the sensor.

[0012] The feature point recognition unit may recognize the second feature point for an object in which the first feature point exists, based on the class information.

[0013] The information processing apparatus may further include a determination unit that determines whether or not the calibration is executable based on a degree of correspondence between the first feature points and the second feature points.

[0014] The determination unit may determine whether or not the calibration is executable based on the number of the first feature points and the second feature points that are associated with each other.

[0015] The execution unit may calculate a calibration parameter based on a difference between the first feature point and the second feature point.

[0016] According to an embodiment of the present technology, there is provided an information processing method executed by a computer system, the information processing method including acquiring map information including first feature points for performing calibration based on position information of a vehicle, and performing calibration of the sensor based on the first feature points and second feature points acquired by a sensor mounted on the vehicle.

[0017] A program according to an embodiment of the present technology causes a computer system to execute the following steps: acquiring map information including first feature points for performing calibration based on position information of a vehicle; and calibrating the sensor based on the first feature points and second feature points acquired by a sensor mounted on the vehicle.

[0018] It is a block diagram showing an example of the configuration of an autonomous driving system.It is a diagram showing a flowchart related to the creation of calibration parameters.It is a schematic diagram showing an example of map information.

[0019] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0020] FIG. 1 is a block diagram showing an example configuration of an autonomous driving system according to the present technology.

[0021] As shown in FIG. 1 , the autonomous driving system 100 is provided in a vehicle 1 and includes a sensor 2 , an I / F 6 , a display device 7 , a communication unit 8 , and an information processing device 20 .

[0022] The sensor 2, the I / F 6, the display device 7, the communication unit 8, and the information processing device 20 are connected to each other via a communication network 15 so as to be able to communicate with each other. The communication network 15 is configured, for example, by an in-vehicle communication network or bus conforming to a digital two-way communication standard such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), FlexRay (registered trademark), or Ethernet (registered trademark). Different communication networks 15 may be used depending on the type of data being transmitted. For example, a CAN may be used for data related to vehicle control, and an Ethernet may be used for large-volume data. Note that the components of the autonomous driving system 100 may be directly connected to each other using wireless communication intended for relatively short-range communication, such as near-field communication (NFC) or Bluetooth (registered trademark), without using the communication network 15.

[0023] The sensor 2 includes a camera 3, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 4, and a GPS receiver 5. For example, the camera 3 includes an image sensor and captures images of the traveling direction of the vehicle 1 and the surrounding environment. The LiDAR 4 performs light detection and distance measurement to acquire a point cloud of objects present around the vehicle 1. The GPS receiver 5 receives signals from multiple satellites that make up a satellite navigation system, and calculates the position information (latitude and longitude) of the GPS receiver 5 (vehicle 1) by performing calculations based on the received signals.

[0024] The method for calculating the position information of the vehicle 1 is not limited, and the Global Navigation Satellite System (GNSS) may be used. The sensor for recognizing the situation outside the vehicle 1 is also not limited, and any sensor such as a radar or an ultrasonic sensor may be installed.

[0025] The information acquired by the sensor 2 is supplied to the information processing device 20. The sensor 2 is not limited to the above and may be configured to include various types of radar. The number of sensors 2 is not particularly limited as long as it is a number that can be realistically installed on the vehicle 1.

[0026] The I / F 6 exchanges information with the information processing device 20 and other devices that make up the autonomous driving system 100.

[0027] The display device 7 is, for example, a display device using a liquid crystal display, an electroluminescence display (EL), or the like. For example, a GUI (Graphical User Interface) for operating the autonomous driving system 100 is displayed on the display device 7, and the display device 7 is also used as an input device such as a touch panel. For example, the display device 7 may display a message announcing the start or completion of acquisition or creation of map information (described later), or may display a dedicated symbol (mark) indicating that calibration has been performed.

[0028] Note that an input device for inputting operations may be installed in addition to the display device 7. For example, a keyboard or buttons may be provided, or a voice recognition system and a microphone for collecting sound may be installed. For example, the user may be able to check the number and type of map information currently stored, the date calibration was performed, etc., via the input device.

[0029] The communication unit 8 wirelessly transmits and receives information between devices external to the vehicle 1 and the information processing device 20. For example, when the map acquisition unit 21 of the information processing device 20 acquires map information stored on the cloud or transmits created map information to the cloud, the map information is transmitted and received via the communication unit 8.

[0030] Furthermore, the exchange of map information is not limited to the cloud, but may be performed via V2V (Vehicle to Vehicle). In this case, even if the two vehicles are not in the same place, the exchange of map information estimated at each location can be performed.

[0031] The information processing device 20 has hardware necessary for configuring a computer, such as a processor such as a CPU, GPU, or DSP, a memory such as a ROM or RAM, a storage device such as an HDD, etc. For example, the CPU loads a program according to the present technology, which is pre-recorded in the ROM or the like, into the RAM and executes the program, thereby executing the information processing method according to the present technology.

[0032] For example, the information processing device 20 can be realized by any computer such as a PC. Of course, hardware such as an FPGA or an ASIC may also be used. In this embodiment, the CPU executes a predetermined program to configure the determination unit 25 and other functional blocks. Of course, dedicated hardware such as an IC (integrated circuit) may also be used to realize the functional blocks.

[0033] The program is installed in the information processing device 20 via, for example, various recording media. Alternatively, the program may be installed via the Internet, etc. In this embodiment, a program for executing the autonomous driving system 100 (described later) is stored in the ROM, and is expanded and executed in the RAM.

[0034] The type of recording medium on which the program is recorded is not limited, and any computer-readable recording medium may be used. For example, any computer-readable non-transitory storage medium may be used.

[0035] As shown in FIG. 1, the information processing device 20 includes a map acquisition unit 21 , a self-position estimation unit 22 , a feature point recognition unit 23 , a class information generation unit 24 , a determination unit 25 , an execution unit 26 , and a storage unit 27 .

[0036] The map acquisition unit 21 acquires map information including feature points for performing calibration. The map information includes location information (e.g., latitude and longitude) of a predetermined location and feature points of characteristic objects in the predetermined location. For example, the map information includes point clouds associated with various objects such as buildings such as landmarks, traffic lanes, streetlights, signs, or parking barriers. In other words, the map information includes location information of the objects.

[0037] Furthermore, the map information may include only feature points necessary for performing calibration extracted from the point cloud. That is, only feature points suitable for performing calibration may be included in the map information. Whether or not a feature point is suitable for performing calibration may be determined based on feature points used when calibration was performed by the previous vehicle 1 or another vehicle, or may be determined based on the accuracy of the generated calibration parameters.

[0038] In the following description, a feature point included in map information will be referred to as a first feature point. The first feature point includes location information (x, y, z coordinates) of the feature point and information about the object to which the feature point is assigned.

[0039] In this embodiment, acquiring a map includes acquiring a map that has already been created and preparing (creating) a map on your own. For example, the map acquisition unit 21 can create map information about the area around the vehicle 1 based on sensing information acquired from the sensor 2.

[0040] The self-position estimation unit 22 estimates the self-position of the vehicle 1. For example, the self-position estimation unit 22 estimates the position of the vehicle 1 on a map based on sensing information acquired from the sensor 2. Note that the method for estimating the self-position is not limited, and for example, the self-position may be estimated by matching surrounding stationary objects with feature points of objects in map information.

[0041] The feature point recognition unit 23 recognizes feature points present around the vehicle 1. In this embodiment, the feature point recognition unit 23 recognizes the feature points based on sensing information acquired from the sensor 2.

[0042] The class information generation unit 24 generates class information for each object from the sensing information acquired from the sensor 2. In this embodiment, the class information generation unit 24 generates various class information for objects around the vehicle 1, such as parked cars, people, and buildings, based on the sensing information. For example, the class information generation unit 24 classifies the class of each object using 3D semantic segmentation.

[0043] The feature point recognition unit 23 may recognize all of the feature points present around the vehicle 1, or may extract only some of them. For example, only feature points that match feature points included in map information, such as signs, white lines, three-dimensional structures, or traffic lights, may be extracted. Specifically, the positions (x, y, z coordinates), orientations (azimuths), or classes (traffic lights, signs, etc.) of the feature points acquired from the sensor 2 may be compared with first feature points, and only feature points that correspond to the first feature points may be extracted.

[0044] In the following description, the feature points present around the vehicle 1 acquired from the sensor 2 will be referred to as second feature points.

[0045] That is, the feature point recognition unit 23 associates the first feature point with the second feature point. For example, the feature point recognition unit 23 sets corresponding feature points as pairs using IDs or the like. Note that the method of association is not limited, and any method such as Euclidean distance, shortest ICP of Mahalanobis distance, or Hungarian distance may be used.

[0046] The determination unit 25 determines whether calibration can be performed. In this embodiment, the determination unit 25 determines whether the necessary correspondences for performing calibration have been performed.

[0047] The execution unit 26 calibrates the sensor 2 based on the first feature points and the second feature points. In this embodiment, the execution unit 26 matches pairs of the first feature points and the second feature points associated by the feature point recognition unit 23. In this embodiment, the map information including the first feature points is taken as correct, and the second feature points (sensor side) are matched. Any method may be used to match the first feature points and the second feature points. For example, optimization may be performed so that all of the associated feature points match, or weighting may be applied to the associated feature points and optimization may be performed based on the weighting.

[0048] Calibration in this embodiment refers to checking where the sensor 2 is attached to the vehicle 1 and how it is attached (such as the orientation of the camera 3). That is, the execution unit 26 specifies the position of the mounted sensor 2 using the vehicle 1 as a reference (for example, the rear wheel axle of the wheels).

[0049] The storage unit 27 stores map information. In this embodiment, the storage unit 27 stores map information acquired by the map acquisition unit 21 or already created map information acquired via the communication unit 8.

[0050] In this embodiment, the timing for acquiring already created map information may be, for example, when approaching a toll booth installed on a highway, etc., surrounding map information may be acquired wirelessly. In addition, map information around a destination may be acquired depending on a situation such as when the destination is set in a navigation system or the like.

[0051] Fig. 2 is a diagram showing a flowchart related to the creation of calibration parameters. Fig. 3 is a schematic diagram showing an example of map information. In this embodiment, the vehicle 1 is in a traveling state, and the recognition of feature points and the creation of calibration parameters are performed while the vehicle 1 is traveling. Of course, if a place suitable for performing calibration is a place where the vehicle can stop, the calibration may be performed while the vehicle is stopped.

[0052] 2, the map acquisition unit 21 determines whether or not there is any map information that matches the search criteria near the vehicle 1 (step 101). If the storage unit 27 stores the map information that matches the search criteria (YES in step 101), the stored map information is selected. If there are multiple pieces of map information that match the search criteria, the newer map information may be selected.

[0053] For example, as shown in Fig. 3, in map information showing a parking lot, first feature points (see dotted line 30) are assigned to parking lot numbers, white lines for guidance, white lines indicating stopping positions, etc. In addition to these, feature points may also be assigned to buildings outside the parking lot. Note that Fig. 3 illustrates some of the first feature points and second feature points, and does not include vehicles other than vehicle 1.

[0054] In this embodiment, when there are multiple pieces of map information that correspond to the map information to be acquired, or when the initial position of the vehicle 1 is to be selected, the map information is acquired using GNSS or the like.

[0055] If no map information is stored (NO in step 101), the map acquisition unit 21 creates map information for the vicinity of the vehicle 1 based on the sensing information (step 102).

[0056] In this embodiment, as shown in FIG. 3 , a parking lot is exemplified as an example of map information, but the map may also include a three-dimensional map of a multi-story parking lot or the like. Furthermore, in addition to the first feature point, the map information may also include a calibration execution point, which is the most suitable point for performing calibration. For example, calibration may be performed when the vehicle 1 reaches the calibration execution point. In other words, information that serves as a trigger for performing calibration may be included in the map information.

[0057] The self-position estimation unit 22 estimates the location of the vehicle 1 on the map from the sensing information (step 103). The feature point recognition unit 23 also recognizes second feature points (see solid line 40) that exist around the vehicle 1 from the sensing information (step 104).

[0058] In this embodiment, class information of surrounding objects is generated from the sensing information by the class information generation unit 24. For example, in Fig. 3, the class information generation unit 24 generates class information such as parking number plates, white lines for guidance, and white lines indicating stopping positions, thereby classifying the classes of the second feature points.

[0059] Further, the feature point recognition unit 23 associates the first feature point with the second feature point (step 105). For example, in Fig. 3, a pair 45a of the first feature point 30 and the second feature point 40 assigned to the parking number, a pair 45b of the first feature point 30 and the second feature point 40 assigned to the guiding white line, a pair 45c of the first feature point 30 and the second feature point 40, and a pair 45d of the first feature point 30 and the second feature point 40 assigned to the white line indicating the stopping position are associated with each other.

[0060] Note that only position information may be used to associate the first feature points and the second feature points. In this embodiment, class information is assigned to the first feature points and the second feature points, which makes it possible to improve the accuracy of the association.

[0061] The determination unit 25 determines whether calibration is executable based on the associated feature points (step 106). For example, the determination unit 25 determines that calibration is executable when there are four or more pairs 45 of first feature points 30 and second feature points 40 associated by the feature point recognition unit 23.

[0062] If calibration is possible (YES in step 106), the execution unit 26 performs matching between pairs of the first feature points and the second feature points (step 107). In this embodiment, the execution unit 26 estimates the relative relationship between the position of the vehicle 1 and the position of the sensor mounted on the vehicle 1, based on map information including the first feature points. In other words, the amount of change in the matching result (the difference between the feature points of the vehicle 1 and the second feature points) is the value to be adjusted by this calibration.

[0063] The execution unit 26 also creates calibration parameters from the amount of change in the matching result calculated from the map information (step 108). In this embodiment, the calibration parameters are converted into a coordinate system required for the vehicle 1. For example, if an arbitrary coordinate system has not been established at the stage of step 107, the calibration parameters are adjusted to a coordinate system required by the rear wheel axle, the center of the bumper, etc. of the vehicle 1.

[0064] As described above, the information processing device 20 according to this embodiment acquires map information including the first feature points 30 for performing calibration based on the position information of the vehicle 1. Furthermore, calibration of the sensor 2 is performed based on the first feature points 30 and the second feature points 40 acquired by the sensor 2 mounted on the vehicle 1. This makes it possible to perform calibration more easily.

[0065] Conventionally, when calibration is performed, it is necessary to prepare calibration spot data in advance using dedicated equipment, which places a heavy burden on the user and causes problems in terms of cost.

[0066] This technology matches feature points in map information created by other vehicles with feature points recognized by the vehicle itself, allowing for easy calibration without requiring the user to take any special action. In addition, because the user can create their own map information, there are fewer limitations, such as not being able to perform calibration due to the lack of map information.

[0067] Furthermore, because map information created by other vehicles can be used, updates are frequent, and calibration can be performed even when the environment changes, such as signs changing or disappearing, roads changing, or vegetation growing and hiding signs, making existing map information unusable.In addition, because map information is created by other vehicles, the range of use can be expanded widely, even to places that users do not usually go to.

[0068] Furthermore, since map information can be created by the vehicle itself or other vehicles, there is no need for a dedicated vehicle to create dedicated map information, which also reduces costs.

[0069] The configurations of the map acquisition unit, determination unit, execution unit, etc. described with reference to the drawings are merely one embodiment and can be modified as desired without departing from the spirit of the present technology. In other words, any other configurations, algorithms, etc. for implementing the present technology may be adopted.

[0070] It should be noted that the effects described in this disclosure are merely examples and are not limiting, and other effects may also be present. The description of multiple effects above does not necessarily mean that these effects are exhibited simultaneously. It means that at least one of the effects described above can be obtained depending on the conditions, etc., and of course, effects not described in this disclosure may also be exhibited.

[0071] It is also possible to combine at least two of the characteristic features of each embodiment described above. In other words, the various characteristic features described in each embodiment may be combined in any manner without distinguishing between the embodiments.

[0072] The present technology may also have the following configurations. (1) An information processing device comprising: a map acquisition unit that acquires map information including first feature points for performing calibration based on vehicle position information; and an execution unit that calibrates the sensor based on the first feature points and second feature points acquired by a sensor mounted on the vehicle. (2) The information processing device described in (1), wherein the map acquisition unit creates map information indicating a map of the periphery of the vehicle when the map information does not exist. (3) The information processing device described in (1), wherein the map information includes map information created by a vehicle different from the vehicle. (4) The information processing device described in (1), further comprising: a feature point recognition unit that recognizes the second feature points based on sensing information acquired from the sensor. (5) The information processing device described in (4), further comprising: a class information generation unit that generates class information for objects present around the vehicle based on sensing information acquired from the sensor. (6) The information processing device according to (5), wherein the feature point recognition unit recognizes the second feature point for an object in which the first feature point exists based on the class information. (7) The information processing device according to (1), further comprising a determination unit that determines whether the calibration is executable based on a degree of correspondence between the first feature point and the second feature point. (8) The information processing device according to (7), wherein the determination unit determines whether the calibration is executable based on the number of corresponding first feature points and second feature points. (9) The information processing device according to (1), wherein the execution unit calculates calibration parameters based on a difference between the first feature point and the second feature point.(10) An information processing method executed by a computer system, which comprises: acquiring map information including first feature points for performing calibration based on vehicle position information; and calibrating the sensor based on the first feature points and second feature points acquired by a sensor mounted on the vehicle. (11) A program that causes a computer system to execute the steps of: acquiring map information including first feature points for performing calibration based on vehicle position information; and calibrating the sensor based on the first feature points and second feature points acquired by a sensor mounted on the vehicle.

[0073] REFERENCE SIGNS LIST 1 vehicle 2 sensor 3 camera 5 GPS receiver 20 information processing device 21 map acquisition unit 23 feature point recognition unit 24 class information generation unit 25 determination unit 26 execution unit 30 first feature point 40 second feature point 100 automatic driving system

Claims

1. An information processing device comprising: a map acquisition unit that acquires map information including first feature points for calibration based on vehicle position information; and an execution unit that calibrates the sensor based on the first feature points and second feature points acquired by a sensor mounted on the vehicle.

2. An information processing device according to claim 1, wherein the map acquisition unit creates map information showing a map of the area around the vehicle when the map information does not exist.

3. An information processing device according to claim 1, wherein the map information includes map information created by a vehicle other than the vehicle.

4. An information processing device according to claim 1, further comprising: a feature point recognition unit that recognizes the second feature point based on sensing information acquired from the sensor.

5. An information processing device according to claim 4, further comprising a class information generating unit that generates class information for objects present around the vehicle based on sensing information acquired from the sensor.

6. An information processing device according to claim 5, wherein the feature point recognition unit recognizes the second feature point for an object in which the first feature point exists, based on the class information.

7. An information processing device according to claim 1, further comprising: a determination unit that determines whether or not the calibration can be performed based on the degree of correspondence between the first feature points and the second feature points.

8. An information processing device according to claim 7, wherein the determination unit determines whether or not the calibration can be performed based on the number of associated first feature points and second feature points.

9. An information processing device according to claim 1, wherein the execution unit calculates a calibration parameter based on a difference between a first feature point and a second feature point.

10. An information processing method executed by a computer system, which comprises: acquiring map information including first feature points for calibration based on vehicle position information; and calibrating the sensor based on the first feature points and second feature points acquired by a sensor mounted on the vehicle.

11. A program that causes a computer system to execute the steps of: acquiring map information including first feature points for calibration based on vehicle position information; and calibrating the sensor based on the first feature points and second feature points acquired by a sensor mounted on the vehicle.

Images