Information processing device, information processing method and information processing program

Abstract

An information processing device (100) includes a calibration execution unit (195) that performs calibration between two or more sensors, the two or more sensors being attached at different positions and having fields of view that at least partially overlap each other. In a region where the fields of view of the two or more sensors overlap each other, the calibration execution unit (195) performs calibration between the sensors when a feature point that enables calibration of each of the two or more sensors is obtainable.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. Background Art

[0002] In recent years, in order to recognize information required for autonomous driving and driving assistance, driving assistance systems that use information acquired from a plurality of on-vehicle sensors in combination have been commercialized.

[0003] In the above-mentioned driving assistance system, when various pieces of information acquired from a plurality of vehicle-mounted sensors are used in combination, detection errors may be included in the information to be used due to deviations in distance and azimuth between the vehicle-mounted sensors. In view of this, a technique for calibrating the vehicle-mounted sensors based on map information has been proposed, for example.

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: JP 2018-96715A Summary of the invention

[0007] Technical issues

[0008] Since the above-mentioned vehicle-mounted sensor calibration technology is based on map information, calibration cannot be performed in some cases. For example, calibration cannot be performed in areas without map information. In addition, calibration cannot be performed when the environment changes due to construction, etc.

[0009] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program that can perform calibration even for an area without map information without being affected by environmental changes.

[0010] Solution to the problem

[0011] In order to solve the above-mentioned problem, an information processing device according to an embodiment of the present disclosure includes: a calibration execution unit, which performs calibration between two or more sensors, wherein the two or more sensors are attached at different positions and their field of view areas at least partially overlap with each other, wherein in the area where the fields of view of the two or more sensors overlap with each other, the calibration execution unit performs calibration between the sensors when feature points that enable calibration of each of the two or more sensors are obtainable. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is an overview diagram showing information processing according to an embodiment of the present disclosure.

[0013] Figure 2is a diagram showing an example of a configuration of a management device according to an embodiment of the present disclosure.

[0014] Figure 3 is a diagram showing an example of a configuration of an information processing device according to an embodiment of the present disclosure.

[0015] Figure 4 is a flowchart showing the flow of information processing according to an embodiment of the present disclosure.

[0016] Figure 5 is a diagram showing an example of detection by LiDAR.

[0017] Figure 6 1 is a block diagram showing a configuration example of schematic functions of a moving body control system to which information processing according to an embodiment of the present disclosure can be applied.

[0018] Figure 7 is a hardware configuration diagram showing an example of a computer that realizes the function of the information processing device of the present disclosure. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the following embodiments, the same reference numerals denote the same parts, and repeated descriptions may be omitted.

[0020] Furthermore, the present disclosure will be described in the following order.

[0021] 1. Example

[0022] 1-1. Overview of information processing according to the embodiment

[0023] 1-2. Configuration of management device according to embodiment

[0024] 1-3. Configuration of information processing apparatus according to embodiment

[0025] 1-4. Information processing process according to the embodiment

[0026] 1-5. Modification examples according to the embodiment

[0027] 2. Other embodiments

[0028] 2-1. Other sensors

[0029] 2-2. Configuration of mobile objects

[0030] 2-3. Others

[0031] 3. Effects of the information processing device according to the present disclosure

[0032] 4. Hardware Configuration

[0033] (1. Example)

[0034] [1-1. Overview of information processing according to the embodiment]

[0035] Figure 1 1 is a schematic diagram showing information processing according to an embodiment of the present disclosure. The information processing according to an embodiment of the present disclosure involves, for example, calibration of a vehicle-mounted sensor installed in a vehicle. The vehicle in the following embodiment is, for example, a car. The information processing according to this embodiment is performed by an information processing device 100 installed in the car.

[0036] Generally, in a driving assistance system using information acquired from a plurality of on-vehicle sensors, a detection error may be included in the information acquired from the plurality of on-vehicle sensors due to deviations in distances and azimuths between the on-vehicle sensors.

[0037] On the other hand, for example, although a technique for calibrating an on-vehicle sensor based on map information has been proposed, since the calibration is performed based on map information, in some cases, the calibration cannot be performed when there is no map information. For example, since the calibration technique is based on map information, the calibration cannot be performed in a car traveling in an area where there is no map information. In addition, in the calibration technique, the calibration cannot be performed in the case where an environmental change occurs due to construction or the like and a deviation occurs between the actual environment and the map information.

[0038] In this regard, a method is proposed as follows: the information processing device 100 according to the present embodiment performs the following information processing so as to perform calibration even for an area without map information without being affected by environmental changes. That is, the outline of the information processing performed by the information processing device 100 is to make a specific object mounted on a vehicle serve as a feature point for calibration between vehicle-mounted sensors.

[0039] The information processing device 100 is installed in Figure 1 In each of the vehicles 1 and 2 shown, similar information processing is performed. In the following description, the following case will be described: the information processing device 100 on the vehicle 1 performs calibration between the on-board sensors installed on the vehicle 1 by using the feature points of a specific object (such as the calibration jig 4 described below) installed on the vehicle 2. In the following description, the vehicle 1 may be referred to as a "host vehicle", and the vehicle 2 on which the specific object having the feature points for calibration between the on-board sensors installed on the vehicle 1 may be referred to as a "target vehicle".

[0040] Figure 1 The illustrated vehicle 1 includes a receiving section 12 , an imaging section 13 , and a measuring section 14 which serve as external sensors, and an information processing device 100 .

[0041] The receiving unit 12 is, for example, a GPS receiver. The information processing device 100 transmits the position information to the management device 3 based on the signal received by the receiving unit 12 .

[0042] The imaging unit 13 and the measuring unit 14 are attached at different positions on the vehicle 1. The field of view area FV1 of the imaging unit 13 and the field of view area FV2 of the measuring unit 14 form a region FV3 in which the field of view area FV1 and the field of view area FV2 at least partially overlap each other. The imaging unit 13 is, for example, a camera, and the measuring unit 14 is, for example, a millimeter wave radar. The imaging unit 13 and the measuring unit 14 are examples of two or more sensors.

[0043] The information processing device 100 transmits and receives various information by, for example, vehicle-to-vehicle communication (V2X communication) performed between the information processing device 100 and the management device 3. The management device 3 manages the positional relationship between the vehicles 1 and 2 based on the positional information received from the vehicles 1 and 2.

[0044] In addition, the information processing device 100 obtains information related to the target vehicle from the management device 3. When the vehicle 2 is the target vehicle, the information processing device 100 can obtain information such as the position and posture of the target vehicle, the installation position and posture of the calibration jig 4 in the target vehicle, and the positional relationship between the checkerboard 41 and the corner reflector 42.

[0045] The calibration jig 4 includes a checkerboard 41 and a corner reflector 42, from which feature points 41P that enable calibration of the imaging unit 13 can be extracted, and a corner reflector 42 having feature points 42P that enable calibration of the measuring unit 14. The checkerboard 41 is installed at a position and direction where at least an oncoming vehicle can capture its image. The corner reflector 42 is installed at a position and direction where at least radar light emitted from an oncoming vehicle can be reflected.

[0046] The information processing device 100 acquires information about whether a target vehicle having feature points that enable calibration between the imaging unit 13 and the measuring unit 14 exists within the acquisitive range around the host vehicle from the management device 3. Then, in the case where the feature points 41P and 42P can be acquired in the area FV3 where the field of view areas of the imaging unit 13 and the measuring unit 14 overlap each other, the information processing device 100 performs calibration between the imaging unit 13 and the measuring unit 14.

[0047] Specifically, the information processing device 100 performs coordinate transformation on the installation position of the calibration jig 4 in the target vehicle based on the position of the target vehicle and the installation position of the calibration jig 4 in the target vehicle, and specifies the position of the calibration jig 4 in the global coordinate system. Next, the information processing device 100 determines whether the target vehicle exists in the area where calibration can be performed based on information about the area FV3 where the field of view areas of the imaging unit 13 and the measuring unit 14 overlap with each other and the position information of the calibration jig 4 installed on the target vehicle. For example, when the calibration jig 4 installed on the target vehicle is positioned in the area FV3 where the field of view areas of the imaging unit 13 and the measuring unit 14 overlap with each other, the information processing device 100 determines that the target vehicle exists in the area where calibration can be performed.

[0048] Then, the information processing device 100 determines whether the feature point 41P and the feature point 42P can be acquired from the calibration jig 4 based on the position information and the posture information of the host vehicle and the position information and the posture information of the target vehicle. Then, in the case where it is determined that the feature point 41P and the feature point 42P can be acquired from the calibration jig 4, the information processing device 100 acquires the feature point 41P and the feature point 42P, and performs distance and azimuth calibration between the imaging unit 13 and the measuring unit 14.

[0049] In this way, the information processing apparatus 100 according to the present embodiment can perform calibration even for an area without map information, without being affected by environmental changes.

[0050] [1-2. Configuration of management device according to embodiment]

[0051] Will refer to Figure 2 The configuration of the management device 3 according to the embodiment of the present disclosure is described. Figure 2 FIG. 1 is an exemplary diagram showing a configuration of a management device according to an embodiment of the present disclosure. Figure 2 As shown, the management device 3 includes a communication unit 31, a storage unit 32, and a control unit 33. The management device 3 manages the positional relationship of vehicles based on information acquired from roadside devices such as wireless stations installed on the road.

[0052] The communication unit 31 is implemented by, for example, a network interface card (NIC) or the like. The communication unit 31 may be a wired interface or a wireless interface. The communication unit 31 may be a wireless communication interface of a wireless LAN system or a cellular communication system. The communication unit 31 is connected to a network N (Internet, etc.) in a wired or wireless manner, and transmits information to and receives information from other devices such as the information processing device 100 via the network N.

[0053] The storage unit 32 is implemented by, for example, a random access memory (RAM) or a semiconductor storage element such as a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 32 stores programs and various data that provide functions for implementing information processing performed by the control unit 33. The storage unit 32 can be used as a buffer memory when the control unit 33 performs information processing. The storage unit 32 stores the position information of the vehicle (vehicle 1, vehicle 2, etc.) received from the information processing device 100.

[0054] The control section 33 is implemented by, for example, a central processing unit (CPU), a microprocessor (MPU), a graphics processing unit (GPU), etc. executing a program (for example, an information processing program according to the present disclosure) stored in the information processing device 100 using a random access memory (RAM) or the like as a work area. In addition, the control section 33 is a controller and can be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0055] The control section 33 specifies the positional relationship between the vehicles based on the positional information of the vehicles (vehicle 1, vehicle 2, etc.) received from the information processing device 100. Then, the control section 33 sends a notification indicating the existence of the target vehicle to each of the information processing devices 100 mounted on the vehicles in the positional relationship in which calibration can be performed. Examples of being in the positional relationship in which calibration can be performed include a case where the distance between the vehicles is within a predetermined distance range.

[0056] In addition, the control unit 33 relays vehicle information exchanged between vehicles based on the positional relationship between the vehicles. For example, when the vehicle 1 and the vehicle 2 are in a positional relationship that allows calibration, the control unit 33 transmits the vehicle information received from the vehicle 1 to the vehicle 2, and transmits the vehicle information received from the vehicle 2 to the vehicle 1.

[0057] [1-3. Configuration of information processing device according to embodiment]

[0058] Will refer to Figure 3 The configuration of the information processing device 100 according to the embodiment of the present disclosure is described. Figure 3 is an example diagram showing the configuration of the information processing device 100 according to an embodiment of the present disclosure. In the following description, when the vehicle 1 and the vehicle 2 are in a positional relationship in which calibration can be performed and the subject vehicle described is the vehicle 1, the vehicle 2 is the target vehicle. Conversely, in the following description, when the vehicle 1 and the vehicle 2 are in a positional relationship in which calibration can be performed and the subject vehicle described is the vehicle 2, the vehicle 1 is the target vehicle. Since the vehicle 1 and the vehicle 2 have the same configuration, an example of the configuration of the vehicle 1 will be described below.

[0059] like Figure 3As shown, the vehicle 1 includes a receiving unit 12 , an imaging unit 13 , a measuring unit 14 , a detecting unit 15 , and an information processing device 100 .

[0060] The receiving section 12 receives a radio wave signal of a predetermined frequency band used for a satellite positioning system such as a GPS from a GPS satellite. The receiving section 12 performs demodulation processing on the received radio wave signal and transmits the processed signal to the information processing device 100 or the like.

[0061] The imaging unit 13 is a camera capable of capturing a dynamic image or a static image of the condition outside the host vehicle. The imaging unit 13 is implemented, for example, by a stereo camera, a monocular camera, or an infrared camera. The imaging unit 13 may be a camera other than the above, such as a time-of-flight (ToF) camera. The imaging unit 13 sends the captured image to the information processing device 100, etc.

[0062] The measuring unit 14 is a sensor for measuring the distance to objects around the host vehicle. The measuring unit 14 is realized by a distance measurement system using a millimeter wave radar, for example.

[0063] The detection unit 15 detects information indicating the state of the host vehicle. The detection unit 15 is implemented, for example, by an inertial measurement system, such as an inertial measurement unit (IMU) including a vehicle speed sensor, an acceleration sensor, an angular velocity sensor, etc. The detection unit 15 acquires detection values ​​as information indicating the state of the host vehicle, such as the running speed of the host vehicle, the acceleration acting on the host vehicle, and the angular velocity acting on the host vehicle. The detection unit 15 transmits the information indicating the state of the host vehicle to the information processing device 100, etc.

[0064] Back to Figure 3 The information processing device 100 mounted on the vehicle 1 includes a communication unit 101 , an input unit 103 , an output unit 105 , a storage unit 107 , and a control unit 109 .

[0065] The communication unit 101 is implemented by, for example, a network interface card (NIC) or the like. The communication unit 101 may be a USB interface including a universal serial bus (USB) host controller, a USB port, or the like. In addition, the communication unit 101 may be a wired interface or a wireless interface. The communication unit 101 may be a wireless communication interface of a wireless LAN system or a cellular communication system. The communication unit 101 is connected to a network N (such as the Internet) in a wired or wireless manner, and transmits and receives information with other devices such as the management device 3 via the network N.

[0066] The input unit 103 is a processing unit for receiving various operations from a user or the like who uses the information processing apparatus 100. The input unit 103 receives input of various types of information via, for example, a keyboard, a touch panel, or the like.

[0067] The output section 105 is a processing section for outputting various types of information. The output section 105 is, for example, a display, a speaker, etc. For example, the output section 105 can display an image captured by the imaging section 13 .

[0068] The storage unit 107 is implemented by, for example, a semiconductor storage element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 107 stores programs and various data that provide functions for implementing information processing performed by the control unit 109. When the control unit 109 performs information processing, the storage unit 107 can be used as a buffer memory.

[0069] The storage unit 107 includes a host vehicle information storage unit 171 that stores host vehicle information. The host vehicle information includes an area FV3 where the field of view areas of the imaging unit 13 and the measuring unit 14 overlap each other, an installation position of the calibration jig 4 in the host vehicle, and information on the relative positional relationship between the imaging unit 13 and the measuring unit 14 when they are installed in the vehicle.

[0070] The control unit 109 is implemented, for example, by a central processing unit (CPU), a microprocessor (MPU), a graphics processing unit (GPU), etc. executing a program (for example, an information processing program according to the present disclosure) stored in the information processing device 100 using a random access memory (RAM) or the like as a work area. In addition, the control unit 109 is a controller and can be implemented, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0071] like Figure 3 As shown, the control unit 109 includes a providing unit 191, an acquiring unit 192, a recognizing unit 193, a determining unit 194, and an executing unit 195, and implements or executes the functions or actions of information processing described below. Note that the internal configuration of the control unit 109 is not limited to Figure 3 The configuration shown is not limited to the configuration shown, and may be another configuration as long as the information processing described later is performed. The determination section 194 is an example of a determination section, and the execution section 195 is an example of a calibration execution section.

[0072] The providing unit 191 periodically communicates with the management device 3 through V2X communication (road-to-vehicle communication) and provides information about the host vehicle to the management device 3. The information about the host vehicle includes the position information and posture information of the host vehicle, and information about the calibration jig 4 in the host vehicle (host vehicle information). The position information and posture information of the host vehicle are acquired by the identification unit 193 described later.

[0073] The acquisition unit 192 receives information about the following through V2X communication (road-to-vehicle communication): whether there is a specific object with characteristic points that enable calibration between two or more sensors within the obtainable range around the main vehicle. For example, the acquisition unit 192 receives a notification from the management device 3 indicating the presence of a target vehicle. In addition, the acquisition unit 192 acquires information about a specific object with characteristic points that enable calibration of each of the two or more sensors. For example, the acquisition unit 192 can acquire information about the position and posture of the target vehicle, the installation position and posture of the calibration jig 4 in the target vehicle, and the positional relationship of the calibration jig 4 (chessboard 41 and corner reflector 42).

[0074] The recognition unit 193 recognizes the position of the host vehicle based on the signal acquired by the receiving unit 12. Also, the recognition unit 193 recognizes the posture of the host vehicle based on the information detected by the detecting unit 15, and acquires posture information. The posture information includes information about the travel speed and orientation (posture) during travel of the host vehicle.

[0075] In the case where a specific object having a characteristic point exists within the obtainable range around the main vehicle, the determination unit 194 uses the target vehicle information received from the management device 3 to determine whether the target vehicle exists in the area where calibration can be performed between the imaging unit 13 and the measuring unit 14. Specifically, the determination unit 194 performs coordinate transformation on the installation position of the calibration jig 4 in the target vehicle based on the position information of the target vehicle and the position information of the calibration jig 4 installed on the target vehicle, and specifies the position in the global coordinate system. Next, the information processing device 100 determines whether the target vehicle exists in the area where calibration can be performed based on the information of the area FV3 where the field of view areas of the imaging unit 13 and the measuring unit 14 overlap with each other and the position information of the calibration jig 4 installed on the target vehicle. That is, when the calibration jig 4 installed on the target vehicle is positioned in the area FV3 where the field of view areas of the imaging unit 13 and the measuring unit 14 overlap with each other, the determination unit 194 determines that the target vehicle exists in the area where calibration can be performed.

[0076] Next, when it is determined that the target vehicle exists in the area where calibration can be performed, the determination unit 194 determines whether feature points can be acquired from the calibration jig 4 mounted on the target vehicle. Specifically, the determination unit 194 acquires the position information and posture information of the host vehicle, and the position information and posture information of the target vehicle. Then, the determination unit 194 determines whether feature points can be acquired from the target vehicle based on the position information and posture information of the host vehicle, and the position information and posture information of the target vehicle. That is, the determination unit 194 determines whether feature points 41P can be acquired from the checkerboard 41 as the calibration jig 4 mounted on the target vehicle, and determines whether feature points 42P can be acquired from the corner reflector 42.

[0077] For example, the determination section 194 may determine whether a feature point can be acquired from the calibration jig 4 based on the positional relationship between the host vehicle and the target vehicle, the relative speed between the host vehicle and the target vehicle, the driving postures of the host vehicle and the target vehicle, etc. For example, as an example of a condition under which a feature point can be acquired, the relative speed between the host vehicle and the target vehicle is a speed at which an image of the calibration jig 4 mounted on the target vehicle can be captured by the imaging section 13 of the host vehicle. In addition, as an example of a condition under which a feature point can be acquired, the driving postures of the host vehicle and the target vehicle are straight postures from which it is relatively easy to extract a feature point from the calibration jig 4 mounted on the target vehicle.

[0078] Furthermore, when the calibration jig 4 mounted on the target vehicle is positioned in the region FV3 where the field of view areas of the imaging section 13 and the measuring section 14 overlap each other, the determination section 194 may determine that the feature point can be acquired from the target vehicle.

[0079] Note that the determination unit 194 may use the position information and posture information of the imaging unit 13 and the measuring unit 14 installed in the host vehicle instead of the position information and posture information of the host vehicle. The determination unit 194 may also use the position information and posture information of the calibration jig 4 installed on the target vehicle as the position information and posture information of the target vehicle.

[0080] When the determination unit 194 determines that the feature point can be acquired from the calibration jig 4 mounted on the target vehicle, the execution unit 195 performs calibration between sensors (calibration between the imaging unit 13 and the measuring unit 14 mounted on the host vehicle). That is, the execution unit 195 performs calibration between the imaging unit 13 and the measuring unit 14 so as to maintain the relative positional relationship between the imaging unit 13 and the measuring unit 14 when mounted on the vehicle.

[0081] Specifically, the execution unit 195 recognizes the image captured by the imaging unit 13, and acquires feature points that enable calibration of each of the imaging unit 13 (camera) and the measuring unit 14 (millimeter wave radar) from the captured image. That is, the execution unit 195 acquires feature points 41P from the checkerboard 41 of the calibration jig 4 mounted on the target vehicle, and acquires feature points 42P from the corner reflector 42.

[0082] Subsequently, the execution unit 195 recognizes the current relative positional relationship between the imaging unit 13 and the measurement unit 14 based on the positional relationship between the feature point 41P and the feature point 42P acquired from the captured image. Subsequently, the execution unit 195 acquires information about the relative positional relationship between the imaging unit 13 and the measurement unit 14 when mounted on the vehicle from the main vehicle information storage unit 171. Subsequently, the execution unit 195 calculates the deviation between the current relative positional relationship between the imaging unit 13 and the measurement unit 14 and the relative positional relationship between the imaging unit 13 and the measurement unit 14 when mounted on the vehicle. Then, the execution unit 195 sends a control signal to the actuator for adjusting the attachment position and attachment angle of the imaging unit 13 to absorb the deviation from the relative positional relationship between the imaging unit 13 and the measurement unit 14 when mounted on the vehicle. In this way, the execution unit 195 performs calibration between the imaging unit 13 and the measurement unit 14 so that the relative positional relationship between the imaging unit 13 and the measurement unit 14 approaches the optimal state when mounted on the vehicle.

[0083] Even when a single sensor is displaced from the position at the time of shipment (when installed in the vehicle) as the vehicle travels, the imaging unit 13 and the measuring unit 14 can be used as sensors as long as the relative positional relationship between the sensors is maintained by the calibration between the sensors performed by the execution unit 195. That is, when the driving direction (orientation) of the host vehicle can be obtained from the GPS signal received by the receiving unit 12 or the detection result of the detection unit 15, and the relative positional relationship can be transformed into a coordinate system around the host vehicle, the imaging unit 13 and the measuring unit 14 can be used as sensors. Note that in order to know the driving direction of other vehicles, each vehicle needs to obtain the driving direction from the same reference system, such as a geographic coordinate system from a dual-antenna GPS or the like.

[0084] [1-4. Information processing process according to the embodiment]

[0085] Will refer to Figure 4 A procedure of information processing according to an embodiment of the present disclosure is described. Figure 4 is a flowchart showing the flow of information processing according to an embodiment of the present disclosure. Figure 4 The processing shown is executed by, for example, each unit included in the control unit 109. Figure 4 Processing shown.

[0086] like Figure 4 As shown, the determination unit 194 determines whether the target vehicle is present in an area where calibration can be performed (step S101 ).

[0087] Specifically, when the mounting position of the calibration jig 4 mounted on the target vehicle is included in the region FV3 where the field of view areas of the imaging unit 13 and the measuring unit 14 overlap each other, the determination unit 194 determines that the target vehicle exists in the region where calibration can be performed.

[0088] When it is determined that the target vehicle exists in the area where calibration is possible (step S101 ; Yes), the determination unit 194 determines whether feature points can be acquired from the calibration jig 4 mounted on the target vehicle (step S102 ).

[0089] Specifically, the determination section 194 determines whether feature points can be acquired from the calibration jig 4 mounted on the target vehicle based on the position information and posture information of the host vehicle and the position information and posture information of the target vehicle. As an example of a condition under which feature points can be acquired, the relative speed is a speed at which an image of the calibration jig 4 mounted on the target vehicle can be captured by the host vehicle (imaging section 13), and the driving postures of the host vehicle and the target vehicle are straight postures.

[0090] When the determination unit determines that the feature points can be acquired (step S103; Yes), the execution unit 195 acquires the feature points from the calibration jig 4 mounted on the target vehicle, performs calibration (step S107), and ends the process. Figure 4 Processing shown.

[0091] When the determination unit 194 determines in step S103 that the feature point cannot be acquired (step S103; No), the execution unit 195 ends the process. Figure 4 Processing shown without calibration.

[0092] When the determination unit 194 determines in step S101 that the target vehicle is not present in the calibration-capable area (step S101; No), Figure 4 The shown processing ends.

[0093] [1-5. Modification example according to the embodiment]

[0094] In the above embodiment, an example has been described in which the information processing device 100 acquires various types of information about the target vehicle through road-to-vehicle communication (V2X communication) via the management device 3, but the present disclosure is not particularly limited to this example. The information processing device 100 can acquire various types of information about the target vehicle by performing vehicle-to-vehicle communication with the target vehicle.

[0095] Furthermore, in the above-described embodiment, the calibration between different types of sensors, the imaging section 13 (camera), and the measuring section 14 (millimeter wave radar) has been described as an example of information processing performed by the information processing device 100. However, the present disclosure is not particularly limited to this example, and the information processing performed by the information processing device 100 can be similarly applied to the case of performing calibration between sensors of the same type.

[0096] Furthermore, in the above-described embodiment, it is desirable that the vehicle 1 travels near the center of the travel lane when calibration is performed.

[0097] In addition, in the above embodiment, the feature point used for calibration between sensors is not necessarily the calibration jig 4 installed in the target vehicle for calibration, and any feature point that can be extracted to enable calibration is sufficient. For example, the feature point can be an existing structure in the vehicle, such as a taillight.

[0098] Furthermore, in the above-described embodiment, an example has been described in which the information processing device 100 performs calibration (calibration between sensors) using captured images of other moving bodies (e.g., target vehicles). The processing performed by the information processing device 100 can be similarly applied to a case in which one sensor is calibrated using captured images of other moving bodies (e.g., target vehicles).

[0099] (2. Other embodiments)

[0100] [2-1. Other sensors]

[0101] An example in which the information processing device 100 according to the above-mentioned embodiment includes, for example, a millimeter wave radar as the measuring unit 14 has been described, but the present disclosure is not particularly limited to this example. The information processing device 100 may include light detection and ranging (laser imaging detection and ranging (LiDAR)) that reads the three-dimensional structure of the surrounding environment of the host vehicle. LiDAR detects the distance and relative speed to the surrounding objects by irradiating the surrounding objects with a laser beam such as an infrared laser beam and measuring the time until the laser beam is reflected and returns. The information processing device 100 can perform calibration between LiDARs by using the information about the target vehicle acquired by the LiDAR as a feature point.

[0102] Figure 5 is a diagram showing an example of detection by LiDAR. Figure 5 As shown, for example, it is assumed that two LiDARs, namely LiDAR_EX1 and LiDAR_EX2, are installed on vehicle 1. At this time, LiDAR_EX1 installed on vehicle 1 detects detection point groups PG1 and PG2 representing the shape of vehicle 2 as a target vehicle. Similarly, LiDAR_EX2 installed on vehicle 1 detects detection point groups PG11 and PG21 representing the shape of vehicle 2 as a target vehicle.

[0103] Then, the information processing device 100 performs calibration between the LiDARs based on the information obtained from the two LiDARs, namely LiDAR_EX1 and LiDAR_EX2. For example, the information processing device 100 adjusts the attachment position and attachment angle of the LiDAR so that the two straight lines L1 and L2 formed by the detection point groups PG1 and PG2 overlap with the two straight lines L11 and L21 formed by the detection point groups PG11 and PG21. In this way, the information processing device 100 can perform calibration between the LiDARs based on the target vehicle information obtained by the LiDAR without the need for additional feature points, such as a calibration jig.

[0104] [2-2. Configuration of moving objects]

[0105] Furthermore, in the above-described embodiment, an example has been described in which the information processing device 100 is installed on a vehicle (car). However, the information processing implemented by the information processing device 100 may be performed by a moving body itself such as a vehicle (car) that performs autonomous driving. In this case, in addition to Figure 2 In addition to the configuration shown in , the information processing apparatus 100 may also have the following configuration. Note that each section described below may be included in, for example, Figure 2 In the configuration shown in .

[0106] That is, the information processing device 100 according to the embodiment of the present disclosure may be configured as a moving body control system as described below. Figure 6 : is a block diagram showing a configuration example of schematic functions of a mobile body control system to which information processing according to an embodiment of the present disclosure can be applied.

[0107] The automatic driving control unit 212 of the vehicle control system 200, as an example of a mobile body control system, corresponds to the control unit 109 of the information processing device 100 of the present embodiment. In addition, the detection unit 231 and the self-position estimation unit 232 of the automatic driving control unit 212 correspond to the recognition unit 193 of the control unit 109 of the information processing device 100 according to the present embodiment. In addition, the status analysis unit 233 of the automatic driving control unit 212 corresponds to the recognition unit 193 of the control unit 109. In addition, the planning unit 234 of the automatic driving control unit 212 corresponds to the determination unit 194 and the execution unit 195 of the control unit 109. The operation control unit 235 of the automatic driving control unit 212 corresponds to the determination unit 194 and the execution unit 195 of the control unit 109. In addition, except Figure 6 In addition to the blocks shown in , the automatic driving control unit 212 may also include blocks corresponding to the various processing units of the control unit 130.

[0108] Note that, hereinafter, the vehicle in which the vehicle control system 200 is provided is referred to as a host vehicle to distinguish it from other vehicles.

[0109] The vehicle control system 200 includes an input unit 201, a data acquisition unit 202, a communication unit 203, an on-board equipment 204, an output control unit 205, an output unit 206, a drive system control unit 207, a drive system 208, a main system control unit 209, a main system 210, a storage unit 211, and an automatic driving control unit 212. The input unit 201, the data acquisition unit 202, the communication unit 203, the output control unit 205, the drive system control unit 207, the main system control unit 209, the storage unit 211, and the automatic driving control unit 212 are connected to each other via a communication network 221. Examples of the communication network 221 include an on-board communication network, a bus, etc. according to any standard, such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), or FlexRay (registered trademark). Note that the various components of the vehicle control system 200 may be directly connected to each other without passing through the communication network 221.

[0110] Note that, hereinafter, when the components of the vehicle control system 200 communicate with each other via the communication network 221, the description of the communication network 221 will be omitted. For example, in the case where the input unit 201 and the automatic driving control unit 212 communicate with each other via the communication network 221, it is simply described that the input unit 201 and the automatic driving control unit 212 communicate with each other.

[0111] The input unit 201 includes a device used by a passenger to input various data, instructions, etc. For example, the input unit 201 includes an operation device such as a touch panel, a button, a microphone, a switch, or a joystick, and an operation device capable of receiving input by a method other than manual operation, such as by voice or gesture. In addition, for example, the input unit 201 may be a remote control device using infrared or other radio waves, or an external connection device such as a mobile device or a wearable device corresponding to the operation of the vehicle control system 200. The input unit 201 generates an input signal based on the data, instructions, etc. input by the passenger, and provides the input signal to each component of the vehicle control system 200.

[0112] The data acquisition section 202 includes various sensors and the like that acquire data used for processing performed by the vehicle control system 200 , and supplies the acquired data to each component of the vehicle control system 200 .

[0113] For example, the data acquisition unit 202 includes various sensors for detecting the state of the host vehicle, etc. Specifically, for example, the data acquisition unit 202 includes a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU), and sensors for detecting an accelerator pedal operation amount, a brake pedal operation amount, a steering wheel steering angle, an engine speed, a motor rotation speed, a wheel rotation speed, etc.

[0114] In addition, for example, the data acquisition unit 202 includes various sensors for detecting information about the exterior of the main vehicle. Specifically, for example, the data acquisition unit 202 includes an imaging device, such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, or other cameras. In addition, for example, the data acquisition unit 202 includes an environmental sensor for detecting weather or climate, and a surrounding information detection sensor for detecting objects around the main vehicle. Environmental sensors include, for example, raindrop sensors, fog sensors, sunlight sensors, snow sensors, etc. Surrounding information detection sensors include, for example, ultrasonic sensors, radars, light detection and ranging (laser imaging detection and ranging (LiDAR)), sonars, etc.

[0115] In addition, for example, the data acquisition unit 202 includes various sensors for detecting the current position of the host vehicle. Specifically, for example, the data acquisition unit 202 includes a GNSS receiver that receives a GNSS signal from a GNSS satellite.

[0116] In addition, for example, the data acquisition unit 202 includes various sensors for detecting information inside the vehicle. Specifically, for example, the data acquisition unit 202 includes an imaging device that captures an image of the driver, a biosensor that detects biometric information of the driver, a microphone that collects sounds inside the vehicle, etc. The biosensor is, for example, provided on the surface of a seat or a steering wheel, and detects biometric information of a passenger sitting on the seat or a driver holding the steering wheel.

[0117] The communication unit 203 communicates with the vehicle-mounted equipment 204 and various equipment outside the vehicle, a server, a base station, etc., transmits data provided from each component of the vehicle control system 200, and transmits received data to each component of the vehicle control system 200. Note that the communication protocol supported by the communication unit 203 is not particularly limited, and the communication unit 203 can support multiple types of communication protocols.

[0118] For example, the communication unit 203 performs wireless communication with the in-vehicle equipment 204 by using wireless LAN, Bluetooth (registered trademark), near field communication (NFC), wireless USB (WUSB), etc. In addition, for example, by using a universal serial bus (USB), a high-definition multimedia interface (HDMI) (registered trademark), a mobile high-definition link (MHL), etc., the communication unit 203 performs wired communication with the in-vehicle equipment 204 via a connection terminal (not shown) (and a cable as needed).

[0119] In addition, for example, the communication unit 203 communicates with equipment (e.g., an application server or a control server) existing on an external network (e.g., the Internet, a cloud network, or a network unique to an enterprise operator) via a base station or an access point. In addition, for example, the communication unit 203 communicates with a terminal (e.g., a pedestrian, a store terminal, or a machine type communication (MTC) terminal) existing near the main vehicle using a peer-to-peer (P2P) technology. In addition, for example, the communication unit 203 performs V2X communication, such as vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication. In addition, for example, the communication unit 203 includes a beacon receiving unit that receives radio waves or electromagnetic waves transmitted from a wireless station installed on a road, etc., and obtains information such as the current location, traffic congestion, traffic control, or required time.

[0120] The in-vehicle equipment 204 includes, for example, mobile equipment or wearable equipment of passengers, information equipment carried on or attached to the host vehicle, and a navigation device for searching for a route to an arbitrary destination.

[0121] The output control section 205 controls the output of various information to the passengers of the host vehicle or the outside of the vehicle. For example, the output control section 205 generates an output signal including at least one of visual information (e.g., image data) and auditory information (e.g., audio data), and provides the generated output signal to the output section 206 to control the output of the visual information and the auditory information from the output section 206. Specifically, for example, the output control section 205 combines the image data captured by the different imaging devices of the data acquisition section 202 to generate a bird's-eye view, a panoramic image, etc., and provides the output signal including the generated image to the output section 206. In addition, for example, the output control section 205 generates audio data including a warning sound, a warning message, etc. for dangers such as collision, contact, and entry into a danger zone, and provides the output signal including the generated audio data to the output section 206.

[0122] The output unit 206 includes a device capable of outputting visual information or auditory information to a passenger of the host vehicle or outside the vehicle. For example, the output unit 206 includes a display device, an instrument panel, a speaker, a headset, a wearable device such as a glasses-type display worn by a passenger, a projector, a lamp, etc. In addition to a device with a normal display, the display device included in the output unit 206 may also be a device that displays visual information in the driver's field of view, such as a head-up display, a transmissive display, and a device with an augmented reality (AR) display function.

[0123] The drive system control unit 207 generates various control signals and provides the generated control signals to the drive system 208 to control the drive system 208. In addition, the drive system control unit 207 provides control signals to each component other than the drive system 208 as needed, and performs notification of the control state of the drive system 208, etc.

[0124] The drive system 208 includes various devices related to the drive system of the host vehicle. For example, the drive system 208 includes a drive force generating device for generating a drive force of an internal combustion engine, a drive motor, etc., a drive force transmitting mechanism for transmitting the drive force to wheels, a steering mechanism for adjusting a steering angle, a braking device for generating a braking force, an anti-lock braking system (ABS), an electronic stability control (ESC), an electric steering device, etc.

[0125] The main body system control unit 209 generates various control signals and supplies the control signals to the main body system 210 to control the main body system 210. In addition, the main body system control unit 209 supplies control signals to each component other than the main body system 210 as needed, and performs notification of the control state of the main body system 210, etc.

[0126] The main system 210 includes various devices of the main system installed on the vehicle body. For example, the main system 210 includes a keyless entry system, a smart key system, a power window device, a power seat, a steering wheel, an air conditioning device, various lights (for example, headlights, taillights, brake lights, turn indicators, and fog lights), etc.

[0127] The storage unit 211 includes, for example, a magnetic storage device such as a read-only memory (ROM), a random access memory (RAM) or a hard disk drive (HDD), a semiconductor storage device, an optical storage device, an optical magnetic storage device, etc. The storage unit 211 stores various programs, data, etc. used by each component of the vehicle control system 200. For example, the storage unit 211 stores map data, for example, a three-dimensional high-precision map such as a dynamic map, a global map with lower precision than the high-precision map and a wide coverage, and a local map including information about the surroundings of the host vehicle.

[0128] The automatic driving control unit 212 performs control related to automatic driving, such as autonomous driving or driving assistance. Specifically, for example, the automatic driving control unit 212 performs cooperative control to realize the functions of the advanced driver assistance system (ADAS), which include collision avoidance or buffering of the main vehicle, follow-up driving based on the distance between vehicles, driving to maintain vehicle speed, warning of collision of the main vehicle, warning of lane deviation of the main vehicle, etc. In addition, for example, the automatic driving control unit 212 performs cooperative control for the purpose of automatic driving, in which the main vehicle drives autonomously without relying on the operation of the driver. The automatic driving control unit 212 includes a detection unit 231, a self-position estimation unit 232, a status analysis unit 233, a planning unit 234, and an operation control unit 235.

[0129] The detection unit 231 detects various information required for controlling the automatic driving. The detection unit 231 includes an external information detection unit 241 , an internal information detection unit 242 , and a vehicle state detection unit 243 .

[0130] The vehicle exterior information detection unit 241 performs a process of detecting information about the exterior of the host vehicle based on data or signals from each component of the vehicle control system 200. For example, the vehicle exterior information detection unit 241 performs a process of detecting objects around the host vehicle, a process of identifying objects, a process of tracking objects, and a process of detecting the distance to the objects. Examples of objects to be detected include vehicles, people, obstacles, buildings, roads, traffic lights, traffic signs, road signs, etc. In addition, for example, the vehicle exterior information detection unit 241 performs a process of detecting the surrounding environment of the host vehicle. Examples of the surrounding environment to be detected include weather, temperature, humidity, brightness, road surface conditions, etc. The vehicle exterior information detection unit 241 provides data representing the detection processing result to the self-position estimation unit 232, the map analysis unit 251 of the situation analysis unit 233, the traffic rule recognition unit 252 and the situation recognition unit 253, and the emergency situation avoidance unit 271 of the operation control unit 235.

[0131] The in-vehicle information detection unit 242 performs processing to detect information about the interior of the vehicle based on data or signals from each component of the vehicle control system 200. For example, the in-vehicle information detection unit 242 performs processing to authenticate and identify the driver, processing to detect the driver's state, processing to detect passengers, processing to detect the in-vehicle environment, etc. Examples of driver states to be detected include physical state, alertness, concentration, fatigue, line of sight direction, etc. Examples of in-vehicle environments to be detected include temperature, humidity, brightness, odor, etc. The in-vehicle information detection unit 242 provides data representing the result of the detection processing to the situation recognition unit 253 of the situation analysis unit 233, the emergency situation avoidance unit 271 of the operation control unit 235, etc.

[0132] The vehicle state detection unit 243 performs processing to detect the state of the host vehicle based on the data or signal from each component of the vehicle control system 200. Examples of the state of the host vehicle to be detected include speed, acceleration, steering angle, presence and content of abnormality, driving operation state, position / tilt of the power seat, door lock state, state of other vehicle-mounted equipment, etc. The vehicle state detection unit 243 provides data indicating the result of the detection processing to the state recognition unit 253 of the state analysis unit 233, the emergency avoidance unit 271 of the operation control unit 235, etc.

[0133] The own position estimation unit 232 performs processing to estimate the position, posture, etc. of the main vehicle based on data or signals from each component of the vehicle control system 200 such as the external information detection unit 241 or the situation recognition unit 253 of the situation analysis unit 233. In addition, the own position estimation unit 232 generates a local map for self-position estimation (hereinafter referred to as the own position estimation map) as needed. For example, the own position estimation map is a high-precision map using technologies such as simultaneous localization and mapping (SLAM). The own position estimation unit 232 provides data representing the estimation processing result to the map analysis unit 251, the traffic rule recognition unit 252, and the situation recognition unit 253 of the situation analysis unit 233. In addition, the own position estimation unit 232 stores the self-position estimation map in the storage unit 211.

[0134] The situation analysis unit 233 performs a process of analyzing the situation of the host vehicle and the surroundings. The situation analysis unit 233 includes a map analysis unit 251 , a traffic regulation recognition unit 252 , a situation recognition unit 253 , and a situation prediction unit 254 .

[0135] The map parsing unit 251 uses data or signals from each component of the vehicle control system 200 such as the own position estimation unit 232 and the vehicle external information detection unit 241 as needed, and performs processing to parse various maps stored in the storage unit 211 to construct a map including information required for the automatic driving process. The map parsing unit 251 provides the constructed map to, for example, the traffic regulation recognition unit 252, the situation recognition unit 253, the situation prediction unit 254, and the route planning unit 261, the action planning unit 262, and the operation planning unit 263 of the planning unit 234.

[0136] The traffic regulation recognition unit 252 performs a process of recognizing traffic regulations in the area around the host vehicle based on data or signals from each component of the vehicle control system 200, such as the own position estimation unit 232, the vehicle external information detection unit 241, and the map analysis unit 251. Through this recognition process, for example, the position and state of the signal around the host vehicle, the content of the traffic regulations in the area around the host vehicle, the lanes in which the vehicle can travel, etc. are recognized. The traffic regulation recognition unit 252 provides data indicating the result of the recognition process to the situation prediction unit 254 and the like.

[0137] The situation recognition unit 253 performs a process of recognizing the situation of the host vehicle based on data or signals from each component of the vehicle control system 200, such as the own position estimation unit 232, the vehicle exterior information detection unit 241, the vehicle interior information detection unit 242, the vehicle state detection unit 243, and the map analysis unit 251. For example, the situation recognition unit 253 performs a process of recognizing the situation of the host vehicle, the situation in the area surrounding the host vehicle, the situation of the driver of the host vehicle, and the like. In addition, the situation recognition unit 253 generates a local map (hereinafter referred to as a situation recognition map) for recognizing the situation in the area surrounding the host vehicle as needed. The situation recognition map is, for example, an occupancy grid map.

[0138] Examples of host vehicle conditions to be identified include the host vehicle's position, posture, and movement (e.g., speed, acceleration, direction of movement, etc.), as well as the presence and content of abnormalities. Examples of conditions in the area surrounding the host vehicle to be identified include the types and positions of surrounding stationary objects in the area surrounding the host vehicle, the types, positions, and movements (e.g., speed, acceleration, direction of movement) of surrounding moving objects, the composition of surrounding roads, road surface conditions, weather, temperature, humidity, brightness, etc. Examples of driver conditions to be identified include physical condition, alertness, concentration, fatigue, line of sight movement, driving operation, etc.

[0139] The situation recognition unit 253 supplies data indicating the result of the recognition process (including a situation recognition map if necessary) to the own position estimation unit 232 , the situation prediction unit 254 , etc. Furthermore, the situation recognition unit 253 stores the situation recognition map in the storage unit 211 .

[0140] The situation prediction unit 254 performs a situation prediction process on the host vehicle based on data or signals from each component of the vehicle control system 200, such as the map analysis unit 251, the traffic rule recognition unit 252, and the situation recognition unit 253. For example, the situation prediction unit 254 performs a process of predicting the situation of the host vehicle, the situation in the area around the host vehicle, the situation of the driver, and the like.

[0141] Examples of the host vehicle's conditions to be predicted include the host vehicle's behavior, the occurrence of anomalies, the distance the host vehicle can travel, etc. Examples of conditions in the area surrounding the host vehicle to be predicted include the behavior of moving objects around the host vehicle, changes in signal status, environmental changes such as weather, etc. Examples of the driver's conditions to be predicted include the driver's behavior, physical condition, etc.

[0142] The situation prediction unit 254 provides data indicating the prediction processing result together with the data from the traffic regulation recognition unit 252 and the situation recognition unit 253 to the route planning unit 261 , the action planning unit 262 , the operation planning unit 263 , etc. of the planning unit 234 .

[0143] The route planning unit 261 plans a route to a destination based on data or signals from each component of the vehicle control system 200, such as the map parsing unit 251 and the condition prediction unit 254. For example, the route planning unit 261 sets a route from the current location to the designated destination based on a global map. In addition, for example, the route planning unit 261 appropriately changes the route based on conditions such as traffic jams, accidents, traffic control, construction, and the physical condition of the driver. The route planning unit 261 provides data representing the planned route to the action planning unit 262, etc.

[0144] The action planning unit 262 plans the action of the host vehicle to travel safely along the route planned by the route planning unit 261 within the planning time based on the data or signal from each component of the vehicle control system 200 such as the map analysis unit 251 and the situation prediction unit 254. For example, the action planning unit 262 plans departure, stop, travel direction (e.g., forward, backward, left turn, right turn, turning, etc.), travel lane, travel speed, overtaking, etc. The action planning unit 262 provides data representing the planned action of the host vehicle to the operation planning unit 263 and the like.

[0145] The operation planning unit 263 plans the operation of the host vehicle for realizing the action planned by the action planning unit 262 based on the data or signal from each component of the vehicle control system 200 such as the map parsing unit 251 and the situation prediction unit 254. For example, the operation planning unit 263 plans acceleration, deceleration, driving trajectory, etc. The operation planning unit 263 provides data representing the planned operation of the host vehicle to the acceleration / deceleration control unit 272 and the direction control unit 273 of the operation control unit 235.

[0146] The operation control unit 235 controls the operation of the host vehicle and includes an emergency situation avoidance unit 271 , an acceleration / deceleration control unit 272 , and a direction control unit 273 .

[0147] The emergency avoidance unit 271 detects emergency situations such as collision, contact, entry into a dangerous area, driver abnormality, and vehicle abnormality based on the detection results of the vehicle exterior information detection unit 241, the vehicle interior information detection unit 242, and the vehicle state detection unit 243. When an emergency situation is detected, the emergency avoidance unit 271 plans the operation of the host vehicle to avoid emergency situations such as sudden stops or sharp turns. The emergency avoidance unit 271 provides data indicating the planned operation of the host vehicle to the acceleration / deceleration control unit 272, the direction control unit 273, and the like.

[0148] The acceleration / deceleration control unit 272 performs acceleration / deceleration control for realizing the host vehicle operation planned by the operation planning unit 263 or the emergency situation avoidance unit 271. For example, the acceleration / deceleration control unit 272 calculates the target control value of the driving force generating device or the braking device for realizing the planned acceleration, deceleration or emergency stop, and provides a control instruction representing the calculated target control value to the drive system control unit 207.

[0149] The direction control unit 273 performs direction control for realizing the operation of the host vehicle planned by the operation planning unit 263 or the emergency avoidance unit 271. For example, the direction control unit 273 calculates a target control value of a steering mechanism for realizing a driving track or a sharp turn planned by the operation planning unit 263 or the emergency avoidance unit 271, and provides a control instruction representing the calculated target control value to the drive system control unit 207.

[0150] [2-3. Others]

[0151] In addition, in each process described in the above embodiments, all or part of the processes described as being automatically performed can be performed manually. Alternatively, all or part of the processes described as being manually performed can be performed automatically by known methods. In addition, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters shown in the specification and the accompanying drawings can be changed arbitrarily. For example, the various information shown in each accompanying drawing is not limited to the information shown.

[0152] In addition, each illustrated component of each device is conceptual in function and does not necessarily have to be physically configured as shown in the drawings. That is, the specific mode of distribution / integration of each device is not limited to the mode shown in the drawings. Depending on various loads or usage conditions, all or part of the devices may be functionally or physically distributed / integrated in any part. For example, Figure 2 The determination unit 194 and the execution unit 195 shown in FIG. 1 may be functionally or physically integrated. Figure 2The various units (providing unit 191, acquiring unit 192, identifying unit 193, determining unit 194, and executing unit 195) included in the control unit 109 shown may be functionally or physically integrated into one unit. In addition, the information processing performed by the information processing device 100 of the present disclosure may be performed by the management device 3.

[0153] In addition, the above-mentioned embodiments and modified examples can be appropriately combined as long as the processing contents do not contradict each other. In addition, in the above-mentioned embodiments, a car is an example of a mobile body, but the information processing performed by the information processing device 100 of the present disclosure can also be applied to mobile bodies other than cars. For example, the mobile body can be a small vehicle such as a motorcycle or a motor tricycle, a large vehicle such as a bus or a truck, or an autonomous mobile body such as a robot or a drone. In addition, the information processing device 100 is not necessarily integrated with the mobile body, but can be a cloud server or the like that obtains information from the mobile body via a network N and performs information processing based on the obtained information.

[0154] In addition, the effects in each embodiment described in this specification are merely examples. The effects of the present disclosure are not limited thereto, and other effects may be obtained.

[0155] (3. Effects of the information processing device according to the present disclosure)

[0156] As described above, the information processing device according to the present disclosure (the information processing device 100 in the present embodiment) includes a calibration execution unit (the execution unit 195 in the present embodiment). The calibration execution unit performs calibration between two or more sensors (the imaging unit 13 and the measuring unit 14 in the present embodiment) which are attached at different positions and whose field of view areas at least partially overlap with each other. In addition, in the case where feature points that enable calibration of each of the two or more sensors can be acquired in the area where the fields of view of the two or more sensors overlap with each other, the calibration execution unit performs calibration between the sensors.

[0157] As described above, in a case where feature points that enable calibration of each of two or more sensors can be acquired in an area where the fields of view of two or more sensors overlap each other, the information processing device according to the present disclosure performs calibration between sensors. Thus, the information processing device according to the present disclosure can perform calibration even for an area without map information, without being affected by environmental changes.

[0158] The two or more sensors are sensors attached to the vehicle. Thus, the information processing device according to the present disclosure can perform calibration between sensors mounted on the vehicle even for an area without map information without being affected by environmental changes.

[0159] At least one of the two or more sensors is a camera. Then, the calibration execution unit recognizes an image captured by the camera, and acquires feature points in the captured image that enable calibration of each of the camera and sensors other than the camera. Thus, the information processing device according to the present disclosure can perform calibration between sensors.

[0160] In addition, when a specific object having feature points that enable calibration of each of two or more sensors exists in an area where the fields of view of the two or more sensors overlap each other, the calibration execution unit performs calibration between the sensors. Thus, the information processing device according to the present disclosure can obtain feature points required for calibration between the sensors from the specific object.

[0161] In addition, the specific object having the characteristic points that enable calibration is another vehicle. Thus, the information processing device according to the present disclosure can acquire the characteristic points required for calibration between sensors from the other vehicle.

[0162] In addition, the information processing device according to the present disclosure receives information about whether there are other vehicles with feature points that enable calibration between two or more sensors within the obtainable range around the host vehicle through vehicle-to-vehicle communication. Thus, the information processing device according to the present disclosure can quickly grasp the position of other vehicles through vehicle-to-vehicle communication.

[0163] In addition, the information processing device according to the present disclosure receives information about whether there are other vehicles with feature points that enable calibration between two or more sensors within the available range around the host vehicle through V2X communication. Thus, the position of other vehicles can be quickly grasped through V2X communication.

[0164] In addition, the information processing device according to the present disclosure receives position information of other vehicles having feature points that enable calibration. Thus, the information processing device according to the present disclosure can grasp the positional relationship between the host vehicle and the other vehicles.

[0165] In addition, the information processing device according to the present disclosure receives posture information of other vehicles having feature points that enable calibration. Thus, the information processing device according to the present disclosure can grasp whether the feature points can be acquired from other vehicles.

[0166] In addition, the information processing device according to the present disclosure further includes a determination unit (determination unit 194 in the present embodiment). The determination unit acquires the position information and posture information of two or more sensors or vehicles to which the two or more sensors are attached, and the position information and posture information of a specific object having a feature point that enables calibration of each of the two or more sensors. Then, the determination unit determines whether the feature point that enables calibration of each of the two or more sensors is in an acquisitive state. Thus, the information processing device according to the present disclosure can accurately determine whether the feature point can be acquired from a specific object.

[0167] In addition, the sensor is a camera, and the specific object having feature points that enable calibration is a checkerboard installed in another vehicle. Thus, the information processing device according to the present disclosure can easily acquire feature points for calibration of the camera.

[0168] In addition, the sensor is a millimeter wave radar, and the specific object having characteristic points that enable calibration is a corner reflector installed in another vehicle. Thus, the information processing device according to the present disclosure can easily acquire characteristic points for calibration of the millimeter wave radar.

[0169] (4. Hardware Configuration)

[0170] By having, for example, Figure 7 The computer 1000 of the illustrated configuration realizes information processing performed by the information processing apparatus 100 according to each of the above-described embodiments. Hereinafter, the information processing apparatus 100 according to an embodiment of the present disclosure will be described as an example. Figure 7 1 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the information processing device 100 of the present disclosure. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each component of the computer 1000 is connected by a bus 1050.

[0171] The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each component. For example, the CPU 1100 loads a program stored in the ROM 1300 or the HDD 1400 onto the RAM 1200 and performs processing corresponding to various programs.

[0172] The ROM 1300 stores programs such as a boot program of a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, programs depending on the hardware of the computer 1000 , and the like.

[0173] The HDD 1400 is a recording medium readable by the computer 1000, in which a program executed by the CPU 1100, data used by the program, and the like are recorded non-temporarily. Specifically, the HDD 1400 is a recording medium in which data for implementing, for example, Figure 3 The procedure of the antenna switching process is shown.

[0174] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from other equipment via the communication interface 1500 or transmits data generated by the CPU 1100 to other equipment.

[0175] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000 to each other. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. In addition, the CPU 1100 sends data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium, etc. Examples of the medium include optical recording media such as a digital versatile disk (DVD) or a phase change rewritable disk (PD), magneto-optical recording media such as a magneto-optical disk (MO), a magnetic tape medium, a magnetic recording medium, and a semiconductor memory.

[0176] For example, in the case where the computer 1000 is used as the receiving device 10 according to the first embodiment, the CPU 1100 of the computer 1000 executes a program loaded on the RAM 1200 (such as a program for realizing the information processing of the present disclosure). Thereby, functions such as various types of processing performed by the control section 109 of the information processing device 100 are realized. In addition, the HDD 1400 stores the program for realizing the information processing of the present disclosure, the data stored in the storage section 111, and the like. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data 1450, but as another example, these programs may be acquired from other devices via the external network 1550.

[0177] Note that the present technology may also have the following configurations. (1)

[0179] An information processing device, comprising:

[0180] a calibration execution section that performs calibration between two or more sensors that are attached at different positions and whose fields of view at least partially overlap each other,

[0181] The calibration execution unit performs calibration between the sensors when a feature point enabling calibration of each of the two or more sensors is obtainable in a region where the fields of view of the two or more sensors overlap each other. (2)

[0183] The information processing device according to (1), wherein the two or more sensors are sensors attached to a vehicle. (3)

[0185] The information processing device according to (2), further comprising:

[0186] an acquisition section that acquires information as to whether a specific object having a feature point that enables calibration between the two or more sensors exists within an acquisible range around the vehicle; and

[0187] a determination unit that determines, when the specific object exists within a retrievable range around the vehicle, whether a feature point is retrievable from the specific object in an area where the fields of view of the two or more sensors overlap each other,

[0188] wherein, in a case where the specific object having a feature point that enables calibration of each of the two or more sensors exists in a region where fields of view of the two or more sensors overlap each other, the determination section determines whether the feature point is acquirable from the specific object, and

[0189] The calibration execution section acquires the feature point when the determination section determines that the feature point is acquisible. (4)

[0191] The information processing device according to (3),

[0192] wherein at least one of the two or more sensors is a camera, and

[0193] The calibration execution section recognizes an image captured by the camera, and acquires feature points in the captured image that enable calibration of each of the camera and sensors other than the camera. (5)

[0195] The information processing device according to (3),

[0196] The acquisition unit receives information about whether the specific object exists within an acquisitive range around the vehicle through V2X communication. (6)

[0198] The information processing device according to (3),

[0199] The acquisition unit receives information about whether the specific object exists within an acquisitive range around the vehicle through vehicle-to-vehicle communication. (7)

[0201] The information processing device according to (3),

[0202] The acquisition unit receives the position information and posture information of the specific object, and

[0203] The determination unit determines whether feature points are obtainable from the specific object based on the position information and posture information of the two or more sensors or the vehicle to which the two or more sensors are attached, and the position information and posture information of the specific object having feature points that enable calibration of each of the two or more sensors. (8)

[0205] The information processing device according to (3),

[0206] wherein at least one of the two or more sensors is a camera, and

[0207] The specific object is a checkerboard installed in other vehicles. (9)

[0209] The information processing device according to (3),

[0210] wherein at least one of the two or more sensors is a millimeter wave radar, and

[0211] The specific object is a corner reflector installed in another vehicle. (10)

[0213] The information processing device according to (3),

[0214] The specific object is another vehicle. (11)

[0216] An information processing method executed by a computer, the information processing method comprising:

[0217] In an area where the fields of view of two or more sensors attached at different positions and whose fields of view at least partially overlap each other overlap with each other, calibration is performed between the two or more sensors when feature points enabling calibration of each of the two or more sensors are obtainable. (12)

[0219] An information processing program that enables a computer to:

[0220] In an area where the fields of view of two or more sensors attached at different positions and whose fields of view at least partially overlap each other overlap with each other, calibration is performed between the two or more sensors when feature points enabling calibration of each of the two or more sensors are obtainable.

[0221] Reference Mark List

[0222] 1, 2 Vehicles

[0223] 3 Management Device

[0224] 12 Receiving Department

[0225] 13 Imaging Department

[0226] 14 Measurement Department

[0227] 15 Detection Department

[0228] 100 Information processing device

[0229] 101 Ministry of Communications

[0230] 103 Input section

[0231] 105 Output

[0232] 107 Storage

[0233] 109 Control Department

[0234] 191 Provide Department

[0235] 192 Acquisition Department

[0236] 193 Identification Department

[0237] 194 Judgment Department

[0238] 195 Executive Department

Claims

1. An information processing device, comprising: a calibration execution section that performs calibration between two or more sensors that are attached at different positions and whose fields of view at least partially overlap each other, wherein, in a region where the fields of view of the two or more sensors overlap each other, the calibration execution unit performs calibration between the sensors in the case where a feature point enabling calibration of each of the two or more sensors is obtainable, wherein the two or more sensors are sensors attached to a vehicle, The information processing device further includes: an acquisition section that acquires information as to whether a specific object having a feature point that enables calibration between the two or more sensors exists within an acquisible range around the vehicle; and a determination unit that determines, when the specific object exists within a retrievable range around the vehicle, whether a feature point is retrievable from the specific object in an area where the fields of view of the two or more sensors overlap each other, wherein, in a case where the specific object having a feature point that enables calibration of each of the two or more sensors exists in a region where fields of view of the two or more sensors overlap each other, the determination section determines whether the feature point is acquirable from the specific object, and When the determination unit determines that the feature point is obtainable, the calibration execution unit acquires the feature point, The acquisition unit receives the position information and posture information of the specific object, and The determination unit determines whether feature points are obtainable from the specific object based on the position information and posture information of the two or more sensors or the vehicle to which the two or more sensors are attached, and the position information and posture information of the specific object having feature points that enable calibration of each of the two or more sensors.

2. The information processing device according to claim 1, in, At least one of the two or more sensors is a camera, and The calibration execution section recognizes an image captured by the camera, and acquires feature points in the captured image that enable calibration of each of the camera and sensors other than the camera.

3. The information processing device according to claim 1, wherein: The acquisition section receives information on whether the specific object exists within an acquisible range around the vehicle through V2X communication.

4. The information processing device according to claim 1, wherein: The acquisition unit receives information on whether the specific object exists within an acquisible range around the vehicle through vehicle-to-vehicle communication.

5. The information processing device according to claim 1, in, At least one of the two or more sensors is a camera, and The specific object is a checkerboard installed in other vehicles.

6. The information processing device according to claim 1, in, At least one of the two or more sensors is a millimeter wave radar, and The specific object is a corner reflector installed in another vehicle.

7. The information processing device according to claim 1, in, The specific object is another vehicle.

8. An information processing method executed by a computer, the information processing method comprising: performing calibration between two or more sensors attached at different positions and having their fields of view at least partially overlapping each other in an area where fields of view overlap with each other, in a case where a feature point enabling calibration of each of the two or more sensors is obtainable, wherein the two or more sensors are sensors attached to a vehicle, The information processing method further includes: acquiring information as to whether a specific object having a feature point that enables calibration between the two or more sensors exists within an acquisitive range around the vehicle; and In a case where the specific object exists within the retrievable range around the vehicle, determining whether a feature point is retrievable from the specific object in an area where the fields of view of the two or more sensors overlap each other, wherein, in a case where the specific object having a feature point that enables calibration of each of the two or more sensors exists in a region where fields of view of the two or more sensors overlap with each other, determining whether the feature point is acquirable from the specific object, and When it is determined that the feature point is obtainable, the feature point is obtained. wherein receiving the position information and posture information of the specific object, and Based on the position information and posture information of the two or more sensors or the vehicle to which the two or more sensors are attached, and the position information and posture information of the specific object having feature points that enable calibration of each of the two or more sensors, it is determined whether the feature points are obtainable from the specific object.

9. A computer program product having a program stored thereon, the program causing a computer to: performing calibration between two or more sensors attached at different positions and having their fields of view at least partially overlapping each other in an area where fields of view overlap with each other, in a case where a feature point enabling calibration of each of the two or more sensors is obtainable, wherein the two or more sensors are sensors attached to a vehicle, The program also causes the computer to: acquiring information as to whether a specific object having a feature point that enables calibration between the two or more sensors exists within an acquisitive range around the vehicle; and In a case where the specific object exists within the retrievable range around the vehicle, determining whether a feature point is retrievable from the specific object in an area where the fields of view of the two or more sensors overlap each other, in, determining whether the feature point is acquirable from the specific object if the specific object having the feature point that enables calibration of each of the two or more sensors exists in a region where the fields of view of the two or more sensors overlap each other, and When it is determined that the feature point is obtainable, the feature point is obtained. wherein receiving the position information and posture information of the specific object, and Based on the position information and posture information of the two or more sensors or the vehicle to which the two or more sensors are attached, and the position information and posture information of the specific object having feature points that enable calibration of each of the two or more sensors, it is determined whether the feature points are obtainable from the specific object.

Images