Information processing device and information processing method

Abstract

To provide an information processing device and information processing method that can reduce effort required to calibrate a radar installed on a main body of a moving object.SOLUTION: A device information processing unit 13 of an analysis device 12 derives a first angle formed, on the basis of a photographing result of a camera 5, by an optical axis of the camera 5 on a virtual plane and an object plane in a situation where an obstacle on which the object plane is formed is located in a position that belongs to both a photographing range of the camera 5 and an irradiation range of a radar, with the object plane facing the camera 5 and the radar 6; derives, on the basis of a detection result of the radar 6, a second angle formed by a perpendicular line extending from an irradiation base point of the radar 6 on the virtual plane toward the object plane and a radar axis pointing toward the front of the radar on the virtual plane; and derives, on the basis of the first angle and the second angle, deviation of the radar axis from the optical axis of the camera 5 on the virtual plane.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method, and is particularly suitable for use in an information processing system and an information processing method that executes processing related to a radar attached to the main body of a mobile body. [Background technology]

[0002] In recent years, radars have been installed on the bodies of vehicles (including senior cars and other vehicles that travel on roads other than public roads), autonomous robots, and other mobile objects. Radars are used to detect the presence or absence of obstacles, measure the distance to obstacles, recognize the surrounding environment of the mobile object, and perform other measurements. To accurately perform radar measurements, the installation status of the radar relative to the mobile object must be understood in advance through calibration. Understanding the installation status enables physical and software adjustments of the radar to perform accurate measurements. The installation status of the radar refers to the state of the radar relative to the vehicle (the attitude that affects the direction in which radio waves are emitted). While the term "calibration" generally includes or may refer to adjustments made after the installation status of the equipment is understood, in this specification, the term "calibration" refers to the understanding of the installation status of the equipment, but does not include (or refer to) subsequent adjustments.

[0003] Conventionally, radar calibration has been performed in the following manner. First, a mobile object is moved into a room dedicated to calibration. In the room, the mobile object is then placed at a predetermined position taking into consideration its positional relationship with a target set on the floor, wall, or the like. This placement must be performed precisely because an inappropriate placement will result in inaccurate calibration. Then, radar measurements are performed on the target, and the gap between the desired measurement result and the actual measurement result is derived, and the installation status of the radar is understood based on this gap.

[0004] Patent Document 1 describes the following technology. Specifically, a camera 10 and a LIDAR 20 are provided on a vehicle 1. An axis deviation detection device 100 extracts feature portions of an integrated object detected by both the camera 10 and the LIDAR 20. The axis deviation detection device 100 derives the direction of the feature portion related to the camera 10 in the coordinate system of the camera 10 and the direction of the feature portion related to the LIDAR 20 in the coordinate system of the LIDAR 20. If the difference between these directions exceeds a threshold, the axis deviation detection device 100 determines that an axis deviation has occurred in at least one of the camera 10 and the LIDAR 20. This technology is described. Patent Document 1 enables accurate detection of axis deviation in at least one of the camera 10 and the LIDAR 20. Patent Document 2 also describes the following technology. Specifically, a signal processing device 43 estimates the positional relationship between a first coordinate system and a second coordinate system based on the relationship between a plurality of planes in a first coordinate system of a first sensor and a plurality of planes in a second coordinate system of a second sensor. This technology is described. According to Patent Document 2, the relative positional relationship between sensors can be grasped with higher accuracy. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-020684 [Patent Document 2] International Publication No. 2017 / 159382 Summary of the Invention [Problem to be solved by the invention]

[0006] The conventional calibration method described above requires the moving object to be moved to a room dedicated to calibration, and then the moving object must be placed in a specific position in the room with strict accuracy, which makes calibration a labor-intensive process.

[0007] The present invention has been made to solve such problems, and has an object to reduce the effort required to perform calibration of a radar provided on the body of a moving object. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the present invention has the following configuration. That is, a camera is installed in addition to a radar on the main body of a moving object. In a situation where an obstacle having a planar object plane is located in a location that belongs to both the camera's imaging range and the radar's irradiation range, with the object plane facing the camera and the radar, the present invention derives a first angle formed by the optical axis of the camera on a virtual plane and the object plane based on the imaging result of the camera, derives a second angle formed by a perpendicular line extending from the radar's irradiation base point on the virtual plane toward the object plane and the radar axis pointing toward the front of the radar on the virtual plane based on the radar detection result, and derives a deviation of the radar axis from the optical axis of the camera on the virtual plane based on the first angle and the second angle. [Effects of the Invention]

[0009] According to the present invention configured as described above, if a situation occurs in which the conditions that an obstacle is within the camera's imaging range and the radar's irradiation range and the object plane faces these are satisfied, it becomes possible to derive and grasp the deviation of the radar axis from the camera's optical axis (i.e., the radar installation state) based on the camera's imaging results and the radar's detection results. Therefore, during calibration, there is no need to place the mobile object in a predetermined position with strict accuracy, and it becomes possible to reduce the effort required to perform calibration of the radar installed on the mobile object's body. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a vehicle according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of an analysis apparatus according to an embodiment. [Figure 3]FIG. 2 is a view of a vehicle and a specific obstacle viewed from above the camera of the vehicle toward below the camera. [Figure 4] 10 is a flowchart illustrating an example of the operation of an analysis device according to an embodiment. [Figure 5] FIG. 10 is a view of the vehicle and a specific obstacle as viewed from the left side of the vehicle's camera toward the right side of the camera. [Figure 6] FIG. 10 is a diagram showing a radar, a specific obstacle, and a sub-obstacle in a modified example. [Figure 7] 10 is a flowchart showing an example of the operation of an analysis device according to a modified example. [Figure 8] FIG. 10 is a diagram used to explain coordinate transformation from a first intersection to a transformed first intersection. [Figure 9] FIG. 10 is a view of the radar and the sub-obstacles as viewed from the rear side of the vehicle camera toward the front side of the camera. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of the present invention will now be described with reference to the drawings. FIG. 1 is a diagram showing a vehicle 1 (mobile body) according to this embodiment. The vehicle 1 is a type of automobile known as a senior car, capable of traveling on sidewalks with a person aboard. The vehicle 1 includes a main body 2 and tires 3 that support the main body 2. For ease of explanation, FIG. 1 shows the vehicle 1 and other elements in a simplified and schematic manner. This also applies to FIGS. 3, 5, 6, and 9, which will be described later. Furthermore, in each of the drawings depicting elements related to the vehicle 1 (particularly a camera 5 and radar 6, which will be described later), the dimensions, placement positions, and positional relationships of the elements have been changed as appropriate for the sake of explanation, and do not necessarily match between the drawings.

[0012] As shown in FIG. 1, a camera 5 is installed on the main body 2. The camera 5 has the function of taking images facing the straight-ahead direction of the vehicle 1, generating captured image data based on the image capture results, and outputting the captured image data to a vehicle-side control device 7 (described later). In this embodiment, the "side of a specific direction" does not mean a direction that completely matches the specific direction, but rather means a direction to which the specific direction belongs. For example, the "side of the straight-ahead direction of the vehicle 1" (in this case, the "direction of travel of the vehicle body" corresponds to the specific direction) means a direction to which the straight-ahead direction of the vehicle 1 belongs.

[0013] In this embodiment, a camera coordinate system C1, which is a three-dimensional coordinate system, is defined in correspondence with the camera 5. FIGS. 1A and 1B show the camera coordinate system C1 in correspondence with the camera 5. As shown in FIGS. 1A and 1B, in the camera coordinate system C1, the camera optical axis 8 (optical axis) of the camera 5 is defined as the roll axis, and a camera yaw axis C1y corresponding to the yaw axis, a camera pitch axis C1p corresponding to the pitch axis, and a camera roll axis C1r corresponding to the roll axis are defined. Hereinafter, the upward direction on the camera yaw axis C1y will be referred to as the "upward direction of the camera," and the downward direction will be referred to as the "downward direction of the camera." On the camera roll axis C1r, the direction toward the front (the direction in which the camera 5 takes a photograph) will be referred to as the "forward direction of the camera," and the backward direction will be referred to as the "backward direction of the camera." On the camera pitch axis C1p, the direction toward the left when facing the forward direction of the camera will be referred to as the "leftward direction of the camera," and the direction toward the right will be referred to as the "rightward direction of the camera."

[0014] In this embodiment, the position and orientation of the camera 5 relative to the main body 2 are known and unchanging. In addition, the relationship between the camera coordinate system C1 and the three-dimensional coordinate system of the main body 2 is known in advance, and coordinate transformation between these coordinate systems is possible.

[0015] As shown in Figure 1, the main body 2 is provided with a radar 6. The radar 6 is a millimeter-wave radar that detects reflection intensity at each point in three-dimensional space, as well as the distance, angle (azimuth), and speed to a target. The radar 6 outputs radar detection data indicating the detection results to a vehicle-side control device 7. A radar axis 10 is defined for the radar 6, extending from a radar irradiation base point 9 (irradiation base point) of the radar 6 toward the front of the radar 6. The radar axis 10 is an axis that serves as a reference for the orientation / attitude of the radar 6 in the main body 2.

[0016] In this embodiment, a radar coordinate system C2, which is a three-dimensional coordinate system, is defined in association with the radar 6. (A) and (B) of FIG. 1 show the radar coordinate system C2 in association with the radar 6. As shown in (A) and (B) of FIG. 1, in the radar coordinate system C2, the radar axis 10 is defined as the roll axis, and a radar yaw axis C2y corresponding to the yaw axis, a radar pitch axis C2p corresponding to the pitch axis, and a radar roll axis C2r corresponding to the roll axis are defined. Hereinafter, the upward direction on the radar yaw axis C2y is referred to as "radar upward," and the downward direction is referred to as "radar downward." On the radar roll axis C2r, the direction toward the front (the direction in which the radar 6 irradiates) is referred to as "radar forward," and the backward direction is referred to as "radar rearward." On the radar pitch axis C2p, the direction toward the left when looking toward the radar forward is referred to as "radar leftward," and the direction toward the right is referred to as "radar rightward."

[0017] In this embodiment, the relationship between the radar axis 10 and the camera optical axis 8 is unknown, and the radar axis 10 is not necessarily parallel to the camera optical axis 8. One of the purposes of the calibration described below is to derive the deviation between the radar axis 10 and the camera optical axis 8. The type of radar 6 is not limited, and the radar type related to distance can be, for example, a pulse type or an MFCW type, and the radar type related to angle can be, for example, a mechanical scan type, a lens scan type, or an electronic scan type. Furthermore, the use of a millimeter-wave radar as the radar 6 is merely an example.

[0018] The vehicle-side control device 7 is a computer mounted on the main body 2. The vehicle-side control device 7 has a function of inputting image data captured by the camera 5 and a function of inputting radar detection data from the radar 6. The vehicle-side control device 7 also has a function of communicating with the analysis device 12.

[0019] 2 is a block diagram showing an example of the functional configuration of an analysis device 12 (information processing device) according to this embodiment. As shown in FIG. 1, the analysis device 12 has, as its functional configuration, a device information processing unit 13 (information processing unit), a communication unit 14, a display unit 15, an input unit 16, and a storage unit 17. The device information processing unit 13 has a processing device and a primary storage device. The processing device includes a processor, and the primary storage device includes RAM. The device information processing unit 13 executes processing by having the processing device read out a program stored in a storage area of ​​the storage unit 17 (or another storage area) into the primary storage device and execute the program. In other words, the device information processing unit 13 executes processing through cooperation between hardware and software.

[0020] The communication unit 14 includes a communication device and communicates with external devices under the control of the device information processing unit 13. In particular, the communication unit 14 has a function of communicating with the vehicle-side control device 7 according to a predetermined protocol. The communication between the communication unit 14 and the vehicle-side control device 7 may be communication via a LAN, the Internet, or other network, or may be direct communication. When direct communication is performed, the direct communication may be wireless communication using Bluetooth (registered trademark) or other wireless communication standards, or may be wired communication via a cable.

[0021] The display unit 15 includes a liquid crystal panel, an organic EL panel, or other display device. The display unit 15 displays an image on the display device under the control of the device information processing unit 13. The input unit 16 includes a keyboard, a mouse, a touch panel, a camera, or other input device. The input unit 16 detects input to the input device and outputs the detection result to the device information processing unit 13. The storage unit 17 includes a nonvolatile memory and stores data in the nonvolatile memory. The nonvolatile memory is, for example, a hard disk drive (or may be another magnetic storage device), a ROM, or a flash memory.

[0022] The type of analytical device 12 is not limited. For example, a desktop PC, a notebook PC, a tablet device (including a smartphone), or a server connected to the Internet, a LAN, or other networks can function as analytical device 12. Analytical device 12 may also be configured from multiple devices. In this case, the multiple devices function as "information processing devices," and the functional units that perform information processing in the multiple devices each function as an "information processing unit."

[0023] In order to perform accurate measurements using the radar 6, it is necessary to understand the installation state of the radar 6 relative to the main body 2 through calibration. Regarding calibration, in this embodiment, a unique function of the analysis device 12 is used to reduce the labor required to perform calibration. The functions and processing of the analysis device 12 regarding calibration will be described in detail below.

[0024] For calibration, the vehicle 1 is first moved near a specific obstacle 20 (obstacle). The specific obstacle 20 is an obstacle on which a planar object plane 21 is formed. The specific obstacle 20 may be any object in which the object plane 21 having a certain size of area is formed in a position appropriate for performing the calibration described below. In this embodiment, the object plane 21 is a surface that stands perpendicular to the ground. For example, a "dedicated object for calibration," such as a wall, a guardrail, a sign, or the cargo compartment of a truck, can be used as the specific obstacle 20.

[0025] The vehicle 1 is placed in a location where a situation occurs in relation to the specific obstacle 20 that satisfies the following conditions. Condition J1: The specific obstacle 20 is located in a location that is within both the photographing range of the camera 5 and the irradiation range of the radar 6. Condition J2: The object plane 21 of the specific obstacle 20 faces the camera 5 and the radar 6.

[0026] Hereinafter, a situation that satisfies conditions J1 and J2 will be referred to as a "corresponding situation." Hereinafter, the person in charge of moving vehicle 1 (hereinafter, the person involved in calibration, including the person in charge of moving vehicle 1, will be referred to as the "person in charge") does not need to position vehicle 1 precisely and accurately in a specific location during calibration. The set of vehicle 1 positions that satisfies conditions J1 and J2 has a certain degree of breadth. Therefore, the person in charge can roughly move / position vehicle 1 to place vehicle 1 in a location that satisfies conditions J1 and J2, thereby completing the work related to moving vehicle 1. In this respect, the workload of the person in charge is small.

[0027] After moving the vehicle 1 so as to present the corresponding situation, the person in charge inputs a predetermined command to the input unit 16 to instruct the start of calibration. In response to this command, the device information processing unit 13 communicates with the vehicle-side control device 7 and instructs the start of control of the camera 5 and the radar 6. In response to this command, the vehicle-side control device 7 causes the camera 5 to capture images and transmits captured image data based on the capture results of the camera 5 to the device information processing unit 13. The vehicle-side control device 7 also drives the radar 6 and transmits radar detection data based on the detection results of the radar 6 to the device information processing unit 13. The radar detection data includes reflection intensity in three-dimensional space, the distance to the target, the angle (azimuth) to the target, and other information related to the three-dimensional space. In other words, the radar detection data includes all the information necessary for the device information processing unit 13 to perform the processing described below. The device information processing unit 13 receives and acquires the captured image data of the camera 5 and the radar detection data of the radar 6. The device information processing unit 13 performs "yaw angle-related processing" and "pitch angle-related processing" based on the acquired various data. Each process will be explained below.

[0028] ● Yaw angle related processing First, the yaw angle-related processing will be described. FIG. 3 is a view of the vehicle 1 and the specific obstacle 20 viewed from above the camera of the vehicle 1 toward below the camera, for use in explaining the yaw angle-related processing. Flowchart FA in FIG. 4 is a flowchart illustrating the details of the yaw angle-related processing. As shown in flow chart FA, the device information processing unit 13 derives a first yaw system angle θy1 (first angle, the first angle corresponding to the first imaginary plane) based on the image capture results of the camera 5 (step SA1). The first yaw system angle θy1 is the angle between the "camera optical axis 8 on the first imaginary plane" and the object plane 21. The first imaginary plane is a virtual plane that is perpendicular to the camera yaw axis C1y in the camera coordinate system C1. The camera optical axis 8 on the first imaginary plane refers to the axis projected onto the first imaginary plane when the camera optical axis 8 is projected in the direction of the camera yaw axis C1y relative to the first imaginary plane. Hereinafter, the camera optical axis 8 on the first imaginary plane will be referred to as the "yaw system camera optical axis 8y." FIG. 3 shows an example of the yaw system camera optical axis 8y and the yaw system first angle θy1.

[0029] Unless otherwise specified below, "axis (line / line segment) on an imaginary plane" means the following: That is, it means the axis (line / line segment) projected onto the imaginary plane when that axis (line / line segment) is projected onto the imaginary plane in a "direction perpendicular to the imaginary plane."

[0030] The processing of step SA1 will be described in detail below. The device information processing unit 13 analyzes the captured image data through image processing (particularly, image processing related to image recognition / object recognition) and identifies an image of the specific obstacle 20 from the captured image data, as well as an image of the object plane 21. Next, the device information processing unit 13 reflects various elements of the captured image data to identify the area (three-dimensional area) of the image of the object plane 21 in the camera coordinate system C1. These elements include the appearance of the image of the specific obstacle 20 in the captured image data and the appearance of the image of the object plane 21 in the captured image data (particularly the appearance of the boundary between the object plane 21 and the ground). Next, the device information processing unit 13 identifies the vector of the yaw-system camera optical axis 8y in the camera coordinate system C1. Next, the device information processing unit 13 derives the angle formed on the first virtual plane between the yaw-system camera optical axis 8y and the object plane 21 based on the area of ​​the image of the object plane 21 in the camera coordinate system C1 and the vector of the yaw-system camera optical axis 8y. The angle derived here is the first yaw-system angle θy1. The above is the details of the processing in step SA1.

[0031] The device information processing unit 13 executes the process of step SA1 using a program that has a function of analyzing the captured image data. The program may include a model learned by machine learning. This also applies to other processes.

[0032] After step SA1, the device information processing unit 13 derives the second yaw-system angle θy2 based on the detection result of the radar 6 (step SA2). The second yaw-system angle θy2 is the angle between the "perpendicular line extending from the radar irradiation base point 9 toward the object plane 21" on the first imaginary plane and the "radar axis 10 on the first imaginary plane." Hereinafter, the "perpendicular line extending from the radar irradiation base point 9 toward the object plane 21" on the first imaginary plane will be referred to as the "yaw-system perpendicular line Py." Also, the "radar axis 10 on the first imaginary plane" will be referred to as the "yaw-system radar axis 10y." FIG. 3 shows an example of the yaw-system perpendicular line Py, the yaw-system radar axis 10y, and the second yaw-system angle θy2. The processing of step SA2 will be described in detail below.

[0033] The device information processing unit 13 analyzes the radar detection data and identifies the detection point with the strongest reflection intensity from the radar 6. Next, the device information processing unit 13 identifies a line connecting the detection point and the radar irradiation base point 9 (hereinafter referred to as the "yaw-system provisional perpendicular line"). Here, the intersection of the perpendicular line drawn from the radar irradiation base point 9 to the object plane 21 and the object plane 21 is the point on the object plane 21 closest to the radar irradiation base point 9, and therefore has the strongest reflection intensity. Therefore, the yaw-axis provisional perpendicular line coincides with the perpendicular line drawn from the radar irradiation base point 9 to the object plane 21. Next, the device information processing unit 13 identifies the yaw-axis provisional perpendicular line on the first imaginary plane as the yaw-system perpendicular line Py. Next, the device information processing unit 13 identifies the radar axis 10 on the first imaginary plane as the yaw-system radar axis 10y. Next, the device information processing unit 13 derives the angle formed on the first imaginary plane by the yaw-system perpendicular line Py and the yaw-system radar axis 10y. The angle derived here is the second yaw system angle θy2. The above is the details of the processing in step SA2.

[0034] After step SA2, the device information processing unit 13 derives the third yaw system angle θy3 (step SA3). The third yaw system angle θy3 is the angle between the radar axis 10 on the first imaginary plane (= yaw system radar axis 10y) and the object plane 21. FIG. 3 shows an example of the third yaw system angle θy3. As shown in FIG. 3, on the first imaginary plane, the sum of the interior angles of the triangle formed by the radar irradiation base point 9, the intersection Ey1 of the yaw system perpendicular line Py and the object plane 21, and the intersection Ey2 of the yaw system radar axis 10y and the object plane 21 is 180 degrees, and the angle corresponding to the intersection Ey1 is 90 degrees. Based on this, in step SA3, the device information processing unit 13 subtracts the second yaw system angle θy2 from 90 degrees to derive the third yaw system angle θy3.

[0035] After step SA3, the device information processing unit 13 derives the yaw system deviation amount θyd (step SA4). In step SA4, the device information processing unit 13 derives the yaw system deviation amount θyd by subtracting the third yaw system angle θy3 from the first yaw system angle θy1. The yaw system deviation amount θyd derived in this manner corresponds to the angle of the radar axis 10 about the camera yaw axis C1y with respect to the camera optical axis 8. Therefore, the yaw system deviation amount θyd can be used as an index / external parameter that indicates the degree of deviation of the radar axis 10 about the camera yaw axis C1y with respect to the camera optical axis 8 in the first imaginary plane. That is, in step SA4, the device information processing unit 13 derives the deviation of the radar axis 10 from the camera optical axis 8 in the first imaginary plane (imaginary plane) based on the first yaw system angle θy1 and the second yaw system angle.

[0036] After step SA4, the device information processing unit 13 provides the yaw system deviation amount θyd to the person in charge (step SA5). For example, the device information processing unit 13 displays information indicating the yaw system deviation amount θyd on the display unit 15. This allows the person in charge to understand the yaw system deviation amount θyd.

[0037] ● Pitch angle related processing Next, the pitch angle-related processing will be explained. Figure 5 is a view of the vehicle 1 and the specific obstacle 20 viewed from the left side of the camera of the vehicle 1 toward the right side of the camera, which is used to explain the pitch angle-related processing. The flow of the pitch angle-related processing is similar to the flow of the yaw angle-related processing, and below, the pitch angle-related processing will be briefly explained while appropriately referring to the yaw angle-related processing.

[0038] In the pitch angle-related processing, the device information processing unit 13 derives the pitch system first angle θp1 (first angle, first angle corresponding to the second imaginary plane) based on the captured image data. The pitch system first angle θp1 is the angle formed on the second imaginary plane between the pitch system camera optical axis 8p and the object plane 21. The pitch system camera optical axis 8p is the camera optical axis 8 on the second imaginary plane. The second imaginary plane is a virtual plane that is perpendicular to the camera pitch axis C1p in the camera coordinate system C1. FIG. 5 shows an example of the pitch system camera optical axis 8p and the pitch system first angle θp1. The device information processing unit 13 derives the pitch system first angle θp1 in a manner similar to the method for deriving the yaw system first angle θy1.

[0039] Next, the device information processing unit 13 derives the pitch system second angle θp2 based on the radar detection data. The pitch system second angle θp2 is the angle between the pitch system perpendicular line Pp and the pitch system radar axis 10p. The pitch system perpendicular line Pp is a perpendicular line extending from the radar irradiation base point 9 toward the object plane 21 on the second imaginary plane. The pitch system radar axis 10p is the radar axis 10 on the second imaginary plane. FIG. 5 shows an example of the pitch system perpendicular line Pp, the pitch system radar axis 10p, and the pitch system second angle θp2. The device information processing unit 13 derives the pitch system second angle θp2 in a manner similar to the manner of deriving the yaw system second angle θy2.

[0040] Next, the device information processing unit 13 derives the pitch system third angle θp3. The pitch system third angle θp3 is the angle between the "radar axis 10 on the second imaginary plane (=pitch system radar axis 10p)" and the object plane 21. FIG. 5 shows an example of the pitch system third angle θp3. The device information processing unit 13 derives the pitch system third angle θp3 in a manner similar to the method for deriving the yaw system third angle θy3.

[0041] Next, the device information processing unit 13 derives the pitch system deviation amount θpd. The device information processing unit 13 derives the pitch system deviation amount θpd by subtracting the pitch system third angle θp3 from the pitch system first angle θp1. The pitch system deviation amount θpd derived in this manner corresponds to the angle of the radar axis 10 about the camera pitch axis C1p relative to the camera optical axis 8. Therefore, the pitch system deviation amount θpd can be used as an index / external parameter indicating the degree of deviation of the radar axis 10 about the camera pitch axis C1p relative to the camera optical axis 8 in the second imaginary plane. In other words, in the pitch angle-related processing, the device information processing unit 13 derives the deviation of the radar axis 10 from the camera optical axis 8 in the second imaginary plane (imaginary plane) based on the pitch system first angle θp1 and the pitch system second angle θp2.

[0042] Next, the device information processing unit 13 provides the pitch system deviation amount θpd to the person in charge. This concludes the description of the pitch angle related processing.

[0043] As described above, the device information processing unit 13 (information processing unit) of the analysis device 12 (information processing device) according to this embodiment derives, in a corresponding situation, a first angle formed between the camera optical axis 8 on the virtual plane and the object plane 21 based on the imaging results of the camera 5, derives a second angle formed between the radar axis 10 on the virtual plane and a perpendicular line extending from the radar irradiation base point 9 on the virtual plane toward the object plane 21, based on the detection results of the radar 6, and derives a deviation of the radar axis 10 from the camera optical axis 8 on the virtual plane based on the first angle and the second angle. According to this configuration, if a situation occurs in which the conditions that a specific obstacle 20 (obstacle) belongs to the imaging range of the camera 5 and the irradiation range of the radar 6 and the object plane 21 faces these are satisfied, it becomes possible to derive the deviation of the radar axis 10 from the camera optical axis 8 (i.e., the installation state of the radar) based on the imaging results of the camera 5 and the detection results of the radar, and grasp the deviation. Therefore, when performing calibration, there is no need to place the vehicle 1 (moving body) in a predetermined position precisely and accurately, which makes it possible to reduce the effort required to perform calibration of the radar 6 provided on the main body 2 of the vehicle 1.

[0044] <Modification> Next, a modification of the above embodiment will be described. The device information processing unit 13 of the analysis device 12 according to this modification has a function of executing roll angle-related processing. The roll angle-related processing will be described in detail below.

[0045] ●Roll angle related processing In this modified example, calibration is first performed by moving the vehicle 1 near the specific obstacle 20 and the sub-obstacle 23 (another obstacle). As in the above embodiment, the specific obstacle 20 is located on the front side of the radar along the radar roll axis C2r (the vehicle 1 is moved so that this state is achieved). On the other hand, the sub-obstacle 23 is located on the left side (one of the facing sides) of the radar along the radar pitch axis C2p (the vehicle 1 is moved so that this state is achieved). The sub-obstacle 23 is located in a location that is within both the imaging range of the camera 5 and the irradiation range of the radar 6, with its object plane (hereinafter referred to as the "sub-object plane 24") facing the camera 5 and the radar 6. For example, two walls connected at an angle in a room can be used as the specific obstacle 20 and the sub-obstacle 23.

[0046] FIG. 6 is a diagram schematically illustrating the camera 5, radar 6, specific obstacle 20, and sub-obstacle 23 for use in explaining the roll angle-related processing. In FIG. 6, the camera 5 and radar 6 are extracted and depicted from among the elements of the vehicle 1. Flowchart FB in FIG. 7 is a flowchart illustrating the roll angle-related processing in detail. As shown in flow chart FB, the device information processing unit 13 derives the yaw system deviation amount θyd using the specific obstacle 20 (step SB1). The device information processing unit 13 executes the yaw angle-related processing described in the above embodiment to derive the yaw system deviation amount θyd. That is, the device information processing unit 13 derives a first angle (first yaw system angle) and a second angle (second yaw system angle) corresponding to the first imaginary plane, and derives the angle of the radar axis 10 about the camera yaw axis C1y with respect to the camera optical axis 8 on the first imaginary plane as the yaw system deviation amount θyd based on the first angle and the second angle corresponding to the first imaginary plane.

[0047] After step SB1, the device information processing unit 13 derives the pitch system deviation θpd using the specific obstacle 20 (step SB2). The device information processing unit 13 executes the pitch angle-related processing described in the above embodiment to derive the pitch system deviation θpd. That is, the device information processing unit 13 derives a first angle (pitch system first angle) and a second angle (pitch system second angle) corresponding to the second imaginary plane, and derives the angle of the radar axis 10 about the camera pitch axis C1p with respect to the camera optical axis 8 on the second imaginary plane as the pitch system deviation θpd based on the first angle and the second angle corresponding to the second imaginary plane.

[0048] After step SB2, the device information processing unit 13 derives the coordinates of the first intersection point Er1 using the sub-obstacle 23 (step SB3). The first intersection point Er1 is the intersection point between the "axis extending from the radar irradiation base point 9 along the camera pitch axis C1p" and the sub-object plane 24. Hereinafter, the "axis extending from the radar irradiation base point 9 along the camera pitch axis C1p" will be referred to as the "roll system camera corresponding axis 8r." FIG. 6 shows an example of the roll system camera corresponding axis 8r and the first intersection point Er1. More specifically, the device information processing unit 13 identifies the area (three-dimensional area) of the image of the sub-obstacle 23 on the sub-object plane 24 in the camera coordinate system C1 based on the image data captured by the camera 5. Next, the device information processing unit 13 identifies the vector of the roll system camera corresponding axis 8r in the camera coordinate system C1. Next, based on the area of ​​the image of the sub-object plane 24 in the camera coordinate system C1 and the vector of the roll system camera corresponding axis 8r, the device information processing unit 13 derives the coordinates of the first intersection Er1 where the plane and the axis intersect.

[0049] After step SB3, the device information processing unit 13 performs coordinate transformation on the first intersection so that the yaw system deviation amount θyd and the pitch system deviation amount θpd are corrected, thereby deriving a transformed first intersection point Er1' (step SB4). FIG. 8 illustrates, in a manner suitable for explanation, how the first intersection point Er1 is transformed into the transformed first intersection point Er1'. As shown in FIG. 8, the device information processing unit 13 performs coordinate transformation on the first intersection point Er1 about the camera yaw axis C1y corresponding to the yaw system deviation amount θyd, and coordinate transformation on the camera pitch axis C1p corresponding to the pitch system deviation amount θpd, thereby deriving the transformed first intersection point Er1'. Specifically, the device information processing unit 13 performs coordinate transformation using the following coordinate transformation equation that uses the yaw system deviation amount θyd and the pitch system deviation amount θpd:

[0050]

number

[0051] where θyd is the yaw misalignment, θpd is the pitch misalignment, (x, y, z) are the coordinates of the first intersection point Er1, and (x', y', z') are the coordinates of the first intersection point Er1' after transformation.

[0052] After step SB4, the device information processing unit 13 derives the coordinates of the second intersection point Er2 using the sub-obstacle 23 (step SB5). The second intersection point Er2 is the intersection point between the "axis extending from the radar irradiation base point 9 along the radar pitch axis C2p" and the sub-object plane 24. Hereinafter, the "axis extending from the radar irradiation base point 9 along the radar pitch axis C2p" will be referred to as the "roll system radar corresponding axis 10r." FIG. 6 shows an example of the roll system radar corresponding axis 10r and the second intersection point Er2. The device information processing unit 13 derives the coordinates of the second intersection point Er2 in a manner similar to the method for deriving the coordinates of the first intersection point Er1. In brief, the device information processing unit 13 identifies the vector of the roll-type radar corresponding axis 10r in the camera coordinate system C1, and derives the coordinates of the second intersection point Er2 where the sub-object plane 24 and the roll-type radar corresponding axis intersect based on the vector and the area of ​​the image of the sub-object plane 24.

[0053] After step SB5, the device information processing unit 13 performs coordinate transformation on the second intersection point Er2 so that the yaw system deviation amount θyd and the pitch system deviation amount θpd are corrected, and derives a transformed second intersection point Er2' (step SB6). The device information processing unit 13 performs coordinate transformation using the above-mentioned coordinate transformation formula that uses the yaw system deviation amount θyd and the pitch system deviation amount θpd, and derives a transformed second intersection point Er2'.

[0054] After step SB6, the device information processing unit 13 derives the angle between the "line connecting the radar irradiation base point 9 and the converted first intersection point Er1' on the third imaginary plane" and the sub-object plane 24 as the roll system first angle θr1 (first angle, the first angle corresponding to the third imaginary plane) (step SB7). The third imaginary plane is a virtual plane perpendicular to the camera roll axis C1r. Hereinafter, the "line connecting the radar irradiation base point 9 and the converted first intersection point Er1 on the third imaginary plane" will be referred to as the "converted roll system camera-corresponding axis 8r'." FIG. 9 is a view of the radar 6 and the sub-obstacle 23 viewed from the rear side of the camera of the vehicle 1 toward the front side of the camera. FIG. 9 shows an example of the converted first intersection point Er1', the converted roll system camera-corresponding axis 8r', and the roll system first angle θr1.

[0055] After step SB7, the device information processing unit 13 derives the angle between the "perpendicular line extending from the radar illumination origin point 9 toward the sub-object plane 24" on the third imaginary plane and the "line connecting the radar illumination origin point 9 and the converted second intersection point Er2'" on the third imaginary plane as the roll system second angle θr2 (second angle, second angle corresponding to the third imaginary plane) (step SB8). Hereinafter, the "perpendicular line extending from the radar illumination origin point 9 toward the sub-object plane 24" on the third imaginary plane will be referred to as the roll system perpendicular line Pr. Also, the "line connecting the radar illumination origin point 9 and the converted second intersection point Er2'" on the third imaginary plane will be referred to as the "converted roll system radar-corresponding axis 10r'." FIG. 9 shows an example of the roll system perpendicular line Pr, the converted roll system radar-corresponding axis 10r', and the roll system second angle θr2.

[0056] After step SB8, the device information processing unit 13 subtracts the second roll system angle θr2 from 90 degrees to derive the third roll system angle θr3 (step SB9). The third roll system angle θr3 is the angle formed on the third virtual plane between the converted roll system radar corresponding axis 10r' and the sub-object plane 24. Figure 9 shows the third roll system angle θr3.

[0057] After step SB9, the device information processing unit 13 derives the roll system deviation amount θrd by subtracting the roll system third angle θr3 from the roll system first angle θr1 (step SB10). The roll system deviation amount θrd derived in this manner corresponds to the angle of the radar pitch axis C2p about the camera roll axis C1r relative to the camera pitch axis C1p. Therefore, the roll system deviation amount θrd can be used as an index / external parameter indicating the degree of deviation of the radar pitch axis C2p relative to the camera pitch axis C1p in the third imaginary plane.

[0058] After step SB10, the apparatus information processor 13 provides the roll-system deviation amount θrd.

[0059] Although one embodiment of the present invention has been described above, the above embodiment merely illustrates one example of a specific embodiment for carrying out the present invention, and the technical scope of the present invention should not be interpreted as being limited thereby. In other words, the present invention can be carried out in various forms without departing from the gist or main features thereof. Below, variations of the above embodiment are presented. When multiple variations can be combined for implementation, multiple variations may be combined for implementation.

[0060] For example, in the above embodiment, the device information processing unit 13 may be configured to execute various processes in cooperation with an external device. In this case, the external device is, for example, a cloud server capable of communicating with the vehicle-side control device 7 or the analysis device 12. Also, the vehicle-side control device 7 may be configured to execute some or all of the processes described as being executed by the analysis device 12. In this case, the vehicle-side control device 7 functions as an "information processing device," and the functional unit that executes information processing in the vehicle-side control device 7 functions as an "information processing unit."

[0061] In the above embodiment, the device information processing unit 13 acquires various data from the vehicle-side control device 7 through communication. However, the method by which the device information processing unit 13 acquires various data required for processing is not limited to the method exemplified in the above embodiment. As an example, the captured image data and radar detection data may be uploaded to a server, and the device information processing unit 13 may download them. Alternatively, the various data may be stored in the storage unit 17 via a storage medium such as a USB memory, and the device information processing unit 13 may acquire the various data from the storage unit 17.

[0062] Furthermore, the vehicle 1 is not limited to a mobility scooter. For example, the vehicle 1 may be another type of vehicle that travels on sidewalks, such as an electric wheelchair, an autonomous robot, a vehicle that travels on roadways, or an air vehicle such as a drone. In other words, the present invention can be widely applied to any mobile object equipped with a radar.

[0063] The camera 5 may also be an imaging device equipped with a function for measuring the distance to a target or a function for acquiring three-dimensional information about space. For example, the camera 5 may be a stereo camera, a TOF camera, or a monocular camera that measures distance by utilizing the blurring of an image due to lens aberration. For example, if the camera 5 is configured as a stereo camera, the device information processing unit 13 analyzes the three-dimensional image data generated by the stereo camera, recognizes the state of the image of the specific obstacle 20 in three-dimensional space deployed in the camera coordinate system C1, and derives the first yaw angle θy1 and other information based on this recognition. This also applies to TOF cameras and monocular cameras related to lens aberration.

[0064] Furthermore, the device information processing unit 13 may be configured to have a function of executing only one of the yaw angle-related processing, the pitch angle-related processing, and the roll angle-related processing.

[0065] The functional blocks shown in the above embodiments can be realized by any hardware or by a combination of any hardware and any software, and are not limited to specific hardware.

[0066] The embodiments may also include providing a program executed by the analysis device 12 (computer). The embodiments may also include providing a recording medium on which the program is recorded so as to be readable by a computer. The recording medium may be a magnetic or optical recording medium or a semiconductor memory device. Specific examples include portable or fixed recording media such as flexible disks, HDDs (Hard Disk Drives), CD-ROMs (Compact Disk Read Only Memory), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, magneto-optical disks, flash memories, and card-type recording media. [Explanation of symbols]

[0067] 1. Vehicle (moving object) 2 Main unit 5. Camera 6. Radar 12 Analytical equipment (information processing equipment) 13 Device information processing section (information processing section) 21 Object plane 20 Specific Obstacles (Obstacles)

Claims

1. An information processing device that executes processing related to radar installed in a main body of a moving body, A camera is installed on the main body, In a situation where an obstacle having a planar object plane is disposed in a location that belongs to both the imaging range of the camera and the irradiation range of the radar, with the object plane facing the camera and the radar, deriving a first angle formed between an optical axis of the camera on a virtual plane and the object plane based on the image capturing result of the camera; deriving a second angle between a perpendicular line extending from an irradiation base point of the radar on the virtual plane toward the object plane and a radar axis directed toward a front of the radar on the virtual plane, based on a detection result of the radar; an information processing unit that derives a deviation of the radar axis from the optical axis of the camera on the virtual plane based on the first angle and the second angle; 1. An information processing device comprising:

2. The information processing unit derives, as the deviation, a difference between a third angle obtained by subtracting the second angle from 90 degrees and the first angle.

2. The information processing apparatus according to claim 1, wherein:

3. The information processing unit specifies, as the perpendicular line, a line on the virtual plane that connects the detection point with the strongest reflection intensity by the radar and the irradiation base point of the radar.

2. The information processing apparatus according to claim 1, wherein:

4. In the camera, a camera yaw axis corresponding to the yaw axis, a camera pitch axis corresponding to the pitch axis, and a camera roll axis corresponding to the roll axis are defined, with the optical axis of the camera being defined as a roll axis; the virtual plane is a virtual plane perpendicular to the camera yaw axis, The information processing unit derives, as the deviation, an angle of the radar axis about the camera yaw axis with respect to the optical axis of the camera on the virtual plane based on the first angle and the second angle.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. In the camera, a camera yaw axis corresponding to the yaw axis, a camera pitch axis corresponding to the pitch axis, and a camera roll axis corresponding to the roll axis are defined, with the optical axis of the camera being defined as a roll axis; the virtual plane is a virtual plane orthogonal to the camera pitch axis, The information processing unit derives, as the deviation, an angle of the radar axis about the camera pitch axis with respect to the optical axis of the camera on the virtual plane based on the first angle and the second angle.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. In the camera, a camera yaw axis corresponding to the yaw axis, a camera pitch axis corresponding to the pitch axis, and a camera roll axis corresponding to the roll axis are defined, with the optical axis of the camera being defined as a roll axis; In the radar, a radar yaw axis corresponding to the yaw axis, a radar pitch axis corresponding to the pitch axis, and a radar roll axis corresponding to the roll axis are defined, with the radar axis being defined as a roll axis; the first virtual plane is a virtual plane perpendicular to the camera yaw axis, the second virtual plane is a virtual plane perpendicular to the camera pitch axis, the third virtual plane is a virtual plane perpendicular to the camera roll axis, another obstacle is disposed on one side of the radar in the radar pitch axis with its object plane facing the radar; The information processing unit deriving the first angle and the second angle corresponding to the first imaginary plane, and deriving an angle of the radar axis about the camera yaw axis with respect to the optical axis of the camera on the first imaginary plane as a yaw system deviation amount based on the first angle and the second angle corresponding to the first imaginary plane; deriving the first angle and the second angle corresponding to the second imaginary plane, and deriving an angle of the radar axis about the camera pitch axis with respect to the optical axis of the camera on the second imaginary plane as a pitch system deviation amount based on the first angle and the second angle corresponding to the second imaginary plane; deriving a first intersection point that is an intersection point between an axis extending from the irradiation base point of the radar along the camera pitch axis and an object plane of the other obstacle, and deriving a transformed first intersection point by performing coordinate transformation on the first intersection point so that the yaw system deviation amount and the pitch system deviation amount are corrected; deriving a second intersection point that is an intersection point between an axis extending from the irradiation base point of the radar along the radar pitch axis and an object plane of the other obstacle, and deriving a transformed second intersection point by performing coordinate transformation on the second intersection point so that the yaw system deviation amount and the pitch system deviation amount are corrected; deriving an angle formed by a line connecting the irradiation base point on the third imaginary plane with the transformed first intersection point and an object plane of the other obstacle as the first angle corresponding to the third imaginary plane; deriving, as the second angle corresponding to the third imaginary plane, an angle formed by a perpendicular line extending from the irradiation base point of the radar on the third imaginary plane toward the object plane of the other obstacle and a line connecting the irradiation base point of the radar on the third imaginary plane and the second intersection point after conversion; Deriving a deviation of the radar pitch axis from the camera pitch axis on the third imaginary plane based on the first angle and the second angle corresponding to the third imaginary plane.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

7. An information processing method by an information processing device that executes processing related to a radar installed in a main body of a moving body, A camera is installed on the main body, The information processing unit of the information processing device In a situation where an obstacle having a planar object plane is disposed in a location that belongs to both the imaging range of the camera and the irradiation range of the radar, with the object plane facing the camera and the radar, deriving a first angle between an optical axis of the camera on a virtual plane and the object plane based on an image capturing result of the camera, and deriving a second angle between a perpendicular line extending from an irradiation base point of the radar on the virtual plane toward the object plane and a radar axis pointing toward a front of the radar on the virtual plane based on a detection result of the radar; and deriving a deviation of the radar axis from the optical axis of the camera on the virtual plane based on the first angle and the second angle.

1. An information processing method comprising:

Images