Calibration system, calibration device, calibration method, calibration program

Abstract

Providing a system for accurate calibration between an imaging camera and observation radar while the vehicle is in motion. [Solution] The processor in the system for calibrating the relationship between the imaging camera and the observation radar in the host vehicle is configured to perform the following actions: acquire image data Dc of the camera field of view captured by the imaging camera, along with point cloud data Dr of the radar field of view superimposed on the camera field of view by the observation radar; monitor the amount of attitude angle change Δψ of the host vehicle that appears when the peak point in the correlation distribution of vertical angle, horizontal angle and Doppler velocity represented by the point cloud data Dr for each observation point shifts relative to the reference point of the point cloud data Dr; and construct a calibration parameter Cp corrected according to the amount of attitude angle change Δψ as a parameter for calibrating the image data Dc to be used for fusion with the point cloud data Dr.

Description

Technical Field

[0001] The present disclosure relates to a calibration technique for calibrating between an imaging camera and an observation radar in a vehicle.

Background Art

[0002] The technique disclosed in Patent Document 1 realizes calibration between an imaging camera and an observation radar in a vehicle by using a dedicated calibration plate.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technique disclosed in Patent Document 1 requires a dedicated calibration plate, so the stopped offline state of the vehicle becomes a timing requirement for calibration. However, for calibration between an imaging camera and an observation radar in a running vehicle, errors will occur depending on the vehicle posture that changes at any time.

[0005] An object of the present disclosure is to provide a calibration system that accurately calibrates between an imaging camera and an observation radar while the vehicle is running. Another object of the present disclosure is to provide a calibration device that accurately calibrates between an imaging camera and an observation radar while the vehicle is running. Another object of the present disclosure is to provide a calibration method that accurately calibrates between an imaging camera and an observation radar while the vehicle is running. Yet another object of the present disclosure is to provide a calibration program that accurately calibrates between an imaging camera and an observation radar while the vehicle is running. It should be noted that there seems to be an incorrect "十七" in the original text at line 17 which is retained as is in the translation. Also, there is an unclear "<6メートル以上の距離で," at line 20 which is translated as best as possible while keeping the original form. [Means for solving the problem]

[0006] The following describes the technical means of solving the problem described in this disclosure. Note that the claims and the reference numerals in parentheses in this section indicate the correspondence with the specific means described in the embodiments detailed later, and do not limit the technical scope of this disclosure.

[0007] The first aspect of this disclosure is, A calibration system having a processor (12) for calibrating between an imaging camera (3) and an observation radar (4) in a host vehicle (2), The processor is The system acquires image data (Dc) captured by the imaging camera within the camera's field of view (Ac), along with point cloud data (Dr) observed by the observation radar within the radar's field of view (Ar) that overlaps with the camera's field of view. This involves monitoring the change in the host vehicle's attitude angle (Δψ) that occurs when the peak point (Pp) in the correlation distribution (α) between the vertical angle (θv), horizontal angle (θh), and Doppler velocity (Vd) represented by the point cloud data for each observation point shifts relative to the reference point (Pb) in the point cloud data. The system is configured to construct calibration parameters (Cp) that are corrected according to the amount of attitude angle change, as parameters for calibrating image data to be used for fusion with point cloud data.

[0008] A second aspect of this disclosure is, A calibration device having a processor (12) for calibrating between an imaging camera (3) and an observation radar (4) in a host vehicle (2), and configured to be mounted on the host vehicle, The processor is The system acquires image data (Dc) captured by the imaging camera within the camera's field of view (Ac), along with point cloud data (Dr) observed by the observation radar within the radar's field of view (Ar) that overlaps with the camera's field of view. This involves monitoring the change in the host vehicle's attitude angle (Δψ) that occurs when the peak point (Pp) in the correlation distribution (α) between the vertical angle (θv), horizontal angle (θh), and Doppler velocity (Vd) represented by the point cloud data for each observation point shifts relative to the reference point (Pb) in the point cloud data. The system is configured to construct calibration parameters (Cp) that are corrected according to the amount of attitude angle change, as parameters for calibrating image data to be used for fusion with point cloud data.

[0009] A third aspect of this disclosure is: A calibration method performed by a processor (12) to calibrate the relationship between an imaging camera (3) and an observation radar (4) in a host vehicle (2), The system acquires image data (Dc) captured by the imaging camera within the camera's field of view (Ac), along with point cloud data (Dr) observed by the observation radar within the radar's field of view (Ar) that overlaps with the camera's field of view. This involves monitoring the change in the host vehicle's attitude angle (Δψ) that occurs when the peak point (Pp) in the correlation distribution (α) between the vertical angle (θv), horizontal angle (θh), and Doppler velocity (Vd) represented by the point cloud data for each observation point shifts relative to the reference point (Pb) in the point cloud data. This includes constructing calibration parameters (Cp) that are corrected according to the amount of attitude angle change, as parameters for calibrating image data to be used for fusion with point cloud data.

[0010] The fourth aspect of this disclosure is: A calibration program stored in a storage medium (10) for calibrating between an imaging camera (3) and an observation radar (4) in a host vehicle (2), and including instructions for causing a processor (12) to perform said calibration, The system acquires image data (Dc) captured by the imaging camera within the camera's field of view (Ac), along with point cloud data (Dr) observed by the observation radar within the radar's field of view (Ar) that overlaps with the camera's field of view. This involves monitoring the change in the host vehicle's attitude angle (Δψ) that occurs when the peak point (Pp) in the correlation distribution (α) between the vertical angle (θv), horizontal angle (θh), and Doppler velocity (Vd) represented by the point cloud data for each observation point shifts relative to the reference point (Pb) in the point cloud data. This includes a command to construct calibration parameters (Cp) that are corrected according to the change in attitude angle, as parameters for calibrating image data to be used for fusion with point cloud data.

[0011] As described above, according to the first to fourth embodiments, along with image data captured by the imaging camera capturing the camera field of view, point cloud data is acquired by the observation radar observing the radar field of view superimposed on the camera field of view. Here, with respect to the reference point of the point cloud data, if the peak point in the correlation distribution between the vertical angle, horizontal angle, and Doppler velocity represented by the point cloud data for each observation point shifts, the attitude angle change amount corresponding to this shift will appear in the host vehicle. Therefore, the attitude angle change amount of the host vehicle can be properly monitored based on the shift in the peak point of the correlation distribution with respect to the reference point. Consequently, as parameters for calibrating the image data in order to fuse it with the point cloud data, it is possible to output calibration parameters that can be accurately corrected according to the attitude angle change amount that is properly monitored even while the host vehicle is in motion. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing the overall configuration of the calibration system according to the first embodiment. [Figure 2] It is a block diagram for explaining the functional configuration of the calibration system according to the first embodiment. [Figure 3] It is a bird's-eye view showing the fields of view of the imaging camera and the observation radar according to the first embodiment. [Figure 4] It is a flowchart showing the calibration flow of the first embodiment. [Figure 5] It is a graph for explaining the calibration flow of the first embodiment. [Figure 6] It is a schematic diagram for explaining the calibration flow of the first embodiment. [Figure 7] It is a flowchart showing the calibration flow of the second embodiment.

Modes for Carrying Out the Invention

[0013] Hereinafter, a plurality of embodiments of the present disclosure will be described based on the drawings. In addition, in each embodiment, the same reference numerals may be assigned to corresponding components, and redundant explanations may be omitted. Also, when only a part of the configuration is described in each embodiment, the configuration of other embodiments described previously can be applied to other parts of the said configuration. Furthermore, not only the combinations of configurations explicitly shown in the description of each embodiment, but also the configurations of a plurality of embodiments can be partially combined with each other as long as there is no problem with the combination.

[0014] (First Embodiment) The calibration system 1 of the first embodiment shown in FIGS. 1 and 2 calibrates between the imaging camera 3 and the observation radar 4 in the host vehicle 2. The host vehicle 2 can be said to be an ego-vehicle from the perspective centered on the vehicle itself. The host vehicle 2 is a moving body such as an automobile that can travel on a road in the state where a passenger is on board. Therefore, the directions in the following description are defined based on the host vehicle 2 on the horizontal plane. Note that FIGS. 1 and 2 typically show an example in which the entire calibration system 1 is configured to be mounted on the host vehicle 2 as an example implemented in the form of a calibration device such as a processing circuit (e.g., a processing ECU or the like) or a semiconductor unit (e.g., a semiconductor chip or the like).

[0015] In the host vehicle 2, an automatic driving mode is provided, which is classified according to the degree of manual intervention of the passenger in the dynamic driving task. The automatic driving mode may be realized by autonomous driving control in which the system during operation executes all dynamic driving tasks, such as conditional driving automation, highly automated driving, or fully automated driving. The automatic driving mode may be realized by advanced driving assistance control in which the passenger executes some or all of the dynamic driving tasks, such as driving assistance or partial driving automation. The automatic driving mode may be realized by either one, combination, or switching of the autonomous driving control and the advanced driving assistance control.

[0016] The host vehicle 2 is equipped with at least one set of the imaging camera 3 and the observation radar 4 that are calibration targets for each other. The set of the imaging camera 3 and the observation radar 4 is arranged in the host vehicle 2 so that their respective visual fields Ac and Ar shown in FIG. 3 overlap each other. Particularly in the horizontal plane view of FIG. 3, the overlap between the visual fields Ac and Ar is realized such that the camera visual field Ac of the imaging camera 3 is within the radar visual field Ar of the observation radar 4. Hereinafter, for the sake of easy understanding of the description, the set of the imaging camera 3 which is the front camera 30 and the observation radar 4 which is the front radar 40 among the sets shown in FIGS. 1 to 3 will be typically described.

[0017] The imaging camera 3 includes an image sensor unit 300 and an imaging circuit unit 302. The image sensor unit 300 is mainly composed of semiconductor elements such as CMOS, which have multiple pixels arranged in a two-dimensional array. The image sensor unit 300 captures light images received from targets within the camera's field of view Ac, pixel by pixel. The imaging circuit unit 302 is a semiconductor chip, such as an image processing circuit, that processes the imaging signals from each pixel of the image sensor unit 300. The imaging circuit unit 302 outputs image data Dc by converting the brightness values ​​of each pixel according to the light intensity received from targets within the camera's field of view Ac into two-dimensional data.

[0018] The observation radar 4 includes a transmit / receive antenna unit 400 and a transmit / receive circuit unit 402. The transmit / receive antenna unit 400 is mainly composed of an antenna array, such as a microstrip antenna. The transmit / receive circuit unit 402 is composed of an IC chip, such as a DSP (Digital Signal Processor), combined with an RF (Radio Frequency) circuit. The transmitting antenna in the transmit / receive antenna unit 400 generates a transmit wave, such as a millimeter wave, directed into the radar field of view Ar by converting the transmit signal modulated by the transmit / receive circuit unit 402 into a radio wave. The receiving antenna in the transmit / receive antenna unit 400 receives the reflected wave from the transmit wave from a target located within the radar field of view Ar and converts it into a received signal.

[0019] The transmitting and receiving circuit unit 402 mixes the converted received signal with the transmitted signal, then performs FFT (Fast Fourier Transform) analysis and angle measurement processing to output point cloud data Dr representing multiple observation points that observed a target within the radar field of view Ar. The point cloud data Dr output at this time is generated to represent the three-dimensional distance, as well as the vertical angle θv, the horizontal angle θh, and the Doppler velocity Vd for each observation point, as shown in Figure 5 below.

[0020] To enable the generation of such point cloud data Dr, the transmitting and receiving antenna unit 400 may be constructed such that multiple transmitting antennas are arranged in one of the vertical and horizontal directions, and multiple receiving antennas are arranged in the other of the vertical and horizontal directions. The transmitting and receiving antenna unit 400 may be constructed such that multiple transmitting antennas are arranged in both the vertical and horizontal directions, and multiple receiving antennas are arranged in one of the vertical and horizontal directions. The transmitting and receiving antenna unit 400 may be constructed such that multiple transmitting antennas are arranged in one of the vertical and horizontal directions, and multiple receiving antennas are arranged in both the vertical and horizontal directions. The transmitting and receiving antenna unit 400 may be constructed such that multiple transmitting antennas are arranged in both the vertical and horizontal directions, and multiple receiving antennas are also arranged in both the vertical and horizontal directions.

[0021] As shown in Figure 1, the calibration system 1 is configured to include at least one dedicated computer. The calibration system 1 is connected to the imaging camera 3 and the observation radar 4 via at least one of the following: a LAN (Local Area Network) line, a wire harness, an internal bus, and a wireless communication line. If the calibration system 1 consists of multiple dedicated computers, the connections between those dedicated computers are similar.

[0022] The dedicated computer constituting the calibration system 1 may be an electronic control unit (ECU) that controls the operation of the host vehicle 2. The dedicated computer constituting the calibration system 1 may be a navigation ECU that navigates the travel path of the host vehicle 2. The dedicated computer constituting the calibration system 1 may be a locator ECU that estimates the self-state quantities of the host vehicle 2. The dedicated computer constituting the calibration system 1 may be an actuator ECU that controls the travel actuators of the host vehicle 2. The dedicated computer constituting the calibration system 1 may be a human-machine interface (HMI) control unit (HCU) that controls information presentation in the host vehicle 2. The dedicated computer constituting the calibration system 1 may be a computer other than the host vehicle 2 that constructs an external center and / or mobile terminal that can communicate via the host vehicle 2's communication system.

[0023] The dedicated computer comprising the calibration system 1 has at least one memory 10 and one processor 12. The memory 10 is at least one type of non-transitory tangible storage medium, such as semiconductor memory, magnetic media, and optical media, which non-temporarily stores programs and data that can be read by the computer. Here, storage may be an accumulation where data is retained even when the host vehicle 2 is turned off, or it may be a temporary storage where data is erased when the host vehicle 2 is turned off. The processor 12 includes at least one type as a core, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), RISC (Reduced Instruction Set Computer)-CPU, DFP (Data Flow Processor), and GSP (Graph Streaming Processor).

[0024] The processor 12 in the calibration system 1 executes multiple instructions included in the calibration program stored in memory 10 for calibrating the relationship between the imaging camera 3 and the observation radar 4 in the host vehicle 2. This allows the calibration system 1 to construct multiple functional blocks for calibrating the relationship between the imaging camera 3 and the observation radar 4 in the host vehicle 2. The multiple functional blocks constructed in the calibration system 1 include, as shown in Figure 2, a data acquisition block 100, an attitude monitoring block 110, a calibration block 120, a fusion block 130, and a recognition block 140.

[0025] Through the combined efforts of blocks 100, 110, 120, 130, and 140, the calibration method by which the calibration system 1 calibrates between the imaging camera 3 and the observation radar 4 in the host vehicle 2 is performed according to the calibration flow shown in Figure 4. This calibration flow is repeatedly executed for each frame that is periodically controlled for calibration in the host vehicle 2. In this calibration flow, each "S" represents a step executed by multiple instructions included in the calibration program.

[0026] In S10 of Figure 4, the data acquisition block 100 (see Figure 2) acquires image data Dc captured by the imaging camera 3 within the camera field of view Ac, along with point cloud data Dr observed by the observation radar 4 within the radar field of view Ar, in the current frame. At this time, the image data Dc and point cloud data Dr may be acquired if they are generated within an acceptable time difference that is permissible for synchronization on the control time axis. Alternatively, if the generation times of the image data Dc and point cloud data Dr are shifted by more than the permissible time difference, synchronization may be achieved by performing position shift compensation processing corresponding to the movement of the host vehicle 2 so that one of the data Dc or Dr is adjusted to match the other. In addition, in S10 or S40 described later, prior to the bird's-eye view conversion described in detail in S40, image processing such as distortion correction may be performed on the image data Dc.

[0027] As shown in Figure 4, in S20 following S10, the attitude monitoring block 110 (see Figure 2) monitors the change in attitude angle Δψ of the host vehicle 2. Specifically, in S20, the attitude monitoring block 110 fits the correlation distribution α between the vertical angle θv, horizontal angle θh, and Doppler velocity Vd, which are represented by the point cloud data Dr acquired in S10 for each observation point, to a three-dimensional approximation surface as shown in Figure 5. At this time, the observation points used for fitting the correlation distribution α are limited to observation points that represent stationary targets and can be recognized from, for example, the point cloud data Dr of the current and past frames. The attitude monitoring block 110 then extracts the peak point Pp in the approximation surface of the surface-fitted correlation distribution α, where the value of the Doppler velocity Vd peaks as shown in Figure 5. If the extracted peak point Pp deviates from the reference point Pb in Figure 5, the angular deviation that appears in at least one of the vertical angle θv and horizontal angle θh at the peak point Pp relative to the reference point Pb will represent the change in attitude angle Δψ in the host vehicle 2.

[0028] Here, the reference point Pb in S20 is defined as the point where the Doppler velocity Vd, when the vertical angle θv and the horizontal angle θh are zero, is matched with the travel speed of the host vehicle 2. Under this definition of the reference point Pb, the angular deviation appearing at the vertical angle θv of the peak point Pp relative to the reference point Pb is monitored in S20 as the attitude angle change amount Δψp (see equation 4 below), which is equivalent to the pitch angle of the host vehicle 2 from an ideal state with no attitude change. At the same time, under the definition of the reference point Pb, the angular deviation appearing at the horizontal angle θh of the peak point Pp relative to the reference point Pb is monitored in S20 as the attitude angle change amount Δψy (see equation 5 below), which is equivalent to the yaw angle of the host vehicle 2 from an ideal state with no attitude change. However, the amount of attitude angle change Δψr (see equation 6 below) that appears in the roll angle of the host vehicle 2 from the ideal state where there is no change in attitude angle should be monitored based on sensing information from, for example, an inertial sensor or a gyro sensor mounted on the host vehicle 2.

[0029] As shown in Figure 4, in S30 following S20, the calibration block 120 (see Figure 2) constructs a calibration parameter Cp for calibrating the image data Dc, which will be used for fusion with the point cloud data Dr in S30, as described later. The calibration parameter Cp constructed at this time is corrected according to the attitude angle change amount Δψ (i.e., Δψp, Δψy, Δψr) monitored in S20. Specifically, in S30, the calibration block 120 constructs the calibration parameter Cp by performing a matrix multiplication operation between the internal parameter Ci and the external parameter Co according to Equation 1.

number

[0030] Among the calibration parameters Cp constructed by S30, the external parameter Co is a matrix parameter for transforming the three-dimensional orthogonal coordinate system of the road surface on which the host vehicle 2 travels to the three-dimensional orthogonal camera coordinate system of the imaging camera 3 in the host vehicle 2. This external parameter Co is defined to include a rotation matrix component R and a translation matrix component T according to Equation 2. In particular, the rotation matrix component R that constitutes the external parameter Co is expressed by the matrix product of the yaw rotation component Ry, the pitch rotation component Rp, and the roll rotation component Rr in the host vehicle 2, according to Equation 3. The three-dimensional orthogonal coordinate system of the road surface that serves as the basis for setting the external parameter Co is defined by the three axes along the pitch axis, yaw axis, and roll axis of the host vehicle 2, but it may also be defined using the world coordinate system of the road surface.

number

number

[0031] Therefore, in the construction of the calibration parameter Cp using S30, the pitch rotation component Rp, which constitutes the matrix product of the rotation matrix components R, is corrected by the attitude angle change amount Δψp according to Equation 4. At the same time, the yaw rotation component Ry, which constitutes the matrix product of the rotation matrix components R, is corrected by the attitude angle change amount Δψy according to Equation 5. Furthermore, the roll rotation component Rr, which constitutes the matrix product of the rotation matrix components R, is corrected by the attitude angle change amount Δψr according to Equation 6.

number

number

number

[0032] Of the calibration parameters Cp constructed by S30, the internal parameter Ci is a matrix parameter for transforming the three-dimensional orthogonal coordinate system of the imaging camera 3 in the host vehicle 2 to the two-dimensional orthogonal coordinate system of the image data Dc. The calibration parameter Cp constructed from this internal parameter Ci and the external parameter Co described above can be used in the form of an inverse matrix to project the image data Dc to a bird's-eye view from above relative to the host vehicle 2.

[0033] As shown in Figure 4, in S40 following S30, the fusion block 130 (see Figure 2) generates fusion data Df by fusing image data Dc with point cloud data Dr. Specifically, in S40, the fusion block 130 calibrates the image data Dc acquired in S10 according to the calibration parameter Cp constructed in S30, and simultaneously performs a bird's-eye view transformation. At the same time, the fusion block 130 in S40 transforms the coordinates of the point cloud data Dr acquired in S10 to match the image data Dc after the bird's-eye view transformation. Thus, in S40, fusion data Df is generated by fusion, which combines the calibrated and bird's-eye view transformed image data Dc with the coordinate-transformed point cloud data Dr through three-dimensional matching.

[0034] In S40, prior to the generation of fusion data Df, the fusion block 130 may extract feature data Dis for each of the 5 feature quantities from the image data Dc acquired in S10, as shown in Figure 6. In this case, feature quantity 5 is extracted using, for example, a machine learning model, to represent the characteristic distribution of pixel brightness values ​​that appear in the image data Dc corresponding to targets within the camera field of view Ac. The feature data Dis extracted for each of the 5 feature quantities in this way is then fused in S40 with the point cloud data Dr, which has undergone coordinate transformation after calibration and bird's-eye view transformation as described above, to synthesize each of the 5 feature quantity fusion data Df individually.

[0035] As shown in Figure 4, in S50 following S40, the recognition block 140 (see Figure 2) recognizes a target in the fusion data Df and outputs recognition data Do. Specifically, in S50, the recognition block 140 performs recognition processing such as clustering and model matching on the fusion data Df fused in S40 to generate recognition data Do to be output in the current frame. At this time, if fusion data Df for each of the 5 features has been generated in S40, it is desirable that the recognition data Do in the current frame be generated in such a way that it collectively represents the targets recognized for each of those fusion data Df.

[0036] The recognition data Do output in S50 is stored in memory 10 and used for driving control in the host vehicle 2. The recognition data Do output in S50 may also be transmitted to, for example, an external center and / or a mobile terminal via a communication unit installed in the host vehicle 2. Once the execution of S50 is completed, the current execution of the calibration flow is finished.

[0037] (Effects and Benefits) The effects and advantages of the first embodiment described above will be explained below.

[0038] According to the first embodiment, along with image data Dc captured by the imaging camera 3 capturing the camera field of view Ac, point cloud data Dr is obtained by the observation radar 4 observing the radar field of view Ar superimposed on the camera field of view Ac. Here, with respect to the reference point Pb of the point cloud data Dr, if the peak point Pp in the correlation distribution α between the vertical angle θv, horizontal angle θh, and Doppler velocity Vd represented by the point cloud data Dr for each observation point shifts, an attitude angle change amount Δψ corresponding to this shift appears in the host vehicle 2. Therefore, the attitude angle change amount Δψ of the host vehicle 2 can be properly monitored based on the shift in the peak point of the correlation distribution α with respect to the reference point Pb. Accordingly, as a parameter for calibrating the image data Dc in order to fuse it with the point cloud data Dr, it is possible to output a calibration parameter Cp that can be accurately corrected according to the properly monitored attitude angle change amount Δψ even while the host vehicle 2 is in motion.

[0039] According to the first embodiment, the angular deviation appearing at the peak point Pp relative to the reference point Pb in at least one of the vertical angle θv and horizontal angle θh is monitored as the attitude angle change amount Δψ. This allows for accurate determination of the angular deviation of the peak point Pp relative to the reference point Pb from the vertical angle θv and horizontal angle θh that constitute the correlation distribution α, and enables proper monitoring as the attitude angle change amount Δψ. Therefore, even while the host vehicle 2 is in motion, it is possible to accurately correct and output the calibration parameter Cp according to the attitude angle change amount Δψ.

[0040] According to the first embodiment, the rotation matrix component R of the external parameter Co defined in the imaging camera 3, which is part of the calibration parameter Cp, is corrected according to the attitude angle change amount Δψ. Such correction makes it possible to output a calibration parameter Cp that accurately reflects the attitude angle change amount Δψ.

[0041] According to the first embodiment, image data Dc, which has been accurately calibrated according to the calibration parameter Cp, is fused with point cloud data Dr to generate fusion data Df. Therefore, by recognizing the target in the thus generated fusion data Df, it becomes possible to output recognition data Do with high recognition accuracy.

[0042] According to the first embodiment, image data Dc, which has been converted to a bird's-eye view according to the calibration parameter Cp constructed from the external parameter Co and internal parameter Ci defined in the imaging camera 3, is fused with point cloud data Dr to generate fusion data Df. This makes it possible to improve the accuracy of the fusion process between image data Dc and point cloud data Dr, which have different dimensions, and consequently, the accuracy of target recognition in the fusion data Df that has undergone the fusion process.

[0043] According to the first embodiment, multiple feature data Dis, extracted from image data Dc according to feature quantity 5 and converted to a bird's-eye view, may be fused with point cloud data Dr to generate fusion data Df for each of the feature quantity 5. Therefore, in this case, recognition data Do, which is output to represent the recognized target for each of the generated fusion data Df for each of the feature quantity 5, can ensure high accuracy in the recognition.

[0044] (Second embodiment) The second embodiment is a modification of the first embodiment. In the calibration flow of the second embodiment shown in Figure 7, after the execution of S10, which is the same as in the first embodiment, S220 and S230 are executed instead of S20 and S30 in the first embodiment, respectively.

[0045] In S220, the attitude monitoring block 110 sets the monitored attitude angle change amount Δψ as the current monitored value Δψcm in the current frame, similar to S20 in the first embodiment. Then, in S220, the attitude monitoring block 110 estimates the current estimated value Δψce of the attitude angle change amount Δψ in the current frame from the current monitored value Δψcm in the current frame and the past estimated value Δψpe of the attitude angle change amount Δψ in past frames. At this time, the past estimated value Δψpe of the attitude angle change amount Δψ is used by first storing the current estimated value Δψce at the time of the previous or earlier execution of the calibration flow in memory 10 and then reading it out. Furthermore, the estimation calculation of the current estimated value Δψce from the past estimated value Δψpe and the current monitored value Δψcm may be performed by, for example, a time-moving average filter or a Kalman filter.

[0046] In the subsequent S230, the calibration block 120 corrects the calibration parameter Cp to be constructed for the current frame according to the currently estimated value Δψce of the attitude angle change amount Δψ estimated in S220. That is, in S230, the calibration parameter Cp is constructed that has been corrected in accordance with S30 of the first embodiment, except that the currently estimated value Δψce is used instead of the attitude angle change amount Δψ in the first embodiment which corresponds to the currently monitored value Δψcm. After S230, in the second embodiment, S40 and S50 are executed in the same manner as in the first embodiment.

[0047] Thus, in this second embodiment, the current estimated value Δψce of the attitude angle change Δψ in the current frame is estimated from the currently monitored value Δψcm of the attitude angle change Δψ monitored in the current frame and the past estimated value Δψpe of the attitude angle change Δψ estimated in past frames. According to this, it is possible to accurately correct the calibration parameter Cp according to the current estimated value Δψce, which can be estimated with higher accuracy than the currently monitored value Δψcm, which is susceptible to noise in relation to the attitude angle change Δψ. Moreover, by using fusion data Df, which is fused with point cloud data Dr, to image data Dc calibrated according to the accurate calibration parameter Cp, it is possible to improve the target recognition accuracy. (Other embodiments) Although several embodiments have been described above, this disclosure is not limited to those embodiments and can be applied to various embodiments and combinations without departing from the spirit of this disclosure.

[0048] In the modified example, the dedicated computer constituting the calibration system 1 may have at least one of the digital circuit and the analog circuit as a processor. Here, the digital circuit is at least one of the following, for example, ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). Such a digital circuit may also have a memory that stores a program.

[0049] In the modified configurations shown in Figures 1-3, the calibration technique of the first or second embodiment may be applied to the set of rear camera 31 (imaging camera 3) and rear radar 41 (observation radar 4). In the modified configurations shown in Figures 1-3, the calibration technique of the first or second embodiment may be applied to the set of left camera 32 (imaging camera 3) and left radar 42 (observation radar 4). In the modified configurations shown in Figures 1-3, the calibration technique of the first or second embodiment may be applied to the set of right camera 33 (imaging camera 3) and right radar 43 (observation radar 4). However, the angular deviation appearing in the vertical angle θv at S20 and S220, in the case of application to the set of left camera 32 and left radar 42 and in the case of application to the set of right camera 33 and right radar 43, should be monitored as the attitude angle change amount Δψr appearing in the roll angle of the host vehicle 2. In addition, the attitude angle change Δψp that appears in the pitch angle in S20 and S220, in the case of application to the left camera 32 and left radar 42 set and in the case of application to the right camera 33 and right radar 43 set, should be monitored based on sensing information from, for example, an inertial sensor or a gyro sensor.

[0050] In the modified examples, the reference point Pb in S20 and S220 may be defined in a way other than the point where the Doppler velocity Vd, when the vertical angle θv and horizontal angle θh are zero, is matched with the travel speed of the host vehicle 2. In addition to the forms described so far, the host vehicle 2 to which the calibration system 1 is applied in the above-described embodiments and modified examples may be, for example, an autonomous robot capable of transporting goods or collecting information by autonomous or remote driving.

[0051] (Additional note) This specification discloses several technical concepts and several combinations thereof, as listed below. The symbols in parentheses in this supplementary section indicate correspondences with the specific means described in the embodiments detailed above, and do not limit the technical scope of this disclosure.

[0052] (Technical thought 1) A calibration system having a processor (12) for calibrating between an imaging camera (3) and an observation radar (4) in a host vehicle (2), The aforementioned processor, The acquisition of, along with image data (Dc) captured by the imaging camera, including the camera field of view (Ac) of the imaging camera, and point cloud data (Dr) observed by the observation radar, including the radar field of view (Ar) that overlaps with the camera field of view of the observation radar, is to be obtained. The change in attitude angle (Δψ) of the host vehicle, which occurs when the peak point (Pp) in the correlation distribution (α) between the vertical angle (θv), horizontal angle (θh), and Doppler velocity (Vd) represented by the point cloud data for each observation point shifts relative to the reference point (Pb) of the point cloud data, is monitored. A calibration system configured to perform the following actions: construct a calibration parameter (Cp) corrected according to the amount of change in attitude angle, as a parameter for calibrating the image data to be used for fusion with the point cloud data.

[0053] (Technical thought 2) The aforementioned monitoring of attitude angle change is performed by A calibration system according to technical concept 1, which includes monitoring the angular deviation appearing in at least one of the vertical angle and the horizontal angle at the peak point relative to the reference point as the attitude angle change amount.

[0054] (Technical Thought 3) The construction of the aforementioned calibration parameters is A calibration system according to technical concept 1 or 2, which includes correcting the rotation matrix component (R) of the external parameter (Co) defined for the imaging camera, among the calibration parameters, in accordance with the amount of change in attitude angle.

[0055] (Technical Thought 4) The aforementioned monitoring of attitude angle change is performed by This includes estimating the current estimated value (Δψce) of the attitude angle change in the current frame from the currently monitored value (Δψcm) of the attitude angle change observed in the current frame and the past estimated value (Δψpe) of the attitude angle change estimated in past frames. The construction of the aforementioned calibration parameters is A calibration system according to any one of the technical ideas 1 to 3, which includes constructing the calibration parameters corrected according to the current estimated values.

[0056] (Technical Thought 5) The aforementioned processor, The image data, which has been calibrated according to the aforementioned calibration parameters, is fused with the point cloud data to generate fusion data (Df). A calibration system according to any one of the technical concepts 1 to 4, further configured to output recognition data (Do) by recognizing a target in the fusion data.

[0057] (Technical Thought 6) The generation of the aforementioned fusion data is A calibration system according to technical concept 5, which includes fusing the image data converted to a bird's-eye view according to the calibration parameters constructed from the external parameters (Co) and internal parameters (Ci) defined in the imaging camera with the point cloud data to generate the fusion data.

[0058] (Technical Thought 7) The generation of the aforementioned fusion data is This includes generating fusion data for each feature quantity by fusing each of the multiple feature data (Dis) extracted from the image data according to feature quantity (5) and converted to the bird's-eye view with the point cloud data, The output of the aforementioned recognition data is: A calibration system according to technical concept 6, which includes outputting the recognition data so as to represent the recognized target for each of the fusion data for each of the features.

[0059] Furthermore, the technical concepts 1 to 7 described above may also be understood within the technical concepts of the apparatus, method, and program, respectively. [Explanation of Symbols]

[0060] 1: Calibration system, 2: Host vehicle, 3: Imaging camera, 4: Observation radar, 5: Features, 10: Memory, 12: Processor, Ac: Camera field of view, Ar: Radar field of view, Ci: Internal parameters, Co: External parameters, Cp: Calibration parameters, Dc: Image data, Df: Fusion data, Dis: Feature data, Do: Recognition data, Dr: Point cloud data, Pb: Reference point, Pp: Peak point, R: Rotation matrix component, Vd: Doppler velocity, Δψ: Attitude angle change, Δψce: Current estimate, Δψcm: Current monitored value, Δψpe: Past estimate, α: Correlation distribution, θh: Horizontal angle, θv: Vertical angle

Claims

1. A calibration system having a processor (12) for calibrating between an imaging camera (3) and an observation radar (4) in a host vehicle (2), The aforementioned processor, The acquisition of, along with image data (Dc) captured by the imaging camera, including the camera field of view (Ac) of the imaging camera, and point cloud data (Dr) observed by the observation radar, including the radar field of view (Ar) that overlaps with the camera field of view of the observation radar, is to be obtained. The change in attitude angle (Δψ) of the host vehicle, which occurs when the peak point (Pp) in the correlation distribution (α) between the vertical angle (θv), horizontal angle (θh), and Doppler velocity (Vd) represented by the point cloud data for each observation point shifts relative to the reference point (Pb) of the point cloud data, is monitored. A calibration system configured to perform the following actions: construct a calibration parameter (Cp) corrected according to the amount of change in attitude angle, as a parameter for calibrating the image data to be used for fusion with the point cloud data.

2. The aforementioned monitoring of attitude angle change is performed by The calibration system according to claim 1, comprising monitoring the angular deviation appearing at least one of the vertical angle and the horizontal angle at the peak point relative to the reference point as the attitude angle change amount.

3. The construction of the aforementioned calibration parameters is The calibration system according to claim 1, further comprising correcting the rotation matrix component (R) of the external parameter (Co) defined for the imaging camera, among the calibration parameters, in accordance with the amount of change in attitude angle.

4. The aforementioned monitoring of attitude angle change is performed by This includes estimating the current estimated value of the attitude angle change in the current frame (Δψce) from the currently monitored value (Δψcm) of the attitude angle change observed in the current frame and the past estimated value (Δψpe) of the attitude angle change estimated in past frames, The construction of the aforementioned calibration parameters is The calibration system according to claim 1, further comprising constructing the calibration parameters corrected according to the current estimated values.

5. The aforementioned processor, The image data, which has been calibrated according to the aforementioned calibration parameters, is fused with the point cloud data to generate fusion data (Df). The calibration system according to any one of claims 1 to 4, further configured to output recognition data (Do) by recognizing a target in the fusion data.

6. The generation of the aforementioned fusion data is The calibration system according to claim 5, further comprising fusing the image data converted to a bird's-eye view according to the calibration parameters constructed from the external parameters (Co) and internal parameters (Ci) defined in the imaging camera with the point cloud data to generate the fusion data.

7. The generation of the aforementioned fusion data is This includes generating fusion data for each feature quantity by fusing each of the multiple feature data (Dis) extracted from the image data according to feature quantity (5) and converted to the bird's-eye view with the point cloud data, The output of the aforementioned recognition data is: The calibration system according to claim 6, further comprising outputting the recognition data so as to represent the recognized target for each of the fusion data for each of the features.

8. A calibration device having a processor (12) for calibrating between an imaging camera (3) and an observation radar (4) in a host vehicle (2), and configured to be mounted on the host vehicle, The aforementioned processor, The acquisition of, along with image data (Dc) captured by the imaging camera, including the camera field of view (Ac) of the imaging camera, and point cloud data (Dr) observed by the observation radar, including the radar field of view (Ar) that overlaps with the camera field of view of the observation radar, is to be obtained. The change in attitude angle (Δψ) of the host vehicle, which occurs when the peak point (Pp) in the correlation distribution (α) between the vertical angle (θv), horizontal angle (θh), and Doppler velocity (Vd) represented by the point cloud data for each observation point shifts relative to the reference point (Pb) of the point cloud data, is monitored. A calibration device configured to perform the following actions: construct a calibration parameter (Cp) corrected according to the amount of change in attitude angle, as a parameter for calibrating the image data to be used for fusion with the point cloud data.

9. A calibration method performed by a processor (12) to calibrate the relationship between an imaging camera (3) and an observation radar (4) in a host vehicle (2), The acquisition of, along with image data (Dc) captured by the imaging camera, including the camera field of view (Ac) of the imaging camera, and point cloud data (Dr) observed by the observation radar, including the radar field of view (Ar) that overlaps with the camera field of view of the observation radar, is to be obtained. The change in attitude angle (Δψ) of the host vehicle, which occurs when the peak point (Pp) in the correlation distribution (α) between the vertical angle (θv), horizontal angle (θh), and Doppler velocity (Vd) represented by the point cloud data for each observation point shifts relative to the reference point (Pb) of the point cloud data, is monitored. A calibration method comprising constructing a calibration parameter (Cp) corrected according to the amount of change in attitude angle, as a parameter for calibrating the image data to be used for fusion with the point cloud data.

10. A calibration program stored in a storage medium (10) for the purpose of calibrating between an imaging camera (3) and an observation radar (4) in a host vehicle (2), and which includes instructions for causing a processor (12) to perform said calibration, The acquisition of, along with image data (Dc) captured by the imaging camera, including the camera field of view (Ac) of the imaging camera, and point cloud data (Dr) observed by the observation radar, including the radar field of view (Ar) that overlaps with the camera field of view of the observation radar, is to be obtained. The change in attitude angle (Δψ) of the host vehicle, which occurs when the peak point (Pp) in the correlation distribution (α) between the vertical angle (θv), horizontal angle (θh), and Doppler velocity (Vd) represented by the point cloud data for each observation point shifts relative to the reference point (Pb) of the point cloud data, is monitored. A calibration program including the command to construct a calibration parameter (Cp) corrected according to the amount of change in attitude angle, as a parameter for calibrating the image data to be used for fusion with the point cloud data.

Images