Calibration system, calibration device, calibration method, calibration program

Abstract

We provide a calibration system that accurately calibrates multiple cameras. [Solution] The processor of the calibration system for a first camera whose first field of view extends including the front-rear axis and a second camera whose second field of view extends including the left-right axis, acquires first image data Di1 of the first field of view captured by the first camera, along with second image data Di2 of the second field of view captured by the second camera, which has a superimposed area that partially superimposes on the first field of view; monitors the driving scene in which the road pattern of interest P is reflected in the superimposed area in both the first image data Di1 and the second image data Di2; and outputs calibration data Dc to calibrate between the first camera and the second camera in accordance with the amount of displacement ΔP between the road patterns of interest reflected in the superimposed area in each of the first image data Di1 and the second image data Di2, as the amount of displacement ΔP between the road patterns of interest reflected in the superimposed area increases outside the allowable range ωP.

Description

Technical Field

[0001] The present disclosure relates to calibration technology for calibrating a camera in a vehicle.

Background Art

[0002] The technology disclosed in Patent Document 1 obtains a vanishing line from feature points of road markings reflected in an image captured by a camera in a vehicle, and calibrates camera parameters based on the vanishing line.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when the technology disclosed in Patent Document 1 is applied to a plurality of cameras in a vehicle, an individual vanishing line is obtained for each captured image by each camera. As a result, between a camera whose field of view spreads on the front and rear axes of the vehicle and a camera whose field of view spreads on the left and right axes of the host vehicle outside the field of view on the front and rear axes, due to the difference in the reference vanishing lines, it becomes difficult to accurately calibrate the camera parameters with each other.

[0005] An object of the present disclosure is to provide a calibration system that accurately calibrates between a plurality of cameras. Another object of the present disclosure is to provide a calibration device that accurately calibrates between a plurality of cameras. Still another object of the present disclosure is to provide a calibration method that accurately calibrates between a plurality of cameras in a field of view. Yet another object of the present disclosure is to provide a calibration program that accurately calibrates between a plurality of cameras in a field of view. [Means for solving the problem]

[0006] The following describes the technical means of solving the problem described in this disclosure. Note that the claims and the reference numerals in parentheses in this section indicate the correspondence with the specific means described in the embodiments detailed later, and do not limit the technical scope of this disclosure.

[0007] The first aspect of this disclosure is, A calibration system having a processor (12) for calibrating a first camera (31) whose first field of view (Av1) extends along the longitudinal axis (X) of a host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the lateral axis (Y) of the host vehicle, The processor is, The system acquires first image data (Di1) of the first field of view captured by the first camera, and second image data (Di2) of the second field of view captured by the second camera, which has a superimposed area (Avo) that is partially superimposed on the first field of view. In both the first and second image data, the driving scene in which the road pattern of interest (P) appears in the superimposed area is monitored, The system is configured to output calibration data (Dc) to calibrate the relationship between the first and second cameras as the amount of misalignment (ΔP) between the target road patterns projected in the superimposed area of ​​each of the first and second image data increases beyond the acceptable range (ωP).

[0008] A second aspect of this disclosure is, A calibration device having a processor (12) for calibrating a first camera (31) whose first field of view (Av1) extends along the longitudinal axis (X) of the host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the lateral axis (Y) of the host vehicle, and configured to be mounted on the host vehicle, The processor is, The system acquires first image data (Di1) of the first field of view captured by the first camera, and second image data (Di2) of the second field of view captured by the second camera, which has a superimposed area (Avo) that is partially superimposed on the first field of view. In both the first and second image data, the driving scene in which the road pattern of interest (P) appears in the superimposed area is monitored, The system is configured to output calibration data (Dc) to calibrate the relationship between the first and second cameras as the amount of misalignment (ΔP) between the target road patterns projected in the superimposed area of ​​each of the first and second image data increases beyond the acceptable range (ωP).

[0009] A third aspect of this disclosure is: A calibration method performed by a processor (12) to calibrate a first camera (31) whose first field of view (Av1) extends along the longitudinal axis (X) of a host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the lateral axis (Y) of the host vehicle, wherein The system acquires first image data (Di1) of the first field of view captured by the first camera, and second image data (Di2) of the second field of view captured by the second camera, which has a superimposed area (Avo) that is partially superimposed on the first field of view. In both the first and second image data, the driving scene in which the road pattern of interest (P) appears in the superimposed area is monitored, This includes outputting calibration data (Dc) to calibrate the relationship between the first and second cameras in response to the amount of misalignment (ΔP) between the road patterns of interest projected in the superimposed area of ​​each of the first and second image data, which increases beyond the acceptable range (ωP).

[0010] The fourth aspect of this disclosure is: A calibration program is stored in a storage medium (10) for the purpose of calibrating a first camera (31) whose first field of view (Av1) extends along the front-rear axis (X) of a host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the left-right axis (Y) of the host vehicle, and includes instructions for causing a processor (12) to perform said calibration, The system acquires first image data (Di1) of the first field of view captured by the first camera, and second image data (Di2) of the second field of view captured by the second camera, which has a superimposed area (Avo) that is partially superimposed on the first field of view. In both the first and second image data, the driving scene in which the road pattern of interest (P) appears in the superimposed area is monitored, The command includes instructions to output calibration data (Dc) to calibrate the relationship between the first and second cameras in response to the amount of misalignment (ΔP) between the road patterns of interest projected in the superimposed area of ​​each of the first and second image data increasing outside the acceptable range (ωP).

[0011] Thus, the first to fourth embodiments calibrate a first camera whose first field of view extends to include the front and rear axes of the host vehicle, and a second camera whose second field of view extends to include the left and right axes of the host vehicle. For this purpose, first image data is acquired by the first camera, capturing the first field of view, and second image data is acquired by the second camera, capturing the second field of view, which has a superimposed area that partially overlaps the first field of view.

[0012] Therefore, according to the first to fourth aspects, in both the first image data and the second image data, the driving scene in which the on-road pattern appears in the overlapping area is monitored. As a result of the monitoring, when the deviation amount between the on-road patterns appearing in the overlapping area in each of the first image data and the second image data increases beyond the allowable range, it can be said that calibration between the first camera and the second camera is necessary. Therefore, if calibration data is output according to the deviation amount between the on-road patterns common to the first camera and the second camera, it is possible to accurately calibrate between the first camera and the second camera.

Brief Description of the Drawings

[0013] [Figure 1] It is a block diagram showing the overall configuration of a calibration system according to an embodiment. [Figure 2] It is a block diagram for explaining the functional configuration of a calibration system according to an embodiment. [Figure 3] It is a bird's-eye view showing the fields of view of a plurality of cameras according to an embodiment. [Figure 4] It is a flowchart showing the calibration flow of an embodiment. [Figure 5] It is a schematic diagram for explaining the calibration flow of an embodiment. [Figure 6] It is a bird's-eye view for explaining the calibration flow of an embodiment. [Figure 7] It is a schematic diagram for explaining the calibration flow of an embodiment. [Figure 8] It is a schematic diagram for explaining the calibration flow of an embodiment. [Figure 9] It is a schematic diagram for explaining the calibration flow of an embodiment. [Figure 10] It is a schematic diagram for explaining the calibration flow of an embodiment. [Figure 11] It is a bird's-eye view for explaining the calibration flow of an embodiment. [Figure 12] It is a bird's-eye view for explaining the calibration flow of one embodiment. [Figure 13] It is a bird's-eye view for explaining the calibration flow of one embodiment. [Figure 14] It is a graph for explaining the calibration flow of one embodiment. [Figure 15] It is a graph for explaining the calibration flow of one embodiment. [Figure 16] It is a graph for explaining the calibration flow of one embodiment.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, one embodiment of the present disclosure will be described based on the drawings.

[0015] The calibration system 1 of one embodiment shown in FIGS. 1 and 2 calibrates between a plurality of cameras 3 in the host vehicle 2. The host vehicle 2 can be said to be an ego-vehicle from the perspective centered on the vehicle itself. The host vehicle 2 is a moving body such as an automobile that can travel on a road in the state where a passenger is on board. Therefore, the directions in the following description are defined based on the host vehicle 2 on the horizontal plane. Note that FIGS. 1 and 2 typically show an example in which the entire calibration system 1 is configured to be mounted on the host vehicle 2 as an example implemented in the form of a calibration device such as a processing circuit (for example, a processing ECU or the like) or a semiconductor unit (for example, a semiconductor chip or the like).

[0016] In the host vehicle 2, an automated driving mode is provided, which is categorized into levels according to the degree of manual intervention by the occupant in dynamic driving tasks. The automated driving mode may be implemented by autonomous driving control, such as conditional driving automation, highly automated driving, or fully automated driving, in which the system performs all dynamic driving tasks when in operation. The automated driving mode may also be implemented by advanced driver assistance control, such as driver assistance or partial driving automation, in which the occupant performs some or all of the dynamic driving tasks. The automated driving mode may be implemented by either one of these autonomous driving controls or advanced driver assistance controls, in combination, or by switching between them.

[0017] The host vehicle 2 is equipped with multiple cameras 3 that are subject to mutual calibration. Each camera 3 consists of an image sensor 300 and an imaging circuit 302. The image sensor 300 is a semiconductor element such as a CMOS, having multiple pixels arranged in two dimensions. The image sensor 300 captures a light image received from a target within the field of view Av (see Figure 3 below) pixel by pixel. The imaging circuit 302 is a semiconductor chip such as an image processing circuit that processes the imaging signals from each pixel of the image sensor 300. The imaging circuit 302 outputs image data Di by converting the brightness values ​​of each pixel according to the light reception intensity of the light image from within the field of view Av into two-dimensional data.

[0018] As shown in Figure 3, among the multiple cameras 3, the front camera 3a and the rear camera 3b constitute a set of first cameras 31 whose first field of view Av1 extends along the longitudinal axis X, which coincides with or is parallel to the roll axis of the host vehicle 2. Among the multiple cameras 3, the left camera 3c and the right camera 3d constitute a set of second cameras 32 whose second field of view Av2 extends along the left-right axis Y, which coincides with or is parallel to the pitch axis of the host vehicle 2.

[0019] The second field of view Av2 of the left camera 3c, which is a component of the second camera 32, is partially superimposed on the first field of view Av1 of the front camera 3a, which is a component of the first camera 31, thereby setting the superimposed area Avo. The second field of view Av2 of the left camera 3c is also partially superimposed on the first field of view Av1 of the rear camera 3b, which is a component of the first camera 31 that is different from the front camera 3a, thereby setting a different superimposed area Avo from that of the front camera 3a.

[0020] The second field of view Av2 of the right camera 3d, which is a separate element from the left camera 3c within the second camera 32, is partially superimposed on the first field of view Av1 of the front camera 3a, which is a separate element from the first camera 31, thereby setting an overlap area Avo. The second field of view Av2 of the right camera 3d is also partially superimposed on the first field of view Av1 of the rear camera 3b, which is a separate element from the front camera 3a within the first camera 31, thereby setting an overlap area Avo separate from that of the front camera 3a.

[0021] As shown in Figure 1, the calibration system 1 is configured to include at least one dedicated computer. The calibration system 1 is connected to each camera 3 via at least one of the following: a LAN (Local Area Network) line, a wire harness, an internal bus, and a wireless communication line. If the calibration system 1 consists of multiple dedicated computers, the connections between those dedicated computers are similar.

[0022] The dedicated computer constituting the calibration system 1 may be an electronic control unit (ECU) that controls the operation of the host vehicle 2. The dedicated computer constituting the calibration system 1 may be a navigation ECU that navigates the driving path of the host vehicle 2. The dedicated computer constituting the calibration system 1 may be a locator ECU that estimates the self-state quantities of the host vehicle 2. The dedicated computer constituting the calibration system 1 may be an actuator ECU that controls the driving actuators of the host vehicle 2. The dedicated computer constituting the calibration system 1 may be a human-machine interface (HMI) control unit (HCU) that controls information presentation in the host vehicle 2. The dedicated computer constituting the calibration system 1 may be a computer other than the host vehicle 2 that constructs an external center or mobile terminal that can communicate via the host vehicle 2's communication system.

[0023] The dedicated computer comprising the calibration system 1 has at least one memory 10 and one processor 12. The memory 10 is at least one type of non-transitory tangible storage medium, such as semiconductor memory, magnetic media, and optical media, which non-temporarily stores programs and data that can be read by the computer. Here, storage may be an accumulation where data is retained even when the host vehicle 2 is turned off, or it may be a temporary storage where data is erased when the host vehicle 2 is turned off. The processor 12 includes at least one type as a core, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), RISC (Reduced Instruction Set Computer)-CPU, DFP (Data Flow Processor), and GSP (Graph Streaming Processor).

[0024] In the calibration system 1, the processor 12 executes multiple instructions contained in the calibration program stored in memory 10 to calibrate the multiple cameras 3 in the host vehicle 2. This causes the calibration system 1 to construct multiple functional blocks for calibrating the multiple cameras 3 in the host vehicle 2. The multiple functional blocks constructed in the calibration system 1 include a data acquisition block 100, a scene monitoring block 110, and a data output block 120, as shown in Figure 2.

[0025] Through the combined efforts of blocks 100, 110, and 120, the calibration method by which the calibration system 1 calibrates between multiple cameras 3 in the host vehicle 2 is performed according to the calibration flow shown in Figure 4. This calibration flow is repeated whenever the host vehicle 2 determines that calibration is necessary. In this calibration flow, each "S" represents a step executed by multiple instructions included in the calibration program.

[0026] As shown in Figure 4, in S10, the data acquisition block 100 acquires image data Di from all cameras 3 on the front, rear, left, and right sides of the host vehicle 2, capturing their respective fields of view Av. Specifically, in S10, the data acquisition block 100 individually acquires first image data Di1 (see Figures 1 and 2) from the front camera 3a and rear camera 3b, which are a set of first cameras 31, capturing their corresponding first fields of view Av1. At the same time, in S10, the data acquisition block 100 individually acquires second image data Di2 (see Figures 1 and 2) from the left camera 3c and right camera 3d, which are a set of second cameras 32, capturing their corresponding second fields of view Av2.

[0027] In S10, the data acquisition block 100 may acquire image data Di from each camera 3 with substantially synchronized imaging timing. Alternatively, in S10, the data acquisition block 100 may compensate for positional displacement corresponding to the movement of the host vehicle 2 between image data Di from each camera 3 with asynchronous imaging timing. For such positional displacement compensation processing, the amount of positional displacement may be recognized based, for example, on image data Di from past frames in past executions of the calibration flow, and / or sensing data from the speed sensor in the host vehicle 2.

[0028] In S10, the data acquisition block 100 converts the image data Di from each camera 3 into a bird's-eye view from above relative to the host vehicle 2, as shown in Figure 5. At this time, the image data Di from each camera 3 is subjected to viewpoint transformation processing by transformation parameters that transform the orthogonal camera coordinate system defined based on the optical axis of each camera 3 (see L1 and L2 in Figure 6, described later) into an orthogonal bird's-eye view coordinate system defined based on the attitude axes of the host vehicle 2 (i.e., roll axis, pitch axis, and yaw axis).

[0029] In S10, prior to the viewpoint transformation process, the data acquisition block 100f calibrates the image data Di captured by each camera 3 based on the calibration data Dc (described in detail later) output by S40 in Figure 4 in the previously executed calibration flow. In S30, in addition to this calibration process and the positional shift compensation process described above, image processing such as distortion correction may be performed on the image data Di captured by each camera 3 prior to the viewpoint transformation process.

[0030] As shown in Figure 4, in S20 following S10, the scene monitoring block 110 monitors driving scenes in which the road pattern of interest P is visible in at least one of the superimposed areas Avo that partially overlap with the field of view Av of another camera 3 within the field of view Av of each camera 3. At this time, monitoring of driving scenes in which the road pattern of interest P is visible is performed on each bird's-eye view image data Di acquired in S10 by capturing images of the field of view Av including the superimposed area Avo of each camera 3. In addition to the image data Di, sensing data from sensors other than each camera 3, such as steering sensors in the host vehicle 2, may also be used for monitoring driving scenes in which the road pattern of interest P is visible.

[0031] Specifically, in S20, the scene monitoring block 110 determines whether the road pattern P (see Figure 10 described later) is visible in the superposition area Avo of both the corresponding first image data Di1 and second image data Di2 for the front camera 3a of the first camera 31 and the left camera 3c of the second camera 32. In S20, the scene monitoring block 110 determines whether the road pattern P (see Figure 9 described later) is visible in the superposition area Avo of both the corresponding first image data Di1 and second image data Di2 for the rear camera 3b of the first camera 31 and the left camera 3c of the second camera 32.

[0032] In S20, the scene monitoring block 110 determines whether the road pattern P (see Figure 7 below) is visible in the overlapping area Avo of the corresponding first image data Di1 and second image data Di2 for the front camera 3a of the first camera 31 and the right camera 3d of the second camera 32. In S20, the scene monitoring block 110 determines whether the road pattern P (see Figure 8 below) is visible in the overlapping area Avo of the corresponding first image data Di1 and second image data Di2 for the rear camera 3b of the first camera 31 and the right camera 3d of the second camera 32.

[0033] In monitoring driving scenes using S20, the road pattern of interest P that appears in the superimposed area Avo of the first image data Di1 and the second image data Di2 refers to the road pattern of interest that appears in driving scenes effective for calibration. Therefore, the road pattern of interest P includes at least the structural pattern Pr of the road surface 4, as shown in Figures 6-10. Here, the structural pattern Pr is preferably a lane marking pattern in which, in a bird's-eye view, at least one of the inclined lane markings 40 on the road surface 4 extends to the area Avo with respect to either the optical axes L1 or L2 of the first field of view Av1 and the second field of view Av2 that give rise to the superimposed area Avo. Driving scenes in which such a lane marking pattern appears in the superimposed area Avo include, for example, scenes of turning right or left at an intersection, or scenes of driving around a curve on a curved road. Furthermore, the structural pattern Pr may be a lane line pattern in which lane lines 40 other than those inclined with respect to the optical axis L1 or L2 are projected onto the superimposed area Avo, or it may be a marking pattern in which markings other than lane lines 40 are projected onto the superimposed area Avo.

[0034] The road surface pattern P may be anything other than the structural pattern Pr of the road surface 4, but may also be the three-dimensional pattern Po of three-dimensional objects 5 around the road surface 4, as shown in Figure 11. The road surface pattern P may also be the shading pattern Ps of three-dimensional objects 5 that appear on the road surface 4, as shown in Figure 12. Here, the three-dimensional objects 5 that are the subject of the three-dimensional pattern Po and / or shading pattern Ps are at least one of the following: buildings, road structures, traffic lights, signs, and planted trees, etc. The road surface pattern P may also be the shape pattern Pp of puddles 6 that accumulate on the road surface 4, as shown in Figure 13.

[0035] In S20, the scene monitoring block 110 sets the combinations of set elements between the set elements of the first camera 31 and the set of the second camera 32 that result in the road pattern of interest P being captured in the superimposed area Avo (i.e., between cameras 31 and 32) as calibration candidates. As shown in Figure 4, if no combination is recognized as a calibration candidate that captures the road pattern of interest P, a negative judgment is made in S20, and the current execution of the calibration flow ends. On the other hand, if at least one pair of calibration candidates that captures the road pattern of interest P is recognized, an positive judgment is made in S20, and the calibration flow proceeds to S30.

[0036] In S30, the data output block 120 determines the amount of displacement ΔP between the target road pattern P projected in the superposition area Avo of the first image data Di1 and the second image data Di2 for the first camera 31 and the second camera 32, which are calibration candidates recognized in S20. This displacement amount ΔP represents the physical error caused by the misalignment of at least one of the optical axes L1 and L2 of the cameras 31 and 32, which are determined to be calibration candidates. Therefore, it is preferable that the displacement amount ΔP be determined based on the distance error between pairs of feature points that match on the orthogonal bird's-eye view coordinate system common to each of the image data Di1 and Di2, after the feature points of the target road pattern P are extracted from each image data Di1 and Di2. In this case, the maximum value of the distance errors between multiple pairs of corresponding feature points may be determined as the displacement amount ΔP. Alternatively, the average value of the distance errors between multiple pairs of matching feature points may be determined as the displacement amount ΔP. The amount of deviation ΔP may be determined to be the reciprocal of the similarity (i.e., the degree of difference) between multiple pairs of matching feature points.

[0037] In S30, the data output block 120 determines whether the amount of deviation ΔP related to the calibration candidates, the first camera 31 and the second camera 32, has increased to outside the acceptable range ωP shown in Figure 14. In this case, "outside the acceptable range ωP" may be defined as a range above the minimum value of the deviation amount ΔP that requires calibration, with that threshold being used as the threshold. Alternatively, "outside the acceptable range ωP" may be defined as a range exceeding the maximum value of the deviation amount ΔP that does not require calibration, with that threshold being used as the threshold. Here, Figure 14 shows a representative example of the amount of deviation ΔP that occurs when all combinations of each set element of the first camera 31 and the second camera 32 are calibration candidates. As shown in Figure 4, if at least one set of calibration candidates is recognized in S30 such that the amount of deviation ΔP is outside the acceptable range ωP, an affirmative judgment is made, and the calibration flow moves to S40.

[0038] In S40, when the deviation amount ΔP in at least one pair of calibration candidates falls outside the acceptable range ωP, the data output block 120 outputs calibration data Dc to calibrate the first camera 31 and second camera 32 of that candidate. At this time, among the combinations of calibration candidates recognized in S20, the calibration target for generating calibration data Dc may be selected by narrowing it down to the pair of first camera 31 and second camera 32 whose deviation amount ΔP is determined to be outside the acceptable range ωP in S30. Alternatively, the calibration target for generating calibration data Dc may be selected for each pair of first camera 31 and second camera 32, which represent all combinations of calibration candidates recognized in S20.

[0039] Therefore, calibration in S40 is achieved by optimizing the external parameters of the first camera 31 and the second camera 32, which are selected as calibration targets, so that the amount of displacement ΔP between them is all within the allowable range ωP as shown in Figure 15. The external parameters at this time should be estimated and adjusted by an optimization simulation calculation that minimizes the sum, a weighted sum in which the larger the value of ΔP, or the average value thereof, with respect to the amount of displacement ΔP between the first camera 31 and the second camera 32 that are being calibrated. Here, the external parameters are defined as matrix parameters that include rotation and translation components, which transform the orthogonal camera coordinate system that the image data Di follows into the orthogonal world coordinate system of the road surface 4.

[0040] Thus, in S40, calibration data Dc is generated and output to represent the external parameters that have been calibrated for the first camera 31 and the second camera 32 that are the targets of calibration. The output calibration data Dc is used to calibrate the image data Di in the steady-state imaging process of the first camera 31 and the second camera 32 that are the targets of calibration, enabling, for example, the synthesis process of image data Di converted to a bird's-eye view and the recognition process based thereon. The output calibration data Dc is also used to calibrate the image data Di acquired in S10 in subsequent executions of the calibration flow.

[0041] However, in S40, if the amount of deviation ΔP between at least one pair of calibration targets falls outside the acceptable range ωP as shown in Figure 16, even after optimization simulation calculations, specific data identifying cameras 31 and / or 32 that require physical maintenance among those targets may be output instead of or as calibration data Dc. In this case, the output calibration data Dc can be used, for example, to warn the user from the display device in the host vehicle 2. Upon completion of S40, the current execution of the calibration flow is finished.

[0042] Now, in S30 shown in Figure 4, if no calibration target is recognized whose displacement amount ΔP is outside the acceptable range ωP, a negative judgment is made and the calibration flow moves to S50. In S50, the data output block 120 outputs judgment data indicating that the displacement amount ΔP is within the acceptable range ωP for all combinations of the elements of each set of the first camera 31 and the second camera 32, either in place of the calibration data Dc or as the same data Dc. Upon completion of S50, the current execution of the calibration flow is finished.

[0043] (Effects and Benefits) The effects and advantages of this embodiment, as described above, will be explained below.

[0044] This embodiment calibrates a first camera 31 whose first field of view Av1 extends along the front-rear axis X of the host vehicle 2, and a second camera 32 whose second field of view Av2 extends along the left-right axis Y of the host vehicle 2. For this purpose, a first image data Di1 is acquired by the first camera 31 capturing the first field of view Av1, and a second image data Di2 is acquired by the second camera 32 capturing the second field of view Av2, which has a superimposed area Avo set to partially superimpose on the first field of view Av1.

[0045] Therefore, according to this embodiment, the driving scene in which the road pattern of interest P is reflected in the superimposed area Avo is monitored in both the first image data Di1 and the second image data Di2. If the amount of misalignment ΔP between the road patterns of interest P reflected in the superimposed area Avo in each of the first image data Di1 and the second image data Di2 increases beyond the allowable range ωP, then calibration between the first camera 31 and the second camera 32 is required. Accordingly, by outputting calibration data Dc according to the amount of misalignment ΔP between the road patterns of interest P common to the first camera 31 and the second camera 32, it is possible to accurately calibrate the cameras 31 and 32.

[0046] According to the calibration data Dc of this embodiment, the external parameters of cameras 31 and 32 are set so that the amount of displacement ΔP between the target road patterns P projected in the superimposed area Avo in each of the first image data Di1 and second image data Di2 is kept within the allowable range ωP. As a result, even when the amount of displacement ΔP between the common target road patterns P increases beyond the allowable range ωP between cameras 31 and 32, accurate calibration can be performed by setting the external parameters to keep the displacement ΔP within the allowable range ωP.

[0047] According to this embodiment, all possible combinations of set elements are assumed between the set of front camera 3a and rear camera 3b, which constitute the set of first cameras 31, and the set of left camera 3c and right camera 3d, which constitute the set of second cameras 32. Therefore, by narrowing down the set elements from all possible combinations of these set elements to those combinations in which the road pattern of interest P is projected onto the superimposed area Avo, it becomes possible to accurately achieve calibration.

[0048] According to this embodiment, the driving scene in which the road pattern of interest P is visible in the superimposed area Avo is monitored in both the first image data Di1 and the second image data Di2, which have been converted to a bird's-eye view relative to the host vehicle 2. As a result, the positional relationship of the road pattern of interest P in both the first image data Di1 and the second image data Di2 can be more easily recognized accurately through conversion to a bird's-eye view. Therefore, it becomes possible to ensure high-precision calibration accuracy corresponding to the amount of displacement ΔP between the road patterns of interest P common to the first camera 31 and the second camera 32.

[0049] Therefore, in this embodiment, the monitoring target is a driving scene in which, in a bird's-eye view, the inclined lane markings 40 on the road surface 4 extend to the superimposed area Avo for either the optical axes L1 or L2 of the first field of view Av1 or the second field of view Av2. As a result, the road surface pattern P of interest that is reflected in the superimposed area Avo, which is common to both the first image data Di1 and the second image data Di2, can be accurately recognized, for example, the lane marking pattern that intersects the optical axes L1 and / or L2 in driving scenes such as over an intersection or on a curved road. Thus, it becomes possible to ensure high accuracy the calibration accuracy corresponding to the amount of misalignment ΔP between the lane marking patterns common to the first camera 31 and the second camera 32.

[0050] From a different perspective, this embodiment can be said to monitor driving scenes in which the structural pattern Pr of the road surface 4 is reflected in the superimposed area Avo as the road pattern of interest P. This means that the road structure pattern Pr, which is relatively easy to recognize, is selected as the road pattern of interest P reflected in the superimposed area Avo, which is common to both the first image data Di1 and the second image data Di2. Therefore, it becomes possible to achieve high-precision calibration according to the amount of displacement ΔP between the structural patterns Pr of the road surface 4, which are common to both the first camera 31 and the second camera 32.

[0051] In addition to the above, this embodiment may also monitor driving scenes in which the three-dimensional pattern Po of three-dimensional objects 5 surrounding the road surface 4 is reflected in the superimposed area Avo as the road pattern of interest P. In this case, the road pattern of interest P reflected in the superimposed area Avo, which is common to both the first image data Di1 and the second image data Di2, is to be a three-dimensional pattern Po that is relatively easy to recognize. Therefore, such selection enables accurate calibration according to the amount of displacement ΔP between the three-dimensional patterns Po common to the first camera 31 and the second camera 32.

[0052] This embodiment may also monitor driving scenes in which the shadow patterns Ps of three-dimensional objects 5 appearing on the road surface 4 as a road pattern of interest P are reflected in the superimposed area Avo. In this case, the shadow patterns Ps of three-dimensional objects 5 are selected as a road pattern of interest P that is easily reflected in the common superimposed area Avo in both the first image data Di1 and the second image data Di2. Therefore, such selection enables accurate calibration according to the amount of displacement ΔP between the shadow patterns Ps of three-dimensional objects 5 that are common to the first camera 31 and the second camera 32.

[0053] In this embodiment, driving scenes in which the shape pattern Pp of puddles 6 accumulating on the road surface 4 is reflected in the superimposed area Avo may also be monitored as a road pattern of interest P. In this case, the shape pattern Pp of puddles 6 is selected as a road pattern of interest P that is easily reflected in the common superimposed area Avo in both the first image data Di1 and the second image data Di2. Therefore, such selection enables accurate calibration according to the amount of difference ΔP between the shape patterns Pp of puddles 6 common to the first camera 31 and the second camera 32.

[0054] (Other embodiments) Although one embodiment has been described above, this disclosure is not to be construed as being limited to the embodiment described herein, and can be applied to various embodiments and combinations without departing from the spirit of this disclosure.

[0055] In the modified example, the dedicated computer constituting the calibration system 1 may have at least one of the digital circuit and the analog circuit as a processor. Here, the digital circuit is at least one of the following, for example, ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). Such a digital circuit may also have a memory that stores a program.

[0056] In the modified example, in S10, the viewpoint conversion process for the image data Di to a bird's-eye view may be omitted. In this case, in at least S30 of S20, S30, and S40, a viewpoint conversion process may be performed to convert the viewpoint of one of the image data Di1 and the second image data Di2 to the viewpoint of the other. In the modified example, the first camera 31 may be either the front camera 3a or the rear camera 3b, and one of them may not be subject to calibration. In addition to the embodiments described so far, the host vehicle 2 to which the calibration system 1 is applied in the above embodiments and modified examples may be, for example, an autonomous robot capable of transporting cargo or collecting information by autonomous driving or remote driving.

[0057] (Additional note) This specification discloses several technical concepts and several combinations thereof, as listed below. The symbols in parentheses in this supplementary section indicate correspondences with the specific means described in the embodiments detailed above, and do not limit the technical scope of this disclosure.

[0058] (Technical thought 1) A calibration system having a processor (12) for calibrating a first camera (31) whose first field of view (Av1) extends along the front-rear axis (X) of a host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the left-right axis (Y) of the host vehicle, The aforementioned processor, The process involves acquiring first image data (Di1) captured by the first camera capturing the first field of view, along with second image data (Di2) captured by the second camera capturing the second field of view, which has a superimposed area (Avo) that partially superimposes on the first field of view. In both the first image data and the second image data, the driving scene in which the road pattern of interest (P), which is the road pattern of interest, is reflected in the superimposed area is monitored. A calibration system configured to output calibration data (Dc) for calibrating the relationship between the first camera and the second camera in response to the amount of displacement (ΔP) between the road patterns of interest projected in the superimposed area in each of the first image data and the second image data increasing outside the acceptable range (ωP).

[0059] (Technical thought 2) The output of the aforementioned calibration data is: A calibration system according to technical concept 1, which includes outputting calibration data for calibrating the relationship between the first camera and the second camera by setting the external parameters of the first camera and the second camera so that the amount of deviation is kept within the allowable range.

[0060] (Technical Thought 3) The output of the aforementioned calibration data is: A calibration system according to technical idea 1 or 2, which includes outputting calibration data between a set of set elements that, out of all possible combinations of set elements, the elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of all possible combinations of set elements, the set elements that, out of

[0061] (Technical Thought 4) Monitoring of the aforementioned driving scene, A calibration system according to any one of the technical ideas 1 to 3, which includes monitoring the driving scene in which the road pattern of interest is reflected in the superimposed area in both the first image data and the second image data converted to a bird's-eye view of the host vehicle.

[0062] (Technical Thought 5) Monitoring of the aforementioned driving scene, A calibration system according to technical concept 4, which includes monitoring the driving scene in which, in the bird's-eye view, the inclined lane markings (40) on the road surface (4) extend to the superimposed area for each optical axis (L1, L2) of the first field of view and the second field of view.

[0063] (Technical Thought 6) Monitoring of the aforementioned driving scene is A calibration system according to any one of the technical concepts 1 to 5, which includes monitoring the driving scene in which the structural pattern (Pr) of the road surface (4) is projected onto the superimposed area as the aforementioned road pattern of interest.

[0064] (Technical Thought 7) Monitoring of the aforementioned driving scene, A calibration system according to any one of the technical concepts 1 to 6, which includes monitoring the driving scene in which the three-dimensional pattern (Po) of three-dimensional objects (5) around the road surface (4) is projected onto the superimposed area as the aforementioned road surface pattern of interest.

[0065] (Technical Thought 8) Monitoring of the aforementioned driving scene, A calibration system according to any one of the technical concepts 1 to 7, which includes monitoring the driving scene in which the shadow pattern (Ps) of a three-dimensional object (5) appearing on the road surface (4) as the aforementioned road pattern of interest is projected onto the superimposed area.

[0066] (Technical Thought 9) Monitoring of the aforementioned driving scene, A calibration system according to any one of the technical concepts 1 to 8, which includes monitoring the driving scene in which the shape pattern (Pp) of puddles (6) accumulating on the road surface (4) is projected onto the superimposed area.

[0067] Furthermore, the technical concepts 1 to 9 described above may also be understood within the technical concepts of the apparatus, method, and program, respectively. [Explanation of Symbols]

[0068] 1: Calibration system, 2: Host vehicle, 3: Camera, 3a: Front camera, 3b: Rear camera, 3c: Left camera, 3d: Right camera, 4: Road surface, 5: 3D object, 6: Puddle, 10: Memory, 12: Processor, 3: Camera, 31: First camera, 32: Second camera, 40: Lane markings, Av1: First field of view, Av2: Second field of view, Avo: Overlay area, Dc: Calibration data, Di1: First image data, Di2: Second image data, L1, L2: Optical axis, P: Road pattern of interest, Po: 3D pattern, Pp: Shape pattern, Pr: Structural pattern, Ps: Shading pattern, X: Front-to-back axis, Y: Left-to-right axis, ΔP: Amount of displacement

Claims

1. A calibration system having a processor (12) for calibrating a first camera (31) whose first field of view (Av1) extends along the front-rear axis (X) of a host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the left-right axis (Y) of the host vehicle, The aforementioned processor, The first camera captures the first field of view and, together with the first image data (Di1) captured by the first camera, the second image data (Di2) is obtained by the second camera capturing the second field of view, which has a superimposed area (Avo) that partially superimposes on the first field of view. In both the first image data and the second image data, the driving scene in which the road pattern of interest (P), which is the road pattern of interest, is reflected in the superimposed area is monitored. A calibration system configured to output calibration data (Dc) for calibrating the relationship between the first camera and the second camera in response to the amount of displacement (ΔP) between the road patterns of interest projected in the superimposed area in each of the first image data and the second image data increasing outside the acceptable range (ωP).

2. The output of the aforementioned calibration data is: The calibration system according to claim 1, which includes outputting calibration data for calibrating the relationship between the first camera and the second camera by setting the external parameters of the first camera and the second camera such that the amount of deviation is kept within the allowable range.

3. The output of the aforementioned calibration data is: The calibration system according to claim 1, further comprising outputting calibration data between a set of front cameras (3a) and rear cameras (3b), which constitute the set of first cameras, and a set of left cameras (3c) and right cameras (3d), which constitute the set of second cameras, for calibration between set elements of a combination in which the road pattern of interest is projected onto the superimposed area, out of all possible combinations of set elements.

4. Monitoring of the aforementioned driving scene, The calibration system according to claim 1, which includes monitoring the driving scene in which the road pattern of interest is reflected in the superimposed area in both the first image data and the second image data converted to a bird's-eye view of the host vehicle.

5. Monitoring of the aforementioned driving scene, The calibration system according to claim 4, which includes monitoring the driving scene in which, in the bird's-eye view, the inclined lane markings (40) on the road surface (4) extend to the superimposed area with respect to either of the optical axes (L1, L2) of the first field of view and the second field of view.

6. Monitoring of the aforementioned driving scene, A calibration system according to any one of claims 1 to 5, comprising monitoring the driving scene in which the structural pattern (Pr) of the road surface (4) is projected onto the superimposed area as the aforementioned road pattern of interest.

7. Monitoring of the aforementioned driving scene, A calibration system according to any one of claims 1 to 5, which includes monitoring the driving scene in which the three-dimensional pattern (Po) of three-dimensional objects (5) around the road surface (4) is projected onto the superimposed area as the aforementioned road surface pattern of interest.

8. Monitoring of the aforementioned driving scene, A calibration system according to any one of claims 1 to 5, which includes monitoring the driving scene in which the shadow pattern (Ps) of a three-dimensional object (5) appearing on the road surface (4) as the aforementioned road pattern of interest is projected onto the superimposed area.

9. Monitoring of the aforementioned driving scene, A calibration system according to any one of claims 1 to 5, which includes monitoring the driving scene in which the shape pattern (Pp) of puddles (6) accumulating on the road surface (4) is projected onto the superimposed area as the aforementioned road pattern of interest.

10. A calibration device having a processor (12) for calibrating a first camera (31) whose first field of view (Av1) extends along the front-rear axis (X) of a host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the left-right axis (Y) of the host vehicle, and configured to be mountable on the host vehicle, The aforementioned processor, The first camera captures the first field of view and, together with the first image data (Di1) captured by the first camera, the second image data (Di2) is obtained by the second camera capturing the second field of view, which has a superimposed area (Avo) that partially superimposes on the first field of view. In both the first image data and the second image data, the driving scene in which the road pattern of interest (P), which is the road pattern of interest, is reflected in the superimposed area is monitored. A calibration device configured to output calibration data (Dc) for calibrating the relationship between the first camera and the second camera in response to the amount of displacement (ΔP) between the road patterns of interest projected in the superimposed area in each of the first image data and the second image data increasing outside the acceptable range (ωP).

11. A calibration method performed by a processor (12) to calibrate a first camera (31) whose first field of view (Av1) extends along the front-rear axis (X) of a host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the left-right axis (Y) of the host vehicle, wherein The first camera captures the first field of view and, together with the first image data (Di1) captured by the first camera, the second image data (Di2) is obtained by the second camera capturing the second field of view, which has a superimposed area (Avo) that partially superimposes on the first field of view. In both the first image data and the second image data, the driving scene in which the road pattern of interest (P), which is the road pattern of interest, is reflected in the superimposed area is monitored. A calibration method comprising outputting calibration data (Dc) for calibrating the relationship between the first camera and the second camera in accordance with the amount of displacement (ΔP) between the road patterns of interest projected in the superimposed area in each of the first image data and the second image data, as the displacement increases outside the acceptable range (ωP).

12. A calibration program is stored in a storage medium (10) for the purpose of calibrating a first camera (31) whose first field of view (Av1) extends along the front-rear axis (X) of a host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the left-right axis (Y) of the host vehicle, and includes instructions for causing a processor (12) to perform said calibration, wherein the calibration is performed by a first camera (31) whose first field of view (Av1) extends along the front-rear axis (X) of a host vehicle (2), and a second camera (32) whose second field of view (Av2) extends along the left-right axis (Y) of the host vehicle, and the calibration program includes instructions for causing a processor (12) to perform said calibration. The first camera captures the first field of view and, together with the first image data (Di1) captured by the first camera, the second image data (Di2) is obtained by the second camera capturing the second field of view, which has a superimposed area (Avo) that partially superimposes on the first field of view. In both the first image data and the second image data, the driving scene in which the road pattern of interest (P), which is the road pattern of interest, is reflected in the superimposed area is monitored. A calibration program including the command to output calibration data (Dc) for calibrating the relationship between the first camera and the second camera in response to the amount of displacement (ΔP) between the road patterns of interest projected in the superimposed area in each of the first image data and the second image data increasing outside the acceptable range (ωP).

Images