On-vehicle camera calibration method, and on-vehicle camera calibration device

Abstract

This on-vehicle camera calibration method comprises: a step for capturing images by a plurality of on-vehicle cameras 2 mounted on a vehicle 1; a step for acquiring each image captured by the plurality of on-vehicle cameras 2; a step for extracting feature point information indicating a calibration marker 3 from the acquired images; a step for calculating, with respect to each image, a deviation between the extracted feature point information and reference information obtained from a design value; a step for estimating the cause of the deviation on the basis of a correlation between deviations in the images captured by at least two or more on-vehicle cameras 2; and a step for calibrating the on-vehicle cameras 2 on the basis of the calculated deviation and the estimated cause of the deviation.

Description

Calibration method for in-vehicle cameras and calibration device for in-vehicle cameras CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Japanese Application No. 2024-049584, filed on March 26, 2024, the contents of which are incorporated herein by reference.

[0002] The present disclosure relates to a calibration method and a calibration device for calibrating an in-vehicle camera.

[0003] In recent years, the number of vehicles equipped with on-board cameras capable of capturing images of the exterior of the vehicle has been increasing. Such on-board cameras are used for driving assistance, such as for positioning the vehicle while parking or driving, by processing the captured video and images. Therefore, high calibration accuracy is required. For example, Patent Document 1 describes a method for appropriately calibrating on-board cameras.

[0004] Patent No. 5091902

[0005] Calibration of an on-board camera is generally performed using a calibration marker having a predetermined size of figure, symbol, or letter, etc. Specifically, the calibration marker and the vehicle are installed so as to have a predetermined positional relationship, feature point information indicating the position, size, etc. of the calibration marker is extracted from an image captured by the on-board camera, and the feature point information is compared with reference information indicating the position, size, etc. of the designed calibration marker, which is obtained based on design values ​​such as the installation position of the on-board camera, thereby calibrating the on-board camera.

[0006] However, in the past, each on-board camera was calibrated independently, and the only information used during calibration was feature point information extracted from the image captured by that on-board camera. In this case, for example, if the installation position of the vehicle is misaligned, the appearance of the calibration marker changes, resulting in errors in the extracted feature point information. Furthermore, in the past, when there was a discrepancy between the feature point information and the reference information, it was not possible to determine whether the cause of the discrepancy was an installation error due to the installation state of the on-board camera or an installation error due to the installation state of the vehicle.

[0007] The present disclosure has been made in consideration of the above circumstances, and its purpose is to provide a calibration method and calibration device for an on-board camera that can isolate the cause of the deviation and calibrate the on-board camera after isolating the cause of the deviation.

[0008] A method for calibrating an on-board camera according to one aspect of the present disclosure includes the steps of capturing images using multiple on-board cameras mounted on a vehicle, acquiring each of the images captured by the multiple on-board cameras, extracting feature point information indicating calibration markers from the acquired images, calculating, for each image, the deviation between the extracted feature point information and reference information determined from design values, estimating a cause of the deviation based on the correlation between the deviations in images captured by at least two or more on-board cameras, and calibrating the on-board camera based on the calculated deviation and the estimated cause of the deviation.

[0009] In addition, an on-board camera calibration device according to one aspect of the present disclosure includes an imaging instruction unit that executes a process to cause each of multiple on-board cameras mounted on a vehicle to capture an image; an image acquisition unit that executes a process to acquire each of the images captured by the multiple on-board cameras; an extraction unit that executes a process to extract feature point information indicating a calibration marker from the acquired images; a deviation calculation unit that executes a process to calculate, for each image, the deviation between the extracted feature point information and reference information determined from design values; a factor estimation unit that executes a process to estimate a factor of the deviation based on the correlation of the deviation in images captured by at least two or more on-board cameras; and a calibration unit that executes a process to calibrate the on-board camera based on the calculated deviation and the estimated factor of the deviation.

[0010] The above and other objects, features, and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which Fig. 1 is a diagram schematically illustrating an example of an installation mode of an on-board camera according to an embodiment, Fig. 2 is a diagram schematically illustrating an example of the angle of view of the on-board camera, Fig. 3 is a diagram schematically illustrating an example of a reference position during calibration and an installation mode of a vehicle, Fig. 4 is a diagram schematically illustrating an example of a calibration marker, Fig. 5 is a diagram schematically illustrating an example of the electrical configuration of a calibration device, Fig. 6 is a diagram briefly explaining the calibration work, Fig. 7 is a diagram explaining an example of a cause of deviation, Fig. 8 is a diagram showing the flow of the calibration process, Fig. 9 is a diagram explaining an example of correcting the reference information, and Fig. 10 is a diagram schematically illustrating another example of an installation mode of the on-board camera.

[0011] Hereinafter, an embodiment will be described with reference to the drawings. As shown in Fig. 1, a vehicle 1 is equipped with a plurality of on-board cameras 2 to be calibrated. In this embodiment, the vehicle 1 is equipped with an on-board camera 2A mounted in an orientation to capture an image of the front, an on-board camera 2B mounted in an orientation to capture an image of the rear of the vehicle 1, an on-board camera 2C mounted in an orientation to capture an image of the left side of the vehicle 1, and an on-board camera 2D mounted in an orientation to capture an image of the right side of the vehicle 1.

[0012] 1 is defined as a reference position (P), a reference line passing through the reference position (P) and extending in the longitudinal direction of the vehicle 1 is defined as a virtual line (LX), and a reference line passing through the reference position (P) and extending in the lateral direction of the vehicle 1 is defined as a virtual line (LY). Note that the number, mounting positions, and angles of the vehicle-mounted cameras 2 shown in FIG. 1 are merely examples and are not intended to be limiting.

[0013] The mounting position of each vehicle-mounted camera 2 is determined by a design value. For example, as shown in a plan view, vehicle-mounted camera 2A is mounted on a virtual line (LX) at a mounting position that is a distance (L1) rearward from a reference position (P), with the angle of view facing forward of vehicle 1. Also, as shown in a side view, vehicle-mounted camera 2A is mounted at a position that is a predetermined height (H1) from the installation surface of vehicle 1, with an installation angle that makes its optical axis parallel to the installation surface. As shown in FIG. 2 , the imaging range of vehicle-mounted camera 2A is a predetermined angle of view (RA) ahead of vehicle 1.

[0014] The vehicle-mounted camera 2B is mounted on a virtual line (LX) at a mounting position a distance (L2) rearward from the reference position (P), with the angle of view facing rearward of the vehicle 1. The vehicle-mounted camera 2B is mounted at a predetermined height (H2) from the installation surface of the vehicle 1, as shown in a side view, at a mounting angle such that its optical axis is parallel to the installation surface. The imaging range of the vehicle-mounted camera 2B is a predetermined angle of view (RB) behind the vehicle 1, as shown in FIG. 2 .

[0015] The vehicle-mounted camera 2C is mounted at a mounting position that is a distance (W1) to the left of the virtual line (LX) and a distance (L3) rearward from the reference position (P), with the angle of view facing the left side of the vehicle 1. The vehicle-mounted camera 2C is mounted at a position that is a predetermined height (H3) from the installation surface of the vehicle 1, as shown in a side view, with an installation angle that makes its optical axis parallel to the installation surface. The imaging range of the vehicle-mounted camera 2C is a predetermined angle of view (RC) on the left side of the vehicle 1, as shown in FIG. 2 .

[0016] The vehicle-mounted camera 2D is mounted at a mounting position that is a distance (W1) to the right of the virtual line (LX) and a distance (L3) rearward from the reference position (P), with the field of view oriented toward the right side of the vehicle 1. The vehicle-mounted camera 2D is mounted at a predetermined height (H3) from the installation surface of the vehicle 1, as shown in a side view, with the optical axis of the vehicle-mounted camera 2D mounted at an installation angle that is parallel to the installation surface. As shown in FIG. 2 , the imaging range of the vehicle-mounted camera 2D is a predetermined field of view (RD) on the right side of the vehicle 1. Note that the field of view shown in FIG. 2 is an example and is not limiting. The vehicle-mounted camera 2D may be mounted inside the vehicle cabin, for example, near the back of the rearview mirror, or outside the vehicle cabin, for example, at the tip of a side mirror.

[0017] Calibration of such an on-board camera 2 is performed by placing the vehicle 1 in a predetermined positional relationship with respect to calibration markers 3 that are placed at predetermined positions as shown in Fig. 3. In this embodiment, a reference position (K), a virtual line (KX) that passes through the reference position (K), and a virtual line (KY) that passes through the reference position (K) and intersects the virtual line (KX) at right angles are set in the work area where calibration is performed. The calibration markers 3 are then placed so as to have a predetermined positional relationship with respect to the reference position (K), the virtual line (KX), and the virtual line (KY). Note that the number and placement of the calibration markers 3 and their positional relationship with the vehicle 1 shown in Fig. 3 are merely examples and are not limiting.

[0018] For example, the calibration marker 3 is positioned in front of the vehicle 1 such that the marker center (M1) described below is located at a distance (L11) above the reference position (K) and a distance (W11) to the left of the virtual line (KX) in the figure, and the marker center (M1) described below is located at a distance (L11) above the reference position (K) and a distance (W11) to the right of the virtual line (KX) in the figure.

[0019] In addition, the calibration marker 3 is positioned at the rear of the vehicle 1 such that its center (M1) is a distance (L12) downward from the reference position (K) in the figure and a distance (W12) to the left of the virtual line (KX) in the figure, and its center (M1) is a distance (L12) downward from the reference position (K) in the figure and a distance (W12) to the right of the virtual line (KX) in the figure.

[0020] In addition, the calibration marker 3 is positioned on the left side of the vehicle 1 so that its center (M1) is a distance (L13) downward from the reference position (K) in the figure and a distance (W13) to the left from the virtual line (KX) in the figure, and so that its center (M1) is a distance (L14) downward from the reference position (K) in the figure and a distance (W13) to the left from the virtual line (KX) in the figure.

[0021] In addition, the calibration marker 3 is positioned on the right side of the vehicle 1 so that its center (M1) is a distance (L13) downward from the reference position (K) and a distance (W13) to the right from the virtual line (KX), and so that its center (M1) is a distance (L14) downward from the reference position (K) and a distance (W13) to the right from the virtual line (KX).

[0022] 4, the calibration marker 3 is formed in the shape of a plate with a width of L21 and a length of L22, for example, and has a black and white image drawn on its surface. In this embodiment, white rectangles with a width of L21a and a length of L22a and black rectangles with a width of L21a and a length of L22a are arranged alternately. The intersection of a virtual line (CL21) extending vertically in the figure and a virtual line (CL22) extending horizontally in the figure, which are the boundaries between the images, is the marker center (M1) indicating the center of the calibration marker 3.

[0023] These calibration markers 3 are supported by support members 4 as shown in a side view in Fig. 3, and are arranged as shown in a plan view with the marker centers (M1) at a predetermined height (H11) from the installation surface. Note that the shapes of the calibration markers 3 and the drawn figures shown in Fig. 4 are merely examples and are not limiting. For example, circular figures, polygonal figures, or drawn characters, symbols, etc. may also be used.

[0024] Now, the calibration of the vehicle-mounted camera 2 is performed in a state where the reference position (P) of the vehicle 1 and the reference position (K) of the calibration marker 3 are aligned, as shown in a plan view in Figure 3. Specifically, the calibration of the vehicle-mounted camera 2 is performed in a state where the reference position (P) and the reference position (K) are aligned, the virtual lines (LX) and the virtual lines (KX) are aligned, and the virtual lines (LY) and the virtual lines (KY) are aligned. Hereinafter, the reference position (P) will also be referred to as the reference on the vehicle-mounted camera 2 side, and the reference position (K) will also be referred to as the reference on the calibration marker 3 side. For convenience, the state in which the vehicle 1 is installed so that the reference on the vehicle-mounted camera 2 side and the reference on the calibration marker 3 side are aligned will be referred to as the appropriate position.

[0025] In this embodiment, each calibration marker 3 is placed at the work site in advance, and the vehicle 1 is moved to the work site and positioned so that the reference on the vehicle-mounted camera 2 side and the reference on the calibration marker 3 side coincide with each other, and then the calibration is performed using the calibration device 10 shown in Figure 5.

[0026] The calibration device 10 is entirely controlled by a microcomputer (not shown), and includes functional units such as an image input unit 11, an image capture instruction unit 12, an image processing unit 13, a memory unit 14, a display control unit 15, and an operation control unit 16. In this embodiment, the image capture instruction unit 12 and the image processing unit 13 are realized by software by executing a program on the microcomputer. The calibration device 10 can be configured as a dedicated device, or some or all of the functional units can be implemented in an electronic control device such as a camera ECU (not shown) for controlling the on-board camera 2 mounted on the vehicle 1 or an HMI ECU (not shown) for controlling a human-machine interface.

[0027] The image input unit 11 is an interface to which images captured by the vehicle-mounted camera 2 are input, and receives a video signal from the vehicle-mounted camera 2. The image input unit 11 also transmits a signal to instruct the vehicle-mounted camera 2 to capture an image. Note that when the vehicle-mounted camera 2 is controlled by a camera ECU, the image input unit 11 is configured as a so-called communication unit that communicates with the camera ECU.

[0028] The imaging instruction unit 12 generates an instruction to capture an image for calibration and outputs it to the vehicle-mounted camera 2 via the image input unit 11. The storage unit 14 is configured with a storage medium such as a semiconductor memory, and stores various programs for controlling the calibration device 10 and for realizing each functional unit, as well as a reference information DB 17 in which reference information for calibration is aggregated, the details of which will be described later.

[0029] The display control unit 15 controls the display of various information on the display 18 during the calibration work. At this time, a center display mounted on the vehicle 1 can be used as the display 18.

[0030] The operation control unit 16 accepts operations input by an operator to the operation unit 19, such as an operation to instruct imaging or an operation to perform calibration work. In this case, the operation unit 19 may be configured to utilize a touch panel or the like mounted on the vehicle 1. Note that the display control unit 15 and the operation control unit 16 are not essential for the calibration device 10, and the calibration device 10 may be configured to be connected to an external device such as a personal computer via a communication line, in which case, for example, a display or keyboard on the external device side may be used.

[0031] The image processing unit 13 has various functional units that execute various processes related to the calibration of the vehicle-mounted cameras 2. Among these, the image acquisition unit 20 executes a process of acquiring images captured by the multiple vehicle-mounted cameras 2. At this time, the image acquisition unit 20 acquires the images by temporarily storing the video signals input to the image input unit 11 in the storage unit 14 or the like.

[0032] The extraction unit 21, details of which will be described later, executes a process of extracting feature point information indicating the calibration marker 3 from the acquired image. For example, as shown in an image capture example in FIG. 6 , when the calibration marker 3 is captured in an actual image 30 at a certain angle of view (R) by the vehicle-mounted camera 2 to be calibrated, the extraction unit 21 extracts feature point information indicating the position of the marker center (M1) in the actual image 30. The extraction unit 21 can also extract other information that can be obtained from the actual image 30, such as the distance from the vehicle-mounted camera 2 to the calibration marker 3, which is obtained from the size of the calibration marker 3 captured in the actual image 30, and the orientation of the vehicle-mounted camera 2 with respect to the calibration marker 3, which is obtained from the direction and length ratio of each side of the calibration marker 3, as actual image information. This actual image information is used as appropriate in processes such as estimating the cause of deviation, which will be described later.

[0033] Well-known techniques can be used to extract the feature point information, and will be explained briefly here, but the image is scanned horizontally and / or vertically to detect the edges of the figure, the figure shown in the image is detected based on the detected edges, the detected figure is compared with the figure of the calibration marker 3 by, for example, image matching to identify the positions of the feature points indicating the calibration marker 3, and feature point information indicating the positional relationship between the vehicle-mounted camera 2 and the calibration marker 3 is extracted from the identified positions of the feature points in the image. Note that the feature point information is extracted as so-called numerical data within the calibration device 10, but it is also possible to overlay on the display 18 a colored frame image indicating the calibration marker 3 or a round image indicating the marker center (M1), etc., to make it easier for the worker to understand the work content.

[0034] The deviation calculation unit 22 executes a process for calculating the deviation between the extracted feature point information and reference information obtained from the design values ​​for each image captured by the multiple on-board cameras 2. This reference information is information indicating the position and size of the calibration marker 3 that should appear in the angle of view (R) when the on-board camera 2 to be calibrated is attached at the position according to the design values ​​and the vehicle 1 is in an appropriate position. This reference information is basically treated as numerical data within the calibration device 10. However, to make it easier for the worker to confirm the work content, for example, as shown in FIG. 6 as an example of reference information, images such as a reference image v30 indicating the angle of view in the reference information, a virtual frame (v3) virtually indicating the calibration marker 3 in the reference image v30, and a virtual center (vM1) indicating the center of the virtual frame (v3) are overlaid and displayed.

[0035] The factor estimation unit 23 executes a process of estimating the cause of the deviation that occurred in the image captured by the in-vehicle camera 2 to be calibrated, based on the correlation of deviations in the images captured by at least two or more in-vehicle cameras 2. Details of the correlation will be described later. The factor estimation unit 23 also estimates whether the cause of the deviation is an installation error caused by the installation state of the in-vehicle camera 2, or whether the cause of the deviation includes an installation error caused by the installation state of the vehicle 1.

[0036] 7 as a comparative example, suppose that a deviation occurs between the feature point information and the reference information in an actual image 30 captured by the vehicle-mounted camera 2 to be calibrated. In this comparative example, the marker center (M1) and virtual center (vM1) of the calibration marker 3 coincide, but the size of the calibration marker 3 is smaller than the virtual frame (v3). In this case, a possible cause of the deviation is an attachment error in which the attachment state of the vehicle-mounted camera 2 deviates from the design value. However, when calibration is performed by moving the vehicle 1 as in this embodiment, an installation error in which the vehicle 1 is deviated from the appropriate position may also be included.

[0037] Therefore, the factor estimation unit 23 checks whether there is a correlation between the factors of the deviation of the actual image 30 captured by the in-vehicle camera 2 to be calibrated and the deviation of an image captured by another in-vehicle camera 2, and estimates the factors based on the correlation. In other words, the factor estimation unit 23 determines whether the deviation factors include an installation error based on the correlation of the deviations.

[0038] The calibration unit 24 calibrates the vehicle-mounted camera 2 based on the calculated deviation and the estimated cause of the deviation. Calibration of the vehicle-mounted camera 2 can be performed using well-known techniques, so a brief explanation will be given here. When calibrating the vehicle-mounted camera 2, the calibration unit 24 first identifies the installation state of the vehicle-mounted camera 2, such as the installation position, installation angle, or installation height, based on the position, size, distortion, etc. of the measurement markers shown in the image. The calibration unit 24 also identifies the installation position and installation orientation of the vehicle 1, as will be described later.

[0039] Then, when there is a misalignment between the actual image 30 and the reference image v30, as shown in the calibration example in Fig. 6, the calibration unit 24 calculates parameters to correct the misalignment. The calculated parameters are used, for example, when providing driving assistance to measure the distance and positional relationship with surrounding objects, and enable appropriate assistance to be provided according to the actual installation state. Note that lens distortion correction, etc. may also be performed during calibration.

[0040] The correction value calculation unit 25, the details of which will be described later, executes a process of calculating a correction value for removing the deviation caused by the installation error when it is estimated that the deviation is caused by an installation error.

[0041] The new reference generating unit 26, as will be described in detail later, corrects the reference information using the calculated correction value and executes a process of generating new reference information from which the installation error has been removed. In other words, the new reference generating unit 26 recreates the reference information based on the design values ​​so that it corresponds to the positional relationship between the actual vehicle 1 and the calibration marker 3.

[0042] Next, the operation and effect of the above-described configuration will be described. First, the correlation between factors that cause deviation and deviation will be described in detail. For example, as shown in the RA comparison in Fig. 7 , in an actual image 30A in which the calibration marker 3 is captured by an in-vehicle camera 3A that captures an image ahead of the vehicle 1, the marker center (M1) and the virtual center (vM1) generally coincide with each other, but the calibration marker 3 is smaller than the virtual frame (v3).

[0043] In this case, the cause of the deviation is thought to be that, although the mounting position of the vehicle-mounted camera 2A is appropriate, as shown schematically in Example 1, the installation position of the vehicle 1 is shifted rearward by a distance (ΔL) from the reference position (P), and that this distance (ΔL) causes the calibration marker 3 to appear smaller than the virtual frame (v3).

[0044] On the other hand, as a cause of the deviation, as shown schematically in Example 2, it is possible that the installation position of the vehicle 1 is appropriate, but the mounting position of the vehicle-mounted camera 2A is shifted a distance (ΔL) rearward from the appropriate distance (L1), and that this distance (ΔL) causes the calibration marker 3 to appear smaller than the virtual frame (v3).

[0045] In this case, it is difficult to distinguish whether the cause of the deviation is an installation error or an attachment error in the actual image 30A captured by one vehicle-mounted camera 2A as shown in the RA comparison. Also, if the installation position of the vehicle 1 is misaligned as in factor example 1, the installation position of the vehicle-mounted camera 2A may also be misaligned, and if the installation position of the vehicle-mounted camera 2A is misaligned as in factor example 2, the installation position of the vehicle 1 may also be misaligned.

[0046] In other words, it is difficult to determine whether the cause of the deviation is an installation error of the vehicle 1 from only the actual image 30 captured by one on-board camera 2A. Therefore, the conventional method cannot isolate the cause of the deviation, and there is a possibility that calibration will be performed in a state that includes an installation error of the vehicle 1. If calibration is performed in a state that includes an error, there is a risk that this will affect the accuracy of driving assistance, such as estimating the distance to a surrounding object, using the on-board camera 2.

[0047] Therefore, in this embodiment, the correlation between two or more images is utilized. For example, as shown in RB Comparison Example 1 in Fig. 7, in an actual image 30B captured by an in-vehicle camera 2B capturing an image of the rear of the vehicle 1, the marker center (M1) and the virtual center (vM1) are approximately aligned, and the calibration marker 3 is larger than the virtual frame (v3). In this case, the distance between the in-vehicle camera 2B and the calibration marker 3 is closer than the reference distance. The distance can be determined from the actual image information.

[0048] In other words, by referring to the images captured by on-board camera 2A and on-board camera 2B, it is possible to obtain a correlation that is presumed to be caused by the installation state of vehicle 1, in which on-board camera 2A is farther away from calibration marker 3 than the reference, and on-board camera 2B is closer to calibration marker 3 than the reference. From this correlation, it can be estimated that the installation error of vehicle 1, shown as factor example 1, is included in the factors that cause the deviation.

[0049] Furthermore, as shown in RB Comparison Example 2, in the actual image 30 captured by the vehicle-mounted camera 2B, the marker center (M1) and the virtual center (vM1) are approximately aligned, and their sizes are also approximately aligned. In this case, the vehicle-mounted camera 2B is positioned as per the reference relative to the calibration marker 3. In other words, the actual image 30 captured by the vehicle-mounted cameras 2A and 2B shows that the vehicle-mounted camera 2A is farther away from the calibration marker 3 than the reference position, while the vehicle-mounted camera 2B is attached at the reference position.

[0050] In this case, it can be determined that there is no correlation that is caused by the installation state of the vehicle 1. In other words, from the correlation of the deviations in the multiple actual images 30, it can be estimated that the deviation is caused by the installation error of the vehicle-mounted camera 2, which was shown as factor 2.

[0051] Such a correlation is not limited to the longitudinal direction. For example, if the vehicle 1 is installed so as to be displaced counterclockwise around the on-board camera 2A shown in Fig. 2, a correlation is observed in each captured image in which the calibration marker 3 is displaced to the right of the reference. Furthermore, if the vehicle 1 is displaced entirely to the left of the virtual line (LX), a correlation is observed in which the calibration marker 3 is displaced to the right in the image captured by the on-board camera 2A, while the calibration marker 3 is displaced to the left in the image captured by the on-board camera 2B. Note that such a correlation is not limited to between the on-board cameras 2A and 2B, but may also be observed between the on-board cameras 2C and 2D.

[0052] Incidentally, calibration of the vehicle-mounted camera 2 is basically performed for all vehicles 1 equipped with the vehicle-mounted camera 2. Furthermore, careful work is required to accurately install the vehicles 1 at the reference positions, and the work time inevitably increases as the number of vehicles 1 to be worked on increases. Therefore, even if installation errors of the vehicles 1 are included, it is thought that work efficiency can be significantly improved if the vehicle-mounted cameras 2 can be calibrated with the installation errors removed.

[0053] Therefore, the calibration device 10 makes it possible to isolate the cause of the deviation as described below, and even if there is an installation error of the vehicle 1, it makes it possible to perform calibration while removing the installation error. A calibration method using the calibration device 10 will be described below. Note that although the processes described below are performed by the functional units described above sharing or cooperating with each other, for the sake of simplicity, the description will be centered on the calibration device 10. It is also assumed that the vehicle 1 has already been installed.

[0054] 8 , the calibration device 10 instructs each vehicle-mounted camera 2 to capture an image (S1) and acquires the images captured by each vehicle-mounted camera 2 (S2). Next, the calibration device 10 identifies the calibration marker 3 in each image (S3). That is, the calibration device 10 checks whether the calibration marker 3 appears in the acquired images. If the calibration device 10 cannot identify the calibration marker 3 in any of the images (S4: NO), it executes error processing. This error processing may, for example, notify the operator that the calibration marker 3 does not appear in the image.

[0055] On the other hand, if the calibration device 10 can identify the calibration markers 3 for all the acquired images (S4: YES), it extracts feature point information for each image (S5) and calculates the deviation from the reference information for each image (S6).The calibration device 10 then determines whether there is a correlation between the deviations of each image (S7), and if it determines that there is no correlation between the deviations (S7: NO), it determines that there is no installation error that would affect the deviation, and estimates that the cause of the deviation is an installation error of the vehicle-mounted camera 2 (S8).

[0056] Next, the calibration device 10 determines whether the deviation is within the tolerance range (S9). For example, if the mounting state of the vehicle-mounted camera 2 is significantly different from the design value, such as if the screws for mounting the vehicle-mounted camera 2 are loose, calibration cannot be performed in that state. Therefore, if the deviation exceeds the tolerance range (S9: NO), the calibration device 10 executes error processing. In this error processing, for example, a prompt to the operator to check the mounting state of the vehicle-mounted camera 2 is executed. The tolerance range for deviation may be set as appropriate.

[0057] On the other hand, if the deviation is within the allowable range (S9: YES), the calibration device 10 identifies (S10) the mounting state of the vehicle-mounted camera 2. At this time, the calibration device 10 identifies the mounting position, mounting orientation, mounting height, and mounting angle of the vehicle-mounted camera 2. Note that the calibration device 10 may be configured to identify at least one of the mounting position, mounting orientation, mounting height, and mounting angle of the vehicle-mounted camera 2.

[0058] Next, the calibration device 10 calibrates the vehicle-mounted camera 2 based on the identified mounting state (S11), and further determines whether the calibration result is within an acceptable range (S12). In other words, the calibration device 10 checks whether the mounting state of the vehicle-mounted camera 2 generally matches the design value, i.e., whether the vehicle-mounted camera 2 is properly mounted. The acceptable range may be set as appropriate. Then, if the calibration result is outside the acceptable range (S12: NO), the calibration device 10 executes error processing. This error processing includes processing such as notifying the operator that the camera mounting state is significantly different from the design value.

[0059] On the other hand, if the calibration result is within the allowable range (S12: YES), the calibration device 10 stores the calibration result, for example, in the storage unit 14 (S13). This saves parameters for correcting the deviation between the actual installation state and the design value, and makes it possible to use the parameters when performing driving assistance, for example.

[0060] If it is determined in step S7 that there is a correlation (S7: YES), the calibration device 10 determines whether the amount of deviation is less than a reference value (S14). This reference value is used to determine whether the deviation is caused by an attachment error or whether the deviation is caused by an installation error, and is set based on design values ​​such as the attachment position of the vehicle-mounted camera 2 and its positional relationship with the calibration marker 3.

[0061] For example, when the vehicle-mounted camera 2 is properly installed, installation errors are thought to occur due to individual differences in the screws, screw holes, and other components. The deviations caused by individual differences are thought to be within approximately a few millimeters. Therefore, the range of deviations that are expected to result from installation errors can be set based on design values ​​such as the installation position of the vehicle-mounted camera 2 and the placement position of the calibration marker 3. Note that when installing the vehicle 1, deviations of approximately 10 millimeters may occur.

[0062] If the deviation amount is less than the reference value (S14: YES), the calibration device 10 determines that there is no installation error of the vehicle 1 that would affect the deviation, and proceeds to step S8, after which it performs the processing from step S9 onwards as described above.

[0063] On the other hand, if the amount of deviation is not less than the reference value (S14: NO), the calibration device 10 estimates that the cause of the deviation includes an installation error of the vehicle 1 (S15), and identifies the installation state of the vehicle 1 (S16). At this time, the calibration device 10 determines the positional relationship of the vehicle 1 with respect to the reference position based on the correlation between the deviations of the images, thereby identifying the actual installation position and installation orientation of the vehicle 1.

[0064] Next, the calibration device 10 calculates a correction value for removing deviations due to installation errors (S17) based on the identified installation position and installation orientation of the vehicle 1. This correction value is used to correct the reference information determined based on the design values ​​to correspond to the actual installation state of the vehicle 1.

[0065] 9 as an example of an image, it is assumed that the calibration marker 3 in the actual image 30 captured by the in-vehicle camera 2A is shifted to the right from the virtual frame (v3), that the shift exceeds a reference value, and that this includes an installation error of the vehicle 1. It is also assumed that it is determined from the identified installation state of the vehicle 1 that the actual vehicle 1 is installed at an angle slightly to the left from the appropriate position illustrated in FIG.

[0066] In this case, although the calibration marker 3 will be shifted to the right from the frame image (v3) serving as reference information in the captured actual image 30, in consideration of the actual installation state of the vehicle 1, the reference frame image (v3) itself should be located further to the right. In other words, if the vehicle 1 is installed at a position shifted from the appropriate position, the positional relationship between the vehicle 1 and the calibration marker 3, that is, the positional relationship between the on-board camera 2 and the calibration marker 3, will also deviate from the design value. In this case, since the positional relationship between the on-board camera 2 and the calibration marker 3 is already shifted from the design value, appropriate calibration cannot be performed if the reference information obtained from the design value is used as is.

[0067] Therefore, the calibration device 10 determines the position of the calibration marker 3 corresponding to the identified installation state based on the installation state, and calculates a correction value (ΔX) for making the reference information based on the design value correspond to the actual installation state (S18), as shown as an example of calculation of the correction value in Fig. 9. Note that the calculation of this correction value is performed for each image of the vehicle-mounted camera 2.

[0068] Then, the calibration device 10 corrects the frame image (v3) based on the design values ​​with the calculated correction value (ΔX), and for example, by shifting the frame image (v3) to the right as shown in Fig. 9 as a correction example, generates a frame image (v3a) that corresponds to the current actual installation state of the vehicle 1. In other words, the calibration device 10 corrects the reference information based on the design values ​​to match the actual installation state of the vehicle 1, thereby generating new reference information from which the installation error has been removed.

[0069] Next, the device for vehicle 1 calculates the deviation using the new reference information (S19), and then proceeds to step S9 and executes the processes from step S9 onward as described above. Note that the same processes are also executed for the other vehicle-mounted cameras 2 to be calibrated.

[0070] In this way, the calibration device 10 estimates the cause of the deviation and makes it possible to calibrate the on-board camera 2 according to the estimated cause of the deviation. This makes it possible to calibrate the on-board camera 2 even if the installation position of the vehicle 1 is misaligned. Furthermore, since a certain degree of deviation in the installation position of the vehicle 1 can be eliminated by correction, there is no need to move the vehicle 1, and it is possible to complete preparations for the calibration work without spending a lot of time on installing the vehicle 1.

[0071] According to the embodiment described above, the following effects can be obtained: The calibration method according to the embodiment includes the steps of capturing images with multiple on-board cameras 2 mounted on a vehicle 1, acquiring each of the images captured by the multiple on-board cameras 2, extracting feature point information indicating calibration markers 3 from the acquired images, calculating, for each image, the deviation between the extracted feature point information and reference information determined from design values, estimating a cause of the deviation based on the correlation between the deviations in the images captured by at least two or more on-board cameras 2, and calibrating the on-board cameras 2 based on the calculated deviation and the estimated cause of the deviation.

[0072] By performing calibration in this way by referring to images captured by multiple vehicle-mounted cameras 2, it becomes possible to estimate the cause of any discrepancy between the feature point information and the reference information when calibrating the vehicle-mounted cameras 2, and to perform appropriate calibration in accordance with the estimated cause. Therefore, it is possible to isolate the cause of the discrepancy, and to calibrate the vehicle-mounted cameras 2 after isolating the cause of the discrepancy.

[0073] Furthermore, in the process of estimating the cause of the deviation, it is estimated whether the cause of the deviation is an installation error of the on-board camera 2 or is due to the installation state of the vehicle 1. As a result, if the cause of the deviation is due to the installation state of the on-board camera 2, calibration can be performed in the same manner as in the past, and if an installation error of the vehicle 1 is included, calibration can be performed according to the installation error.

[0074] Furthermore, if it is estimated that the installation state of the vehicle 1 is one of the causes of the deviation, the method includes a step of calculating a correction value to remove the deviation caused by the installation error, and a step of correcting the reference information using the calculated correction value to generate new reference information from which the installation error has been removed. This makes it possible to calibrate the vehicle-mounted camera 2 in a state where the deviation caused by the installation error has been removed. Furthermore, since it is possible to remove errors caused by slight deviations when installing the vehicle 1, the installation work of the vehicle 1 can be easily performed, and the efficiency of the calibration work can be greatly improved.

[0075] Furthermore, in the process of estimating the cause of the misalignment, if there is a correlation between the misalignments of the images, it is assumed that the cause of the misalignment is an installation error, and if there is no correlation, it is assumed that the cause of the misalignment is an attachment error. This reduces the risk of misjudging the cause of the misalignment.

[0076] Furthermore, in the process of estimating the cause of the misalignment, if the amount of misalignment is less than a predetermined reference value, it is estimated that the cause of the misalignment is an attachment error, and if the amount of misalignment is not less than the reference value, it is estimated that the cause of the misalignment includes an installation error, thereby reducing the risk of erroneously determining the cause of the misalignment.

[0077] The method also includes at least one of a step of identifying at least one of the mounting position, mounting orientation, mounting height, and mounting angle of the vehicle-mounted camera 2, which are factors that cause mounting errors, and a step of identifying the mounting position and mounting orientation of the vehicle 1, which are factors that cause mounting errors. This makes it possible to identify the factors of the errors, and to calibrate the vehicle-mounted camera 2 by making appropriate corrections based on the factors of the errors.

[0078] Furthermore, the process of estimating the cause of the deviation can be configured to use feature point information extracted from images of multiple vehicle-mounted cameras 2 that are installed to capture different fields of view. In this case, the different fields of view may be those in which the installation positions or optical axes of the vehicle-mounted cameras 2 are different, and the fields of view may overlap partially or entirely. Note that the field of view here refers to the imaging range of the vehicle-mounted cameras 2.

[0079] For example, as shown in Fig. 10 as another installation example 1, a configuration in which multiple vehicle-mounted cameras 2 are arranged one above the other, that is, in which the positions are the same in the plan view of Fig. 1 but the heights are different in the side view, the optical axes (J11) and (J12) of the cameras are generally parallel, and the fields of view can be partially overlapped. Also, as shown in Fig. 10 as another installation example 2, the optical axes (J21) and (J22) of the cameras can intersect, and the fields of view can be substantially entirely overlapped.

[0080] 10 as another example of mounting configuration 3, a configuration in which multiple vehicle-mounted cameras 2 are arranged on the left and right, that is, the left and right positions in the plan view of FIG. 1 are approximately the same, the heights in the side view are approximately the same, the optical axes (J31) and (J32) are approximately parallel, and the fields of view do not overlap. Also, as shown in another example of mounting configuration 4, the optical axes (J41) and (J42) of the vehicle-mounted cameras 2 may be arranged to intersect and partially overlap each other.

[0081] By using this configuration, if there is a correlation between the deviations in different fields of view, it can be determined that the correlation is likely to be caused by an installation error of the vehicle 1, and the cause of the deviation can be appropriately estimated.

[0082] The calibration device 10 also includes an imaging instruction unit 12 that executes a process to cause each of the multiple on-board cameras 2 mounted on the vehicle 1 to capture an image; an image acquisition unit 20 that executes a process to acquire each of the images captured by the multiple on-board cameras 2; an extraction unit 21 that executes a process to extract feature point information indicating the calibration markers 3 from the acquired images; a deviation calculation unit 22 that executes a process to calculate, for each image, the deviation between the extracted feature point information and reference information obtained from the design values; a factor estimation unit 23 that executes a process to estimate the cause of the deviation based on the correlation of the deviation in images captured by at least two or more on-board cameras 2; and a calibration unit 24 that executes a process to calibrate the on-board cameras 2 based on the calculated deviation and the estimated cause of the deviation.

[0083] With this configuration, if there is a discrepancy between the feature point information and the reference information when calibrating the vehicle-mounted camera 2, it is possible to estimate the cause of the discrepancy and to perform appropriate calibration in accordance with the estimated cause. Therefore, it is possible to isolate the cause of the discrepancy and, after isolating the cause of the discrepancy, to calibrate the vehicle-mounted camera 2, thereby obtaining the various effects described above, similar to the calibration method.

[0084] Furthermore, the calibration device 10 includes a correction value calculation unit 25 that, when it is estimated that the installation state of the vehicle 1 is included in the causes of the deviation, executes a process of calculating a correction value for removing deviations caused by installation errors, and a new reference generation unit 26 that executes a process of correcting the reference information using the calculated correction value and generating new reference information from which the installation errors have been removed. This makes it possible to calibrate the vehicle-mounted camera 2 in a state in which deviations caused by installation errors have been removed. Furthermore, because errors caused by slight deviations when installing the vehicle 1 can be removed, the installation work of the vehicle 1 can be easily performed, and the efficiency of the calibration work can be greatly improved, thereby achieving the various effects described above, similar to the calibration method.

[0085] In addition to the claims, the present disclosure also includes the following inventions: [1] A method for calibrating an on-board camera, comprising: capturing images with multiple on-board cameras (2) mounted on a vehicle (1); acquiring each of the images captured by the multiple on-board cameras; extracting feature point information indicating a calibration marker (3) from the acquired images; calculating, for each image, a deviation between the extracted feature point information and reference information determined from design values; estimating a cause of the deviation based on a correlation between the deviations in the images captured by at least two or more on-board cameras; and calibrating the on-board camera based on the calculated deviation and the estimated cause of the deviation.

[0086] [2] A method for calibrating an on-board camera according to [1], in which the process of estimating the cause of the deviation estimates whether the cause of the deviation is an installation error of the on-board camera or whether it includes an installation error due to the installation state of the vehicle.

[0087] [3] A method for calibrating an on-board camera according to [2], comprising the steps of: calculating a correction value for removing the deviation caused by the installation error when it is estimated that the cause of the deviation is a vehicle installation error; and correcting the reference information using the calculated correction value to generate new reference information from which the installation error has been removed.

[0088] [4] A method for calibrating an on-board camera according to [2] or [3], in which, in the process of estimating the cause of the deviation, if there is a correlation between the deviations of each image, it is estimated that the cause of the deviation is an installation error, and if there is no correlation, it is estimated that the cause of the deviation is an attachment error.

[0089] [5] A method for calibrating an in-vehicle camera described in any one of [2] to [4], in which, in the process of estimating the cause of the deviation, if the amount of deviation is less than a predetermined reference value, it is estimated that the cause of the deviation is an installation error, and if the amount of deviation is not less than the reference value, it is estimated that the cause of the deviation includes an installation error.

[0090] [6] A method for calibrating an onboard camera according to any one of [2] to [5], including at least one of a step of identifying at least one of the mounting position, mounting orientation, mounting height, and mounting angle of the onboard camera that are factors that cause mounting errors, or a step of identifying the mounting position and mounting orientation of the vehicle that are factors that cause mounting errors.

[0091] [7] A method for calibrating an on-board camera according to any one of [1] to [6], wherein the process of estimating the cause of the deviation uses feature point information extracted from images of multiple on-board cameras mounted to capture different fields of view.

[0092] [8] An imaging instruction unit (12) that executes a process of causing a plurality of vehicle-mounted cameras (2) mounted on a vehicle (1) to capture images,

[0093] An on-board camera calibration device comprising: an image acquisition unit (20) that executes a process to acquire each of images captured by a plurality of on-board cameras; an extraction unit (21) that executes a process to extract feature point information indicating calibration markers from the acquired images; a deviation calculation unit (22) that executes a process to calculate, for each image, a deviation between the extracted feature point information and reference information obtained from design values; a factor estimation unit (23) that executes a process to estimate a factor of deviation based on a correlation between deviations in images captured by at least two or more on-board cameras; and a calibration unit (24) that executes a process to calibrate the on-board camera based on the calculated deviation and the estimated factor of deviation.

[0094] [9] A calibration device for an onboard camera as described in [8], comprising: a correction value calculation unit (25) that, when it is estimated that the cause of the deviation includes an installation error due to the installation state of the vehicle, executes a process of calculating a correction value to remove the deviation due to the installation error; and a new reference generation unit (26) that executes a process of correcting the reference information using the calculated correction value and generating new reference information from which the installation error has been removed.

[0095] The control unit and the method described herein may be implemented by a special-purpose computer configured by configuring a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the control unit and the method described herein may be implemented by a special-purpose computer configured by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method described herein may be implemented by one or more special-purpose computers configured by combining a processor and memory programmed to perform one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory tangible storage medium.

Claims

1. A method for calibrating an on-board camera, comprising: a step of capturing images using a plurality of on-board cameras (2) mounted on a vehicle (1); a step of acquiring each of the images captured by the plurality of on-board cameras; a step of extracting feature point information indicating a calibration marker (3) from the acquired images; a step of calculating, for each image, the deviation between the extracted feature point information and reference information obtained from design values; a step of estimating the cause of the deviation based on the correlation between the deviations in images captured by at least two or more of the on-board cameras; and a step of calibrating the on-board camera based on the calculated deviation and the estimated cause of the deviation.

2. A method for calibrating an on-board camera as described in claim 1, wherein the step of estimating the cause of the deviation estimates whether the cause of the deviation is an installation error of the on-board camera or an installation error caused by the installation state of the vehicle.

3. A method for calibrating an in-vehicle camera as described in claim 2, comprising the steps of: calculating a correction value to remove the deviation caused by the installation error when it is estimated that the cause of the deviation is an installation error of the vehicle; and correcting the reference information using the calculated correction value to generate new reference information from which the installation error has been removed.

4. A method for calibrating an on-board camera as described in claim 2, wherein in the step of estimating the cause of the deviation, if there is a correlation between the deviations of each image, it is estimated that the cause of the deviation is an installation error, and if there is no correlation, it is estimated that the cause of the deviation is an attachment error.

5. A method for calibrating an in-vehicle camera as described in claim 2, wherein in the step of estimating the cause of the deviation, if the amount of deviation is less than a predetermined reference value, it is estimated that the cause of the deviation is an installation error, and if the amount of deviation is not less than the reference value, it is estimated that the cause of the deviation includes an installation error.

6. A method for calibrating an on-board camera as described in claim 2, including at least one of a step of identifying at least one of the mounting position, mounting orientation, mounting height and mounting angle of the on-board camera that are factors that cause mounting errors, or a step of identifying the mounting position and mounting orientation of the vehicle that are factors that cause mounting errors.

7. A method for calibrating an on-board camera according to claim 1, wherein the step of estimating the cause of the deviation uses feature point information extracted from images taken by a plurality of on-board cameras mounted so as to capture different fields of view.

8. A vehicle-mounted camera calibration device comprising: an imaging instruction unit (12) that executes a process to cause each of multiple vehicle-mounted cameras (2) mounted on a vehicle (1) to capture an image; an image acquisition unit (20) that executes a process to acquire each of the images captured by the multiple vehicle-mounted cameras; an extraction unit (21) that executes a process to extract feature point information indicating a calibration marker from the acquired image; a deviation calculation unit (22) that executes a process to calculate, for each image, the deviation between the extracted feature point information and reference information obtained from design values; a factor estimation unit (23) that executes a process to estimate a factor of deviation based on the correlation of deviation in images captured by at least two or more of the vehicle-mounted cameras; and a calibration unit (24) that executes a process to calibrate the vehicle-mounted camera based on the calculated deviation and the estimated factor of deviation.

9. A vehicle-mounted camera calibration device as described in claim 8, comprising: a correction value calculation unit (25) that, when it is estimated that the cause of the deviation includes an installation error due to the installation state of the vehicle, executes a process of calculating a correction value to remove the deviation due to the installation error; and a new reference generation unit (26) that executes a process of correcting the reference information using the calculated correction value and generating new reference information from which the installation error has been removed.

Images