Calibration method for on-vehicle camera and calibration device for on-vehicle camera

Abstract

To provide a method and a device for calibrating an on-vehicle camera, which can isolate the cause of deviation and calibrate the on-vehicle camera after isolating the cause of deviation.SOLUTION: A calibration method according to an embodiment includes the steps of capturing images using multiple on-vehicle cameras 2 mounted on a vehicle 1, acquiring each of the images captured by the multiple on-vehicle cameras 2, extracting feature point information indicating a calibration marker 3 from the acquired images, calculating the deviation between the extracted feature point information and reference information obtained from design values for each image, estimating the cause of the deviation on the basis of the correlation between the deviations in images captured by at least two or more on-vehicle cameras 2, and calibrating the on-vehicle cameras 2 on the basis of the calculated deviation and the estimated cause of the deviation.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to a calibration method and a calibration device for calibrating an in-vehicle camera. [Background technology]

[0002] In recent years, the number of vehicles equipped with on-board cameras capable of capturing images of the exterior of the vehicle has been increasing. Such on-board cameras are used for driving assistance, such as for positioning the vehicle while parking or driving, by processing the captured video and images. Therefore, high calibration accuracy is required. For example, Patent Document 1 describes a method for appropriately calibrating on-board cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5091902 Summary of the Invention [Problem to be solved by the invention]

[0004] Calibration of an on-board camera is generally performed using a calibration marker having a predetermined size of figure, symbol, or letter, etc. Specifically, the calibration marker and the vehicle are installed so as to have a predetermined positional relationship, feature point information indicating the position, size, etc. of the calibration marker is extracted from an image captured by the on-board camera, and the feature point information is compared with reference information indicating the position, size, etc. of the designed calibration marker, which is obtained based on design values ​​such as the installation position of the on-board camera, thereby calibrating the on-board camera.

[0005] However, in the past, each on-board camera was calibrated independently, and the only information used during calibration was feature point information extracted from the image captured by that on-board camera. In this case, for example, if the installation position of the vehicle is misaligned, the appearance of the calibration marker changes, resulting in errors in the extracted feature point information. Furthermore, in the past, when there was a discrepancy between the feature point information and the reference information, it was not possible to determine whether the cause of the discrepancy was an installation error due to the installation state of the on-board camera or an installation error due to the installation state of the vehicle.

[0006] The present disclosure has been made in consideration of the above circumstances, and its purpose is to provide a calibration method and calibration device for an on-board camera that can isolate the cause of the deviation and calibrate the on-board camera after isolating the cause of the deviation. [Means for solving the problem]

[0007] The method for calibrating an on-board camera according to the embodiment includes the steps of capturing images using multiple on-board cameras (2) mounted on a vehicle (1), acquiring each of the images captured by the multiple on-board cameras, extracting feature point information indicating a calibration marker (3) from the acquired images, calculating the deviation between the extracted feature point information and reference information obtained from design values ​​for each image, estimating the cause of the deviation based on the correlation between the deviations in images captured by at least two or more on-board cameras, and calibrating the on-board camera based on the calculated deviation and the estimated cause of the deviation.

[0008] In addition, the vehicle-mounted camera calibration device according to the embodiment includes an imaging instruction unit (12) that executes a process to cause each of multiple vehicle-mounted cameras (2) mounted on the vehicle (1) to capture an image, an image acquisition unit (20) that executes a process to acquire each of the images captured by the multiple vehicle-mounted cameras, an extraction unit (21) that executes a process to extract feature point information indicating a calibration marker from the acquired image, a deviation calculation unit (22) that executes a process to calculate, for each image, the deviation between the extracted feature point information and reference information obtained from design values, a factor estimation unit (23) that executes a process to estimate a factor of the deviation based on the correlation of the deviation in images captured by at least two or more vehicle-mounted cameras, and a calibration unit (24) that executes a process to calibrate the vehicle-mounted camera based on the calculated deviation and the estimated factor of the deviation. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of an installation mode of an in-vehicle camera according to an embodiment; [Figure 2] FIG. 1 is a diagram showing an example of the angle of view of an in-vehicle camera; [Figure 3] FIG. 10 is a diagram showing an example of a reference position and a vehicle installation state during calibration work; [Figure 4] FIG. 1 is a diagram schematically illustrating an example of a calibration marker; [Figure 5] FIG. 1 is a diagram illustrating an example of an electrical configuration of a calibration device. [Figure 6] A simple diagram explaining the calibration process [Figure 7] FIG. 1 is a diagram illustrating an example of a cause of deviation. [Figure 8] Flow of calibration process [Figure 9] FIG. 10 is a diagram illustrating an aspect of correcting reference information. [Figure 10] FIG. 10 is a diagram showing another example of an installation mode of the vehicle-mounted camera; DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment will be described with reference to the drawings. As shown in Fig. 1, a vehicle 1 is equipped with a plurality of on-board cameras 2 to be calibrated. In this embodiment, the vehicle 1 is equipped with an on-board camera 2A mounted in a direction to capture an image ahead, an on-board camera 2B mounted in a direction to capture an image behind the vehicle 1, an on-board camera 2C mounted in a direction to capture an image to the left side of the vehicle 1, and an on-board camera 2D mounted in a direction to capture an image to the right side of the vehicle 1.

[0011] 1 is defined as a reference position (P), a reference line passing through the reference position (P) and extending in the longitudinal direction of the vehicle 1 is defined as a virtual line (LX), and a reference line passing through the reference position (P) and extending in the lateral direction of the vehicle 1 is defined as a virtual line (LY). Note that the number, mounting positions, and angles of the vehicle-mounted cameras 2 shown in FIG. 1 are merely examples and are not limiting.

[0012] The mounting position of each vehicle-mounted camera 2 is determined by a design value. For example, as shown in a plan view, vehicle-mounted camera 2A is mounted on a virtual line (LX) at a mounting position a distance (L1) rearward from a reference position (P) so that the angle of view is directed forward of vehicle 1. Also, as shown in a side view, vehicle-mounted camera 2A is mounted at a position a predetermined height (H1) from the installation surface of vehicle 1, at an installation angle such that its optical axis is parallel to the installation surface. As shown in FIG. 2, the imaging range of vehicle-mounted camera 2A is a predetermined angle of view (RA) forward of vehicle 1.

[0013] The vehicle-mounted camera 2B is mounted on a virtual line (LX) at a mounting position a distance (L2) rearward from the reference position (P) so that the angle of view faces rearward of the vehicle 1. The vehicle-mounted camera 2B is mounted at a position a predetermined height (H2) from the installation surface of the vehicle 1 as shown in a side view, at a mounting angle such that its optical axis is parallel to the installation surface. The imaging range of the vehicle-mounted camera 2B is a predetermined angle of view (RB) behind the vehicle 1, as shown in FIG.

[0014] The vehicle-mounted camera 2C is mounted at a mounting position that is a distance (W1) to the left of the virtual line (LX) and a distance (L3) behind the reference position (P), with the angle of view facing the left side of the vehicle 1. The vehicle-mounted camera 2C is mounted at a position that is a predetermined height (H3) from the installation surface of the vehicle 1, as shown in a side view, with an installation angle that makes its optical axis parallel to the installation surface. The imaging range of the vehicle-mounted camera 2C is a predetermined angle of view (RC) on the left side of the vehicle 1, as shown in FIG. 2.

[0015] The vehicle-mounted camera 2D is mounted at a mounting position that is a distance (W1) to the right of the virtual line (LX) and a distance (L3) rearward from the reference position (P), with the angle of view oriented toward the right side of the vehicle 1. The vehicle-mounted camera 2D is mounted at a position that is a predetermined height (H3) from the installation surface of the vehicle 1, as shown in a side view, with an installation angle that makes its optical axis parallel to the installation surface. As shown in FIG. 2, the vehicle-mounted camera 2D has an imaging range that is a predetermined angle of view (RD) to the right side of the vehicle 1. The angle of view shown in FIG. 2 is an example and is not limiting. The vehicle-mounted camera 2D may be mounted inside the vehicle, for example, near the back of the rearview mirror, or outside the vehicle, for example, at the tip of a side mirror.

[0016] Calibration of such an in-vehicle camera 2 is performed by placing the vehicle 1 in a predetermined positional relationship with respect to calibration markers 3 that are placed at predetermined positions as shown in Fig. 3. In this embodiment, a reference position (K), a virtual line (KX) that passes through the reference position (K), and a virtual line (KY) that passes through the reference position (K) and intersects the virtual line (KX) at right angles are set in the work area where calibration is performed. The calibration marker 3 is placed so as to have a predetermined positional relationship with respect to the reference position (K), virtual line (KX), and virtual line (KY). Note that the number and placement of the calibration markers 3 and their positional relationship with the vehicle 1 shown in Fig. 3 are merely examples and are not limited to these.

[0017] For example, the calibration marker 3 is positioned in front of the vehicle 1 such that the marker center (M1) described below is located at a distance (L11) above the reference position (K) and a distance (W11) to the left of the virtual line (KX) in the figure, and the marker center (M1) described below is located at a distance (L11) above the reference position (K) and a distance (W11) to the right of the virtual line (KX) in the figure.

[0018] In addition, the calibration marker 3 is positioned behind the vehicle 1 so that its center (M1) is a distance (L12) below the reference position (K) in the figure and a distance (W12) to the left of the virtual line (KX) in the figure, and so that its center (M1) is a distance (L12) below the reference position (K) in the figure and a distance (W12) to the right of the virtual line (KX) in the figure.

[0019] In addition, the calibration marker 3 is positioned on the left side of the vehicle 1 so that its center (M1) is a distance (L13) downward from the reference position (K) in the figure and a distance (W13) to the left from the virtual line (KX) in the figure, and so that its center (M1) is a distance (L14) downward from the reference position (K) in the figure and a distance (W13) to the left from the virtual line (KX) in the figure.

[0020] In addition, the calibration marker 3 is positioned on the right side of the vehicle 1 so that its marker center (M1) is a distance (L13) downward from the reference position (K) in the figure and a distance (W13) to the right from the virtual line (KX) in the figure, and so that its marker center (M1) is a distance (L14) downward from the reference position (K) in the figure and a distance (W13) to the right from the virtual line (KX) in the figure.

[0021] 4, the calibration marker 3 is formed in the shape of a plate with a width of L21 and a height of L22, for example, and has a black and white image drawn on its surface. In this embodiment, white rectangles with a width of L21a and a height of L22a and black rectangles with a width of L21a and a height of L22a are arranged alternately. The intersection of an imaginary line (CL21) extending vertically in the figure, which is the boundary between each image, and an imaginary line (CL22) extending horizontally in the figure, is the marker center (M1) indicating the center of the calibration marker 3.

[0022] These calibration markers 3 are supported by support members 4 as shown in a side view in Fig. 3, and are arranged as shown in a plan view with the marker centers (M1) at a predetermined height (H11) from the installation surface. Note that the shapes of the calibration markers 3 and the drawn figures shown in Fig. 4 are merely examples and are not limiting. For example, circular figures, polygonal figures, or drawn characters, symbols, etc. may also be used.

[0023] Now, the calibration of the vehicle-mounted camera 2 is performed in a state where the reference position (P) of the vehicle 1 and the reference position (K) of the calibration marker 3 are aligned, as shown in the plan view of Fig. 3. Specifically, the calibration of the vehicle-mounted camera 2 is performed in a state where the reference position (P) and the reference position (K) are aligned, the virtual lines (LX) and the virtual lines (KX) are aligned, and the virtual lines (LY) and the virtual lines (KY) are aligned. Hereinafter, the reference position (P) will also be referred to as the reference on the vehicle-mounted camera 2 side, and the reference position (K) will also be referred to as the reference on the calibration marker 3 side, and for convenience, the state where the vehicle 1 is installed so that the reference on the vehicle-mounted camera 2 side and the reference on the calibration marker 3 side are aligned will be referred to as the appropriate position.

[0024] In this embodiment, each calibration marker 3 is placed at the work site in advance, and the vehicle 1 is moved to the work site and positioned so that the reference on the in-vehicle camera 2 side and the reference on the calibration marker 3 side coincide with each other, and then the calibration is performed using the calibration device 10 shown in Figure 5.

[0025] The calibration device 10 is entirely controlled by a microcomputer (not shown), and includes functional units such as an image input unit 11, an image capture instruction unit 12, an image processing unit 13, a memory unit 14, a display control unit 15, and an operation control unit 16. In this embodiment, the image capture instruction unit 12 and the image processing unit 13 are realized by software by executing a program on the microcomputer. The calibration device 10 can be configured as a dedicated device, but some or all of the functional units can also be implemented in an electronic control device such as a camera ECU (not shown) for controlling the on-board camera 2 mounted on the vehicle 1 or an HMIECU (not shown) for controlling the human-machine interface.

[0026] The image input unit 11 is an interface to which images captured by the vehicle-mounted camera 2 are input, and receives a video signal from the vehicle-mounted camera 2. The image input unit 11 also transmits a signal to instruct the vehicle-mounted camera 2 to capture an image. Note that when the vehicle-mounted camera 2 is controlled by a camera ECU, the image input unit 11 is configured as a so-called communication unit that communicates with the camera ECU.

[0027] The imaging instruction unit 12 generates an instruction to capture an image for calibration and outputs it to the vehicle-mounted camera 2 via the image input unit 11. The storage unit 14 is configured with a storage medium such as a semiconductor memory, and stores various programs for controlling the calibration device 10 and for realizing each functional unit, as well as a reference information DB 17 in which reference information for calibration is collected, the details of which will be described later.

[0028] During the calibration work, the display control unit 15 controls the display of various information on the display 18. At this time, a center display mounted on the vehicle 1 can be used as the display 18.

[0029] The operation control unit 16 accepts operations input by an operator to the operation unit 19, such as an operation to instruct imaging or an operation to perform calibration work. In this case, the operation unit 19 may be configured to use a touch panel or the like mounted on the vehicle 1. Note that the display control unit 15 and the operation control unit 16 are not essential for the calibration device 10, and the calibration device 10 may be configured to be connected to an external device such as a personal computer via a communication line, in which case, for example, a display or keyboard on the external device side may be used.

[0030] The image processing unit 13 has various functional units that execute various processes related to the calibration of the vehicle-mounted cameras 2. Among these, the image acquisition unit 20 executes a process of acquiring each of the images captured by the multiple vehicle-mounted cameras 2. At this time, the image acquisition unit 20 acquires the images by temporarily storing the video signal input to the image input unit 11 in the storage unit 14 or the like.

[0031] The extraction unit 21, the details of which will be described later, executes a process of extracting feature point information indicating the calibration marker 3 from the acquired image. For example, as shown in an image capture example in FIG. 6 , when the calibration marker 3 is captured in an actual image 30 at a certain angle of view (R) by the vehicle-mounted camera 2 to be calibrated, the extraction unit 21 extracts feature point information indicating the position of the marker center (M1) in the actual image 30. The extraction unit 21 can also extract other information that can be obtained from the actual image 30, such as the distance from the vehicle-mounted camera 2 to the calibration marker 3, which is obtained from the size of the calibration marker 3 captured in the actual image 30, and the orientation of the vehicle-mounted camera 2 with respect to the calibration marker 3, which is obtained from the direction and length ratio of each side of the calibration marker 3, as actual image information. This actual image information is used as appropriate in processes such as estimating the cause of deviation, which will be described later.

[0032] Well-known techniques can be used to extract feature point information, and will be explained briefly here: the edges of a figure are detected by horizontally scanning and vertically manipulating the image, the figure shown in the image is detected based on the detected edges, the detected figure is compared with the figure of the calibration marker 3 by, for example, image matching to identify the position of the feature point indicating the calibration marker 3, and feature point information indicating the positional relationship between the vehicle-mounted camera 2 and the calibration marker 3 is extracted from the identified position of the feature point in the image. Note that the feature point information is extracted as so-called numerical data within the calibration device 10, but it is also possible to overlay on the display 18 a colored frame image indicating the calibration marker 3 or a round image indicating the marker center (M1), etc., to make it easier for the worker to understand the work content.

[0033] The deviation calculation unit 22 executes a process for calculating the deviation between the extracted feature point information and reference information obtained from the design values ​​for each image captured by the multiple vehicle-mounted cameras 2. This reference information is information indicating the position and size of the calibration marker 3 that should appear in the angle of view (R) when the vehicle-mounted camera 2 to be calibrated is attached at a position according to the design values ​​and the vehicle 1 is in an appropriate position. This reference information is basically handled as numerical data within the calibration device 10, but to make it easier for the worker to check the work content, images such as a reference image v30 indicating the angle of view in the reference information, a virtual frame (v3) that virtually shows the calibration marker 3 in the reference image v30, and a virtual center (vM1) that shows the center of the virtual frame (v3) are overlaid and displayed as an example of reference information in FIG. 6 .

[0034] The factor estimation unit 23 executes a process to estimate the factor of the deviation that occurred in the image captured by the in-vehicle camera 2 to be calibrated, based on the correlation of deviations in the images captured by at least two or more in-vehicle cameras 2. Details of the correlation will be described later. The factor estimation unit 23 also estimates whether the factor of the deviation is an installation error caused by the installation state of the in-vehicle camera 2, or whether the factor of the deviation includes an installation error caused by the installation state of the vehicle 1.

[0035] 7 as a comparative example, suppose that there is a discrepancy between the feature point information and the reference information in an actual image 30 captured by the vehicle-mounted camera 2 to be calibrated. In this comparative example, the marker center (M1) of the calibration marker 3 and the virtual center (vM1) coincide with each other, but the size of the calibration marker 3 is smaller than the virtual frame (v3). In this case, the cause of the discrepancy may be an attachment error in which the attachment state of the vehicle-mounted camera 2 deviates from the design value. However, when calibration is performed by moving the vehicle 1 as in this embodiment, there is also a possibility that an installation error in which the vehicle 1 is displaced from the appropriate position may also be included.

[0036] Therefore, the factor estimation unit 23 checks whether there is a correlation between the factors of the deviation of the actual image 30 captured by the in-vehicle camera 2 to be calibrated and the deviation of an image captured by another in-vehicle camera 2, and estimates the factors based on the correlation. In other words, the factor estimation unit 23 determines whether the deviation factors include installation error based on the correlation of the deviations.

[0037] The calibration unit 24 calibrates the vehicle-mounted camera 2 based on the calculated deviation and the estimated cause of the deviation. Calibration of the vehicle-mounted camera 2 can be performed using well-known techniques, so a brief explanation will be given here. When calibrating the vehicle-mounted camera 2, the calibration unit 24 first identifies the installation state of the vehicle-mounted camera 2, such as the installation position, installation angle, or installation height, based on the position, size, distortion, etc. of the measurement markers shown in the image. The calibration unit 24 also identifies the installation position and installation orientation of the vehicle 1, as will be described later.

[0038] Then, when there is a misalignment between the actual image 30 and the reference image v30, as shown in the calibration example in Fig. 6, the calibration unit 24 calculates parameters to correct the misalignment. The calculated parameters are used, for example, when providing driving assistance to measure the distance and positional relationship with surrounding objects, and enable appropriate assistance to be provided according to the actual installation state. Note that lens distortion correction, etc. may also be performed during calibration.

[0039] The correction value calculation unit 25, the details of which will be described later, executes a process of calculating a correction value for removing the deviation caused by the installation error when it is estimated that the deviation is caused by an installation error.

[0040] The new reference generation unit 26, the details of which will be described later, corrects the reference information using the calculated correction value and executes a process of generating new reference information from which the installation error has been removed. In other words, the new reference generation unit 26 recreates the reference information based on the design values ​​so that it corresponds to the positional relationship between the actual vehicle 1 and the calibration marker 3.

[0041] Next, the operation and effects of the above-described configuration will be described. First, the details of the correlation between the factors that cause the deviation and the deviation will be described. For example, as shown in the RA comparison in Fig. 7, in an actual image 30A in which the calibration marker 3 is captured by an in-vehicle camera 3A that captures an image ahead of the vehicle 1, the marker center (M1) and the virtual center (vM1) roughly coincide with each other, but the calibration marker 3 is smaller than the virtual frame (v3).

[0042] In this case, the cause of the deviation may be that, although the mounting position of the vehicle-mounted camera 2A is appropriate, as shown schematically in Example 1, the installation position of the vehicle 1 is shifted rearward by a distance (ΔL) from the reference position (P), and due to this distance (ΔL), the calibration marker 3 appears smaller than the virtual frame (v3).

[0043] On the other hand, as a cause of the deviation, as shown schematically in Example 2, it is possible that the installation position of the vehicle 1 is appropriate, but the mounting position of the vehicle-mounted camera 2A is shifted a distance (ΔL) further back than the appropriate distance (L1), and that this distance (ΔL) causes the calibration marker 3 to appear smaller than the virtual frame (v3).

[0044] In this case, it is difficult to distinguish whether the cause of the deviation is an installation error or an attachment error in the actual image 30A captured by one vehicle-mounted camera 2A as shown in the RA comparison. Also, if the installation position of the vehicle 1 is misaligned as in factor example 1, the installation position of the vehicle-mounted camera 2A may also be misaligned, and if the installation position of the vehicle-mounted camera 2A is misaligned as in factor example 2, the installation position of the vehicle 1 may also be misaligned.

[0045] In other words, it is difficult to determine whether the cause of the deviation is an installation error of the vehicle 1 from only the actual image 30 captured by one in-vehicle camera 2A. Therefore, conventional methods cannot isolate the cause of the deviation, and there is a possibility that calibration will be performed in a state that includes an installation error of the vehicle 1. If calibration is performed in a state that includes an error, there is a risk that this will affect the accuracy of driving assistance, such as estimating the distance to a surrounding object using the in-vehicle camera 2.

[0046] Therefore, in this embodiment, the correlation between two or more images is utilized. For example, as shown in RB comparison example 1 in Fig. 7, in an actual image 30B captured by an in-vehicle camera 2B capturing an image of the rear of the vehicle 1, the marker center (M1) and the virtual center (vM1) are approximately aligned, and the calibration marker 3 is larger than the virtual frame (v3). In this case, the distance between the in-vehicle camera 2B and the calibration marker 3 is closer than the reference distance. The distance can be determined from the actual image information.

[0047] That is, by referring to the images captured by the on-board camera 2A and the on-board camera 2B, it is possible to obtain a correlation that is presumed to be caused by the installation state of the vehicle 1, in which the on-board camera 2A is farther away from the calibration marker 3 than the reference, and the on-board camera 2B is closer to the calibration marker 3 than the reference. From this correlation, it can be presumed that the installation error of the vehicle 1, shown as factor example 1, is included in the factors that cause the deviation.

[0048] Furthermore, as shown in RB comparison example 2, in actual image 30 captured by in-vehicle camera 2B, the marker center (M1) and the virtual center (vM1) roughly coincide with each other, and their sizes also roughly coincide with each other. In this case, in-vehicle camera 2B is positioned as per the reference relative position to calibration marker 3. In other words, actual image 30 captured by in-vehicle camera 2A and in-vehicle camera 2B shows that in-vehicle camera 2A is farther away from calibration marker 3 than the reference position, while in-vehicle camera 2B is attached at the reference position.

[0049] In this case, it can be determined that there is no correlation that is caused by the installation state of the vehicle 1. In other words, from the correlation of the deviations in the multiple photographed images 30, it can be estimated that the deviation is caused by the installation error of the vehicle-mounted camera 2, which is shown as factor 2.

[0050] Such a correlation is not limited to the longitudinal direction. For example, if the vehicle 1 is installed so as to be displaced counterclockwise around the vehicle-mounted camera 2A shown in Fig. 2, a correlation is observed in each captured image in which the calibration marker 3 is displaced to the right of the reference. Furthermore, if the vehicle 1 is displaced entirely to the left of the virtual line (LX), a correlation is observed in which the calibration marker 3 is displaced to the right in the image captured by the vehicle-mounted camera 2A, while the calibration marker 3 is displaced to the left in the image captured by the vehicle-mounted camera 2B. Note that such a correlation is not limited to between the vehicle-mounted cameras 2A and 2B, but may also be observed between the vehicle-mounted cameras 2C and 2D.

[0051] Incidentally, calibration of the vehicle-mounted camera 2 is basically performed for all vehicles 1 equipped with the vehicle-mounted camera 2. Furthermore, careful work is required to accurately install the vehicles 1 at the reference positions, and the work time inevitably increases as the number of vehicles 1 to be worked on increases. Therefore, even if there is an installation error in the vehicles 1, it is believed that work efficiency can be significantly improved if the vehicle-mounted cameras 2 can be calibrated with the installation error removed.

[0052] Therefore, the calibration device 10 makes it possible to isolate the cause of the deviation as described below, and even if there is an installation error of the vehicle 1, it makes it possible to perform calibration with the installation error removed. Below, a calibration method using the calibration device 10 will be described. Note that although each process described below is performed by the functional units described above sharing or cooperating with each other, for the sake of simplicity, the description will be centered on the calibration device 10. It is also assumed that the vehicle 1 has already been installed.

[0053] 8, the calibration device 10 instructs each vehicle-mounted camera 2 to capture an image (S1) and acquires the images captured by each vehicle-mounted camera 2 (S2). Next, the calibration device 10 identifies the calibration marker 3 for each image (S3). That is, the calibration device 10 checks whether the calibration marker 3 appears in the acquired image. If the calibration device 10 cannot identify the calibration marker 3 for any of the images (S4: NO), the calibration device 10 executes error processing. In this error processing, for example, a process is executed to notify the operator that the calibration marker 3 does not appear in the image.

[0054] On the other hand, if the calibration device 10 can identify the calibration markers 3 for all the acquired images (S4: YES), it extracts feature point information for each image (S5) and calculates the deviation from the reference information for each image (S6).The calibration device 10 then determines whether there is a correlation between the deviations of each image (S7), and if it determines that there is no correlation between the deviations (S7: NO), it determines that there is no installation error that would affect the deviation, and estimates that the cause of the deviation is an installation error of the vehicle-mounted camera 2 (S8).

[0055] Next, the calibration device 10 determines whether the deviation is within an allowable range (S9). For example, if the mounting state of the vehicle-mounted camera 2 is significantly different from the design value, such as if the screws for mounting the vehicle-mounted camera 2 are loose, calibration cannot be performed in that state. Therefore, if the deviation exceeds the allowable range (S9: NO), the calibration device 10 executes error processing. In this error processing, for example, a prompt to the operator to check the mounting state of the vehicle-mounted camera 2 is carried out. The allowable range of deviation may be set as appropriate.

[0056] On the other hand, if the deviation is within the allowable range (S9: YES), the calibration device 10 determines (S10) the installation state of the vehicle-mounted camera 2. At this time, the calibration device 10 determines the installation position, installation orientation, installation height, and installation angle of the vehicle-mounted camera 2. Note that the calibration device 10 may be configured to determine at least one of the installation position, installation orientation, installation height, and installation angle of the vehicle-mounted camera 2.

[0057] Next, the calibration device 10 calibrates the vehicle-mounted camera 2 based on the identified mounting state (S11), and further determines whether the calibration result is within an acceptable range (S12). In other words, the calibration device 10 checks whether the mounting state of the vehicle-mounted camera 2 generally matches the design value, i.e., whether the vehicle-mounted camera 2 is properly mounted. The acceptable range may be set as appropriate. If the calibration result is outside the acceptable range (S12: NO), the calibration device 10 executes error processing. This error processing includes processing to notify the operator that the camera mounting state is significantly different from the design value.

[0058] On the other hand, if the calibration result is within the allowable range (S12: YES), the calibration device 10 stores the calibration result, for example, in the storage unit 14 (S13). This saves parameters that correct the deviation between the actual installation state and the design value, and makes it possible to use the parameters when performing driving assistance, for example.

[0059] If it is determined in step S7 that there is a correlation (S7: YES), the calibration device 10 determines whether the amount of deviation is less than a reference value (S14). This reference value is used to determine whether the deviation is caused by an attachment error or whether the deviation is caused by an installation error, and is set based on design values ​​such as the attachment position of the vehicle-mounted camera 2 and its positional relationship with the calibration marker 3.

[0060] For example, if the vehicle-mounted camera 2 is properly installed, installation errors are thought to occur due to individual differences in the screws, screw holes, and other components. The deviations caused by individual differences are thought to be within approximately a few millimeters. Therefore, the range of deviations that are expected to result from installation errors can be set based on design values ​​such as the installation position of the vehicle-mounted camera 2 and the placement position of the calibration marker 3. Note that when installing the vehicle 1, deviations of approximately 10 millimeters may occur.

[0061] If the deviation amount is less than the reference value (S14: YES), the calibration device 10 determines that there is no installation error in the vehicle 1 that is large enough to affect the deviation, and then proceeds to step S8, and performs the processing from step S9 onwards as described above.

[0062] On the other hand, if the amount of deviation is not less than the reference value (S14: NO), the calibration device 10 estimates that the cause of the deviation is an installation error of the vehicle 1 (S15) and identifies the installation state of the vehicle 1 (S16). At this time, the calibration device 10 determines the positional relationship of the vehicle 1 with respect to the reference position based on the correlation between the deviations of each image, thereby identifying the actual installation position and installation orientation of the vehicle 1.

[0063] Next, the calibration device 10 calculates a correction value for removing deviations due to installation errors based on the identified installation position and installation orientation of the vehicle 1 (S17). This correction value is used to correct the reference information determined based on the design values ​​to correspond to the actual installation state of the vehicle 1.

[0064] 9 as an example of an image, it is assumed that the calibration marker 3 in the actual image 30 captured by the in-vehicle camera 2A is shifted to the right from the virtual frame (v3), that the shift exceeds a reference value, and that this includes an installation error of the vehicle 1. Then, it is assumed that it is determined from the identified installation state of the vehicle 1 that the actual vehicle 1 is installed at an angle slightly to the left from the appropriate position illustrated in FIG.

[0065] In this case, although the calibration marker 3 will be shifted to the right from the frame image (v3) serving as reference information in the captured actual image 30, the frame image (v3) serving as reference information should itself be located further to the right in consideration of the actual installation state of the vehicle 1. In other words, if the vehicle 1 is installed at a position shifted from the appropriate position, the positional relationship between the vehicle 1 and the calibration marker 3, that is, the positional relationship between the on-board camera 2 and the calibration marker 3, will also deviate from the design value. In this case, since the positional relationship between the on-board camera 2 and the calibration marker 3 is already shifted from the design value, appropriate calibration cannot be performed if the reference information obtained from the design value is used as is.

[0066] Therefore, the calibration device 10 determines the position of the calibration marker 3 corresponding to the identified installation state based on the installation state, and calculates a correction value (ΔX) for making the reference information based on the design value correspond to the actual installation state (S18), as shown as an example of calculation of the correction value in Fig. 9. Note that the calculation of this correction value is performed for each image of the vehicle-mounted camera 2.

[0067] Then, the calibration device 10 corrects the frame image (v3) based on the design values ​​with the calculated correction value (ΔX), and for example, by shifting the frame image (v3) to the right as shown in Fig. 9 as a correction example, generates a frame image (v3a) that corresponds to the current actual installation state of the vehicle 1. In other words, the calibration device 10 corrects the reference information based on the design values ​​to match the actual installation state of the vehicle 1, thereby generating new reference information from which the installation error has been removed.

[0068] Next, the device for vehicle 1 calculates the deviation using the new reference information (S19), and then proceeds to step S9 and executes the processes from step S9 onward as described above. Note that the same processes are also executed for the other vehicle-mounted cameras 2 to be calibrated.

[0069] In this way, the calibration device 10 estimates the cause of the deviation and makes it possible to calibrate the vehicle-mounted camera 2 according to the estimated cause of the deviation. This makes it possible to calibrate the vehicle-mounted camera 2 even if the installation position of the vehicle 1 is misaligned. Furthermore, since a certain degree of deviation in the installation position of the vehicle 1 can be eliminated by correction, there is no need to move the vehicle 1, and it is possible to complete preparations for the calibration work without spending a lot of time setting up the vehicle 1.

[0070] According to the embodiment described above, the following effects can be obtained. The calibration method according to the embodiment includes the steps of capturing images using multiple on-board cameras 2 mounted on a vehicle 1, acquiring each of the images captured by the multiple on-board cameras 2, extracting feature point information indicating a calibration marker 3 from the acquired images, calculating the deviation between the extracted feature point information and reference information obtained from design values ​​for each image, estimating the cause of the deviation based on the correlation between the deviations in images captured by at least two or more on-board cameras 2, and calibrating the on-board cameras 2 based on the calculated deviation and the estimated cause of the deviation.

[0071] By performing calibration in this way with reference to images captured by multiple vehicle-mounted cameras 2, it becomes possible to estimate the cause of any discrepancy between the feature point information and the reference information when calibrating the vehicle-mounted cameras 2, and to perform appropriate calibration in accordance with the estimated cause. Therefore, it is possible to isolate the cause of the discrepancy, and to calibrate the vehicle-mounted cameras 2 after isolating the cause of the discrepancy.

[0072] Furthermore, in the process of estimating the cause of the deviation, it is estimated whether the cause of the deviation is an installation error of the on-board camera 2 or is due to the installation state of the vehicle 1. As a result, if the cause of the deviation is due to the installation state of the on-board camera 2, calibration can be performed in the same way as in the past, and if an installation error of the vehicle 1 is included, calibration can be performed according to the installation error.

[0073] Furthermore, if it is estimated that the installation state of the vehicle 1 is one of the causes of the deviation, the method includes a step of calculating a correction value to remove the deviation caused by the installation error, and a step of correcting the reference information using the calculated correction value to generate new reference information from which the installation error has been removed. This allows the vehicle-mounted camera 2 to be calibrated in a state in which the deviation caused by the installation error has been removed. Furthermore, since the error caused by a slight deviation when installing the vehicle 1 can be removed, the installation work of the vehicle 1 can be easily performed, and the efficiency of the calibration work can be greatly improved.

[0074] Furthermore, in the process of estimating the cause of the misalignment, if there is a correlation between the misalignments of the images, it is assumed that the cause of the misalignment is an installation error, and if there is no correlation, it is assumed that the cause of the misalignment is an attachment error. This reduces the risk of misjudging the cause of the misalignment.

[0075] Furthermore, in the process of estimating the cause of the misalignment, if the amount of misalignment is less than a predetermined reference value, it is estimated that the cause of the misalignment is an attachment error, and if the amount of misalignment is not less than the reference value, it is estimated that the cause of the misalignment includes an installation error, thereby reducing the risk of erroneously determining the cause of the misalignment.

[0076] The method also includes at least one of a step of identifying at least one of the mounting position, mounting orientation, mounting height, and mounting angle of the vehicle-mounted camera 2, which are factors that cause mounting errors, and a step of identifying the mounting position and mounting orientation of the vehicle 1, which are factors that cause mounting errors. This makes it possible to identify the factors of the errors, and to calibrate the vehicle-mounted camera 2 by making appropriate corrections based on the factors of the errors.

[0077] Furthermore, the process of estimating the cause of the deviation can be configured to use feature point information extracted from images of multiple vehicle-mounted cameras 2 that are installed to capture different fields of view. In this case, the different fields of view only need to have different installation positions or optical axes for the vehicle-mounted cameras 2, and the fields of view may overlap partially or entirely. Note that the field of view here refers to the imaging range of the vehicle-mounted cameras 2.

[0078] For example, as shown in Fig. 10 as another installation example 1, a configuration can be adopted in which multiple vehicle-mounted cameras 2 are arranged one above the other, that is, they are positioned at the same position in the plan view of Fig. 1 but at different heights in the side view, their optical axes (J11) and (J12) are generally parallel, and their fields of view can partially overlap, as shown in Fig. 10 as another installation example 2, in which their optical axes (J21) and (J22) intersect, and their fields of view can almost entirely overlap.

[0079] 10 shows a third example of another mounting configuration, in which multiple vehicle-mounted cameras 2 are arranged on the left and right, i.e., the left and right positions in the plan view of FIG. 1 are approximately the same, the heights in the side view are approximately the same, the optical axes (J31) and (J32) are approximately parallel, and the fields of view do not overlap. Also, as shown in a fourth example of another mounting configuration, the optical axes (J41) and (J42) of the vehicle-mounted cameras 2 may intersect, causing the fields of view to partially overlap.

[0080] By using this configuration, if there is a correlation between the deviations in different fields of view, it can be determined that the correlation is likely to be caused by an installation error of the vehicle 1, and the cause of the deviation can be appropriately estimated.

[0081] The calibration device 10 also includes an imaging instruction unit 12 that executes a process to cause each of the multiple on-board cameras 2 mounted on the vehicle 1 to capture an image; an image acquisition unit 20 that executes a process to acquire each of the images captured by the multiple on-board cameras 2; an extraction unit 21 that executes a process to extract feature point information indicating the calibration markers 3 from the acquired images; a deviation calculation unit 22 that executes a process to calculate the deviation between the extracted feature point information and reference information obtained from the design values ​​for each image; a factor estimation unit 23 that executes a process to estimate the cause of the deviation based on the correlation of the deviation in images captured by at least two or more on-board cameras 2; and a calibration unit 24 that executes a process to calibrate the on-board cameras 2 based on the calculated deviation and the estimated cause of the deviation.

[0082] With this configuration, if there is a discrepancy between the feature point information and the reference information when calibrating the vehicle-mounted camera 2, it is possible to estimate the cause of the discrepancy and to perform appropriate calibration in accordance with the estimated cause. Therefore, it is possible to isolate the cause of the discrepancy and, after isolating the cause of the discrepancy, to calibrate the vehicle-mounted camera 2, thereby obtaining the various effects described above, similar to the calibration method.

[0083] Furthermore, the calibration device 10 includes a correction value calculation unit 25 that, when it is estimated that the installation state of the vehicle 1 is one of the causes of the deviation, executes a process of calculating a correction value for removing deviations caused by installation errors, and a new reference generation unit 26 that executes a process of correcting the reference information using the calculated correction value and generating new reference information from which the installation errors have been removed. This makes it possible to calibrate the vehicle-mounted camera 2 in a state in which deviations caused by installation errors have been removed. Furthermore, because it is possible to remove errors caused by slight deviations when installing the vehicle 1, it is possible to easily install the vehicle 1, greatly improving the efficiency of the calibration work, and achieving the various effects described above, similar to the calibration method.

[0084] In addition to the claims, the present disclosure also includes the following inventions. [1] A step of capturing images using a plurality of on-board cameras (2) mounted on a vehicle (1); acquiring images captured by a plurality of vehicle-mounted cameras; A step of extracting feature point information indicating the calibration marker (3) from the acquired image; a step of calculating a deviation between the extracted feature point information and reference information obtained from design values ​​for each image; a step of estimating a cause of the deviation based on a correlation between deviations in images captured by at least two or more vehicle-mounted cameras; and calibrating the vehicle-mounted camera based on the calculated deviation and the estimated cause of the deviation.

[0085] [2] In the step of estimating the cause of the deviation, the method for calibrating an on-board camera described in [1] is to estimate whether the cause of the deviation is an installation error of the on-board camera or whether it includes an installation error due to the installation state of the vehicle.

[0086] [3] A step of calculating a correction value for removing the deviation caused by the installation error when it is estimated that the deviation is caused by the installation error of the vehicle;

[0023] The method for calibrating an in-vehicle camera according to [2], further comprising the step of correcting the reference information using the calculated correction value and generating new reference information from which the installation error has been removed.

[0087] [4] In the process of estimating the cause of the deviation, if there is a correlation between the deviations of each image, it is estimated that the cause of the deviation is an installation error, and if there is no correlation, it is estimated that the cause of the deviation is an attachment error. [2] A method for calibrating an in-vehicle camera according to [3].

[0088] [5] A method for calibrating an in-vehicle camera according to any one of [2] to [4], wherein in the step of estimating the cause of the deviation, if the amount of deviation is less than a predetermined reference value, it is estimated that the cause of the deviation is an installation error, and if the amount of deviation is not less than the reference value, it is estimated that the cause of the deviation includes an installation error.

[0089] [6] A method for calibrating an onboard camera according to any one of [2] to [5], including at least one of a step of identifying at least one of the mounting position, mounting orientation, mounting height and mounting angle of the onboard camera that are factors that cause mounting error, or a step of identifying the mounting position and mounting orientation of the vehicle that are factors that cause mounting error.

[0090] [7] A method for calibrating an on-board camera according to any one of [1] to [6], wherein the process of estimating the cause of the deviation uses feature point information extracted from images of multiple on-board cameras mounted to capture different fields of view.

[0091] [8] An imaging instruction unit (12) that executes a process of causing a plurality of vehicle-mounted cameras (2) mounted on the vehicle (1) to capture images,

[0092] an image acquisition unit (20) that executes a process of acquiring images captured by a plurality of vehicle-mounted cameras; an extraction unit (21) that executes a process of extracting feature point information indicating a calibration marker from the acquired image; a deviation calculation unit (22) that executes a process of calculating a deviation between extracted feature point information and reference information obtained from design values ​​for each image; a factor estimation unit (23) that executes a process of estimating a factor of the deviation based on a correlation between deviations in images captured by at least two or more vehicle-mounted cameras; and a calibration unit (24) that executes a process for calibrating the vehicle-mounted camera based on the calculated deviation and the estimated cause of the deviation.

[0093] [9] a correction value calculation unit (25) that, when it is estimated that the cause of the deviation includes an installation error due to the installation state of the vehicle, executes a process of calculating a correction value for removing the deviation due to the installation error; [8] The calibration device for an in-vehicle camera described in [8] is provided with a new reference generation unit (26) that corrects the reference information using the calculated correction value and performs a process of generating new reference information that removes the installation error.

[0094] The control unit and the method described herein may be implemented by a special-purpose computer configured by configuring a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the control unit and the method described herein may be implemented by a special-purpose computer configured by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method described herein may be implemented by one or more special-purpose computers configured by combining a processor and memory programmed to perform one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory tangible storage medium. [Explanation of symbols]

[0095] In the drawing, 1 indicates a vehicle, 2, 2A, 2B, 2C, and 2D indicate on-board cameras, 3 indicates a calibration marker, 10 indicates a calibration device, 11 indicates an image input unit, 12 indicates an image capture instruction unit, 20 indicates an image acquisition unit, 21 indicates an extraction unit, 22 indicates a calculation unit, 23 indicates a factor estimation unit, 24 indicates a calibration unit, 25 indicates a correction value calculation unit, and 26 indicates a new standard generation unit.

Claims

1. A step of capturing images with a plurality of on-board cameras (2) mounted on a vehicle (1) in a work area where a proofing marker (3) on which a figure, character, or symbol is drawn in a predetermined shape is arranged in a predetermined positional relationship with respect to a predetermined reference position, A step of acquiring each image captured by multiple in-vehicle cameras, A step of extracting feature point information indicating the calibration marker from the acquired image, For each image, the process involves calculating the difference between the extracted feature point information and the reference information obtained from the design values, A step of estimating the cause of the misalignment based on the correlation of misalignments in images captured by at least two of the in-vehicle cameras, The process includes calibrating the in-vehicle camera based on the calculated deviation and the estimated cause of the deviation, A calibration method for an in-vehicle camera, which involves a step in estimating the cause of the deviation, in which the cause of the deviation is estimated to be either an installation error of the in-vehicle camera from the design value, or an installation error caused by the installation condition of the vehicle.

2. If it is estimated that the installation error of the vehicle is a factor in the deviation, the process of calculating a correction value to remove the deviation caused by the installation error, The process involves correcting the reference information using the calculated correction value to generate new reference information from which installation errors have been removed, and A method for calibrating an in-vehicle camera according to claim 1, including the following:

3. The method for calibrating an in-vehicle camera according to claim 1, wherein in the step of estimating the cause of the misalignment, if there is a correlation between the misalignments of each image, it is estimated that the cause of the misalignment is an installation error, and if there is no correlation, it is estimated that the cause of the misalignment is an installation error.

4. A method for calibrating an in-vehicle camera according to claim 1, wherein in the step of estimating the cause of the deviation, if the amount of deviation is less than a predetermined reference value, it is estimated that the cause of the deviation is an installation error, and if the amount of deviation is not less than a reference value, it is estimated that the cause of the deviation includes an installation error.

5. A method for calibrating an in-vehicle camera according to claim 1, comprising the steps of identifying at least one of the mounting position, mounting orientation, mounting height, and mounting angle of the in-vehicle camera that are factors in mounting errors, or identifying the mounting position and mounting orientation of the vehicle that are factors in mounting errors.

6. The method for calibrating an in-vehicle camera according to claim 1, wherein the step of estimating the cause of the misalignment uses feature point information extracted from images of a plurality of in-vehicle cameras that are mounted to capture different fields of view.

7. A work area in which a proofreading marker (3) on which a figure, character, or symbol is drawn in a predetermined shape is arranged in a predetermined positional relationship with a predetermined reference position, and an imaging instruction unit (12) that performs the process of causing each of the multiple on-board cameras (2) mounted on the vehicle (1) to capture an image, An image acquisition unit (20) that performs the process of acquiring images captured by multiple in-vehicle cameras, An extraction unit (21) performs a process to extract feature point information indicating the calibration marker from the acquired image, For each image, a deviation calculation unit (22) performs a process to calculate the difference between the extracted feature point information and the reference information obtained from the design value, A factor estimation unit (23) performs a process to estimate the cause of the misalignment based on the correlation of misalignments in images captured by at least two of the in-vehicle cameras, The system includes a calibration unit (24) that performs a calibration process for the in-vehicle camera based on the calculated deviation and the estimated cause of the deviation, The factor estimation unit is a calibration device for an in-vehicle camera that estimates whether the cause of the deviation is an installation error of the in-vehicle camera from the design value, or whether it includes an installation error caused by the vehicle's installation condition.

8. If it is estimated that the cause of the misalignment includes installation errors due to the installation state of the vehicle, the correction value calculation unit (25) executes a process to calculate a correction value to remove the misalignment caused by the installation error, A new standard generation unit (26) performs a process to correct the standard information using the calculated correction value and generate new standard information from which installation errors have been removed, Calibration device for an in-vehicle camera according to claim 7, comprising:

Images