Camera calibration system, camera calibration processing device, and camera calibration processing method

Abstract

This invention provides a camera calibration system, a camera calibration processing device, and a camera calibration processing method that enable easier camera calibration. [Solution] The camera calibration system 1 comprises: multiple cameras 20A to 20D mounted on a vehicle 200 that capture imaging areas A1 to A4 around the vehicle 200; projectors 10A to 10D mounted on the vehicle 200 that project a calibration pattern Pt onto overlap regions OL1 to OL4 where the imaging areas A1 to A4 of the multiple cameras 20A to 20D overlap; and a control unit that, while the vehicle 200 is in motion, projects the calibration pattern Pt onto the overlap regions OL1 to OL4 via the projectors 10A to 10D, captures the projected calibration pattern Pt with cameras 20A and 20B, and performs calibration of cameras 20A and 20B based on the captured calibration pattern Pt.

Description

Technical Field

[0001] The present disclosure relates to a camera calibration system, a camera calibration processing device, and a camera calibration processing method.

Background Art

[0002] The surround view camera automatic calibration system described in Patent Document 1 calibrates cameras by having a plurality of cameras mounted on a vehicle image test boards such as a plurality of checkerboards installed around the vehicle and based on the imaging results.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the configuration described in Patent Document 1 above, in order to calibrate the cameras, it was necessary to arrange a physical test board (checkerboard) around the vehicle, which was troublesome.

[0005] The present disclosure has been made in view of the above actual situation, and an object thereof is to provide a camera calibration system, a camera calibration processing device, and a camera calibration processing method that can more easily calibrate a camera.

Means for Solving the Problems

[0006] To achieve the above object, a camera calibration system according to a first aspect of the present disclosure includes: a plurality of cameras mounted on a vehicle and imaging an imaging area around the vehicle; a projector mounted on the vehicle and projecting a calibration pattern onto an overlap area where the imaging areas of the plurality of cameras overlap; The system includes a control unit that projects the calibration pattern onto the overlap area via the projector while the vehicle is in motion, captures the projected calibration pattern with the first and second cameras among the plurality of cameras, and performs calibration of the first and second cameras based on the captured calibration pattern.

[0007] To achieve the above objectives, the camera calibration processing device relating to the second aspect of this disclosure is: A camera calibration processing device that controls a plurality of cameras mounted on a vehicle and capturing an imaging area around the vehicle, and a projector mounted on the vehicle and projecting a calibration pattern onto an overlapping region where the imaging areas of each of the plurality of cameras overlap, While the vehicle is in motion, the calibration pattern is projected onto the overlap area via the projector, and the projected calibration pattern is captured by the first and second cameras among the plurality of cameras. The first and second cameras are then calibrated based on the captured calibration pattern.

[0008] To achieve the above objectives, the camera calibration processing method relating to the third aspect of this disclosure is: A camera calibration processing method utilizing a plurality of cameras mounted on a vehicle and capturing an imaging area around the vehicle, and a projector mounted on the vehicle and projecting a calibration pattern onto the overlapping region where the imaging areas of each of the plurality of cameras overlap, The procedure includes projecting the calibration pattern onto the overlap area via the projector while the vehicle is in motion, capturing the projected calibration pattern with the first and second cameras among the plurality of cameras, and calibrating the first and second cameras based on the captured calibration pattern. [Effects of the Invention]

[0009] According to this disclosure, cameras can be calibrated more easily. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic plan view of a vehicle according to the first embodiment of this disclosure. [Figure 2] This is a block diagram of a vehicle according to the first embodiment of this disclosure. [Figure 3] This is a flowchart of the calibration process according to the first embodiment of this disclosure. [Figure 4] This is a timing chart of the on / off switching of the emission of display light from the projector and the opening and closing of the camera shutter according to the first embodiment of this disclosure. [Figure 5] This is a schematic front view of a vehicle with a wall close by according to the first embodiment of this disclosure. [Figure 6] This is a schematic plan view of a vehicle according to a second embodiment of the present disclosure. [Figure 7] This is a schematic plan view showing an enlarged portion of a vehicle according to the second embodiment of this disclosure. [Modes for carrying out the invention]

[0011] (First Embodiment) A camera calibration system, a camera calibration processing device, and a camera calibration processing method according to the first embodiment of this disclosure will be described with reference to the drawings.

[0012] As shown in Figure 2, the vehicle 200 includes a camera calibration system 1 and a display unit 30. The camera calibration system 1 includes a plurality of cameras 20A to 20D, projectors 10A to 10D, and a control unit 50. Vehicle 200 is a construction machine vehicle, a motorcycle vehicle, or a passenger vehicle.

[0013] As shown in Figure 1, multiple (four) cameras 20A to 20D are mounted on the vehicle 200 and capture images of the area outside the vehicle. Each camera 20A to 20D is a fisheye camera with a field of view of 180°. Note that the angle of view of a fisheye camera is not limited to 180°; other angles are also acceptable. Furthermore, each camera 20A to 20D is not limited to a fisheye camera; it may also be a camera capable of capturing a wide area, such as a wide-angle camera.

[0014] The camera 20A is located in front of the vehicle 200 and can image an imaging area A1 set in front of the vehicle 200. The camera 20B is located on the left side of the vehicle 200 and can image an imaging area A2 set on the left side of the vehicle 200. The camera 20C is located on the right side of the vehicle 200 and can image an imaging area A3 set on the right side of the vehicle 200. The camera 20D is located behind the vehicle 200 and can image an imaging area A4 set behind the vehicle 200.

[0015] The imaging areas A1 to A4 have overlapping areas OL1 to OL4 that overlap with each other. The overlapping area OL1 is an area where the imaging areas A1 and A2 overlap and is set on the front left side of the vehicle 200. The overlapping area OL2 is an area where the imaging areas A1 and A3 overlap and is set on the front right side of the vehicle 200. The overlapping area OL3 is an area where the imaging areas A2 and A4 overlap and is set on the rear left side of the vehicle 200. The overlapping area OL4 is an area where the imaging areas A3 and A4 overlap and is set on the rear right side of the vehicle 200.

[0016] The projectors 10A to 1OD are attached outside the vehicle compartment of the vehicle 20O and project projection images onto the road surface G outside the vehicle under the control of the control unit 50. The projectors 10A to 1OD are projectors of the laser DMD (Digital Micro-mirror Device) method. Note that the projectors 10A to 1OD may be projectors of any method other than this method. For example, they may be projectors of the LCOS (Liquid Crystal on Silicon) method.

[0017] Projector 10A projects display light L onto overlap region OL1, and projects the calibration pattern Pt as a projected image onto overlap region OL1. Projector 10B projects display light L onto overlap region OL2, and projects the calibration pattern Pt as a projected image onto overlap region OL2. Projector 10C projects display light L onto overlap region OL3, and projects the calibration pattern Pt as a projected image onto overlap region OL3. Projector 10D projects display light L onto overlap region OL4, and projects the calibration pattern Pt as a projected image onto overlap region OL4. Calibration pattern Pt is a chessboard pattern, consisting of black and white squares arranged periodically in a matrix. Within the rectangular outer frame of calibration pattern Pt, black and white squares are arranged alternately in both the vertical and horizontal directions. The display light L from projectors 10A to 10D is visible light.

[0018] The display unit 30 is mounted inside the vehicle and displays vehicle information. The display unit 30 comprises a TFT (Thin Film Transistor) type liquid crystal display panel (not shown) and an illumination unit (not shown) that illuminates the liquid crystal display panel. As vehicle information, the display unit 30 displays a bird's-eye view image of the vehicle as seen from above, generated by the control unit 50. This bird's-eye view image is displayed when the vehicle 200 is parked (reversing). Furthermore, this bird's-eye view image may be displayed in all driving scenes except when reversing, and even when the power is stopped. Furthermore, the display unit 30 may be an organic electroluminescent display (OELD) or a head-up display device.

[0019] The control unit 50 consists of a CPU (Central Processing Unit), a GDC (Graphics Display Controller), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The control unit 50 includes, as functional blocks, a calibration processing unit 51 that performs calibration of cameras 20A to 20D, an image generation unit 53 that generates a bird's-eye view image, and a display processing unit 52 that displays images such as the bird's-eye view image.

[0020] The image generation unit 53 corrects the distortion of each image captured by the calibrated cameras 20A to 20D and combines them to generate a composite image (bird's-eye view image) that appears as if viewed from above the vehicle. The display processing unit 52 displays the bird's-eye view image generated by the image generation unit 53 on the display unit 30. The calibration processing unit 51 captures the calibration patterns Pt projected onto each overlap region OL1 to OL4 with cameras 20A to 20D, calculates calibration parameters (internal parameters and external parameters) from the captured calibration patterns Pt, and calibrates cameras 20A to 20D using the calculated calibration parameters. By calibrating cameras 20A to 20D, distortion or misalignment of the bird's-eye view image generated by the image generation unit 53 can be suppressed.

[0021] Next, the calibration process performed by the calibration processing unit 51 will be described with reference to the flowchart in Figure 3. This calibration process is repeatedly performed during the period when the control unit 50 is supplied with operating power.

[0022] The calibration processing unit 51 determines whether the vehicle 200 is in motion (step S1). This determination of whether the vehicle is in motion is made based on whether the vehicle speed is above a threshold speed (for example, 5 km / h). However, this determination of whether the vehicle is in motion is not limited to the vehicle speed; for example, it may also be made based on whether the shift position is in the drive position.

[0023] If the calibration processing unit 51 determines that the vehicle 200 is not in motion (step S1; NO), it terminates the calibration process without performing camera calibration.

[0024] When the calibration processing unit 51 determines that the vehicle 200 is in motion (step S1; YES), it determines whether or not camera calibration is necessary (step S2). This determination of whether or not to perform camera calibration is made based on whether a predetermined period (e.g., 1 year) has passed since the last camera calibration, and / or whether or not repairs have just been carried out at a car dealer or the like.

[0025] If the calibration processing unit 51 determines that camera calibration is unnecessary, for example, if a predetermined period has not elapsed since the last camera calibration (step S2; NO), it terminates the calibration process without performing camera calibration. This avoids unnecessary camera calibration.

[0026] If the calibration processing unit 51 determines that it is necessary to perform camera calibration, for example, because a predetermined period of time has elapsed since the last camera calibration (step S2; YES), it projects a calibration pattern Pt onto the overlap region OL1 via the projector 10A and simultaneously images this calibration pattern Pt with cameras 20A and 20B (step S3). Specifically, in step S3, as shown in the lower part of Figure 4, the projector 10A projects the calibration pattern Pt multiple times at regular intervals. The projection time T1 of one calibration pattern Pt is set to a time when a person cannot perceive the projected image (calibration pattern Pt), for example, 10 ms or less. Furthermore, in step S3, cameras 20A and 20B, synchronized with projector 10A, capture images of the calibration pattern Pt projected multiple times. Specifically, as shown in the upper part of Figure 4, the shutter opening time T2 of cameras 20A and 20B corresponds to each projection time T1 and is set to include projection time T1. The shutter opening time T2 is also called the shutter speed. The projection time T1 is set to be less than or equal to the shutter opening time T2. As a result, cameras 20A and 20B capture images of multiple calibration patterns Pt projected at different times. In this example, cameras 20A and 20B simultaneously captured the calibration pattern Pt, but cameras 20A and 20B may capture the calibration pattern Pt at different times.

[0027] The calibration processing unit 51 takes the average of images containing the calibration pattern Pt that have been captured multiple times, and generates an average image (step S4). This average image is generated by superimposing the multiple captured images. Specifically, the RGB value of each pixel in this average image is the average value of the RGB values ​​of each pixel in the multiple captured images. This makes it possible to generate an average image that smooths out the unevenness of the road surface G onto which the calibration pattern Pt is projected. For example, even if a stone or curb is reflected in one of the multiple images, if the stone or curb is not reflected in the remaining images, the calibration pattern Pt can be flattened in the average image.

[0028] The calibration processing unit 51 calculates calibration parameters (internal parameters and external parameters, described later) for cameras 20A and 20B based on the acquired average image, and performs calibration of cameras 20A and 20B using the calculated calibration parameters (step S5).

[0029] Next, the calibration processing unit 51 determines whether all camera calibrations have been completed (step S6). All camera calibrations include the calibration of cameras 20A and 20B by projecting the calibration pattern Pt onto overlap region OL1 (first camera calibration), the calibration of cameras 20A and 20C by projecting the calibration pattern Pt onto overlap region OL2 (second camera calibration), the calibration of cameras 20B and 20D by projecting the calibration pattern Pt onto overlap region OL3 (third camera calibration), and the calibration of cameras 20C and 20D by projecting the calibration pattern Pt onto overlap region OL4 (fourth camera calibration). In the above case, the first camera calibration has been completed, but the second to fourth camera calibrations have not been completed.

[0030] If the calibration processing unit 51 determines that not all camera calibrations are complete (step S6; NO), it returns to the process in step S3 and executes steps S3 to S5, performing the camera calibrations of the first to fourth camera calibrations that have not yet been completed. In this way, by repeating steps S3 to S6, the first to fourth camera calibrations are performed in order. When the calibration processing unit 51 determines that all camera calibrations are complete (step S6; YES), it terminates the calibration process.

[0031] The intrinsic parameters include the distortion of cameras 20A to 20D individually, and parameters for calibrating the position of cameras 20A to 20D relative to the vehicle 200. The intrinsic parameters may be determined by methods implemented in the image processing library, which is a library for image processing and image analysis. Specifically, the intrinsic parameters are calculated by minimizing the reprojection error of the calibration point cloud (the point cloud located at each corner of the black and white squares of the calibration pattern Pt) in the calibration pattern Pt within the acquired average image, relative to the training data.

[0032] The external parameters include relative position parameters for calibrating the relative position and relative orientation of cameras 20A to 20D. One of cameras 20A to 20D is a reference camera whose mounting position and orientation are determined in the coordinate system of the vehicle 200, and the relative positions of cameras 20A to 20D are relative to this reference camera. The reference camera is, for example, camera 20A, located at the top center of the windshield. These relative position parameters are used to calibrate the relative position and relative orientation of cameras 20A and 20B by comparing a common calibration pattern Pt captured by cameras 20A and 20B.

[0033] (effect) The first embodiment described above provides the following effects. (1-1) The camera calibration system 1 comprises: a plurality of cameras 20A to 20D mounted on a vehicle 200 that capture imaging areas A1 to A4 around the vehicle 200; projectors 10A to 10D mounted on the vehicle 200 that project a calibration pattern Pt onto overlap regions OL1 to OL4 where the imaging areas A1 to A4 of the plurality of cameras 20A to 20D overlap; and a control unit 50 that, while the vehicle 200 is in motion, projects the calibration pattern Pt onto the overlap regions OL1 to OL4 via the projectors 10A to 10D, captures the projected calibration pattern Pt with the first and second cameras 20A and 20B among the plurality of cameras 20A to 20D, and performs calibration of the first and second cameras 20A and 20B based on the captured calibration pattern Pt. With this configuration, the calibration pattern Pt is projected by projectors 10A to 10D, eliminating the need to place a physical test board (check board) around the vehicle, making camera calibration easier. Furthermore, conventionally, camera calibration was performed at the time of shipment or while the vehicle was stationary in a well-equipped location. In contrast, with the above configuration, the first and second cameras 20A and 20B are calibrated while the vehicle 200 is in motion. Therefore, camera calibration becomes possible in a less restrictive environment.

[0034] (1-2) The control unit 50 projects the calibration pattern Pt via the projectors 10A to 10D for a projection time T1 that is not perceptible to humans, and simultaneously captures the calibration pattern Pt with the cameras 20A to 20D in synchronization with the projectors 10A to 10D. With this configuration, the calibration pattern Pt is not visible to people, so it does not cause any discomfort to the occupants of the vehicle 200 or people outside the vehicle during camera calibration.

[0035] (1-3) The control unit 50 projects the calibration pattern Pt via the projectors 10A to 10D for a projection time T1 that is less than or equal to the shutter opening time T2 of the cameras 20A to 20D, and simultaneously captures the calibration pattern Pt with the cameras 20A to 20D in synchronization with the projectors 10A to 10D. This configuration allows the projection time T1 to be set to a short duration, thereby suppressing any discomfort caused to the occupants of the vehicle 200 or to people outside the vehicle during camera calibration.

[0036] (1-4) While the vehicle 200 is in motion, the control unit 50 projects calibration patterns Pt multiple times via projectors 10A to 10D, and captures multiple calibration patterns Pt with the first and second cameras 20A and 20B each time the calibration patterns Pt are projected, and performs calibration of the first and second cameras 20A and 20B based on the captured multiple calibration patterns Pt. This configuration allows for the generation of an image in which the unevenness of the road surface G onto which the calibration pattern Pt is projected has been smoothed out, thereby suppressing distortion and misalignment in the composite image (bird's-eye view image) caused by the unevenness of the road surface G.

[0037] (1-5) A control unit 50, which is an example of a camera calibration processing device, is mounted on a vehicle 200 and controls a plurality of cameras 20A to 20D that capture imaging areas A1 to A4 around the vehicle 200, and projectors 10A to 10D that are mounted on the vehicle 200 and project a calibration pattern Pt onto overlap regions OL1 to OL4 where the imaging areas A1 to A4 of the plurality of cameras 20A to 20D overlap. While the vehicle 200 is in motion, the control unit 50 projects the calibration pattern Pt onto the overlap regions OL1 to OL4 via the projectors 10A to 10D, captures the projected calibration pattern Pt with the first and second cameras 20A and 20B among the plurality of cameras 20A to 20D, and performs calibration of the first and second cameras 20A and 20B based on the captured calibration pattern Pt. With this configuration, the calibration pattern Pt is projected by projectors 10A to 10D, eliminating the need to place a physical test board (check board) around the vehicle, making camera calibration easier.

[0038] (1-6) The camera calibration processing method utilizes a plurality of cameras 20A to 20D mounted on a vehicle 200 that image imaging areas A1 to A4 around the vehicle 200, and projectors 10A to 10D mounted on the vehicle 200 that project a calibration pattern Pt onto overlap regions OL1 to OL4 where the respective image imaging areas A1 to A4 of the plurality of cameras 20A to 20D overlap. The camera calibration processing method includes steps S3 to S6 in which, while the vehicle 200 is in motion, the calibration pattern Pt is projected onto the overlap regions OL1 to OL4 via the projectors 10A to 10D, the projected calibration pattern Pt is imaged by the first and second cameras 20A and 20B among the plurality of cameras 20A to 20D, and the first and second cameras 20A and 20B are calibrated based on the imaged calibration pattern Pt. With this configuration, the calibration pattern Pt is projected by projectors 10A to 10D, eliminating the need to place a physical test board (check board) around the vehicle, making camera calibration easier.

[0039] (Second Embodiment) A camera calibration system, a camera calibration processing device, and a camera calibration processing method according to the second embodiment of this disclosure will be described with reference to the drawings. This embodiment differs from the first embodiment in that a two-dimensional code is projected instead of a calibration pattern. The differences from the first embodiment will be described below.

[0040] As shown in Figure 6, projectors 10A to 10D project a 2D code Pc onto overlapping regions OL1 to OL4, respectively. The 2D code Pc is rectangular in shape and smaller in size than the calibration pattern Pt of the first embodiment described above. The area of ​​the 2D code Pc is set to 1 / 4 to 1 / 20 of the total area of ​​overlapping regions OL1 to OL4. Furthermore, the 2D code Pc does not have to be a planar image like the calibration pattern in the first embodiment; it can be any image that can identify at least one point (so to speak, a calibration marker). In addition to the 2D code Pc, the calibration marker can be a round bright spot, the intersection of a cross-shaped line, etc. Note that images with a regular planar shape like the calibration pattern require a relatively large projection area, so a calibration marker that requires a relatively small projection area is preferable because it makes the calibration of this embodiment easier.

[0041] The 2D code Pc consists of a rectangular frame in which black and white squares are arranged in a matrix in a predetermined combination. In the areas of the black squares, the brightness of the display light L is zero, and in the areas of the white squares, the brightness of the display light L is at its maximum. The 2D code Pc is constructed to be rotationally symmetric so that its orientation can be determined. Therefore, when the 2D code Pc is rotated by 360° / n (where n is a natural number greater than or equal to 2) around its center position, the appearance of the 2D code Pc changes.

[0042] Vehicle 200 is equipped with a notification means 45 for notifying the user (driver) of a camera malfunction, as shown by the dashed line in Figure 2. The notification means 45 is an indicator that lights up when a malfunction occurs, and / or a speaker that emits sound. The notification means 45 may be configured as a display unit 30.

[0043] In step S3 of the first embodiment described above, the calibration processing unit 51 changes the position of the two-dimensional code Pc within the overlap region OL1 between the first position P1 and the second position P2, as shown in Figure 7. The two-dimensional code Pc1 at the first position P1 and the two-dimensional code Pc2 at the second position P2 are set so that they do not overlap each other if projected simultaneously. The first position P1 and the second position P2 are located on opposite sides of the overlap region OL1, with the center position Cp of the overlap region OL1 in between. The two-dimensional code Pc2 is projected to a position closer than the two-dimensional code Pc1. Note that the first position P1 and the second position P2 are not limited to this example and may be located anywhere within the overlap region OL1. Also, the 2D code Pc may be moved between three or more positions within the overlap region OL1.

[0044] The calibration processing unit 51 projects the 2D code Pc1 onto the first position P1 and executes the processes in steps S3 to S5 described above, and then projects the 2D code Pc2 onto the second position P2 and executes the processes in steps S3 to S5 described above. In this way, by executing the processes in steps S3 to S5 described above for each 2D code Pc projected at different positions P1 and P2, the camera can be calibrated with higher accuracy.

[0045] The control unit 50 includes a camera abnormality determination unit 54 that determines whether or not there is an abnormality in the cameras 20A to 20D that image the 2D code Pc, as shown by the dashed line in Figure 2. In step S3 described above, if the camera abnormality determination unit 54 determines that either camera 20A or 20B cannot image the 2D code Pc, it determines that an abnormality has occurred in the camera that cannot image the 2D code Pc. Also, in step S3 described above, if the camera abnormality determination unit 54 determines that both cameras 20A and 20B can image the 2D code Pc, it determines that there is no abnormality in either camera 20A or 20B. Furthermore, in step S3 described above, if the camera abnormality determination unit 54 determines that neither camera 20A nor 20B can image the 2D code Pc, it determines that an abnormality has occurred in either camera 20A or 20B. Here, "unable to capture a 2D code Pc" by the camera means that a 2D code Pc does not exist in the image captured by this camera, and "able to capture a 2D code Pc" by the camera means that a 2D code Pc exists in the image captured by this camera.

[0046] The camera anomaly detection unit 54 determines whether or not there is an anomaly in cameras 20A and 20B when a two-dimensional code Pc1 is projected onto a first position P1 and when a two-dimensional code Pc2 is projected onto a second position P2. The anomaly detection by the camera anomaly detection unit 54 is performed when cameras 20A and 20B are capturing images of two-dimensional codes Pc1 and Pc2, when cameras 20A and 20C are capturing images of two-dimensional codes Pc1 and Pc2, when cameras 20B and 20D are capturing images of two-dimensional codes Pc1 and Pc2, or when cameras 20C and 20D are capturing images of two-dimensional codes Pc1 and Pc2. When the camera abnormality detection unit 54 determines that there is an abnormality in cameras 20A to 20D, it notifies the user (occupants, including the driver) of the camera abnormality via the notification means 45.

[0047] (effect) The second embodiment described above provides the following effects. (2-1) The camera calibration system 1 comprises: a plurality of cameras 20A to 20D mounted on a vehicle 200 that capture imaging areas A1 to A4 around the vehicle 200; projectors 10A to 10D mounted on the vehicle 200 that project a two-dimensional code Pc, which is an example of a calibration marker, onto a portion of the overlap region OL1 to OL4 where the imaging areas A1 to A4 of the plurality of cameras 20A to 20D overlap; and a control unit 50 that projects the two-dimensional code Pc via the projectors 10A to 10D, captures the projected two-dimensional code Pc with the first and second cameras 20A and 20B among the plurality of cameras 20A to 20D, and performs calibration of the first and second cameras 20A and 20B based on the captured two-dimensional code Pc. With this configuration, the 2D code PC is projected by projectors 10A-10D, eliminating the need to place a physical test board (check board) around the vehicle, making camera calibration easier.

[0048] (2-2) The control unit 50 calculates the relative attitude and position of the first and second cameras 20A and 20B from the two-dimensional code Pc captured by the first and second cameras 20A and 20B, and performs calibration of the first and second cameras 20A and 20B based on the calculated relative attitude and position of the first and second cameras 20A and 20B. With this configuration, even if the relative attitude and position of the first and second cameras 20A and 20B are shifted due to aging deterioration of the first and second cameras 20A and 20B or manufacturing tolerances at the time of shipment, distortion and shifting of the composite image (bird's-eye view image) are suppressed.

[0049] (2-3) The control unit 50 projects a 2D code Pc with a rotationally asymmetric shape via the projectors 10A to 10D. In this configuration, the 2D code Pc has a rotationally asymmetrical shape. Therefore, the control unit 50 can perform high-precision camera calibration based on the orientation of the captured 2D code Pc.

[0050] (2-4) The control unit 50 projects the two-dimensional code Pc at first position P1 and second position P2, which are different positions within the overlap region OL1 to OL4, via projectors 10A to 10D, and captures images of the two-dimensional code Pc at first position P1 and second position P2, respectively, with first and second cameras 20A and 20B. This configuration enables high-precision calibration of the first and second cameras 20A and 20B.

[0051] (2-5) The control unit 50 determines that a camera malfunction has occurred if it is unable to capture the two-dimensional code Pc with at least one of the first and second cameras 20A and 20B, and notifies the user of the camera malfunction via the notification means 45. This configuration allows for rapid detection of anomalies in cases where the camera has not been properly calibrated due to aging or manufacturing tolerances at the time of shipment.

[0052] (2-6) The control unit 50 projects the 2D code Pc via the projectors 10A to 10D for a projection time T1 that is not perceptible to humans, and simultaneously captures the 2D code Pc with the cameras 20A to 20D in synchronization with the projectors 10A to 10D. With this configuration, the 2D code PC is not visible to people, so it does not cause any discomfort to the occupants of vehicle 200 or people outside the vehicle during camera calibration.

[0053] (2-7) The control unit 50 projects the 2D code Pc via the projectors 10A to 10D for a projection time T1 that is less than or equal to the shutter opening time T2 of the cameras 20A to 20D, and simultaneously captures the 2D code Pc with the cameras 20A to 20D in synchronization with the projectors 10A to 10D. This configuration allows the projection time T1 to be set to a short duration, ensuring that the occupants of the vehicle 200 or people outside the vehicle do not experience any discomfort during camera calibration.

[0054] (2-8) A control unit 50, which is an example of a camera calibration processing device, is mounted on a vehicle 200 and controls a plurality of cameras 20A to 20D that capture imaging areas A1 to A4 around the vehicle 200, and projectors 10A to 10D that are mounted on the vehicle 200 and project a two-dimensional code Pc onto a portion of the overlap region OL1 to OL4 where the imaging areas A1 to A4 of each of the plurality of cameras 20A to 20D overlap. The control unit 50 projects the two-dimensional code Pc via the projectors 10A to 10D and captures the projected two-dimensional code Pc with the first and second cameras 20A and 20B among the plurality of cameras 20A to 20D, and performs calibration of the first and second cameras 20A and 20B based on the captured two-dimensional code Pc. With this configuration, the 2D code PC is projected by projectors 10A-10D, eliminating the need to place a physical test board (check board) around the vehicle, making camera calibration easier.

[0055] (2-9) The camera calibration processing method utilizes a plurality of cameras 20A to 20D mounted on a vehicle 200 that image imaging areas A1 to A4 around the vehicle 200, and projectors 10A to 10D mounted on the vehicle 200 that project a two-dimensional code Pc onto a portion of the overlap region OL1 to OL4 where the respective image imaging areas A1 to A4 of the plurality of cameras 20A to 20D overlap. The camera calibration processing method includes steps S3 to S6 in which the two-dimensional code Pc is projected via the projectors 10A to 10D, the projected two-dimensional code Pc is imaged by the first and second cameras 20A and 20B among the plurality of cameras 20A to 20D, and the first and second cameras 20A and 20B are calibrated based on the imaged two-dimensional code Pc. With this configuration, the 2D code PC is projected by projectors 10A-10D, eliminating the need to place a physical test board (check board) around the vehicle, making camera calibration easier.

[0056] This disclosure is not limited to the embodiments and drawings described above. Modifications (including the deletion of components) can be made as appropriate, provided they do not alter the essence of this disclosure. An example of such a modification is described below.

[0057] (modified version) In each of the above embodiments, if a wall surface is detected around the vehicle 200, the calibration pattern Pt may be projected only onto the wall surface. Specifically, the vehicle 200 is equipped with multiple proximity sensors 40, as shown by the dashed line in Figure 2. The multiple proximity sensors 40 are mounted outside the vehicle's interior and detect the approach of obstacles, including a wall W (see Figure 5), around the vehicle 200, and output the detected results to the control unit 50. Each proximity sensor 40 detects the approach of an obstacle using infrared light or millimeter-wave radar, etc. Note that each proximity sensor 40 may be a TOF (Time of Flight) sensor. If the calibration processing unit 51 determines that camera calibration is necessary in the calibration process described above (step S2; YES), it uses each proximity sensor 40 to determine whether or not the wall W is approaching. If the calibration processing unit 51 determines that the wall W is approaching, in step S3, as shown in Figure 5, it projects a calibration pattern Pt or a two-dimensional code Pc onto the wall W from each projector 10A to 10D. Specifically, the projectors 10A to 10D emit display light L to the illumination range B2 on the wall W, out of the maximum illumination range B1 of the display light L, and do not emit display light L to the area of ​​the maximum illumination range B1 that includes the road surface G. This modification produces the following effects: The camera calibration system 1 includes a proximity sensor 40, which is an example of a detection unit, that detects whether or not there is a wall W, which is an example of an obstacle, at the projection destination of the calibration pattern Pt or 2D code Pc of the projectors 10A to 10D. When the proximity sensor 40 detects that there is a wall W at the projection destination, the control unit 50 projects the calibration pattern Pt or 2D code Pc onto the wall W via the projectors 10A to 10D. With this configuration, the first and second cameras 20A and 20B can be calibrated by projecting a calibration pattern Pt or a two-dimensional code Pc onto the wall surface W.

[0058] Furthermore, the control unit 50 may stop projecting the calibration pattern Pt or 2D code Pc via the projectors 10A to 10D when the proximity sensor 40 detects that there is a wall W at the projection destination, and start projecting the calibration pattern Pt or 2D code Pc via the projectors 10A to 10D when the proximity sensor 40 detects that there is no longer a wall W at the projection destination. With this configuration, camera calibration can be reliably performed using a calibration pattern Pt or a 2D code Pc projected onto the road surface G.

[0059] In each of the embodiments described above, visible light, which is display light L, was emitted from the projectors 10A to 10D. However, the invention is not limited to this, and infrared light, which has a wavelength of light that humans cannot perceive, may also be emitted as display light L. In this case, each camera 20A to 20D is configured to capture images in the infrared light region. For example, each camera 20A to 20D may be configured to capture images in the infrared light region and the visible light region simultaneously, or it may be configured to switch between capturing areas in the infrared light region and the visible light region using a filter. This modification produces the following effects: Projectors 10A to 10D project a calibration pattern Pt or a two-dimensional code Pc by emitting display light L, which is infrared light. Multiple cameras 20A to 20D are configured to capture infrared light. This configuration ensures that the calibration pattern Pt or the two-dimensional code Pc are not visible to occupants or people outside the vehicle.

[0060] Furthermore, the camera calibration system 1 may include a drive unit (not shown) that rotates the projectors 10A to 10D to change the direction in which the display light L emitted from the projectors 10A to 10D is emitted. In this case, when the calibration processing unit 51 determines that a wall W is approaching, it drives the projectors 10A to 10D via this drive unit so that the display light L reaches only the wall W or only the road surface G. Alternatively, the display light L may be made to reach only the wall W or only the road surface G by changing the position of the illumination range B2 within the maximum illumination range B1 without rotating the projectors 10A to 10D.

[0061] In each of the embodiments described above, the projection time T1 of a single calibration pattern Pt was set to a time that is not perceptible to humans, for example, 10 ms (= 0.01 s) or less. However, it is not limited to this and may be set to a time that is perceptible to humans. In the embodiments described above, the projection time T1 was set to be less than or equal to the shutter opening time T2, but it is not limited to this and may be set to be longer than the shutter opening time T2.

[0062] In the embodiments described above, the calibration pattern Pt or the two-dimensional code Pc was projected while the vehicle 200 was in motion. However, the calibration pattern Pt or the two-dimensional code Pc may be projected not only while the vehicle 200 is in motion, but also while the vehicle 200 is stationary.

[0063] In each of the above embodiments, the calibration pattern Pt or 2D code Pc was projected multiple times in step S3, but the calibration pattern Pt or 2D code Pc may be projected only once. In this case, the process of averaging the captured images (step S4) is omitted.

[0064] In the embodiments described above, the first to fourth camera calibrations were performed sequentially, but they may also be performed simultaneously. In this case, the color of the calibration pattern Pt or 2D code Pc may be changed for each overlap region OL1 to OL4. Alternatively, the shape of the calibration pattern Pt or 2D code Pc may be changed for each overlap region OL1 to OL4. This makes it easier for the calibration processing unit 51 to identify which of the overlap regions OL1 to OL4 the captured calibration pattern Pt or 2D code Pc will be projected onto.

[0065] In each of the above embodiments, the calibration processing unit 51 may perform camera calibration only when stable driving of the vehicle 200 is expected (for example, when driving on an expressway or on a road with a legal speed limit of 60 km / h or higher).

[0066] In each of the above embodiments, the process of determining whether or not camera calibration is necessary (step S2) may be omitted. In each of the above embodiments, the calibration processing unit 51 calculated the internal and external parameters of cameras 20A to 20D and calibrated the cameras 20A to 20D using the calculated internal and external parameters. However, the cameras 20A to 20D may also be calibrated using either the internal or external parameters.

[0067] In each of the above embodiments, the number of projectors 10A to 10D is not limited to four, but may be one to three, or five or more. In each of the above embodiments, the number of cameras 20A to 20D is not limited to four, but may be one to three, or five or more. In each of the above embodiments, projectors 10A to 10D may display guidance or provide illumination on the road surface G or wall surface W, in addition to the calibration pattern Pt or the two-dimensional code Pc.

[0068] In the second embodiment described above, the two-dimensional code Pc was moved between the first position P1 and the second position P2 within the overlap region OL1, but it is not necessary to move it. In the second embodiment described above, the two-dimensional code Pc may be in other shapes such as multiple or single dots or lines. In the second embodiment described above, the two-dimensional code Pc may have a rotationally symmetrical shape.

[0069] The first and second embodiments described above may be combined. For example, in the first embodiment, projectors 10A to 10D may project a two-dimensional code Pc, or conversely, in the second embodiment, projectors 10A to 10D may project a calibration pattern Pt. Also, in the first embodiment, the calibration pattern Pt may be moved within the overlap region OL1 to OL4. [Explanation of Symbols]

[0070] 1…Camera calibration system 10A~10D...Projector 20A~20D…Camera 30...Indicator 40… Proximity sensor 45…Notification means 50...Control unit, 51...Calibration processing unit, 52...Display processing unit, 53...Image generation unit, 54...Camera abnormality detection unit 200...vehicles A1-A4: Imaging area, B1: Maximum illumination range, B2: Illumination range, G: Road surface, L: Indicator light, P1: First position, P2: Second position, T1: Projection time, T2: Shutter open time, W: Wall surface, OL1-OL4: Overlap area, Pc, Pc1, Pc2: 2D code, Cp: Center position, Pt: Calibration pattern

Claims

1. Multiple cameras mounted on the vehicle capture images of the surrounding area of ​​the vehicle, A projector mounted on the vehicle, which projects a calibration pattern onto the overlap region where the imaging areas of each of the multiple cameras overlap, The system includes a control unit that projects the calibration pattern onto the overlap region via the projector while the vehicle is in motion, captures the projected calibration pattern with the first and second cameras among the plurality of cameras, and performs calibration of the first and second cameras based on the captured calibration pattern. Camera calibration system.

2. The control unit projects the calibration pattern via the projector for a period of time that is imperceptible to human perception, and simultaneously captures the calibration pattern with the camera in synchronization with the projector. The camera calibration system according to claim 1.

3. The control unit projects the calibration pattern via the projector for a time less than or equal to the time the camera shutter is open, and simultaneously captures the calibration pattern with the camera in synchronization with the projector. The camera calibration system according to claim 1 or 2.

4. The control unit projects the calibration pattern multiple times via the projector while the vehicle is in motion, captures multiple calibration patterns with the first and second cameras each time the calibration pattern is projected multiple times, and performs calibration of the first and second cameras based on the captured multiple calibration patterns. The camera calibration system according to claim 1 or 2.

5. The camera calibration system includes a detection unit that detects whether or not there is an obstacle at the projection destination of the calibration pattern of the projector. When the control unit detects that the obstacle is present at the projection destination, it projects the calibration pattern onto the obstacle via the projector. The camera calibration system according to claim 1 or 2.

6. The camera calibration system includes a detection unit that detects whether or not there is an obstacle at the projection destination of the calibration pattern of the projector. The control unit stops projecting the calibration pattern via the projector when the detection unit detects that the obstacle is present at the projection destination, and starts projecting the calibration pattern via the projector when the detection unit detects that the obstacle is gone from the projection destination. The camera calibration system according to claim 1 or 2.

7. The projector projects the calibration pattern by emitting display light having a wavelength of light that is imperceptible to humans. The plurality of cameras are configured to capture the display light. The camera calibration system according to claim 1 or 2.

8. A camera calibration processing device that controls a plurality of cameras mounted on a vehicle and capturing an imaging area around the vehicle, and a projector mounted on the vehicle and projecting a calibration pattern onto an overlapping region where the imaging areas of each of the plurality of cameras overlap, While the vehicle is in motion, the calibration pattern is projected onto the overlap region via the projector, and the projected calibration pattern is captured by the first and second cameras among the plurality of cameras, and the first and second cameras are calibrated based on the captured calibration pattern. Camera calibration processing device.

9. A camera calibration processing method utilizing a plurality of cameras mounted on a vehicle and capturing an imaging area around the vehicle, and a projector mounted on the vehicle and projecting a calibration pattern onto the overlapping region where the imaging areas of each of the plurality of cameras overlap, The process includes the steps of projecting the calibration pattern onto the overlap region via the projector while the vehicle is in motion, capturing the projected calibration pattern with the first and second cameras among the plurality of cameras, and calibrating the first and second cameras based on the captured calibration pattern. Camera calibration processing method.

Images