Calibration device and calibration method for camera unit
The camera unit calibration device uses movable collimators and mirrors to correct geometric distortions across the entire image range, addressing space and cost limitations of conventional setups, ensuring precise measurements and corrections.
Patent Information
- Application Number
- JP2023083897
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-12-04
AI Technical Summary
Existing camera units with multiple cameras face challenges in accurately measuring and correcting geometric distortions across the entire range of wide-angle images due to the limitations of conventional collimator setups, which require large spaces and increased costs when trying to expand the collimator diameter.
A camera unit calibration device and method that utilizes a system of movable collimators and plane mirrors, controlled by a collimator control unit, to form collimated images at desired positions within the entire image range, allowing for accurate geometric distortion correction without the need for extensive space.
Enables highly accurate measurement and correction of geometric distortions across the entire image range in a space-saving manner, ensuring precise parallax information and distance calculations.
Smart Images

Figure 2024167670000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an apparatus and method for calibrating a camera unit that performs a predetermined measurement based on a plurality of images acquired by using a plurality of cameras to capture an image of the same object from different viewpoints. [Background technology]
[0002] Conventionally, various proposals have been made for camera units that are configured with multiple cameras and use these multiple cameras to capture images of approximately the same predetermined area containing the same object from different viewpoints, and that can perform various measurements or generate various images by performing predetermined image processing based on the multiple image data obtained, and these units are widely and generally in practical use.
[0003] For example, in a camera unit consisting of two cameras arranged horizontally at a predetermined distance from each other, a technology has become widespread in which the distance to an object is calculated using, for example, disparity information based on two sets of image data obtained by capturing images of the same object, or the position of the object on the screen, etc.
[0004] When this type of camera unit is installed and used in a predetermined location, a case or the like that covers the entire camera unit may be used, for example, for the purpose of protecting the camera unit.
[0005] When used in this manner, the case may be provided with a window member formed using a substantially colorless and transparent plate-like optical member made of a material such as glass or acrylic, for example, in order to ensure visibility in the area in front of the camera unit.
[0006] A specific example of the configuration is a camera unit mounted inside a vehicle such as an automobile to acquire image data of the external situation around (mainly in front of) the vehicle. In this type of vehicle-mounted camera unit, for example, the windshield of the vehicle is disposed in the area between the imaging lens of the camera unit and the object to be imaged. Therefore, in this case, the camera unit images the object to be imaged through the windshield.
[0007] Generally, when an imaging operation is performed in a situation where an optical component such as a windshield is placed between the imaging lens of the camera unit and the object to be imaged, the image obtained may be affected by optical refraction when the light beam from the object passes through the windshield, etc., which may result in adverse effects such as geometric distortion on the image.
[0008] In this way, if geometric distortion or the like occurs in the image acquired by the camera unit, accurate parallax information or the like cannot be acquired. Therefore, the calculation results (e.g., distance to an object, etc.) calculated based on inaccurate parallax information or the like contain errors, making accurate measurements impossible. For this reason, in order to obtain accurate calculation results using the camera unit, it is important to always acquire accurate information.
[0009] In view of this, it has been considered in the past to measure the geometric distortion occurring in a captured image acquired by a camera unit and to perform image correction based on the measurement result. As a means for measuring the geometric distortion of a captured image, there has been a means for measuring the geometric distortion occurring due to a windshield or the like by capturing an image of an object such as a chart board placed at an infinity position. However, it is known that using such a means requires a vast space, and is difficult to realize.
[0010] On the other hand, various methods for generating a chart image placed at infinity, such as using equipment such as a collimator, have been disclosed, for example, in International Patent Publication WO2021 / 024612, JP 2012-132739, and JP 2019-90755.
[0011] The technology disclosed in International Patent Publication WO2021 / 024612 and other publications uses a collimator, a finite distance chart board, and an infinite distance chart to correct the image shift that occurs between the two cameras of a camera unit mounted on a vehicle.
[0012] The technologies disclosed in the above-mentioned JP 2012-132739 A, JP 2019-90755 A, etc. all use a collimator to generate an infinitely distant chart image and correct image shift in the horizontal direction or optical axis direction of the two cameras of the camera unit. [Prior art documents] [Patent documents]
[0013] [Patent Document 1] International Patent Publication No. WO2021 / 024612 [Patent Document 2] JP 2012-132739 A [Patent Document 3] JP 2019-90755 A Summary of the Invention [Problem to be solved by the invention]
[0014] However, the technologies disclosed in International Patent Publication WO2021 / 024612, JP 2012-132739, JP 2019-90755, etc. all have a configuration in which the collimator is fixed to the camera unit.
[0015] For this reason, the image formed by the collimator (hereinafter referred to as a collimated image) is captured only in a predetermined area within the range of the image captured by the camera unit. Therefore, there are image areas where the collimated image is not captured, which causes a problem that distortion correction cannot be performed in those areas.
[0016] To achieve this, for example, the collimator is made larger in diameter so that a collimated image can be captured within the range of the image, even in peripheral areas, as disclosed in the above-mentioned JP2012-132739A.
[0017] However, increasing the diameter of the collimator is not realistic because it leads to an increase in size and cost of the entire apparatus including the collimator.
[0018] In particular, in recent years, the imaging angle of view of the camera applied to a vehicle-mounted camera unit has been widening, which poses a problem that the collimator needs to have a larger aperture in order to reliably perform distortion correction within the entire range of the wide-angle image captured by the camera unit.
[0019] The present invention provides a camera unit calibration device and calibration method that performs specified measurements based on multiple images acquired by using multiple cameras to capture the same object from different viewpoints, and aims to provide a camera unit calibration device and calibration method that can measure (calibrate) and correct, with high precision, geometric distortion that occurs in an image acquired by capturing an object placed at infinity over the entire range of the acquired image, in a space-saving manner without requiring a large space. [Means for solving the problem]
[0020] In order to achieve the above object, a camera unit calibration device according to one aspect of the present invention is a camera unit calibration device that performs a predetermined measurement based on a plurality of images obtained by capturing images of the same object from different viewpoints using a plurality of cameras, the device comprising a plurality of collimators provided corresponding to the plurality of cameras, a plurality of collimator drive devices provided corresponding to the plurality of collimators, a plurality of plane mirrors that reflect collimated light from the plurality of collimators in a predetermined direction, a plurality of mirror drive devices provided corresponding to the plurality of plane mirrors, and a plurality of collimators that are supported so as to be independently movable in parallel within a predetermined plane and a plurality of mirror drive devices that drive the plurality of collimators. The collimator unit includes a support unit that supports a plane mirror so that it can swing freely, and a collimator control unit that controls the multiple collimators, the multiple collimator driving devices, and the multiple mirror driving devices, and a control unit that controls the collimator unit, wherein the control unit controls the multiple collimator driving devices and the multiple mirror driving devices to separately set each of the multiple collimators and the multiple plane mirrors to a predetermined position and orientation, and forms a plurality of collimated images formed by the multiple collimators at desired positions within the range of each imaging plane of the multiple cameras.
[0021] A camera unit calibration method according to one embodiment of the present invention is a camera unit calibration method that performs a predetermined measurement based on a plurality of images acquired by using a plurality of cameras to capture an image of the same object from different viewpoints, and includes controlling the plurality of collimator driving devices and the plurality of mirror driving devices to separately set each of the plurality of collimators and the plurality of plane mirrors to a predetermined position and orientation, forming a plurality of collimated images formed by the plurality of collimators at desired positions within the range of each imaging plane of the plurality of cameras, estimating actual imaging coordinates of the plurality of collimated images formed on each imaging plane of the plurality of cameras, and performing a calibration process to compare the actual imaging coordinates of the plurality of collimated images, and if the actual imaging coordinates of the plurality of collimated images do not match as a result of the calibration process, performing an image correction process to correct geometric distortion of the image so that the actual imaging coordinates of the plurality of collimated images match. Effect of the Invention
[0022] According to the present invention, it is possible to provide a camera unit calibration device and calibration method that can measure (calibrate) and correct geometric distortion that occurs in an image acquired by imaging an object placed at infinity with high precision over the entire range of the acquired image, in a space-saving manner without requiring a large amount of space. [Brief description of the drawings]
[0023] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a camera unit calibration device according to an embodiment of the present invention; [Diagram 2] A schematic diagram showing how a camera unit captures an object placed at infinity and the image that is captured (reference image during normal imaging) [Diagram 3] A schematic diagram showing how to capture an image of an object placed at infinity using a camera unit and the image that is captured (image of calibration target) [Figure 4] FIG. 1 is a schematic diagram illustrating the operation of calculating an image shift amount in an image to be calibrated, during calibration processing performed using the calibration device of an embodiment of the present invention; [Diagram 5] 1 is a flowchart showing an outline of the operation of a calibration device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] The present invention will be described below with reference to the illustrated embodiments. The drawings used in the following description are schematic, and the dimensional relationship and scale of each component may be different for each component in order to show each component at a size that can be recognized on the drawing. Therefore, the present invention is not limited to the illustrated form with respect to the quantity of each component, the shape of each component, the size ratio of each component, the relative positional relationship of each component, etc. shown in each drawing.
[0025] In the following description of this embodiment, "calibration" (also written as "calibration") refers to, for example, measuring the amount of image shift between the position (actual imaging coordinates) of an image of an object in multiple images (referred to as actual images) acquired by multiple cameras included in a camera unit to be calibrated when mounted on a vehicle, and the amount of image shift between the assumed imaging positions (assumed imaging coordinates) of the image of the object in multiple reference images (described later).
[0026] In addition, in the following description, "correction" refers to correcting the geometric distortion of an object in an image by, for example, applying a predetermined image processing based on the above-mentioned calibration results to multiple actual image data acquired by multiple cameras included in the camera unit to be calibrated.
[0027] First, a schematic configuration of a camera unit calibration device according to one embodiment of the present invention will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing a schematic configuration of a camera unit calibration device according to one embodiment of the present invention.
[0028] The arrows x, y, and z shown in Fig. 1 indicate three mutually orthogonal spatial axes. Here, the x-axis is a horizontal direction in a plane parallel to the paper surface in Fig. 1, and indicates a straight line perpendicular to the y-axis and z-axis. The y-axis is a direction perpendicular to the paper surface in Fig. 1, and indicates a straight line perpendicular to the x-axis and z-axis. The z-axis is a vertical direction in a plane parallel to the paper surface in Fig. 1, and indicates a straight line perpendicular to the x-axis and y-axis.
[0029] As shown in FIG. 1, the calibration device 1 of this embodiment is mainly composed of a collimator unit 10 and a control unit 20.
[0030] FIG. 1 conceptually illustrates a state in which the calibration device 1 of this embodiment is installed on a vehicle equipped with a camera unit 30 to be calibrated.
[0031] In Fig. 1, reference numeral 30 denotes a camera unit to be calibrated. The camera unit 30 illustrated in this embodiment is a so-called stereo camera unit having two cameras (31L, 31R) that are multiple imaging devices arranged side by side in the horizontal direction at a predetermined distance. The configuration of the camera unit 30 will be described later.
[0032] 1, reference numeral 100 denotes a windshield or the like of a vehicle (not shown) such as an automobile on which the camera unit 30 to be calibrated is mounted. The windshield or the like 100 is made of a plate-shaped optical member. The windshield or the like 100 will be described later.
[0033] As shown in FIG. 1, the collimator unit 10 is mainly composed of a plurality of collimators (11L, 11R), a plurality of collimator driving devices 15, a support unit 16, a collimator control unit 17, a plurality of plane mirrors (18LA, 18LB, 18RA, 18RB), and a plurality of mirror driving devices 19, etc.
[0034] A required number of collimators (11L, 11R) are provided corresponding to the two cameras (31L, 31R; described later) included in the camera unit 30 to be calibrated.
[0035] In this embodiment, an example configuration is shown in which two collimators are provided: a first collimator 11L corresponding to the left camera (first camera 31L) of the two cameras (31L, 31R) of the camera unit 30, and a second collimator 11R corresponding to the right camera (second camera 31R).
[0036] In the following description, the letters "L" and "R" added to the numerals accompanying the names of components are symbols to specify "left" and "right." Therefore, when indicating a component without specifying the left or right, the letters "L" and "R" may be omitted and only the numerals may be written.
[0037] Similarly, for similar components that are provided in multiple locations, the letter code "A," "B," etc. is added to the numeral code to indicate one specific component. Therefore, in this case, too, the letter notation may be omitted unless a specific component is being referred to.
[0038] For example, there are a first collimator 11L and a second collimator 11R as a plurality of collimators, but when the collimators are not specified, they may be simply referred to as collimators 11.
[0039] The collimators 11 are each configured as follows. The basic configuration of the collimators 11 themselves is substantially the same as that of conventionally known collimators. Therefore, the description of the configuration of the collimators 11 will be limited to the following brief description.
[0040] The collimator 11 is composed of a light source 12, a chart board 13, a collimator lens 14, etc. The collimator 11 is a device that generates and emits a parallel light beam (collimated light) by transmitting a light beam (diffused light) emitted from the light source 12 through the collimator lens 14.
[0041] In this case, a chart board 13 is disposed between the light source 12 and the collimator lens 14 at the focal position of the collimator lens 14. The chart board 13 is, for example, a board member on which a predetermined graphic pattern is formed, and is a plate-like member for forming a predetermined pattern image by transmitting or reflecting a light beam and forming an image on a predetermined imaging surface. The chart board 13 used in the multiple collimators 11 included in the calibration device 1 of this embodiment is assumed to be, for example, a transmissive chart board.
[0042] In the collimator 11 configured as described above, the light beam emitted from the light source 12 passes through the chart board 13 to form a chart image of a predetermined graphic pattern. The light beam that forms this chart image is collimated by the collimator lens 14 and emitted. The emitted light at this time is called collimated light. In Fig. 1, symbols C1 and C2 indicate the optical axes of the collimated light. 1 indicates the collimated light emitted from the collimator 11 and reflected by the plane mirror 18. Also, C2 in Fig. 1 indicates the collimated light reflected by the plane mirror 18 and then focused at a predetermined position on the imaging surface of each of the imaging elements 33 (described later) of the two cameras 31 of the camera unit 30. The chart image formed by this collimated light can be treated as equivalent to an image of an object placed at infinity, for example.
[0043] The collimator driving device 15 is connected to the base end of each collimator 11 and is a driving source that drives each collimator 11. Each collimator driving device 15 adjusts the direction in which the collimated light C1 of each collimator 11 is emitted. Each collimator driving device 15 can rotate each collimator 11 within a predetermined range indicated by the symbol R1 in FIG. 1 in a direction along a plane including the x-axis, with the base end of each collimator 11 as the rotation center. Although not shown in FIG. 1, each collimator 11 can also be rotated within the same predetermined range as in the x-axis direction in a direction along a plane including the y-axis.
[0044] With this configuration, each collimator 11 is configured to be able to form a collimated image at a predetermined position in a predetermined region that mainly includes the central portion of each image acquired by the two cameras 31.
[0045] Furthermore, each collimator driving device 15 and each collimator 11 are configured to be freely movable in parallel in the x-axis direction and the y-axis direction in FIG. 1, that is, in the xz plane and the xy plane.
[0046] The support unit 16 is a support unit that supports each collimator driving device 15 and allows each collimator driving device 15 and each collimator 11 to move in parallel within the xz plane and the xy plane.
[0047] Further, the support unit 16 is provided with a plurality of mirror support arms 16a. These mirror support arms 16a are support members that support each of the plurality of plane mirrors 18 so that they can swing independently. The mirror support arms 16a are provided at positions facing each other across each collimator 11. The mirror support arms 16a are formed to extend in a direction perpendicular to the surface of the support unit 16 on which the collimators 11 are supported.
[0048] A mirror drive device 19 is disposed at each tip of the plurality of mirror support arms 16a. A plane mirror 18 is disposed on each mirror drive device 19 so as to be capable of swinging freely.
[0049] The multiple plane mirrors 18 are components having mirror surfaces that receive collimated light C1 emitted from the collimator 11 and reflect it in a predetermined direction (specifically, toward the imaging element). To this end, each plane mirror 18 is configured to be driven by a mirror drive device 19. As a result, each plane mirror 18 is freely swingable in the direction of arrow R2 in FIG. 1 around the tip of the mirror support arm 16a as the center of rotation.
[0050] In the configuration of this embodiment, the multiple plane mirrors 18 include four plane mirrors provided corresponding to the two cameras 31 of the camera unit 30. The four plane mirrors 18 include two plane mirrors (first plane mirror 18LA, second plane mirror 18LB) corresponding to the first camera 31L, and two plane mirrors (third plane mirror 18RA, fourth plane mirror 18RB) corresponding to the second camera 31R.
[0051] Among these, the first plane mirror 18LA reflects the collimated image mainly toward a predetermined position in the left half area of the image acquired by the first camera 31L. The second plane mirror 18LB reflects the collimated image mainly toward a predetermined position in the right half area of the image acquired by the first camera 31L. The third plane mirror 18RA reflects the collimated image mainly toward a predetermined position in the left half area of the image acquired by the second camera 31R. The fourth plane mirror 18RB reflects the collimated image mainly toward a predetermined position in the right half area of the image acquired by the second camera 31R.
[0052] The multiple plane mirrors 18 are configured to reflect collimated images toward predetermined positions in predetermined areas mainly near the periphery within the entire range of each image captured by the two cameras 31.
[0053] The multiple mirror driving devices 19 are connected to the multiple plane mirrors 18, respectively, and are driving sources for swinging each of the plane mirrors 18 in a predetermined direction (the direction of the arrow R2 in FIG. 1).
[0054] The collimator control unit 17 is a component unit or electronic circuit that controls each collimator 11, each collimator driver 15, and each mirror driver 19 under the control of the control unit 20. The collimator control unit 17 controls the on / off control and light amount adjustment of each collimator 11. The collimator control unit 17 also controls each collimator driver 15 and mirror driver 19 to adjust the position and orientation of each collimator 11 and the orientation of the plane mirror 18. By performing this adjustment, the collimated light C1 from each collimator 11 is reflected at a predetermined position on the mirror surface of each plane mirror 18, and then forms an image as collimated light C2 at a desired position within the range of each imaging surface of the two cameras 31 of the camera unit 30 (within the entire range of the captured image).
[0055] The control unit 20 is a configuration unit or electronic circuit that comprehensively controls the entire calibration device 1. That is, the control unit 20 is connected to the collimator unit 10 and controls the collimator unit 10. The control unit 20 is also connected to an image recognition unit 35 (described later) in the camera unit 30 to be calibrated.
[0056] The control unit 20 includes a calibration circuit 20a and a correction circuit 20b. The calibration circuit 20a and the correction circuit 20b are components or electronic circuits that perform predetermined calibration and correction processes, respectively.
[0057] The calibration circuit 20a receives image data acquired by the camera unit 30 to be calibrated, and determines the position (actual imaging coordinates) of the image (collimated image) of the object in the image. The calibration circuit 20a also receives setting information (information on position, orientation, etc.) from each collimator 11, and determines the assumed imaging position (assumed imaging coordinates) of the image (collimated image) of the object corresponding to the setting information. The calibration circuit 20a then performs a calibration process to compare the actual imaging coordinates with the assumed imaging coordinates and measure the amount of image shift.
[0058] The correction circuit 20b performs a predetermined image correction process (i.e., a process such as correcting geometric distortion of an object in an image) on a plurality of actual image data acquired by the camera unit 30 to be calibrated, based on the result of the calibration process. The configuration of the calibration device 1 of this embodiment is as described above.
[0059] Incidentally, the camera unit 30 to be calibrated by the calibration device 1 of this embodiment is configured as a so-called stereo camera unit, as described above. For this reason, the camera unit 30 is mainly configured by two cameras 31, an image processing unit (IPU) 34, an image recognition unit 35, etc., as shown in FIG.
[0060] The two cameras 31 include a first camera 31L which is a left camera and a second camera 31R which is a right camera. The first camera 31L and the second camera 31R are arranged side by side in the horizontal direction at a predetermined distance. In this case, the two cameras 31 are arranged so that their optical axes O (see FIG. 1) are approximately parallel to each other.
[0061] The basic configuration of the camera 31 itself is substantially the same as that of a conventionally known camera, and therefore the configuration of the camera 31 itself will be merely briefly described below.
[0062] The camera 31 is mainly composed of an imaging lens 32, an imaging element 33, an imaging board (not shown), etc. The imaging lens 32 is composed of an optical lens that forms an optical image of an object. The imaging element 33 is a photoelectric conversion element that receives the optical image formed by the imaging lens 32, generates and outputs an electrical image signal. For example, a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, etc. are used as the imaging element 33. This imaging element 33 is generally mounted on an imaging board (not shown) on which a predetermined imaging circuit, etc. are mounted.
[0063] Image processing unit (hereinafter referred to as IPU) 34 is a component unit or electronic circuit that performs predetermined image processing on image data acquired by camera 31 and performs image processing such as edge detection processing for objects within the image range. IPU 34 also performs processing such as acquiring distance information from the amount of positional deviation of corresponding edges on the left and right images and generating image information including the distance information (distance image information).
[0064] Based on distance image information received from IPU 34, the image recognition unit 35 obtains various information such as the road curvature [1 / m] of the left and right dividing lines of the road on which the vehicle is traveling and the width between the left and right dividing lines (lane width).
[0065] Furthermore, the image recognition unit 35 performs predetermined pattern matching on the distance image information to recognize various three-dimensional objects on the road as well as road surface conditions, etc. Here, the recognition of three-dimensional objects includes, for example, the type of the three-dimensional object, the height of the three-dimensional object, the distance to the three-dimensional object, the speed of the three-dimensional object, the relative speed between the three-dimensional object and the vehicle, and the relative distance between the three-dimensional objects.
[0066] The camera unit 30 configured in this manner uses two cameras 31 to capture images of a predetermined range in front, including the same object, from different viewpoints at approximately the same time at a predetermined imaging period that is synchronized with each other, thereby obtaining two captured image data and generating a stereo image.
[0067] On the other hand, the windshield etc. 100 is a plate-like optical member provided in a vehicle (not shown) such as an automobile in which the camera unit 30 to be calibrated is mounted. Here, the windshield etc. 100 corresponds to a so-called front window glass.
[0068] The windshield 100 is made of a material such as almost colorless and transparent glass or acrylic, and has a complex curved surface. The windshield 100 is disposed in the front area of the imaging lens 32 of the camera unit 30 between the camera unit 30 to be calibrated and the object (corresponding to the collimated image formed by the collimator 11 in this embodiment).
[0069] All or part of the control unit 20, the calibration circuit 20a, the correction circuit 20b, the collimator control unit 17, etc. are configured by a processor including hardware. The same is true for the IPU 34, the image recognition unit 35, etc. included in the camera unit 30.
[0070] Here, the processor is configured by a well-known configuration including, for example, a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), a non-volatile memory, a non-volatile storage device, a non-transitory computer readable medium, and peripheral devices.
[0071] Software programs executed by the CPU and fixed data such as data tables are stored in advance in ROM, non-volatile memory, non-volatile storage devices, etc. Then, the CPU reads out the software programs stored in ROM, etc., expands them into RAM, and executes them, and the software programs appropriately refer to various data, etc., thereby realizing the functions of the above-mentioned components and units, etc.
[0072] The processor may be configured with a semiconductor chip such as a Field Programmable Gate Array (FPGA), etc. Each of the above components and units may be configured with an electronic circuit.
[0073] Furthermore, the software program may be in a form in which the whole or part is recorded as a computer program product on a portable disk medium such as a flexible disk, CD-ROM, or DVD-ROM, or on a non-transitory computer readable medium such as a card-type memory, HDD (Hard Disk Drive) device, or SSD (Solid State Drive) device.
[0074] In the calibration device 1 of the present embodiment configured as described above, the positions and orientations of the multiple collimators 11 of the collimator unit 10 and the orientations of the multiple plane mirrors 18 can be arbitrarily adjusted at a desired timing. In other words, the collimated light C2 obtained after the collimated light C1 emitted from the multiple collimators 11 is reflected by the plane mirror 18 and the optical axes O of the two cameras 31 of the camera unit 30 can be set non-parallel to each other. With this configuration, it is possible to capture a collimated image at a desired position within the entire range of the image acquired by the camera 31.
[0075] In the collimator unit 10 of this embodiment, the collimated light C1 emitted from the collimator 11 is reflected by the plane mirror 18, and the collimated light C2 reflected by the plane mirror 18 is guided to a desired position on the imaging surface of the imaging element 33 to form a collimated image. In this case, the collimated light C1 does not lose its properties as collimated light (parallel light beam) even after being reflected by the plane mirror 18. Therefore, the collimated image formed by the collimated light C2 reflected by the plane mirror 18 can also be treated as equivalent to an image of an object placed at infinity.
[0076] The operation of the calibration device 1 of this embodiment configured as above will be described below. First, the basic concept of the calibration process by the calibration device 1 of this embodiment will be described with reference to FIGS.
[0077] 2 and 3 are conceptual diagrams that show the state when an object placed at infinity is imaged using a camera unit and the image acquired at that time. Of these, Fig. 2 shows the state during normal imaging by the camera unit and the image acquired at that time (reference image). Also, Fig. 3 shows the state during imaging by the camera unit through the windshield and the image acquired at that time (calibration target image). Here, normal imaging in Fig. 2 refers to imaging in a situation where there is no windshield, as opposed to the imaging situation through the windshield in Fig. 3.
[0078] In the examples of Fig. 2 and Fig. 3, the symbol Obj indicates an image capturing object. In the examples of Fig. 2 and Fig. 3, the image capturing object Obj is shown in an arbitrary object shape (e.g., a vehicle). However, the image capturing object Obj is not limited to this illustrated form. The image capturing object Obj may be, for example, a chart consisting of a predetermined graphic pattern or the like that is defined in advance.
[0079] The image capturing object Obj is positioned directly opposite each of the imaging lenses 32 of the two cameras 31 of the camera unit 30, and is disposed at a position substantially at infinity. Here, it is assumed that the image capturing object Obj is a collimated image formed by a collimator (not shown).
[0080] Therefore, at this time, the optical axes O of the two cameras 31 of the camera unit 30 are parallel to each other and to the collimated light C that forms a collimated image of the image pickup object Obj.
[0081] In other words, a state in which the object to be imaged Obj and the two cameras 31 of the camera unit 30 are directly facing each other refers to a state in which the optical axes O of the imaging lenses 32 of the two cameras 31 coincide with the light flux (collimated light C) from the object to be imaged Obj.
[0082] 2 and 3, it is assumed that the image capturing object Obj (collimated image) is directly opposed to the camera unit 30. Therefore, at this time, the image capturing object Obj (collimated image) is formed in an approximately central region of each camera 31 of the camera unit 30. This situation corresponds to a situation in which collimated light (see symbol C) is emitted with the collimator 11 facing the camera unit 30 in the calibration device 1 of this embodiment. That is, this illustrates a situation in which a collimator image is formed on the imaging surface of each camera 31 without using the plane mirror 18.
[0083] In this way, when the imaging target object Obj is disposed facing the camera unit 30, the two images acquired by the two cameras 31 of the camera unit 30 are indicated by symbols 200L and 200R in Fig. 2. Here, symbol 200L indicates the left image acquired by the first camera (left camera) 31L. Also, symbol 200R indicates the right image acquired by the second camera (right camera) 31R.
[0084] In these two images 200, the image of the object (collimated image) is formed in the approximate center of each image. Here, reference symbol d1(L) in Fig. 2 indicates the image shift amount (assumed image coordinates) from the image center position of the image of the object (collimated image) in the left image 200L. Similarly, reference symbol d1(R) in Fig. 2 indicates the image shift amount (assumed image coordinates) from the image center position of the image of the object (collimated image) in the right image 200R.
[0085] Here, the image shift amounts d1(L) and d1(R) may cause image shifts in the x-axis direction and the y-axis direction on the xy plane of the image plane (i.e., the imaging surface). However, in this embodiment, only the image shift amount in the x-axis direction is considered.
[0086] That is, the image shift in the x-axis direction affects the accuracy of the parallax information. This affects the accuracy of the distance measurement. Therefore, in this embodiment, attention is focused only on the image shift in the x-axis direction, and the purpose is to calibrate and correct the image shift in the x-axis direction, as described later.
[0087] The image shift amounts d1(L) and d1(R) can be expressed as coordinates on the x-axis (shifts on the y-axis are ignored). In this case, the image shift amount is expressed in units of pixels that form the image.
[0088] When these two images 200 are superimposed, the result is as shown by reference symbol 200LR in Fig. 2. In Fig. 2, reference symbol d1 indicates the amount of image shift occurring between the image of the object in the left image 200L and the image of the object in the right image 200R.
[0089] Here, the image shift amount d1 can be expressed as the difference between the image shift amount d1(L) in the left image 200L and the image shift amount d1(R) in the right image 200R. That is, d1 = d1 (L) - d1 (R) In this case, the image shift amount d1 is the parallax occurring between the two images 200.
[0090] Generally, during normal imaging in which the object to be imaged Obj is positioned at an infinity position facing the camera unit 30 (at this time, the optical axis O and the collimated light C are parallel), the image shift (parallax) d1 between the two images 200 is basically d1 = 0 (zero).
[0091] In the normal imaging situation shown in Fig. 2, a windshield or the like (described later) is not present between the camera unit 30 and the imaging target object Obj. Therefore, the images acquired in such a situation are called reference images (200L, 200R). It is desirable that the parallax in the reference image 200 is zero.
[0092] On the other hand, the situation shown in Fig. 3 shows a state in which the camera unit 30 is mounted on a vehicle (not shown) and captures an image. In the example of Fig. 3, as in the example of Fig. 2, the image capture target Obj is positioned directly opposite each of the imaging lenses 32 of the two cameras 31 of the camera unit 30, and is positioned at a position almost at infinity. Also, it is assumed that the image capture target Obj is a collimated image.
[0093] Reference numeral 100 in Fig. 3 indicates a windshield or the like of a vehicle. It is assumed that this windshield or the like 100 exists between the camera unit 30 and the imaging target object Obj. An image acquired under such circumstances is referred to as a calibration target image. In addition, since this calibration target image is an image based on actual image data acquired by the camera unit 30 to be calibrated, it is also referred to as a real image.
[0094] As shown in FIG. 3, when the camera unit 30 and the object to be imaged Obj are faced directly to each other to capture an image, if an optical component such as a windshield 100 is present between the camera unit 30 and the object to be imaged Obj, distortion may occur in the image of the object to be imaged Obj.
[0095] 3, reference numerals 201L and 201R are examples of two images captured by the two cameras 31 of the camera unit 30. Here, reference numeral 201L indicates an actual image of the left image (left actual image) captured by the first camera (left camera) 31L. Also, reference numeral 201R indicates an actual image of the right image (right actual image) captured by the second camera (right camera) 31R.
[0096] In these two actual images 201, the image of the object (collimated image) is formed at a position shifted from the approximate center of each image. This indicates that the image is distorted due to the presence of, for example, a windshield 100.
[0097] Here, the reference character d0(L) in Fig. 3 indicates the image shift amount (actual imaging coordinates) from the image center position of the object image (collimated image) in the left actual image 201L. Similarly, the reference character d0(R) in Fig. 3 indicates the image shift amount (actual imaging coordinates) from the image center position of the object image (collimated image) in the right image 200R.
[0098] When these two real images 201 are superimposed, the result is as shown by reference symbol 201LR in Fig. 3. In Fig. 3, reference symbol d0 indicates the amount of image shift occurring between the image of the object in the left real image 201L and the image of the object in the right real image 201R.
[0099] 3 can be expressed as the difference between the image shift amount d0(L) in the left real image 201L and the image shift amount d0(R) in the right real image 201R. d0 = d0 (L) - d0 (R) In this case, the image shift amount d0 is the parallax occurring between the two images 201.
[0100] However, as described above, the image shift amount d0 in this case also includes the image shift caused by, for example, the image distortion caused by the windshield 100. Therefore, the image shift amount d0 does not accurately indicate the parallax occurring between the two images 201, and when a calculation such as measuring the distance to an object is performed based on this image shift amount d0, the calculated measurement result will be inaccurate.
[0101] Therefore, in the calibration device 1 of this embodiment, the image shift caused by the image distortion resulting from the presence of a windshield or the like 100 between the camera unit 30 and the object to be imaged Obj is measured (calibrated), and a predetermined image correction process is performed.
[0102] As described above, in Figs. 2 and 3, an example is described in which the camera unit 30 is disposed so as to directly face the image capture object Obj located at infinity.
[0103] However, in the calibration device 1 of this embodiment, the collimated light C2 of the multiple collimators 11 is further set non-parallel to each optical axis O of the two cameras 31 of the camera unit 30, so that a collimated image can be captured at a desired position within the entire range of the two images acquired by the two cameras 31.
[0104] With this configuration, by using the calibration device 1 of this embodiment, it is possible to perform calibration and correction over the entire range of the image captured by the camera unit 30.
[0105] Next, the operation of the calibration device 1 of this embodiment will be described below with reference to Fig. 4 and Fig. 5. Fig. 4 is a schematic diagram for explaining the operation when obtaining the amount of image shift in a calibration target image (actual image) during the calibration process performed using the calibration device of one embodiment of the present invention. Fig. 5 is a flow chart showing an outline of the operation of the calibration device of one embodiment of the present invention.
[0106] First, the calibration device 1 of this embodiment is installed at a predetermined position relative to a vehicle equipped with a camera unit 30 to be calibrated. At this time, a plate-shaped optical member equivalent to a windshield 100 or the like is placed between the camera unit 30 and the collimator unit 10 in the calibration device 1. In this state, an inspection (calibration process and correction process) is performed using the calibration device 1 of this embodiment. Figure 4 shows the state of each device at this time.
[0107] In step S1 of FIG. 5, the control unit 20 controls the collimator unit 10 through the collimator control unit 17 to adjust the positions and orientations of the collimators 11 and the plane mirrors 18.
[0108] This adjustment is made to form a collimated image by each collimated light emitted from each collimator 11 at a desired position within the range of the imaging surface of each image sensor 33 of each camera 31 of the camera unit 30 to be calibrated.
[0109] In other words, this adjustment is made so that the collimated image of each collimator 11 is captured at a desired position within the range of each image acquired by the camera unit 30.
[0110] In this case, the collimators 11 and the plane mirrors 18 are adjusted according to the desired position at which the collimated image is to be formed. For example, when the desired position at which the collimated image is to be formed is in a substantially central region of the imaging surface, it may be sufficient to adjust only the collimators 11. On the other hand, when the desired position at which the collimated image is to be formed is in a region closer to the periphery of the imaging surface, in addition to adjusting the collimators 11, the plane mirrors 18 are adjusted.
[0111] Next, in step S2, the control unit 20 estimates the actual imaging coordinates d0(L), d0(R) on the x-axis of the collimator image on the two actual images 201 (see Figure 4) acquired by the two cameras 31 of the camera unit 30.
[0112] 4 indicate two collimated images captured on the two real images 201. Of these, the symbol [Lx] is the first collimated image of the first collimator 11L captured on the left real image 201L. The symbol [Rx] is the second collimated image of the second collimator 11R captured on the right real image 201R.
[0113] 4, the symbol d0(L) denotes the actual image coordinate on the x-axis of the first collimated image [Lx], and the symbol d0(R) denotes the actual image coordinate on the x-axis of the second collimated image [Rx].
[0114] In addition, reference character d0 in FIG. 4 indicates the amount of image shift occurring between the collimated image [Lx] of the left real image 201L and the collimated image [Rx] of the right real image 201R.
[0115] In this case, basically, the angle θ(L) of the first collimator 11L and the angle θ(R) of the second collimator 11R are set to be equal values. Therefore, at this time, the collimated light C2 emitted from the first collimator 11L and reflected by the second flat plate 18LB and the collimated light C2 emitted from the second collimator 11R and reflected by the fourth flat plate 18RB are approximately parallel to each other.
[0116] By setting in this way, it is ideal that the actual image coordinate d0(L) on the x-axis of the collimated image [Lx] of the first collimator 11L and the actual image coordinate d0(R) on the x-axis of the collimated image [Rx] of the second collimator 11R are approximately equal to each other. However, in reality, due to the distortion caused by the windshield 100 or the like, d0(L)≠d0(R), and a certain image shift amount d0 occurs between the two.
[0117] 5, the control unit 20 compares the actual imaging coordinates d0(L) and d0(R) to check whether they match. If it is confirmed that d0(L)=d0(R), the process proceeds to step S5. If it is confirmed that d0(L)=d0(R) does not match, the process proceeds to step S4.
[0118] In step S4, the control unit 20 executes a predetermined image correction process including geometric distortion correction of the captured collimated image and alignment for the two actual images 201 through the correction circuit 20b. This image correction process is, for example, a correction process for matching the actual imaging coordinates d0(L) and d0(R). As various conventionally known correction processes are applied to this image correction process, detailed description thereof will be omitted.
[0119] In step S5, the control unit 20 checks whether the calibration and correction processes have been completed for the entire range of the two actual images 201 captured by the two cameras 31. If it is not confirmed that the calibration and correction processes have been completed for the entire range, the process returns to step S1 described above and repeats the subsequent processes. If it is confirmed that the calibration and correction processes have been completed for the entire range, the process ends (END).
[0120] As described above, according to the embodiment, by applying the collimator 11, it is possible to obtain an image of the object equivalent to the image of the object placed at infinity. In this case, the plane mirror 18 is used to form the collimator image mainly in the peripheral region of the image. This configuration can contribute to saving the space required for the examination.
[0121] Furthermore, by setting the collimated light C2 of the multiple collimators 11 to be non-parallel to the optical axis O of the camera unit 30 to be calibrated, a collimated image can be captured at a desired position within the entire range of the image acquired by the camera unit 30.
[0122] Therefore, accurate calibration (measurement) can be performed in a small space within the entire imaging range, and appropriate image correction processing based on the calibration results can be performed within the entire range of the image. Therefore, the calibration device 1 of this embodiment can realize more accurate calibration processing and correction processing of the camera unit 30.
[0123] The present invention is not limited to the above-mentioned embodiment, and various modifications and applications can be implemented within the scope of the gist of the invention. Furthermore, the above-mentioned embodiment includes inventions at various stages, and various inventions can be extracted by appropriate combinations of the disclosed multiple constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the above-mentioned one embodiment, if the problem to be solved by the invention can be solved and the effect of the invention can be obtained, the configuration from which the constituent elements are deleted can be extracted as the invention. Furthermore, the constituent elements across different embodiments may be appropriately combined. The present invention is not restricted by its specific embodiment except as limited by the attached claims. [Explanation of symbols]
[0124] 1...Calibration device 10...Collimator unit 11L…First collimator 11R…Second collimator 12...Light source 13. Chart board 14...Collimator lens 15...Collimator drive unit 16…Support unit 16a…Mirror support arm 17...Collimator control unit 18LA…1st plane mirror 18LB…Second plane mirror 18RA...Third plane mirror 18RB…4th plane mirror 19...Mirror drive mechanism 20…Control unit 20a…Calibration circuit 20b…Compensation circuit 30…Camera unit 31L…Left camera (first camera) 31R…Right camera (second camera) 32...imaging lens 33...Image sensor 34...Image processing unit (IPU) 35…Image recognition unit 100…Windshield, etc. 200L…Left image 200R…Right image 201L…Left actual image 201R…Right image C, C1, C2...Collimated light (optical axis) O: Optical axis of the imaging optical system Obj…Image subject R1: Collimator rotation direction R2: Swing direction of plane mirror d1: Image shift amount (reference image) d1(L), d1(R)…Assumed imaging coordinates d0: Image shift amount (calibration target image, actual image) d0(L),d0(R)…Real image coordinates
Claims
1. A camera unit calibration device that performs a predetermined measurement based on a plurality of images obtained by capturing images of the same object from different viewpoints using a plurality of cameras, a collimator unit including: a plurality of collimators provided corresponding to the plurality of cameras, a plurality of collimator drive devices provided corresponding to the plurality of collimators, a plurality of plane mirrors that reflect collimated light from the plurality of collimators in a predetermined direction, a plurality of mirror drive devices provided corresponding to the plurality of plane mirrors, a support unit that supports the plurality of collimators so that they can move in parallel independently within a predetermined plane and supports the plurality of plane mirrors so that they can swing freely; and a collimator control unit that controls the plurality of collimators, the plurality of collimator drive devices, and the plurality of mirror drive devices; A control unit for controlling the collimator unit; Equipped with The control unit Controlling the plurality of collimator drive devices and the plurality of mirror drive devices to set the plurality of collimators and the plurality of plane mirrors to respective predetermined positions and orientations; A plurality of collimated images formed by the plurality of collimators are formed at desired positions within the range of each of the imaging planes of the plurality of cameras.
13. A camera unit calibration device comprising:
2. an optical axis of each of the collimated lights from the plurality of collimators and an optical axis of each of the plurality of cameras are set to be non-parallel to each other; The optical axes of the plurality of collimated beams are set parallel to each other.
2. The camera unit calibration device according to claim 1.
3. The control unit further comprises: Estimating actual imaging coordinates of the collimated images formed on the imaging planes of the cameras; a calibration process for comparing actual image coordinates of the plurality of collimated images; an image correction process for performing a geometric distortion correction of an image so that the actual image coordinates of the plurality of collimated images coincide with each other when the actual image coordinates of the plurality of collimated images do not coincide with each other as a result of the calibration process; 2. The camera unit calibration device according to claim 1, wherein the following is executed.
4. 2. The camera unit calibration device according to claim 1, wherein the camera unit is a stereo camera unit.
5. A method for calibrating a camera unit, which performs a predetermined measurement based on a plurality of images obtained by capturing images of a same object from different viewpoints using a plurality of cameras, comprising the steps of: Controlling the plurality of collimator drive devices and the plurality of mirror drive devices to set the plurality of collimators and the plurality of plane mirrors to respective predetermined positions and orientations; forming a plurality of collimated images formed by the plurality of collimators at desired positions within the range of each of the imaging planes of the plurality of cameras; Estimating actual imaging coordinates of the collimated images formed on the imaging planes of the cameras; A calibration process is performed to compare the actual image coordinates of the plurality of collimated images; A camera unit calibration method characterized in that, if the calibration process results in the actual imaging coordinates of the multiple collimated images not matching, an image correction process is executed to correct geometric distortion of the image so that the actual imaging coordinates of the multiple collimated images match.
Citation Information
Patent Citations
Stereo camera calibrating device and calibrating method
JP2012132739A
Calibration method and calibration device
JP2019090755A
Method for correcting stereocamera and device for correcting stereocamera
WO2021024612A1
