Stereoscopic camera device and calibration method

Abstract

A stereoscopic camera device according to the present invention is provided with: a storage unit that stores information relating to a parallax error in a horizontal direction with respect to a vertical shift in a plurality of images captured by a plurality of cameras respectively through a refractor that refracts light; a matching unit that matches a plurality of images obtained by imaging the same subject by the plurality of cameras, and determines vertical shifts of the plurality of images; and a parallax error estimation unit that estimates a parallax error of the plurality of matched images on the basis of the vertical offset and the correlation information. And a distance calculation unit that corrects the parallax of the matched image using the parallax error.

Description

Technical Field

[0001] The invention relates to a stereo camera device and a calibration method. Background Art

[0002] With the development of computer image technology, stereo cameras are becoming more and more widely used. On the other hand, in recent years, with the AI ​​(Artificial Intelligence) of automobiles, the technology of recognizing images taken by stereo cameras is also widely used in ADAS (Advanced Driver Assistance Systems) and autonomous driving. Unlike ordinary cameras, stereo cameras can realize three-dimensional object recognition of images taken by stereo cameras. Three-dimensional object recognition is a technology that uses two cameras set at a certain distance (baseline length) to obtain images of the same object respectively, and calculates the distance from the two cameras to the object based on the parallax of the object according to the principle of triangulation to realize the three-dimensionalization of the image.

[0003] Existing stereo cameras mainly support the technology of enabling the vehicle to avoid objects located in front of the vehicle by obtaining distance information to the objects. This technology is mainly used when the car is driving straight, and the vehicle can avoid the objects in front. However, as safety standards increase, the technology of detecting and avoiding pedestrians and bicycles at intersections before the vehicle enters the intersection is also required.

[0004] In order for the vehicle to avoid pedestrians and bicycles at intersections, a wider field of view than previous stereo cameras for ADAS is required. On the other hand, if a wide-angle lens is used to expand the field of view, the distortion of the captured image becomes larger. Therefore, if the calibration to correct the distortion of the image is incomplete, pixel offsets will occur in the images captured by the left camera and the right camera of the stereo camera. As a result, sometimes problems may occur in the process of detecting objects by measuring the distance of objects projected on the image, and the distance is measured incorrectly. In addition, in the following description, the pixel offset in the vertical direction of the images captured by the left camera and the right camera is referred to as "vertical offset". In addition, the pixel offset in the horizontal direction affects the parallax detection of the stereo camera, so it is called "parallax error".

[0005] In order to cope with such a wide field of view, there are the following problems. That is, in the wide-angle region, the incident angle to the front windshield of the vehicle becomes larger, and the pixel shift caused by the refraction of the front windshield, that is, the parallax, becomes larger. On the other hand, in the existing narrow-angle stereo camera, since the incident angle to the front windshield is small, it is almost unnecessary to consider the pixel shift caused by the front windshield. Therefore, there is a need for a parallax calibration method for a stereo camera with a wide field of view that can capture the outside world through the front windshield.

[0006] As a calibration method for such a wide angle of view, the techniques described in the following Patent Documents 1 and 2 have been proposed. Patent document 1 states that "the wider the field of view angle, the greater the parallax deviation caused by the refraction of the light path when passing through the front windshield", and states that "the correction parameters of the parallax deviation are summarized as a function or table of the horizontal position within the field of view angle, and stored as an optical axis characteristic in a storage device", and "the image is further corrected using the optical axis characteristic obtained by calibration".

[0007] Patent document 2 states that "the front windshield differs according to the type of vehicle, and therefore the deformation of the glass also differs according to the type of vehicle. In order to evaluate the effect of glass deformation on the captured image, it is necessary to actually drive or simulate all vehicle types, which inadvertently requires a huge amount of time and money." With this as the subject, it is recorded that "a plurality of geometric tables are combined, and the geometric distortion of the stereo camera and each protective cover is removed, or the geometric distortion of different protective covers is reproduced." Prior art literature Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2021-25868 Patent Document 2: Japanese Patent Application Publication No. 2017-62198 Summary of the invention Problem that the invention aims to solve

[0009] In the technology disclosed in Patent Document 1, which corrects images based on the shape of the designed windshield (window of the vehicle cabin), the technology is valid as long as the shapes of all windshields installed in the vehicle are the same. However, in reality, the positional relationship between the windshield and the camera varies depending on their respective installation positions. Therefore, when there are manufacturing deviations in the horizontal and vertical curvature radii of the windshield, the thickness of the windshield, and the relative relationship between the installation position of the windshield and the stereo camera, the allowable value (for example, the deviation described below) cannot be met. Figure 5 The parallax error shown is within 0.25 pixels).

[0010] In addition, there is a problem with the widening of the angle of the stereo camera in the technology disclosed in Patent Document 2. In this technology, a calibration chart (hereinafter referred to as a "chart") is used to correct the image, but as the angle of the stereo camera becomes wider, a large chart needs to be used.

[0011] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to perform calibration for correcting the parallax of a stereo camera that captures the outside world through a refracting body without using a graph. Technical means of solving problems

[0012] The stereo camera device of the present invention comprises: a storage unit that stores information related to parallax errors in the horizontal direction relative to vertical offsets of multiple images respectively photographed by multiple cameras through a refractor that refracts light, the relevant information being obtained based on refractor characteristics including the posture of the refractor installed on a vehicle and the shape of the refractor, and the postures of multiple cameras arranged on the vehicle; a matching unit that matches multiple images obtained by photographing the same subject by multiple cameras to obtain vertical offsets of the multiple images; a parallax error estimating unit that estimates the parallax errors of the matched multiple images based on the vertical offset and the relevant information; and a distance calculating unit that uses the parallax errors to correct the parallax of the matched images. Effects of the Invention

[0013] According to the present invention, it is possible to perform calibration for correcting the parallax of a stereo camera that captures the outside world through a refractor without using a graph. The other objects, structures, and effects than those described above will become clear from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Represents the relationship between the width of the chart and the field of view of the stereo camera. Figure 2 This is an explanatory diagram showing an example of a subject photographed by a stereo camera according to one embodiment of the present invention. Figure 3 This is a block diagram showing an example of the overall configuration of a stereo camera device according to one embodiment of the present invention. Figure 4 This is a block diagram showing an example of the hardware configuration of a computer according to one embodiment of the present invention. Figure 5 This is a diagram showing an example of a simulation result of a parallax error calculated when a front windshield is sandwiched between a stereo camera and a subject according to an embodiment of the present invention. Figure 6 is a graph showing an example of delta parallax error with respect to the horizontal field of view of a stereo camera. Figure 7 This is a flowchart showing an example of a calibration method according to an embodiment of the present invention. Figure 8 This is a diagram showing an example of horizontal stereo matching of left and right images according to one embodiment of the present invention. Fig. 9 This is a diagram showing an example of matching of left and right images in the vertical direction according to one embodiment of the present invention. Fig.10 This is a schematic diagram showing an example of a calculation process of a parallax error according to one embodiment of the present invention. Fig.11 This is a diagram showing the structure of a correlation table according to one embodiment of the present invention. Fig.12 This is a diagram for explaining the principle of estimating a parallax error based on a vertical offset by a parallax error estimation unit according to one embodiment of the present invention. Fig.13 This is a diagram showing nine types of deviation examples according to one embodiment of the present invention. Fig.14 1 is a diagram showing the correlation between the delta vertical offset and the delta parallax error according to one embodiment of the present invention. Fig.15 It is a diagram showing simulation results of residual values ​​of parallax errors obtained for horizontal viewing angles of 20 degrees, 40 degrees, and 60 degrees according to one embodiment of the present invention. DETAILED DESCRIPTION

[0015] Hereinafter, the mode for implementing the present invention will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, sometimes when there are multiple components having substantially the same or similar functions, different subscripts are added to the same symbols for description. In addition, sometimes when there is no need to distinguish these multiple components, the subscripts are omitted for description. The embodiments of the present invention described below are examples for illustrating the present invention, and appropriate omissions and simplifications have been made for clarity of description. The present invention may also be implemented in various other ways. Unless otherwise specified, each constituent element may be singular or plural. Furthermore, the present invention can be applied to a computing device for vehicle control that can communicate with an on-vehicle ECU (Electronic Control Unit) for, for example, an advanced driver assistance system (ADAS) or autonomous driving (AD).

[0016] [One embodiment] A stereo camera device and a calibration method according to an embodiment of the present invention are an example of an invention for solving the above-mentioned problems. The contents of the stereo camera device and the calibration method according to this embodiment are described below. In addition, in the following description, correcting the parallax of two images is referred to as "calibration".

[0017] The stereo camera is calibrated at four stages: (1) manufacturing plant, (2) vehicle factory, (3) dealership, and (4) while the vehicle is in motion. (1) Calibration at the manufacturing plant When the stereo camera is manufactured at a manufacturing factory, optical axis adjustment of the stereo camera is performed at the manufacturing factory. (2) Calibration at the vehicle factory After the stereo camera is shipped from the manufacturing factory, it is mounted on the vehicle at the vehicle factory and calibrated. Calibration performed at the vehicle factory is also called "calibration." Information such as a calibration table obtained by calibration is stored in the stereo camera.

[0018] (3) Calibration at the dealer After the vehicle factory calibration is performed, a dealer calibration is performed when the vehicle is sent to a dealer. Dealer calibration is mainly performed after the glass is replaced when the vehicle is repaired. (4) Calibration while driving Finally, calibration is performed while the vehicle is moving. Calibration while the vehicle is moving is called "automatic adjustment." The calibration method of this embodiment is mainly used during (3) dealer calibration and (4) vehicle driving. Since dealers often have a small space for installing equipment such as a chart, the calibration without a chart of this embodiment is preferred.

[0019] Here, refer to Figure 1 The graph width of the graph used in calibration is described below. Figure 1 Represents the relationship between the width of the chart and the field of view of the stereo camera. Figure 1 The horizontal axis is the field of view [degrees], and the vertical axis is the required chart width [m]. Here, the distance between the stereo camera and the chart is set to 3m. Figure 1 As shown, for example, when the viewing angle is 40 degrees, the required chart width is about 2 meters. On the other hand, when the viewing angle is 120 degrees, the required chart width exceeds 10 meters. Charts with such wide chart widths are difficult to handle and require significant modifications to vehicle manufacturing plants and dealer inspection plants.

[0020] Figure 2 1 is an explanatory diagram showing an example in which the stereo camera 21 captures an image of the subject 24 . The stereoscopic panel camera 21 is mounted on the inner side of the front windshield 22 (inside the vehicle 20), and is composed of two cameras arranged in the horizontal direction and arranged on the left and right. The refractor (front windshield 22) has the property of refracting the light passing through it, and can be either transparent or translucent. In addition, of the two cameras that constitute the stereoscopic camera 21, the camera on the left side is called the left camera 21L, and the camera on the right side is called the right camera 21R. In addition, the left camera 21L and the right camera 21R are sometimes referred to as "left and right cameras", and are collectively referred to as the stereoscopic camera 21.

[0021] Figure 2 21 shows a scene where the stereo camera 21 detects the subject 24 through the front windshield 22. Figure 2It can be seen that the horizontal field angle of the stereo camera 21 (about 120 degrees) is wider than that of conventional cameras (about 40 degrees). In addition, the left camera 21L and the right camera 21R can each capture the outside world including the subject 24 at a horizontal field angle of about 120 degrees.

[0022] The posture of the windshield glass 22 installed on the vehicle (vehicle 20) varies depending on the model. In addition, even if it is the same model, the posture of the windshield glass 22 installed on the vehicle (vehicle 20) is slightly different.

[0023] Next, refer to Figure 3 A configuration example of a stereo camera device 10 according to an embodiment will be described. Figure 3 It is a block diagram showing an overall configuration example of a stereo camera device 10 according to one embodiment. The stereo camera device 10 includes an imaging unit 100, an external input unit 101, a correction unit 102, a storage unit 103, a matching unit 104, a noise removal unit 105, a parallax error estimation unit 106, a distance calculation unit 107, and a control unit 108. The stereo camera device 10 may be configured as a part of an ECU (Electronic Control Unit) mounted on a vehicle.

[0024] The camera unit 100 is connected to the calibration unit 102. The camera unit (camera unit 100) acquires a plurality of images captured by a plurality of cameras (stereo cameras 21) arranged on the vehicle (vehicle 20) through the refractor (front windshield 22). The camera unit 100 is a device that can acquire images obtained by capturing the outside of the vehicle, such as a visible light camera or an infrared camera, and is composed of a plurality of cameras arranged in a horizontal direction. Therefore, the camera unit 100 is configured not only as a camera that can detect the outside world through visible light, but also as a camera that can detect the outside world based on light other than visible light, such as infrared light. However, the camera unit 100 is not limited to a visible light camera or an infrared camera. In this case, the camera unit 100 may be replaced by an outside detection unit that can detect the outside world with light of a specified wavelength. In addition, the object (e.g., subject 24) captured by the camera unit 100 may be a person, another vehicle, a landscape, or a conventional calibration chart.

[0025] The camera unit 100 is composed of Figure 2 The left camera 21L and the right camera 21R (stereo camera 21) shown in the figure are composed. Then, the camera unit 100 outputs the left and right images of the scenery in front of the vehicle to the correction unit 102. Each pixel of the image represents a brightness value. In addition, the image is not limited to the RGB image separated from the color image, and can also be a black and white image.

[0026] The external input unit 101 is connected to the storage unit 103. The external input unit 101 inputs information related to the parallax error and vertical offset of the two images, which is pre-calculated based on design values ​​such as the deviation of the installation posture of the stereo camera 21 and the installation posture and shape of the front windshield 22, and sends it to the storage unit 103. The information related to the parallax error and vertical offset is described later. Fig.11 The correlation table shown in FIG. 1 is stored in the storage unit 103. In this specification, the displacement of the image in the horizontal direction is referred to as "parallax error". In addition, as the correlation information between the parallax error and the vertical displacement, Fig.10 The image diagram is shown in FIG. Fig.14 The proportional relationship is illustrated in FIG.

[0027] In addition, the external input unit 101 inputs a reference parallax error and a reference vertical offset, which are calculated in advance by an external calculation device based on design values ​​such as the deviation of the installation posture of the stereo camera 21 and the installation posture and shape of the front windshield 22, and sends them to the storage unit 103. Here, the reference parallax error and the reference vertical offset are values ​​generated by the refraction of the front windshield 22 (design value), which are different from the above-mentioned related information on the parallax error and the vertical offset calculated based on the installation posture, shape, etc. of the stereo camera 21 and the front windshield 22. The reference parallax error and the reference vertical offset are stored in the storage unit 103. Regarding the reference parallax error and the reference vertical offset, the reference parallax error and the reference vertical offset will be used. Figure 8 and Fig. 9 This will be described later.

[0028] Furthermore, the external input unit 101 inputs a calibration table obtained by calibration at the factory and transmits it to the storage unit 103. The storage unit 103 stores the calibration table obtained by calibration at the factory.

[0029] The correction unit 102 is connected to the camera unit 100, the storage unit 103, and the matching unit 104. The correction unit 102 uses the correction table read from the storage unit 103 to correct both the left and right images obtained from the camera unit 100. The correction of the left and right images here is a process of correcting the influence of the distortion of each lens of the left camera 21L and the right camera 21R, rather than a process of correcting the parallax of the left and right images. Then, the correction unit 102 sends the corrected left and right images to the matching unit 104.

[0030] The storage unit 103 is connected to the external input unit 101, the correction unit 102, the matching unit 104, the noise removal unit 105, the parallax error estimation unit 106 and the distance calculation unit 107. The storage unit 103 is a non-volatile memory configured inside the stereo camera 21, and can store various information. For example, the storage unit (storage unit 103) stores the reference parallax error and reference vertical offset pre-calculated based on the design value of the refractor (front windshield 22). The reference parallax error and reference vertical offset are information input from the external input unit 101, and represent the parallax error and vertical offset before the front windshield 22 is installed on the vehicle 20. In addition, the reference parallax error and reference vertical offset are stored in the storage unit (storage unit 103) when the multiple cameras (stereo cameras 21) are shipped.

[0031] In addition, the storage unit 103 stores the correction table input from the external input unit 101, and sends the correction table to the correction unit 102. In addition, the storage unit (storage unit 103) stores the related information of the horizontal parallax error relative to the vertical offset of the multiple images of the outside world of the vehicle (vehicle 20) respectively captured by the multiple cameras (left camera 21L and right camera 21R) through the refractor (windshield 22) that refracts the light. The related information is obtained based on the refractor characteristics (characteristics of the windshield 22) including the posture of the refractor (windshield 22) installed on the vehicle (vehicle 20) and the shape of the refractor (windshield 22), and the postures of the multiple cameras (left camera 21L and right camera 21R) arranged on the vehicle (vehicle 20). The windshield 22 has the refractor characteristics (characteristics of the windshield 22) such as the horizontal curvature of the windshield 22, the vertical curvature of the windshield 22, and the thickness of the windshield 22.

[0032] In addition, the storage unit 103 stores the parallax and vertical offset input from the matching unit 104. Here, the storage unit 103 sends the vertical offset of a plurality of frames input from the matching unit 104 in a plurality of processes to the noise removal unit 105. In addition, the storage unit 103 sends the information related to the vertical offset and the parallax error to the parallax error estimation unit 106. In addition, the storage unit 103 sends the parallax input from the matching unit 104 to the distance calculation unit 107.

[0033] In addition, the reference parallax error and the reference vertical offset may be stored in the storage unit (storage unit 103) before the refractor (front windshield 22) is mounted on the vehicle (vehicle 20). In this case, the amount of change relative to the reference parallax error after the refractor (front windshield 22) is actually mounted on the vehicle (vehicle 20) is δ parallax error, and the amount of change relative to the reference vertical offset is δ vertical offset.

[0034] The matching unit 104 is connected to the correction unit 102 and the storage unit 103. The matching unit (matching unit 104) matches multiple images of the same subject obtained by multiple cameras (stereo cameras 21) to obtain the vertical offset of the multiple images. For example, the matching unit 104 obtains the corrected left and right images from the correction unit 102, and matches the left and right images to calculate the parallax and vertical offset. The matching unit 104 sends the calculated parallax and vertical offset to the storage unit 103, and stores the parallax and vertical offset in the storage unit 103.

[0035] The noise removal unit 105 is connected to the storage unit 103 and the parallax error estimation unit 106. The noise removal unit 105 takes the average value of the vertical offsets of the plurality of frames read from the storage unit 103. Next, the noise removal unit 105 uses a filter to remove invalid vertical offsets mixed as noise from the calculation of the average value, and sends the average value of the vertical offsets to the parallax error estimation unit 106. In addition, if noise removal is not required, the noise removal unit 105 may not be provided in the stereo camera device 10.

[0036] The parallax error estimation unit 106 is connected to the storage unit 103, the noise removal unit 105, and the distance calculation unit 107. The parallax error estimation unit (parallax error estimation unit 106) estimates the parallax error of the matched multiple images based on the vertical offset and related information read from the storage unit 103. Therefore, when receiving the average value of the vertical offset from the noise removal unit 105, the parallax error estimation unit 106 reads the related information between the parallax error and the vertical offset from the storage unit 103. In addition, the parallax error estimation unit 106 reads the reference parallax error and the reference vertical offset from the storage unit 103, calculates the delta vertical offset based on the average value of the vertical offset received from the noise removal unit 105, and then calculates the delta parallax error.

[0037] Therefore, the parallax error estimation unit (parallax error estimation unit 106) calculates a δ vertical offset, which is the amount of change in the vertical offset caused by the refractor (front windshield 22) actually mounted on the vehicle (vehicle 20), based on the reference vertical offset and the vertical offset read from the storage unit (storage unit 103). Next, the parallax error estimation unit (parallax error estimation unit 106) calculates a δ parallax error, which is the amount of change in the parallax error caused by the refractor (front windshield 22) actually mounted on the vehicle (vehicle 20), based on the calculated δ vertical offset and the related information read from the storage unit (storage unit 103). Then, the parallax error estimation unit 106 sends the estimated δ parallax error to the distance calculation unit 107.

[0038] In addition, the parallax error estimation unit 106 sends the calculated δ parallax error and δ vertical offset to the storage unit 103, and stores the δ parallax error and δ vertical offset in the storage unit 103. As described above, the δ parallax error and δ vertical offset represent the amount of change before and after the front windshield glass 22 is installed on the vehicle 20. That is, the amount of change of the plurality of cameras (stereo cameras 21) relative to the reference parallax error after shipment is the δ parallax error, and the amount of change relative to the reference vertical offset is the δ vertical offset. For details of the δ vertical offset and the δ parallax error, the following will be described. Figure 8 and Fig. 9 This will be described later.

[0039] When the noise removal unit 105 is not provided in the stereo camera device 10, the parallax error estimation unit (parallax error estimation unit 106) accumulates the vertical offsets obtained at the same position of the plurality of images captured at different times by the matching unit 104 a plurality of times and calculates the average value of the vertical offsets. The parallax error estimation unit (parallax error estimation unit 106) estimates the parallax error based on the average value of the vertical offsets and the related information. By calculating the average value of the vertical offsets in this way, it is possible to prevent the parallax error estimated from the vertical offset from becoming excessively large when the vertical offset calculated only once is an abnormal value.

[0040] The distance calculation unit 107 is connected to the storage unit 103 and the parallax error estimation unit 106. The distance calculation unit (distance calculation unit 107) corrects the parallax of the matched multiple images using the parallax error. At this time, the distance calculation unit (distance calculation unit 107) corrects the parallax using the reference parallax error and the δ parallax error. Therefore, the distance calculation unit 107 receives the δ parallax error from the parallax error estimation unit 106, and receives the parallax and the reference parallax error from the storage unit 103. Then, the distance calculation unit 107 corrects the parallax by removing the reference parallax error and the δ parallax error from the parallax. In addition, the distance calculation unit (distance calculation unit 107) calculates the distance from the vehicle (vehicle 20) to the external object using the parallax corrected by the parallax error. Thereafter, the distance calculation unit 107 sends the distance information including the distance calculated for each object to the control unit 108.

[0041] The control unit 108 is connected to the distance calculation unit 107. When receiving the distance information from the distance calculation unit 107, the control unit 108 issues a warning and performs automatic control such as calling the driver's attention or applying the brakes according to the distance of each object.

[0042] The functional blocks 100 to 106 are responsible for processing performed before the vehicle travels, for example, by a vehicle dealer. On the other hand, the functional blocks 100 to 108 are responsible for processing performed while the vehicle travels.

[0043] <Computer Hardware Configuration Example> Next, a hardware configuration example of the computer 50 constituting the stereo camera device 10 will be described. Figure 4 1 is a block diagram showing an example of the hardware configuration of a computer 50. The computer 50 is an example of hardware used as a computer capable of operating as the stereo camera device 10 of the present embodiment. The stereo camera device 10 of the present embodiment is implemented by the computer 50 (computer) executing a program. Figure 3 The functional blocks shown cooperate to perform a parallax error correction method.

[0044] The computer 50 includes a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, and a RAM (Random Access Memory) 53, each of which is connected to a bus 54. In addition, the computer 50 includes a nonvolatile memory 55 and a network interface 56.

[0045] The CPU 51 reads the program code of the software that realizes each function of the present embodiment from the ROM 52, loads it into the RAM 53, and executes it. Variables or parameters generated in the middle of the calculation processing of the CPU 51 are temporarily written in the RAM 53, and these variables or parameters are read appropriately by the CPU 51. However, the CPU 51 and the GPU (Graphics Processing Unit) may be used together. Figure 3 The functions of the functional blocks 100 to 107 shown are implemented by the CPU 51 .

[0046] The nonvolatile memory 55 is, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), a floppy disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory, etc. In the nonvolatile memory 55, in addition to the OS (Operating System) and various parameters, a program for making the computer 50 function is recorded. The ROM 52 and the nonvolatile memory 55 record programs and data necessary for the operation of the CPU 51, and are used as an example of a computer-readable non-transient storage medium that stores programs executed by the computer 50. Figure 3 The function of the storage unit 103 shown is implemented by the RAM 53 , but may be implemented by the nonvolatile memory 55 .

[0047] The network interface 56 uses, for example, a NIC (Network Interface Card) and can transmit and receive various data between devices via an in-vehicle LAN (Local Area Network) connected to a terminal of the NIC, a dedicated line, or the like.

[0048] Here, we will provide a quantitative description of existing issues. Figure 5 This is a diagram showing an example of a simulation result of a parallax error calculated when a front windshield is sandwiched between a stereo camera and a subject. Figure 5 The vertical axis represents the parallax error [pixel], and the horizontal axis represents the horizontal field angle [degree] of the stereo camera. In addition, the calculation conditions in the simulation processing of the parallax error are as follows. In addition, the allowable value of the parallax error is 0.25 pixel. <Calculation conditions (design conditions)> Lens focal length: 5.41mm Lens projection: ftanθ Glass curvature radius (horizontal): 5.5m Glass curvature radius (vertical): 3.5m Glass refractive index: 1.52 Glass thickness: 4.7mm Glass tilt: 30 degrees Sensor pixel pitch: 2.25μm Camera baseline length: 180mm Distance between lens and glass: 40mm

[0049] like Figure 5 As shown, for example, when the horizontal viewing angle exceeds 70 degrees (35 degrees left + 35 degrees right), the parallax error exceeds the allowable value of 0.25 pixels. Therefore, as disclosed in Patent Document 1, it is also possible to correct the image based on the shape of the designed windshield.

[0050] Figure 6 : is a graph showing an example of the delta parallax error with respect to the horizontal viewing angle of the stereo camera. The deviation with respect to the parallax in the horizontal direction obtained based on the above-mentioned design conditions is referred to as the "delta parallax error". Figure 6 The vertical axis represents the delta parallax error [pixel], and the horizontal axis represents the horizontal viewing angle [degree]. In the figure, the dotted line represents the allowable value 41 of the delta parallax error of -0.25 pixel and +0.25 pixel.

[0051] Figure 6 The graph 40 shown shows Figure 5 The simulation result of the change in the parallax error shown is calculated when the stereo camera is rotated 3 degrees in the yaw direction to represent the deviation of the installation posture of the stereo camera. Figure 6The results shown show that even if the image is corrected using the technology disclosed in Patent Document 1, when the installation posture of the stereo camera is deviated, the delta parallax error cannot satisfy the allowable range of ±0.25 pixels represented by the allowable value 41. This is a unique problem for the imaging unit 100 that captures the outside world at a wide angle of view like a stereo camera.

[0052] On the other hand, in the calibration method according to the present embodiment, the parallax error of the design value of the front windshield (referred to as "reference parallax error" in the following description) and the delta parallax error caused by the change in the installation posture of the stereo camera are calculated separately. Thereafter, by correcting the parallax using the delta parallax error, high-precision calibration can be achieved. Therefore, the details of the calibration method according to the present embodiment will be described.

[0053] Figure 7 is a flowchart showing an example of a calibration method according to this embodiment. Figure 2 The example of the stereo camera 21 photographing the subject 24 is explained in Figure 3 Processing performed in each functional block shown.

[0054] Before the calibration of the present embodiment is performed, an external computing device is used to calculate information related to the parallax error and the vertical offset based on the installation posture of the stereo camera 21 of the corresponding vehicle type, the installation posture and shape design value of the front windshield 22, and the reference parallax error. Then, the external input unit 101 records the relevant information of the parallax error and the vertical offset (see the following). Fig.11 ) is stored in the storage unit 103 (S1). In addition, in step S1, the external input unit 101 also stores the reference vertical offset and the reference parallax error as design values ​​corresponding to the vehicle type in the storage unit 103.

[0055] Next, the stereo camera device 10 starts calibration for reducing the influence of the front windshield 22 on the parallax of the two images captured by the stereo camera 21 ( S2 ). Therefore, the stereo camera device 10 activates the stereo camera 21 of the imaging unit 100 .

[0056] Next, the camera unit 100 performs an image acquisition process (S3) to acquire two left and right images. At this time, the camera unit 100 acquires two images of the same subject 24 or scenery captured by the left camera 21L and the right camera 21. In the following description, the two images captured by the left camera 21L and the right camera 21R are also referred to as "left and right images."

[0057] Next, the correction unit 102 performs a correction process for correcting the two images acquired from the imaging unit 100 using the correction table read out from the storage unit 103 (the correction table obtained in the factory calibration) ( S4 ).

[0058] Next, the matching unit 104 performs a matching process for calculating the parallax and vertical offset of the two images corrected by the correction unit 102 (S5). Here, the matching unit 104 performs horizontal stereo matching on the corrected left and right images to calculate the parallax. In addition, the matching unit 104 calculates the vertical offset by matching the left and right images in the vertical direction. The matching unit 104 stores the calculated parallax and vertical offset in the storage unit 103.

[0059] Then, the processing of steps S3 to S5 is repeated (S5A). The matching unit 104 stores the vertical offsets of the plurality of frames calculated in the iterative processing in the storage unit 103. The iterative processing of step S5A is performed only as many times as the number of frames required for adding the vertical offset in step S6 described later.

[0060] Next, the noise removal unit 105 performs a process of adding the vertical offsets of the plurality of frames read from the storage unit 103 and removing noise (S6). Therefore, the noise removal unit 105 repeatedly processes steps S3 to S5, and reads the vertical offsets of the plurality of frames stored in the storage unit 103 from the storage unit 103. The noise removal unit 105 performs filtering before adding the vertical offsets, and removes the invalid vertical offsets that become the noise of the calculation result. Then, the noise removal unit 105 adds the plurality of vertical offsets from which the invalid vertical offsets have been removed, and calculates the average value of the vertical offsets.

[0061] Next, the parallax error estimation unit 106 performs a process of calculating a δ vertical offset using the vertical offset from which noise has been removed by the noise removal unit 105 and the reference vertical offset read from the storage unit 103 (S7). Next, the parallax error estimation unit 106 performs a process of calculating a δ parallax error from the δ vertical offset using a correlation table in which information related to the parallax error and the vertical offset is recorded (S8). Next, the parallax error estimation unit 106 stores the calculated δ parallax error in the storage unit 103 as correction information (S9).

[0062] Then, the distance calculation unit 107 removes the reference parallax error and the delta parallax error from the parallax read from the storage unit 103 to correct the parallax, and then calculates the distance information to the image pickup object using the corrected parallax (S10). Figure 7 The calibration process shown ends.

[0063] After the present process is completed, the distance information to the imaged object is output to the control unit 108. The control unit 108 recognizes the outside world based on the distance information and controls the automatic driving such as avoiding obstacles ahead of the vehicle.

[0064] Figure 82 is a diagram showing an example of horizontal stereo matching of left and right images. Here, the left camera 21L and the right camera 21R constituting the stereo camera 21 each capture an object 24. In the following description, the front windshield 22 is simply referred to as "glass".

[0065] Figure 8 The upper side of represents the image PL of the subject 24 projected by the left camera 21L. Figure 8 The lower side represents the image PR of the subject 24 projected by the right camera 21R. Image PL1 is the image of the subject 24 captured by the left camera 21L in the absence of glass. Image PR1 is the image of the subject 24 captured by the right camera 21R in the absence of glass. The parallax DPX is the true value of the parallax (horizontal offset) calculated by stereo matching in the horizontal direction between the image PL1 captured without glass and the image PR1. The parallax DPX is expressed as "Dx" in the formula described below. In addition, the glass in the state where the glass is installed on the vehicle at the designed position is referred to as "glass (design value)".

[0066] Image PL2 is an image of the subject 24 captured by the left camera 21L through the glass when there is glass (design value). Image PR2 is an image of the subject 24 captured by the right camera 21R through the glass when there is glass (design value). Image PL2 is offset to the lower right relative to image PL1, and image PR2 is offset to the lower right relative to image PR1.

[0067] Pixel offset LS1 indicates the parallax error of image PL2 photographed by left camera 21L through glass (design value) relative to image PL1. Pixel offset RS1 indicates the parallax error of image PR2 photographed by right camera 21R through glass (design value) relative to image PR1. Therefore, when horizontal stereo matching is performed on image PL2 and image PR2, parallax DPXE1 is generated. Parallax DPXE1 indicates the parallax error generated due to the influence of glass (design value).

[0068] Here, in the formula, "DEx" represents the parallax DPXE1 when there is glass (design value), and "εx" which is the parallax error caused by the influence of glass (design value) represents the reference parallax error, and the formula "DEx=Dx+εx" is obtained. That is, the parallax DEx when the glass (design value) is present is a value obtained by adding the parallax true value Dx and the reference parallax error εx caused by the influence of the glass (design value).

[0069] However, the case where there is glass (design value) here is an ideal state. In reality, there is a deviation in the installation posture of the camera relative to the glass (design value), or there is a deviation in the installation posture and shape of the glass. Here, the glass in the state where the glass is actually installed on the vehicle is referred to as "glass (actual value)". Figure 8The image PL3 captured by the left camera 21L through the glass (actual value) and the image PR3 captured by the right camera 21R through the glass (actual value) are shown. The image PL3 is offset to the lower right relative to the image PL2, and the image PR3 is offset to the lower right relative to the image PR2.

[0070] Pixel offset LS2 indicates the parallax error of image PL3 captured by left camera 21L through glass (actual value) relative to image PL2. Pixel offset RS2 indicates the parallax error of image PR3 captured by right camera 21R through glass (actual value) relative to image PR2. Therefore, when horizontal stereo matching is performed on image PL3 and image PR3, parallax DPXE2 is generated. Parallax DPXE2 indicates the parallax error generated due to the influence of glass (actual value).

[0071] In this case, "DE'x" is used in the formula to represent the parallax DPXE2 when there is glass (actual value), and "δεx" is used to represent the δ parallax error caused by the deviation of the glass and the camera relative to the design value, and the formula "DE'x=Dx+εx+δεx" is obtained. That is, the parallax DE'x when there is glass (actual value) is the value obtained by adding the above-mentioned parallax true value Dx, the reference parallax error εx caused by the influence of glass (design value), and the δ parallax error δεx. That is, the formula becomes (parallax) = (parallax true value) + (reference parallax error) + (δ parallax error). In addition, the δ parallax error δεx may be negative.

[0072] Fig. 9 This is a diagram showing an example of matching between left and right images in the vertical direction. exist Fig. 9 The left side of the left camera 21L shows the image PL of the subject 24. Fig. 9 The right side of the right camera 21R shows the image PR of the subject 24. The images PL1, PL2, PL3, pixel shifts LS1, LS2 shown in the image PL, the images PR1, PR2, PR3, pixel shifts RS1, RS2 shown in the image PR are as shown in FIG. Figure 8 As described.

[0073] The vertical offset DPY is the true value of the vertical offset calculated by stereo matching of the image PL1 and the image PR1 in the vertical direction without glass. The vertical offset DPY is expressed as "Dy" in the formula. The true value of the vertical offset when there is no glass is adjusted when the stereo camera 21 is shipped. Therefore, the true value of the vertical offset between the left camera 21L and the left camera 21L is zero, which is expressed by the formula Dy=0.

[0074] On the other hand, when there is glass (design value), the positions of the light rays incident through the glass (design value) are different in the left camera 21L and the right camera 21R. Therefore, the vertical pixel offset LS1 of the image captured by the left camera 21L and the vertical pixel offset RS1 of the image captured by the right camera 21R are different. Therefore, when there is glass (design value), if vertical stereo matching is performed on the image PL2 and the image PR2, a vertical offset DPYE1 is generated. That is, the vertical offset DPYE1 represents the vertical offset generated by the influence of the glass (design value).

[0075] Here, if "DEy" is used in the formula to represent the vertical offset DPYE1 when there is glass (design value), and "εy" which is the vertical offset caused by the influence of glass (design value) is used to represent the reference vertical offset, then since Dy=0, the equation "DEy=εy" is obtained. That is, the vertical offset DEy when the glass (design value) is present is a value obtained by adding the vertical offset true value Dy (=0) and the reference vertical offset εy caused by the influence of the glass (design value).

[0076] As reference Figure 8 As shown, the case with glass (design value) is an ideal state. Fig. 9 An image PL3 captured by the left camera 21L through the glass (actual value) and an image PR3 captured by the right camera 21R through the glass (actual value) are shown.

[0077] Pixel offset LS2 indicates the vertical offset of image PL3 captured by left camera 21L through glass (actual value) relative to image PL2. Pixel offset RS2 indicates the vertical offset of image PR3 captured by right camera 21R through glass (actual value) relative to image PR2. Therefore, when vertical stereo matching is performed between image PL3 and image PR3, vertical offset DPYE2 is generated. Vertical offset DPYE2 indicates the vertical offset value of the vertical offset generated due to the influence of glass (actual value).

[0078] Here, "DE'y" represents the vertical offset DPYE2 when there is glass (actual value), and "δεy" represents the δ vertical offset caused by the deviation of the glass and the camera relative to the design value, and the equation "DE'y=εy+δεy" is obtained. That is, the vertical offset DE'y when there is glass (actual value) is the sum of the vertical offset true value Dy (=0), the reference vertical offset εy caused by the influence of glass (design value), and the δ vertical offset. That is, the formula becomes (vertical offset) = (reference vertical offset) + (δ vertical offset). In addition, the δ vertical offset (δεy) may also be a negative value.

[0079] However, the value of DE'x in the horizontal direction changes depending on the distance from the camera to the subject 24. Therefore, it is necessary to perform measurement under predetermined conditions such as when the distance to the object is known.

[0080] On the other hand, the stereo camera device 10 of the present embodiment is characterized in that it uses an offset in the vertical direction that does not depend on the distance, rather than an offset in the horizontal direction. In the absence of glass, since the vertical offset is zero, even if the distance to the subject 24 changes, the vertical offset does not change and becomes the same offset amount. Therefore, the stereo camera device 10 can directly calculate the vertical offset and can calculate the parallax error using the vertical offset.

[0081] Next, the calculation process of the parallax error will be described. Fig.10 : is a schematic diagram showing an example of a calculation process of a parallax error according to the present embodiment. (1) Shooting of Scenery 1001 First, the stereo camera 21 captures the scenery 1001 through the front windshield 22. Figure 7 The processing of steps S3 to S5 in .

[0082] (2) Vertical Stereo Matching Next, the matching unit 104 performs vertical stereo matching of the two images captured by the left and right cameras to detect the vertical offset 10DPY. Figure 7 Here, the vertical offset in the image is represented by the vertical arrows in the vertical offset 10DPY. The direction of each arrow represents the direction of the vertical offset, and the length represents the offset amount of the vertical offset.

[0083] (3) Calculation of parallax error The parallax error estimation unit 106 estimates the parallax error 10DPX in the horizontal direction. Here, the parallax error in the image is indicated by the horizontal arrow in the parallax error 10DPX. The direction of each arrow indicates the direction of the parallax error, and the length indicates the offset of the parallax error.

[0084] An arrow of the parallax error 10DPX at the same position as an arrow of the vertical offset 10DPY has a correlation. The correlation of these arrows is stored in the correlation table of the storage unit 103 as related information c1, c2. In addition, as shown in the related information c1, c2, near the center of the image, the offset amount of the vertical offset and the offset amount of the parallax error hardly change. However, on the lower side where the curvature of the front windshield 22 becomes larger, the offset amount of the parallax error becomes larger relative to the offset amount of the vertical offset. Therefore, multiple correlations are stored in the storage unit 103 as related information. As described later, the correlation between the vertical offset and the parallax error can be used as the correlation between the δ vertical offset and the δ parallax error. In addition, the parallax error estimation unit 106 is capable of estimating the δ parallax error based on the δ vertical offset based on the related information containing multiple correlations read from the storage unit 103.

[0085] In addition, when the front windshield 22 is formed of a part of a spherical surface, it is expected that the correlation between the vertical offset and the parallax error is constant in the entire front windshield 22. In this case, one correlation is stored as the related information in the storage unit 103. Then, the parallax error estimation unit 106 can estimate the δ parallax error from the δ vertical offset based on the related information including the one correlation read from the storage unit 103.

[0086] As described above, if the parallax error estimation unit 106 can calculate the δ vertical offset, it can estimate the δ parallax error at each viewing angle indicated by the parallax error 10DPX. Figure 7 The processing of steps S6 to S8 in .

[0087] Fig.11 This is a diagram of the structure of the correlation table. The correlation table records the information related to the parallax error and the vertical offset.

[0088] The correlation table is represented by, for example, a table of horizontal viewing angles [degrees] and vertical viewing angles [degrees] of the stereo camera 21. In the cell where each viewing angle intersects, two values, "e(m)" for the parallax error and "v(n)" for the vertical offset, are stored as correlation information. Here, "m" of the parallax error e(m) corresponds to the horizontal viewing angle and takes a value from "-60" to "+60". In addition, "n" of the vertical offset v(n) corresponds to the vertical viewing angle and takes a value from "-30" to "+30".

[0089] The storage unit (storage unit 103) stores a plurality of related information corresponding to the shooting angle of view (horizontal angle of view and vertical angle of view) of the camera (stereo camera 21) in the related table. Fig.11The correlation table shown stores information related to when the horizontal and vertical viewing angles are changed by 1 degree each within the range of the horizontal viewing angle from -60 degrees to +60 degrees and the vertical viewing angle from -30 degrees to +30 degrees. Therefore, the parallax error estimation unit 106 can easily read the information related to the parallax error and the vertical offset at a certain horizontal and vertical viewing angles from the correlation table. The parallax error estimation unit (parallax error estimation unit 106) can estimate the parallax error using a plurality of related information read from the storage unit (storage unit 103) in comparison with the shooting viewing angle.

[0090] In addition, the correlation table may store relevant information when the horizontal and vertical viewing angles are changed by 0.5 degrees each, or may store relevant information when the horizontal and vertical viewing angles are changed by 10 degrees each.

[0091] In addition, the storage unit (storage unit 103) may store one piece of related information obtained for the refractor (front windshield 22) when the shape of the refractor (front windshield 22) is uniform. If the related information of the horizontal field of view angle and the vertical field of view angle are the same everywhere, only one piece of related information of the parallax error and one piece of related information of the vertical offset may be stored in the correlation table. The parallax error estimating unit (parallax error estimating unit 106) can estimate the parallax error using one piece of related information read from the storage unit (storage unit 103).

[0092] Next, the reason why the parallax error estimation unit 106 can estimate the parallax error in the horizontal direction based on the pixel shift in the vertical direction will be described. Fig.12 1 is a diagram for explaining the principle of how the parallax error estimation unit 106 estimates the parallax error based on the vertical offset.

[0093] Fig.12 The instructions shown on the upper side Fig.12 P0 is a stereogram showing the situation of light incident on a glass without deviation and then emitted. Here, it shows the situation where a certain incident light 11in is incident on the front windshield 22 and then emitted as an outgoing light 11out. The X-axis in the figure represents the horizontal viewing angle direction, the Y-axis represents the vertical viewing angle direction, and the Z-axis represents the direction of travel of the vehicle. When the incident light 11in and the outgoing light 11out are not distinguished, they are referred to as "light".

[0094] When the incident light 11in is projected onto the XY plane, it is represented as XY plane projection light 1111. Also, when the outgoing light 11out is projected onto the XY plane, it is represented as XY plane projection light 1113. Here, the cut surface 1112 is obtained by projecting the portion where the light irradiates the glass 22 onto the XY plane.

[0095] Next, a description will be given of a case where light passes through glass with no deviation in mounting posture. exist Fig.12 The instructions shown on the lower left Fig.12 P1 shows the projection to illustrate Fig.12 Example of light on the XY plane shown in P0 Here, the XY plane projection light 1113 is decomposed into light 1115 and light 1114 on the X axis and Y axis, which respectively represent the horizontal component and vertical component of the XY plane projection light 1113.

[0096] The following describes the case where light passes through glass that is installed in a deviated manner. exist Fig.12 The instructions shown in the lower right Fig.12 In P2, the light passing through the glass with a deviation in installation posture is projected onto the Fig.12 Example on the XY plane shown in P0. Fig.12 In P2, it also means that Fig.12 P1 shows the displacement of light rays on the same section 1112.

[0097] For example, when there is a roll rotation deviation B1130 in the glass, the cut surface 1122 (solid line) without the deviation is shown at the position changed to the cut surface 1122 (dashed line). In addition, the outgoing light 1123 (dashed line) is changed from the XY plane projection light (solid line).

[0098] In addition, light 1124 (dashed line) and light 1125 (dashed line) respectively represent the vertical component and horizontal component of the changed outgoing light 1123 (dashed line). The vertical difference between light 1124 (dashed line) and light 1114 (solid line) is the delta vertical offset δεy caused by the deviation of the glass. In addition, the horizontal difference between light 1125 (dashed line) and light 1115 (solid line) is the delta parallax error δεx caused by the deviation of the glass.

[0099] The actual deviation of the glass installation posture is much smaller than the design value. Therefore, the ratio of the delta parallax error ε'x to the delta vertical offset ε'y when the glass installation posture is deviated does not change much from the ratio of the reference parallax error εx to the reference vertical offset εy calculated from the glass design value. That is, there exists a relationship of |εx| / |εy|≈|ε'x| / |ε'y|.

[0100] For example, if the correlation between the reference parallax error εx and the reference vertical offset εy is obtained in advance based on the design value of the glass, it can be expressed as |εx| / |εy|=k. On the other hand, when the δ parallax error when the installation posture of the glass is deviated is set to "ε'x", and the δ vertical offset is set to "ε'y", and the correlation between the δ parallax error ε'x and the δ vertical offset ε'y is obtained, it can be expressed as |ε'x|≈k*|ε'y|. That is, since it can be expressed as |ε'x|-|εx|≈k*(|ε'y|-|εy|), the relationship between the δ parallax error δεx and the δ vertical offset δεy can be expressed as δεx≈k*δεy.

[0101] Therefore, the parallax error estimation unit 106 can estimate the actual parallax error by the calculation formula of (actual parallax error) = (reference parallax error (glass design value)) + (δ parallax error (with deviation)). Here, the reference parallax error and the correlation coefficient k are values ​​obtained from the design value of the glass. In addition, as described above, the δ parallax error δεx can be calculated by (δ vertical offset δεy*k).

[0102] In this way, in the stereo camera device 10 of the present embodiment, even if there is a deviation in the installation posture of the glass, by using the correlation between the δ vertical offset and the δ parallax error, that is, the correlation between the vertical offset and the parallax error, the δ parallax error (with deviation) is calculated, and the parallax is corrected using the actual parallax error, it is possible to achieve high-precision calibration. In addition, although the correlation is described as linear here, as the amount of change increases, the correlation becomes nonlinear. In this case, of course, there is also an effect.

[0103] Next, the correlation between the delta vertical deviation and the delta parallax error when the deviation of the installation posture of the glass is taken into account is confirmed. The following describes nine examples of deviations taking into account the deviation conditions. Fig.13 It is a figure which shows the example of 9 types of deviations. Fig.13 The deviations (1) to (3) shown are based on the following calculation conditions.

[0104] <Calculation conditions (deviation)> X, Y, Z displacement: +2mm Pitch, yaw, roll rotation: +3 degrees Glass curvature radius (horizontal, vertical): -1.5m Glass thickness: +1mm

[0105] Fig.13 The deviation (1) shows an example of the displacement deviation of the stereo camera relative to the front windshield 22. Fig.13Starting from the left side of , examples of displacement in the X-axis direction (X displacement), displacement in the Y-axis direction (Y displacement), and displacement in the Z-axis direction (Z displacement) are shown in sequence. Fig.13 The deviation (2) represents an example of the rotation deviation of the stereo camera relative to the front windshield 22. Fig.13 Starting from the left side of , examples of rotation in the pitch direction (pitch rotation), rotation in the yaw direction (yaw rotation), and rotation in the roll direction (roll rotation) are shown in sequence. Fig.13 The deviation (3) shows an example of the characteristic deviation of the front windshield glass 22. Fig.13 Starting from the left side of , examples of the horizontal curvature of the front windshield 22 , the vertical curvature of the front windshield 22 , and the thickness of the front windshield 22 are sequentially shown.

[0106] Fig.14 3 is a diagram showing the correlation between the δ vertical offset and the δ parallax error. Fig.14 The horizontal axis represents the δ vertical offset, and the vertical axis represents the δ parallax error. Then, Fig.14 Under the following calculation conditions shown in the legend of FIG. , the correlation between the δ vertical offset and the δ parallax error is obtained.

[0107] <Calculation conditions> · Left and right translation (X axis): +2mm ·Up and down translation (Y axis): +2mm Front and rear translation (Z axis): +2mm Pitch angle (X-axis rotation): +3 degrees Roll angle (Y-axis rotation): +3 degrees Roll angle (Z-axis rotation): +3 degrees Horizontal curvature radius: -1500mm Vertical curvature radius: -1500mm Thickness: +0.2mm

[0108] Here, the stereo camera device 10 calculates the δ vertical offset and the δ parallax error when detecting the object 24 50 m ahead, thereby obtaining the correlation between the δ vertical offset and the δ parallax error. Fig.14 The curve RT20 on the upper side represents the correlation between the δ vertical offset and the δ parallax error when the horizontal field angle is 20 degrees. Fig.14 The curve RT40 in the middle side represents the correlation between the δ vertical offset and the δ parallax error when the horizontal field angle is 40 degrees. Fig.14 The curve RT60 on the lower side represents the correlation between the δ vertical offset and the δ parallax error when the horizontal field angle is 60 degrees.

[0109] As shown in the graphs RT20, RT40, and RT60, it can be seen that the delta vertical offset and the delta parallax error have a high correlation. In addition, the delta parallax error shifts with respect to the deviation of the front windshield 22, but the delta vertical offset also changes with the shift of the delta parallax error. Therefore, it can be seen that if the delta vertical offset is calculated, the delta parallax error can be obtained.

[0110] In addition, it can be seen that the correlation coefficient k representing the correlation between the δ vertical offset and the δ parallax error varies depending on the width of the horizontal field of view. For example, if the horizontal field of view is 60 degrees, the correlation coefficient k is approximately "+1". And, as the horizontal field of view changes to 40 degrees and 20 degrees, the correlation coefficient k takes a value smaller than "+1". Therefore, in order to perform high-precision calibration, it is preferred to use a different correlation coefficient k for each horizontal field of view angle of the stereo camera 21.

[0111] Fig.15 Consider all Fig.14 The correction effect under the composite conditions of the calculation conditions (deviations) shown. Fig.15 , the simulation results of the parallax errors obtained for horizontal viewing angles of 20 degrees, 40 degrees, and 60 degrees are shown. Here, the parallax errors before correction are represented by black circles, the parallax errors corrected using the correlation coefficient k for each horizontal viewing angle are represented by black triangles, and the parallax errors corrected using a correlation coefficient of 60 degrees for a horizontal viewing angle are represented by white quadrilaterals.

[0112] When the horizontal viewing angle is 40 degrees and 60 degrees, the parallax error corrected by a correlation coefficient for a horizontal viewing angle of 60 degrees is greater than 0 pixels. On the other hand, the parallax error corrected by the correlation coefficient k for each horizontal viewing angle of the present embodiment is approximately 0 pixels for a horizontal viewing angle of 40 degrees. However, when the horizontal viewing angle is 60 degrees, the corrected parallax error is approximately 0.5 pixels.

[0113] It can be seen that by performing the correction of this embodiment, the parallax error can be greatly reduced. In addition, when the correlation coefficient k for each horizontal field of view angle is used for correction, the residual parallax can be further reduced, but even if the correlation coefficient k is 60 degrees for the horizontal field of view angle, a sufficient effect can be obtained compared with the parallax error before correction exceeding 1 pixel.

[0114] In the stereo camera device 10 of one embodiment described above, the reference parallax error calculated from the design value of the windshield 22 and the delta parallax error caused by the deviation of the windshield 22 are calculated separately. Therefore, even without a calibration chart, the stereo camera device 10 can accurately correct (calibrate) the parallax of the stereo camera 21 with a wide field of view. For example, even if there is a manufacturing deviation in the curvature radius of the windshield 22 in the horizontal and vertical directions or a deviation in the relative value relationship between the windshield 22 and the stereo camera 21, the stereo camera device 10 can accurately correct the parallax of the stereo camera 21.

[0115] For example, when the windshield 22 is replaced at a dealer's repair shop, even if there are deviations in characteristics such as the installation posture of the stereo camera 21 and the installation posture and shape of the windshield 22, high ranging accuracy can be obtained by performing calibration according to this embodiment.

[0116] In addition, the method of this embodiment does not require a large chart for calibration. Therefore, the image of the scenery on the road can be calibrated, and the influence of the front windshield 22 can be corrected even when the vehicle 20 is running.

[0117] In addition, the user can use calibration to correct the influence of the front windshield glass 22 by the stereo camera 21 while the vehicle 20 is stopped or running. For example, in the calibration mode, the stereo camera 21 performs calibration in real time while the vehicle 20 is stopped or running. Here, the calibration while the vehicle 20 is running is performed by Figure 7 The flowchart shown in the figure is used to accumulate the vertical offsets of the plurality of frames of images acquired during driving. For example, the vertical offsets are accumulated based on the changes in the scenery reflected in the images taken when the vehicle 20 is driving or stopped.

[0118] For example, after the vehicle 20 is parked in a certain direction and photographs a scene, the vehicle 20 may be parked in another direction and photographs the scene in that direction, thereby accumulating the vertical offset based on the scene change. For example, the stereo camera device 10 acquires the left and right images of the same subject 100 times, performs stereo matching 100 times, accumulates the vertical offset 100 times, and obtains the average value. Then, the parallax error estimation unit 106 estimates the parallax error based on the vertical offset using the correlation table, and the distance calculation unit 107 performs calibration to remove the parallax error from the parallax.

[0119] in addition, Fig.10 The scenery 1001 shown is not limited to scenery of people or animals, etc. For example, it can be anywhere as long as it is scenery such as a road surface or a graph where parallax can be obtained.

[0120] in addition, Figure 7The iterative process (S5A) of steps S3 to S5 shown is not limited to the accumulation of vertical offsets. For example, a plurality of delta parallax errors may be obtained to correct the parallax multiple times. In addition, other parameters may be accumulated to improve the accuracy of parallax correction.

[0121] The invention completed by the present inventors has been specifically described above based on the embodiments, but the present invention is not limited to the above embodiments, and various modifications are of course possible. For example, the present invention can be applied to calibration in a factory, and parallax can be corrected without using a chart. In addition, even when calibration is performed using a chart in a factory, it is sufficient to use a chart with a narrow chart width compared to conventional calibration, and there is no need to modify the factory or prepare a large chart installation space.

[0122] Furthermore, the stereo camera device 10 can correct the influence on the glass even when the vehicle 20 is running. Furthermore, since calibration can be performed even when the vehicle 20 is running, even when the installation posture of the stereo camera 21 is shifted due to vibration of the vehicle 20, parallax error can be appropriately corrected.

[0123] In addition, if the stereo camera 21 is installed inside the vehicle 20, it can also be installed on the inside side of the rear windshield. In this case, the stereo camera device 10 can also correct the parallax based on the multiple images of the outside world behind the vehicle 20 captured by the stereo camera 21, and calculate the distance to the object in the outside world behind the vehicle 20.

[0124] In the above-described embodiment, the stereo camera 21 is described as being provided on the vehicle interior side of the windshield 22 as an example of a refractor. However, in addition to glass, a transparent organic compound such as transparent plastic may be used as the refractor.

[0125] Thus, the present invention is not limited to the above-described embodiment, and various other application examples and modified examples can of course be adopted without departing from the gist of the present invention described in the claims.

[0126] The positions, sizes, shapes, and ranges of the components shown in the drawings may not represent actual positions, sizes, shapes, and ranges in order to facilitate the understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, and ranges disclosed in the drawings. In addition, the following embodiments are embodiments that describe the configuration of the system in detail and specifically in order to facilitate the understanding of the present invention, and are not necessarily limited to embodiments that have all the configurations described. In addition, for a part of the configuration of the present embodiment, other configurations may be added, deleted, or replaced. In addition, the control lines and information lines are those that are considered necessary for the purpose of explanation, and not all control lines and information lines are necessarily shown in the product. In reality, it can be considered that almost all components are connected to each other. Explanation of symbols

[0127] 10…stereo camera device, 20…vehicle, 21…stereo camera, 22…front windshield, 100…imaging unit, 101…external input unit, 102…correction unit, 103…storage unit, 104…matching unit, 105…noise removal unit, 106…parallax error estimation unit, 107…distance calculation unit, 108…control unit.

Claims

1. A stereo camera device, characterized in that: have: a storage unit storing information related to parallax errors in the horizontal direction relative to vertical offsets of a plurality of images respectively captured by a plurality of cameras through a refractor that refracts light, the information being obtained based on refractor characteristics including a posture of the refractor mounted on a vehicle and a shape of the refractor, and the postures of a plurality of cameras arranged on the vehicle; a matching unit, which matches the plurality of images obtained by photographing the same object by the plurality of cameras, and obtains a vertical offset of the plurality of images; a parallax error estimating unit configured to estimate a parallax error of the matched plurality of images based on the vertical offset and the related information; and A distance calculation unit is configured to correct the parallax of the matched images using the parallax error.

2. The stereo camera device according to claim 1, characterized in that The storage unit stores a plurality of related information corresponding to the shooting angle of view of the camera, The parallax error estimation unit estimates the parallax error using the plurality of pieces of related information read from the storage unit in comparison with the shooting angle of view of the camera.

3. The stereo camera device according to claim 1, characterized in that The storage unit stores the related information obtained for the refractor when the refractor has a uniform shape. The parallax error estimating section estimates the parallax error using the one piece of related information read from the storage section.

4. The stereo camera device according to claim 2, characterized in that The parallax error estimation unit accumulates the vertical offsets obtained by the matching unit at the same position of the plurality of images captured at different times a plurality of times to calculate an average value of the vertical offsets, and estimates the parallax error based on the average value of the vertical offsets and the related information.

5. The stereo camera device according to claim 3, characterized in that: The storage unit stores a reference parallax error and a reference vertical offset calculated in advance based on the design value of the refractor. The parallax error estimating unit calculates a δ vertical offset, which is a change amount of the vertical offset caused by the refractor actually mounted on the vehicle, based on the reference vertical offset and the vertical offset read from the storage unit, and calculates a δ parallax error, which is a change amount of the parallax error caused by the refractor actually mounted on the vehicle, based on the δ vertical offset and the related information read from the storage unit. The distance calculation section corrects the parallax using the reference parallax error and the delta parallax error.

6. The stereo camera device according to claim 5, characterized in that The reference parallax error and the reference vertical offset are stored in the storage unit when the plurality of cameras are shipped. The amount of change of the plurality of cameras after leaving the factory relative to the reference parallax error is the δ parallax error, and the amount of change relative to the reference vertical offset is the δ vertical offset.

7. The stereo camera device according to claim 5, characterized in that The reference parallax error and the reference vertical offset are stored in the storage unit before the refractor is mounted on the vehicle, The amount of change of the refractor relative to the reference parallax error after it is actually mounted on the vehicle is the δ parallax error, and the amount of change relative to the reference vertical offset is the δ vertical offset.

8. The stereo camera device according to claim 2, characterized in that The distance calculation unit calculates the distance from the vehicle to an external object using the parallax corrected by the parallax error.

9. A calibration method, characterized in that: include: A process of acquiring a plurality of images respectively captured by a plurality of cameras arranged on a vehicle through a refractor installed on the vehicle and refracting light; Matching the plurality of images obtained by taking photos of the same object by the plurality of cameras to obtain a vertical offset of the plurality of images; A process of estimating the parallax errors of the matched images based on the related information about the parallax errors of the horizontal directions of the plurality of images with respect to the vertical offset, and the vertical offset, the related information being obtained based on the refractor characteristics including the posture of the refractor and the shape of the refractor, and the postures of the plurality of cameras; as well as The process of using the parallax error to correct the parallax of the matched images.

Images