Processing device and vehicle camera device

Abstract

The present disclosure provides a processing device that detects a setting deviation of a camera with respect to a marker caused by a setting deviation of a vehicle or a mounting deviation of the camera when a monocular camera or a stereo camera is mounted to the vehicle to take a shot of the marker for calibration, corrects an image deviation caused by glass in a manner that is not affected by the factor, and thereby accurately measures a distance. The processing device of the present disclosure acquires a setting deviation of a vehicle-mounted camera based on images taken of a plurality of markers (101, 102, 103) that differ in distance from the vehicle-mounted camera (stereo camera (104)) and data of a window (front windshield (112)) of a modeled vehicle (111), and uses the setting deviation to change correction data for correcting images of the vehicle-mounted camera. The setting deviation is a positional deviation with respect to a position at which a card of the marker (101, 102, 103) should be disposed.

Description

TECHNICAL FIELD

[0001] The present application relates to a processing device that calculates data for correcting one or more camera images to be distortion-free images, and a vehicle-mounted camera device. BACKGROUND

[0002] The prior art of Patent Literature 1, after a stereo camera having a first camera and a second camera is provided to a vehicle, calculates a correction parameter of the first camera for correcting an image of the first camera taken through a windshield (a transparent body). Then, a correction parameter of the second camera for correcting a parallax deviation of a subject of the first camera and the second camera is calculated, thereby removing a deviation of an image caused by the windshield.

[0003] The prior art of Patent Literature 2, using a stereo camera to take a subject with and without a windshield (a transparent body), calculates coordinates of the subject on an image, and calculates a correction parameter for constructing an image from the difference between the coordinates. A deviation of an image caused by the windshield is removed based on the correction parameter.

[0004] Prior Art Documents

[0005] Patent Literature

[0006] Patent Literature 1: Japanese Patent Laid-Open No. 2019-132855

[0007] Patent Literature 2: Japanese Patent Laid-Open No. 2015-169583 SUMMARY

[0008] Problems to be Solved by the Invention

[0009] In a case where a vehicle in which a camera is installed has a setting deviation with respect to a card or a case where the camera has a mounting deviation with respect to the vehicle, a deviation of an image caused by a windshield is included in a correction parameter. Thus, when an object is taken at a distance different from a distance at which a subject is taken for correction, the object on images of the first camera and the second camera has a vertical direction deviation, and images of the object do not match normally in a pattern matching process, so that an accurate parallax cannot be detected. In addition, the object on the images of the first camera and the second camera has a horizontal direction deviation, so that a parallax error occurs. Thus, a distance of the object cannot be accurately measured due to the parallax error.

[0010] The present application provides a processing device that detects a setting deviation of a vehicle or a mounting deviation of a camera caused by a setting deviation of the camera with respect to a marker when a monocular camera or a stereo camera is mounted to the vehicle to capture the marker for calibration, corrects an image deviation caused by a glass in a manner that is not affected by the factor, and thereby accurately measures a distance.

[0011] Technical means for solving the problem

[0012] The processing device of the present application acquires a setting deviation of a vehicle-mounted camera from images captured of a plurality of markers that differ in distance from the vehicle-mounted camera and data modeling a window of the vehicle, and uses the setting deviation to change correction data for correcting images captured by the vehicle-mounted camera, wherein the setting deviation is a positional deviation with respect to a position at which the vehicle-mounted camera should be disposed with respect to a card on which the markers exist.

[0013] Effects of the invention

[0014] According to the present application, a magnification and a shift deviation of a model of a glass of a vehicle that are independent of distance, and a positional and attitude deviation between a marker and a camera that depend on distance can be detected from a separation and a positional deviation of the markers on the images that differ in distance. Then, assuming that there is a vehicle setting deviation and a camera mounting deviation in the position of the camera, the data for correcting the images are changed in a manner that corrects a deviation of the images caused by a front glass, and thereby the deviation of the images caused by the vehicle setting deviation and the camera mounting deviation can be removed by only one capturing, and thus a distance can be accurately measured. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 FIGS. 1 to 5 are diagrams for illustrating an embodiment of a processing device and a vehicle-mounted camera device of the present application.

[0016] Figure 2 FIG. 6 is a diagram for illustrating an example of a configuration region of a marker.

[0017] Figure 3 FIGS. 7 to 11 are diagrams for illustrating an embodiment of a processing device and a vehicle-mounted camera device of the present application.

[0018] Figure 4 FIG. 12 is a diagram for illustrating an action of a processing device and a vehicle-mounted camera device.

[0019] Figure 5 FIG. 13 is a diagram for illustrating a position of a marker on a corrected image.

[0020] Figure 6 FIG. 14 is a diagram for illustrating an example of a configuration of a marker, a stereo camera, and a vehicle viewed from above.

[0021] Figure 7FIG. 1 is a diagram for illustrating an example of a configuration of a marker, a stereo camera, and a windshield.

[0022] Figure 8 FIG. 2 is a diagram for illustrating an operation of the stereo camera.

[0023] Figure 9 FIG. 3 is a diagram for illustrating an example of calculating a disparity on a rectified image of the two imaging system sections.

[0024] Figure 10 FIG. 4 is a diagram for illustrating an example of a configuration of a processing device and a vehicle-mounted camera device. Figure 3 FIG. 5 is a diagram for illustrating a modification example of the processing device and the vehicle-mounted camera device shown in FIG. 4.

[0025] Figure 11 FIG. 6 is a diagram for illustrating an example of a configuration of a processing device and a vehicle-mounted camera device. Figure 3 FIG. 7 is a diagram for illustrating a modification example of the processing device and the vehicle-mounted camera device shown in FIG. 6.

[0026] Figure 12 FIG. 8 is a diagram for illustrating an operation of the camera.

[0027] Figure 13 FIG. 9 is a diagram for illustrating an example of an image of an object obtained by a shooting distance measurement. DETAILED DESCRIPTION

[0028] (Embodiment 1)

[0029] Figure 1 FIG. 10 is a diagram for illustrating a configuration of an image correction device in a mode of an embodiment of the present application. Figure 2 FIG. 11 is a diagram for illustrating an example of a configuration region of a marker. Figure 3 FIG. 12 is a diagram for illustrating a configuration of a stereo camera of an image correction device in a mode of an embodiment of the present application.

[0030] The processing device of the present embodiment is, for example, an image correction device. The image correction device has image correction data for correcting distortion and deviation of an image captured in a state where the windshield 112 is not penetrated, and the vehicle 111 is disposed in a position and an attitude (direction 123) deviated from a prescribed position 121 and a prescribed attitude (direction 122). In a state where the stereo camera 104 is installed in a position and an attitude (direction 124) deviated from the prescribed position and the prescribed attitude (direction 122) with respect to the vehicle 111, the effects of the disposition deviation of the vehicle 111 and the installation deviation of the stereo camera 104 are removed, and the image correction data is changed in a manner to correct the image deviation caused by the windshield 112.

[0031] Figure 1 and Figure 3The configuration of one embodiment of the image correction apparatus of the present invention is shown below. Here, the corrected image storage unit 316, the parallax image storage unit 317, and the distance detection unit 330, shown in gray, are not included in the image correction apparatus and operate in the case of detecting the distance of a three-dimensional object. Their operation will be described later.

[0032] The image correction device includes multiple markers, a vehicle setting deviation detection unit 105, and a stereo camera 104.

[0033] like Figure 2 As shown, multiple markings are circular in shape. Multiple markings 201 are positioned on the front of the vehicle, while multiple markings 202, 203, 204, and 205 are positioned relative to markings 201 at the upper left front, upper right front, lower left front, and lower right front, respectively. Here, markings 201, 202, 203, 204, and 205 are depicted on cards 206, 207, 208, 209, and 210, respectively. Furthermore, marking 101 represents marking 201, marking 102 represents marking 202 or 204, and marking 103 represents marking 203 or 205. The arrangement of markings 201 to 205 can also be varied depending on the distance from the camera they are positioned at. Figure 2 Configurations other than those shown.

[0034] The vehicle setting deviation detection unit 105, which consists of LiDAR (Light Detection and Ranging) and a computing unit, has surface shape data (design value) of the vehicle when it is set according to the design and position of the vehicle (design position). It measures the surface position (measured value) of the vehicle 111 and compares the design value of the surface shape data of the vehicle 111 with the measured value of the surface position of the vehicle 111 to detect the position deviation and attitude deviation of the vehicle 111 from the design position.

[0035] A stereo camera 104 is mounted on the vehicle 111 and captures images of markings 201 to 205 through the windshield 112 of the vehicle 111. In addition, the stereo camera 104 includes a camera system unit 300a, a camera system unit 300b, and a computing unit 310, and is connected to the vehicle position deviation detection unit 105.

[0036] The camera system 300a includes an optical element section 301a and an image sensor section 302a. The optical element section 301a refracts light to form an image on the image sensor section 302a. The image sensor section 302a receives light from the image of the light refracted by the optical element section 301a and generates an image corresponding to the intensity of that light.

[0037] The imaging system section 300b, such as a camera, includes an optical element section 301b and an imaging element section 302b. The focal length of the imaging system section 300a is designed to be the same as that of the imaging system section 300b. The optical axes of the imaging system section 300a and the imaging system section 300b are substantially the same. The optical element section 301b, such as a lens, refracts light to form an image on the imaging element section 302b. The imaging element section 302b, such as an imaging element, receives light from the optical element section 301b to generate an image corresponding to the intensity of the light.

[0038] The image captured by the imaging system section 300a is an image that is used as a reference when a parallax image is created, and is therefore referred to as a reference image. The image captured by the imaging system section 300b is an image that is used to search for an area that matches an area extracted from the reference image when a parallax image is created, and is therefore referred to as a reference image.

[0039] The CPU (central processing unit) and a memory, or the like, of the arithmetic section 310 includes a captured image storage section 311, an image correction information storage section 312, a correction information storage section 313, a synchronization signal generation section 314, a reference image import section 315b, an image correction section 320, a correction image storage section 316, a parallax image storage section 317, a deviation correction information storage section 318, and a distance detection section 330.

[0040] The captured image storage section 311, such as a memory, stores images output from the imaging system section 300a and the imaging system section 300b.

[0041] The image correction information storage section 312, such as a memory, stores two-dimensional coordinates (image correction data) on a correction image (a post-capture image with distortion) corresponding to each pixel on a captured image (an image without distortion) for the image of the imaging system section 300a and the image of the imaging system section 300b. The image correction information storage section 312 stores image correction data of the imaging system section 300a and the imaging system section 300b that is calculated in advance in a state in which the windshield 112 is not present, and stores image correction data of an image that is corrected to be an image without distortion through the windshield 112, which is created by the image correction data modification section. Here, the image without distortion is an image that is perspective projected using a pinhole model. The image correction data is used for a process of correcting lens distortion or optical axis misalignment of a post-capture image, and indicates distortion of an image at the time of capture.

[0042] The correction information storage section 313 stores the focal length, the pixel pitch, the optical axis position on the captured image, the optical axis position on the corrected image, the three-dimensional position of the markers 201 to 205 with respect to the stereo camera 104 when the markers 201 to 205 are disposed in a bias-free manner on the vehicle 111, the design value of the data of the model of the image distortion caused by the windshield 112 in the imaging system section 300a and the imaging system section 300b (glass distortion data), and the magnification and the shift bias in the horizontal direction and the vertical direction of the model of the image distortion caused by the windshield 112, the positional bias and the attitude bias of the vehicle, and the positional bias and the attitude bias of the camera installation at the time of the respective marker images on the corrected image.

[0043] Here, the magnification and the shift bias in the horizontal direction and the vertical direction of the image distortion caused by the windshield 112, the positional bias and the attitude bias of the vehicle, and the positional bias and the attitude bias of the camera installation are calculated in the image bias factor detection section 322. The initial value of the magnification in the horizontal direction and the vertical direction of the image distortion caused by the windshield 112 is 1, and the initial value of the other is zero. The glass distortion data is a two-dimensional coordinate of the position on the image after passing through the windshield 112 corresponding to each pixel on the image before passing through the windshield 112.

[0044] The synchronization signal generation section 314 generates and transmits a synchronization signal.

[0045] The reference image import section 315a transmits the synchronization signal and the information of the exposure time to the imaging element section 302a in accordance with the synchronization signal of the synchronization signal generation section 314, and then acquires the image generated by the imaging element section 302a, and stores the image to the captured image storage section 311.

[0046] The reference image import section 315a transmits the synchronization signal and the information of the exposure time to the imaging element section 302a in accordance with the synchronization signal of the synchronization signal generation section 314, and then acquires the image generated by the imaging element section 302a, and stores the image to the captured image storage section 311.

[0047] The image correction section 320 includes the marker position detection section 321, the image bias factor detection section 322, and the image correction data change section 323.

[0048] The mark position detection section 321 calculates image correction data for correcting the captured image through the windshield 112, based on the design values of the image correction data and the image distortion data caused by the windshield 112, which are stored in advance in the image correction information storage section 312, and corrects the captured images of the imaging system section 300a and the imaging system section 300b using the image correction data. The mark position detection section 321 then performs pattern matching processing to detect the mark positions (detection positions) on the corrected images in a manner consistent with the marks on the mark images and their corrected images stored in the correction information storage section 313.

[0049] The image deviation factor detection section 322 calculates the mark positions (design positions) on the corrected images in a manner consistent with the design values when the image deviation caused by the windshield 112 and the design values are consistent when the marks are disposed in a manner free from deviation on the vehicle 111, based on the focal lengths, the pixel pitches, the optical axis positions on the captured images, the optical axis positions on the corrected images, the three-dimensional positions of the marks 201 to 205 with respect to the stereo camera 104 when disposed in a manner free from deviation on the vehicle 111, and the design values of the glass distortion data of the imaging system section 300a and the imaging system section 300b, which are stored in the correction information storage section 313.

[0050] Further, the image deviation factor detection section 322 calculates the magnification and the shift deviation of the image distortion caused by the windshield 112 in the horizontal direction and the vertical direction, and the camera setting deviation, based on the detection positions and the design positions of the marks 201 to 205 on the corrected images, the fact that the image deviation caused by the windshield 112 does not depend on the distance between the stereo camera 104 and the marks 201 to 205, and the fact that the camera setting deviation depends on the distance between the stereo camera 104 and the marks 201 to 205. Here, the camera setting deviation is the positional deviation and the attitude deviation (deviation from the design position) between the marks 201 to 205 and the stereo camera 104. Then, the camera installation deviation is calculated based on the vehicle setting deviation detected by the vehicle setting deviation detection section 105 and the camera setting deviation calculated by the image deviation factor detection section 322. Here, the camera installation deviation is the positional deviation and the attitude deviation (deviation from the design position) of the stereo camera 104 with respect to the vehicle 111 when the stereo camera 104 is installed on the vehicle 111.

[0051] The image correction data change section 323 reads in the focal lengths, the pixel pitches, the optical axis positions on the captured images, the optical axis positions on the corrected images, and the three-dimensional positions of the marks 201 to 205 with respect to the stereo camera 104 when disposed in a manner free from deviation on the vehicle 111, which are stored in the correction information storage section 313.

[0052] Further, the image correction data changing section 323 calculates camera setting deviation data indicating an image deviation caused by the camera setting deviation, camera mounting direction deviation data indicating an image deviation caused by the camera mounting direction deviation, magnification of the model of the windshield 112 in the horizontal direction and the vertical direction, and glass deviation data in the presence of the shift deviation, using the camera setting deviation detected by the image deviation factor detection section 322, the camera mounting deviation, the magnification of the model of the windshield 112 in the horizontal direction and the vertical direction, and the shift deviation.

[0053] Further, the image correction data changing section 323 sequentially performs, for each pixel, a transformation based on the camera mounting direction deviation data, a transformation based on the glass deviation data, a transformation based on the glass deformation data, a transformation based on the camera setting deviation data, and a transformation based on the image correction data in the absence of the windshield 112 (indicating a distortion of an image at the time of photographing) using two-dimensional linear interpolation. Thus, the image correction data changing section 323 removes the pixel deviation caused by the camera setting deviation, and calculates the image correction data across the windshield 112 toward the traveling direction of the vehicle 111 for the reference image and the reference image, and stores these image correction data to the image correction information storage section 312.

[0054] The storage or the like correction image storage section 316 stores a correction image obtained by the image correction section 331 correcting the respective photographed images using the image correction data of the image pickup system section 300a and the image pickup system section 300b. The storage or the like parallax image storage section 317 stores a parallax image. The storage or the like deviation correction information storage section 318 stores time series data of the position of the region having both the longitudinal edge and the lateral edge in the reference image and the reference image, the design value of the vanishing point position, the vanishing point position data (detection value), and the image deviation in the horizontal direction and the vertical direction caused by the camera mounting deviation.

[0055] The distance detection section 330 includes the image correction section 331, the parallax calculation section 332, the recognition section 333, and the camera mounting deviation detection section 334.

[0056] The image correction section 331 transforms the reference image and the reference image at the time of photographing read from the photographed image storage section 311 into images without distortion using the image correction data of the reference image and the reference image read from the image correction information storage section 312. These corrected reference image and reference image are stored to the correction image storage section 316.

[0057] The parallax calculation section 332 reads the corrected reference image and the reference image from the corrected image storage section 316, searches for a region on the corrected reference image that corresponds to a region of a prescribed size (referred to as a reference region) extracted from the corrected reference image, and that is at the same height on the corrected reference image. The parallax calculation section 332 calculates the parallax, which is the difference between the position of the reference region and the position of a region on the reference image that coincides with the reference region (referred to as a reference region). The parallax is calculated for each reference region, and thus a parallax image is calculated.

[0058] The recognition section 333 reads the parallax image from the parallax image storage section 317, and calculates the distance from the stereo camera 104 to the object on the image in the optical axis direction of the imaging system section 300a and the imaging system section 300b, based on the parallax, the distance between the focal points of the imaging system section 300a and the imaging system section 300b (the baseline length), the focal length, and the size of one pixel. The recognition section 333 calculates the distance for each region, and thus a distance image is calculated.

[0059] The recognition section 333 then reads the corrected reference image from the corrected image storage section 316, and uses the corrected reference image and the distance image to recognize the object that appears in the reference image and the position of the object on the reference image, and calculates the three-dimensional relative position and the relative velocity of the object with respect to the stereo camera 104. Here, the three-dimensional relative position coordinate system with respect to the stereo camera 104 is such that the x coordinate is taken in the right direction, the y coordinate is taken in the downward direction, and the z coordinate is taken in the optical axis direction, with respect to the imaging system section 300a, with the center of the entrance pupil of the imaging system section 300a as the origin. Furthermore, the recognition section 333 calculates the time until a collision based on the relative position and the relative velocity of the stereo camera 104 and the object, and determines whether a collision will occur within a prescribed time.

[0060] The camera mounting deviation detection section 334 creates edge images in the horizontal direction and the vertical direction for the corrected reference images of the previous time and the current time read from the corrected image storage section 316, and detects regions that contain both the vertical edge (the edge in the vertical direction) and the horizontal edge (the edge in the horizontal direction). Next, the camera mounting deviation detection section 334 performs pattern matching processing, searches for a region on the corrected reference image of the previous time that corresponds to a region on the corrected reference image of the current time, and calculates time series data of the position of the region that contains both the vertical edge and the horizontal edge.

[0061] Furthermore, the camera mounting deviation detection unit 334 calculates multiple approximate curves of the time-series data of the aforementioned region's location, and detects the position (detection value) of the vanishing point on the reference image based on their intersection points. The camera mounting deviation detection unit 334 stores the difference between the detected value and the design value of the vanishing point position on the reference image as the camera mounting deviation in the deviation correction information storage unit 318. Finally, the camera mounting deviation detection unit 334 corrects the degree of camera mounting deviation in the image correction data of the reference image read from the image correction information storage unit 312, and stores it in the image correction information storage unit 312. The same processing is performed on the reference image.

[0062] use Figure 4 ,right Figures 1-3 The operation sequence of one embodiment of the stereo camera 104 shown in this embodiment will be explained.

[0063] Step 401: The synchronization signal generator 314 generates a synchronization signal and sends it to the reference image import unit 315a and the reference image import unit 315b.

[0064] Upon receiving a synchronization signal from the synchronization signal generator 314, the reference image import unit 315a sends the synchronization signal and exposure time information to the imaging element unit 302a. Upon receiving the synchronization signal and exposure time information from the reference image import unit 315a, the imaging element unit 302a receives light of the image refracted by the optical element unit 301a at the required exposure time, generates an image corresponding to the intensity of that light, and sends the image to the reference image import unit 315a. The reference image import unit 315a receives the image from the imaging element unit 302a and stores the image in the captured image storage unit 311.

[0065] Upon receiving the synchronization signal from the synchronization signal generator 314, the reference image import unit 315b sends the synchronization signal and exposure time information to the imaging element unit 302b. Upon receiving the synchronization signal and exposure time information from the reference image import unit 315b, the imaging element unit 302b receives light of the image refracted by the optical element unit 301b at an exposure time level, generates an image corresponding to the intensity of that light, and sends the image to the reference image import unit 315b. The reference image import unit 315b receives the image from the imaging element unit 302b and stores the image in the captured image storage unit 311.

[0066] Step 402: The mark position detection unit 321 reads the reference image and reference image at the time of shooting from the image storage unit 311, reads the image correction data of the reference image and reference image stored in advance from the image correction information storage unit 312, and reads the glass deformation data of the reference image and reference image stored in advance from the correction information storage unit 313.

[0067] The image correction table (F12x, F12y) for the reference image when not passing through the windshield 112 is shown in equations (1) and (2). Here, (X1, Y1) is the position on the captured image (after passing through the optical element), and (X2, Y2) is the position on the corrected image (before passing through the optical element) when not passing through the windshield 112.

[0068] X1= F12x(X2, Y2) • • • (1)

[0069] Y1= F12y(X2, Y2) • • • (2)

[0070] Using equations (3) and (4), the position (X2, Y2) on the corrected reference image after passing through the windshield 112 corresponding to the position (X3, Y3) on the corrected reference image before passing through the windshield 112 is calculated from the glass deformation data (Gx, Gy).

[0071] X2= Gx(X3, Y3) • • • (3)

[0072] Y2= Gy(X3, Y3) • • • (4)

[0073] The position (X2, Y2) on the corrected reference image after passing through the windshield 112 is a real number. Therefore, the position on the reference image after capturing (after passing through the optical element) corresponding to the position on the reference image after correction (before passing through the optical element) after passing through the windshield 112 is calculated using equations (1) and (2) for four positions in which the values near the position (X2, Y2) are integers, and two-dimensional linear interpolation is performed. In this way, the position on the reference image after capturing corresponding to the position (X3, Y3) on the corrected reference image across the windshield 112 is calculated. This process is performed for each pixel, and the image correction data (equations (5) and (6)) for transforming the captured image into the corrected image across the windshield 112 is calculated.

[0074] X1= F13x(X3, Y3) • • • (5)

[0075] Y1= F13y(X3, Y3) • • • (6)

[0076] The luminance values of the four pixels around the position (X1, Y1) of the reference image after capturing are subjected to two-dimensional linear interpolation using the image correction data, and the luminance value of the pixel (X3, Y3) of the reference image after image correction is calculated. The above steps are performed for each pixel of the corrected reference image, and the luminance value of the corrected reference image in the case of the windshield 112 in accordance with the design is calculated.

[0077] The marker position detection unit 321 reads the focal length, pixel pitch, optical axis position on the captured image, optical axis position on the corrected image, and the three-dimensional position of markers 201-205 relative to the stereo camera 104 when they are set on the vehicle 111 without deviation, stored in the correction information storage unit 313, for both camera system units 300a and 300b. The marker position detection unit 321 then calculates the marker position (design position) on the corrected image when the stereo camera 104 has no setting deviation, does not penetrate the windshield 112, and has no optical element distortion, based on the values ​​of these parameters.

[0078] Next, the marker position detection unit 321 reads each marker image stored in the correction information storage unit 313 and calculates the sum of the absolute values ​​of the differences between the brightness values ​​of the corrected reference image near the design position of the marker and the brightness values ​​of the marker image (SAD). The marker position detection unit 321 determines the position where the SAD is minimum near the design positions of markers 201 to 205.

[0079] The marker position detection unit 321 performs isometric straight line fitting based on the SAD of the left and right pixels at the position with the minimum SAD, and calculates the sub-pixel at the position where the corrected reference image and the marker image are most consistent. The marker position detection unit 321 adds this sub-pixel to the horizontal position with the minimum SAD, thereby calculating the horizontal positions of markers 201 to 205.

[0080] Next, the marker position detection unit 321 performs isometric line fitting based on the SAD of the pixels above and below the position with the minimum SAD, and calculates the sub-pixel at the position where the corrected reference image and the marker image are most consistent. This sub-pixel is added to the vertical position with the minimum SAD, thereby calculating the vertical positions of markers 201-205. Similarly, the same processing is performed on the other markers 201-205 to calculate the positions of markers 201-205 on the corrected reference image. The positions of markers 201-205 detected through the above processing are called detection positions. Furthermore, the detection positions of markers 201-205 are... Figure 5 This is equivalent to correcting markers 502 and 504 on image 501.

[0081] The reference image is also processed using the image correction and marker location detection methods described above.

[0082] Step 403: The image deviation factor detection section 322 reads in the focal length, the pixel pitch, the optical axis position on the photographed image, the optical axis position on the corrected image, the three-dimensional position of the marks 201 to 205 with respect to the stereo camera 104 when the marks 201 to 205 are disposed in the vehicle 111 in a deviation-free manner, the design value of the glass distortion data of the imaging system section 300a and the imaging system section 300b from the correction information storage section 313. The image deviation factor detection section 322 calculates the position (design position) of the marks 201 to 205 on the corrected image in the case where the image deviation caused by the front windshield 112 when the marks 201 to 205 are disposed in the vehicle 111 in a deviation-free manner coincides with the design value, based on these pieces of information. The design position of the marks 201 to 205 corresponds to the marks 503 and 505 on the corrected image 501 in Figure 5

[0083] The image deviation factor detection section 322 estimates the magnification of the front windshield 112 and the camera setting deviation and the like, based on the design position of the marks 201 to 205 on the corrected image calculated as described above and the detected position of the marks 201 to 205 detected in step 402, as described below. Here, the fact that the image deviation caused by the front windshield 112 is not dependent on the distance between the stereo camera 104 and the marks 201 to 205 and the fact that the camera setting deviation is dependent on the distance between the stereo camera 104 and the marks 201 to 205 are utilized based on the positions of the marks 201 to 205 that differ in distance from the stereo camera 104.

[0084] Step 403: The image deviation factor detection section 322 estimates the magnification of the front windshield 112 and the camera setting deviation and the like, based on the design position of the marks 201 to 205 on the corrected image calculated as described above and the detected position of the marks 201 to 205 detected in step 402, as described below. Here, the fact that the image deviation caused by the front windshield 112 is not dependent on the distance between the stereo camera 104 and the marks 201 to 205 and the fact that the camera setting deviation is dependent on the distance between the stereo camera 104 and the marks 201 to 205 are utilized based on the positions of the marks 201 to 205 that differ in distance from the stereo camera 104.

[0085] ​First, for the reference image, the image deviation factor detection section 322 calculates the average of the intervals of the marks 201 adjacent in the horizontal direction and the vertical direction with respect to the detected positions and the design positions of the marks 201 on the card 206 in the correction image. Similarly, the average of the intervals of the detected positions and the design positions of the marks 202 to 205 on the cards 207 to 210 is calculated. Thus, the fact that the image deviation caused by the windshield 112 does not depend on the distance between the stereo camera 104 and the marks 201 to 205 and the fact that the camera setting deviation depends on the distance between the stereo camera 104 and the marks 201 to 205 are used. Thereby, the horizontal direction and the vertical direction magnification Mgh, Mgv caused by the windshield 112, and the distance dLc between the stereo camera 104 and the mark 201 are calculated using the formulas (7) to (10). Here, the far mark is the mark 201, and the near marks are the marks 202 to 205.

[0086] Mgh = Mifh Minh (Lf - Ln) / (Minh Lf - Mifh Ln) · · · (7)

[0087] Mgv = Mifv Minv (Lf - Ln) / (Minv Lf - Mifv Ln) · · · (8)

[0088] Mifh = Wifh / Wdfh · · · (9)

[0089] Mifv = Wifv / Wdfv · · · (10)

[0090] Minh = Winh / Wdnh · · · (11)

[0091] Minv = Winv / Wdnv · · · (12)

[0092] dLc = (dLch + dLcv) / 2 · · · (13)

[0093] dLch = (Mifh - Minh) Lf Ln / (Minh Lf - Mifh Ln) · · · (14)

[0094] dLcv = (Mifv - Minv) Lf Ln / (Minv Lf - Mifv Ln) · · · (15)

[0095] • Design value of the distance of the far mark 201 from the stereo camera 104: Lf

[0096] • Design value of the distance of the near marks 202 to 205 from the stereo camera 104: Ln

[0097] • Distance deviation of the markers 201-205 from the stereo camera 104: dLc

[0098] • Magnification in the horizontal direction by the front windshield 112: Mgh

[0099] • Magnification in the vertical direction by the front windshield 112: Mgv

[0100] • Average of the horizontal intervals of the design positions of the far markers 201 on the corrected image: Wdfh

[0101] • Average of the horizontal intervals of the design positions of the near markers 202-205 on the corrected image: Wdnh

[0102] • Average of the vertical intervals of the design positions of the far markers 201 on the corrected image: Wdfv

[0103] • Average of the vertical intervals of the design positions of the near markers 202-205 on the corrected image: Wdnv

[0104] • Average of the horizontal intervals of the detection positions of the far markers 201 on the corrected image: Wifh

[0105] • Average of the horizontal intervals of the detection positions of the near markers 202-205 on the corrected image: Winh

[0106] • Average of the vertical intervals of the detection positions of the far markers 201 on the corrected image: Wifv

[0107] • Average of the vertical intervals of the detection positions of the near markers 202-205 on the corrected image: Winv

[0108] • Magnification in the horizontal direction of the intervals of the far markers 201 on the corrected image: Mifh

[0109] • Magnification in the horizontal direction of the intervals of the near markers 202-205 on the corrected image: Minh

[0110] • Magnification in the vertical direction of the intervals of the far markers 201 on the corrected image: Mifv

[0111] • Magnification in the vertical direction of the intervals of the near markers 202-205 on the corrected image: Minv

[0112] Marker 201 on card 206, which is closer to marker 202, is designated as the distant marker, and marker 202 is designated as the near marker. Equations (13) to (15) are used to calculate the distance deviation dLclu between marker 202 (located in the upper left of the image) and the stereo camera 104. The same calculation is performed to calculate the distance deviations dLcru (upper right of the image), dLcld (lower left of the image), and dLclu (lower right of the image) between markers 203 to 205 and the stereo camera 104. Figure 6 As shown, the yaw angle deviation between the marker and the stereo camera 104 can be calculated based on the distance deviation between the left and right markers of lines 602 and 603 in the horizontal direction perpendicular to the optical axis direction 601 of the stereo camera 104. Therefore, the yaw angle deviation dψc between the marker and the stereo camera 104 is calculated using equations (16) to (18).

[0113] dψc=atan((dLcl-dLcr) / Llr) · · · (16)

[0114] dLcl=(dLcul+dLcdl) / 2 · · · (17)

[0115] dLcr=(dLcur+dLcdr) / 2 · · · (18)

[0116] • The average dLcl of the distance deviation between markers 202 and 204 on the left

[0117] • The average dLcr of the distance deviation between markers 203 and 205 on the right.

[0118] • The average Llr of the design value of the distance between the left and right markers

[0119] like Figure 7 As shown, the tilt angle deviation between the markers and the stereo camera 104 can be calculated based on the distance deviation between the upper and lower marks of lines 704 and 705 in a vertical direction perpendicular to the optical axis direction 703 of the stereo camera 104. Therefore, the tilt angle deviation dφc between the markers and the stereo camera 104 is calculated using equations (19) to (21).

[0120] dφc=atan((dLcd-dLcu) / Lud) · · · (19)

[0121] dLcu=(dLcul+dLcur) / 2 · · · (20)

[0122] dLcd=(dLcdl+dLcdr) / 2 · · · (21)

[0123] • The average distance deviation between the upper markings 202 and 203 and the stereo camera: dLcu

[0124] • The average distance deviation between the lower markings 204 and 205 and the stereo camera: dLcd

[0125] • The average design value of the distance between the upper and lower marks: Lud

[0126] The tilt angle deviation between the marker and the stereo camera 104 is equivalent to the rotational deviation between the marker's detection position and its designed position, centered on the optical axis position (or center position) on the corrected image. Therefore, the tilt angle deviation dθc is calculated using equation (22) on the corrected image.

[0127] dθc=-atan[Σ{(Vi-Vo)(Ud-Uo)-(Ui-Uo)(Vd-vo)} / Σ{(Ui-Uo)(Ud-Uo)+(Vi-Vo)(Vd-Vo)}]···(22)

[0128] • Detection positions marked 201-205: (Ui, Vi)

[0129] • Mark the design locations 201 to 205: (Ud, Vd)

[0130] • Correct the optical axis position (center position) on the image: (Uo, Vo)

[0131] The horizontal and vertical positional deviations between the marker and the stereo camera 104 depend on the distance between the stereo camera 104 and the marker, but the offset deviation caused by the windshield 112 does not depend on the distance between the stereo camera 104 and the marker. Using this fact, equations (23) to (26) are used to calculate the horizontal and vertical offset deviations dUg and dVg caused by the windshield 112, and the horizontal and vertical positional deviations dXc and dYc between the marker and the stereo camera 104. Here, the distant marker corresponds to marker 201, and the nearby markers correspond to markers 202 to 205.

[0132] dXc=(dUin-dUif)c Lf Ln / {f(Lf-Ln)} · · · (23)

[0133] dYc=(dVin-dVif)c Lf Ln / {f(Lf-Ln)} · · · (24)

[0134] dUg=(dUin Ln-dUif Lf) / (Lf-Ln) · · · (25)

[0135] dVg = (dVin Ln - dVif Lf) / (Lf - Ln) · · · (26)

[0136] • focal length of lens: f

[0137] • pixel pitch of image pickup element: c

[0138] • average of difference between detected position and design position of far mark on rectified image in horizontal direction: dUif

[0139] • average of difference between detected position and design position of near mark on rectified image in horizontal direction: dUin

[0140] • average of difference between detected position and design position of far mark on rectified image in vertical direction: dVif

[0141] • average of difference between detected position and design position of far mark on rectified image in vertical direction: dVin

[0142] The above-described processing performed on the reference image is also performed on the reference image.

[0143] The average of the horizontal and vertical position deviations, distance deviations, pitch angle deviations, yaw angle deviations, roll angle deviations between the stereoscopic camera 104 and the marks calculated using the reference image and the reference image are calculated. These are taken as the setting deviations of the stereoscopic camera 104. Furthermore, by the above calculation, the horizontal and vertical magnification and shift deviations caused by the windshield 112 of the reference image and the reference image are obtained.

[0144] Step 404: The vehicle setting deviation detection section 105 irradiates the vehicle 111 with laser light, and detects the distances and light receiving angles to a plurality of portions of the surface of the vehicle 111 from the light receiving times of the reflected light thereof. The detected positions of the plurality of portions of the surface of the vehicle 111 are converted from these distances and light receiving angles.

[0145] The vehicle setting deviation detection section 105 has the surface shape data of the vehicle 111 when set as designed and the position (design position) and attitude (design attitude) of the vehicle 111, and calculates the design positions of a plurality of portions of the surface of the vehicle 111 from these data.

[0146] The design position closest to the detected position of each portion of the surface of the vehicle 111 is searched for, and this design position is set as the correspondence point of the detected position. From the detected positions (Xvm, Yvm, Zvm) and the design positions (Xvd, Yvd, Zvd) of each portion of the surface of the vehicle 111, the position deviations (dXv, dYv, dLv) and the attitude deviations (dφv, dψv, dθv) of the vehicle 111 are calculated using Equations (27) to (32).

[0147] dXv = ∑(Xvd- Xvm) / Nm · · · (27)

[0148] dYv = ∑(Yvd- Yvm) / Nm · · · (28)

[0149] dLv = ∑(Zvd- Zvm) / Nm · · · (29)

[0150] dφv = -atan[∑{(Zvm- Zo)(Yvd- Yvo) - (Yvm- Yvo)(Zvd- Zvo)} / ∑{(Yvm- Yvo)(Yvd- Yvo) + (Zvm- Zvo)(Zvd- Zvo)}] · · · (30)

[0151] dψv = -atan[∑{(Xvm- Xo)(Zvd- Zvo) - (Zvm- Zvo)(Xvd- Xvo)} / ∑{(Zvm- Zvo)(Zvd- Zvo) + (Xvm- Xvo)(Xvd- Xvo)}] · · · (31)

[0152] dθv = -atan[∑{(Yvm- Yo)(Xvd- Xvo) - (Xvm- Xvo)(Yvd- Yvo)} / ∑{(Xvm- Xvo)(Xvd- Xvo) + (Yvm- Yvo)(Yvd- Yvo)}] · · · (32)

[0153] • the number of measurement points Nm

[0154] • the center of rotation of the vehicle 111 (Xvo, Yvo, Zvo)

[0155] The design position and the design posture of the vehicle 111 are added to the position deviation and the posture deviation of the vehicle 111 calculated as described above, and the design positions of the plurality of portions of the surface of the vehicle 111 are calculated again from the surface shape data of the vehicle 111. Then, the following processing is repeated: the design position closest to the detection position of each portion of the surface of the vehicle 111 is searched for, the design position is set as the corresponding point of the detection position, and the position deviation and the posture deviation of the vehicle 111 are calculated using the equations (27) to (32). In a case where the values of the deviations calculated using the equations (27) to (32) are smaller than a prescribed threshold value, the processing is ended, and the value obtained by adding the deviations up to this point is set as the position deviation and the posture deviation of the vehicle setting.

[0156] Step 405: The camera mounting deviation detection section 334 calculates the position deviation (dXa, dYa, dZa) and the attitude deviation (dφa, dψa, dθa) of the camera mounting by subtracting the position deviation and the attitude deviation of the vehicle setting from the position deviation and the attitude deviation of the camera setting using the equations (33) to (38).

[0157] dXa = dXc - dXv... (33)

[0158] dYa = dYc - dYv... (34)

[0159] dZa = dZc - dZv... (35)

[0160] dφa = dφc - dφv... (36)

[0161] dψa = dψc - dψv... (37)

[0162] dθa = dθc - dθv... (38)

[0163] Step 406: The image correction data changing section 323 reads the focal length, the pixel pitch, the optical axis position on the captured image, the optical axis position on the corrected image, and the three-dimensional position of the marker with respect to the stereo camera 104 when the marker is set on the vehicle 111 without deviation, which are stored in the correction information storage section 313, for the camera system section 300a and the camera system section 300b. The image correction data changing section 323 calculates the marker position (design position) on the corrected image when the stereo camera 104 is set without deviation, does not pass through the windshield 112, and has no distortion of the optical element, based on the values of these parameters.

[0164] The image correction data changing section 323 calculates the marker position on the corrected image when the stereo camera 104 passes through the windshield 112, based on the position deviation and the attitude deviation of the setting calculated by the image deviation factor detection section 322. The image correction data changing section 323 creates the camera setting deviation data indicating the image deviation caused by the camera setting deviation, based on the change in the marker position on the two corrected images.

[0165] Next, the image correction data changing section 323 calculates the mark position on the corrected image when there is no (removing) the deviation of the installation direction of the camera with respect to the traveling direction of the vehicle 111, based on the positional deviation and the attitude deviation of the stereo camera 104 set calculated by the image deviation factor detecting section 322. Then, the image correction data changing section 323 calculates the camera installation direction deviation data based on the mark position on the corrected image when there is the positional deviation of the stereo camera set and when the deviation of the installation direction of the camera with respect to the traveling direction of the vehicle 111 is removed therefrom.

[0166] The image correction data changing section 323 calculates the magnification in the horizontal direction and the vertical direction of the model of the windshield 112 calculated by the image deviation factor detecting section 322, the glass deviation data when there is the positional deviation. The image correction data changing section 323 sequentially performs the transformation based on the camera installation direction deviation data, the transformation based on the glass deviation data, the transformation based on the glass deformation data, the transformation based on the camera set deviation data, and the transformation based on the image correction data (indicating the distortion caused by the optical element) when there is no penetration through the windshield 112 for each pixel using the two-dimensional linear interpolation shown in step 402.

[0167] Thus, the image correction data changing section 323 removes the pixel deviation caused by the camera set deviation, and calculates the image correction data through the windshield 112 when there is the traveling direction of the vehicle 111 for the reference image and the reference image. The image correction data changing section 323 stores these image correction data to the image correction information storage section 312.

[0168] Figure 3 The stereo camera 104 of the present embodiment shown performs the above-described operation during the traveling of the vehicle 111. Figure 8 By the operation shown, the distance of the stereo object is detected using the image correction data produced, and the image correction data is changed by detecting the camera installation deviation.

[0169] Step 801: The synchronization signal generating section 314 generates a synchronization signal and sends it to the reference image introducing section 315a and the reference image introducing section 315b.

[0170] The reference image import section 315a sends the synchronization signal and the information of the exposure time to the imaging element section 302a immediately after receiving the synchronization signal from the synchronization signal generation section 314. The imaging element section 302a performs light reception of the image of the light refracted by the optical element section 301a for the exposure time, generates an image corresponding to the intensity of the light, and sends the image to the reference image import section 315a immediately after receiving the synchronization signal and the information of the exposure time from the reference image import section 315a. The reference image import section 315a receives the image from the imaging element section 302a and stores the image in the captured image storage section 311.

[0171] The reference image import section 315a sends the synchronization signal and the information of the exposure time to the imaging element section 302a immediately after receiving the synchronization signal from the synchronization signal generation section 314. The imaging element section 302a performs light reception of the image of the light refracted by the optical element section 301a for the exposure time, generates an image corresponding to the intensity of the light, and sends the image to the reference image import section 315a immediately after receiving the synchronization signal and the information of the exposure time from the reference image import section 315a. The reference image import section 315a receives the image from the imaging element section 302a and stores the image in the captured image storage section 311.

[0172] Step 802: The image correction section 331 reads the captured reference image and the captured reference image from the captured image storage section 311. The image correction section 331 reads the image correction data of the reference image and the reference image from the image correction information storage section 312.

[0173] Using the formula (39) and the formula (40), the position (Xl, Yl) of the captured reference image corresponding to the pixel (X4, Y4) of the corrected reference image is calculated from the coordinates (Fx(X4, Y4), Fy(X4, Y4)) on the captured image corresponding to each pixel on the corrected reference image.

[0174] Xl = Fx(X4, Y4)... (39)

[0175] Yl = Fy(X4, Y4)... (40)

[0176] The luminance value of the pixel (X4, Y4) of the corrected reference image is calculated by performing two-dimensional linear interpolation on the luminance values of the 4 pixels around the position (Xl, Yl) of the reference image. The luminance value of the corrected reference image is calculated by performing the above step on each pixel of the corrected reference image. Further, the luminance value of the corrected image of the reference image is calculated by performing the above step on the reference image. The corrected reference image and the reference image are stored in the corrected image storage section 316.

[0177] Step 803: The disparity calculation unit 332 reads the corrected image of the reference image and the reference image from the corrected image storage unit 316. The disparity calculation unit 332... Figure 9 As shown, a region 903 (referred to as the reference region) of a specified size is extracted from the corrected reference image 901. The parallax calculation unit 332 explores the image of the region of the same object as the reference region 903 on the corrected reference image 902 by pattern matching.

[0178] The disparity calculation unit 332 extracts a region 904 (reference region) of a predetermined size on a reference image 902 at the same height as the reference image 903, and calculates the sum of the absolute values ​​of the differences between the brightness values ​​of the reference region 903 and the brightness values ​​of the reference region 904 (SAD). The disparity calculation unit 332 calculates the SAD for each reference region 904 on the reference image 902 at the same height as the reference region 903, and explores the reference region 905 with the minimum SAD value. The disparity calculation unit 332 uses the SAD of the reference region 905 and the SADs of the left and right adjacent reference regions one pixel away from the reference region 905 to perform isoangular line fitting, calculating the sub-pixel of the reference region 905 on the reference image that best matches the reference region 903. The disparity calculation unit 332 adds the sub-pixel to the difference in position between the reference region 903 and the reference region 905, thereby calculating the disparity of the reference region 903 on the corrected reference image 901.

[0179] The disparity calculation unit 332 performs this processing on all regions of the image-corrected reference image 901 to calculate the overall disparity of the reference image 901. The disparity calculation unit 332 then stores the calculated disparity image in the disparity image storage unit 317.

[0180] Step 804: The recognition unit 333 reads the parallax image from the parallax image storage unit 317.

[0181] The recognition unit 333 uses formula (41) to calculate the distance L between the region on the parallax image and the optical axis of the stereo camera 104. Here, f is the design value of the focal length of the camera system unit 300a and the camera system unit 300b, B is the distance between the principal points of the camera system unit 300a and the camera system unit 300b (baseline length), d is the parallax, and c is the pixel pitch of the camera element unit 302a and the camera element unit 302b.

[0182] L=f×B / (d×c) · · · (41)

[0183] The recognition unit 333 performs this processing on all areas of the parallax image, calculates the distance of the entire parallax image to the optical axis direction of the stereo camera 104, and produces a distance image.

[0184] The recognition section 333 calculates the three-dimensional position within the region on the distance image using formulas (42) to (44). Here, the three-dimensional coordinate system is such that the X coordinate is taken in the right direction, the Y coordinate is taken in the upward direction, and the Z coordinate is taken in the traveling direction, with the origin being a point on the road surface vertically downward from the principal point of the optical element section 301a of the imaging system section 300a of the stereo camera 104. Further, (U, V) is each position on the distance image, (U0, V0) is the optical axis position on the reference image, and H is the installation height of the stereo camera 104 from the road surface.

[0185] X = L x c x (U - U0) / f · · · (42)

[0186] Y = H + L x c x (V - V0) / f · · · (43)

[0187] Z = L · · · (44)

[0188] The recognition section 333 performs this processing on all regions of the parallax image, and calculates the three-dimensional position on the entire parallax image.

[0189] The recognition section 333 detects regions on the distance image for which the ratio of the distance is within the threshold of the processing and which are adjacent, and in the case where the area of the region is above the threshold and the height (Y coordinate) of the regions from the road surface is above the threshold, infers that it is a stereo object.

[0190] Step 805: The camera installation deviation detection section 334 reads in the corrected reference images produced in the previous and current processing from the correction image storage section 316. The camera installation deviation detection section 334 produces edge images in the horizontal direction and the vertical direction with respect to the previous and current corrected reference images. The camera installation deviation detection section 334 detects regions in which both a vertical edge and a horizontal edge exist within regions of a prescribed size on these edge images. The camera installation deviation detection section 334 uses a pattern matching process to search for a region on the previous corrected reference image that coincides with this region.

[0191] In the case where edges that coincide are detected on the previous and current correction images, the camera installation deviation detection section 334 detects the edge positions of the previous and current, and additionally stores the edge position of the current in the deviation correction information storage section 318 in correspondence with the detected edge position of the previous. In the case where there are no edges that coincide on the previous and current correction images, the camera installation deviation detection section 334 newly stores the edge position of the current in the deviation correction information storage section 318. In this way, the camera installation deviation detection section 334 continuously stores the time-series positions of the edges.

[0192] When the number of time-series data at a certain edge position is above a prescribed threshold, the camera installation deviation detection section 334 obtains an approximate straight line that links the time-series data that are edge-uniform. The camera installation deviation detection section 334 performs this processing on each region that contains both the vertical edge and the horizontal edge. The camera installation deviation detection section 334 obtains the position of the intersection on the image of each combination of these approximate straight lines, and averages the positions of the intersections.

[0193] When the average of the positions of the intersections is within a distance of a threshold from the designed vanishing point position, the camera installation deviation detection section 334 determines that it is straight ahead, takes the average of the positions of the intersections as the vanishing point position, increments the number of vanishing point position data, and stores it to the deviation correction information storage section 318. When the average of the positions of the intersections is above a distance of a threshold from the designed vanishing point position, the camera installation deviation detection section 334 determines that it is a turn, and thus determines that the vanishing point position data is not accurately detected and discards it.

[0194] When the number of vanishing point position data stored in the deviation correction information storage section 318 is above a prescribed threshold, the camera installation deviation detection section 334 averages the vanishing point position data stored in the deviation correction information storage section 318, obtains a detected value of the vanishing point position, and takes the difference between the detected value of the vanishing point position and the designed value as the image deviation in the horizontal direction and the vertical direction due to the camera installation deviation.

[0195] The camera installation deviation detection section 334 performs the above processing on the reference image. The camera installation deviation detection section 334 averages the image deviation in the horizontal direction and the vertical direction due to the camera installation deviation in the reference image and the reference image, and stores it to the deviation correction information storage section 318.

[0196] Step 806: The camera installation deviation detection section 334 reads the image correction data of the reference image from the image correction information storage section 312, and reads the image deviation in the horizontal direction and the vertical direction due to the camera installation deviation from the deviation correction information storage section 318. The camera installation deviation detection section 334 sequentially performs, for each of the pixels, a transformation based on the image correction data of the reference image, and a transformation that corrects the image deviation in the horizontal direction and the vertical direction due to the camera installation deviation, using two-dimensional linear interpolation as shown in step 402. Thus, the camera installation deviation detection section 334 obtains the image correction data that removes the camera installation deviation.

[0197] The camera installation deviation detection section 334 performs the above processing on the reference image. The camera installation deviation detection section 334 stores the image correction data of the reference image and the reference image thus obtained to the image correction information storage section 312.

[0198] When the stereo camera 104 is installed on the vehicle 111, if there is a vehicle setting deviation or a stereo camera installation deviation, it cannot be distinguished from the image deviation caused by the windshield 112. Thus, the image deviation caused by the vehicle setting deviation or the stereo camera installation deviation is included in the image deviation caused by the windshield 112, resulting in an error in the changed image correction data. Then, horizontal and vertical image deviations are generated in the corrected reference image and the reference image of the object at a distance different from the marker. Then, the vertical image deviation causes the areas of the reference image and the reference image to be inconsistent in the pattern matching process, and the parallax cannot be normally calculated. Furthermore, even if the vertical deviation is corrected, the parallax error is generated due to the horizontal image deviation.

[0199] According to the present embodiment, the magnification and the shift deviation of the model of the glass of the vehicle 111 independent of the distance, and the position and the attitude deviation between the marker dependent on the distance and the camera are detected from the interval and the position deviation of the markers on the image at different distances. Then, assuming that there is a vehicle setting deviation and a camera installation deviation in the camera position, the data for correcting the image are changed in a manner to correct the deviation of the image caused by the windshield 112. Thus, the deviation of the image caused by the vehicle setting deviation and the camera installation deviation can be removed by only one time of photographing, and thus the distance can be accurately measured.

[0200] According to Figure 3 In the present embodiment shown in FIG. 8, the position of the vanishing point on the corrected image is detected by the camera installation deviation detection section 334 in steps 805 and 806 of the operation sequence shown in FIG. 7 during the travel of the vehicle 111. Figure 8 In the present embodiment shown in FIG. 8, the position of the vanishing point on the corrected image is detected by the camera installation deviation detection section 334 in steps 805 and 806 of the operation sequence shown in FIG. 7 during the travel of the vehicle 111.

[0201] Further, the processing device and the vehicle-mounted camera device of the present application are not limited to the above-described embodiments, and can be variously modified and used. Hereinafter, modified examples of the embodiments of the processing device and the vehicle-mounted camera device of the present application will be described.

[0202] (Modified Example 1-1) (Embodiment of processing of image correction by calculation section)

[0203] Figure 1 and Figure 10 The configuration of an embodiment of the processing device and the vehicle-mounted camera device of the present application is shown in FIG. 7. The present modified example is different from the above-described embodiments in that the image correction data is changed in a manner to correct the deviation of the image caused by the windshield 112 in steps 805 and 806 of the operation sequence shown in FIG. 7 during the travel of the vehicle 111. Figure 1 and Figure 3In one embodiment of the processing device and vehicle-mounted camera device shown, the correction information storage unit 313 and the image correction unit 320 are removed from the stereo camera 104 and the correction information storage unit 313 and the image correction unit 320 are equipped in the newly added computing unit 106 (1000).

[0204] The computing unit 106 (1000) is connected to the stereo camera 104 and the vehicle setup deviation detection unit 105. The operation of each unit is... Figure 1 and Figure 3 The processing device and the vehicle-mounted camera device shown are implemented in the same way. Thus, by removing the correction information storage unit 313 and the image correction unit 320 from the stereo camera 104, the execution file for image correction processing can be removed from the stereo camera 104, thereby reducing the memory capacity used within the stereo camera 104. Furthermore, the newly added computing unit 106 (1000) is specifically installed during the installation process onto the vehicle 111, and the computing power of the computing unit 106 (1000) is faster than that of the stereo camera 104. Therefore, the time spent on image correction processing can be shortened, thereby increasing the number of images that can be corrected per hour.

[0205] (Variations 1-2) (Detection of camera setting deviation only)

[0206] This variation is in Figure 1 and Figure 3 In the embodiments of the processing apparatus and vehicle-mounted camera device of the present invention shown, the vehicle setting deviation detection unit 105 is excluded. Furthermore, it is not implemented... Figure 4 Steps 404 and 405 in the sequence of actions shown. Furthermore, in step 406, instead of sequentially performing transformations based on camera mounting orientation deviation data, glass deviation data, glass deformation data, camera setting deviation data, and image correction data (representing distortion caused by optical elements) for each pixel, the transformations are performed sequentially based on glass deviation data, glass deformation data, camera setting deviation data, and image correction data (representing distortion caused by optical elements) for each pixel.

[0207] Therefore, even without the vehicle setup deviation detection unit 105, the correction error caused by the setup deviation of the stereo camera 104 due to the vehicle setup deviation can be removed from the image correction data. In this modified example, the vanishing point position on the image caused by the stereo camera installation deviation will produce an error, but... Figure 8The camera mounting deviation is detected and the image correction data for correcting the image deviation caused by the camera mounting deviation is calculated in steps 805 and 806 of the illustrated action sequence, whereby the image deviation caused by the stereo camera mounting deviation can be corrected.

[0208] (Modified Example 1-3) (Embodiment of detecting the position of the center of the mark)

[0209] In the present modified example, in the embodiments of the processing device and the in-vehicle camera device of the present application illustrated in Figure 1 and Figure 3 In the embodiments of the processing device and the in-vehicle camera device of the present application illustrated in

[0210] (Modified Example 1-4) (Image correction device of monocular camera)

[0211] In the present modified example, in the embodiments of the processing device and the in-vehicle camera device of the present application illustrated in Figure 1 and Figure 3 In the embodiments of the processing device and the in-vehicle camera device of the present application illustrated in Figure 1 the stereo camera 104 is changed to a monocular camera 104, the monocular camera illustrated in Figure 11 is provided instead of the stereo camera illustrated in Figure 3 , the action sequence illustrated in Figure 12 is performed instead of the action sequence during driving illustrated in Figure 8 , and in the action sequences illustrated in Figure 4 and Figure 12 , each section performs the image-related processing of the imaging system 1000 instead of the image-related processing of the imaging system 300a and the imaging system 300b.

[0212] Figure 1 and Figure 11 The configuration of an embodiment of the image correction device of the present application is shown in

[0213] A monocular camera is installed in the vehicle 111, and captures the marks 201 to 205 through the front glass 112 of the vehicle 111. Further, the monocular camera has a camera system section 1000 and an arithmetic section 310, and is connected to the vehicle setting deviation detection section 105.

[0214] The camera system section 1000 such as a camera has an optical element section 1001 and a camera element section 1002. The optical element section 1001 such as a lens refracts light to form an image on the camera element section 1002. The camera element section 1002 such as a camera element photoelectrically converts the image of the light refracted by the optical element section 1001 to generate an image corresponding to the intensity of the light.

[0215] The arithmetic section 310 such as a CPU (central processing unit) and a memory has a captured image storage section 311, an image correction information storage section 312, a correction information storage section 313, a corrected image storage section 316, a deviation correction information storage section 318, an image import section 1015, an image correction section 320, and a distance detection section 1030.

[0216] The distance detection section 1030 has an image correction section 331, an identification section 1032, a three-dimensional object distance detection section 1033, and a camera installation deviation detection section 334.

[0217] The captured image storage section 311 such as a memory, the image correction information storage section 312, the correction information storage section 313, the corrected image storage section 316, and the deviation correction information storage section 318 store information of the camera system section 1000, not information of the camera system section 300a and the camera system section 300b.

[0218] The image import section 1015 acquires an image generated by the camera element section 1002, and stores the image in the captured image storage section 311.

[0219] The mark position detection section 321, the image deviation factor detection section 322, the image correction data change section 323, the image correction section 331, and the camera installation deviation detection section 1034 perform the same processing related to the image of the camera system section 1000, not the camera system section 300a and the camera system section 300b.

[0220] The identification section 1032 calculates a feature amount of the vehicle 111 in each region of the image corrected by the image correction section 331, and detects a region in which the feature of the vehicle 111 is consistent. Further, the identification section 1032 calculates a feature amount of a pedestrian in each region of the corrected image, and detects a region in which the feature of the pedestrian is consistent.

[0221] The stereoscopic object distance detection section 1033 calculates the distance of the vehicle 111 and the pedestrian from the position of the lower part of the region of the vehicle 111 and the region of the pedestrian as the position in contact with the road surface.

[0222] Figure 1 and Figure 11 The present modification shown in Figure 4 performs the following processing in step 401, and the marker position detection section 321, the image deviation factor detection section 322, and the image correction data changing section 323 perform the image-related processing of the imaging system section 1000 instead of the image-related processing of the imaging system section 300a and the imaging system section 300b, thereby detecting the camera setting deviation, the vehicle setting deviation, the camera installation deviation, and the magnification in the horizontal direction and the vertical direction of the windshield 112.

[0223] Step 401: The image introduction section 1015 transmits the imaging signal and the information of the exposure time to the imaging element section 1002. The imaging element section 1002 performs the light-receiving of the image of the light refracted by the optical element section 1001 to the extent of the exposure time immediately after receiving the imaging signal and the information of the exposure time from the image introduction section 1015, generates the image corresponding to the intensity of the light, and sends the image to the image introduction section 1015. The image introduction section 1015 receives the image from the imaging element section 1002, and stores the image to the captured image storage section 311.

[0224] Figure 1 and Figure 11 In the embodiment of the monocular camera of the present modification shown in Figure 12 instead of the action sequence shown in Figure 8 thereby detecting the distance of the stereoscopic object and changing the image correction data by detecting the camera installation deviation. Here, the same processing as that of step 401 (described above) of the modification example 1-4 not using the synchronization signal generation section is performed in step 801. Further, in steps 802 and 805 to 806, instead of the image-related processing of the imaging system section 300a and the imaging system section 300b, the same image-related processing of the imaging system section 1000 is performed, so the explanation of these steps is omitted, and only the explanation of steps 1103 and 1104 is shown.

[0225] Step 1103: The recognition section 1032 reads the corrected image stored in the correction image storage section 316. The characteristic amount of the vehicle 111 is calculated in each region in the corrected image 1201, as shown in Figure 13As shown, the region 1202 in which the feature of the vehicle 111 is detected. Further, the recognition section 1032 calculates the feature amount of the pedestrian in each region in the corrected image 1201, and detects the region 1203 in which the feature of the pedestrian is detected.

[0226] Step 1104: The stereoscopic object distance detection section 1033 calculates the distance of the vehicle 111 and the pedestrian from the position of the lower part of the region 1202 of the vehicle 111 and the region 1203 of the pedestrian as the position in contact with the road surface, and calculates the distance of the vehicle 111 and the pedestrian from the position in contact with the road surface.

[0227] When a monocular camera is installed on the vehicle 111, if there is a vehicle setting deviation or a monocular camera installation deviation, it cannot be distinguished from the image deviation caused by the windshield 112, and the image deviation caused by the vehicle setting deviation or the monocular camera installation deviation is included in the image deviation caused by the windshield 112, resulting in an error in the corrected image of the changed image correction data, and thus horizontal and vertical image deviations are generated in the corrected image of the object at a distance different from the marker, and the vertical image deviation causes an error in their distance.

[0228] In Modification 1-4, the magnification and shift deviation of the model of the windshield 112 of the vehicle 111 independent of the distance, and the position and attitude deviation between the marker and the camera dependent on the distance are detected from the interval and position deviation of the markers on the image at different distances. Then, assuming that there is a vehicle setting deviation and a camera installation deviation in the camera position, the data for correcting the deviation of the image caused by the windshield 112 is changed in a manner to correct the deviation of the image caused by the windshield 112, and thus the deviation of the image caused by the vehicle setting deviation and the camera installation deviation can be removed by only one shooting, and thus the distance can be accurately measured.

[0229] Further, the monocular camera of Modification 1-4 is not limited to the above-described embodiments, and can be used as various modifications as shown in Modifications 1-1 to 1-3, and the same effects can be obtained.

[0230] The above-described embodiments of the processing device and the vehicle-mounted camera device of the present disclosure are described in detail using the drawings, but the specific configuration is not limited to the embodiments, and even if there are design changes and the like within the scope of the main idea of the present disclosure, they are included in the present disclosure.

[0231] Symbol Explanation

[0232] 101…marker, 102…marker, 103…marker, 104…stereo camera, 105…vehicle setting deviation detection unit, 106…calculation unit, 111…vehicle, 112…front windshield, 121…predetermined vehicle position, 122…predetermined vehicle posture (direction), 123…vehicle posture (direction), 124…stereo camera posture (direction), 201…marker, 202…marker, 203…marker, 204…marker, 205…marker, 206…card, 207…card, 208…card, 209…card, 210…card, 300b…imaging system unit, 301a…optical element unit, 301b…optical element unit, 302a…imaging element unit, 302b…imaging element unit, 310…operation unit, 311…captured image storage unit, 312…image correction information storage unit, 313…correction information storage unit, 314…synchronization signal generation unit, 315a…reference image import unit, 315b…reference image import unit, 316…corrected image storage unit, 317…parallax image storage unit, 318…deviation correction information storage unit, 320…image correction unit, 321…marker position detection unit, 322…image deviation factor detection unit, 323…image correction data change unit, 330…distance detection unit, 331…image correction unit, 332…parallax calculation unit, 333…recognition unit, 334…camera installation deviation detection unit, 501…corrected image, 502…detected position of marker, 503…design position of marker, 504…detected position of marker, design position of marker, 601…optical axis direction, 602…line in horizontal direction perpendicular to optical axis direction, 603…line in horizontal direction perpendicular to optical axis direction, 701…marker, 702…marker, 703…optical axis direction, 704…line in vertical direction perpendicular to optical axis direction, 705…line in vertical direction perpendicular to optical axis direction, 901…corrected reference image, 902…corrected reference image, 903…reference region, 904…reference region, 905…reference region most coincident with reference region, 1000…calculation unit, 1000…imaging system unit, 1001…optical element unit, 1002…imaging element unit, 1015…image import unit, 1030…distance detection unit, 1032…recognition unit, 1033…stereo object distance detection unit, 1201…corrected image, 1202…vehicle, 1203…pedestrian.

Claims

1. A processing apparatus, characterized in that, The setting deviation of the vehicle-mounted camera is obtained by capturing images from multiple markers at different distances from the camera and by using data from a modeled vehicle window. This setting deviation is then used to adjust the correction data for refining the images captured by the vehicle-mounted camera. The setting deviation refers to the positional deviation relative to the position that the card with the marker should be positioned for, as indicated by the vehicle-mounted camera. A vehicle setting deviation detection unit is provided for detecting the setting deviation of the vehicle. The installation deviation of the vehicle-mounted camera relative to the vehicle is calculated based on the setting deviation of the vehicle and the setting deviation of the vehicle-mounted camera. The correction data for correcting the image of the vehicle-mounted camera is then changed using the setting deviation and the installation deviation of the vehicle-mounted camera.

2. The processing apparatus according to claim 1, characterized in that, The markers are configured to have different distances from the vehicle-mounted camera in the vertical direction. The processing device obtains the setting deviation based on the interval and distance deviation of the marks. The setting deviation is the pitch angle deviation of the vehicle.

3. The processing apparatus according to claim 1, characterized in that, The markers are configured to be at different distances from the vehicle-mounted camera in the left and right directions. The processing device obtains the setting deviation based on the interval and distance deviation of the marks. The setting deviation is the yaw angle deviation of the vehicle.

4. The processing apparatus according to claim 1, characterized in that, The processing device obtains the horizontal and vertical magnification of the vehicle's windows based on the horizontal and vertical spacing deviations of the marks on the image.

5. A vehicle-mounted camera device, characterized in that, The vehicle-mounted camera device, equipped with the processing apparatus according to claim 1, includes a first camera and a second camera. The processing device acquires a first image of the marker captured by the first camera and a second image of the marker captured by the second camera. The magnification deviation, offset deviation, and common vehicle setting deviation of the vehicle windows related to the first camera and the second camera are obtained based on the marker interval deviation and position deviation obtained from the first image and the second image. The correction data used for the first camera and the second camera are changed based on the magnification deviation, the offset deviation, and the common vehicle setting deviation.

Images