Vehicle driving control device

Abstract

To provide a vehicle travel control device that can reliably calibrate a stereo camera device that recognizes the front environment with a simple configuration, always acquires three-dimensional image information including accurate distance information, and can ensure higher safety and reliability.SOLUTION: A vehicle travel control device 1 includes at least a stereo camera device 11 that acquires three-dimensional image information including distance information on the basis of image data acquired by recognizing a predetermined range of a vehicle front region as an image, a lidar device 12 that scans a range that is substantially the same as the recognition range of the stereo camera device and acquires three-dimensional point cloud data including distance information for each of a plurality of detection points, and a control unit 10 that controls the stereo camera device and the lidar device, and further includes a camera calibration unit 10a that calibrates the stereo camera device by correcting the positional deviation of the three-dimensional image information acquired by the stereo camera device on the basis of the three-dimensional point cloud data acquired by the lidar device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for calibrating a stereo camera device that recognizes an environment ahead in the traveling direction in a cruise control device of a vehicle. [Background technology]

[0002] In recent years, automatic driving control technology has been developed for vehicles such as automobiles, which allows the vehicle to travel automatically without the driver's operation. In addition, various proposals have been made for vehicle travel control devices that assist the driver's driving operation by utilizing such automatic driving control technology, with the aim of reducing the burden of the driver's driving operation and improving safety when the vehicle is travelling, and these devices are being generally put into practical use.

[0003] For example, some conventional vehicle driving control devices are configured with a surrounding environment recognition device using various sensors (autonomous sensor devices) such as an imaging device (camera), a radar device, and a sonar device. This surrounding environment recognition device recognizes obstacles (e.g., other vehicles, pedestrians, bicycles, motorcycles, and other moving objects and fixed objects) that exist in the vehicle's traveling direction (mainly the forward area) and in the rear or lateral areas of the vehicle. Based on the recognition results by the surrounding environment recognition device, the vehicle driving control device realizes various driving controls for avoiding collision or contact of the vehicle with the recognized obstacles, such as emergency braking (AEB (Autonomous Emergency Braking); so-called collision damage mitigation brake) control, steering control, or rear side vigilance support control when changing lanes.

[0004] In this type of conventional vehicle cruise control device, for example, a stereo camera device has been put to practical use as a type of surrounding environment recognition device for recognizing the surrounding environment.

[0005] In a vehicle driving control device to which this type of stereo camera device is applied, for example, a displacement in the imaging position may occur due to mechanical factors such as a mounting position shift caused by the mounting accuracy when mounting the stereo camera device to the vehicle or aging, or a mounting position shift of internal components (such as an imaging element and an imaging lens) caused by the manufacturing and assembly accuracy of the stereo camera device itself or aging. Furthermore, a distortion of the optical image may occur due to the optical performance of the imaging lens or the occurrence of an imaging position shift caused by capturing an image through the windshield of the vehicle. When such an image distortion occurs, there is a problem that it becomes difficult to obtain accurate distance information to the target object. As a result, there is a possibility that a malfunction may occur in the vehicle driving control performed using image information including distance information obtained by the stereo camera device.

[0006] Therefore, in conventional vehicle driving control devices that use a stereo camera device, various technical means have been disclosed, for example in JP 2017-32356 A, for suppressing erroneous detection of information such as distance to an object caused by the imaging position shift due to the mechanical position shift described above, the imaging position shift due to the optical performance of the imaging lens, image distortion, etc., and correcting the detection value.

[0007] The technical means disclosed in the above-mentioned JP 2017-32356 A and the like includes an image processing device that calculates distance information to the same object based on the disparity for the same object in a pair of image information acquired using a stereo camera device, and a map information acquisition means that acquires map information, and is a technology that compares a first inter-object distance for two stationary objects calculated based on the image information with a second inter-object distance for the two stationary objects calculated based on the map information, and corrects the disparity so that the two match based on the second inter-object distance. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2017-32356 A Summary of the Invention [Problem to be solved by the invention]

[0009] However, the technical means disclosed in the above-mentioned JP 2017-32356 A and the like are configured to utilize map information, and therefore require an acquisition means for acquiring the map information. In addition, there is a problem in that the information processing becomes complicated, involving calculation processing based on image information, calculation processing based on map information, and comparison processing for comparing the two, resulting in a large load on the information processing.

[0010] The present invention has been made in consideration of the above-mentioned points, and an object of the present invention is to provide a vehicle driving control device equipped with a surrounding environment recognition device, which can reliably calibrate a stereo camera device that recognizes the environment ahead in the direction of travel with a simple configuration, can always obtain three-dimensional image information including accurate distance information, and can ensure higher safety and reliability. [Means for solving the problem]

[0011] In order to achieve the above object, a vehicle driving control device according to one aspect of the present invention is a vehicle driving control device including at least a stereo camera device that recognizes a predetermined range of a region in front of the vehicle as an image and acquires three-dimensional image information including distance information based on the acquired image data, a LiDAR device that scans a region in front of the vehicle within approximately the same range as the recognition range of the stereo camera device and acquires three-dimensional point cloud data including distance information for each of a plurality of detection points, and a control unit that controls the stereo camera device and the LiDAR device, and the vehicle driving control device is configured to detect a positional deviation of the three-dimensional image information acquired by the stereo camera device based on the three-dimensional point cloud data acquired by the LiDAR device. is interpolated for each pixel. correct Then, the position error correction process is performed. A camera calibration method for calibrating the stereo camera device a storage memory that sequentially accumulates and stores the correction data for each pixel generated when the position deviation correction process is performed by the camera calibration unit, Ingredients The camera calibration unit generates a correction map corresponding to a recognition range of the stereo camera device based on the correction data stored in the storage memory. do. Effect of the Invention

[0012] According to the present invention, a driving control device for a vehicle equipped with a surrounding environment recognition device can be provided, which can reliably calibrate a stereo camera device that recognizes the environment ahead in the direction of travel with a simple configuration, can always obtain three-dimensional image information including accurate distance information, and can ensure higher safety and reliability. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a vehicle cruise control device according to an embodiment of the present invention; [Diagram 2] A conceptual diagram explaining the principle of distance measurement, which calculates the distance to an object based on parallax information, in the stereo camera device included in the driving control device of FIG. [Diagram 3] A conceptual diagram showing point cloud data detected by a LIDAR device included in the cruise control device of FIG. 1. [Figure 4] A conceptual diagram showing a positional shift of the point cloud data frame of the stereo camera device relative to the point cloud data frame of the LIDAR device when a mechanical positional shift occurs in the stereo camera device in the driving control device of FIG. [Diagram 5] A conceptual diagram showing a positional shift of the point cloud data frame of the stereo camera device relative to the point cloud data frame of the LIDAR device when an imaging position shift occurs due to the optical performance of the imaging lens of the stereo camera device in the driving control device of FIG. [Figure 6] A conceptual diagram showing the positional shift of the point cloud data frame of the stereo camera device relative to the point cloud data frame of the lidar device when an imaging position shift occurs in the stereo camera device in the driving control device of FIG. 1 due to the windshield. [Figure 7] 2 is a diagram for explaining the concept of position shift correction (calibration process) when an image formation position shift occurs in the stereo camera device of the driving control device of FIG. 1. [Figure 8]FIG. 2 is a diagram showing a specific example in which point cloud data from a corresponding LIDAR device is superimposed on an image captured by a stereo camera device of the cruise control device in FIG. 1; [Figure 9] A flowchart showing an operation of calibrating a stereo camera device based on data acquired by a lidar device, among the operations of the cruise control device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The present invention will be described below with reference to the illustrated embodiments. The drawings used in the following description are schematic, and the dimensional relationship and scale of each component may be different for each component in order to show each component at a size that can be recognized on the drawing. Therefore, the present invention is not limited to the illustrated form with respect to the quantity of each component, the shape of each component, the size ratio of each component, the relative positional relationship of each component, etc. shown in each drawing.

[0015] A cruise control device according to one embodiment of the present invention is a device that is mounted on a vehicle such as an automobile and executes cruise control to assist the driving operation by a driver of the vehicle.

[0016] Generally, in order to realize a highly advanced autonomous driving control of, for example, level 3 or higher in a vehicle driving control device, it is necessary to ensure higher safety and reliability while taking into consideration device failures, etc. For this reason, it is necessary to consider preventing the occurrence of false detection and malfunction by configuring an autonomous sensor device as a surrounding environment recognition device that reliably detects the surrounding environment of the vehicle and always obtains accurate surrounding environment information, for example, as a dual system.

[0017] One example of such a dual-system surrounding environment recognition device (autonomous sensor device) that can be applied is a stereo camera device that serves as a passive sensor device (passive sensor device) that recognizes objects within a specified range in the area in front of the vehicle as three-dimensional images and acquires three-dimensional image information (three-dimensional point cloud data) including distance information to the objects within the specified range.

[0018] Furthermore, in recent years, the adoption of a LiDAR (Light Detection And Ranging) device, which is a type of laser sensor device, has come to be considered as another type of surrounding environment recognition device (autonomous sensor device).

[0019] This LIDAR device is a next-generation laser sensor device that, for example, emits a laser pulse to scan a predetermined range ahead and receives return pulses reflected from forward objects within the predetermined range to obtain detailed three-dimensional point cloud data corresponding to the forward objects contained within the predetermined range in the forward region, thereby obtaining three-dimensional point cloud data including highly accurate distance information.In this way, since the LIDAR device is an active sensor device, it can function as an alternative sensor to passive sensor devices such as a stereo camera device.

[0020] For example, a stereo camera device, which is a passive sensor device, tends to be unable to acquire clear image information in dark places, etc., and the distance measurement accuracy tends to decrease. On the other hand, a LIDAR device, which is an active sensor device, can always perform accurate distance measurement without a decrease in distance measurement accuracy even in dark places, etc. Therefore, when realizing a high level of autonomous driving control, detection is mainly performed using a LIDAR device (active sensor device) in the surrounding environment such as a dark place. In this way, a dual system surrounding environment recognition device is configured to recognize the forward area using two sensor devices (a stereo camera device and a LIDAR device) that can acquire three-dimensional point cloud data.

[0021] In the vehicle driving control device of this embodiment, in order to realize more advanced autonomous driving control, a surrounding environment recognition device for acquiring three-dimensional point cloud data including distance information is configured as a dual system, and its component units are configured by combining, for example, a stereo camera device, which is a passive sensor device, and a LIDAR device, which is an active sensor device.

[0022] That is, the driving control device of this embodiment acquires information about the surrounding environment of the vehicle using various sensors such as a stereo camera device and a LIDAR device. In this case, these various sensors are configured as autonomous sensor devices that operate autonomously. The stereo camera device and the LIDAR device are set to recognize approximately the same range in the area in front of the vehicle, forming a dual-system surrounding environment recognition device.

[0023] The information on the surrounding environment of the vehicle refers to information on various objects present around the vehicle, such as other vehicles, including a preceding vehicle, a following vehicle, an oncoming vehicle, and a vehicle traveling alongside the vehicle, moving objects, such as pedestrians and bicycles, or fixed objects, including buildings and various structures. Hereinafter, information on the surrounding environment of the vehicle will be referred to as surrounding environment information, etc.

[0024] The driving control device of this embodiment appropriately uses surrounding environment information acquired using various sensors including a stereo camera device and a LIDAR device as information for performing driving control to assist the driving operation of the vehicle driver.

[0025] Incidentally, in a vehicle driving control device to which a stereo camera device is applied, the mounting position may deviate from the correct position due to, for example, the mounting accuracy when mounting the stereo camera device to the vehicle, aging, etc. In addition, in the stereo camera device itself, the mounting positions of the imaging element, imaging lens, etc. fixed inside the device may deviate from the correct position due to the assembly accuracy during manufacturing, aging, etc. In this way, when a mechanical positional deviation occurs in the stereo camera device and the imaging position of the optical image is shifted, distortion occurs in the optical image (mechanical imaging position deviation).

[0026] Furthermore, in addition to the image formation position shift caused by these mechanical factors, in a stereo camera device, a shift in the image formation position of an optical image due to the optical performance of the imaging lens, etc., may cause distortion of the optical image (optical image formation position shift). It is well known that the distortion of the optical image in this case tends to appear more as the imaging lens has a wider angle, and tends to appear more in the peripheral parts of the image.

[0027] In addition, a stereo camera device applied to a vehicle driving control device is generally attached and fixed to a fixed portion inside the vehicle cabin, and in this case, the stereo camera device is configured to obtain image information of the forward area through, for example, the windshield of the vehicle.

[0028] In general, the windshield used in a vehicle is not a uniform flat glass, but is formed in a complex shape with different curvatures depending on the part. In addition, the windshield may have glass distortion in some parts due to manufacturing reasons. Therefore, when light rays from an object in front pass through the windshield, different refraction occurs depending on the part, which may cause the imaging position of the optical image formed on the imaging surface to deviate from the normal position. As a result, the optical image may be distorted (optical imaging position deviation).

[0029] In this way, in the case where a stereo camera device is used as a surrounding environment recognition device in a vehicle driving control device, if image distortion occurs in the optical image of the target due to an imaging position shift caused by the windshield in addition to the imaging position shift caused by the mechanical factors described above and the imaging position shift caused by the optical performance of the imaging lens, etc., the accuracy of the distance information to the target obtained based on the optical image will be reduced.

[0030] In addition, in a stereo camera device applied to a vehicle driving control device, it is desired to obtain image information of a wide range of the area in front of the vehicle, so that the imaging lens used in the stereo camera device tends to have a wider angle.

[0031] However, in general, when the imaging lens of a stereo camera device is used at a wide angle, there is a tendency for distortion of the optical image, particularly in the peripheral area, to occur within the image area captured by the imaging lens, and such distortion of the optical image can cause a decrease in distance measurement accuracy.

[0032] As described above, in a stereo camera device, distortion of an optical image due to various factors reduces the distance measurement accuracy, which may cause a malfunction in the vehicle driving control that is executed using image information including distance information acquired by the stereo camera device.

[0033] This type of distortion of an optical image can be corrected by performing a predetermined signal processing on the image data, but performing such a correction process each time would place a large load on the processing circuitry and the like.

[0034] On the other hand, when the imaging lens of a stereo camera device is made wider in angle, the resolution of distant objects decreases, and the image tends to become unclear. Therefore, in a camera unit using a wide-angle imaging lens, it is necessary to increase the pixel count of the imaging element in order to form a clearer image of distant objects. Increasing the pixel count of the imaging element tends to increase the amount of image data.

[0035] Therefore, when the imaging lens in a stereo camera device is made wider in angle, in addition to increasing the load of signal processing related to distortion correction of the optical image, the load of data processing tends to increase due to the increase in the amount of image data to be handled.

[0036] On the other hand, LIDAR devices have limitations in terms of high-speed scanning due to the structural characteristics of the device, which scans a given area using mechanical drive while emitting laser pulses.The sampling rate of typical LIDAR devices is usually around 100 milliseconds (10 Hz).

[0037] In contrast, stereo camera devices that have a large number of pixels (for example, more than 2 million pixels) and can easily capture images at high speeds of about 120 frames per second (fps) are now widely available.

[0038] Considering these factors, there is a demand for the stereo camera device to be used as a sensor device that can always quickly calculate the distance to an object in order to quickly respond to rapidly changing environmental situations such as a pedestrian suddenly jumping out, another vehicle cutting in, etc. To achieve this, there is a strong demand for the stereo camera device to always maintain its function as a high-speed and highly accurate distance measuring sensor device.

[0039] Therefore, in order to minimize signal processing and the like that can cause these overloads, the driving control device of this embodiment is configured to automatically perform calibration processing of the stereo camera device based on three-dimensional point cloud data including distance information acquired by a LIDAR device, which is a laser sensor device, when the load on the driving control device is low, the vehicle is in a stable state, and there are few changes in the surrounding environment.

[0040] A schematic configuration of a cruise control device according to one embodiment of the present invention will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing a schematic configuration of a cruise control device for a vehicle according to one embodiment of the present invention.

[0041] The cruise control device 1 of this embodiment is a device mounted on a vehicle M such as an automobile. The cruise control device 1 basically has a configuration substantially similar to that of a conventional cruise control device of the same type. Therefore, in FIG. 1, only the configurations of the components of the cruise control device 1 of this embodiment that are directly related to the present invention are illustrated, and the illustration of the components that are not directly related to the present invention is omitted. In the following description, the components other than those directly related to the present invention are assumed to be substantially similar to those of the conventional cruise control device, and detailed description of only the components directly related to the present invention is omitted.

[0042] As shown in FIG. 1, the driving control device 1 of this embodiment is configured to include at least a stereo camera device 11 as an autonomous sensor device provided to recognize the surrounding environment of the vehicle M, a LIDAR device 12, etc., and a control unit 10 that comprehensively controls the entire driving control device 1, etc.

[0043] As various sensors (autonomous sensor devices) for recognizing the surrounding environment of the vehicle M, in addition to the above-mentioned stereo camera device 11 and the LIDAR device 12, for example, a longitudinal acceleration sensor for detecting the longitudinal acceleration of the vehicle itself, a wheel speed sensor for detecting the rotational speed of each of the front, rear, left and right wheels, a gyro sensor for detecting the angular velocity or angular acceleration of the vehicle itself, a GNSS (Global Navigation Satellite System) receiver for receiving positioning signals transmitted from a plurality of positioning satellites, and the like are provided, though not shown in the figure. These various sensors are connected to, for example, a control unit 10. The control unit 10 receives outputs from these various sensors and appropriately executes a predetermined driving control via a predetermined control unit (not shown).

[0044] The control unit 10 is configured by a well-known microcomputer including, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), a non-volatile storage unit, and its peripheral devices, etc. The ROM stores in advance programs executed by the CPU, fixed data such as data tables, and the like.

[0045] The stereo camera device 11 is a passive sensor device that recognizes a predetermined range in the area ahead of the vehicle M as a three-dimensional image and acquires three-dimensional image information including distance information of an object included in the three-dimensional image. The stereo camera device 11 is connected to the control unit 10. As a result, the stereo camera device 11 is driven and controlled by the control unit 10, and outputs acquired data in a predetermined format to the control unit 10.

[0046] The stereo camera device 11 is configured to include a camera unit including a pair of left and right cameras, a main camera 11a and a sub-camera 11b, which are fixed to the upper center of the front part of the interior of the vehicle M, for example.

[0047] The main camera 11a and the sub-camera 11b in the stereo camera device 11 are, for example, autonomous sensor devices that sense the real space ahead of the vehicle M and acquire image information of the surrounding driving environment mainly ahead of the vehicle. For this purpose, the two cameras 11a and 11b are each configured to have an imaging lens (see reference numerals 101a and 101b in FIG. 2), an imaging element (see reference numerals 102a and 102b in FIG. 2), and the like.

[0048] Each camera (11a, 11b) itself has a general configuration in which an optical image of an object is formed by an imaging lens (101a, 101b), the optical image is focused on an imaging surface of an imaging element (102a, 102b), and a pair of image data corresponding to the object is obtained by performing a predetermined photoelectric conversion process using the imaging element (102a, 102b). Therefore, detailed configurations and detailed descriptions of each camera 11a, 11b are omitted.

[0049] The main camera 11a and the sub camera 11b in the stereo camera device 11 are disposed at symmetrical positions on either side of the center in the vehicle width direction. As a result, the stereo camera device 11 acquires a pair of left and right image data captured by the two cameras 11a and 11b of a predetermined area in front of the vehicle M from different viewpoints. The pair of image data is transmitted to an image processing unit (IPU) 10b provided in the control unit 10.

[0050] Here, the symbol R in Fig. 1 conceptually indicates the horizontal imaging range of the main camera 11a which is the right camera unit in the stereo camera device 11. The symbol L in Fig. 1 conceptually indicates the horizontal imaging range of the sub-camera 11b which is the left camera unit in the stereo camera device 11. And the symbol S in Fig. 1 conceptually indicates the horizontal imaging range of the pair of left and right cameras (11a, 11b) in the stereo camera device 11.

[0051] The IPU 10b is composed of a processing circuit and the like that receives a pair of left and right image data acquired by the two cameras 11a and 11b and performs a predetermined image data processing based on the pair of image data. In the IPU 10b, for example, stereo image information (three-dimensional image information) is generated based on the pair of left and right image data acquired by the two cameras 11a and 11b, and for the same object captured in each of the pair of left and right image data, a positional deviation amount (parallax information) occurring between the two images is acquired, and three-dimensional image information (three-dimensional point cloud data; 3D point cloud data) including distance information from the stereo camera device 11 to the object is generated based on the parallax information and the like. The three-dimensional image information and the like generated in this way are transmitted to the image recognition unit 10c, which is an image recognition device provided in the control unit 10.

[0052] The image recognition unit 10c receives the three-dimensional image information and the like transmitted from the IPU 10b, and recognizes, for example, the surface condition of the road on which the vehicle M is traveling (hereinafter referred to as the road surface condition) based on the three-dimensional image information and the like. The image recognition unit 10c then recognizes the lane markings dividing the left and right of the driving lane of the road on which the vehicle M is traveling, and obtains various information such as the road curvature [1 / m] of each of the left and right lane markings and the width between the left and right markings (lane width).

[0053] There are various known methods for calculating road curvature and lane width. For example, a binarization process based on brightness differences in image information is performed to recognize the left and right dividing lines, and the curvature of the left and right dividing lines is then calculated for each specified section using a curve approximation formula based on the least squares method.

[0054] Furthermore, the image recognition unit 10c performs predetermined pattern matching based on the three-dimensional image information to recognize guardrails, curbs, and the like along the road, as well as three-dimensional objects such as pedestrians, motorcycles, and other vehicles other than motorcycles that are present in the forward area on the road on which the vehicle M is traveling. Specifically, the image recognition unit 10c recognizes various information such as the type of three-dimensional object, the distance to the three-dimensional object, the moving speed of the three-dimensional object, and the relative speed between the three-dimensional object and the vehicle M.

[0055] Here, the distance measurement principle for calculating the distance to a forward object based on parallax information by the stereo camera device 11 in the driving control device 1 of this embodiment will be described with reference to Fig. 2. Fig. 2 is a conceptual diagram for explaining the distance measurement principle for calculating the distance to an object based on parallax information in the stereo camera device included in the driving control device of this embodiment.

[0056] 2, the stereo camera device 11 in the driving control device 1 of this embodiment is configured with a pair of left and right cameras (11a, 11b). Each of the cameras (11a, 11b) includes an imaging lens (101a, 101b) and an imaging element (102a, 102b).

[0057] In this case, the cameras (11a, 11b) in the stereo camera device 11 are arranged such that the optical axes (Oa, Ob) of the imaging lenses (101a, 101b) are spaced a predetermined distance apart in the horizontal direction (see symbol B in FIG. 2) and are substantially parallel to each other. Here, the horizontal distance (symbol B in FIG. 2) between the optical axes (Oa, Ob) of the two cameras (11a, 11b) is referred to as the baseline length of the stereo camera device 11.

[0058] 2, the focal length of each of the imaging lenses (101a, 101b) is indicated by the symbol F. Here, each of the imaging elements (102a, 102b) is disposed in a position where the imaging surface is parallel to a plane perpendicular to the optical axis (Oa, Ob) and coincides with the focal length F. Each of the imaging lenses (101a, 101b) is disposed such that the optical axis (Oa, Ob) is located at the center of the imaging surface of each of the imaging elements (102a, 102b).

[0059] It is assumed that an object Obj exists at a position a predetermined distance Z away in front of the stereo camera device 11 configured as above. At this time, reflected light from the object Obj passes through each imaging lens (101a, 101b) and forms an image at each predetermined position on the imaging surface of each imaging element (102a, 102b) that is a focal length F away. As a result, an optical image of the object Obj is formed on the imaging surface of each imaging element (102a, 102b).

[0060] In the example shown in Figure 2, the optical image of the object Obj is formed at the position indicated by the symbol Obj1 on the imaging surface of the imaging element 102a of the main camera 11a, and at the position indicated by the symbol Obj2 on the imaging surface of the imaging element 102b of the sub-camera 11b.

[0061] In this case, the position of the object image Obj2 formed on the imaging surface of the imaging element 102b of the sub-camera 11b relative to the imaging surface of the imaging element 102a of the main camera 11a is indicated by the position indicated by the symbol Obj2'.

[0062] Here, the symbol Ga in Fig. 2 is an example of an image including an object image Obj1 formed on the imaging surface of the imaging element 102a of the main camera 11a, and the symbol Gb in Fig. 2 is an example of an image including an object image Obj2 formed on the imaging surface of the imaging element 102b of the sub-camera 11b.

[0063] When the image Gb of the sub-camera 11b is superimposed on the image Ga of the main camera 11a, the image is as shown by the symbol Gab in Fig. 2. In this image Gab, the object image Obj1 of the main camera 11a and the object image Obj2 of the sub-camera 11b are shown separated by a distance D in the horizontal direction. This distance D is the parallax between the two cameras (11a, 11b). Note that the parallax between the two cameras (11a, 11b) tends to become larger as the distance to the object becomes shorter, and to become smaller as the distance to the object becomes longer.

[0064] In this case, when the base line length is B, the focal length is F, and the parallax is D, the distance Z from the stereo camera device 11 to the object Obj is expressed by the following formula (1): Z = B × F / D … (1) It can be calculated by:

[0065] In the stereo camera device 11, a stereo matching process is performed on two images (Ga, Gb) captured by two cameras (11a, 11b) to estimate the disparity of the same object for each corresponding region. Then, the stereo camera device 11 obtains the disparity distribution for the entire image to generate a disparity image.

[0066] The stereo camera device 11 obtains distance information for each pixel using the above-mentioned formula (1) based on the parallax image generated as described above. This allows the stereo camera device 11 to obtain three-dimensional point cloud data including distance information for each pixel.

[0067] On the other hand, the LIDAR device 12, which is a type of laser sensor device, is an active sensor device that acquires three-dimensional point cloud data including highly accurate distance information to objects included within a predetermined range in the area ahead of the vehicle M. The LIDAR device 12 is connected to the control unit 10. As a result, the LIDAR device 12 is driven and controlled by the control unit 10, and outputs acquired data in a predetermined format to the control unit 10.

[0068] The LIDAR device 12, for example, scans a predetermined range (see symbol T in FIG. 1) while irradiating a laser pulse forward of the vehicle M and receives a return pulse reflected from the forward object, thereby acquiring detailed three-dimensional point cloud data corresponding to the forward object within the predetermined range in the area forward of the vehicle M. Then, the three-dimensional point cloud data acquired by the LIDAR device 12 is sent to the control unit 10 as described above. In response to this, the control unit 10 recognizes various moving objects such as pedestrians, bicycles, and the like, as well as other vehicles and the like present in front of the vehicle M, as three-dimensional objects.

[0069] Here, FIG. 3 is a conceptual diagram showing point cloud data detected by a LIDAR device included in the driving control device of this embodiment.

[0070] The LIDAR device 12 scans a predetermined range (see symbol T in FIG. 1) in the area in front of the vehicle M as, for example, a rectangular point cloud data frame GL as shown in FIG. 3. That is, the LIDAR device 12 scans the predetermined range (point cloud data frame GL) by, for example, a raster scan method while emitting a laser pulse. In this way, point cloud data on three-dimensional objects in the predetermined range (point cloud data frame GL) is acquired. The example shown in FIG. 3 conceptually represents point cloud data in which each intersection point of a lattice is a detection point.

[0071] In the driving control device 1 of this embodiment, as shown in FIG. 1, the forward recognition range by the stereo camera device 11 (see symbol S in FIG. 1) and the forward recognition range by the LIDAR device 12 (see symbol T in FIG. 1) are set to be approximately the same range.

[0072] As described above, both the stereo camera device 11 and the LIDAR device 12 acquire three-dimensional point cloud data including distance information to a forward object. Therefore, in the driving control device 1 of this embodiment, the stereo camera device 11 (passive sensor device) and the LIDAR device 12 (active sensor device) are combined to configure a dual-system surrounding environment recognition device.

[0073] On the other hand, the control unit 10 performs, for example, drive control of the stereo camera device 11 and the LIDAR device 12, receives output data (such as three-dimensional point cloud data) as detection results from each device 11, 12, and stores it in a specified storage memory (not shown) as appropriate. At the same time, it performs specified signal processing to contribute to various driving controls, and sends the output data as a result of the processing to a specified component unit (not shown) as appropriate.

[0074] The control unit 10 also includes a camera calibration unit 10a, an image processing unit (IPU) 10b, an image recognition unit 10c, and the like (see FIG. 1). The camera configuration unit 10a includes a processing circuit and the like that performs a calibration process on the output data of the stereo camera device 11. For example, the camera calibration unit 10a performs a calibration process to match the three-dimensional point cloud data acquired by the stereo camera device 11 with the three-dimensional point cloud data acquired by the LIDAR device 12 as a reference. The IPU 10b is an image processing unit that includes a processing circuit and the like that receives and appropriately processes a pair of left and right image data acquired by the stereo camera device 11 (two cameras 11a, 11b). The image recognition unit 10c includes a processing circuit and the like that receives the processed image data output from the IPU 10c and performs a predetermined image recognition process and the like.

[0075] As described above, the LIDAR device 12 can always obtain 3D point cloud data including highly accurate distance information. In contrast, the stereo camera device 11 has a possibility that errors may occur in the obtained distance information due to the above-mentioned mechanical image formation position shift or optical image formation position shift.

[0076] For example, Figures 4 to 6 are diagrams conceptually showing the positional shift of the point cloud data frame of the stereo camera device relative to the point cloud data frame of the LIDAR device when the stereo camera device in the driving control device of this embodiment causes an imaging position shift due to the influence of various factors.

[0077] 4 to 6, the symbol GL indicates a point cloud data frame (predetermined scanning range frame) acquired by the LIDAR device 12. Also, in Fig. 4 to 6, the symbol GS (indicated by a dotted line) indicates a point cloud data frame (predetermined image range frame) obtained based on the parallax image generated by the stereo camera device 11.

[0078] Specifically, the mechanical positional deviation in the stereo camera device 11 includes, for example, a positional deviation in the installation position caused by an inaccuracy in mounting the stereo camera device 11 on the vehicle M or a positional deviation in the mounting positions of components inside the device (such as an imaging element and an imaging lens) caused by an inaccuracy in assembly or a change over time during the manufacture of the stereo camera device 11. This type of mechanical positional deviation in the stereo camera device 11 generates a translational deviation in the up / down and left / right directions of the image frame and a rotational deviation around the optical axis.

[0079] Fig. 4 conceptually illustrates the positional shift of the point cloud data frame GS of the stereo camera device relative to the point cloud data frame GL of the LIDAR device when a mechanical image formation position shift occurs in the stereo camera device 11. Note that Fig. 4 illustrates an example in which translational shift and rotational shift of the image frame occur simultaneously.

[0080] Moreover, one type of optical imaging position shift in the stereo camera device 11 is an imaging position shift caused by, for example, the optical performance of the imaging lens. Fig. 5 conceptually shows the position shift of the point cloud data frame GS of the stereo camera device relative to the point cloud data frame GL of the LIDAR device when an optical imaging position shift occurs due to the imaging lens of the stereo camera device 11. Note that Fig. 5 illustrates a so-called barrel distortion.

[0081] Another type of optical imaging position shift in the stereo camera device 11 is an imaging position shift caused by the influence of, for example, the windshield of a vehicle. Figure 6 conceptually shows the position shift of the point cloud data frame GS of the stereo camera device relative to the point cloud data frame GL of the LIDAR device when an imaging position shift occurs due to the windshield placed in front of the stereo camera device 11.

[0082] Therefore, as described above, the camera calibration unit 10a of the control unit 10 in the driving control device 1 of this embodiment performs a calibration process of the data (point cloud data) acquired by the stereo camera device 11 based on the data (point cloud data) acquired by the LIDAR device 12. In order to perform this calibration process, information on the positional relationship between the stereo camera device 11 and the LIDAR device 12 in the vehicle is required. The information on the positional relationship between the stereo camera device 11 and the LIDAR device 12 in the vehicle is fixed information for each vehicle. The information is assumed to be stored in advance in a predetermined storage unit (for example, a storage unit not shown in the control unit 10) or the like. Then, the camera configuration unit 10a appropriately reads out and uses the necessary information from the storage unit or the like as necessary.

[0083] The camera calibration unit 10a performs a position correction process for each detection point of the point cloud data acquired by the LIDAR device 12 so as to match the corresponding point of the point cloud data acquired by the stereo camera device 11.

[0084] In this case, the camera calibration unit 10a acquires the amount of positional deviation between each detection point of the point cloud data by the LIDAR device 12 and each corresponding point of the point cloud data by the stereo camera device 11 over the entire overlapping range in the forward recognition range (see symbols S and T in FIG. 1) of both devices (11, 12). That is, the camera calibration unit 10a acquires the amount of positional deviation between each detection point in the point cloud data frame GL of the LIDAR device 12 and the corresponding point in the point cloud data frame GS of the stereo camera device corresponding to each detection point.

[0085] 7 is a diagram for explaining the concept of position shift correction (calibration process) when an imaging position shift occurs in the stereo camera device 11 of the driving control device 1 of this embodiment. The example shown in FIG. 7 shows that a windshield 110 of the vehicle M is disposed in front of the stereo camera device 11. Note that the configuration shown in FIG. 7 other than the windshield 110 is exactly the same as the configuration in FIG. 2.

[0086] Here, an optical image of the object Obj is formed at a position indicated by symbol Obj1'' on the imaging surface of the imaging element 102a of the main camera 11a, and at a position indicated by symbol Obj2'' on the imaging surface of the imaging element 102b of the sub-camera 11b. In this case, each of the imaging positions Obj1'' and Obj2'' is mainly a result of being affected by the windshield 110 (such as refraction or reflection when passing through the glass). In FIG. 7, the light rays after passing through the windshield 110 are indicated by two-dot chain lines.

[0087] On the other hand, in the configuration of Fig. 7, if we consider the image formation position when the windshield 110 does not exist, the optical image of the object Obj should be formed at the position indicated by the symbol Obj1 on the imaging surface of the imaging element 102a of the main camera 11a, and at the position indicated by the symbol Obj2 on the imaging surface of the imaging element 102b of the sub-camera 11b, as explained in Fig. 2. The light rays from the object Obj at this time are indicated by solid lines.

[0088] Here, the distance information obtained from the parallax D (see Figure 2) between the imaging positions Obj1 and Obj2 where the images are supposed to be formed can be estimated to be approximately the same as the distance information of the corresponding detection points in the point cloud data obtained by the LIDAR device 12 corresponding to the imaging positions Obj1 and Obj2.

[0089] Therefore, in the main camera 11a, the horizontal distance S1 between the imaging position Obj1 (the detection point of the corresponding LIDAR device 12) where the image is supposed to be formed and the imaging position Obj1'' resulting from the influence of the windshield 110, etc. is obtained as the imaging position shift amount. Similarly, in the sub-camera 11b, the distance S2 between the imaging position Obj2 (the detection point of the corresponding LIDAR device 12) where the image is supposed to be formed and the imaging position Obj2'' resulting from the influence of the windshield 110, etc. is obtained as the imaging position shift amount. Also, at this time, the directions indicated by the symbols X1 and X2 in FIG. 7 are obtained as the respective imaging position shift directions.

[0090] In this case, each imaging position can be specified by an XY coordinate system in which the horizontal axis is the X axis and the vertical axis is the Y axis, with the center point of each imaging element 102a, 102b as the origin, and therefore the imaging position shift amounts S1, S2 and the imaging position shift directions X1, X2 can be expressed by the XY coordinate system. In this manner, the imaging position shift amounts S1, S2 and the imaging position shift directions X1, X2 caused by the influence of the windshield 110, etc. can be obtained.

[0091] The camera calibration unit 10a performs position correction using the information on the imaging position shift amounts S1, S2 and the imaging position shift directions X1, X2 obtained as described above so that the imaging positions Obj1'', Obj2'', when influenced by the windshield 110, etc., coincide with the imaging positions Obj1, Obj2 (corresponding detection points of the LIDAR device 12) where the images are originally supposed to be formed. The stereo camera device 11 can be calibrated by such a procedure.

[0092] In the above description, an example is given in which image data acquired by the stereo camera device 11 is affected by the windshield 110, causing an imaging position shift. However, as described above, the distortion of the optical image in the stereo camera device 11 may also occur due to, for example, the optical performance of the imaging lenses (101a, 101b). The imaging position shift and the imaging position shift direction in such a case can also be found from the imaging position shift information in exactly the same way as described above.

[0093] In other words, even if the influence of the windshield 110 and the influence of the imaging lens overlap to cause an imaging position shift, the amount and direction of the imaging position shift can be determined from the imaging position shift between the original imaging position and the imaging position when the imaging position is shifted due to optical distortion.

[0094] Incidentally, a plurality of points (detection points) forming the point cloud data of the LIDAR device 12 corresponds to each intersection of a lattice in a point cloud data frame GL as shown in Fig. 3. In contrast, a plurality of points forming the point cloud data of the stereo camera device 11 corresponds to a plurality of pixels arranged on the imaging surface of an imaging element.

[0095] For this reason, the detection points (pixels) on the imaging surface of the stereo camera device 11 are arranged more finely than the multiple detection points within the scanning range (point cloud data frame GL) of the LIDAR device 12. Therefore, there is no one-to-one correspondence between the detection points of the LIDAR device 12 and the pixels of the stereo camera device 11.

[0096] Taking this into consideration, the amount of image position shift in each pixel (point cloud) in the intermediate region between the corresponding points of the stereo camera device 11 corresponding to each detection point of the LIDAR device 12 may be calculated by, for example, linear interpolation processing, etc. That is, for the internal region of a substantially square formed by connecting four lattice points in the LIDAR device 12, the amount of image position shift corresponding to each pixel is calculated based on the four sides having the four lattice points as the four corners.

[0097] In addition, the camera calibration unit 10a does not always perform the calibration process of the stereo camera device 11. For example, since the LIDAR device 12 performs sequential scanning, it may not be possible to obtain a stable distance measurement result for a moving object that moves at high speed.

[0098] In other words, since the LIDAR device 12 sequentially scans within a specified recognition range while emitting laser pulses, it may not be possible to obtain accurate distance information for a specific object in an environment where the vehicle M equipped with the driving control device 1 is moving at high speed, there are many moving objects in the vicinity, or the object itself is moving at high speed.

[0099] For this reason, in the cruise control device 1 of this embodiment, the calibration process is executed only when the vehicle M is in a predetermined state. Here, the predetermined state of the vehicle M when the cruise control device 1 of this embodiment executes the calibration process is preferably, for example, a state in which the load on the cruise control device 1 of this embodiment is small, the vehicle M equipped with the cruise control device 1 is in a stable state, and there is little change in the surrounding environment.

[0100] For example, when the cruise control device 1 is activated and the vehicle M is stopped, the load on the cruise control device 1 is small, and the vehicle M is in a stable state. In such a case, it is more desirable if there is little change in the surrounding environment. Therefore, specifically, the predetermined calibration process may be executed immediately after starting the engine of the parked vehicle M, or before stopping the engine after the traveling vehicle M is stopped in a predetermined parking space or the like. In addition, in the cruise control device 1 of this embodiment, it is considered that the predetermined calibration process is not executed at least while the vehicle M is traveling at high speed.

[0101] Here, FIG. 8 shows an example in which point cloud data from a corresponding LIDAR device is superimposed on an image displayed based on image data acquired by the stereo camera device 11 when the driving control device 1 is activated and the vehicle M is stopped (for example, vehicle M immediately after the engine is started).

[0102] In Fig. 8, reference numeral 120 denotes an image capturing range frame by the stereo camera device 11 included in the driving control device 1 of this embodiment. Fig. 8 also shows a point cloud data detection frame 121 (indicated by a dotted line) by the LIDAR device 12, which has a scanning range that is approximately the same as the image capturing range frame 120 by the stereo camera device 11.

[0103] In the cruise control device 1 of this embodiment, when the calibration process of the stereo camera device 11 is performed, it is desirable that various objects are present at a predetermined distance in the forward area of ​​the vehicle M when the cruise control device 1 is in an activated state and the vehicle M equipped with the cruise control device 1 is in a stable stopped state. At this time, it is desirable that the forward object is a structure from which the stereo camera device 11 can reliably acquire disparity information points. It is also desirable that the forward objects include a mixture of a short-distance object (see symbol Obj21) that is stationary and located relatively close to the vehicle M, and a long-distance object Obj22 that is stationary and located far away.

[0104] In such a surrounding environment, it would be even more desirable if the LIDAR device 12 could stably and reliably acquire point cloud data of multiple detection points, and if stereo matching processing could be performed on the images acquired by the stereo camera device 11 for points corresponding to each detection point of the LIDAR device 12, and if distance information for the corresponding points could be acquired.

[0105] Under such circumstances, the driving control device 1 of this embodiment executes a calibration process for the stereo camera device 11 based on the point cloud data of the LIDAR device 12 as a reference.

[0106] However, the point cloud data acquired by the LIDAR device 12 does not always detect all detection points within the scanning range of the LIDAR device 12 even under predetermined constant conditions (for example, see the situation in FIG. 8).

[0107] For example, the LIDAR device 12 may have difficulty receiving the reflected light (return pulse) of the irradiated laser pulse from an object with low reflectivity. As a result, it may not be possible to obtain a detection result in an area where an object with such a predetermined condition (e.g., a condition with low reflectivity, etc.) exists within a predetermined scanning range. In addition, the LIDAR device 12 cannot receive reflected light from an area such as the sky, and therefore cannot obtain a detection result in such a case.

[0108] For this reason, when the LIDAR device 12 performs sequential scanning within a predetermined scanning range, undetectable points may occur in some areas within the predetermined scanning range.

[0109] Therefore, in the driving control device 1 of this embodiment, each time a calibration process is performed, positional deviation data regarding the amount and direction of positional deviation between the points detected by the LIDAR device 12 and the corresponding points detected by the stereo camera device 11 is accumulated, and a correction map for the entire specified scanning range is generated.

[0110] This correction map is stored in a predetermined storage memory (not shown) or the like of the specific vehicle M equipped with the driving control device 1 as information unique to the vehicle M. Then, the stereo camera device 11 refers to the generated correction map to acquire subsequent point cloud data.

[0111] The correction map generated as described above is updated whenever necessary by a calibration process that is performed as appropriate. The schematic configuration of the cruise control device 1 of this embodiment has been described above.

[0112] In the cruise control device 1 of this embodiment configured as described above, the operation of performing the calibration process of the stereo camera device based on the point cloud data acquired by the LIDAR device will be described below. Fig. 9 is a flowchart showing the operation related to the present invention (calibration process of the stereo camera device) among the operations of the cruise control device of one embodiment of the present invention.

[0113] The processing sequence shown in Fig. 9 is a process that is continuously executed at all times in parallel with various control processes of the driving control device 1 of this embodiment. That is, the processing sequence of Fig. 9 is started to be executed at the same time as the driving control device 1 operates, and is executed continuously while the driving control device 1 is operating. Then, the processing sequence is stopped at the same time as the operation of the driving control device 1 is stopped.

[0114] First, the vehicle M equipped with the cruise control device 1 of this embodiment is started, and the components such as the stereo camera device 11, the LIDAR device 12, the control unit 10, etc. start operating, and the vehicle M becomes ready to travel. This starts the flow chart of FIG.

[0115] First, in step S11 of FIG. 9, the control unit 10 performs a surrounding environment recognition process to drive and control various sensors such as the stereo camera device 11 and the LIDAR device 12, and recognize the surrounding environment including the area ahead of the vehicle through these various sensors.

[0116] Here, the stereo camera device 11 and the LIDAR device 12 recognize the surrounding environment such as various three-dimensional objects contained within a predetermined range mainly in the area ahead of the vehicle M. Here, the control unit 10 detects the state of the vehicle itself (for example, the vehicle speed and the type (mode) of cruise control that has been set). The surrounding environment recognition process carried out here is constantly performed while the cruise control device 1 is running.

[0117] Next, in step S12, the control unit 10 checks whether the vehicle M is in a stopped state. If it is confirmed that the vehicle M is in a stopped state, the process proceeds to step S13. If it is confirmed that the vehicle M is not in a stopped state, for example, if the vehicle M is in a running state, the process returns to step S11, and the subsequent process sequence is repeated.

[0118] In step S13, the control unit 10 generates point cloud data by the LIDAR device 12.

[0119] Next, in step S14, the control unit 10 generates point cloud data by the stereo camera device 11.

[0120] Here, the process of acquiring point cloud data by the stereo camera device 11 is roughly as follows. That is, the stereo camera device 11 first acquires a pair of left and right image data by the two cameras (11a, 11b). Next, a predetermined stereo matching process is performed based on the pair of image data to obtain disparity information of detection points on the same object. Based on this disparity information, point cloud data covering the entire image range is generated.

[0121] Next, in step S15, the control unit 10 compares the point cloud data acquired by the LIDAR device 12 with the point cloud data acquired by the stereo camera device 11, and checks for each detection point whether or not there is a positional deviation between corresponding positions of the same object (a position corresponding to a detection point of the LIDAR device 12 and a position of a corresponding point on the image of the stereo camera device 11 corresponding to this detection point). If a positional deviation is confirmed, the process proceeds to the next step S16. If a positional deviation is not confirmed, the process returns to the above-mentioned step S11, and the subsequent processes are repeated.

[0122] In step S16, the control unit 10 determines the amount and direction of positional deviation between the position corresponding to the detection point of the LIDAR device 12 and the corresponding point on the image of the stereo camera device 11 that corresponds to this detection point, and generates position correction data.

[0123] Next, in step S17, the control unit 10 corrects the point cloud data of the stereo camera device 11 based on the position correction data including the positional deviation amount and positional deviation direction obtained in the process of step S16 described above.

[0124] In step S18, the control unit 10 accumulates the position correction data obtained in the process of step S16 described above in a predetermined storage memory (not shown or an internal memory (not shown) of the control unit 10, etc.) and generates a correction map specific to the stereo camera device 11. The generated correction map is stored in a predetermined storage memory or the like (not shown). The stereo camera device 11 then corrects the acquired image data by referring to the correction map. The stereo camera device 11 then performs a process of calculating distance information, etc., using the correction data thus obtained. The correction map stored in the storage memory is updated as appropriate as a result of a calibration process that is executed at an appropriate predetermined timing each time the vehicle M on which the driving control device 1 is mounted is used.

[0125] As described above, according to the above embodiment, the vehicle driving control device is equipped with a dual-system surrounding environment recognition device that recognizes the environment ahead in the traveling direction by combining the stereo camera device 11 (passive sensor device) and the LIDAR device 12 (active sensor device), and is configured to include a camera calibration unit 10a that simply and reliably calibrates the stereo camera device 11 by correcting the positional deviation of the three-dimensional image information acquired by the stereo camera device 11 based on the three-dimensional point cloud data acquired by the LIDAR device 12.

[0126] In this case, the camera calibration unit 10a performs a position shift correction process for correcting the position shift of the three-dimensional image information of the stereo camera device 11 for each pixel, thereby enabling more precise calibration.

[0127] In addition, the correction data for each pixel generated when the camera calibration unit 10a performs the position deviation correction process is sequentially accumulated and stored in a storage memory (not shown) or in the control unit. The camera calibration unit 10a generates a correction map corresponding to the recognition range of the stereo camera device 11 based on the correction data accumulated and stored in the storage memory. This configuration allows more reliable calibration to be performed over the entire recognition range of the stereo camera device 11. Furthermore, the camera calibration unit 10a appropriately executes the position deviation correction process at a timing suitable for the process and accumulates the acquired correction data each time, which contributes to reducing the processing load and reduces the load on the control circuit in the driving control device 1.

[0128] Furthermore, the correction process by the camera calibration unit 10a is executed when the vehicle M equipped with the driving control device 1 is in a stable stopped state, so that reliable correction data can be obtained.

[0129] In this way, the stereo camera device 11 is calibrated based on the data acquired by the LIDAR device 12, so that the stereo camera device 11 can always acquire three-dimensional image information including accurate distance information. Therefore, the driving control device 1 of this embodiment can ensure higher safety and reliability.

[0130] The present invention is not limited to the above-mentioned embodiment, and various modifications and applications can be implemented within the scope of the gist of the invention. Furthermore, the above-mentioned embodiment includes inventions at various stages, and various inventions can be extracted by appropriate combinations of the disclosed multiple constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the above-mentioned one embodiment, if the problem to be solved by the invention can be solved and the effect of the invention can be obtained, the configuration from which the constituent elements are deleted can be extracted as the invention. Furthermore, the constituent elements across different embodiments may be appropriately combined. The present invention is not restricted by its specific embodiment except as limited by the attached claims. [Explanation of symbols]

[0131] 1...Drive control device 10…Control unit 10a...Camera calibration section 10b…Image Processing Unit (IPU) 10c...Image recognition unit (image recognition device) 11...Stereo camera device 11a…Main camera 11b…Sub camera 12... LiDAR device 101a, 101b...imaging lens 102a, 102b...imaging element 110…Windshield 120…Image capture range frame 121...Point cloud data detection frame B…baseline length D...Parallax F…focal length Z: Distance to the object GL: Lidar point cloud data frame GS: Point cloud data frame of stereo camera device M…Vehicle Obj…Object

Claims

1. a stereo camera device that recognizes a predetermined range in a forward area of ​​a vehicle as an image and acquires three-dimensional image information including distance information based on the acquired image data; A LiDAR device that scans a forward area of ​​the vehicle within approximately the same range as the recognition range of the stereo camera device to acquire three-dimensional point cloud data including distance information for each of a plurality of detection points; A control unit that controls the stereo camera device and the LIDAR device; A vehicle driving control device comprising at least a camera calibration unit that calibrates the stereo camera device by performing a position shift correction process to correct a position shift of the 3D image information acquired by the stereo camera device for each pixel based on the 3D point cloud data acquired by the LIDAR device; and a storage memory that sequentially accumulates and stores the correction data for each pixel generated when the position deviation correction process is performed by the camera calibration unit; Equipped with The vehicle driving control device according to claim 1, wherein the camera calibration unit generates a correction map corresponding to a recognition range of the stereo camera device based on the correction data accumulated and stored in the storage memory.

2. The vehicle driving control device according to claim 1 , wherein the correction process by the camera calibration unit is executed when the vehicle is in a stable stopped state.

Images