Sensing device and vehicle control device

Abstract

To provide a sensing device and a vehicle control device for highly-accurately estimating the relative postures between a road surface and a vehicle.SOLUTION: In a vehicle mounted with a sensing device, a vehicle on-board processing device 102 comprises: a first common image pickup region observation unit that observes a first region around a subject vehicle from information that is acquired by at least a first sensor and a second sensor having a common image pickup region and relates to the common image pickup region; a second common image pickup region observation unit that observes a second region, which is different from the first region, from information that is acquired by at least a third sensor and a fourth sensor having a common image pickup region and relates to the common image pickup region; a coordinate integration unit that integrates a geometric relationship among the sensors with coordinates of information observed in the first region and the second region; and a road surface estimation unit (road surface model matching unit) that, based on point cloud information calculated from the integrated coordinates, estimates relative postures for each of the sensors and a road surface, including a pitch angle and roll angle of the subject vehicle.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a sensing device and a vehicle control device. [Background technology]

[0002] In-vehicle cameras are becoming more and more common for driver assistance and autonomous driving, and in some cases, multiple monocular cameras are installed for surrounding monitoring. One of the requirements for in-vehicle cameras in driver assistance and autonomous driving is to measure the distance to an object. When measuring the distance to an object using a monocular camera, it is assumed that the road surface is flat, and the distance corresponding to each pixel of the in-vehicle camera can be calculated from the relationship between the installation state of the in-vehicle camera and the road surface. However, if the road surface has a slope, there is a problem that errors will occur in the measured distance if the calculation is made assuming a flat surface. In particular, the error becomes larger the further away the object is. Therefore, it is necessary to reduce the effect of the slope error.

[0003] Patent Document 1 discloses a gradient estimation device based on an image captured by a camera, the device comprising a camera, a road contact point calculation unit, a distance measurement sensor, and a gradient estimation unit, in which the road contact point calculation unit calculates a distance L2 to the road contact point of an object reflected in the captured image based on the image captured by the camera, the gradient estimation device uses the distance measurement sensor to calculate a distance L1 to the object, and the gradient estimation unit estimates a gradient β of a straight line passing through a predetermined point A1 indicating the object and an intersection point between a perpendicular line dropped from the camera position and a horizontal plane, based on the depression angle α of the camera, the distance L2, and the distance L1. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-165933 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, it is assumed that an active laser type distance measuring sensor is used, and if there is no sensor capable of measuring distance regardless of gradient, it is difficult to perform distance correction.

[0006] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide a sensing device that estimates the relative attitude of a vehicle relative to a road surface with high accuracy. [Means for solving the problem]

[0007] A representative example of the invention disclosed in the present application is as follows: That is, the vehicle control system includes a first common imaging area observation unit that observes a first area around the vehicle from information of the common imaging area acquired by at least a first sensor and a second sensor having a common imaging area, a second common imaging area observation unit that observes a second area different from the first area from information of the common imaging area acquired by at least a third sensor and a fourth sensor having a common imaging area, a coordinate integration unit that integrates a geometric relationship between the sensors and coordinates of information observed in the first area and the second area, and a road surface estimation unit that estimates a relative attitude between the sensors and the road surface, including a pitch angle and a roll angle of the vehicle, based on point cloud information calculated from the integrated coordinates. The common imaging area is disposed at a position where the optical axes of the sensors intersect. It is characterized by the above. Effect of the Invention

[0008] According to one aspect of the present invention, the influence of road gradients and vehicle attitudes can be reduced, and distance measurement accuracy can be improved. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of a vehicle equipped with a sensing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a relationship between a camera and a common imaging area. [Diagram 3] 1 is a diagram illustrating an example of gradient and vehicle attitude fluctuation. [Figure 4] FIG. 4 is a functional block diagram of a distance estimation program executed by a CPU. [Diagram 5] FIG. 2 is a schematic diagram showing the processes of coordinate integration and road surface model matching. [Figure 6] 13 is a flowchart of a process executed by a first common imaging area observation unit. [Figure 7] 13 is a schematic diagram showing a modified example of the process of coordinate integration and road surface model matching of Modification 1. FIG. [Figure 8] 13 is a flowchart of a process executed by a first common imaging area observing unit in the first modified example. [Figure 9] 13 is a schematic diagram showing a further modified example of the process of coordinate integration and road surface model matching of the modified example 2. FIG. [Figure 10] FIG. 11 is a functional block diagram of a distance estimation program executed by a CPU in Modification 2. [Figure 11] FIG. 11 is a block diagram showing a configuration of a vehicle equipped with a sensing device according to a third modified example. [Figure 12] 13 is a diagram showing an example of the relationship between the camera and the common imaging area in Modification 3. FIG. [Figure 13] FIG. 13 is a functional block diagram of a distance estimation program executed by a CPU in Modification 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of a sensing device 101 according to the present invention will be described with reference to FIGS.

[0011] FIG. 1 is a block diagram showing a configuration of a vehicle 100 equipped with a sensing device 101 according to an embodiment of the present invention.

[0012] The sensing device 101 has cameras 121 to 124, a vehicle speed sensor 131, a steering angle sensor 132, a display device 161, and an in-vehicle processing device 102. The cameras 121 to 124, the vehicle speed sensor 131, the steering angle sensor 132, and the display device 161 are connected to the in-vehicle processing device 102 via signal lines, and exchange various types of data with the in-vehicle processing device 102.

[0013] The cameras 121-124, which will be described in detail later, are attached to the periphery of the vehicle 100 and capture images of the periphery of the vehicle 100. The capture range of the cameras 121-124 includes the road surface on which the vehicle 100 runs. The position and orientation relationship between the cameras 121-124 and the vehicle 100 is stored in the ROM 111 as camera parameter initial values ​​141.

[0014] The cameras 121 to 124 are equipped with lenses and imaging elements, and their characteristics, such as a lens distortion coefficient which is a parameter indicating lens distortion, the optical axis center, focal length, the number of pixels and dimensions of the imaging element, and other internal parameters are also stored in the ROM 111 as camera parameter initial values ​​141.

[0015] The vehicle speed sensor 131 and the steering angle sensor 132 respectively measure the vehicle speed and steering angle of the vehicle 100 equipped with the on-vehicle processing device 102, and output them to the CPU 110. The on-vehicle processing device 102 uses the outputs of the vehicle speed sensor 131 and the steering angle sensor 132 to calculate the amount of movement and the direction of movement of the vehicle 100 equipped with the on-vehicle processing device 102 by a known dead reckoning technique.

[0016] The in-vehicle processing device 102 includes a CPU 110, which is a central processing unit, a ROM 111, and a RAM 112. It may be configured such that all or part of the arithmetic processing is executed by another arithmetic processing device such as an FPGA.

[0017] The CPU 110 operates as an execution unit of the in-vehicle processing device 102 by reading and executing various programs and parameters from the ROM 111 .

[0018] The ROM 111 is a read-only storage area, and stores a camera parameter initial value 141, a road surface model 142, and a distance estimation program 150.

[0019] The camera parameter initial value 141 is a numerical value indicating the relationship between the position and the attitude of the cameras 121 to 124 and the vehicle 100. The cameras 121 to 124 are attached to the vehicle 100 at positions and attitudes based on the design, but it is inevitable that an error occurs in the attachment. For this reason, for example, when the vehicle 100 is shipped from a factory, a calibration is performed in the factory using a predetermined test pattern or the like, and a correctly corrected value is calculated. The camera parameter initial value 141 stores the relationship between the position and the attitude correctly corrected after the execution of this calibration. In addition, the correction values ​​after the execution of the calibration of internal parameters such as the lens distortion coefficient, which is a parameter indicating the characteristics of the lens and the image sensor, for example, the lens distortion, the optical axis center, the focal length, the number of pixels and the dimensions of the image sensor, are also stored.

[0020] The road surface model 142 models the road surface gradient, and stores the model type and its parameters in numerical form. The road surface model 142 is used when calculating a road surface relative attitude 153 in a road surface relative attitude estimation program 143 in a distance estimation program 150. The road surface model 142 will be described in detail later.

[0021] The distance estimation program 150 includes a road surface relative attitude estimation program 143, a common visual field distance measurement program 144, and a monocular distance measurement program 145, which will be described later. These programs are read from the ROM 111, loaded into the RAM 112, and executed by the CPU 110.

[0022] The RAM 112 is a readable / writable storage area, and operates as a main storage device of the in-vehicle processing device 102. The RAM 112 stores a first area observation value 151, a second area observation value 152, a road surface relative attitude 153, and external parameters 154, which will be described later.

[0023] The first area observation value 151 is observed by the common view distance measurement program 144, and is observation data including the road surface and three-dimensional objects in the area observed in the common view of the cameras 121 and 122. The first area observation value 151 is obtained by calculating the distance between the feature points of common objects including the road surface and three-dimensional objects in the common view and the known relative relationship and internal parameters of the cameras 121 and 122 recorded in the camera parameter initial value 141, and storing the three-dimensional coordinates calculated from the distance. The first area observation value 151 is used in the road surface relative attitude estimation program 143.

[0024] The second area observation value 152 is observed by the common view distance measurement program 144, has the same processing content as the first area observation value 151, and is observation data including the road surface and three-dimensional objects in the area observed in the common view of the cameras 123 and 124. The second area observation value 152 is obtained by calculating the distance between the feature points of common objects including the road surface and three-dimensional objects in the common view and the known relative relationship and internal parameters of the cameras 123 and 124 recorded in the camera parameter initial value 141, and storing the three-dimensional coordinates calculated from the distance. The second area observation value 152 is used in the road surface relative attitude estimation program 143.

[0025] The road surface relative attitude 153 is a parameter that indicates the relative attitude between the vehicle 100 and the road surface, and is obtained from the road surface relative attitude estimation program 143. The road surface relative attitude 153 is used to calculate the external parameters 154.

[0026] The external parameters 154 are the relationship between the positions and attitudes of the cameras 121 to 124 and the vehicle 100, including the road surface gradient and the attitude change of the vehicle 100. Based on the external parameters of the camera parameter initial values ​​141, they are calculated in the road surface relative attitude estimation program 143 and used in the monocular distance measurement program 145.

[0027] FIG. 2 is a diagram showing an example of the relationship between vehicle 100, cameras 121 to 124, optical axis 201 of camera 121, optical axis 202 of camera 122, optical axis 203 of camera 123, optical axis 204 of camera 124, first common imaging area 211, and second common imaging area 212.

[0028] The camera 121 is attached, for example, under the side mirror on the left side of the vehicle 100 in the traveling direction so as to capture an image of the diagonally rear of the vehicle 100, i.e., in the direction of the optical axis 201. Similarly, the camera 123 is attached, for example, under the side mirror on the right side of the vehicle 100 in the traveling direction so as to capture an image of the diagonally rear of the vehicle 100, i.e., in the direction of the optical axis 203. The camera 122 is attached, for example, near the C pillar on the left side of the vehicle 100 in the traveling direction so as to capture an image of the diagonally forward of the vehicle 100, i.e., in the direction of the optical axis 202. Similarly, the camera 124 is attached, for example, near the C pillar on the right side of the vehicle 100 in the traveling direction so as to capture an image of the diagonally forward of the vehicle 100, i.e., in the direction of the optical axis 204. The area on the left side of the traveling direction of the vehicle 100 captured by both the camera 121 and the camera 122 is a first common imaging area 211. In the first common imaging area 211, the road surface and three-dimensional objects are captured. By arranging the cameras in this manner, there is an advantage that the resolution of the cameras 121 and 122 is high, a common imaging area can be generated near the center of the optical axis with little image distortion, and the diagonally forward and diagonally backward areas, which are important in sensing by an on-board camera, can be imaged. Furthermore, a camera with a matching angle of view, angle, etc. may be selected so that the camera also functions as a peripheral vision camera that displays an overhead view of the surroundings of the vehicle 100. Similarly, the area on the left side of the traveling direction of the vehicle 100 that is imaged by both the cameras 123 and 124 is the second common imaging area 212. The second common imaging area 212 includes the road surface and three-dimensional objects. The imaging results of the first common imaging area 211 and the second common imaging area 212 are used to estimate the relative attitude between the road surface and the vehicle 100.

[0029] FIG. 3(a) is a diagram showing an example of a gradient, FIG. 3(b) is a diagram showing an example of a loading state, and FIGS. 3(c) and 3(d) are diagrams showing an example of a vehicle attitude change.

[0030] A drainage gradient of about 2% is provided in the transverse direction with the road center as the apex so that rainwater does not accumulate on the road. On expressways where the vehicle 100 travels at high speed, a drainage gradient of about 2.5% is provided because it is necessary to make the water film on the road surface thinner. FIG. 3(a) shows an example of a drainage gradient. The vehicle 100 is traveling on a road surface 301, and a drainage gradient is provided on the road surface 301 with the road center as the apex. In the direction of the optical axis 202 from the vehicle 100, the observed object is on the same plane as the vehicle 100, so there is no effect of error due to the drainage gradient on distance measurement in the direction of the optical axis 202. On the other hand, in the direction of the optical axis 201 from the vehicle 100, the observed object is not on the same plane as the vehicle 100, so the distance is calculated incorrectly under the assumption that the observed object is on the same plane as the vehicle 100. Therefore, it is necessary to estimate the relative relationship between the vehicle 100 and the road surface 301 and correct the estimated distance based on the estimation result.

[0031] FIG. 3(b) is a diagram showing an example of a loading state. The attitude of the vehicle 100 with respect to the road surface changes depending on the loading state of the vehicle 100. For example, when a heavy load is loaded on the rear of the vehicle 100, the front of the vehicle 100 is raised as shown in FIG. 3(c). When only the driver is seated in the left front seat of the vehicle 100, the left side of the vehicle 100 is raised as shown in FIG. 3(d). When passengers are seated in the driver's seat and the seat behind the driver's seat of the vehicle 100, the vehicle 100 assumes a different attitude. In this way, the attitude of the vehicle 100 changes in various ways depending on the loading state. When the attitude of the vehicle 100 changes in this way, the relative attitude between the vehicle 100 and the road surface 301 shown in FIG. 3(a) changes, so it is necessary to estimate the relative attitude relationship between the vehicle 100 and the road surface 301, including the attitude change due to the loading state, and correct the estimated distance based on the estimation result.

[0032] Fig. 4 is a functional block diagram of the distance estimation program 150 executed by the CPU 110. Fig. 4 shows the processing order of each functional block of the distance estimation program 150, and the data flows between the functional blocks, and between the functional blocks and the ROM 111 and RAM 112.

[0033] The distance estimation program 150 includes a sensor value acquisition unit 401, a first common imaging area observation unit 402, a second common imaging area observation unit 403, a coordinate integration unit 404, a road surface model matching unit 405, and a monocular distance estimation unit 406. The functional blocks corresponding to the common field of view distance measurement program 144 are the first common imaging area observation unit 402 and the second common imaging area observation unit 403, the functional blocks corresponding to the road surface relative attitude estimation program 143 are the coordinate integration unit 404 and the road surface model matching unit 405, and the functional block corresponding to the monocular distance measurement program 145 is the monocular distance estimation unit 406.

[0034] The sensor value acquisition unit 401 acquires images output from the multiple cameras 121-124. The cameras 121-124 capture images continuously at a predetermined frequency (for example, 30 times per second). Images captured by the cameras are transmitted to the in-vehicle processing device 102 every time they are captured. In this example, the cameras 121 and 122, and the cameras 123 and 124 are synchronized in some way, and it is desirable for them to capture images synchronously. The sensor value acquisition unit 401 outputs images captured by the cameras 121-124 to the first common imaging area observation unit 402 and the second common imaging area observation unit 403. Thereafter, the processes from the first common imaging area observation unit 402 to the monocular distance estimation unit 406 are executed every time an image is received.

[0035] The first common imaging area observation unit 402 measures the three-dimensional coordinate value of the road surface 301 captured in the first common imaging area 211 using the images captured by the multiple cameras 121 and 122 output from the sensor value acquisition unit 401. For example, a common object (e.g., a part that is easy to find as a feature point such as a corner of the road surface paint) captured in the image of the camera 121 and the pixel corresponding to the road surface 301 of the camera 122 can be extracted by image recognition, and the three-dimensional coordinate value of the road surface 301 can be measured by triangulation using a known geometric relationship between the cameras 121 and 122 (e.g., the calibrated camera parameter initial value 141 stored in the ROM 111). This is calculated by the cameras 121 and 122 using a plurality of feature points on the road surface that can be associated with each other. Whether a certain pixel is on the road surface may be determined within a range calculated by providing a certain error range to the known relationship between the attitude angle of the camera parameter initial value 141 and the road surface gradient, or the road surface may be determined by distinguishing three-dimensional objects on the captured image. The measurement of the three-dimensional coordinate values ​​may be performed using other known methods. The calculated distance is calculated, for example, in a camera coordinate system with the optical axis 201 of the camera 121 as one of its axes. The first common image capture area observation unit 402 outputs all the calculated three-dimensional coordinate values ​​of the road surface 301 to the coordinate integration unit 404. The processing flow of the first common image capture area observation unit 402 will be described with reference to FIG. 6.

[0036] The second common imaging area observation unit 403 measures three-dimensional coordinate values ​​of the road surface 301 photographed in the second common imaging area 212, using images captured by the multiple cameras 123, 124 output from the sensor value acquisition unit 401. The processing of the second common imaging area observation unit 403 is the same as the processing of the first common imaging area observation unit 402, except that the cameras that captured the images are different and that the second common imaging area 212 is photographed, so a detailed description will be omitted.

[0037] The coordinate integration unit 404 is a functional block that integrates the three-dimensional coordinates of the road surface 301 output from the first common imaging area observation unit 402 and the three-dimensional coordinates of the road surface 301 output from the second common imaging area observation unit 403 into the same coordinate system.

[0038] The first common imaging area observation unit 402 calculates three-dimensional coordinate values ​​in, for example, the camera coordinate system of the camera 121. Meanwhile, the second common imaging area observation unit 403 calculates three-dimensional coordinate values ​​in, for example, the camera coordinate system of the camera 123. The coordinate integration unit 404 converts and integrates the coordinate values ​​in these different coordinate systems, for example, into a vehicle coordinate system, using the calibrated camera parameter initial values ​​141 stored in the ROM 111. The conversion and integration into the vehicle coordinate system can be performed by a known coordinate conversion calculation using external parameters that represent the position and orientation relationship in the camera parameter initial values ​​141. The coordinate system in which the coordinates are integrated may be another coordinate system. The three-dimensional coordinate values ​​obtained by integrating the first common imaging area 211 and the second common imaging area 212 output from the first common imaging area observation unit 402 and the second common imaging area observation unit 403 are output to the road surface model matching unit 405.

[0039] The road surface model matching unit 405 fits the road surface model 142 using the three-dimensional coordinate values ​​obtained by integrating the first common imaging area 211 and the second common imaging area 212 output from the coordinate integration unit 404, and calculates the relative attitude relationship between the vehicle 100 and the road surface 301. The relative attitude relationship is calculated, including the vehicle attitude fluctuations shown in Fig. 3(c) and (d).

[0040] The road surface model 142 is configured with two planes having a gradient of, for example, about 2%, taking into consideration the road structure in which the drainage gradient is about 2 to 2.5% in the transverse direction, with the center of the road as the apex. Other road surface models will be described later.

[0041] The road surface model matching unit 405 obtains the relative attitude of the road surface model 142 with respect to the vehicle 100 so that the road surface model 142 and the three-dimensional coordinate values ​​obtained by integrating the first common imaging area 211 and the second common imaging area 212 output from the coordinate integration unit 404 are most consistent. For example, the sum of the distances between the three-dimensional coordinate values ​​obtained by integrating the first common imaging area 211 and the second common imaging area 212 output from the coordinate integration unit 404 and the road surface plane of the road surface model 142 is set as an objective function, and the relative position and attitude parameters of the road surface model 142 with respect to the vehicle 100 are calculated so as to minimize the objective function. The relative position and attitude parameters of the vehicle 100 and the road surface model 142 are one or more of three attitude angles, namely, a roll angle, a pitch angle, and a yaw angle, and position parameters of length, width, and height. In order to stabilize the estimation, one or more parameters may be known and other parameters may be estimated. The objective function can be minimized by using a known objective function minimization method such as the steepest descent method or the Levenberg-Marquardt method. The relative position and attitude parameters of the vehicle 100 and the road surface 301 obtained by the road surface model matching unit 405 are output to the monocular distance estimation unit 406 as road surface relative attitude 153. Also, to be used as an initial value for optimization at the next time, it is stored in the RAM 112 as the road surface relative attitude 153. At the next time, the road surface relative attitude 153 is read from the RAM 112 and the objective function is minimized as the initial value. Then, a new road surface relative attitude obtained from the new three-dimensional coordinate values ​​is stored in the RAM 112 as the road surface relative attitude 153. Note that the road surface model matching unit 405 may obtain the relative position and attitude of the vehicle 100 by another method without using a road surface model.

[0042] The monocular distance estimation unit 406 calculates an accurate distance to the object using the road surface relative orientation 153 and road surface model 142 output from the road surface model matching unit 405. The relative relationship between the cameras 121 to 124 and the road surface 301 can be calculated by a known geometric solution from the road surface model 142, the road surface relative orientation 153, and the camera parameter initial value 141. Using the calculated relative relationship, the distance to the object is calculated based on the pixel corresponding to the road surface installation point of the object, and output as an estimated distance. This object is detected, for example, by an image recognition function using AI or the like provided in the cameras 121 to 124. The distance to the object can be calculated in areas other than the common imaging areas 211 and 212 of the cameras 121 to 124, and can be calculated correctly even when the road surface is not flat over a wide range using multiple cameras.

[0043] 5(a) and 5(b) are schematic diagrams showing the processes of coordinate integration and road surface model matching. In Fig. 5(a) and 5(b), the relationship between the first common imaging area observation unit 402, the second common imaging area observation unit 403, the coordinate integration unit 404, and the road surface model matching unit 405, the outline of the processes performed by these units, and their advantages are shown.

[0044] A three-dimensional coordinate value 501 of the road surface 301 in the first common imaging area 211 observed by the first common imaging area observation unit 402 is expressed in the camera coordinate system of the camera 121, and a three-dimensional coordinate value 502 of the road surface 301 in the second common imaging area 212 observed by the second common imaging area observation unit 403 is expressed in the camera coordinate system of the camera 123. The relative relationship between the three-dimensional coordinate value 501 and the three-dimensional coordinate value 502 is unclear. Here, the coordinate integration unit 404 converts the three-dimensional coordinate value 501 in the camera coordinate system into a three-dimensional coordinate value 503 in the coordinate system of the vehicle 100, and converts the three-dimensional coordinate value 502 in the camera coordinate system into a three-dimensional coordinate value 504 in the coordinate system of the vehicle 100, using the external parameters of the camera parameter initial value 141. The three-dimensional coordinate value 503 and the three-dimensional coordinate value 504 are expressed in the same coordinate system, and the relative relationship is clear. The road surface model matching unit 405 can find the relative relationship between the vehicle 100 and the road surface 301 by matching the three-dimensional coordinate values ​​503 and the three-dimensional coordinate values ​​504 with the road surface model 142. By utilizing the areas on both sides of the vehicle 100, not just one side of the vehicle 100, and further fitting the road surface model 142, it is possible to stably estimate the road surface using observation values ​​over a wide range and constraints imposed by the model, and to find the relative relationship between the vehicle 100 and the road surface with high accuracy, compared to estimating the road surface individually in each common imaging area. As a result, even if the road is not flat, it is possible to improve the accuracy of distance measurement using images captured by multiple cameras.

[0045] 5(c), 5(d), and 5(e) are diagrams showing examples of the road surface model 142. FIG. 5(c) is a model in which two planes with a gradient of 2% are connected. FIG. 5(d) is a model in which the tops of two planes with a gradient of 2% are curved. FIG. 5(e) is a model in which the tops of the road surface and the width of the road surface are excluded in order to accommodate various shapes of the tops of the road surface and the width of the road surface plane. Corresponding points close to the excluded parts can be flexibly matched by not including them in the error summation of the objective function when matching the road surface with the three-dimensional coordinate values. In addition, in a highway with three lanes on each side and a median strip, there are cases where the entire width of the three lanes on each side is flat. The road surface models 142 in FIG. 5(c) to FIG. 5(e) can also accommodate cases where the entire width is a single flat surface. In other words, when the road surface model 142 is slid significantly left or right, the entire observation range becomes a flat road surface. In the example shown here, an example of a model in which two planes are connected in accordance with the shape of the road cross slope is shown, but a model in which two or more planes are connected may also be used.

[0046] 6 is a flowchart of the process executed by the first common imaging area observing unit 402. In the first common imaging area observing unit 402, every time an image is received from the sensor value acquiring unit 401, the CPU 110 executes the process of each of the following steps.

[0047] In the feature point detection step 601, feature points (for example, corner points of road paint) that are characteristic points in the images are detected from each of the two images obtained from the sensor value acquisition unit 401. The detection range is the road surface in the common imaging area of ​​the two images. Whether a certain pixel is on the road surface in the common imaging area may be determined by a part calculated by giving a certain error range to the known relationship between the attitude angle of the camera parameter initial value 141 and the road surface gradient, or by discriminating between three-dimensional objects and the road surface on the captured image. The feature points are detected using a known feature point extraction technique such as the Harris operator. Also, information expressing the feature points may be attached by a known technique such as ORB and used in the next feature point correspondence step. Next, proceed to step 602.

[0048] In the feature point matching step 602, points that represent the same object are matched from among the feature points of each of the two images obtained in the feature point detection step 601. For example, the feature points are matched using the feature point information obtained for each feature point in step 601. Specifically, a feature point in one image whose feature point information is closest to that of the feature point in the other image is selected, and the two feature points are matched. Any known technique may be used to improve the feature point coordinates and the matching accuracy of the feature points. Next, the process proceeds to step 603.

[0049] In distance measurement step 603, the three-dimensional distance of each point is calculated using the correspondence between the two images obtained in step 602. The three-dimensional distance can be calculated by applying the principle of triangulation to two corresponding coordinate points using the geometric relationship between the two cameras 121 and 122 obtained from the camera parameter initial values ​​141 stored in ROM 111. For the sake of explanation, the camera 121 is used as the reference and the distance is calculated as seen from the camera 121, but the camera 122 may be used as the reference. In this case, there is no problem if a geometric calculation is performed in a later stage according to the camera that is used as the reference. To ensure accuracy, any known technology may be used for the calculation. Next, proceed to step 604.

[0050] In a three-dimensional coordinate calculation step 604, three-dimensional coordinate values ​​are calculated from the coordinates on the image of the camera 121 and the distance obtained in step 603. Using the camera parameter initial values ​​141, the three-dimensional coordinates can be calculated from the coordinates on the image and the distance by known geometric calculations. The three-dimensional coordinates calculated here are the camera coordinates of the camera 121. The three-dimensional coordinates of each point are output, and the flow chart ends.

[0051] The processing performed by the first common imaging area observation unit 402 has been described above with reference to Figure 6. However, the processing performed by the second common imaging area observation unit 403 is the same as that performed by the first common imaging area observation unit 402, except that the camera image and the associated camera parameters, etc. are different.

[0052] <Variation 1> Fig. 7(a) and Fig. 7(b) are schematic diagrams showing modified examples of the coordinate integration and road surface model matching processes. Fig. 7(a) and Fig. 7(b) show the relationship between the first common imaging area observation unit 402, the second common imaging area observation unit 403, the coordinate integration unit 404, and the road surface model matching unit 405, as well as an overview of the processes performed by these units and their advantages. Fig. 7(a) and Fig. 7(b) are modified examples of Fig. 5(a) and Fig. 5(b), and explanations of common parts will be omitted, and differences will be explained.

[0053] The three-dimensional coordinate value 701 is a three-dimensional coordinate value of a solid object observed in the first common imaging region 211 and expressed in the camera coordinate system of the camera 121. The coordinate integration unit 404 converts the three-dimensional coordinate value 701 of the solid object into a three-dimensional coordinate value 702 in the coordinate system of the vehicle 100 using external parameters of the camera parameter initial value 141. In addition to the three-dimensional coordinate values ​​503 and 504, the three-dimensional coordinate value 702 may be used to match with the road surface model 142. In this case, the three-dimensional coordinate values ​​702 are considered to be aligned vertically, and the objective function of the road surface model 142 is designed so that the evaluation value is high when the coordinate values ​​are vertical.

[0054] Fig. 8 is a flowchart of processing executed by the first common imaging area observing unit 402, and corresponds to the modified example shown in Fig. 7. Since the flowchart in Fig. 8 is almost the same as the flowchart in Fig. 6, a description of the common parts will be omitted and only the differences will be described.

[0055] In the road surface / solid object separation step 801, among the observation points in the common imaging area, the observation points on the road surface and the observation points of solid objects are separated. For example, using image recognition by AI or the like, points recognized as guardrails, utility poles, etc., are labeled as solid objects, and the points recognized by road surface painting are labeled as points on the road surface, and the observation points are separated. Similarly, for solid objects, a different label is attached to each solid object and they are grouped. These grouped points are fitted to a rectangle, and rectangle information is attached. The attached rectangle information is used in the road surface model matching unit 405 in the subsequent stage by designing an objective function that is minimized when this rectangle and the road surface model 142 are perpendicular to each other. Here, the labeled three-dimensional coordinate points are output, and the flowchart ends.

[0056] The processing performed by the first common imaging area observation unit 402 has been described above with reference to Figure 8. However, the processing performed by the second common imaging area observation unit 403 is the same as that performed by the first common imaging area observation unit 402, except that the camera image and the associated camera parameters, etc. are different.

[0057] <Variation 2> Fig. 9(a) and Fig. 9(b) are schematic diagrams showing further modified examples of the coordinate integration and road surface model matching processes. Fig. 9(a) and Fig. 9(b) show the relationship between the first common imaging area observation unit 402, the second common imaging area observation unit 403, the coordinate integration unit 404, and the road surface model matching unit 405, the outline of their processing, and their advantages. Fig. 9(a) and Fig. 9(b) are modified examples of Fig. 7(a) and Fig. 7(b), and explanations of common parts will be omitted and differences will be explained.

[0058] The three-dimensional coordinate value 901 is the three-dimensional coordinate value of a solid object observed in the first common imaging region 211 at the time next to the three-dimensional coordinate values ​​501 and 701. That is, under the assumption that the attitude of the vehicle 100 does not change significantly in a short time, the observation points of the first common imaging region 211 are integrated in a time series. Similarly, the three-dimensional coordinate value 902 is the three-dimensional coordinate value of a solid object observed in the second common imaging region 212 at the time next to the three-dimensional coordinate value 502, and is obtained by integrating the coordinate points of the second common imaging region 212 in a time series. Here, the coordinate integration unit 404 converts the three-dimensional coordinate values ​​901 and 902 into the coordinate system of the vehicle 100 using the external parameters of the camera parameter initial value 141. Not only the three-dimensional coordinate values ​​503 and 504 but also the three-dimensional coordinate values ​​702, 901, and 902 are used to match with the road surface model 142. In this case, the three-dimensional coordinate values ​​901, 902 are also considered to be aligned vertically, and the objective function of the road surface model 142 is designed so that the evaluation value becomes high when they are vertical.

[0059] According to this modification, not only road surface points and three-dimensional object points observed at a certain point in time on both sides of the vehicle 100 are used, but also time-series road surface points and three-dimensional object points are used, and further fitting them to the road surface model 142 allows stable estimation based on observation values ​​over a wide range and constraints by the model, rather than individual estimation based on observation values ​​at a certain point in time in each common imaging area, and makes it possible to obtain the relative relationship between the vehicle 100 and the road surface with high accuracy. As a result, even if the road is not flat, the accuracy of distance measurement using images captured by multiple cameras can be further improved.

[0060] Fig. 10 is a functional block diagram of distance estimation program 150 executed by CPU 110, and corresponds to the modified example shown in Fig. 9. Fig. 10 shows the processing order of each functional block of distance estimation program 150, and the flow of data between the functional blocks, and between the functional blocks and ROM 111 and RAM 112. The functional block diagram of Fig. 10 is almost the same as the functional block diagram of Fig. 4, so a description of the common parts will be omitted and only the differences will be described.

[0061] The distance estimation program 150 includes a sensor value acquisition unit 401, a first common imaging area observation unit 402, a second common imaging area observation unit 403, a coordinate integration unit 404, a time series integration unit 1001, a road surface model matching unit 405, a monocular distance estimation unit 406, and an odometry estimation unit 1002. The time series integration unit 1001 is one of the functional blocks corresponding to the road surface relative attitude estimation program 143, and integrates in a time series manner the three-dimensional coordinate values ​​in which the first common imaging area 211 and the second common imaging area 212 output from the coordinate integration unit 404 are integrated. The three-dimensional coordinate values ​​are integrated under the assumption that the attitude of the vehicle 100 does not change significantly in a short period of time. When integrating, the three-dimensional coordinate values ​​can be integrated by acquiring the amount of movement of the vehicle 100 between the previous time and the current time, and calculating the amount of movement of the camera from the amount of vehicle movement by known geometric calculation. The amount of vehicle movement is acquired from an odometry estimation unit 1002 described later. At the first time, the output of the coordinate integration unit 404 is written and recorded in the RAM 112, and from the next time onwards, the point cloud recorded at the previous time is read from the RAM 112 and added to the point cloud in the coordinate system of the vehicle 100 newly obtained from the coordinate integration unit 404 and stored. It is preferable that the time range for integration can be adjusted as a parameter. The integrated three-dimensional coordinate values ​​are output to the road surface model matching unit 405 and the RAM 112.

[0062] The odometry estimation unit 1002 estimates the motion of the vehicle 100 using the speed and steering angle of the vehicle 100 transmitted from the vehicle speed sensor 131 and the steering angle sensor 132. For example, the odometry estimation unit 1002 may use a known dead reckoning method, a known visual odometry technique using a camera, or a known Kalman filter or the like. The odometry estimation unit 1002 outputs the vehicle motion estimation result to the time series integration unit 1001.

[0063] <Variation 3> Fig. 11 is a block diagram showing the configuration of a vehicle 100 equipped with a sensing device 101 according to a modified example of the present invention. The sensing device 101 shown in Fig. 11 is almost the same as the sensing device 101 shown in Fig. 1, so a description of the common parts will be omitted and only the differences will be described.

[0064] The sensing device 101 shown in FIG. 11 is obtained by adding a camera 1101, a camera 1102, and a vehicle control device 1180 to the sensing device 101 shown in FIG. 1. The cameras 1101 and 1102 are attached to the periphery of the vehicle 100 in the same manner as the cameras 121 to 124, and capture the periphery of the vehicle 100. The method of attaching the cameras 1101 and 1102 will be described later. The shooting range of the cameras 1101 and 1102 includes the road surface on which the vehicle 100 runs. The relationship between the positions and attitudes of the cameras 1101 and 1102 and the vehicle 100 is stored in the ROM 111 as the camera parameter initial value 141. The cameras 1101 and 1102 have a lens and an image sensor. The characteristics of the cameras 1101 and 1102, such as a lens distortion coefficient indicating the lens distortion, the optical axis center, the focal length, the number of pixels of the image sensor, the dimensions, and the like, are also stored in the ROM 111 as the camera parameter initial value 141.

[0065] The vehicle control device 1180 controls the steering device, the driving device, the braking device, the active suspension, and the like, using information output from the CPU 110, for example, the road surface relative attitude 153 output from the distance estimation program 1150. The steering device operates the steering of the vehicle 100. The driving device applies driving force to the vehicle 100. The driving device increases the driving force of the vehicle 100, for example, by increasing the target rotation speed of the engine of the vehicle 100. The braking device applies braking force to the vehicle 100. The active suspension can change the operation of various devices while traveling, such as by expanding and contracting an actuator operated by hydraulic pressure or pneumatic pressure, or by adjusting the strength of the damping force of a spring.

[0066] Figure 12 is a diagram showing an example of the relationship between the vehicle 100, cameras 121 to 124, 1101, 1102, optical axis 201 of camera 121, optical axis 202 of camera 122, optical axis 203 of camera 123, optical axis 204 of camera 124, optical axis 1201 of camera 1101, optical axis 1202 of camera 1102, and the first common imaging area 211, the second common imaging area 212, the third common imaging area 1213, the fourth common imaging area 1214, the fifth common imaging area 1215, and the sixth common imaging area 1216 in the sensing device 101 of the modified example shown in Figure 11.

[0067] The camera 1101 is attached to the front of the vehicle 100 with a camera angle of view selected so as to capture an image in the direction of an optical axis 1201 and to have a common imaging area with the cameras 122 and 124. The common imaging area of ​​the cameras 1101 and 122 is a third common imaging area 1213, and the common imaging area of ​​the cameras 1101 and 124 is a fourth common imaging area 1214. The camera 1102 is attached to the rear of the vehicle 100 with a camera angle of view selected so as to capture an image in the direction of the optical axis 1202 and to have a common imaging area with the cameras 121 and 123. The common imaging area of ​​the cameras 1102 and 121 is a fifth common imaging area 1215, and the common imaging area of ​​the cameras 1102 and 123 is a sixth common imaging area 1216. The road surface and three-dimensional objects are captured in each common imaging area. Furthermore, a camera with a matching angle of view, angle, etc. may be selected so that it also functions as a peripheral vision camera that displays an overhead view of the periphery of the vehicle 100. The imaging results in each of the common imaging areas 211, 212, 1213 to 1216 are used to estimate the relative attitude of the road surface and the vehicle 100. In the modification shown in FIG. 12, a stable estimation can be performed using observation values ​​in an even wider range, and the relative relationship between the vehicle 100 and the road surface can be obtained with high accuracy. As a result, even if the road is not flat, the accuracy of distance measurement using images captured by multiple cameras can be improved.

[0068] Fig. 13 is a functional block diagram of distance estimation program 150 executed by CPU 110, and corresponds to the modified example shown in Fig. 11. Fig. 13 shows the processing order of each functional block of distance estimation program 150, and the flow of data between the functional blocks, and between the functional blocks and ROM 111 and RAM 112. The functional block diagram of Fig. 13 is almost the same as the functional block diagram of Fig. 10, so a description of the common parts will be omitted and only the differences will be described.

[0069] 13 includes a first common imaging area observation unit 402, a second common imaging area observation unit 403, a third common imaging area observation unit 1313, a fourth common imaging area observation unit 1314, a fifth common imaging area observation unit 1315, and a sixth common imaging area observation unit 1316. In addition, images are input to the common visual field distance measurement program 144 from the cameras 121 to 124 and the cameras 1101 and 1102.

[0070] The third common imaging area observation unit 1313 measures three-dimensional coordinate values ​​of the road surface 301 photographed in the third common imaging area 1213, using images captured by the multiple cameras 122, 1101 output from the sensor value acquisition unit 401. The fourth common imaging area observation unit 1314 measures three-dimensional coordinate values ​​of the road surface 301 photographed in the fourth common imaging area 1214, using images captured by the multiple cameras 124, 1101 output from the sensor value acquisition unit 401. The fifth common imaging area observation unit 1315 measures three-dimensional coordinate values ​​of the road surface 301 photographed in the fifth common imaging area 1215, using images captured by the multiple cameras 121, 1102 output from the sensor value acquisition unit 401. The sixth common imaging area observation unit 1316 measures three-dimensional coordinate values ​​of the road surface 301 captured in the sixth common imaging area 1216, using images captured by the multiple cameras 123, 1102 output from the sensor value acquisition unit 401. The three-dimensional coordinate values ​​output from each common imaging area observation unit are expressed in the camera coordinate system of each camera, and the three-dimensional coordinate values ​​in the camera coordinate system are converted into three-dimensional coordinate values ​​in the coordinate system of the vehicle 100, using external parameters of the camera parameter initial values ​​141.

[0071] The vehicle control unit 1320 receives the road surface relative attitude 153 that is the output of the road surface model matching unit 405, and controls the vehicle 100. One example of vehicle control is to improve the ride comfort by controlling an active suspension so as to reduce the influence of the roll angle and pitch angle of the vehicle 100 obtained from the road surface relative attitude 153. The vehicle 100 may be controlled based on other attitude angles of the vehicle 100.

[0072] As described above, the sensing device 101 of an embodiment of the present invention is equipped with a first common imaging area observation unit 402 that observes the first common imaging area 211 around the vehicle from information of the common imaging area acquired by at least a first sensor (camera 121) and a second sensor (camera 122) having a common imaging area, a second common imaging area observation unit 403 that observes the second common imaging area 212, which is different from the first common imaging area 211, from information of the common imaging area acquired by at least a third sensor (camera 123) and a fourth sensor (camera 124) having a common imaging area, a coordinate integration unit 404 that integrates the geometric relationship of each sensor and the coordinates of the information observed in the first common imaging area 211 and the second common imaging area 212, and a road surface estimation unit (road surface model matching unit 405) that estimates the relative attitude of each sensor and the road surface, including the pitch angle and roll angle of the vehicle, based on point cloud information calculated from the integrated coordinates, thereby reducing the effects of road gradient and vehicle attitude and improving distance measurement accuracy. In addition, the accuracy of binocular distance measurement using multiple cameras can be improved.

[0073] In addition, since the common imaging area is positioned at the position where the optical axes of the sensors intersect, the effects of road gradient and vehicle posture can be accurately corrected using images with less distortion near the optical axes.

[0074] In addition, the road surface estimation unit (road surface model matching unit 405) has a road surface model matching unit 405 that matches the point cloud information of the road surface with a road surface model, and the road surface model matching unit 405 fits the point cloud information of the road surface to the road surface model 142 so as to reduce the error between the point cloud information of the road surface and the road surface model 142, thereby making it possible to accurately correct the effects of road gradient and vehicle attitude.

[0075] In addition, the first common imaging area observation unit 402 and the second common imaging area observation unit 403 separate the road surface point cloud information from the object point cloud information representing objects present on the road surface (801), and the road surface estimation unit (road surface model matching unit 405) matches the object point cloud information with the road surface model so that the error between the road surface point cloud information and the road surface is small and the object point cloud information is aligned vertically, so that the effects of road gradient and vehicle attitude can be accurately corrected without causing major errors.

[0076] In addition, the road surface estimation unit (road surface model matching unit 405) has a time series integration unit 1001 that time-series integrates the output of the first common imaging area observation unit and time-series integrates the output of the second common imaging area observation unit, so that the effects of road gradient and vehicle attitude can be accurately corrected, improving the robustness of road surface estimation.

[0077] Moreover, the road surface estimation unit (road surface model matching unit 405) receives information acquired by the six sensors, and estimates the relative attitude of each sensor and the road surface, including the pitch angle and roll angle of the vehicle, based on point cloud information calculated by integrating coordinates of information observed in a common imaging area by a combination of two of the six sensors, so that it is possible to use information from many sensors and a wide area, and to accurately correct the effects of road gradient and vehicle attitude. Also, it is possible to improve the accuracy of binocular distance measurement using multiple cameras, and improve the robustness of road surface estimation.

[0078] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. Furthermore, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0079] Furthermore, the sensing device 101 may be provided with an input / output interface (not shown), and a program may be loaded from another device when necessary via the input / output interface and a medium available to the sensing device 101. Here, the medium refers to, for example, a storage medium that is detachable from the input / output interface, or a communication medium, that is, a network such as a wired, wireless, or optical network, or a carrier wave or digital signal that propagates through the network.

[0080] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits, or may be realized in software by a processor interpreting and executing a program that realizes each function. In addition, some or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

[0081] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0082] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0083] 100 vehicles 101 Sensing Device 102 On-board processing device 110 CPU 111 ROM 112 RAM 121, 122, 123, 124, 1101, 1102 Camera 131 Vehicle speed sensor 132 Steering angle sensor 141 Camera parameter initial value 142 Road surface model 143 Road surface relative attitude estimation program 144 Common Visual Field Distance Measurement Program 145 Monocular distance measurement program 150 Distance Estimation Program 151 First Region Observations 152 Second Region Observations 153 Road surface relative attitude 154 External parameters 161 Display device 201, 202, 203, 204, 1201, 1202 Optical axis 211 First common imaging area 212 Second common imaging area 301 Road surface 401 Sensor value acquisition unit 402 First Common Imaging Area Observation Section 403 Second Common Imaging Area Observation Section 404 Coordinate Integration Unit 405 Road surface model matching unit 406 Monocular Distance Estimation Unit 1001 Time Series Integration Department 1002 Odometry estimation unit 1150 Distance Estimation Program 1180 Vehicle control device 1213 Third common imaging area 1214 Fourth common imaging area 1215 Fifth Common Imaging Area 1216 6th Common Imaging Area 1313 Third Common Imaging Area Observation Section 1314 Fourth Common Imaging Area Observation Section 1315 Fifth Common Imaging Area Observation Section 1316 6th Common Imaging Area Observation Section 1320 Vehicle control unit

Claims

1. A first common imaging area observation unit that observes a first area around the host vehicle from information on the common imaging area acquired by at least a first sensor and a second sensor having the common imaging area; A second common imaging area observation unit that observes a second area different from the first area from information on the common imaging area acquired by at least a third sensor and a fourth sensor having the common imaging area; A coordinate integration unit that integrates the geometric relationships of the respective sensors and the coordinates of the information observed in the first area and the second area; A road surface estimation unit that estimates the relative posture of each sensor including the pitch angle and roll angle of the host vehicle with respect to the road surface based on the point cloud information calculated from the integrated coordinates. The sensing device, wherein the common imaging area is disposed at a position where the optical axes of the sensors intersect.

2. A first common imaging area observation unit that observes a first area around the host vehicle from information on the common imaging area acquired by at least a first sensor and a second sensor having the common imaging area; A second common imaging area observation unit that observes a second area different from the first area from information on the common imaging area acquired by at least a third sensor and a fourth sensor having the common imaging area; A coordinate integration unit that integrates the geometric relationships of the respective sensors and the coordinates of the information observed in the first area and the second area; A road surface estimation unit that estimates the relative posture of each sensor including the pitch angle and roll angle of the host vehicle with respect to the road surface based on the point cloud information calculated from the integrated coordinates. The sensing device, wherein the road surface estimation unit collates the point cloud information of the road surface with a road surface model.

3. The sensing device according to Claim 2, wherein the road surface estimation unit fits the point cloud information of the road surface to the road surface model so that the error between the point cloud information of the road surface and the road surface model is reduced.

4. A first common imaging area observation unit that observes a first area around the host vehicle from information on the common imaging area acquired by at least a first sensor and a second sensor having the common imaging area; A second common imaging area observation unit that observes a second area different from the first area from information on the common imaging area acquired by at least a third sensor and a fourth sensor having the common imaging area; A coordinate integration unit that integrates the geometric relationships of the respective sensors and the coordinates of the information observed in the first area and the second area; Based on the point cloud information calculated from the integrated coordinates, a road surface estimation unit that estimates the relative posture between each sensor including the pitch angle and roll angle of the host vehicle and the road surface; A sensing device comprising: a distance measurement unit that measures distance with reference to a correction value calculated from the estimated relative posture.

5. A first common imaging area observation unit that observes a first area around the host vehicle from the information of the common imaging area acquired by at least a first sensor and a second sensor having a common imaging area; A second common imaging area observation unit that observes a second area different from the first area from the information of the common imaging area acquired by at least a third sensor and a fourth sensor having a common imaging area; A coordinate integration unit that integrates the geometric relationship of each sensor and the coordinates of the information observed in the first area and the second area; Based on the point cloud information calculated from the integrated coordinates, a road surface estimation unit that estimates the relative posture between each sensor including the pitch angle and roll angle of the host vehicle and the road surface; The first common imaging area observation unit and the second common imaging area observation unit separate road surface point cloud information and object point cloud information representing an object existing on the road surface; The road surface estimation unit is characterized in that it collates with a road surface model so that the error between the road surface point cloud information and the road surface becomes small and the object point cloud information is arranged vertically.

6. A first common imaging area observation unit that observes a first area around the host vehicle from the information of the common imaging area acquired by at least a first sensor and a second sensor having a common imaging area; A second common imaging area observation unit that observes a second area different from the first area from the information of the common imaging area acquired by at least a third sensor and a fourth sensor having a common imaging area; A coordinate integration unit that integrates the geometric relationship of each sensor and the coordinates of the information observed in the first area and the second area; Based on the point cloud information calculated from the integrated coordinates, a road surface estimation unit that estimates the relative posture between each sensor including the pitch angle and roll angle of the host vehicle and the road surface; A sensing device comprising: a time-series integration unit that time-series integrates the output of the first common imaging area observation unit and time-series integrates the output of the second common imaging area observation unit.

7. The sensing device according to any one of Claims 1 to 6, The first common imaging area observation unit acquires information on the left side of the traveling direction of the host vehicle. The second common imaging area observation unit is a sensing device characterized by acquiring information on the right side in the traveling direction of the host vehicle.

8. The sensing device according to any one of Claims 1 to 6, receives information acquired by six sensors, the road surface estimation unit estimates the relative posture between each sensor including the pitch angle and roll angle of the host vehicle and the road surface based on point cloud information calculated by integrating the coordinates of information observed in a common imaging area formed by a combination of two of the six sensors. The sensing device is characterized by this.

9. A first common imaging area observation unit that observes a first area around the host vehicle from the information on the common imaging area acquired by at least a first sensor and a second sensor having the common imaging area, a second common imaging area observation unit that observes a second area different from the first area from the information on the common imaging area acquired by at least a third sensor and a fourth sensor having the common imaging area, a coordinate integration unit that integrates the geometric relationship of each sensor and the coordinates of information observed in the first area and the second area, and a road surface estimation unit that estimates the relative posture between each sensor including the pitch angle and roll angle of the host vehicle and the road surface based on point cloud information calculated from the integrated coordinates. The vehicle control device is provided with this, the common imaging area is arranged at a position where the optical axes of the sensors intersect, and the vehicle is controlled using the estimated relative posture. The vehicle control device is characterized by this.

10. A first common imaging area observation unit that observes a first area around the host vehicle from the information on the common imaging area acquired by at least a first sensor and a second sensor having the common imaging area, a second common imaging area observation unit that observes a second area different from the first area from the information on the common imaging area acquired by at least a third sensor and a fourth sensor having the common imaging area, a coordinate integration unit that integrates the geometric relationship of each sensor and the coordinates of information observed in the first area and the second area, and a road surface estimation unit that estimates the relative posture between each sensor including the pitch angle and roll angle of the host vehicle and the road surface based on point cloud information calculated from the integrated coordinates. The vehicle control device is provided with this, the road surface estimation unit collates the point cloud information of the road surface with a road surface model, and the vehicle is controlled using the estimated relative posture. The vehicle control device is characterized by this. [

11. ] A first common imaging area observation unit that observes a first area around the host vehicle from information on the common imaging area acquired by at least a first sensor and a second sensor having a common imaging area; A second common imaging area observation unit that observes a second area different from the first area from information on the common imaging area acquired by at least a third sensor and a fourth sensor having a common imaging area; A coordinate integration unit that integrates the geometric relationships of the respective sensors and the coordinates of the information observed in the first area and the second area; A road surface estimation unit that estimates the relative posture of each sensor including the pitch angle and roll angle of the host vehicle and the road surface based on the point cloud information calculated from the integrated coordinates; A distance measurement unit that measures distance with reference to a correction value calculated from the estimated relative posture, and A vehicle control device characterized by controlling the vehicle using the estimated relative posture. [

12. ] A first common imaging area observation unit that observes a first area around the host vehicle from information on the common imaging area acquired by at least a first sensor and a second sensor having a common imaging area; A second common imaging area observation unit that observes a second area different from the first area from information on the common imaging area acquired by at least a third sensor and a fourth sensor having a common imaging area; A coordinate integration unit that integrates the geometric relationships of the respective sensors and the coordinates of the information observed in the first area and the second area; A road surface estimation unit that estimates the relative posture of each sensor including the pitch angle and roll angle of the host vehicle and the road surface based on the point cloud information calculated from the integrated coordinates, and The first common imaging area observation unit and the second common imaging area observation unit separate road surface point cloud information and object point cloud information representing an object existing on the road surface, The road surface estimation unit collates with a road surface model so that the error between the road surface point cloud information and the road surface becomes small and the object point cloud information is arranged vertically, A vehicle control device characterized by controlling the vehicle using the estimated relative posture. [

13. ] A first common imaging area observation unit that observes a first area around the host vehicle from information on the common imaging area acquired by at least a first sensor and a second sensor having a common imaging area; A second common imaging area observation unit that observes a second area different from the first area from information on the common imaging area acquired by at least a third sensor and a fourth sensor having a common imaging area; A coordinate integration unit that integrates the geometric relationships of the respective sensors and the coordinates of the information observed in the first region and the second region; A road surface estimation unit that estimates the relative postures of each sensor including the pitch angle and roll angle of the host vehicle and the road surface based on the point cloud information calculated from the integrated coordinates; A time series integration unit that integrates the outputs of the first common imaging region observation unit in time series and integrates the outputs of the second common imaging region observation unit in time series; A vehicle control device characterized by controlling the vehicle using the estimated relative posture.

Images