3D road shape estimation device, 3D road shape estimation method, 3D road shape estimation system, and 3D road shape estimation program

Abstract

To provide a road surface shape estimation apparatus and so on which can estimate a three-dimensional shape of a road easily and with high accuracy.SOLUTION: The present invention is directed to a road surface shape estimation apparatus having a specifying unit for comparing a first bird's eye view obtained by converting a first image obtained by capturing a road surface from a first position into a bird's eye view and a second bird's eye view obtained by converting a second image obtained by capturing a road surface from a second position different from the first position into a bird's eye view to specify, with respect to a plurality of parts of the road surface, a first part corresponding to the part in the first bird's eye view and a second pat corresponding thereto in the second bird's eye view, and an estimation unit for calculating movement amounts on the image of the first part and the second part to estimate an uneven shape of the road surface based on the movement amounts.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a road surface three-dimensional shape estimation device, a road surface three-dimensional shape estimation method, a road surface three-dimensional shape estimation system, and a road surface three-dimensional shape estimation program. [Background technology]

[0002] In recent years, various inspection items and methods have been proposed for the purpose of detecting road defects and developing appropriate maintenance methods. Regarding road surface inspection items, the Ministry of Land, Infrastructure, Transport and Tourism has set the crack rate and rut depth as important indicators for determining whether road surface repair is necessary.

[0003] Possible methods for estimating the three-dimensional shape of a road surface (hereinafter also referred to as "road surface shape") include, for example, a method of estimating the longitudinal shape of the road surface by installing a sensor such as an acceleration sensor on a vehicle traveling on the road surface, a method of estimating the uneven shape of the road surface using an MMS (Mobile Mapping System) equipped with a laser scanner, and a method such as Structure from motion (SfM), which estimates the uneven shape of the road surface from multiple images taken from different viewpoints.

[0004] For example, Patent Document 1 discloses a method for estimating the longitudinal shape of a road surface based on the acceleration in the direction perpendicular to the road surface on which a vehicle is in contact and the angular velocity about a pitch axis. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-185276 A Summary of the Invention [Problem to be solved by the invention]

[0006] The inventors have found various problems in existing road surface shape estimation methods. For example, the road surface shape estimation method using SfM requires a huge amount of calculations to construct a 3D model, and it is difficult to estimate the road surface shape with high accuracy. Furthermore, since this method requires road surface images from multiple viewpoints, it tends to take a long time to estimate the road surface shape.

[0007] Therefore, an object of the present invention is to provide a road surface shape estimation device and the like that can estimate the three-dimensional shape of a road surface simply and with high accuracy. [Means for solving the problem]

[0008] One aspect of the present invention provides a road surface shape estimation device that includes an identification unit that compares a first bird's-eye view obtained by converting a first image of the road surface taken from a first position into a bird's-eye view, and a second bird's-eye view obtained by converting a second image of the road surface taken from a second position different from the first position into a bird's-eye view, and identifies, for multiple portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the portions, and an estimation unit that calculates the amount of movement on the image between the first and second portions, and estimates the uneven shape of the road surface based on the amount of movement.

[0009] According to this aspect, when there are unevenness in the road surface, part of the image converted into a bird's-eye view is distorted. By comparing the degree of distortion caused by bird's-eye view conversion of a pair of images taken from different viewpoints, the uneven shape of the road surface can be estimated with high accuracy from only two images.

[0010] In the above aspect, it is preferable that the identification unit identifies the first and second portions by matching the first bird's-eye view and the second bird's-eye view. The identification unit may match feature points in the first and second bird's-eye views, or match all pixels included in at least a portion of the first and second bird's-eye views. According to this aspect, it is possible to identify the first portion in the first bird's-eye view and the second portion in the second bird's-eye view for many points on the road surface, thereby improving the accuracy of road surface shape estimation.

[0011] In the above aspect, it is preferable that matching is performed for each of the plurality of regions in which the specific portion is a plurality of regions having a predetermined size specified in the first and second bird's-eye views, and at least a portion of the region overlaps with adjacent regions. It is also preferable that the identification unit further identifies the first and second parts by matching first and second transformed views obtained by performing homography transformation on the first and second bird's-eye views, respectively. It is also preferable that the identification unit performs matching using a supervised learning model, and the teacher data of the learning model includes at least one of a combination of an image of the road surface and a corresponding unclear image, and a combination of two images in which all pixels in overlapping areas are matched. According to these aspects, it is possible to identify the first portion in the first bird's-eye view and the second portion in the second bird's-eye view for many points on the road surface, thereby improving the accuracy of road surface shape estimation.

[0012] In the above aspect, the first and second images may be images or video frames captured by a capture unit installed in a vehicle traveling on a road surface, the identification unit may identify the first and second portions for a plurality of portions of the road surface having different positions from each other in a direction substantially perpendicular to the traveling direction of the vehicle, and the estimation unit may estimate the uneven shape along the direction substantially perpendicular to the traveling direction of the vehicle. According to this aspect, the shape of the ruts on the road surface can be suitably estimated.

[0013] In the above aspect, it is preferable that the identifying unit further identifies a first and a second portion for a plurality of portions of the road surface having the same position relative to a direction substantially perpendicular to the vehicle's traveling direction, and the estimating unit further calculates an average value of the travel direction components of the calculated movement amounts for the plurality of portions of the road surface having the same position relative to a direction substantially perpendicular to the vehicle's traveling direction. According to this aspect, it is possible to estimate the shape of the rut on the road surface with higher accuracy.

[0014] In the above embodiment, the uneven shape may include at least one of ruts and potholes.

[0015] In the above aspect, the estimation unit may further calculate the dimensions of the estimated unevenness of the road surface using the estimation result for a road surface or an object having an unevenness of known dimensions as reference data. According to this aspect, it is possible to estimate the actual dimensions of the estimated unevenness.

[0016] Another aspect of the present invention provides a method for estimating a road surface shape using the road surface shape estimation device of the above aspect, and a program for causing a computer to function as the road surface shape estimation device.

[0017] Yet another aspect of the present invention provides a road surface shape estimation system including an imaging unit installed in a vehicle, an identification unit that compares a first bird's-eye view obtained by converting a first image of the road surface captured by the imaging unit from a first position into a bird's-eye view and a second bird's-eye view obtained by converting a second image of the road surface captured by the imaging unit from a second position different from the first position into a bird's-eye view, and identifies, for multiple portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the portions, and an estimation unit that calculates the amount of movement on the image between the first and second portions and estimates the uneven shape of the road surface based on the amount of movement. Effect of the Invention

[0018] The present invention provides a road surface shape estimation device and the like that can estimate the three-dimensional shape of a road surface simply and with high accuracy. [Brief description of the drawings]

[0019] [Figure 1] 1 is a schematic diagram of a road surface shape estimation system according to an embodiment of the present invention. [Diagram 2] 1 is a functional block diagram of a road surface shape estimation device according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram showing a physical configuration of a road surface shape estimation device according to an embodiment of the present invention. [Figure 4]4 is a flowchart of a process in the road surface shape estimation system according to the present embodiment. [Diagram 5] 1 shows an example of an image acquired and converted in the road surface shape estimation system according to the present embodiment. (a) is a first image taken from a first position, (b) is a second image taken from a second position, (c) is a first bird's-eye view obtained by bird's-eye-view conversion of the first image, and (d) is a second bird's-eye view obtained by bird's-eye-view conversion of the second image. In (c) and (d), a rectangular area S represents an overlapping area between the first bird's-eye view and the second bird's-eye view. (e) is a graph showing the position on the image of a shadow P of a linear structure included in the area S in (c) and (d) in two-dimensional coordinates. [Figure 6] 3 is a schematic diagram of estimation in an estimation unit of the road surface shape estimation device according to the present embodiment. FIG. [Figure 7] 4 is a flowchart of a process in which the road surface shape estimation system according to the present embodiment estimates a rut shape on a road surface. [Figure 8] Graphs showing examples of output when estimating the shape of a rut using the road surface shape estimation system according to this embodiment. (a) A graph plotting the x coordinate on the image and the y-axis component Δy of the amount of movement from the first part to the second part for multiple points included in a specified area for which a first part and a second part have been identified. (b) A graph in which the average value of Δy for each point with the same x coordinate in graph (a) is calculated. (c) A graph in which graph (b) has been made continuous. (d) A graph in which graph (c) has been smoothed. (e) A graph in which graph (d) has been converted into the depth of a rut. [Figure 9] Fig. 1 is a diagram showing an example of a three-dimensional shape estimated by the road surface shape estimation system according to the present embodiment. (a) is a diagram showing an example of estimating a three-dimensional shape by dividing into a plurality of regions. (b) is a diagram showing the three-dimensional shapes of a rut and (c) a pothole estimated by the road surface shape estimation system according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described with reference to the accompanying drawings, in which the same reference numerals denote the same or similar configurations.

[0021] [Road surface shape estimation system] FIG. 1 is a diagram showing an outline of the configuration of a road surface shape estimation system 1 according to this embodiment. The road surface shape estimation system 1 includes a vehicle 30, a photographing unit 20 installed on the vehicle 30, and a road surface shape estimation device 10. In the road surface shape estimation system 1, the photographing unit 20 on the vehicle 30 photographs a video or a photograph of the road surface. The road surface shape estimation device 10 extracts one or more pairs of images from the video or photograph photographed by the photographing unit 20, and estimates the three-dimensional shape of the road surface based on the pair of images. The road surface shape estimation device 10 converts the pair of images into bird's-eye views and compares the pair of bird's-eye views. The road surface shape estimation device 10 utilizes the fact that a part of the image converted into a bird's-eye view is distorted when there are unevenness on the road surface, and estimates the three-dimensional shape of the road surface by comparing the degree of distortion caused by the bird's-eye view conversion of a pair of images photographed from different viewpoints.

[0022] That is, the road surface shape estimation system 1 includes an imaging unit 20 installed in a vehicle 30, and a road surface shape estimation device 10 which compares a first bird's-eye view obtained by converting a first image taken by the imaging unit 20 of the road surface from a first position into a bird's-eye view, and a second bird's-eye view obtained by converting a second image taken by the imaging unit 20 of the road surface from a second position different from the first position into a bird's-eye view, identifies, for multiple portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to those portions, calculates the amount of movement on the image between the first and second portions, and estimates the uneven shape of the road surface based on the amount of movement.

[0023] In the road surface shape estimation system 1, the road surface shape estimation device 10 is connected to the photographing unit 20 via a communication network N. Here, the communication network N may be a wired or wireless communication network. Note that the road surface shape estimation device 10 does not necessarily have to be an independent device from the photographing unit 20, and may be configured integrally with the photographing unit 20. In that case, the information processing terminal including the photographing unit 20 may also function as the road surface shape estimation device 10 by executing a road surface estimation program installed in the information processing terminal.

[0024] The imaging unit 20 is not particularly limited as long as it has a configuration capable of capturing video or images. The imaging unit 20 may be, for example, a camera or a general-purpose information processing terminal such as a smartphone or a tablet.

[0025] The photographing unit 20 is fixedly installed on the vehicle 30. That is, it moves in response to the movement of the vehicle 30. In the estimation by the road surface shape estimation device 10, the photographing unit 20 does not necessarily need to be installed on the vehicle, but in this specification, a case will be described in which the photographing unit 20 is installed on the vehicle 30 and moves with the movement of the vehicle 30. If the photographing unit 20 is not installed on the vehicle, it is preferable that the photographing unit 20 is held by a holder having a mechanism that can move the photographing unit 20 while keeping the height from the road surface constant. This can improve the accuracy of the estimation by the road surface shape estimation device 10.

[0026] The photographing unit 20 may be installed in any part of the vehicle 30 as long as it is located in a position where it can photograph the road surface. For example, the photographing unit 20 may be set in front or behind the vehicle 30 in the traveling direction, and may be installed on the dashboard, windshield, rearview mirror, etc. of the vehicle 30 to photograph the road surface ahead in the traveling direction. The photographing unit 20 may obtain an image in front or behind the vehicle 30 in the traveling direction. The first position for photographing the first image, the second position for photographing the second image, and the road surface to be photographed may be substantially on the same straight line.

[0027] The photographing unit 20 photographs the road surface as a target for estimating the three-dimensional shape, and obtains a video or image of the road surface. When the photographing unit 20 obtains an image, it obtains at least two images that include the same location on the road surface within the image range. When the photographing unit 20 obtains a video, it obtains the video so that the video includes at least two frames that include the same location on the road surface within the shooting range. In this specification, unless otherwise specified, the term "image" includes frames in a video and still images. Therefore, the photographing unit 20 obtains a pair of images whose shooting ranges overlap. In other words, each of the pair of images obtained by the photographing unit 20 includes a predetermined range on the road surface, and the road surface shape estimation device 10 can estimate the three-dimensional shape of the range on the road surface that is included in both of the pair of images.

[0028] The photographing unit 20 is installed in the vehicle 30, and photographs the road surface while the vehicle 30 is traveling. This results in a first image photographed of the road surface from a first position, and a second image photographed of the road surface from a second position. Here, the heights of the first and second positions from the road surface are approximately equal. By adjusting the traveling speed of the vehicle 30 and the interval at which the photographing unit 20 captures images (this is the frame rate when capturing a video, and the continuous shooting speed when capturing an image), the range of overlapping parts in the first and second images can be changed as desired.

[0029] The first position where the first image is acquired and the second position where the second image is acquired may be spaced apart by, for example, several meters, 0.5 to 10 meters, or 1 to 5 meters. The road surface shape estimation system 1 may acquire position information of the vehicle 30 and the photographing unit 20 from an information processing terminal installed in the vehicle 30 based on the Global Navigation Satellite System (GNSS), and may select a pair of images acquired at an appropriate interval from a plurality of images acquired by the photographing unit 20 based on the position information. The function of selecting the images may be possessed by the information processing terminal installed in the vehicle 30, or may be possessed by the road surface shape estimation device 10. The information processing terminal installed in the vehicle 30 may function as the photographing unit 20.

[0030] The vehicle 30 may be an automobile that runs on a road surface with tires on four wheels. However, the vehicle 30 may be one with three wheels, two wheels, or five or more wheels. The vehicle 30 may be an automobile of any size.

[0031] (Road surface shape estimation device) FIG. 2 is a functional block diagram of the road surface shape estimation device 10. The road surface shape estimation device 10 includes an acquisition unit 11, a conversion unit 12, a determination unit 13, and an estimation unit 14. In the present embodiment, the road surface shape estimation device 10 will be described with an example in which the acquisition unit 11 acquires an image from the photographing unit 20 and the conversion unit 12 converts the image. However, for example, an information processing terminal having the function of the photographing unit 20 may have the function of the conversion unit 12. In this case, the acquisition unit 11 of the road surface shape estimation device 10 acquires a bird's-eye view that is acquired by the information processing terminal including the photographing unit 20 and further converted into a bird's-eye view. Also, as described above, when the information processing terminal including the photographing unit 20 has the function of the road surface shape estimation device 10, the information processing terminal may include the photographing unit 20, the conversion unit 12, the determination unit 13, and the estimation unit 14.

[0032] The acquisition unit 11 acquires two or more images acquired by the photographing unit 20 photographing the road surface. The acquisition unit 11 may be realized by a communication unit 10d described later. The acquisition unit 11 may acquire all of the images photographed by the photographing unit 20, or may acquire only some of the images.

[0033] The acquisition unit 11 acquires at least a pair of images whose shooting ranges overlap. The overlapping portion may be 20% or more, 30% or more, 40% or more, 50% or more, or 60% or more of each of the acquired pair of images, and may be 90% or less, 80% or less, or 70% or less.

[0034] The conversion unit 12 converts the multiple images acquired by the acquisition unit 11 into bird's-eye views. The conversion unit 12 may be realized by an image conversion program stored in the RAM 10b or ROM 10c described below and executed by the CPU 10a. The conversion unit 12 may convert all of the multiple images acquired by the acquisition unit 11, or may convert only some of the images.

[0035] The photographing unit 20 is installed in the vehicle 30 and photographs the road surface. Therefore, the image photographed by the photographing unit 20 and acquired by the acquisition unit 11 is an image of the road surface photographed from diagonally above the road surface. The conversion unit 12 converts the image of the road surface photographed from diagonally above into an image of the road surface photographed from directly above the road surface. The conversion means is not particularly limited, and a specific example will be described later. The conversion unit 12 performs bird's-eye view conversion on at least a pair of images whose photographing ranges overlap.

[0036] The identification unit 13 compares a pair of bird's-eye views, among the bird's-eye views converted by the conversion unit 12, whose shooting ranges overlap, and identifies, for a plurality of portions of the road surface, a first portion in the first bird's-eye view and a second portion in the second bird's-eye view that correspond to the portions. In other words, the identification unit 13 identifies, for a plurality of portions or points in the first bird's-eye view, a portion or point in the second bird's-eye view that corresponds to the portion. The identification unit 13 identifies a plurality of first portions and a plurality of second portions that respectively correspond to the first portions. The identification unit 13 may be realized by an input unit 10e described later, or a program stored in a RAM 10b or a ROM 10c described later and executed by a CPU 10a.

[0037] The portion of the bird's-eye view identified by the identification unit 13 may be a portion consisting of one pixel in the bird's-eye view, or may be a portion consisting of multiple pixels. The multiple pixels may be, for example, 2 to 16 pixels. Furthermore, the identification unit 13 identifies a first portion and a second portion for multiple portions of the road surface, and the multiple portions may be adjacent to each other or may be separated from each other. For example, the portions from which the first portion and the second portion are identified may be feature points of the first and second bird's-eye views, or may be structures or shadows on the road surface specified by the user.

[0038] Identification by the identification unit 13 may be performed by a user of the road surface shape estimation device 10, or may be performed by an algorithm programmed to identify corresponding parts of a pair of images. When the user identifies the first and second parts, the user may compare the first and second bird's-eye views and specify characteristic parts on the road surface, such as lanes, manholes, and shadows, in the first and second bird's-eye views. When identifying the first and second parts by an algorithm, an image matching technique can be used. A specific example will be described later.

[0039] The estimation unit 14 calculates the amount of movement on the image from the first part to the second part for each of the combinations of the first part and the second part identified by the identification unit 13, and estimates the uneven shape of the road surface based on the amount of movement. Although details will be described later, when an image of the road surface having unevenness is converted into a bird's-eye view, distortion occurs in the bird's-eye view depending on the acquisition position of the image and the uneven shape of the road surface. The estimation unit 14 quantifies the distortion by calculating the amount of movement on the image from the first part to the second part, and quantitatively estimates the uneven shape of the road surface based on the amount of movement. The estimation unit 14 may be realized by a program stored in the RAM 10b or ROM 10c described later and executed by the CPU 10a.

[0040] The estimation unit 14 can estimate the three-dimensional shape of the road surface, but does not necessarily need to estimate the three-dimensional shape. For example, the estimation unit 14 may estimate the cross-sectional shape of the road surface in a predetermined direction. The estimation unit 14 may estimate the transverse shape of the road surface, that is, the cross-sectional shape of the road surface in the transverse direction of the road surface (a direction substantially perpendicular to the direction in which the vehicle travels). The estimation unit 14 may estimate, for example, at least one type of ruts and potholes on the road surface.

[0041] The road surface shape estimation device 10 may output the unevenness of the road surface estimated by the estimation unit 14 in an appropriate format. For example, the road surface shape estimation device 10 may display the estimation result on a display unit 10f described below, or may transmit the estimation result to another terminal via the communication unit 10d. The estimation result may be a three-dimensional or two-dimensional graph, or may be numerical data.

[0042] FIG. 3 is a diagram showing a physical configuration of the road surface shape estimation device 10. The road surface shape estimation device 10 has a CPU (Central Processing Unit) 10a equivalent to a processor, a RAM (Random Access Memory) 10b and a ROM (Read Only Memory) 10c equivalent to a storage unit, a communication unit 10d, an input unit 10e, and a display unit 10f. These components are connected to each other via a bus so that they can transmit and receive data. In this example, the road surface shape estimation device 10 is configured by one computer, but the road surface shape estimation device 10 may be realized by combining multiple computers. The configuration shown in FIG. 3 is an example, and the road surface shape estimation device 10 may have other configurations than these, or may not have some of these configurations.

[0043] The CPU 10a is a control unit that controls the execution of programs stored in the RAM 10b or the ROM 10c and calculates and processes data. The CPU 10a is a calculation unit that executes a program (road surface shape estimation program) that estimates the shape of a road surface based on an image acquired by the photographing unit 20. The CPU 10a receives various data from the input unit 10e and the communication unit 10d, and displays the calculation results of the data on the display unit 10f or stores them in the RAM 10b or the ROM 10c.

[0044] The RAM 10b is a rewritable storage device, and may be implemented, for example, by a semiconductor memory device. The RAM 10b may store a road surface shape estimation program executed by the CPU 10a. Note that these are merely examples, and the RAM 10b may store other data.

[0045] The ROM 10c is a storage unit from which data can be read, and may be configured, for example, with a semiconductor memory element. The ROM 10c may store, for example, an image editing program and data that is not rewritten.

[0046] The communication unit 10d is an interface that connects the road surface shape estimation device 10 to other devices. The communication unit 10d may be connected to a communication network N such as the Internet.

[0047] The input unit 10e receives data input from a user, and may include, for example, a keyboard and a touch panel.

[0048] The display unit 10f visually displays the results of the calculations performed by the CPU 10a, and may be configured with, for example, an LCD (Liquid Crystal Display). The display unit 10f may display a graph showing the estimated road surface shape.

[0049] The road surface shape estimation program may be provided by being stored in a computer-readable storage medium such as RAM 10b or ROM 10c, or may be provided via a communication network connected by communication unit 10d. In road surface shape estimation device 10, CPU 10a executes the road surface shape estimation program to realize the operations of acquisition unit 11, conversion unit 12, identification unit 13, and estimation unit 14 described using FIG. 2. Note that these physical configurations are merely examples and do not necessarily have to be independent configurations. For example, road surface shape estimation device 10 may include an LSI (Large-Scale Integration) in which CPU 10a is integrated with RAM 10b and / or ROM 10c.

[0050] (Processing flow) <Summary> 4 is a flowchart of processing in the road surface shape estimation system 1 of this embodiment. First, in the road surface shape estimation system 1, the photographing unit 20 acquires a first image of the road surface captured from a first position, and a second image of the road surface captured from a second position. The photographing unit 20 transmits the first image and the second image to the road surface shape estimation device 10 (S10). Here, the photographing unit 20 may further acquire one or more other images and transmit them to the road surface shape estimation device 10.

[0051] Next, the road surface shape estimation device 10, which has acquired the first image and the second image, converts the first image and the second image into a bird's-eye view in the conversion unit 12 (S11). The conversion unit 12 performs the bird's-eye view conversion, for example, according to the following formula. The following formula is a formula for converting an image acquired by the image capturing unit 20 from diagonally above the road surface into an image acquired by a virtual camera installed directly above the road surface.

[0052]

number

[0053] Here, x and y are pixel positions in the image before transformation, and x' and y' are pixel positions in the image after transformation. Also, f and f' are focal lengths of the image capturing unit 20 and the virtual camera in pixel conversion, respectively, θ is the angle between the image capturing unit 20 and the road surface, and H VC is the height of the virtual camera from the road surface, and D VC is the parallel distance between the image capture unit 20 and the virtual camera, and H C is the height of the image capturing unit 20 from the road surface. The bird's-eye view can be calibrated by adjusting various parameters in the virtual camera. For more details, see S. Tanaka et al., International Journal of Vehicular Technology, Volume 2011, Article ID 279739 (2011).

[0054] In the conversion unit 12, various parameters of the virtual camera may be set by a user of the road surface shape estimation device 10 or may be set by the manufacturer of the road surface shape estimation device 10. Alternatively, the road surface shape estimation device 10 may automatically set the various parameters of the virtual camera using an algorithm designed to improve the accuracy of road surface shape estimation while adjusting the various parameters. Alternatively, various parameters may be adjusted manually or automatically so that parts of the road surface whose shapes are known, such as lanes and manholes, have the known shapes in the converted bird's-eye view.

[0055] FIG. 5 is a diagram showing an example of a pair of images acquired in step S10 and converted in step S11 in the road surface shape estimation system 1 of this embodiment. FIGS. 5(a) and (b) are images acquired by a photographing unit installed in a traveling vehicle. FIG. 5(a) is an image of the road surface photographed from a first position, and FIG. 5(b) is an image photographed from a second position several meters forward in the photographing direction from the first position. FIGS. 5(c) and (d) are a pair of bird's-eye views obtained by bird's-eye conversion of the images in FIGS. 5(a) and (b), respectively. In FIGS. 5(c) and (d), a rectangular area S is an area overlapping between the first bird's-eye view and the second bird's-eye view, and the road surface shape estimation device 10 can estimate the road surface shape of this area S.

[0056] 4, following step S11, the identification unit 13 of the road surface shape estimation device 10 compares the pair of bird's-eye views converted by the conversion unit 12, and identifies, for multiple parts of the road surface, a first part in the first bird's-eye view and a second part in the second bird's-eye view that correspond to the parts (S12). Here, an example will be described in which the user specifies characteristic parts on the road surface, such as lanes, manholes, and shadows, and the identification unit 13 identifies the positions of the characteristic parts.

[0057] In step S12, the identification unit 13 identifies pixels in the first and second bird's-eye views that correspond to a characteristic portion on the road surface specified by the user, and the position of the pixel on the image. For each pixel of the characteristic portion specified in the first bird's-eye view, the identification unit 13 identifies a corresponding pixel of the characteristic portion specified in the second bird's-eye view. The corresponding pixel may be a pixel located at the same position in a predetermined direction on the image, or may be a characteristic point of the characteristic portion.

[0058] In the example of a pair of images shown in FIG. 5, for example, a user may specify a shadow P of a linear structure included in the region S in FIG. 5(c) and (d) as a corresponding location. After the user specifies the shadow P of the linear structure as a corresponding location, the identification unit 13 identifies a pixel corresponding to the shadow and its position on the image. FIG. 5(e) is a diagram showing the position on the image of the shadow P of the linear structure in FIG. 5(c) and (d) in the vertical direction (y direction) and horizontal direction (x direction) of the image. For example, the identification unit 13 may associate each pixel of the shadow P of the linear structure in the first bird's-eye view with a pixel of the shadow P of the linear structure in the second bird's-eye view having the same x coordinate value. Alternatively, for a pixel corresponding to each feature point (for example, a maximum point or a minimum point in the xy coordinates) of the shadow P of the linear structure in the first bird's-eye view, the identification unit 13 may associate a pixel corresponding to the corresponding feature point of the shadow P of the linear structure in the second bird's-eye view with a pixel.

[0059] Returning to Fig. 4, following step S12, the estimation unit 14 of the road surface shape estimation device 10 calculates the amount of movement on the image from the first portion to the second portion for each corresponding portion (each of the first portion and second portion) in the first and second bird's-eye views identified by the identification unit 13 (S13). The amount of movement is, for example, the amount of movement in pixel terms of the image. For example, the amount of movement may be found by calculating the difference between the pixel-converted coordinates of the first portion in the first bird's-eye view and the pixel-converted coordinates of the second portion in the second bird's-eye view.

[0060] Next, the estimation unit 14 of the road surface shape estimation device 10 estimates the uneven shape of the road surface based on the movement amount of each part calculated in step S13 (S14). Here, the following relationship holds between the length in the bird's-eye view and the depth of the road surface, assuming the pinhole camera model shown in FIG. 6. In the following formula, P' (P1', P2') and P'' (P1'', P2'') are the positions on the image of the corresponding points in the second bird's-eye view and the first bird's-eye view, respectively, and f and H VC is the same as in the above formulas (1) and (2), and P1P2 is the depth from the road surface, and d1, d2, α, and β are lengths or angles defined in FIG.

[0061]

number

[0062] Therefore, in step S14, the estimation unit 14 may estimate the relative coordinates in the pixel-converted depth direction of each portion in which the corresponding portion is identified in the first and second bird's-eye views according to the following formula. In the following formula, Δy is a component parallel to the traveling direction of the vehicle (i.e., a component in the direction from the first position to the second position) in the movement amount calculated in step S13, and other variables are the same as in the above formula (3). Therefore, the estimation unit 14 can estimate the depth of the road surface based on the angle at which the photographing unit 20 photographs the road surface at the first position where the first image is acquired, the angle at which the photographing unit 20 photographs the road surface at the second position where the second image is acquired, the focal length of the photographing unit 20, the height of the virtual camera from the road surface set in advance, and the movement amount calculated in step S13. Alternatively, the estimation unit 14 may estimate the depth of the road surface by performing calibration using the value expressed by formula (4-A), which is the coefficient of Δy, as a variable. A specific example of the calibration method will be described later in the section "Estimation of actual dimensions of uneven shape."

[0063]

number

number

[0064] It can be seen from the above formula (4) that the estimation unit 14 can estimate the relative coordinates in the depth direction in pixel terms and / or the depth of the road surface for each point identified by the identification unit 13. In the example shown in Fig. 5, the estimation unit 14 can estimate the uneven shape of the road surface in a portion where the shadow P of a linear structure included in the area S is located. When the identification unit 13 identifies the correspondence between the portions or points in the first bird's-eye view and the portions or points in the second bird's-eye view for a plurality of portions or points in a predetermined range of the road surface, it can estimate the three-dimensional shape of the range.

[0065] <Estimation of actual size of uneven shape> The road surface shape estimation system 1 of this embodiment can estimate the unevenness of a road surface using the above processing flow, but the road surface shape estimation system 1 may further estimate the actual size of the estimated unevenness using the estimation result for a road surface or object having an unevenness with known dimensions as reference data. That is, the road surface shape estimation system 1 may calibrate the value expressed by the above formula (4-A) using the estimation result for a road surface or object having an unevenness with known dimensions as reference data.

[0066] The reference data may be obtained, for example, by placing an object with known dimensions (e.g., a flat plate with known thickness) on the road surface and applying the above estimation as described with reference to Fig. 4 to the object. This allows the pixel-converted length obtained as the estimation result to correspond to the actual size length. Alternatively, the estimation result obtained by an existing road surface shape estimation method such as SfM may be used as the reference data.

[0067] <Processing of specific parts> In step S12 of the above-mentioned processing flow, the identification unit 13 may be executed by an algorithm programmed to identify corresponding parts of a pair of images. For example, the identification unit 13 may match the first bird's-eye view with the second bird's-eye view to identify, for multiple parts of the road surface, a first part in the first bird's-eye view and a second part in the second bird's-eye view that correspond to the part. The matching can be executed by an appropriate algorithm related to image processing.

[0068] In the matching, the specification unit 13 may match feature points in the first and second bird's-eye views, or may match all pixels included in at least a part of the first and second bird's-eye views. A method for matching feature points in the first and second bird's-eye views may be, for example, a method including feature point detection, feature amount descriptor calculation, and feature point matching, or may be a method for performing end-to-end matching using machine learning.

[0069] Examples of algorithms that include feature point detection, feature descriptor calculation, and feature point matching include SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), and Harris features. Examples of algorithms that perform end-to-end matching using machine learning include LoFTR (Local Feature TRansformer), GMflow, and ml-aspanformer.

[0070] As a method for matching all pixels included in at least a part of the first and second bird's-eye views, for example, matching based on optical flow can be given. As an optical flow algorithm, for example, the Lucas-Kanad method, the Horn-Schunck method, and the Farneback method can be given. As a matching algorithm by the specification unit 13, LoFTR is preferable.

[0071] Before or after the matching in the identification unit 13, pre-processing or post-processing may be performed to increase the number of detected feature points or to improve the matching accuracy. Examples of such processing include adjusting the brightness and contrast of the image, cropping the image, and removing mismatches. Examples of image cropping include cropping only the range in which it is desired to identify corresponding parts in the first bird's-eye view and the second bird's-eye view. Examples of removing mismatches include removing feature points for which a predetermined component of the amount of movement is equal to or greater than a threshold value when the estimation unit 14 calculates the amount of movement between the first part and the second part after matching by the identification unit 13.

[0072] Furthermore, the specification unit 13 may divide a pair of bird's-eye views into a plurality of regions each having a specified size and perform matching for each region, instead of performing matching at once. Also, by setting the regions so that they overlap with adjacent regions, many feature points can be matched even in the portions near the boundaries of each region.

[0073] In addition to the first and second parts identified by matching the first bird's-eye view and the second bird's-eye view, the identification unit 13 may match first and second transformed views obtained by performing homography transformation on the first and second bird's-eye views, respectively, in order to identify other first and second parts. In this way, by matching the first and second transformed views obtained by performing homography transformation on the first and second bird's-eye views, respectively, it is possible to match feature points that were not matched before transformation, and it is possible to identify more first and second parts. Examples of such homography transformation include enlargement, reduction, translation, rotation, and skew.

[0074] In addition, when matching is performed in the identification unit 13 using a machine learning model such as LoFTR, more first and second parts can be identified by having the training data include at least one of a combination of an image of the road surface and a corresponding unclear image, and a combination of two images in which all pixels in overlapping areas are matched.

[0075] By including a combination of an image of the road surface and a corresponding unclear image, the first and second parts can be identified with high accuracy even if motion blur occurs in the image acquired by the image capturing unit 20. Such teacher data can be obtained by preparing a combination of a (clear) image of the road surface and an image to which motion blur has been intentionally added.

[0076] By including a combination of two images in which all pixels in the overlapping areas are associated with each other, it is possible to further improve the accuracy of feature point matching in the identification unit 13. Such teacher data can be obtained by first performing feature point matching on a pair of bird's-eye views including an overlapping area to match a plurality of feature points, performing projective transformation on these feature points and other images so that the pair of bird's-eye views overlap, and then matching the overlapping points that have not yet been matched as corresponding points.

[0077] <Estimation of rut shape> The road surface shape estimation system 1 of this embodiment can estimate the rut shape of the road surface when the first and second parts identified by the identification unit 13 include multiple parts having different positions in a direction approximately perpendicular to the direction in which the vehicle travels on the road surface (the transverse direction of the road surface).

[0078] Fig. 7 is a flowchart of a process in which the road surface shape estimation system 1 of this embodiment estimates the shape of ruts on a road surface. Steps S10 to S13 may be the same as the process shown in the flowchart in Fig. 4. However, the image acquired in step S10 is an image captured by the photographing unit 20 installed in the vehicle 30 traveling on the road surface. Also, in step S12, it is preferable that the identifying unit 13 matches the first bird's-eye view with the second bird's-eye view to identify a large number of first and second parts in the overlapping area in the pair of images.

[0079] In the following description, the cross direction of the road surface is defined as the x-axis, and the direction in which the vehicle travels on the road surface is defined as the y-axis. Fig. 8 is a graph showing an example of an output when estimating the shape of a rut by the road surface shape estimation system according to this embodiment. Fig. 8(a) is a graph plotting the x coordinate on the image (the position in the transverse direction of the road surface) and the y-axis component Δy of the amount of movement from the first portion to the second portion for a plurality of points at which the first portion and the second portion are identified and which are included in a predetermined area having a width of 480 pixels in the y-axis direction.

[0080] In estimating the shape of the rut, it is assumed that the shape of the rut is constant within a predetermined range in the y-axis direction, and a plurality of points in which the first and second parts are identified and included in a predetermined range in the y-axis direction of the first or second image are used for the estimation, as shown in FIG. 8(a). In step S20, for a plurality of points in which the first and second parts are identified and included in a predetermined range in the y-axis direction of the first or second image, the average value of the travel direction component of the movement amount calculated in step S13 is calculated for each part having the same position in the transverse direction of the road surface (S20). In the example shown in FIG. 8, the graph shown in FIG. 8(b) is obtained by calculating the average value of Δy for each point having the same x coordinate shown in FIG. 8(a).

[0081] In step S20, discrete data is obtained, and then the discrete data is made continuous and smoothed (S21, S22). Smoothing can be performed by calculating a moving average. In the example shown in Figure 8, the graphs shown in Figures 8(c) and 8(d) are obtained by making the data continuous and smoothing the data.

[0082] Ideally, both ends of the road surface should have a rut depth of 0, so the baseline of the graph obtained in step S22 is then corrected so that the rut depth at both ends of the road surface is 0 (S23). The line segment connecting both ends of the graph obtained in step S22 is assumed to represent a road surface without ruts, and the distance between each point on the graph and the line segment is estimated as the rut depth in pixels. Such a correction can be performed using the following formula. In the following formula, d i is the rut depth (in pixels) at point i, and x pi and y pi are the x and y coordinates of point i before transformation, and x pl and y pl are the x and y coordinates of the leftmost point before transformation, and x pr and y pr are the x and y coordinates of the rightmost point before transformation, and the operations in the numerator and denominator are the determinant and norm of the matrix, respectively.

[0083]

number

[0084] In the example shown in Fig. 8, by correcting the baseline, the graph shown in Fig. 8(e) is obtained as the rut shape in the area including each part shown in Fig. 8(a). As described in detail in <Estimation of actual dimensions of uneven shape>, the actual depth of the rut may be further estimated using the estimation result for a road surface or object having an uneven shape with known dimensions as reference data.

[0085] In the above process flow, any of steps S21 to S23 may be omitted, or another new step may be added.

[0086] <3D shape estimation> In estimating the shape of the rut, a plurality of points at which the first and second parts are identified and included in a predetermined range in the traveling direction of the vehicle are used for the estimation, but by dividing the range into a plurality of areas and performing the estimation multiple times, the three-dimensional shape of the area can be estimated with high accuracy. That is, in estimating the three-dimensional shape, the area to be estimated is divided into a plurality of areas, the cross-sectional shape of the road surface is estimated for each area as described with reference to FIG. 7, and the estimation results are recombined, thereby improving the accuracy of the estimation. The direction in which the areas are divided is not particularly limited, and may be a direction from the first position to the second position, or may be a direction substantially perpendicular to the direction.

[0087] Fig. 9 is a diagram showing the estimation of a three-dimensional shape by the road surface shape estimation system 1. As shown in Fig. 9(a), the area for estimating the three-dimensional shape is divided into a plurality of areas, the cross-sectional shape of the road surface is estimated for each area as described with reference to Fig. 7, and the estimation results are recombined, so that the three-dimensional shape of the road surface can be estimated with high accuracy as shown in Fig. 9(b). Furthermore, according to this method, the shape of potholes on the road surface can also be estimated as shown in Fig. 9(c).

[0088] The above-described embodiments are intended to facilitate understanding of the present invention, and are not intended to limit the present invention. The elements of the embodiments, as well as their arrangements, materials, conditions, shapes, sizes, etc., are not limited to those shown as examples and can be changed as appropriate. In addition, configurations shown in different embodiments can be partially substituted or combined with each other.

[0089] [Note] The present invention includes the following embodiments. [1] an identification unit that compares a first bird's-eye view obtained by converting a first image of a road surface taken from a first position into a bird's-eye view and a second bird's-eye view obtained by converting a second image of the road surface taken from a second position different from the first position into a bird's-eye view, and identifies, for a plurality of portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the portions; an estimation unit that calculates an amount of movement between the first portion and the second portion on an image and estimates an uneven shape of the road surface based on the amount of movement; A road surface shape estimation device comprising: [2] the identification unit identifies the first and second parts by matching the first bird's-eye view with the second bird's-eye view; The road surface shape estimation device according to [1]. [3] the identification unit performs identification by matching feature points in the first and second bird's-eye views, or by matching all pixels included in at least a portion of the first and second bird's-eye views; The road surface shape estimation device according to [2]. [4] the identifying unit performs matching for each of a plurality of regions having a predetermined size specified in the first and second bird's-eye views, at least a portion of each of the regions overlapping with an adjacent region; The road surface shape estimation device according to [2] or [3]. [5] the identification unit further identifies the first and second parts by matching first and second transformed views obtained by performing homography transformation on the first and second bird's-eye views, respectively; The road surface shape estimation device according to any one of [2] to [4]. [6] The identification unit performs matching using a supervised learning model, The training data of the learning model includes at least one of a combination of an image of a road surface and a corresponding unclear image, and a combination of two images in which all pixels in overlapping areas are associated with each other. The road surface shape estimation device according to any one of [2] to [5]. [7] The first and second images are frames of images or videos captured by a capture unit installed in a vehicle traveling on the road surface, The identification unit identifies the first and second portions for a plurality of portions of the road surface having positions different from each other in a direction substantially perpendicular to the traveling direction of the vehicle, The estimation unit estimates an uneven shape along a direction substantially perpendicular to a traveling direction of the vehicle. The road surface shape estimation device according to any one of [1] to [6]. [8] The identification unit further identifies the first and second portions for a plurality of portions of the road surface having the same position relative to a direction substantially perpendicular to the traveling direction of the vehicle, the estimation unit further calculates an average value of the travel direction components of the calculated movement amounts for a plurality of portions of the road surface that are at the same position relative to a direction substantially perpendicular to the vehicle travel direction; The road surface shape estimation device according to [7]. [9] The uneven shape includes at least one of ruts and potholes; The road surface shape estimation device according to any one of [1] to [8].

[10] The estimation unit further calculates dimensions of the estimated unevenness of the road surface using an estimation result for a road surface or an object having an unevenness of known dimensions as reference data. The road surface shape estimation device according to any one of [1] to [9].

[11] comparing a first bird's-eye view obtained by converting a first image of a road surface taken from a first position into a bird's-eye view with a second bird's-eye view obtained by converting a second image of the road surface taken from a second position different from the first position into a bird's-eye view, and identifying, for a plurality of portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the portions; calculating an amount of movement between the first portion and the second portion on an image, and estimating an uneven shape of the road surface based on the amount of movement; A method for estimating road surface shape.

[12] A camera installed in the vehicle; an identification unit that compares a first bird's-eye view obtained by converting a first image of a road surface captured by the photographing unit from a first position into a bird's-eye view and a second bird's-eye view obtained by converting a second image of the road surface captured by the photographing unit from a second position different from the first position into a bird's-eye view, and identifies, for multiple portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the portions; an estimation unit that calculates an amount of movement between the first portion and the second portion on an image and estimates an uneven shape of the road surface based on the amount of movement; A road surface shape estimation system comprising:

[13] Computer, an identification unit that compares a first bird's-eye view obtained by converting a first image of a road surface taken from a first position into a bird's-eye view and a second bird's-eye view obtained by converting a second image of the road surface taken from a second position different from the first position into a bird's-eye view, and identifies, for a plurality of portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the respective portions; and an estimation unit that calculates an amount of movement between the first portion and the second portion on an image and estimates an uneven shape of the road surface based on the amount of movement; A road surface shape estimation program that functions as a road surface shape estimation program. [Explanation of symbols]

[0090] 1...road surface shape estimation system, 10...road surface shape estimation device, 10a...CPU, 10b...RAM, 10c...ROM, 10d...communication unit, 10e...input unit, 10f...display unit, 11...acquisition unit, 12...conversion unit, 13...identification unit, 14...estimation unit, 20...photography unit, 30...vehicle, N...communication network, S...area.

Claims

1. an identification unit that compares a first bird's-eye view obtained by converting a first image of a road surface taken from a first position into a bird's-eye view and a second bird's-eye view obtained by converting a second image of the road surface taken from a second position different from the first position into a bird's-eye view, and identifies, for a plurality of portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the portions; an estimation unit that calculates an amount of movement between the first portion and the second portion on an image and estimates an uneven shape of the road surface based on the amount of movement; Equipped with the first and second images are frames of images or videos captured by a capture unit installed in a vehicle traveling on the road surface, The identification unit identifies the first and second portions for a plurality of portions of the road surface having different positions from each other in a direction substantially perpendicular to the traveling direction of the vehicle, and a plurality of portions of the road surface having the same position from each other in a direction substantially perpendicular to the traveling direction, the estimation unit estimates an uneven shape along a direction substantially perpendicular to the traveling direction by calculating an average value of components of the traveling direction of the calculated movement amounts for a plurality of portions of the road surface having the same positions relative to a direction substantially perpendicular to the traveling direction. Road surface shape estimation device.

2. the identification unit identifies the first and second parts by matching the first bird's-eye view with the second bird's-eye view; The road surface shape estimation device according to claim 1 .

3. the identification unit performs identification by matching feature points in the first and second bird's-eye views, or by matching all pixels included in at least a partial area of ​​the first and second bird's-eye views; The road surface shape estimation device according to claim 2 .

4. the identifying unit performs matching for each of a plurality of regions having a predetermined size specified in the first and second bird's-eye views, at least a portion of each of the regions overlapping with an adjacent region; The road surface shape estimation device according to claim 2 .

5. the identification unit further identifies the first and second parts by matching first and second transformed views obtained by performing homography transformation on the first and second bird's-eye views, respectively; The road surface shape estimation device according to claim 2 .

6. The identification unit performs matching using a supervised learning model, The training data of the learning model includes at least one of a combination of an image of a road surface and a corresponding unclear image, and a combination of two images in which all pixels in overlapping areas are associated with each other. The road surface shape estimation device according to claim 2 .

7. The uneven shape includes at least one of ruts and potholes; The road surface shape estimation device according to any one of claims 1 to 6.

8. The estimation unit further calculates dimensions of the estimated unevenness of the road surface using an estimation result for a road surface or an object having an unevenness of known dimensions as reference data. The road surface shape estimation device according to any one of claims 1 to 6.

9. A computer comprising: comparing a first bird's-eye view obtained by converting a first image of a road surface taken from a first position into a bird's-eye view with a second bird's-eye view obtained by converting a second image of the road surface taken from a second position different from the first position into a bird's-eye view, and identifying, for a plurality of portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the portions; calculating an amount of movement between the first portion and the second portion on an image, and estimating an uneven shape of the road surface based on the amount of movement; the first and second images are frames of images or videos captured by a capture unit installed in a vehicle traveling on the road surface, The identifying step includes identifying the first and second portions for a plurality of portions of the road surface having different positions from each other in a direction substantially perpendicular to the traveling direction of the vehicle, and a plurality of portions of the road surface having the same positions from each other in a direction substantially perpendicular to the traveling direction, the estimating includes estimating an uneven shape along a direction substantially perpendicular to the traveling direction by calculating an average value of components of the traveling direction of the calculated movement amount for a plurality of portions of the road surface having the same position relative to a direction substantially perpendicular to the traveling direction; A method for estimating road surface shape.

10. A camera installed in the vehicle; an identification unit that compares a first bird's-eye view obtained by converting a first image of a road surface captured by the photographing unit from a first position into a bird's-eye view and a second bird's-eye view obtained by converting a second image of the road surface captured by the photographing unit from a second position different from the first position into a bird's-eye view, and identifies, for a plurality of portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the portions; an estimation unit that calculates an amount of movement between the first portion and the second portion on an image and estimates an uneven shape of the road surface based on the amount of movement; Equipped with the first and second images are frames of an image or a video captured by the image capture unit, The identification unit identifies the first and second portions for a plurality of portions of the road surface having different positions from each other in a direction substantially perpendicular to the traveling direction of the vehicle, and a plurality of portions of the road surface having the same position from each other in a direction substantially perpendicular to the traveling direction, the estimation unit estimates an uneven shape along a direction substantially perpendicular to the traveling direction by calculating an average value of components of the traveling direction of the calculated movement amounts for a plurality of portions of the road surface having the same positions relative to a direction substantially perpendicular to the traveling direction. Road surface shape estimation system.

11. Computer, an identification unit that compares a first bird's-eye view obtained by converting a first image of a road surface taken from a first position into a bird's-eye view and a second bird's-eye view obtained by converting a second image of the road surface taken from a second position different from the first position into a bird's-eye view, and identifies, for a plurality of portions of the road surface, first portions in the first bird's-eye view and second portions in the second bird's-eye view that correspond to the portions; and an estimation unit that calculates an amount of movement between the first portion and the second portion on an image and estimates an uneven shape of the road surface based on the amount of movement; Function as a the first and second images are frames of images or videos captured by a capture unit installed in a vehicle traveling on the road surface, The identification unit identifies the first and second portions for a plurality of portions of the road surface having different positions from each other in a direction substantially perpendicular to the traveling direction of the vehicle, and a plurality of portions of the road surface having the same position from each other in a direction substantially perpendicular to the traveling direction, the estimation unit estimates an uneven shape along a direction substantially perpendicular to the vehicle traveling direction by calculating an average value of components of the travel direction of the calculated movement amounts for a plurality of portions of the road surface having the same positions relative to a direction substantially perpendicular to the vehicle traveling direction; Road surface shape estimation program.

Images