Slope estimation device and method of operation thereof

Abstract

An operation method of a gradient estimation device is provided. The operation method of the gradient estimation device including at least one camera includes obtaining a front image through the at least one camera, detecting a lane included in the front image, dividing the front image into a plurality of smaller regions in a horizontal direction, identifying a plurality of lane segments included in each of the plurality of smaller regions, obtaining a plurality of coordinate values forming each of the plurality of lane segments, and obtaining a pitch angle of each of the plurality of smaller regions based on the obtained plurality of coordinate values.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of Korean Patent Application No. 10-2020-0066066, filed on June 1, 2020, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to an apparatus for estimating the slope of a road ahead and a method of operating the apparatus, and more specifically, to an apparatus for estimating the slope of a road ahead based on an image obtained by a camera and a method of operating the apparatus. Background Technology

[0004] Image capturing devices using image sensors can be included in various types of electronic devices, such as smartphones, personal computers (PCs), surveillance cameras, and vehicles, or can all be used as a standalone electronic device.

[0005] In autonomous vehicles, the distance to surrounding vehicles can be identified using image sensors, and based on this distance, the vehicle can be controlled to perform stable driving. However, conventional distance recognition of targets ahead involves the assumption that the driving vehicle and surrounding vehicles in the area ahead are on the same plane, thus making it difficult to apply conventional distance recognition to roads with inclines (e.g., uphill or downhill roads).

[0006] Furthermore, in addition to driving on straight roads, the actual driving environment of a vehicle can also include driving on curved roads. On curved roads, the vanishing point cannot be defined as a point, thus creating a problem where the vehicle's pitch angle cannot be estimated based on the vanishing point. Summary of the Invention

[0007] The present invention provides an apparatus for estimating the slope of a road ahead by horizontally dividing an image obtained by at least one camera, and a method for operating the apparatus.

[0008] According to one aspect of the present invention, a method of operating a slope estimation device including at least one camera is provided, the method comprising: acquiring a forward image of a vehicle accommodating the vehicle including the slope estimation device relative to the at least one camera; detecting lanes included in the forward image; dividing the forward image into a plurality of smaller regions in a horizontal direction; identifying a plurality of lane segments included in each of the plurality of smaller regions; obtaining a plurality of coordinate values ​​forming each of the plurality of lane segments; and obtaining a pitch angle of each of the plurality of smaller regions based on the obtained plurality of coordinate values.

[0009] According to another aspect of the inventive concept, there is provided a bank estimation device including at least one camera configured to obtain a front image with respect to a host vehicle in a situation in which the host vehicle is driving, and an image signal processor configured to detect a lane included in the front image, divide the front image into a plurality of smaller regions in a horizontal direction, identify a plurality of lane segments included in each of the plurality of smaller regions, obtain a plurality of coordinate values forming each of the plurality of lane segments, and obtain a pitch angle of each of the plurality of smaller regions based on the obtained plurality of coordinate values.

[0010] According to another aspect of the inventive concept, there is provided a host vehicle apparatus including at least one camera configured to obtain a front image with respect to a host vehicle in a situation in which the host vehicle is driving, an image signal processor configured to detect a lane included in the front image, divide the front image into a plurality of smaller regions in a horizontal direction, identify a plurality of lane segments included in each of the plurality of smaller regions, obtain a plurality of coordinate values forming each of the plurality of lane segments, obtain a pitch angle of each of the plurality of smaller regions based on the obtained plurality of coordinate values, identify the plurality of smaller regions from a lower end of the front image, identify a lane segment included in each of the plurality of smaller regions, obtain a pitch angle corresponding to each of the plurality of smaller regions based on a plurality of coordinate values forming the identified lane segment, and convert a coordinate of the front image into a coordinate value of a world coordinate based on the obtained pitch angle, and a vehicle controller configured to control a strength of a suspension and / or a speed of the host vehicle based on the coordinate value of the world coordinate. BRIEF DESCRIPTION OF DRAWINGS

[0011] Example embodiments of the inventive concept will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a block diagram of a bank estimation device according to an example embodiment;

[0013] Figure 2 is a side view of a host vehicle including a bank estimation device according to an example embodiment;

[0014] Figure 3 shows an example of a road coordinate according to an example embodiment;

[0015] Figure 4 shows a mutual relationship between an image coordinate and a road coordinate according to an example embodiment;

[0016] Figure 5A is a flowchart showing an operation method of a bank estimation device according to an example embodiment;

[0017] Figure 5B An example of a detection result of a front lane and a front image according to an example embodiment is shown;

[0018] Figure 5C An example of dividing a front image into a plurality of smaller regions according to an example embodiment is shown;

[0019] Figure 5D A road coordinate and an image plane corresponding to one smaller region according to an example embodiment is shown;

[0020] Figure 6 A flowchart for measuring a pitch angle of an accommodating vehicle according to an example embodiment is shown;

[0021] Figure 7A A flowchart for generating a vertical road profile according to an example embodiment is shown;

[0022] Figure 7B An example of continuity between adjacent smaller regions according to an example embodiment is shown;

[0023] Figure 7C An example of a vertical road profile according to an example embodiment is shown;

[0024] Figure 7D An example of sequentially calculating a vertical height of a smaller region from a lower end of a front image according to an example embodiment is shown;

[0025] Figure 8 An example embodiment of detecting a vertical height of a deceleration strip according to an example embodiment is shown;

[0026] Figure 9 An example embodiment using a head-up display (HUD) according to an example embodiment is shown; and

[0027] Figure 10 An example embodiment of measuring a roll of a front road according to an example embodiment is shown. DETAILED DESCRIPTION

[0028] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. Prior to providing a detailed description, the following will describe the terms.

[0029] An accommodating vehicle can mean a vehicle including a gradient estimation device according to an example embodiment. The gradient estimation device can be electrically connected or physically connected to the accommodating vehicle. The gradient estimation device can be embedded in the accommodating vehicle, or can be attached to the outside of the accommodating vehicle. The accommodating vehicle can be referred to as various terms including a self vehicle, a self vehicle, and an autonomous vehicle.

[0030] An image plane can represent a two-dimensional (2D) region onto which a real world is projected by a camera of a distance estimation device. For example, when the camera is a pinhole camera, the real world can be projected in a state in which an upper portion and a lower portion are reversed and a left portion and a right portion are reversed, and thus the real world can be displayed as the image plane.

[0031] The image plane can be a 2D region, and thus can include coordinates divided in units of pixels. The coordinates can be referred to as image coordinates. For example, when an upper left end of the image plane is set as an origin, a right direction can be represented as a u-axis, and a downward direction can be represented as a v-axis.

[0032] A world coordinate can represent a coordinate for representing a real world corresponding to an external environment of a camera of a distance estimation device. According to some example embodiments, when the camera of the slope estimation device is set as an origin, the world coordinate can be referred to as a camera coordinate. The camera coordinate or the world coordinate in which the camera of the slope estimation device is set as the origin can have an X-axis, a Y-axis, and a Z-axis. For example, the X-axis can correspond to a front direction in which the camera focuses, the Y-axis can correspond to a left direction with respect to the front direction of the camera, and the Z-axis can correspond to an upward direction with respect to the front direction of the camera. In other words, the Z-axis (hereinafter, referred to as a Z-axis) of the camera coordinate can correspond to the X-axis of the world coordinate, the Y-axis (hereinafter, referred to as a Y-axis) of the camera coordinate can correspond to the Z-axis of the world coordinate, and the X-axis (hereinafter, referred to as an X-axis) of the camera coordinate can correspond to the Y-axis of the world coordinate. C C C

[0033] A road coordinate can represent a virtual coordinate for calculating a slope of an arbitrary region placed on a front road. For example, with respect to two different points placed on a front road, the road coordinate can correspond to a world coordinate in which the two different points are disposed on an X-axis and a camera is disposed on a Z-axis.

[0034] Figure 1 is a block diagram of a slope estimation device 100 according to an example embodiment.

[0035] Referring to Figure 1 , the slope estimation device 100 can include a camera 110 and an image signal processor (ISP) 120.

[0036] The camera 110 can be embedded into an accommodation vehicle and can recognize an external environment of the accommodation vehicle. For example, the camera 110 can convert light corresponding to the external environment in a front direction or various directions into electrical energy to generate an image, and can transmit the generated image to the ISP 120.

[0037] ​​​The camera 110 can be embedded in an electronic device or can be implemented as an electronic device. For example, the electronic device can be implemented as a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. The portable electronic device can include a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital camera, a digital camcorder, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an electronic book, a wearable device, etc.

[0038] The ISP 120 can detect a lane of a front road based on an image (hereinafter, referred to as a front image) obtained through the camera 110 and can obtain a pitch angle and a vertical height of the front road. The pitch angle can indicate an elevation angle with respect to a plane that supports an accommodation vehicle. For example, the pitch angle can be referred to as various terms, such as an elevation angle and a pitch angle. The vertical height can indicate a distance spaced apart in a vertical direction from the plane that supports the accommodation vehicle. An operation of obtaining the pitch angle and the vertical height will be described below.

[0039] The ISP 120 can divide the front image into a plurality of smaller (e.g., pre-defined) regions. For example, the ISP 120 can divide the front image into a plurality of smaller regions in a horizontal direction. One of the divided plurality of smaller regions can include a portion of the detected lane. The portion of the lane included in the one smaller region can be referred to as a lane segment. For example, a first region can include an image of a first lane segment, and a second region can include an image of a second lane segment. When the first region and the second region are adjacent to each other in a vertical direction, the first lane segment and the second lane segment can be connected in a line.

[0040] The ISP 120 can estimate a pitch angle corresponding to one smaller region based on a lane segment included in each of the plurality of smaller regions. For example, the ISP 120 can estimate a slope of the divided region by using homography and road coordinates. The homography can indicate an algorithm for converting between 2D and three dimensions (3D). An operation of estimating the slope by using the ISP 120 will be described in detail below.

[0041] Figure 2 is a side view of an accommodation vehicle 200 including a slope estimation apparatus according to an example embodiment. Descriptions identical or similar to those of Figure 1 will be omitted.

[0042] Referring to Figure 1 and Figure 2 , the accommodation vehicle 200 can include the slope estimation apparatus 100 and a vehicle controller 210.

[0043] The vehicle controller 210 can control the overall travel of the host vehicle 200. The vehicle controller 210 can obtain the pitch angle of an arbitrary point on the road ahead from the slope estimation device 100, and distance information about the distance to the arbitrary point. For example, the arbitrary point can correspond to a point at which a speed bump is provided. The vehicle controller 210 can decrease the speed of the host vehicle 200 in a direction closer to the speed bump based on the distance information about the distance to the speed bump. To this end, the vehicle controller 210 can generate a control signal indicating deceleration, and can transmit the control signal to a brake system. In addition, the vehicle controller 210 can increase the strength of a suspension when the host vehicle 200 passes through the speed bump. The suspension can be a device that connects a wheel and a chassis of the host vehicle 200, and can be a device for absorbing an impact from a road surface. The vehicle controller 210 can increase the strength of the suspension when the host vehicle 200 passes through the speed bump, and thus can reduce the vertical motion of a passenger and can provide a stable ride quality.

[0044] Any of the elements disclosed herein can include or be implemented in processing circuitry, e.g., hardware including logic circuitry; a hardware / software combination such as a processor executing software; or a combination of both. For example, processing circuitry can more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc.

[0045] Figure 3 An example of road coordinates according to an example embodiment is illustrated.

[0046] Referring to Figure 3 , the host vehicle 200 can travel along a front lane, and Figure 3 may correspond to a side view of a travel situation. Referring to Figure 3 , an uphill road can be in a front view with respect to the host vehicle 200. Two points on the uphill road of the front view can be referred to as a first point p'1 and a second point p'2, respectively. The Z-axis components of the first point p'1 and the second point p'2 can have different values. That is, in world coordinates, the vertical heights of the first point p'1 and the second point p'2 can be different.

[0047] According to some example embodiments, it can be assumed that such road coordinates in which the first point p'1 and the second point p'2 are on a flat surface. In the road coordinates, the Z-axis (hereinafter referred to as Z' axis) components of the first point p'1 and the second point p'2 can each correspond to 0. That is, the first point p'1 and the second point p'2 can be on an X-axis (hereinafter referred to as X' axis) of the road coordinates. Further, it can be assumed that the camera 110 is on the Z' axis of the road coordinates. That is, in the road coordinates, the X' axis component of the camera 110 can be 0.

[0048] The world coordinates of the first point p'1 can be (X1, Y1, Z1), and the road coordinates of the first point p'1 can be (X'1, Y'1, 0). The world coordinates of the second point p'2 can be (X2, Y2, Z2), and the road coordinates of the second point p'2 can be (X'2, Y'2, 0).

[0049] As described above, it can be seen that the camera coordinates are expressed with respect to the world coordinates, and further, the road coordinates are elevated by a pitch angle with respect to the world coordinates. Therefore, the camera coordinates can be expressed with respect to the road coordinates. The mutual relationship between the camera coordinates and the road coordinates can be expressed as the following Equation 1.

[0050] [Equation 1]

[0051]

[0052] In one embodiment, h can represent a separation between the camera and the X' axis on the road coordinates, that is, can correspond to a Z' axis component value of the camera in the road coordinates, and θ can correspond to a pitch angle representing a degree of elevation of the road coordinates with respect to the world coordinates.

[0053] An object in the camera coordinates can be projected by the camera 110, and thus the front image can include an image corresponding to the object. That is, the camera coordinates can be related to image coordinates (or pixel coordinates) for expressing an image plane. The mutual relationship between the camera coordinates and the image coordinates can be expressed as the following Equation 2.

[0054] [Equation 2]

[0055]

[0056] In one embodiment, f X and f Y may represent a focal length, and c X and c Y may each represent a principal point. The internal parameters c X and c Y of the camera can each have a predetermined value or an alternative desired value for correcting a movement displacement of a center of coordinates in the image plane.

[0057] As described above, in the road coordinates, since the first point p'1 and the second point p'2 are assumed to be on a flat surface, the Z' component of the first point p'1 and the second point p'2 can be 0. Thus, when 0 is substituted for Z' in Equation 2, the following Equation 3 can be calculated.

[0058] [Equation 3]

[0059] X' · (u · cos θ - c X · cos θ) + γ' · f X = -u · h · sin θ + c X · h · sin θ

[0060] X' · (f Y · sin θ + v · cos θ - c Y · cos θ) = -v · h · sin θ + f Y · h · cos θ + c Y · h · sin θ

[0061] In one embodiment, by using a first-order Taylor series for approximation with respect to Equation 3, an arbitrary point (X', Y') of the road coordinates can be expressed as the following Equation 4.

[0062] [Equation 4]

[0063]

[0064]

[0065] Referring to Equation 4, an arbitrary point (X', Y') of the road coordinates can be expressed as a coordinate (u, v) and a pitch angle, which both correspond to the arbitrary point in the image coordinates, and can be expressed as a camera height in the road coordinates.

[0066] Figure 4 A mutual relationship between the image coordinates and the road coordinates according to an example embodiment is illustrated.

[0067] Referring to Figure 4 , a left image can correspond to an image plane. That is, the left image can correspond to an image seen through a camera 110 included in a slope estimation device 100 in a host vehicle 200 that is driving on a road.

[0068] Referring to the image plane, it can be seen that two lines are not parallel to each other. Like the two lines of the image plane, parallel lines in the world coordinates can not be parallel to each other and can face one point. This one point can be referred to as a vanishing point.

[0069] Two points having the same v component can be set by two lines of the image plane. For example, when the first coordinate is (u1, v1) and the third coordinate is (u3, v3), v1 and v3 can have the same value. As another example, when the second coordinate is (u2, v2) and the fourth coordinate is (u4, v4), v2 and v4 can have the same value. That is, in the image plane, when two parallel virtual lines are drawn in the horizontal direction, four points intersecting the line corresponding to the lane can be referred to as the first coordinate to the fourth coordinate, respectively.

[0070] According to some example embodiments, the right image can correspond to an aerial view of the world coordinate. That is, the first coordinate to the fourth coordinate can be points on the road ahead, and thus the Z' components of the first coordinate to the fourth coordinate can each be 0. When the first coordinate is converted into the world coordinate in the image plane on which the camera focuses, the first coordinate can be (X'1, Y'1, 0). The other second coordinate to the fourth coordinate are as described above.

[0071] According to some example embodiments, the width of the lane in which the host vehicle 200 is traveling can be constant, and this can be equally applied to a curved road. That is, in the road coordinate, the difference in the Y' component between the third coordinate and the first coordinate can be the same as the difference in the Y' component between the fourth coordinate and the second coordinate. That is, assuming that the width of the lane is constant, the width of the lane can be expressed as the following Equation 5.

[0072] [Equation 5]

[0073]

[0074]

[0075] In one embodiment, with reference to Equation 5, a cubic equation corresponding to the pitch angle can be obtained, and the pitch angle can be expressed as the following Equation 6.

[0076] [Equation 6]

[0077]

[0078] In one embodiment, u and v can represent coordinate values of the image plane, and f Y and c Y may represent values defined as internal parameters of the camera. That is, when the coordinate values of four points generated by the intersection of two lines detected as lanes in the front image of the image plane and two lines parallel to each other in the horizontal direction are identified, the pitch angle can be calculated.

[0079] Figure 5A is a flowchart of an operation method of a slope estimation device according to example embodiments.

[0080] Referring to Figure 5A In operation S110, the ISP 120 can obtain a front image. When the host vehicle 200 is driving along a road, the camera 110 can obtain an image with respect to a front view of the host vehicle 200 (hereinafter, referred to as a front image), and can transmit the obtained image to the ISP 120.

[0081] In operation S120, the ISP 120 can detect a lane included in the obtained front image. The ISP 120 can perform an object detection algorithm on the obtained front image.

[0082] For example, the ISP 120 can first filter yellow or white objects, and after the filtering, the ISP 120 can check whether a shape of each of the objects is a linear shape, thereby detecting only the lane. As another example, the host vehicle 200 can further include an artificial intelligence (AI)-based neural network intellectual property (IP). The neural network IP can include one or more accelerators, and can quickly perform an object detection algorithm.

[0083] In some embodiments including those discussed below, the neural network IP can utilize various artificial neural network organizations and processing models, such as a convolutional neural network (CNN), a deconvolutional neural network, a recurrent neural network (RNN) optionally including long short-term memory (LSTM) units and / or gated recurrent units (GRUs), a stacked neural network (SNN), a state space dynamic neural network (SSDNN), a deep belief network (DBN), a generative adversarial network (GAN), and / or a restricted Boltzmann machine (RBM).

[0084] Alternatively or additionally, such neural network IP can include other forms of machine learning models, such as linear and / or logistic regression, statistical clustering, Bayesian classification, decision trees, dimensionality reduction (e.g., principal component analysis), and expert systems; and / or combinations thereof including ensembles such as random forests.

[0085] Referring to Figure 5A and Figure 5B The upper image can correspond to a front image obtained by the camera, and the lower image can correspond to an image in which lane detection is completed based on an object detection algorithm. With reference to the lower image, it can be checked that objects including vehicles driving in the front view, buildings, and trees other than the lane have been filtered.

[0086] In operation S130, the ISP 120 can divide the front image into a plurality of smaller regions in a horizontal direction. Referring to Figure 5A and Figure 5CThe ISP 120 can divide the front image completed with lane detection (hereinafter, referred to as a lane detection image) in a horizontal direction into a plurality of smaller regions.

[0087] Referring to Figure 5C The pixel length in a vertical direction (hereinafter, referred to as a vertical pixel length) of the plurality of smaller regions can be the same. That is, the interval between a plurality of points parallel to each other in the horizontal direction can be the same. For example, the pixel length can correspond to a pixel length equal to 5, but example embodiments are not limited thereto, and a pixel length having a decimal unit can be used. The vertical pixel length can be flexibly varied based on the complexity of the front image, the speed of the accommodated vehicle 200, and the overload of the ISP 120.

[0088] According to some example embodiments, the vertical pixel length of the plurality of smaller regions can be different. For example, the ISP 120 can reduce the vertical pixel length of each of a plurality of lower smaller regions of the front image. That is, in the front road close to the distance of the accommodated vehicle 200, the pitch angle thereof can be more accurately measured, and thus can be used as the pitch angle of the accommodated vehicle 200. As another example, the ISP 120 can reduce the vertical pixel length of each of a plurality of upper smaller regions of the front image. The upper smaller regions of the front image can correspond to regions far from the accommodated vehicle 200. This is because even if the region is far from the accommodated vehicle 200, the pitch angle needs to be calculated in advance before approaching the region when the speed of the accommodated vehicle 200 is greater than a threshold speed or the load of the ISP 120 is equal to or greater than a predetermined threshold or alternatively equal to or greater than a desired threshold.

[0089] In operation S140, the ISP 120 can identify a lane segment included in each of the plurality of smaller regions, and can obtain coordinates of points forming the lane segment. Referring to Figure 5A and Figure 5D The left image can represent one of the plurality of smaller regions.

[0090] The lane segment can correspond to at least a portion of the lane detected in operation S120. For example, when the front image is divided into a plurality of smaller regions in the horizontal direction, the lane detected in operation S120 can be divided into a plurality of lane segments. That is, the lane segment can represent the lane included in each of the plurality of smaller regions.

[0091] The ISP 120 can obtain coordinate values of points forming the lane segment. For example, referring to Figure 5C When one smaller region is the region 510, the lane segment can represent a plurality of lines 510 and 520. In one embodiment, the plurality of points forming the lane segment can include two end points of the left line 510 and two end points of the right line 520. That is, referring to Figure 4The plurality of points forming the lane segment can include first to fourth coordinates. The two end points forming the left line 510 can include a first coordinate (u1, v1) and a second coordinate (u2, v2), and the two end points forming the right line 520 can include a third coordinate (u3, v3) and a fourth coordinate (u4, v4).

[0092] In operation S150, the ISP 120 can obtain a pitch angle of each of the plurality of smaller regions based on the obtained coordinates.

[0093] The ISP 120 can obtain the pitch angle corresponding to an arbitrary smaller region based on the obtained coordinate values by using Equation 6. For example, referring to Equation 6, the pitch angle of the smaller region 510 can be calculated by inputting the first to fourth coordinates (u1, v1) to (u4, v4) included in the image plane and the internal parameters of the camera 110. Figure 5C

[0094] According to some example embodiments, the ISP 120 can calculate the pitch angle from each of the lower smaller regions of the front image. Each of the lower smaller regions of the front image can be a region corresponding to a distance close to the accommodation vehicle 200, and thus can be assumed to be a region having a flat road with respect to the accommodation vehicle 200. Accordingly, the ISP 120 can calculate the pitch angle of each of the lower smaller regions, and can set the calculated pitch angle as the pitch angle of the accommodation vehicle 200. Accordingly, the ISP 120 can reflect the pitch angle of the accommodation vehicle 200 in real time during the travel of the accommodation vehicle 200.

[0095] Figure 6 A flowchart for measuring a pitch angle of an accommodation vehicle according to an example embodiment is illustrated.

[0096] Referring to Figure 6 , the ISP 120 can identify a predetermined plurality of smaller regions or alternatively desired plurality of smaller regions from a lower end of the front image. For example, the ISP 120 can identify five smaller regions from the lower end of the front image. The predetermined number or alternatively the desired number can be flexibly varied based on the speed of the accommodation vehicle 200 and the overload of the ISP 120.

[0097] ​That is, the ISP 120 can set at least some of the plurality of smaller regions, thereby improving reliability of the pitch angle close to the region in which the vehicle 200 is accommodated. For example, the concave-convex portion can be on the front road with respect to the vehicle 200 being accommodated. For example, when the ISP 120 identifies only one smaller region and sets the calculated pitch angle as the pitch angle of the vehicle 200 being accommodated, an error can occur. For example, this is because when the identified one smaller region is a region including the concave-convex portion, although the pitch angle of the vehicle 200 being accommodated is 0, the pitch angle having a higher value can be set as the pitch angle of the vehicle 200 being accommodated. When the pitch angle of the wide region is set as the pitch angle of the vehicle 200 being accommodated, even if the concave-convex portion is partially disposed on the front road, the total pitch angle of the front road can be set as the pitch angle of the vehicle 200 being accommodated.

[0098] In operation S220, the ISP 120 can detect a lane segment included in the identified region of the front image. The ISP 120 can detect a lane disposed in the plurality of smaller regions. For example, the length of the lane segment of the region having a vertical pixel length of 1 can be short to represent the pitch angle of the front road, but the length of the lane segment of the region having a vertical pixel length of 10 can be long enough to represent the pitch angle of the front road.

[0099] In operation S230, the ISP 120 can estimate the pitch angle of the identified region based on the detected lane segment. That is, the ISP 120 can obtain four coordinate values forming the detected lane segment, and can substitute the obtained coordinate values into Equation 6 to obtain the pitch angle of the identified region. Its description is the same as or similar to that of operations S140 and S150 of Figure 5A The description of operations S140 and S150 of

[0100] Figure 7A A flowchart for generating a vertical road profile according to an example embodiment is illustrated. The vertical road profile can represent information related to a vertical height of a front lane along which the vehicle 200 is traveling.

[0101] Referring to Figure 7A In operation S310, the ISP 120 can obtain the pitch angle of the lowermost smaller region of the front image. For example, in conjunction with Figure 7B Referring to Figure 7AThe lowermost small region can be a small region h0. The lowermost small region can indicate a small region closest to the accommodating vehicle 200 among a plurality of small regions included in the front image. It can be appreciated that the lowermost small region h0 is a small region having the same vertical height as the vertical height of the accommodating vehicle 200 that is traveling. That is, the lowermost small region h0 can indicate a flat region with respect to the ground on which the accommodating vehicle 200 is traveling. The description of the process of obtaining the pitch angle of the lowermost small region h0 is the same as or similar to that of the operation of obtaining the pitch angle in Figure 5A

[0102] In operation S320, the ISP 120 can set the height at which the lowermost small region h0 starts to 0, and can calculate the vertical height at which the lowermost small region h0 ends.

[0103] According to some example embodiments, by using Equation 2 and Equation 4, the ISP 120 can obtain the Z' value in the road coordinates, and additionally, based on the interrelationship between the Z' value in the road coordinates and the Z value in the world coordinates, the ISP 120 can obtain the Z value of an arbitrary small region, that is, can obtain the change in the vertical height in an arbitrary small region. The change in the vertical height can be expressed as Equation 7 below.

[0104] [Equation 7]

[0105]

[0106] In one embodiment, Hc can indicate a height parameter of the camera 110, and can correspond to the vertical height of the camera 110 in the world coordinates.

[0107] Referring to Figure 4 The point at which the lowermost small region starts can include a second coordinate (u2, v2) and a fourth coordinate (u4, v4), and the point at which the lowermost small region ends can include a first coordinate (u1, v1) and a third coordinate (u3, v3). This is because, in the case where the accommodating vehicle 200 travels along the lane, the accommodating vehicle 200 first passes through the point including the second coordinate (u2, v2) and the fourth coordinate (u4, v4), and then passes through the point including the first coordinate (u1, v1) and the third coordinate (u3, v3).

[0108] That is, a line passing through the second coordinate (u2, v2) and the fourth coordinate (u4, v4) can be assumed to be a first line 710, and a line passing through the first coordinate (u1, v1) and the third coordinate (u3, v3) can be assumed to be a second line 720. The point at which the lowermost small region starts can correspond to the first line 710, and the point at which the lowermost small region ends can correspond to the second line 720. ​

[0109] As described above, the first line 710 can be a point close enough to the accommodation vehicle 200, and thus the vertical height can not change with respect to the accommodation vehicle 200. That is, the Z values of the second coordinate (u2, v2) and the fourth coordinate (u4, v4) both included in the first line 710 can be assumed to be 0. Based on the feature that the Z value of the lowermost small region in Equation 7 is 0 (e.g., i = 0), the height of the lowermost small region in the road coordinates by the camera 110 can be expressed as Equation 8 below.

[0110] [Equation 8]

[0111]

[0112] According to some example embodiments, the plurality of small regions adjacent to each other in the vertical direction can be continuous. This is because the small region h1 adjacent upward to the lowermost small region h0 divided from the front image forms one front road, and the front road is continuous. That is, the continuity of the front road can be expressed as Equation 9 below.

[0113] [Equation 9]

[0114]

[0115] That is, by substituting Equation 7 and Equation 8 into Equation 9, the vertical height corresponding to an arbitrary small region can be obtained, and this can be expressed as Equation 10 below.

[0116] [Equation 10]

[0117]

[0118] According to some example embodiments, the conversion between the road coordinates and the world coordinates can be performed, and this can be expressed as Equation 11 below.

[0119] [Equation 11]

[0120]

[0121] Referring to Equation 11, the Z value of the point where the arbitrary small region ends can be checked based on the Z value of the starting point of the arbitrary small region, the height of the camera 110 in the world coordinates, and the pitch angle of the arbitrary small region. That is, the vertical height that is the change in the Z value of the arbitrary small region can be calculated. For example, by substituting i = 0, the change in the Z value of the lowermost small region h0 of the front image can be obtained.

[0122] In operation S330, the ISP 120 can sequentially calculate the vertical heights of the plurality of small regions based on the continuity between the vertically adjacent small regions.

[0123] Referring to Figure 7B and Figure 7D , the Z value of the point at which the first smaller region h0 ends can be the same as the Z value of the point at which the second smaller region h1 starts. As described above, this is because the height of the road ahead is continuous. For example, a concave-convex portion can be on the road ahead, and thus the Z value can abruptly increase. That is, even when the Z value rapidly changes due to the concave-convex portion, the ISP 120 can finely divide the image into a plurality of smaller regions, and thus the Z values of the vertically adjacent smaller regions can be the same.

[0124] The ISP 120 can sequentially calculate the Z value change of the smaller regions from the lowermost smaller region h0 in a direction toward the upper portion. As described above, the Z value change of the first smaller region h0 can be set to the Z value of the starting point of the second smaller region h1. The ISP 120 can calculate the Z value of the point at which the second smaller region h1 ends based on the pitch angle of the second smaller region h1, and can set the calculated Z value as the Z value of the starting point of the third smaller region (not shown). The third smaller region (not shown) can represent a region vertically adjacent to the second smaller region h1 upward. The ISP 120 can repeat the calculation until the uppermost smaller region of the front image to calculate the change in the vertical height.

[0125] In operation S340, the ISP 120 can generate a vertical road profile in which the change in the vertical height with respect to the front view of the vehicle 200 is reflected based on the calculated Z value change.

[0126] Referring to Figure 7C , the ISP 120 can generate a vertical road profile. The horizontal axis can be the same as the traveling direction of the vehicle 200. The vertical axis can represent the change in the vertical height with respect to the vehicle 200. The ISP 120 can sequentially obtain the Z value change of the plurality of smaller regions from the lower end of the front image until the upper end of the front image, and in the vertical road profile, the vertical axis can be displayed by accumulating the Z value change as the vehicle 200 travels along the horizontal axis.

[0127] Figure 8 An example embodiment of detecting a vertical height of a speed bump according to an example embodiment is illustrated.

[0128] Referring to Figure 8According to an example embodiment, the ISP 120 can identify a speed bump disposed in the front view. The ISP 120 can detect a lane based on the front image transmitted from the camera 110. The speed bump in front of the accommodated vehicle 200 can include a shape corresponding to a different direction from the lane along which the accommodated vehicle 200 is traveling. That is, when using a lane detection algorithm, the ISP 120 can detect a lane of a short length facing a different direction from the traveling lane due to the shape of a different direction printed in the speed bump. When the lane facing a different direction is detected from the front image, the ISP 120 can determine that the speed bump is in the front view.

[0129] In the above-described example embodiment, it has been described that the speed bump is identified based on the detection of the lane of a different direction, but the inventive concept is not limited thereto. The ISP 120 can identify the speed bump more quickly by using an object detection algorithm of a neural network IP.

[0130] According to an example embodiment, the ISP 120 can generate a vertical road profile corresponding to the front image. As described above, the ISP 120 can divide the front image into a plurality of smaller regions in a horizontal direction, and can calculate a change in pitch angle and Z value from the lowermost smaller region of the front image in a direction toward the upper end. For example, in a smaller region corresponding to a speed bump, the ISP 120 can calculate a change in pitch angle and Z value according to a lane segment generated based on a pattern printed in the speed bump, rather than a lane segment based on which the accommodated vehicle 200 is traveling. This is because, in the case where the calculation is performed based on the lane segment based on which the accommodated vehicle 200 is traveling, a change in vertical height based on the speed bump is not reflected in the vertical road profile.

[0131] According to some example embodiments, the ISP 120 can more accurately divide a plurality of smaller regions corresponding to the identified speed bump in a horizontal direction. For example, assuming that the ISP 120 divides the front image every 5 times of a vertical pixel length in a horizontal direction, the ISP 120 can divide a region corresponding to the speed bump every 1 time of the vertical pixel length. The ISP 120 can decrease the interval of dividing the region only in a region including a lane of a different direction, and thus can more accurately divide the region corresponding to the speed bump.

[0132] In an example embodiment, the ISP 120 can generate a vertical road profile based on a change in Z value of each of a plurality of smaller regions included in the front image. In a region corresponding to a speed bump in a front region of the vertical road profile, the ISP 120 can use a change in Z value calculated according to a lane segment generated based on a pattern printed in the speed bump.

[0133] Figure 9An example embodiment using a head-up display (HUD) according to an example embodiment is illustrated.

[0134] Referring to Figure 9 , the host vehicle 200 can further include a HUD (not shown). The HUD can be a device that displays a travel lane and additional information such as a travel speed of the host vehicle 200 on a front window of the host vehicle 200.

[0135] According to an example embodiment, the ISP 120 can generate information about a virtual lane. The virtual lane can be indicated with respect to a front road of the host vehicle 200 by using the HUD. For example, the ISP 120 can generate information about a lane that is parallel to a lane shown in a bird's-eye view of FIG. 10 and has a narrower width in a Y' axis direction. Figure 4

[0136] According to an example embodiment, the ISP 120 can reflect a vertical road profile in the generated information about the virtual lane. That is, the ISP 120 can generate a coordinate value of a world coordinate of a lane in the virtual lane based on the vertical road profile including information about a change in a Z value. Figure 4

[0137] According to an example embodiment, the ISP 120 can calculate a coordinate value corresponding to an image plane of the virtual lane based on a conversion relationship of Equation 11. Subsequently, the HUD can display the virtual lane based on the coordinate value corresponding to the image plane. For example, the virtual lane can include information about a change in a Z value of the vertical road profile, and thus the HUD can actually display uneven portions even if the uneven portions exist in the front road. Accordingly, a driver of the host vehicle 200 can recognize information about the uneven portions and a speed bump of the front road in advance.

[0138] Figure 10 An example embodiment of measuring a roll of a front road according to an example embodiment is illustrated.

[0139] Referring to Figure 10 , the front image 1000 can include an image including a plurality of lanes. Hereinafter, for convenience of description, a lane in which the host vehicle 200 travels can be referred to as a second lane, a lane left of the second lane can be referred to as a first lane, and a lane right of the second lane can be referred to as a third lane.

[0140] ​​According to an example embodiment, the ISP 120 can calculate the Z value changes of the first lane to the third lane. That is, the ISP 120 can divide the front image including the first lane to the third lane into a plurality of smaller regions in a horizontal direction. Subsequently, the ISP 120 can detect a lane segment from any smaller region of each of the first lane to the third lane, and can detect the change of the Z value based on four coordinate values forming the lane segment.

[0141] For example, in any smaller region, the Z value change of the first lane can be -10 mm, the Z value change of the second lane can be 10 mm, and the Z value change of the third lane can be 20 mm. The ISP 120 can calculate the roll angle of the front road based on the Z value change of each of the first lane to the third lane. According to an example embodiment, the ISP 120 can calculate the roll angle of the front road based on the Z value change of each of the first lane and the third lane. That is, the ISP 120 can calculate the roll angle of the front road based on 30 mm, which is the vertical height difference between the left region and the right region of the front road. According to an example embodiment, the ISP 120 can calculate a first roll angle and a second roll angle, and based on the first roll angle and the second roll angle, the ISP 120 can calculate the roll angle of the front road. The first roll angle can be determined based on the vertical height difference between the first lane and the second lane, and the second roll angle can be determined based on the vertical height difference between the second lane and the third lane. The ISP 120 can obtain each of the first roll angle and the second roll angle, and can determine the average of the first roll angle and the second roll angle as the roll angle of the front road.

[0142] Although the present inventive concept has been shown and described in detail with respect to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and detail can be made without departing from the spirit and scope of the appended claims.

Claims

1. An operating method of a slope estimation device including at least one camera, the operating method comprising: obtaining a front image through the at least one camera; detecting a lane included in the front image; dividing the front image into a plurality of smaller regions in a horizontal direction; identifying a plurality of lane segments included in each of the plurality of smaller regions; obtaining a plurality of coordinate values forming each of the plurality of lane segments; obtaining a pitch angle of each of the plurality of smaller regions based on the obtained plurality of coordinate values; determining whether a speed of an accommodation vehicle including the slope estimation device is greater than a threshold speed; and when the speed of the accommodation vehicle is greater than the threshold speed, reducing a vertical pixel length of an upper region of the front image. 2.The operating method of claim 1, further comprising: identifying a predetermined plurality of smaller regions from a lower end of the front image; identifying a lane segment included in each of the predetermined plurality of smaller regions; obtaining a pitch angle corresponding to each of the predetermined plurality of smaller regions based on a plurality of coordinate values forming each of the identified lane segments; and setting the obtained pitch angle as a pitch angle of the accommodation vehicle. calculating, for each of the plurality of smaller regions, each of camera heights in a road coordinate corresponding to the plurality of smaller regions based on the pitch angles of the plurality of smaller regions. converting each of the plurality of coordinate values of the front image into a world coordinate value of a world coordinate based on the calculated camera heights and the obtained pitch angles.

3. The operating method of claim 1, further comprising: generating a road profile based on a change in a vertical component of the world coordinate value.

4. The method of operation of claim 3, further comprising: the front image includes images corresponding to a plurality of lanes, and 5. The method of operation of claim 4, further comprising: the operating method comprises:

6. The operating method of claim 5, wherein, obtaining world coordinate values of the plurality of lanes; obtaining a change in a vertical component of one of the plurality of smaller regions; and calculating a roll angle of a front road based on the change in the vertical component of each of the plurality of lanes. vertical pixel lengths of the plurality of smaller regions are the same. based on the speed of the accommodation vehicle, the vertical pixel lengths of the plurality of smaller regions are set to be different.

7. The operating method of claim 1, wherein, 9.The operating method of claim 5, further comprising:

8. The operating method of claim 1, wherein, detecting a lane in a different direction with respect to a lane on which the accommodation vehicle is traveling in the front image; determining that a speed bump is in a region corresponding to the lane in the different direction; obtaining a world coordinate value based on the lane in the different direction; and determining a vertical height of a region corresponding to the speed bump according to the world coordinate value obtained based on the lane in the different direction. 10.A slope estimation device comprising: at least one camera configured to obtain a front image with respect to an accommodation vehicle including the slope estimation device, if the accommodation vehicle is traveling; and ​ ​ ​ an image signal processor configured to: detect a lane included in the front image; divide the front image into a plurality of smaller regions in a horizontal direction; identify a plurality of lane segments included in each of the plurality of smaller regions; obtain a plurality of coordinate values forming each of the plurality of lane segments; obtain a pitch angle of each of the plurality of smaller regions based on the obtained plurality of coordinate values; determine whether a speed of the accommodating vehicle is greater than a threshold speed; and when the speed of the accommodating vehicle is greater than the threshold speed, reduce a vertical pixel length of an upper region in which the front image is divided.

11. The inclination estimation device according to claim 10, wherein the image signal processor is further configured to: identify a predetermined plurality of smaller regions from a lower end of the front image; identify a lane segment included in each of the predetermined plurality of smaller regions; obtain a pitch angle corresponding to each of the predetermined plurality of smaller regions based on a plurality of coordinate values forming each of the identified lane segments; and set the obtained pitch angle as a pitch angle of the accommodating vehicle.

12. The inclination estimation device according to claim 10, wherein the image signal processor is further configured to: calculate, for each of the plurality of smaller regions, each of camera heights corresponding to the plurality of smaller regions in road coordinates based on the pitch angles of the plurality of smaller regions.

13. The inclination estimation device according to claim 12, wherein the image signal processor is further configured to: convert each of the plurality of coordinate values of the front image into a world coordinate value of world coordinates based on the calculated camera heights and the obtained pitch angles.

14. The inclination estimation device according to claim 13, wherein the image signal processor is further configured to: generate a road profile based on a change in a vertical component of the world coordinate value.

15. The inclination estimation device according to claim 14, wherein the front image includes images corresponding to a plurality of lanes, and the image signal processor is further configured to: obtain world coordinate values of the plurality of lanes; obtain a change in a vertical component of one of the plurality of smaller regions; and calculate a roll angle of a front road based on the change in the vertical component of each of the plurality of lanes.

16. The inclination estimation device according to claim 10, wherein vertical pixel lengths of the plurality of smaller regions are the same.

17. The inclination estimation device according to claim 10, wherein the vertical pixel lengths of the plurality of smaller regions are set to be different based on a speed of the accommodating vehicle.

18. The inclination estimation device according to claim 14, wherein the image signal processor is further configured to: detect a lane in a different direction with respect to a lane in which the accommodating vehicle is traveling in the front image; determine that a speed bump is in a region corresponding to the lane in the different direction; obtain a world coordinate value based on the lane in the different direction; and determine a vertical height of a region corresponding to the speed bump according to the world coordinate value obtained based on the lane in the different direction.

19. The inclination estimation device according to claim 14, wherein the image signal processor is further configured to: generate a virtual lane parallel to the detected lane based on the world coordinate value; convert coordinates of the virtual lane into coordinate values of image coordinates based on a conversion relationship of the front image and the world coordinate value; and display the virtual lane based on the coordinate values of the image coordinates by using a head-up display (HUD).

20. An accommodating vehicle apparatus comprising: at least one camera configured to obtain a front image with respect to the accommodating vehicle in a case where the accommodating vehicle is traveling; An image signal processor configured to: detect a lane included in the front image, divide the front image into a plurality of smaller regions in a horizontal direction, identify a plurality of lane segments included in each of the plurality of smaller regions, obtain a plurality of coordinate values forming each of the plurality of lane segments, obtain a pitch angle of each of the plurality of smaller regions based on the obtained plurality of coordinate values, identify a predetermined plurality of smaller regions from a lower end of the front image, identify a lane segment included in each of the predetermined plurality of smaller regions, obtain a pitch angle corresponding to each of the predetermined plurality of smaller regions based on a plurality of coordinate values forming each of the identified lane segments, and convert coordinates of the front image into coordinate values of a world coordinate based on the obtained pitch angles; and and a vehicle controller configured to control a strength of a suspension and / or a speed of the accommodating vehicle based on the coordinate values of the world coordinate, wherein the image signal processor is further configured to determine whether the speed of the accommodating vehicle is greater than a threshold speed, and reduce a vertical pixel length dividing an upper region of the front image when the speed of the accommodating vehicle is greater than the threshold speed.

Images