Environment recognition device

Abstract

To reduce false alarms and erroneous control by drawing attention to Bump and improving accuracy of control timing.SOLUTION: An environment recognition unit 102 includes at least a Bump feature generating unit 202, a travel area specifying unit 203, a Bump recognizing unit 204, and a result output unit 205. The Bump recognizing unit 204 includes a candidate area extracting unit 241 and a visible area determination unit 242.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a driving environment detection technology for detecting surrounding objects in a driving environment based on camera images. [Background technology]

[0002] Patent Document 1 discloses a vehicle control device that can provide information corresponding to the vehicle's position to a vehicle occupant or system even when a Global Navigation Satellite System (GNSS) is unavailable, and that detects an area where an object exists (object area) based on image data or three-dimensional data. As a method for detecting this object area, a method is disclosed in which, to detect road bumps and other road irregularities, a planar road portion in the direction of travel is detected and road surface irregularities are detected from the depth and height information. The document also describes a method for detecting road surface irregularities by detecting maximum and minimum values of depth and height information, tracking corresponding points over time as the vehicle moves, and determining the positions of the bumps based on points where tracking is successful multiple times, thereby enabling accurate detection of bumps. Instead of detecting maximum and minimum values of the road surface, there is also a method in which machine learning is used as training data to learn which positions are bumps. This method makes it possible to detect steps with complex shapes or multiple consecutive steps, and is claimed to be effective in improving detection accuracy (Patent Document 1, paragraphs 0007, 0027, etc.).

[0003] Patent Document 2 also discloses an object recognition device capable of highly accurately recognizing objects captured across overlapping and non-overlapping fields of view in multiple images captured by a stereo camera. Regarding the object recognition device, the document states that "the reliability calculation unit is composed of a first reliability calculation unit that calculates an object identification score (identification result) based on 3D information and a second reliability calculation unit that calculates an object identification score (identification result) based on texture information." It also states that "range images and parallax images may be generated by superimposing the acquired 3D information on a 2D image, and identification processing may be performed based on the generated range images and parallax images." It also discloses that "the object redetection process detects objects by analyzing the texture information of the region. A convolutional neural network is used as the detection method. The convolutional neural network outputs the image positions of the object in the upper left and lower right corners. The output of the convolutional neural network can identify the region of the vehicle captured across overlapping and non-overlapping fields of view." It also discloses that the object to be recognized is determined by comparing the identification score with a predetermined threshold, focusing on the area of the object region, which is the processing region. It is claimed that this method enables highly accurate recognition of objects captured across overlapping and non-overlapping areas of the visual field (Patent Document 2, paragraphs 0015, 0016, 0024 to 0028, etc.). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2023-12649 [Patent Document 2] Patent Publication No. 2022-64388 Summary of the Invention [Problem to be solved by the invention]

[0005] The vehicle control device disclosed in Patent Document 1 describes how, to detect road bumps and other road irregularities, it detects the road portion, which is a plane in the direction of travel, detects the maximum and minimum values of depth and height information, tracks corresponding points over time as the vehicle moves, and sets the point where tracking is successful multiple times as the position of the bump, thereby accurately detecting the bump. However, in an actual driving environment, the mere presence of an uneven road surface is not equivalent to the presence of a bump that requires warnings and various controls. Generally, a bump is a convex protrusion on the road surface that extends across the entire width of the road and cannot be avoided by, for example, steering control. However, the vehicle control device in Patent Document 1 cannot detect only such an object.

[0006] The object recognition device disclosed in Patent Document 2 describes acquiring texture information for an area captured across an overlapping area and a non-overlapping area of the field of view from images captured by two cameras. However, in an actual driving environment, other vehicles besides the vehicle itself are also traveling, and it is not always possible to capture an image of the entire road surface where a bump exists. Blind spots on the road surface caused by other vehicles exist, but Patent Document 2 does not take this into consideration, and there remains the possibility that a bump installed as a convex protrusion across the entire width of the road may not be correctly detected. As a result, it is not possible to properly issue a warning or control the detected bump area.

[0007] Therefore, the present invention aims to detect bumps in the driving environment, while simultaneously suppressing false detection and extending the detection distance, improving the accuracy of the timing of warnings and control for detected bump areas, and reducing false alarms and erroneous control. [Means for solving the problem]

[0008] One aspect of the present disclosure is An environment recognition device that recognizes bumps installed on a road surface based on images captured by a camera mounted on a vehicle, a travel area identification unit that identifies a travel road area in the image in which the travel road on which the vehicle is traveling is captured; a bump feature generation unit that generates a feature map that represents an area including the bump feature from the image together with a degree of likelihood; a bump recognition unit that recognizes the bump based on a bump candidate region in the feature map where the degree of certainty is equal to or greater than a threshold, The bump recognition unit recognizes, as the bump, the bump candidate region that exists on the roadway region and in which at least one of the length in the width direction of the roadway region, the length in the depth direction of the candidate region, or the complexity of the contour of the bump candidate region meets a criterion having a predetermined judgment threshold. The environment recognition device is characterized by the above. [Effects of the Invention]

[0009] According to the above-described aspect of the present disclosure, it is possible to provide an environment recognition device that detects bumps in a driving environment, while simultaneously suppressing false detections and extending the detection distance, improving the accuracy of the timing of warnings and control for detected bump areas, and reducing false alarms and erroneous control.

[0010] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating a first embodiment of an in-vehicle system including an environment recognition device according to the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating the internal configuration of a camera unit and an environment recognition unit according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating the configuration of a candidate area extraction unit. [Figure 4] FIG. 3 is a diagram illustrating the configuration of a viewable area determination unit. [Figure 5]FIG. 10 is a diagram illustrating the internal configuration of a camera unit and an environment recognition unit according to a second embodiment. [Figure 6] FIG. 10 is a diagram showing the processing flow of a bump re-recognition unit. [Figure 7] FIG. 10 is a diagram showing an example of an output determination table inside a result output unit. [Figure 8] FIG. 10 is a diagram illustrating the internal configuration of a camera unit and an environment recognition unit according to a third embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a feature map. [Figure 10] FIG. 10 is a diagram illustrating a fourth embodiment of an in-vehicle system including an environment recognition device according to the present disclosure. [Figure 11] FIG. 10 is a diagram illustrating the internal configuration of a camera unit and an environment recognition unit according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of an environment recognition device according to the present disclosure will be described with reference to the drawings.

[0013] [Embodiment 1] FIG. 1 illustrates one embodiment of an in-vehicle system including an environment recognition unit 102 according to the present disclosure. 1, a camera unit 101 is installed so as to include a field of view in the traveling direction of a vehicle 100. An image of the front of the vehicle captured by this camera unit 101 is input to an environment recognition unit (environment recognition device) .

[0014] The environment recognition unit 102 can be configured, for example, by one or more microcontrollers equipped with a central processing unit (CPU), memories such as ROM and RAM, a timer, and an input / output unit. The environment recognition unit 102 recognizes bumps on the road surface based on an image of the road ahead of the vehicle captured by the camera unit 101. Then, it calculates the distance to the bump, the height of the bump, the relative speed, a detection reliability index, etc., and transmits the calculation results to the control unit 103.

[0015] The control unit 103 determines the risk of an impact or collision based on the distance to the bump, the height of the bump, the relative speed, and a detection reliability index, and outputs commands according to the risk, such as displaying information on a display 104, turning on a warning light, sounding a warning from a speaker 105, reducing or suppressing acceleration with an accelerator 106, decelerating with a brake 107, or changing the shock absorption characteristics of a suspension 108. Similar to the environment recognition unit 102, the control unit 103 can be configured, for example, by one or more microcontrollers equipped with a central processing unit (CPU), memories such as ROM and RAM, a timer, and input / output units.

[0016] In this embodiment, the bump is described as a speed bump, but is not limited to this. The bump in this embodiment may be any uneven surface that causes vibrations of a predetermined magnitude or greater to a vehicle when the vehicle passes over it, and may include, in addition to speed bumps, one or more of a crosswalk, a train track or railroad crossing, and a joint in a bridge or viaduct.

[0017] The speed bump 902 (see Figure 9) that the environment recognition unit 102 recognizes has a length that extends across the entire width of the road surface 901, including the lane in which the vehicle is traveling and, if there are any adjacent lanes that are traveling in the same direction as the lane in which the vehicle is traveling, also the adjacent lanes.Speed bumps in the lane in which oncoming vehicles are traveling are not included in the recognition targets.

[0018] Next, a method in which the environment recognition unit 102 detects a bump based on an image captured by the camera unit 101 will be described with reference to FIG.

[0019] 2 is a block diagram showing the internal configuration of the camera unit 101 and the environment recognition unit 102. Note that, although the camera unit 101 is described here as a stereo camera 201, the camera unit 101 may be configured as a combination of one or more cameras, such as a monocular camera, and a LiDAR, a combination of three or more cameras, or a combination of one or more cameras and a millimeter wave sensor.

[0020] The environment recognition unit 102 has at least a bump feature generation unit 202, a driving area identification unit 203, a bump recognition unit 204, and a result output unit 205. The bump recognition unit 204 also includes a candidate area extraction unit 241 and a visible area determination unit 242.

[0021] The stereo camera 201 consists of two cameras. Each camera is equipped with a CMOS (Complementary Metal Oxide Semiconductor) sensor and a DSP (Digital Signal Processor). The CMOS sensor is an imaging device with a grid of photodiodes that exhibit photoelectric effects. It triggers a shutter at any timing and transmits the accumulated charge to the DSP while performing analog-to-digital (AD) conversion. The DSP converts the transmitted information into an image while performing image quality corrections such as contrast correction, gamma correction, and edge correction. Distance information (distance image) and invalid areas are calculated from these images as parallax information. Note that invalid areas refer to areas where it is difficult to accurately obtain stable parallax information, such as areas with little texture or high similarity. The distance information and invalid areas are collectively referred to as parallax information. Since there are several well-known stereo image processing methods, the details of how to calculate parallax information are omitted here. This parallax information and at least one of the images are sent to the bump feature generation unit 202 and the driving area identification unit 203, respectively.

[0022] The bump feature generation unit 202 generates a feature map that indicates regions containing bump features from images captured by the camera unit 101, along with the degree of likelihood. The bump feature generation unit 202 receives as input disparity information and grayscale images acquired from the stereo camera 201, and outputs a feature map in which bump-likeliness is expressed as a likelihood of 0.0 to 1.0. This feature map can be realized using a convolutional neural network (hereinafter referred to as CNN) or the like, and can be realized by preparing multiple sets of training data in advance, each set including disparity information with a resolution of width W1 × height H1 acquired by the stereo camera 201, grayscale images with a resolution of width W2 × height H2 also acquired by the stereo camera 201, and annotation images with a resolution of width W3 × height H3, and then learning the network structure of the CNN through supervised learning based on this training data. By learning the network structure of the CNN in advance, a feature map of the likelihood of bumps can be obtained by inputting the parallax information and grayscale images obtained from the stereo camera 201 while driving into the CNN.

[0023] In this case, the resolution widths W1, W2, and W3 and heights H1, H2, and H3 may be the same or different. Using the same resolution has the effect of simplifying the network structure and improving learning efficiency. On the other hand, when the baseline of the stereo camera 201 is horizontal, i.e., when the two cameras are installed horizontally spaced apart, the effective resolution of the disparity information is lower in the horizontal direction than in the vertical direction. Therefore, by halving the resolution of only the horizontal widths of the disparity information and the output feature map, i.e., W1 = W3 = W2 ÷ 2 and H1 = H2 = H3, it is possible to reduce the computational cost while suppressing a decrease in the accuracy of the output feature map.

[0024] An example of a feature map is shown in Fig. 9. Fig. 9(a1) shows the input camera image, and Fig. 9(a2) shows the road surface unevenness at that time. Fig. 9(b) shows the input disparity information, and Fig. 9(c) shows the concept of the output likelihood feature map. Here, it is assumed that the grayscale image acquired from the stereo camera 201 shows the road surface 901 in the immediate vicinity, a speed bump 902 installed so as to overlap the position of a pedestrian crossing a little further ahead, and a preceding vehicle 903 further ahead, as shown in Fig. 9(a1).

[0025] According to the road surface unevenness shown in Figure 9(a2), it can be seen that the road surface 901 other than the speed bump 902 is almost flat. In this situation, the disparity information acquired by the stereo camera 201 is as shown in Figure 9(b), and point cloud 904 can be acquired beyond point cloud 905 of the flat road surface. These point clouds can be calculated based on the general principle of stereo vision using the texture of the asphalt and road markings. The bump feature generation unit 202 performs learning using as input the grayscale image (Figure 9(a1)), disparity information (Figure 9(b)), and so-called annotation images in which bump areas are identified by, for example, filling in the area corresponding to the speed bump 902 in Figure 9(a1). As a result, during inference, it is possible to output a feature map such as that shown in Figure 9(c). The feature map shown in Figure 9(c) shows that the likelihood is 0 outside of region 908, which corresponds to the area of the speed bump 902, and is large inside region 908, with even larger likelihoods in regions 906 and 907, which react strongly to the bump. Here, it is expressed as a feature map (Figure 9(c)) of the same size as the input grayscale image (Figure 9(a1)), and shows how likelihood values are output for each corresponding pixel.

[0026] The driving area identification unit 203 identifies a driving road area in which the driving road on which the vehicle 100 is traveling is captured and an object area in which an object existing on the driving road is captured in the captured image captured by the camera unit 101. The driving area identification unit 203 inputs disparity information and a grayscale image acquired from the stereo camera 201, and outputs an area map of the area segmentation result in which the object type to which each pixel belongs is calculated. This area map can be realized using semantic segmentation (SS) or the like. Specifically, the disparity information acquired by the stereo camera 201, a grayscale image also acquired by the stereo camera 201, and an annotation image in which each pixel on the grayscale image is labeled with the type of area to which each pixel belongs are prepared in advance, and multiple sets of these are prepared as one set of training data, and a CNN network structure is trained by supervised learning based on this training data. An annotation image is an image in which each pixel is labeled, for example, road asphalt area (=1), painted area such as road markings like white lines and speed limits (=2), vehicle area such as cars and trucks (=3), two-wheeled vehicle area such as bicycles and motorbikes (=4), pedestrian area (=5), animal area such as cows and sheep (=6), etc. By learning the CNN network structure in advance, disparity information and grayscale images obtained from the stereo camera 201 while driving can be input into the CNN to obtain a region map as an inference result of semantic segmentation.

[0027] The bump recognition unit 204 receives the feature map output from the bump feature generation unit 202 and the area map output from the driving area identification unit 203, determines the presence or absence of a bump, and outputs the bump recognition result and its accompanying information to the result output unit 205. The bump recognition result and accompanying information include information such as whether or not a bump was recognized as being present, and if so, the distance from the vehicle to the bump, the height from the road surface plane to the apex of the bump's protrusion, the width of the bump's protrusion, and the confidence level at the time of recognition.

[0028] The result output unit 205 receives the output result of the bump recognition unit 204 as input, and generates output information for the control unit 103 based on the bump recognition results in time series and the accompanying information. Generally, there is a non-zero possibility of erroneous recognition when recognizing the outside world through image recognition, so the result output unit 205 may determine whether or not to output the final recognition result based on the recognition results obtained multiple times.

[0029] Specifically, the output result of the bump recognition unit 204 and additional information may be input and calculated as follows. For example, it is assumed that the following information is obtained: A recognition possibility flag F takes a value of 1 if a bump is recognized during N time periods from time t0, t-1, . . . , t-(N-1), and takes a value of 0 if not. t0 , F t-1 , , F t-(N-1) , the distance to the recognized bump D t0 , D t-1 ,···,D t-(N-1) , the height of the recognized bump H t0 , H t-1 , , H t-(N-1) , is obtained. In this case, (Condition A) ΣFx≧Thr1, (Condition B) D t0 ≧D t-1 ≧···≧D t-(N-1) , (condition C) Var(H x )≦Thr2, the bump recognition result and the accompanying information may be generated as output information to the control unit 103.

[0030] That is, (Condition A) means that detection must be made at least Thr1 times within N time periods. If detection is made at least Thr1 times while the distance to the object changes as the vehicle travels, it has the effect of eliminating false positives that occur suddenly. (Condition B) means that the distance to the bump becomes shorter over time and the bump approaches monotonically, which has the effect of eliminating false positives that may occur by chance due to light or shadow. (Condition C) means that the variance of the height of the observed bump is below a certain threshold Thr2 and that it is observed stably. Since the physical shape of a bump does not change over a short period of time, observing the bump multiple times while the distance to the object changes as the vehicle travels and verifying that its shape and height do not change has the effect of eliminating false positives that may occur accidentally due to noise, etc. Note that (Condition B) and (Condition C) may use data only for the time period when the bump is recognized.

[0031] Note that the detailed numerical values of the detection conditions are design matters, and results may be output and determined under conditions other than those mentioned above. Also, while the above-mentioned (Condition B) sets the determination condition as simply that the vehicle is monotonically approaching the bump, it is also possible to calculate the vehicle's traveling distance or movement distance using a method generally known as the dead reckoning method, compare the traveling distance or movement distance with the amount of change in the distance to the bump, and use the difference being equal to or less than a certain value as the determination condition.

[0032] The bump recognition unit 204 also includes a candidate area extraction unit 241 and a viewable area determination unit 242 .

[0033] The candidate area extraction section 241 will be described with reference to FIG. The candidate area extraction unit 241 outputs bump candidate areas using as input the bump features (hereinafter also referred to as feature map) generated by the bump feature generation unit 202. The candidate area extraction unit 241 extracts, from among the bump candidate areas present on the road area, areas whose length in the width direction of the road area is equal to or greater than a determination threshold, as bump candidate areas.

[0034] The candidate area extraction unit 241 includes a high likelihood area extraction unit 301 , a width direction feature calculation unit 302 , a depth direction feature calculation unit 303 , a contour shape feature calculation unit 304 , and a bump candidate area estimation unit 305 .

[0035] The high-likelihood region extraction unit 301 determines a bump candidate region based on the feature map generated by the bump feature generation unit 202, and further calculates its relative position with respect to the vehicle. As described above, the feature map is output so that the likelihood is high inside region 908 corresponding to the bump region, and regions 906 and 907 have even higher likelihood in regions with even higher bump probability. Therefore, by comparing the likelihood of each point with a threshold ThrC and extracting points with high likelihood, it is possible to extract a high-likelihood region with a high probability of being a bump. For example, from among the high-likelihood regions with a high bump probability, the region with the largest area is extracted, and the coordinates of the center of gravity of the extracted region are taken to determine the bump candidate region. In this way, it is possible to calculate the position of this bump candidate region, such as how far forward it is from the vehicle or how far it is offset to the left or right from the center of the vehicle. Furthermore, after extracting the region with the largest area, the same process may be performed again, excluding that region, to calculate the region with the next largest area as the second bump candidate region, and then calculate its position relative to the vehicle. In this way, the high likelihood region extraction unit 301 may calculate one or more bump candidate regions and output them as bump candidate regions together with their area information or ranking information indicating how large they are.

[0036] The width direction feature calculation unit 302 calculates the width direction feature FW, which is the length in the direction perpendicular to the vehicle's path, i.e., the width direction length of the bump, for the bump candidate area calculated by the high likelihood area extraction unit 301 described above.

[0037] The depth direction feature calculation unit 303 calculates a depth direction feature FD, which is the length in the direction of the vehicle's travel path, i.e., the depth direction length of the bump, for the bump candidate area calculated by the high likelihood area extraction unit 301 described above.

[0038] The contour shape feature calculation unit 304 calculates a contour shape feature FC, which is a numerical value that indexes the complexity of the contour of the bump candidate region calculated by the high-likelihood region extraction unit 301. Specifically, as the straightness of the width direction side of the bump, the contour of each of the upper and lower sides of the bump candidate region may be approximated as a straight line using Hough transform, and the variance of the points that make up the contour with respect to the approximated straight line may be calculated. If the upper and lower sides of the bump candidate region are straight lines, such as in the case of a perfect trapezoid, the variance of this straightness will be 0.

[0039] The bump candidate region estimation unit 305 receives outputs from the width direction feature calculation unit 302, depth direction feature calculation unit 303, and contour shape feature calculation unit 304, and determines whether or not to make each region extracted by the high likelihood region extraction unit 301 a bump candidate region depending on whether or not each calculated feature satisfies a criterion having a predetermined judgment threshold. If the region is determined to be a bump candidate region, the unit 305 calculates a bump certainty factor that approaches 1 if the region is highly likely to be a bump, and approaches 0 if not.

[0040] Specifically, if the region is a bump, the width direction feature FW is expected to be at least the vehicle width or more, so a threshold value ThrW1 equivalent to the vehicle width and a threshold value ThrW2 equivalent to the road width are used. If the width direction feature FW is smaller than threshold ThrW1, the region is determined to not be a bump candidate region, and if the width direction feature FW is equal to or greater than threshold ThrW1, the region is determined to not be a bump candidate region. If the width direction feature FW is equal to or greater than threshold ThrW1, the region is determined to not be a bump candidate region. If the width direction feature FD is equal to or greater than threshold ThrD1 and threshold ThrD2, which are the depth directions of standard bumps, the region is determined to not be a bump candidate region if it is not within that range, and if it is within that range, the region is determined to not be a bump candidate region. If the contour shape feature FC is equal to or greater than threshold ThrC, the region is determined to be a bump candidate region. If the contour shape feature FC is equal to or greater than threshold ThrC, the region is determined to be a bump candidate region. In this way, bump candidate regions can be extracted based on the feature amounts of the width direction feature calculation unit 302, the depth direction feature calculation unit 303, and the contour shape feature calculation unit 304.

[0041] In the above example, a bump candidate region is extracted based on the feature values of the width direction feature calculation unit 302, the depth direction feature calculation unit 303, and the contour shape feature calculation unit 304. However, a bump candidate region may be extracted based on at least one of the feature values.

[0042] The confidence factor of the bump may be designed to decrease when it approaches each decision threshold. Specifically, the confidence factor of the width direction feature, ConfW, is ConfW=0(FW<ThrW1)、ConfW=1(FW> By using the following formula, such as ConfW=-(FW-ThrW2), ConfW=-(FW-ThrW2)÷(ThrW1-ThrW2)÷2+1(FW≧ThrW1, FW≦ThrW2), the confidence factor can be calculated to be low at around 0.5 when the width direction characteristic FW is almost the same as the vehicle width, and to approach 1 as FW becomes wider. Other confidence factors can also be designed in a similar way.

[0043] The visible area determination unit 242 will be described using Fig. 4. The visible area determination unit 242 receives as input bump candidate areas, bump features (feature map), and information related to road areas (area map), recognizes bumps based on the overlap rate of the bump candidate areas with respect to the visible area, and outputs the recognition results and their accompanying information. The visible area determination unit 242 has a tentative bump candidate estimation unit 401, a distance / TTC calculation unit 402, a road width estimation unit 403, an obstructed area estimation unit 404, a bump visible area estimation unit 405, and a visible area overlap rate determination unit 406.

[0044] The tentative bump candidate estimation unit 401 performs subsequent processing on tentative bump candidates in descending order of reliability among the bump candidate regions extracted by the candidate region extraction unit 241, and can terminate processing when there is no more room for computational resources or processing time, or when all bump candidate regions have been processed. This has the effect of preventing the processing time from significantly exceeding the limit, even when many bump candidate regions exist in the image.

[0045] The distance / TTC calculation unit 402 calculates the distance from the vehicle to the tentative bump candidate output from the tentative bump candidate estimation unit 401. There are several known techniques for calculating this distance. For example, when the vehicle is traveling stably, the mounting height and visual axis of the camera are known from design values, and the camera angle of view and distortion information are also known. Therefore, it is easy to calculate, using the principle of triangulation, which point on the road surface plane on which the vehicle is traveling in real space corresponds to the coordinates observed on the image, and the distance can be calculated. Alternatively, when a stereo camera is used, the calculation can be performed using the principle of triangulation from the coordinates observed on the left and right cameras and the relative positional relationship between the left and right cameras. The TTC (Time To Collision) to the bump can be calculated based on the distance calculated in this way and the vehicle speed. Note that the coordinates observed on the camera at this time may be the center of the area of the tentative bump candidate, or the distance may be calculated by calculating the distance to each of the center of the front side (closer to the vehicle) and the center of the back side (farther from the vehicle) of the tentative bump candidate, and then calculating the average of these distances. By doing this, the perspective transformation reduces the depth of the image and enlarges the front, so if you simply take the center of the candidate area on the image, you will end up calculating a point closer to the center.However, this has the effect of preventing this and making it possible to calculate coordinates closer to the true center.

[0046] The road width estimation unit 403 estimates the width Lane of the road based on the area map of the area division result, which calculates the object type to which each pixel belongs, identified by the driving area identification unit 203. For example, it may calculate the distance between the left and right edges of the road at 10 points 11 m, 12 m, ..., and 20 m ahead and output the most frequent value, or it may extract the left and right road edge boundary lines based on the boundaries between the road area and off-road area from the area map of the area division result, and use the distance between them as the width of the road.

[0047] The occluded area estimation unit 404 estimates the width of the object area of a vehicle or obstacle that is occluding the road surface (blind spot area) at the distance where the tentative bump candidate is located, based on the area map of the area division result identified by the driving area identification unit 203, which calculates the object type to which each pixel belongs. Note that if the vehicle or obstacle is located farther away than the distance where the tentative bump candidate is located, the road surface width occluded by them will be zero. On the other hand, if the vehicle or obstacle is located closer than the distance where the tentative bump candidate is located, the road surface width occluded by them will be wider than the width of the vehicle or obstacle itself. Specifically, when a vehicle or obstacle with width Width is located at a distance Dist1 from the camera, the occluded road surface width Mask (blind spot area) at a distance Dist2 where the tentative bump candidate is located is expressed as follows: = Width × Dist2 ÷ Dist1 (in this case, Dist1 <Dist2)。

[0048] The bump visible area estimation unit 405 estimates the visible area as the area excluding the blind spot area blocked by the object area from the road area. The bump visible area estimation unit 405 calculates the width of the unblocked road surface at the distance where the tentative bump candidate is located. Specifically, the bump visible area estimation unit 405 uses the road width Lane at the distance where the tentative bump candidate is located and the blocked road surface width Mask to calculate the width of the blocked road surface by subtracting the blocked road surface width Mask from the road width Lane (=Lane-Mask).

[0049] The visible area overlap rate determination unit 406 determines to what extent the bump candidate area output from the candidate area extraction unit 241 matches the width of the bump that is visible at that distance estimated by the bump visible area estimation unit 405. Specifically, the width of the bump candidate area is divided by the width over which the bump is estimated to be visible to calculate an overlap rate in the range of 0 to 1. If this overlap rate is greater than a threshold ThrO, it is determined that the candidate area sufficiently overlaps with the visible area, and is output as a recognized bump.

[0050] In this way, the bump recognition unit 204, which has the candidate area extraction unit 241 and the visible area determination unit 242, recognizes bumps. The bump recognition unit 204 recognizes bumps based on the visible area of the road area, excluding blind spot areas that are blocked by object areas.

[0051] The result output unit 205 receives the bump recognition result recognized by the bump recognition unit 204 and its accompanying information, as well as one or more pieces of information on the speed, driving environment, and lighting environment of the vehicle as input, and outputs how control should be performed. A specific example will be described with reference to Fig. 7. Fig. 7 is an example of an output decision table inside the result output unit 205 (not shown).

[0052] For example, No. 1 assumes a scenario in which the vehicle is traveling on a paved road in a city on a clear day. In this case, the parameters are set to output at the standard output confirmation distance (30 m for warning, 20 m for pre-emptive control, and 10 m for deceleration control). No. 2 assumes a scenario in which the vehicle is traveling on a paved road in a city on a clear night. In this case, the vehicle's headlights cast shadows on the bumps, making them easier to detect and less likely to cause false detections. In this case, since the presence of the bumps is difficult for the driver to see, warnings and pre-emptive control are performed as in daytime, but the output parameters are set to perform deceleration control earlier. Also, in an environment where false detections are likely to occur, such as in rainy weather at night, as in No. 3, the output parameters are set to not perform deceleration control to prevent false detections. Furthermore, on an unpaved road as in No. 4, vehicle body sway occurs due to road surface irregularities, making accurate road surface estimation difficult. Therefore, the output parameters are set to shorten the output confirmation distance in order to verify repeated detections from warnings and pre-emptive control to actual control. Also, when traveling at low speeds like No. 5, the risk is low even if no control is performed on the bump, so the output parameter may be set to only issue a warning.

[0053] In this way, the environment recognition unit of this embodiment 1 can detect bumps in the driving environment and output information for appropriate control even in situations where the detection performance of the camera is reduced.

[0054] [Embodiment 2] Hereinafter, a second embodiment of the environment recognition unit according to the present disclosure will be described with reference to FIG.

[0055] 5 shows one embodiment of an in-vehicle system including an environment recognition unit 102 according to the present disclosure. The environment recognition unit 102 of this embodiment differs from the environment recognition unit 102 of the first embodiment in that it additionally includes a bump re-recognition unit 501. Other aspects of the environment recognition unit 102 of this embodiment are similar to those of the environment recognition unit 102 of the first embodiment, and therefore similar parts are denoted by the same reference numerals and description thereof will be omitted.

[0056] In the environment recognition unit 102 of the first embodiment, the bump recognition unit 204 outputs a bump recognition result and its accompanying information, but the bump recognition unit 204 of the present embodiment differs in that even if a bump is not recognized, it outputs a result of non-recognition and accompanying information for each candidate region output by the candidate region extraction unit 241. The bump re-recognition unit 501 has the function of receiving the result of non-recognition and the accompanying information, changing various thresholds according to the control type, performing bump determination again, and outputting the result.

[0057] The bump re-recognition unit 501 changes the judgment threshold value (re-determination setting unit) based on the control type indicating the control state of the vehicle 100 and the distance to the bump candidate area of the vehicle 100. Then, if there is a bump candidate area that has not been recognized as a bump until the previous processing cycle, the bump is re-recognized based on the changed judgment threshold value.

[0058] The processing flow of the bump re-recognition unit 501 is shown in Fig. 6. First, in S101, a control type including one or more of warning, preliminary control, and deceleration control is set. This setting may use the control type included in the recognition request sent from the control unit 103, or may use the control type for which the output determination distance is set in the output determination table shown in Fig. 7 described above.

[0059] Next, in S102, it is determined whether the parallax accuracy is above a certain level for the distance of the candidate area output by the candidate area extraction unit 241. This parallax accuracy can be calculated using one or more of the following: the baseline length of the stereo camera, the resolution of each camera, exposure time, aging and temperature changes, and calibration status. For example, if a point corresponding to point P on the left camera image is captured as point Q on the right camera image, the coordinate difference between points P and Q changes depending on the distance. The distance that changes when this coordinate difference shifts by one unit is the parallax resolution. Here, one unit refers to the search width when performing a correspondence search. If searching using integer coordinates, it is one pixel. If performing a subpixel search, it is a subpixel such as 1 / 4 pixel. This can be calculated from the camera installation status, baseline length, and resolution of each camera. Furthermore, in dark lighting environments, an attempt is made to capture a brighter image of the subject by extending the exposure time. However, extending the exposure time in an in-vehicle camera increases the likelihood of motion blur, which reduces parallax accuracy. In addition, distortion of the stereo camera housing due to aging and temperature changes can occur, resulting in a decrease in parallax accuracy. Furthermore, because calibration is performed using large-scale equipment when the stereo camera is shipped from the factory, the calibration state is good and highly accurate. However, if the stereo camera is replaced or reinstalled at a dealership or other facility for some reason, the calibration state may become poor if calibration is performed using simple equipment, which may result in a decrease in parallax accuracy. Therefore, the parallax accuracy can be calculated by multiplying the parallax resolution by a coefficient of 1 or more depending on the exposure time, aging, temperature changes, and changes in the calibration state. This coefficient can be recorded, for example, as a table during design. If the parallax accuracy at the distance of the candidate area calculated in this way is lower than the height of the bump to be detected (for example, if the parallax accuracy is 10 cm and the bump to be detected is 5 cm), the bump cannot be detected correctly. In such cases, S102 determines that the parallax accuracy is not above a certain level (= No).Conversely, if the calculated parallax accuracy value at the distance of the candidate area is smaller than the height of the bump to be detected, it is determined in S102 that the parallax accuracy is equal to or greater than a certain level (=Yes), and the process proceeds to S103.

[0060] In S103, a determination is made based on the control type and TTC (Time To Collision). If the control type set in S101 is deceleration control and the TTC calculated from the vehicle speed and the distance to the candidate area output by the candidate area extraction unit 241 is equal to or less than the TTC threshold value for deceleration control set in advance (=Yes), the process proceeds to S104; otherwise (=No), the process proceeds to S105.

[0061] In S105, a determination is made based on the control type and TTC, as in S103. If the control type set in S101 is preliminary control and the TTC calculated from the vehicle speed and the distance to the candidate area output by the candidate area extraction unit 241 is equal to or less than the TTC threshold value for preliminary control set in advance (=Yes), the process proceeds to S106; otherwise (=No), the process proceeds to S107.

[0062] In S107, a determination is made based on the control type and TTC, as in S103. If the control type set in S101 is warning, and the TTC calculated from the vehicle speed and the distance of the candidate area output by the candidate area extraction unit 241 is equal to or less than the preset TTC threshold for warning (=Yes), the process proceeds to S108; otherwise (=No), the result of bump re-recognition is output as non-recognition. In S104, the determination threshold is set to a determination threshold A that makes it easier to recognize than the bump recognition unit 204, and the process proceeds to S109. In other words, the determination threshold is set to a determination threshold A that is lower than the determination threshold in the bump recognition unit 204, and the process proceeds to S109. In S106, the process proceeds to S109 by setting a determination threshold B that is easier to recognize than the determination threshold A. In other words, the process proceeds to S109 by setting the determination threshold B to a value lower than the determination threshold in the bump recognition unit 204. In S108, the determination threshold C is set to be easier to recognize than the determination threshold B, and the process proceeds to S109. In other words, the determination threshold C is set to be lower than the determination threshold in the bump recognition unit 204, and the process proceeds to S109. In S109, the bump shape is determined using the set determination threshold value, but since this process is the same as the process of the bump recognition unit 204, details thereof will be omitted.

[0063] In this way, the environment recognition unit 102 of this embodiment 2 waits for output when the detection accuracy of road surface irregularities is low at a distance, and can make a judgment once the resolution of the stereo camera becomes sufficient for bump detection, which has the effect of suppressing false detection.

[0064] In the above embodiment, three determination processes, S103, S105, and S107, are provided, and when the TTC threshold of each determination process is equal to or less than the threshold, the determination threshold is set to a value that makes it easier to recognize the character, in the process of S104, S106, and S108. However, the present invention is not limited to this. For example, only the processes of S103 and S104, or only the processes of S103 to S106 may be performed.

[0065] [Embodiment 3] Hereinafter, a third embodiment of the environment recognition unit 102 according to the present disclosure will be described with reference to FIG.

[0066] 8 shows one embodiment of an in-vehicle system including an environment recognition unit 102 according to the present disclosure. The environment recognition unit 102 of this embodiment differs from the environment recognition unit 102 of the second embodiment in that it additionally includes a surrounding vehicle observation unit 801 and a map processing unit 802. Other aspects of the environment recognition unit 102 of this embodiment are similar to those of the environment recognition unit 102 of the previous embodiment, and therefore similar parts are denoted by the same reference numerals and description thereof will be omitted.

[0067] Since bumps are installed to the left and right ends of the road, if there is another vehicle (surrounding vehicle) traveling ahead in the same direction as the vehicle in an adjacent lane, that other vehicle will often decelerate just before the bump. The environment recognition unit 102 of this embodiment 3 is configured to detect the deceleration of another vehicle traveling ahead in an adjacent lane, and by changing the threshold for bump recognition, it is possible to make it easier to detect bumps even when not many bump features can be detected due to obstructions, etc.

[0068] In the environment recognition unit 102 of the first embodiment, the bump recognition section 204 outputs the bump recognition result and its accompanying information, but the bump recognition section 204 of the third embodiment differs in that it can estimate the presence of a bump by observing the behavior of other vehicles, even if the bump itself is not sufficiently observed by the stereo camera, making it easier to recognize the bump.

[0069] The surrounding vehicle observation unit 801 observes the behavior of preceding vehicles traveling in the same lane as the host vehicle or in adjacent lanes. Based on the area map of the area division results output by the driving area identification unit 203, the surrounding vehicle observation unit 801 extracts vehicle areas such as passenger cars and trucks, and painted areas such as white lines, and tracks the movement of each vehicle area using the time-series extraction results. In addition, the surrounding vehicle observation unit 801 calculates the relative positions of these areas with respect to the host vehicle from the camera installation information and parallax information corresponding to the extracted vehicle areas.

[0070] The map processing unit 802 calculates the behavior of other vehicles on a map based on the information about other vehicles observed by the surrounding vehicle observation unit 801 and the vehicle speed and steering angle of the vehicle itself. Specifically, it calculates on the map the traveling trajectory and traveling behavior of other vehicles, such as how the vehicle speed changed relative to the lane estimated from painted white lines and other markings. The map processing unit 802 places and tracks bump position candidates estimated from the bump features on the map.

[0071] In addition to the information described in the first or second embodiment, the bump recognition unit 204 also uses as input the travel path and travel behavior of other vehicles calculated by the map processing unit 802. The bump recognition unit 204 tracks the behavior of other vehicles on the map obtained by the surrounding vehicle observation unit 801, and when a preset condition is met, performs processing to lower the determination threshold so that the vehicle is more likely to be recognized as a bump. Specifically, when the travel path of the other vehicle is traveling along a lane and the vehicle is decelerating, there is a possibility that the vehicle is not changing lanes but that a bump is present ahead. Therefore, the width direction feature output by the width direction feature calculation unit 302 is used to lower the threshold used by the bump candidate area estimation unit 305 to estimate a bump candidate area, making it easier to recognize the bump. The bump recognition unit 204 performs processing to lower the determination threshold so that the vehicle is more likely to be recognized as a bump when one or more conditions are met, for example, when a vehicle in an adjacent lane has braked, the inter-vehicle distance has shortened, or the tail lamp illuminance has changed.

[0072] In this way, the environment recognition unit of this embodiment 3 has the advantage that even if the bump itself is not sufficiently observed by the stereo camera, it can estimate the presence of the bump by observing the behavior of other vehicles, making it easier to recognize the bump.

[0073] [Embodiment 4] Hereinafter, a fourth embodiment of the environment recognition unit 102 according to the present disclosure will be described with reference to FIGS.

[0074] FIG. 10 is a diagram illustrating a fourth embodiment of an in-vehicle system including an environment recognition unit according to the present disclosure, and FIG. 11 is a diagram illustrating the configuration of the environment recognition unit in the fourth embodiment. The environment recognition unit 102 of this embodiment 4 differs from the environment recognition unit 102 of the above-described embodiment 1 or embodiment 2 in that it additionally includes an environment recording device 1001 and a communication device 1002. Other aspects of the environment recognition unit 102 of this embodiment 4 are similar to the environment recognition unit 102 of the above-described embodiment 1 or embodiment 2, so similar parts are denoted by the same reference numerals and description thereof will be omitted.

[0075] The environment recognition unit 102 in the first embodiment operates independently of the vehicle itself, recognizing the ever-changing environment around the vehicle at any time based on external information obtained by a camera mounted on the vehicle. However, the environment recognition unit 102 in the fourth embodiment differs in that it keeps records and communicates with the outside world as needed.

[0076] The environment recording device 1001 can record the recognition results of the environment recognition unit 102 and the vehicle position output by a locator device (not shown) as a set, and can also read out the recognition results and recorded positions that exist near the input vehicle position from the content recorded using the vehicle position as a query.

[0077] The communication device 1002 may communicate with one or more of other vehicles, road facilities, or map servers, and upload the contents recorded by the vehicle in the environment recording device 1001. The communication device 1002 corresponds to a communication unit in the claims. Conversely, the communication device 1002 may communicate with one or more of other vehicles, road facilities, or map servers, and receive recognition results such as bump positions and heights observed by devices other than the vehicle, and adjust parameters or override the recognition results based on the received information so that the environment recognition unit 102 can more easily recognize the bumps.

[0078] Next, a method in which the environment recognition unit 102 detects a bump based on an image captured by the camera unit 101 will be described.

[0079] 11 is a block diagram showing the internal configuration of the camera unit 101 and the environment recognition unit 102. The environment recognition unit 102 of this embodiment differs from the environment recognition unit 102 of the first to third embodiments described above in that it additionally includes a self-position estimation unit 1101, such as a locator device, an environment recording unit 1102, and a communication device 1103.

[0080] The self-position estimation unit 1101 has a function of estimating the position of the vehicle by receiving latitude, longitude, and altitude obtained by a GNSS receiver, for example, and outputs the result to the environment recording unit 1102. Alternatively, the function may be to recognize the environment around the vehicle using a camera or LiDAR, and estimate the position of the vehicle by comparing it with an internal map (not shown).

[0081] The environment recording unit 1102 receives as input the distance to the bump, height, and recognition reliability calculated by the bump recognition unit 204 through the processing described in the first embodiment, as well as the self-position of the self-position estimation unit 1101, and records these as a set of bump information on a hard disk (HDD), solid state disk (SSD), memory, etc. Furthermore, the environment recording unit 1102 reads out the recorded information, and when the self-position of the self-position estimation unit 1101 is input, outputs to the bump recognition unit 204 one or more of the position, height, and recognition reliability of a nearby bump within a threshold value of the input self-position.

[0082] In the bump recognition unit 204, if the position and height of the bump obtained from the environment recording unit 1102 are available and the recognition reliability is greater than a predetermined threshold, the position, height, and recognition reliability of the corresponding bump may be replaced with the information obtained from the environment recording unit 1102. In this way, instead of performing detection several times from a distance to improve the detection reliability index, it becomes possible to detect bumps from a greater distance with higher detection reliability.

[0083] The communication device 1103 may communicate the contents of the environment recording unit 1102 with one or more of other vehicles, road facilities, and map servers to upload or download bump information indicating the position and height of the bump. This makes it possible to detect bumps that the vehicle has not detected before, such as bumps that exist on roads the vehicle is traveling on for the first time, from a greater distance with a higher degree of detection reliability, and to take action against the bump from a greater distance.

[0084] According to the first to fourth embodiments of the present invention described above, the camera unit 101 installed in the vehicle 100 is used to observe the surroundings, and it is determined from three-dimensional information obtained by the camera unit 101 whether or not a collision will occur even if the vehicle 100 proceeds. Even if it is determined that a collision will not occur, it is possible to analyze from the obtained three-dimensional road surface information whether or not there are any irregularities, such as bumps, ahead of the vehicle 100 that may cause large vibrations to the vehicle 100, and based on the analysis results, it is possible to notify the driver and control the suspension and brakes to suppress vibrations of the vehicle 100. That is, the shape of the road surface ahead of the vehicle 100 is measured using the camera unit 101, and based on the measurement results, it is possible to alert the driver before vibrations are transmitted to the vehicle 100 for safer driving and control the vehicle 100 for more comfortable driving. For example, if it is predicted that vehicle 100 will enter a bump area at a speed higher than the appropriate vehicle speed range, the driver can be alerted by sound or display to slow down vehicle 100, or control can be performed to suppress acceleration or decelerate vehicle 100 just before the bump area, or semi-active control can be implemented to soften the suspension of vehicle 100 just before the bump, thereby suppressing the impact when passing over the bump, thereby mitigating the impact on the vehicle body.

[0085] Furthermore, according to the above-described first to fourth embodiments of the present invention, when using disparity information obtained from, for example, a stereo camera to detect road bumps (e.g., bumps, deceleration bumps, cracks, potholes, etc.), it is possible to suppress false detections and degradation of detection performance due to the fundamental characteristics of the stereo camera. That is, because the resolution of disparity information obtained from a stereo camera is lower at a distance than at a nearby location, false detections may not be eliminated when attempting to confirm a detection at a distance because low-resolution disparity information is used. Furthermore, if the detection confirmation is always waited for until the disparity information has high resolution and the detection is confirmed only when the vehicle is close, the detection distance may be shortened. According to the first to fourth embodiments of the present invention, it is possible to simultaneously suppress false detections and extend the detection distance, thereby improving the accuracy of the timing of warnings and control for detected bump areas and reducing false alarms and erroneous control.

[0086] Although the embodiments of the environment recognition unit according to the present disclosure have been described in detail above using the drawings, the specific configuration is not limited to this embodiment, and design changes and the like that do not deviate from the gist of the present disclosure are also included in the present disclosure. For example, the above-described embodiments have been described in detail to clearly explain the present disclosure, and are not necessarily limited to those that include all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0087] 100···Vehicle, 101···Camera unit (camera), 102···Environment recognition unit (environment recognition device), 103···Control unit, 201··Stereo camera, 202···Bump feature generation unit, 203···Driving area identification unit, 204···Bump recognition unit, 205···Result output unit, 241···Candidate area extraction unit, 242···Visible area determination unit, 301···High likelihood area extraction unit, 302···Width direction feature calculation unit, 303···Depth direction feature calculation unit, 304···Contour shape feature calculation unit, 305···Bump candidate area estimation unit, 401···Bump tentative candidate estimation unit, 402···Distance / TTC calculation unit, 403: Road width estimation unit, 404: Obstructed area estimation unit, 405: Bump visible area estimation unit, 406: Overlap rate determination unit for visible area, 501: Bump re-recognition unit, 801: Surrounding vehicle observation unit, 802: Map processing unit, 901: Road surface, 902: Speed bump, 903: Leading vehicle, 904: Bump area point cloud, 905: Flat road surface point cloud, 906: Area, 907: Area strongly responding to bump, 908: Area corresponding to bump area, 1001: Environmental recording device, 1002: Communication device, 1101: Self-position estimation unit, 1102: Environmental recording unit

Claims

1. An environment recognition device that recognizes bumps installed on a road surface based on an image captured by a camera mounted on a vehicle, a travel area identification unit that identifies a travel road area in the image in which the travel road on which the vehicle is traveling is captured; a bump feature generation unit that generates a feature map that represents an area including the bump feature from the image together with a degree of likelihood; a bump recognition unit that recognizes the bump based on a bump candidate region in the feature map whose degree of likelihood is equal to or greater than a threshold, The bump recognition unit recognizes, as the bump, one of the bump candidate areas present on the roadway area, in which at least one of a length in a width direction of the roadway area, a length in a depth direction of the bump candidate area, or a complexity of a contour line of the bump candidate area satisfies a criterion having a predetermined judgment threshold. An environment recognition device characterized by:

2. the travel area identification unit identifies an object area in the image in which an object present on the travel path is captured, The bump recognition unit recognizes the bump based on a visible area excluding a blind spot area blocked by the object area from the road area.

2. The environment recognition device according to claim 1.

3. a bump re-recognition unit that changes the determination threshold based on a control type indicating a control state of the vehicle and a distance from the vehicle to the bump candidate area, and, if there is a bump candidate area that has not been recognized as a bump up to the previous processing cycle, re-recognizes the bump based on the changed determination threshold; 2. The environment recognition device according to claim 1.

4. A surrounding vehicle observation unit is provided to observe the behavior of other vehicles in adjacent lanes, The bump recognition unit lowers the determination threshold so that the bump is more easily recognized when one or more of the following conditions are met: another vehicle in the adjacent lane brakes; the inter-vehicle distance is shortened; and the tail lamp illuminance is changed.

2. The environment recognition device according to claim 1.

5. a map processing unit that arranges and tracks position candidates of the bump estimated from the feature map on a map; The bump recognition unit tracks the behavior of other vehicles on the map obtained by the surrounding vehicle observation unit, and when the behavior meets a preset condition, lowers the determination threshold so that the bump is more likely to be recognized.

5. The environment recognition device according to claim 4.

6. a communication unit capable of communicating with one or more of another vehicle different from the vehicle, road facilities, or a server; an environment recording unit that records bump information including at least the position of the bump; The communication unit transmits and receives the bump information recorded in the environment recording unit to and from one or more of the other vehicles, road facilities, and servers.

2. The environment recognition device according to claim 1.

7. a self-position estimation unit that estimates a position of the vehicle; The bump recognition unit recognizes the bump based on the vehicle's own position estimated by the self-position estimation unit and the bump information received from one or more of the other vehicles, road facilities, and map servers via the communication unit.

7. The environment recognition device according to claim 6.

Images