Environment recognition device

Abstract

The purpose of this invention is to improve the accuracy of the timing of attention arousal and control of bulges, and to reduce false alarms and false control. The environment recognition unit (102) has at least a bulge feature generation unit (202), a driving area determination unit (203), a bulge recognition unit (204), and a result output unit (205). In addition, the bulge recognition unit (204) includes a candidate area extraction unit (241) and a visible area determination unit (242). The bulge recognition unit (204) identifies a bulge candidate area that meets at least one of the following criteria: its length relative to the width direction of the driving area, its length in the depth direction of the bulge candidate area, or the complexity of the contour line of the bulge candidate area, as a bulge.

Description

Technical Field

[0001] This invention relates to a driving environment detection technology based on camera images to detect surrounding objects in a driving environment. Background Technology

[0002] Patent Document 1 discloses a vehicle control device that can provide occupants and the system with information corresponding to the vehicle's position even when GNSS (Global Navigation Satellite System) is unavailable. This vehicle control device detects areas where objects exist (object regions) based on image data or 3D data. The method for detecting these object regions includes methods such as: detecting road elevation differences, such as speed bumps, by detecting a portion of the road surface in the direction of travel, and detecting the road surface's unevenness from its depth information. Furthermore, it describes that in detecting road surface unevenness, the maximum and minimum values ​​of the depth information are detected separately, and the corresponding points on the time series are tracked as the vehicle moves, with points that are successfully tracked multiple times being taken as the location of the elevation difference, thereby enabling accurate detection of elevation differences. Another method involves not detecting the maximum and minimum values ​​of the road surface, but instead using machine learning with depth data as input to learn which locations represent elevation differences as teacher data. According to this method, it is possible to detect even complex shapes, multiple consecutive height differences, etc., which is effective in improving detection accuracy (Patent Document 1, Paragraph 0007, Paragraph 0027, etc.).

[0003] Furthermore, Patent Document 2 discloses an object recognition device that can accurately identify objects captured across overlapping and non-overlapping regions of the field of view in multiple images acquired by a stereo camera. The device discloses that "the reliability calculation unit is composed of a first reliability calculation unit and a second reliability calculation unit. The first reliability calculation unit calculates the object's recognition score (recognition result) based on three-dimensional information, and the second reliability calculation unit calculates the object's recognition score (recognition result) based on texture information." It also states that "the acquired three-dimensional information can be superimposed on a two-dimensional image to generate a distance image and a disparity image, and recognition processing is performed based on the generated distance image and disparity image." Furthermore, "in the object re-detection processing, the object is detected by analyzing the texture information of the region. As a detection method, a convolutional neural network is used. In the convolutional neural network, the upper-left and lower-right image positions of the object are output. Through the output of the convolutional neural network, the region of the vehicle captured across overlapping and non-overlapping regions of the field of view can be determined." Additionally, it discloses that the area of ​​the object region, which is the processing area, is compared with the recognition score and a predetermined threshold to determine the object as the object to be recognized. According to this method, it is possible to claim high-precision identification of objects captured across overlapping and non-overlapping regions of the field of view (Patent Document 2, paragraphs 0015, 0016, 0024 to 0028, etc.).

[0004] Prior technology documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2023-12649

[0007] Patent Document 2: Japanese Patent Application Publication No. 2022-64388 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] The vehicle control device disclosed in Patent Document 1 describes a method for detecting road height differences, such as speed bumps, by detecting the road portion in the direction of travel, measuring the maximum and minimum values ​​of depth and height information, tracking corresponding points in a time series as the vehicle moves, and using points that are successfully tracked multiple times as the location of the height difference, thereby detecting the height difference with good accuracy. However, in actual driving environments, the mere presence of uneven road surfaces is not equivalent to the presence of bumps that require attention and various controls. Generally, bumps are convex road surface protrusions that span the entire width of the road, and are objects that cannot be avoided by steering control, but the vehicle control device in Patent Document 1 cannot detect only such objects.

[0010] The object recognition device disclosed in Patent Document 2 describes acquiring texture information of areas captured across overlapping and non-overlapping areas of the field of view from images taken by two cameras. However, in actual driving environments, other vehicles are also in motion, and it is not always possible to capture the entire road surface with convex features. Although there are blind spots on the road surface caused by other vehicles, Patent Document 2 does not consider this, and there is still a possibility that convex road surface features spanning the entire width of the road cannot be correctly detected. Therefore, it is impossible to properly attract attention and control the detected convex areas.

[0011] Therefore, the purpose of this invention is to detect bumps in a driving environment while simultaneously suppressing false detections and extending the detection distance, thereby improving the accuracy of the timing of attention arousal and control of detected bump areas and reducing false alarms and miscontrols.

[0012] Methods for solving problems

[0013] One aspect of this disclosure is an environmental recognition device that identifies road surface protrusions based on images captured by a camera mounted on a vehicle. The environmental recognition device is characterized by comprising:

[0014] The driving area determination unit determines, in the image, the driving road area where the vehicle is traveling, as captured in the image;

[0015] A bulge feature generation unit generates a feature map based on the image, the feature map representing the region containing the bulge feature along with the degree of accuracy; and

[0016] The bulge recognition unit identifies the bulge based on bulge candidate regions in the feature map whose accuracy exceeds a threshold.

[0017] The bulge identification unit identifies a bulge candidate region that meets at least one of the following criteria: its length relative to the width direction of the driving road area, its length in the depth direction of the candidate region, or the complexity of the outline of the bulge candidate region, as the bulge candidate region.

[0018] Invention Effects

[0019] According to one of the above-described methods of this disclosure, an environmental recognition device can be provided that detects bumps in a driving environment while simultaneously suppressing false detections and extending the detection distance, thereby improving the accuracy of the timing of attention arousal and control of the detected bump area and reducing false alarms and miscontrols.

[0020] Further features related to the present invention become apparent from the description and drawings herein. Furthermore, issues, structures, and effects other than those described above become apparent from the following description of embodiments. Attached Figure Description

[0021] Figure 1 This is a diagram illustrating Embodiment 1 of an in-vehicle system including the environmental identification device involved in this disclosure.

[0022] Figure 2 This is a diagram illustrating the internal structure of the camera unit and the environment recognition unit in Embodiment 1.

[0023] Figure 3 This is a diagram illustrating the structure of the candidate region extraction section.

[0024] Figure 4 This is a diagram illustrating the structure of the visible area determination section.

[0025] Figure 5 This is a diagram illustrating the internal structure of the camera unit and the environment recognition unit in Embodiment 2.

[0026] Figure 6 This is a diagram illustrating the processing flow of the raised re-identification section.

[0027] Figure 7 This is a diagram showing an example of an output decision table inside the result output section.

[0028] Figure 8 This is a diagram illustrating the internal structure of the camera unit and the environment recognition unit in Embodiment 3.

[0029] Figure 9 This is a diagram showing an example of a feature map.

[0030] Figure 10 This is a figure illustrating embodiment 4 of an in-vehicle system that includes the environmental identification device involved in this disclosure.

[0031] Figure 11 This is a diagram illustrating the internal structure of the camera unit and the environment recognition unit in Embodiment 4. Detailed Implementation

[0032] Hereinafter, embodiments of the environmental identification device involved in this disclosure will be described with reference to the accompanying drawings.

[0033] [Implementation Method 1]

[0034] Figure 1 This is one embodiment of an in-vehicle system that includes the environmental identification unit 102 disclosed herein.

[0035] exist Figure 1In this system, a camera unit 101 is set up with a field of view that includes the direction of travel of the vehicle 100. The image of the front of the vehicle captured by the camera unit 101 is input to an environment recognition unit (environment recognition device) 102.

[0036] The environment recognition unit 102 can be configured, for example, by one or more microcontrollers equipped with a central processing unit (CPU), memory such as ROM and RAM, a timer, and an input / output unit. Based on an image of the area in front of the vehicle captured by the camera unit 101, the environment recognition unit 102 identifies bumps on the road surface. Furthermore, it calculates the distance to the bump, the height of the bump, the relative speed, and the reliability index of the detection, and sends the calculation results to the control unit 103.

[0037] The control unit 103 determines the degree of danger of impact or collision based on the distance to the bump, the height of the bump, the relative speed, and the reliability index of the detection. According to the degree of danger, it outputs commands such as displaying on the display 104, illuminating the warning light, emitting a warning sound with the speaker 105, mitigating or suppressing the acceleration of the accelerator 106, decelerating with the brake 107, and changing the impact absorption characteristics of the suspension 108. Similar to the environmental recognition unit 102, the control unit 103 can be, for example, composed of one or more microcontrollers equipped with a central processing unit (CPU), memory such as ROM and RAM, timers, and input / output units.

[0038] Furthermore, in this embodiment, the case where the raised part is a speed bump is described, but it is not limited thereto. In this embodiment, the raised part can be any one or more of the following: a bump that causes a vibration of a specified magnitude or greater to the vehicle when it passes over it. In addition to speed bumps, it can be any one or more of the following: a pedestrian crossing, a tram line or level crossing, or a joint of a bridge or viaduct.

[0039] Speed ​​bump 902, which is the object of identification by environmental identification unit 102 (see reference) Figure 9 In the presence of a lane for the vehicle to travel in the same direction as the other adjacent lanes, the length extending in the width direction of the road surface 901 that spans and also includes the adjacent lanes is not included in the identification object.

[0040] Next, use Figure 2 This describes the method by which the environmental recognition unit 102 detects bumps based on images captured by the camera unit 101.

[0041] Figure 2This is a block diagram showing the internal configuration of the camera unit 101 and the environment recognition unit 102. Furthermore, this description uses the case where the camera unit 101 is a stereo camera 201 as an example, but it can also be a configuration combining one or more cameras such as a monocular camera with LiDAR, a configuration using three or more cameras, or a configuration combining one or more cameras with a millimeter-wave sensor.

[0042] The environment recognition unit 102 includes at least a raised feature generation unit 202, a driving area determination unit 203, a raised feature recognition unit 204, and a result output unit 205. Furthermore, the raised feature recognition unit 204 includes a candidate region extraction unit 241 and a visible region determination unit 242.

[0043] The stereo camera 201 consists of two cameras. Each camera is equipped with a CMOS (Complementary Metal Oxide Semiconductor) sensor and a DSP (Digital Signal Processor). The CMOS sensor is an imaging device in which photodiodes with photoelectric effect are arranged in a lattice. When the shutter is pressed at any time, the charge accumulated up to that moment is converted to digital (AD) and transmitted to the DSP. In the DSP, the transmitted information is converted into an image while performing image quality corrections such as contrast correction, gamma correction, and edge correction. Distance information (distance image) and invalid regions are calculated from these images as parallax information. Furthermore, invalid regions here refer to areas with little texture, high similarity, or other areas where stable parallax information is difficult to calculate accurately. Distance information and invalid regions are combined and referred to as parallax information. Since the details of how this parallax information is calculated exist in various well-known stereo image processing methods, the explanation here is omitted. The parallax information and at least one of the images are sent to the bulge feature generation unit 202 and the driving area determination unit 203, respectively.

[0044] The bulge feature generation unit 202 generates a feature map representing the region containing bulge features along with the accuracy level based on the image captured by the camera unit 101. The bulge feature generation unit 202 takes disparity information and shading images acquired from the stereo camera 201 as input and outputs a feature map with a likelihood of 0.0 to 1.0 for the bulge features. This feature map can be implemented using a Convolutional Neural Network (CNN) or similar technology. Multiple sets are prepared in advance, using disparity information with a resolution of width W1 × height H1 acquired by the stereo camera 201, shading images with a resolution of width W2 × height H2 acquired by the stereo camera 201, and labeled images with a resolution of width W3 × height H3 as one set of learning data. The CNN network structure is then learned through supervised learning based on this learning data. By learning the CNN network structure in advance, if the disparity information and shading images obtained from the stereo camera 201 are input into the CNN during driving, a feature map representing the likelihood of the bulge can be obtained.

[0045] Furthermore, the widths W1, W2, W3 and the heights H1, H2, H3 of the resolution can be the same or different. By setting them to the same resolution, the network construction becomes simpler and the learning efficiency is improved. On the other hand, when the baseline of the stereo camera 201 is horizontal, that is, when the two cameras are set up separately in the horizontal direction, since the actual resolution of the disparity information in the horizontal direction is lower than that in the vertical direction, the resolution of the disparity information and the horizontal width of the output feature map is only set to half, that is, W1=W3=W2÷2, H1=H2=H3. This has the effect of suppressing the reduction in the accuracy of the output feature map while suppressing the computational cost.

[0046] exist Figure 9 An example of a feature map is shown below. Assume... Figure 9 (a1) is the input camera image, showing the road surface unevenness at that time. Figure 9 (a2) Figure 9 (b) shows the input disparity information. Figure 9 (c) illustrates the concept of a feature map representing the output likelihood. Here, it is assumed to be a density image acquired from the stereo camera 201, such as... Figure 9 As shown in (a1), the nearest road surface 901, the speed bump 902 which is positioned to overlap with the pedestrian crossing slightly ahead, and the leading vehicle 903 further ahead are reflected.

[0047] according to Figure 9As shown in (a2), the road surface unevenness indicates that the road surface 901, excluding the speed bump 902, is almost flat. Under these conditions, the parallax information acquired by the stereo camera 201 becomes as follows: Figure 9 As in (b), point group 904 can be acquired in front of point group 905 on a flat road surface. These point groups can be calculated based on the principles of general stereo vision, utilizing the texture of asphalt and road markings. In the ridge feature generation unit 202, as a method for generating images of varying shades ( Figure 9 (a1) and disparity information ( Figure 9 (b)) Further coating Figure 9 The so-called labeled image, which is equivalent to the speed bump 902 in (a1), was used as input for learning, and during inference, it can output, as... Figure 9 The feature map shown in (c). Figure 9 The feature map shown in (c) depicts a situation where the likelihood is 0 outside region 908 corresponding to the speed bump 902, the likelihood increases inside region 908, and the likelihood is even greater in regions 906 and 907, which strongly respond to the bump. This is represented here by the input shading image ( Figure 9 (a1)) feature maps of the same size Figure 9 (c) shows the situation where the likelihood values ​​are output for their respective corresponding pixels.

[0048] The driving region determination unit 203 determines the driving road region where the vehicle 100 travels and the object region where objects existing on the driving road are captured in the images captured by the camera unit 101. The driving region determination unit 203 inputs disparity information and density images acquired from the stereo camera 201 and outputs a region map showing the object category of each pixel. This region map can be implemented using semantic segmentation (SS). Specifically, multiple sets are prepared using disparity information acquired in advance by the stereo camera 201, density images also acquired by the stereo camera 201, and labeled images indicating the category of each pixel in the density images. These are used as one set of learning data to learn the CNN network structure through supervised learning based on this learning data. Furthermore, the labeled images are, for example, images where each pixel is labeled, such as asphalt areas (=1), painted areas like white lines and speed limits (=2), vehicle areas like passenger cars and trucks (=3), two-wheeled vehicle areas like bicycles and motorcycles (=4), pedestrian areas (=5), and animal areas like cows and sheep (=6). By learning the CNN network structure in advance, if the disparity information and density images obtained from the stereo camera 201 are input into the CNN during driving, a region map can be obtained as the result of semantic segmentation inference.

[0049] The bump recognition unit 204 receives a feature map output from the bump feature generation unit 202 and a region map output from the driving area determination unit 203, determines whether a bump exists, and outputs the bump recognition result and its accompanying information to the result output unit 205. The bump recognition result and its accompanying information include information such as whether a bump was detected, the distance from the vehicle to the bump if it was detected, the height from the road surface to the apex of the bump, the width of the bump, and the confidence level during recognition.

[0050] The result output unit 205 takes the output result of the bulge recognition unit 204 as input, and generates output information for the control unit 103 based on the time-series bulge recognition results and accompanying information. Generally speaking, since the possibility of misidentification by external recognition through image recognition is not zero, the result output unit 205 can also determine whether the final recognition output is acceptable based on multiple recognition results.

[0051] Specifically, the output of the bulge recognition unit 204 and accompanying information can be used as input for the following calculation. For example, suppose the following information is obtained: Suppose the recognition success or failure flag F is obtained for N time intervals t0, t-1, ..., t-(N-1), where the value is 1 when a bulge is recognized and 0 when it is not. t0 F t-1..., F t-（N-1） The distance D to the identified bulge t0 D t-1 ... D t-（N-1） The height H of the identified bulge t0 H t-1 ... H t-（N-1） At this point, it is possible to satisfy (Condition A) ΣFx≧Thr1 and (Condition B) D. t0 ≧D t-1 ≧...≧D t-（N-1） If any one of the three conditions (condition C) Var(H x) ≦ Thr2 is met, a bump identification result and accompanying information are generated as output information for the control unit 103.

[0052] That is, (Condition A) means that Thr1 or more detections need to be performed between N time points. If the distance from the vehicle to the object changes as the vehicle travels and Thr1 or more detections are performed, it can eliminate false detections caused by sudden events. (Condition B) means that the distance to the bulge shortens as time passes, monotonically approaching the bulge, which can eliminate false detections caused by accidental light or shadow. (Condition C) means that the dispersion value of the observed bulge height is below a certain threshold Thr2 and is observed stably. Since the physical shape of the bulge does not change in a short time, if the distance from the vehicle to the object changes as the vehicle travels and the bulge is observed multiple times, verifying that its shape or height is unchanged, it can eliminate false detections that may occur accidentally due to noise. In addition, (Condition B) and (Condition C) can also use only the data from the time the bulge was identified.

[0053] Furthermore, the detailed values ​​of the detection conditions are design considerations, and results can also be determined using conditions other than those mentioned above. Additionally, while condition B above only considers monotonically approaching the bulge as a criterion, a method commonly known as dead reckoning can be used to calculate the vehicle's travel distance or movement distance. The change in travel distance or movement distance compared to the distance to the bulge is then used as a criterion, with the difference being below a certain value and almost identical.

[0054] In addition, the bulge recognition unit 204 has a candidate region extraction unit 241 and a visible region determination unit 242.

[0055] Regarding the candidate region extraction unit 241, using... Figure 3 Please provide an explanation.

[0056] The candidate region extraction unit 241 takes the raised features (hereinafter also referred to as feature maps) generated in the raised feature generation unit 202 as input and outputs raised candidate regions. The candidate region extraction unit 241 extracts regions that exist in the raised candidate regions on the driving road area and whose length in the width direction relative to the driving road area is above the determination threshold as raised candidate regions.

[0057] The candidate region extraction unit 241 includes a high likelihood region extraction unit 301, a width direction feature calculation unit 302, a depth direction feature calculation unit 303, a contour shape feature calculation unit 304, and a raised candidate region estimation unit 305.

[0058] The high-likelihood region extraction unit 301 determines candidate bulge regions based on the feature map generated by the bulge feature generation unit 202, and further calculates their relative positions with respect to the vehicle. As mentioned above, the feature map is output such that the likelihood increases inside region 908 corresponding to the bulge region, and regions 906 and 907 with higher bulge certainty become regions with even greater likelihood. Therefore, by comparing the likelihood of each point with the threshold ThrC and extracting the larger points, high-likelihood regions with high bulge certainty can be extracted. For example, the region with the largest area among the high-likelihood regions with high bulge certainty is extracted, and the centroid coordinates of the extracted region are obtained to determine the candidate bulge region. In this way, the position of the candidate bulge region, such as its distance from the front of the vehicle and its deviation to the left or right relative to the center of the vehicle, can be calculated. Alternatively, after extracting the region with the largest area, the candidate bulge region can be excluded and the same processing can be performed again, thereby calculating the next region with a large area as the second candidate bulge region and calculating its relative position with respect to the vehicle. In this way, the high likelihood region extraction unit 301 can also calculate more than one candidate region for elevation and output it as a candidate region for elevation along with its area information or the order information of which region is the largest.

[0059] The width direction feature calculation unit 302 calculates the length of the candidate region of the raised area calculated by the high likelihood region extraction unit 301, which is orthogonal to the path of the vehicle travel, and is the width direction feature FW, which is the length of the width direction of the raised area.

[0060] The depth direction feature calculation unit 303 calculates the length of the direction of the vehicle's driving path for the region of the candidate region of the raised area calculated by the high likelihood region extraction unit 301, which is the depth direction feature FD that serves as the length of the depth direction of the raised area.

[0061] The contour shape feature calculation unit 304 calculates the contour shape feature FC, which is an indicator of the complexity of the contour lines, for the region of the candidate bulge calculated by the high likelihood region extraction unit 301. Specifically, regarding the straightness of the edges in the width direction of the bulge, the top and bottom edges of the candidate bulge can be approximated by a straight line using a Hough transform, and the dispersion value of the points constituting the contour lines relative to the approximate straight lines can be calculated. When the candidate bulge is a perfect trapezoid, such as one where the top and bottom edges are straight lines, the dispersion value of this straightness is 0.

[0062] The bulge candidate region estimation unit 305 receives outputs from the width direction feature calculation unit 302, the depth direction feature calculation unit 303, and the contour shape feature calculation unit 304. For each region extracted by the high likelihood region extraction unit 301, it determines whether it is a bulge candidate region based on whether the calculated features meet a predetermined judgment threshold. Furthermore, when a region is a bulge candidate region, the bulge confidence level is calculated to be close to 1 if the bulge characteristics are high, and close to 0 if not.

[0063] Specifically, if the region is a ridge, the width-direction feature FW is expected to be at least above the vehicle width, so thresholds ThrW1 (equivalent to vehicle width) and ThrW2 (equivalent to road width) are set. If the value is smaller than ThrW1, it is determined not to be a candidate region for a ridge; otherwise, it proceeds to the next step. If the region is a ridge, the depth-direction feature FD is expected to be between thresholds ThrD1 and ThrD2, which are considered standard ridge depth directions. Therefore, if the value is not included in this range, it is determined not to be a candidate region for a ridge; otherwise, it proceeds to the next step. If the region is a ridge, the contour shape feature FC is expected to have a straight boundary between the road surface and the ridge. Therefore, assuming the contour shape feature FC is the dispersion value of points forming a contour line relative to an approximately straight line, if the shape feature FC is above ThrC, it is determined not to be a candidate region for a ridge; otherwise, it is determined to be a candidate region for a ridge. By doing so, candidate regions for raised areas can be extracted based on the respective feature values ​​of the width direction feature calculation unit 302, the depth direction feature calculation unit 303, and the contour shape feature calculation unit 304.

[0064] Furthermore, in the above example, the case of extracting the candidate region of the bulge based on the respective feature values ​​of the width direction feature calculation unit 302, the depth direction feature calculation unit 303, and the contour shape feature calculation unit 304 has been described. However, the candidate region of the bulge can also be extracted based on at least one feature value.

[0065] Regarding the confidence level for bulges, it can also be designed to decrease as the confidence level approaches its respective judgment threshold. Specifically, by setting the confidence level ConfW for the width-direction feature as ConfW=0 (FW<ThrW1), ConfW=1 (FW>ThrW2), ConfW=-(FW-ThrW2)÷(ThrW1-ThrW2)÷2+1 (FW≥ThrW1, FW≤ThrW2), the confidence level drops to around 0.5 when the width-direction feature FW is almost equal to the vehicle width. This can be calculated as FW getting wider and closer to 1. Other confidence levels can be designed similarly.

[0066] Regarding the visible area determination unit 242, using... Figure 4 The visible area determination unit 242 takes information about candidate ridge regions and ridge features (feature map) and driving road area (area map) as input, identifies ridges based on the repetition rate of candidate ridge regions relative to the visible area, and outputs the identification result and its accompanying information. The visible area determination unit 242 includes a ridge provisional candidate estimation unit 401, a distance / TTC calculation unit 402, a driving road width estimation unit 403, an occlusion area estimation unit 404, a ridge visible area estimation unit 405, and a repetition rate determination unit 406 relative to the visible area.

[0067] The ridge candidate estimation unit 401 selects the ridge candidate regions extracted by the candidate region extraction unit 241, starting with the candidate regions with the highest confidence, and processes them sequentially. This process can terminate when computational resources and processing time are insufficient, or when processing of all ridge candidate regions is complete. Therefore, even when multiple ridge candidate regions exist in the image, it effectively prevents processing time from significantly exceeding the limit.

[0068] The distance / TTC calculation unit 402 calculates the distance to the vehicle based on the provisional candidate for a bump output from the provisional candidate estimation unit 401. Several known techniques exist for calculating this distance. For example, when the vehicle is moving stably, since the camera's mounting height and line of sight are known according to design values, and the camera's field of view and distortion information are also known, the location on the road surface where the vehicle is traveling in physical space corresponding to the coordinates observed on the image can be easily calculated using the principle of triangulation, and the distance can be calculated. Alternatively, when using a stereo camera, the distance can be calculated using the principle of triangulation based on the coordinates observed on the left and right cameras and the relative positional relationship between the left and right cameras. Based on the calculated distance and the vehicle's speed, the TTC (Time To Collision) to the bump can be calculated. Furthermore, the coordinates observed on the camera at this time can be calculated either by using the center of the provisional candidate area for the bulge, or by averaging the centers of the near-front (closest to the vehicle) and inner (farthest from the vehicle) sides of the provisional candidate area as distances. By doing so, the inner side of the image shrinks due to perspective transformation, while the near-front side expands. Therefore, simply taking the center of the candidate area on the image would result in a coordinate closer to the front than the actual center. However, this method can suppress this situation and achieve the effect of calculating coordinates that are closer to the true center.

[0069] The driving road width estimation unit 403 estimates the width of the driving road based on the region map determined by the driving area determination unit 203, which calculates the object category of each pixel. For example, it can calculate the distance between the left and right ends of the road at 10 locations 11m ahead, 12m ahead, ..., 20m ahead and output the mode, or it can extract the left and right road boundary lines based on the boundary between the driving road area and the off-road area from the region map of the region segmentation result, and use the interval between them as the width of the driving road.

[0070] The occlusion area estimation unit 404 estimates the extent to which the object region of a vehicle or obstacle occludes the road surface (blind spot area) at a distance where a preliminary candidate for a raised area exists, based on the region map determined by the driving region determination unit 203, which calculates the object category of each pixel. Furthermore, if the vehicle or obstacle is located further away than the preliminary candidate for a raised area, the width of the road surface occluded by them is zero. On the other hand, if the vehicle or obstacle is located closer than the preliminary candidate for a raised area, the width of the road surface occluded by them becomes wider than the width of the vehicle and obstacle themselves. Specifically, when a vehicle or obstacle with a width of Width is located at a distance of Dist1 from the camera, the occluded road surface width Mask (blind spot area) at a distance of Dist2 where a preliminary candidate for a raised area exists is = Width × Dist2 ÷ Dist1 (in this case, Dist1 < Dist2).

[0071] The visible area estimation unit 405 estimates the visible area of ​​the driving road area, excluding blind spots obscured by object areas. The visible area estimation unit 405 calculates the unobscured road width at a distance where a provisional candidate for a raised area exists. Specifically, it calculates the width of the driving road (Lane) at the distance where a provisional candidate for a raised area exists, and the obscured road width (Mask), by subtracting the obscured road width (Mask) from the driving road width (Lane) (=Lane-Mask).

[0072] The repetition rate determination unit 406 determines the degree to which the candidate region for the raised area output from the candidate region extraction unit 241 is consistent with the width of the visible raised area estimated by the visible region estimation unit 405 at that distance. Specifically, the repetition rate is calculated within the range of 0 to 1 by dividing the estimated width of the visible raised area by the width of the candidate region for the raised area. If the repetition rate is greater than the threshold ThrO, the candidate region is determined to be sufficiently repetitive with respect to the visible area and is output as a detected raised area.

[0073] In this manner, the bulge recognition unit 204, which has a candidate region extraction unit 241 and a visible region determination unit 242, identifies bulges. The bulge recognition unit 204 identifies bulges based on the visible regions that exclude blind spots obscured by object regions from the driving road region.

[0074] The output unit 205 takes the bump recognition result and its accompanying information identified by the bump recognition unit 204, plus one or more pieces of information such as the vehicle's speed, driving environment, and lighting environment as input, and determines how the output should be controlled. Figure 7 Provide specific examples to illustrate. Figure 7 This is an example of an output determination table inside the result output section 205 (not shown).

[0075] For example, scenario No.1 envisions driving on a paved urban road during a clear day. In this case, the output parameters are set to the standard output determination distance (30m for alarm, 20m for preparatory control, and 10m for deceleration control). Scenario No.2 envisions driving on a paved urban road at night. In this case, the vehicle's headlights create shadows on bumps and depressions, making detection easier and reducing the likelihood of false detections. In this scenario, since the driver may have difficulty seeing the bumps, the alarm and preparatory control are performed as in daytime, but the deceleration control is set to be performed earlier. Furthermore, in environments like rainy nights, where false detections are more likely, as in No.3, the output parameters are set to not perform deceleration control to suppress false control. Finally, in unpaved roads, as in No.4, where vehicle swaying due to road surface irregularities makes accurate road surface estimation difficult, the output parameters are set to shorten the output determination distance to verify the accuracy of alarm, preparatory control, and actual control through repeated testing. In addition, as with No.5, the danger level is low when driving at low speeds, even without control over the bumps, so the output parameters can be set to only issue an alarm.

[0076] In doing so, the environment recognition unit of Implementation 1 can detect bumps in the driving environment, and can output information for appropriate control even when the detection performance of the camera is reduced.

[0077] [Implementation Method 2]

[0078] The following is for reference Figure 5 Implementation 2 of the environmental identification unit involved in this disclosure will be described.

[0079] Figure 5 This is one embodiment of an in-vehicle system including the environmental identification unit 102 disclosed herein. The environmental identification unit 102 in this embodiment additionally includes a raised re-identification portion 501, which differs from the environmental identification unit 102 of Embodiment 1 described above. Other aspects of the environmental identification unit 102 in this embodiment are the same as those in Embodiment 1 described above; therefore, the same reference numerals are used for the same parts, and descriptions are omitted.

[0080] In the environment identification unit 102 of Embodiment 1, the bulge identification unit 204 outputs the bulge identification result and its accompanying information. However, in this embodiment, the bulge identification unit 204 outputs the unidentified result and accompanying information for each candidate region output by the candidate region extraction unit 241, even if the bulge is not identified as a bulge. The bulge re-identification unit 501 has the function of taking the unidentified result and accompanying information as input, changing various thresholds according to the control category, and performing bulge determination again, and outputting the result.

[0081] The bump re-identification unit 501 determines the bump based on the control type indicating the control state of the vehicle 100 and the distance change determination threshold (re-determination setting unit) between the vehicle 100 and the bump candidate area. Furthermore, if there is a bump candidate area that was not identified as a bump up to the previous processing cycle, the bump is re-identified based on the changed determination threshold.

[0082] exist Figure 6 The diagram illustrates the processing flow of the bump re-identification unit 501. First, in S101, one or more control types are set, including alarm, pre-operation control, and deceleration control. This setting can use the control type included in the identification request sent from the control unit 103, or it can use the control type mentioned above. Figure 7 The output determination table shown specifies the control categories for determining the output distance.

[0083] Next, in S102, it is determined whether the parallax accuracy is at least a certain level relative to the distance of the candidate region output by the candidate region extraction unit 241. This parallax accuracy can be calculated using at least one of the following: the baseline length of the stereo camera, the resolution of each camera, the exposure time, aging or temperature changes, and the calibration status. For example, when a point P corresponding to a point Q on the left camera image is captured on the right camera image, the coordinate difference between point P and point Q changes with distance. The distance that changes when this coordinate difference shifts by one unit is called the parallax resolution. Here, one unit refers to the exploration width during correlation exploration. If the exploration is performed using integer coordinates, it is one pixel; if the exploration is performed using sub-pixel coordinates, it is one-quarter of a pixel or other sub-pixel coordinates. This can be calculated from the camera mounting status, the baseline length, and the resolution of each camera. Furthermore, in dark lighting environments, it is desirable to extend the exposure time to capture the subject brightly. However, in vehicle-mounted cameras, extending the exposure time can easily cause motion blur, thus reducing parallax accuracy. Furthermore, the stereo camera's casing can sometimes become distorted due to years of degradation or temperature changes, contributing to a decrease in parallax accuracy. Additionally, while the camera is calibrated using large-scale equipment at the factory, resulting in excellent calibration and high accuracy, if it is replaced or reconfigured at a dealership or similar location for some reason, the calibration may become poor if simpler equipment is used, potentially leading to a decrease in parallax accuracy. Therefore, parallax accuracy can be calculated by multiplying the aforementioned parallax resolution by a coefficient of 1 or more, determined based on exposure time, years of degradation or temperature changes, and variations in calibration status. This coefficient can be recorded as a table during the design phase, for example. In cases where the parallax accuracy at the calculated distance to the candidate area is lower than the height of the desired protrusion (e.g., a parallax accuracy of 10cm and a desired protrusion of 5cm), the protrusion cannot be detected correctly. In such cases, in step S102, it is determined that the parallax accuracy is not necessarily higher (= no). Conversely, if the value of the disparity accuracy at the calculated distance of the candidate region is smaller than the height of the ridge to be detected, it is determined in S102 that the disparity accuracy is greater than or equal to 'yes', and proceeds to S103.

[0084] In S103, a determination is made based on the control type and TTC (Time To Collision). If the control type set in S101 is deceleration control, and the TTC calculated based on the vehicle speed and the distance of the candidate area output by the candidate area extraction unit 241 is below the preset TTC threshold for deceleration control (=Yes), then proceed to S104; otherwise (=No), proceed to S105.

[0085] In S105, the determination is made based on the control type and TTC, similar to S103. If the control type set in S101 is preparatory control, and the TTC calculated based on the vehicle speed and the distance of the candidate region output by the candidate region extraction unit 241 is below the preset TTC threshold for preparatory control (=Yes), then proceed to S106; otherwise (=No), proceed to S107.

[0086] In S107, the determination is made based on the control type and TTC, similar to S103. If the control type set in S101 is alarm, and the TTC calculated based on the vehicle speed and the distance of the candidate area output by the candidate area extraction unit 241 is below the preset TTC threshold for alarm (=Yes), then proceed to S108; otherwise (=No), the result of the bulge re-identification is output as no recognition.

[0087] In S104, the determination threshold is set to a determination threshold A that makes it easier to recognize than the bulge recognition unit 204, and the process proceeds to S109. In other words, the determination threshold is set to a determination threshold A that is lower than the determination threshold of the bulge recognition unit 204, and the process proceeds to S109.

[0088] In S106, a determination threshold B is set to make it easier to identify than determination threshold A, and the process proceeds to S109. In other words, the determination threshold is set to a determination threshold B that is lower than the determination threshold of the bulge recognition unit 204, and the process proceeds to S109.

[0089] In S108, a determination threshold C is set to make it easier to identify than the determination threshold B, and the process proceeds to S109. In other words, the determination threshold is set to a determination threshold C that is lower than the determination threshold of the bulge recognition unit 204, and the process proceeds to S109.

[0090] In S109, the shape of the bulge is determined using a pre-set determination threshold. Since this process is the same as that of the bulge recognition unit 204, details are omitted.

[0091] By doing so, the environment recognition unit 102 of this embodiment 2 can standby output when the detection accuracy of road surface bumps is low. Since the determination is made only after the resolution of the stereo camera has become sufficient to be used for bump detection, it has the effect of suppressing false detection.

[0092] Furthermore, in the above embodiments, an example was described where three determination processes, S103, S105, and S107, were set, and determination thresholds, S104, S106, and S108, were set to facilitate identification when the TTC threshold of each determination process was below it. However, this is not a limitation. For example, only processes S103 and S104, or only processes S103 to S106, may be performed.

[0093] [Implementation Method 3]

[0094] The following is for reference Figure 8 Implementation 3 of the environmental identification unit 102 involved in this disclosure will be described.

[0095] Figure 8 This is one embodiment of an in-vehicle system including the environmental recognition unit 102 disclosed herein. The environmental recognition unit 102 in this embodiment additionally includes an ambient vehicle observation unit 801 and a map processing unit 802, which differs from the environmental recognition unit 102 of Embodiment 2 described above. Other aspects of the environmental recognition unit 102 in this embodiment are the same as those in the environmental recognition unit 102 of the above embodiments; therefore, the same reference numerals are used for the same parts, and descriptions are omitted.

[0096] Since the bump extends to the left and right ends of the road, other vehicles (surrounding vehicles) traveling in the same direction as the vehicle in adjacent lanes often slow down when they approach the bump. In the environment recognition unit 102 of this embodiment 3, the bump can be easily detected even if a large number of bump features are not detected due to obstruction or other reasons by detecting the slowing down of other vehicles traveling in adjacent lanes.

[0097] In the environment recognition unit 102 of Embodiment 1, the bump recognition unit 204 outputs the bump recognition result and its accompanying information. However, in this Embodiment 3, the bump recognition unit 204 can infer the existence of the bump by observing the behavior of other vehicles even if the bump itself is not fully observed by the stereo camera, making it easy to recognize the bump. This is different.

[0098] The surrounding vehicle observation unit 801 observes the behavior of preceding vehicles traveling in the same lane and adjacent lanes as the vehicle. Based on the region map resulting from the region segmentation output from the driving area determination unit 203, the surrounding vehicle observation unit 801 extracts vehicle areas such as passenger cars or trucks, as well as painted areas such as white lines, and uses the extraction results in a time series to track the movement of each vehicle area. Furthermore, based on camera setting information and parallax information corresponding to the extracted vehicle areas, it calculates their relative positions to the vehicle.

[0099] The map processing unit 802 calculates the behavior of other vehicles on the map based on information about other vehicles observed by the surrounding vehicle observation unit 801, as well as the vehicle's speed and steering angle. Specifically, for lanes inferred from white lines or other markings, it calculates the vehicle's speed changes and other driving trajectories and behaviors on the map. The map processing unit 802 also selects candidate locations for ridges inferred from ridge features and tracks them on the map.

[0100] In addition to the information described in Embodiment 1 or Embodiment 2, the bump recognition unit 204 also uses the driving trajectories and behaviors of other vehicles calculated by the map processing unit 802 as input. The bump recognition unit 204 tracks the behavior of other vehicles on the map obtained by the surrounding vehicle observation unit 801, and, when pre-set conditions are met, lowers the determination threshold to make it easier to identify a bump. Specifically, it is characterized in that, when other vehicles are traveling along the lane and decelerating, since there may be a bump ahead rather than a lane change, the width direction feature output by the width direction feature calculation unit 302 is used to lower the threshold set by the bump candidate region estimation unit 305 when estimating the bump candidate region, making it easier to identify. The bump recognition unit 204 lowers the determination threshold to make it easier to identify a bump when, for example, at least one of the following conditions is met: a vehicle in an adjacent lane brakes, the inter-vehicle distance decreases, or the taillight illumination changes.

[0101] Therefore, the environmental recognition unit of this embodiment 3 has the following effect: even if the stereo camera fails to fully observe the bulge itself, the presence of the bulge can be inferred by observing the behavior of other vehicles, making the bulge easier to identify.

[0102] [Implementation Method 4]

[0103] The following is for reference Figure 10 and Figure 11 Implementation 4 of the environmental identification unit 102 involved in this disclosure will be described.

[0104] Figure 10 This is a figure illustrating embodiment 4 of an in-vehicle system including the environmental recognition unit disclosed herein. Figure 11 This is a diagram illustrating the configuration of the environment recognition unit in Implementation Method 4.

[0105] The environmental identification unit 102 of this embodiment 4 additionally includes an environmental recording device 1001 and a communication device 1002, which differs from the environmental identification unit 102 of embodiments 1 or 2 described above. Other aspects of the environmental identification unit 102 of this embodiment 4 are the same as those of the environmental identification unit 102 of embodiments 1 or 2 described above; therefore, the same reference numerals are used for the same parts and descriptions are omitted.

[0106] The environment recognition unit 102 in Embodiment 1 is a device that operates independently of the vehicle and identifies the ever-changing environment around the vehicle based on external information obtained from a camera mounted on the vehicle. However, the environment recognition unit 102 in Embodiment 4 differs in that it keeps recording and communicates with the outside world when needed.

[0107] The environmental recording device 1001 can record the recognition result of the environmental recognition unit 102 and the vehicle position output by the locator device (not shown) as a set. In addition, it can read the recognition result and recorded position that exist near the input vehicle position from the recorded content using the vehicle position as a query command.

[0108] Furthermore, the communication device 1002 can communicate with one or more other vehicles, road equipment, or map servers to upload the content recorded by the environmental recording device 1001. The communication device 1002 is equivalent to a communication unit. Conversely, it can also communicate with one or more other vehicles, road equipment, or map servers to receive identification results such as the location and height of bulges observed outside the vehicle. The environmental identification unit 102 adjusts parameters based on the received information to easily identify the bulge or overwrites the identification results.

[0109] Next, the method for detecting bumps based on the image captured by the camera unit 101 will be described.

[0110] Figure 11 This is a block diagram showing the internal configuration of the camera unit 101 and the environment recognition unit 102. The environment recognition unit 102 in this embodiment is additionally equipped with a self-position estimation unit 1101, an environment recording unit 1102, and a communication device 1103, which is different from the environment recognition unit 102 in embodiments 1 to 3 described above.

[0111] The self-position estimation unit 1101 has the function of estimating the vehicle's position by receiving latitude, longitude, and altitude from a GNSS receiver, and outputting the result to the environment recording unit 1102. Alternatively, it may use a camera or LiDAR to identify the environment around the vehicle and compare it with an internal map (not shown) to estimate the vehicle's position.

[0112] The environmental recording unit 1102 takes the distance to the bulge, height, and recognition reliability calculated by the bulge recognition unit 204 as described in Embodiment 1, and the self-position of the self-position estimation unit 1101 as input, and records the set of bulge information in a hard disk (HDD), solid-state drive (SSD), or memory. Furthermore, the environmental recording unit 1102 reads the recorded information, and when the self-position of the self-position estimation unit 1101 is input, outputs to the bulge recognition unit 204 at least one of the following: position, height, and recognition reliability of a bulge within a threshold range of the input self-position.

[0113] In the bulge recognition unit 204, if the location and height of the bulge are obtained from the environment recording unit 1102 and the recognition reliability is greater than a predetermined threshold, the location, height, and recognition reliability of the corresponding bulge can be replaced with the information obtained from the environment recording unit 1102. By doing so, instead of performing several detections from a distance to improve the detection reliability index, bulges can be detected from a greater distance with higher detection reliability.

[0114] The communication device 1103 can communicate with one or more other vehicles, road equipment, or map servers regarding the contents of the environmental recording unit 1102, and upload or download bulge information indicating the location and height of the bulge. By doing so, even for bulges that have not been detected by the vehicle before, such as those present on roads where the vehicle is driving for the first time, bulges can be detected from a greater distance with higher detection reliability, and bulges can be matched from a greater distance.

[0115] According to embodiments 1 to 4 of the present invention, a camera unit 101 installed on a vehicle 100 observes the surroundings. Based on the three-dimensional information obtained from the camera unit 101, it is determined whether the vehicle 100 will avoid a collision while traveling. Even if a collision is determined to be unlikely, the obtained three-dimensional road surface information is used to analyze whether there are bumps or unevenness in front of the vehicle 100 that would cause large vibrations to the vehicle 100. Based on the analysis results, the driver can be notified, or the suspension and brakes can be controlled to suppress vibrations of the vehicle 100. In other words, the camera unit 101 measures the shape of the road surface in front of the vehicle 100, and based on the results, for safer driving, the driver's attention is drawn before vibrations are transmitted to the vehicle 100; for more comfortable driving, the vehicle 100 can be controlled. For example, if it is predicted that vehicle 100 will enter a raised area at a speed higher than the appropriate speed range, the driver's attention is drawn by sound and display to slow down vehicle 100. Control is performed to suppress acceleration or decelerate vehicle 100 in front of the raised area. By softening the suspension of vehicle 100 in front of the height difference, semi-active control is implemented to suppress the impact when passing through the height difference, thereby mitigating the impact on the vehicle body.

[0116] Furthermore, according to embodiments 1 to 4 of the present invention, in order to detect factors of elevation difference present on the road surface (e.g., bumps, deceleration differences, cracks, potholes, etc.), when using parallax information obtained, for example, from a stereo camera, false detections and performance degradation caused by the inherent characteristics of the stereo camera can be suppressed. That is, due to the decreased resolution of parallax information at a distance compared to nearby objects, when attempting to determine detection at a distance, the use of low-resolution parallax information may prevent the elimination of false detections. Additionally, if detection is delayed until the parallax information becomes high-resolution, the detection distance may shorten when attempting to determine detection after approaching the target area. According to embodiments 1 to 4 of the present invention, by taking into account both false detection suppression and extended detection distance, the accuracy of the timing of attention arousal and control of detected bump areas is improved, reducing false alarms and miscontrols.

[0117] The embodiments of the environmental identification unit disclosed herein have been described in detail above using accompanying drawings. However, the specific construction is not limited to these embodiments, and design changes that do not depart from the spirit and scope of this disclosure are also included in this disclosure. For example, the foregoing embodiments have been described in detail for the purpose of easily understanding and explaining this disclosure, and are not necessarily limited to all the described configurations. In addition, a part of the configuration of a certain embodiment can be replaced by the configuration of other embodiments, and the configuration of a certain embodiment can be incorporated into the configuration of other embodiments. Furthermore, regarding a part of the configuration of each embodiment, other configurations can be added, deleted, or replaced.

[0118] Symbol Explanation

[0119] 100...Vehicle, 101...Camera Unit (Camera), 102...Environment Recognition Unit (Environment Recognition Device), 103...Control Unit, 201...Stereo Camera, 202...Raise Feature Generation Unit, 203...Driving Area Determination Unit, 204...Raise Recognition Unit, 205...Result Output Unit, 241...Candidate Region Extraction Unit, 242...Visible Area Determination Unit, 301...High Likelihood Region Extraction Unit, 302...Width Direction Feature Calculation Unit, 303...Depth Direction Feature Calculation Unit, 304...Contour Shape Feature Calculation Unit, 305...Raise Candidate Region Estimation Unit, 401...Raise Provisional Candidate Estimation Unit, 402...Distance / TTC Calculation Unit 403... Road width estimation unit, 404... Obscured area estimation unit, 405... Visible raised area estimation unit, 406... Repetition rate determination unit relative to the visible area, 501... Raised area re-identification unit, 801... Surrounding vehicle observation unit, 802... Map processing unit, 901... Road surface, 902... Speed ​​bump, 903... Leading vehicle, 904... Point group of raised areas, 905... Point group of flat road surface, 906... Area, 907... Area with strong reaction to raised areas, 908... Area corresponding to raised area, 1001... Environmental recording device, 1002... Communication device, 1101... Self-position estimation unit, 1102... Environmental recording unit.

Claims

1. An environment recognition device that recognizes a bump provided on a road surface on the basis of an image captured by a camera mounted on a vehicle, characterized by comprising: a bump recognition unit that recognizes the bump on the basis of a feature amount of the image captured by the camera. The environmental identification device includes: The driving area determination unit determines, in the image, the driving road area where the vehicle is traveling, as captured in the image; A bulge feature generation unit generates a feature map based on the image, the feature map representing the region containing the bulge feature along with the degree of accuracy; as well as The bulge recognition unit identifies the bulge based on bulge candidate regions in the feature map whose accuracy exceeds a threshold. The bulge identification unit identifies a bulge candidate region that meets at least one of the following criteria: its length relative to the width direction of the driving road area, its length in the depth direction of the bulge candidate region, or the complexity of the outline of the bulge candidate region, as the bulge candidate region.

2. The environmental identification device as described in claim 1, characterized in that, The driving area determination unit determines the area of ​​the object that exists on the driving road and is captured in the image. The bulge recognition unit identifies the bulge based on the visible area after excluding blind spots obscured by the object area from the driving road area.

3. The environmental identification device as described in claim 1, characterized in that, The system includes a bump re-identification unit. Based on the control category representing the control state of the vehicle and the distance from the vehicle to the bump candidate region, the bump re-identification unit changes the determination threshold. If there is a bump candidate region that was not identified as a bump up to the previous processing cycle, the system re-identifies the bump based on the changed determination threshold.

4. The environmental identification device as described in claim 1, characterized in that, It has a surrounding vehicle observation unit that observes the behavior of other vehicles in adjacent lanes. If at least one of the following conditions is met: other vehicles in the adjacent lane have braked, the inter-vehicle distance has decreased, or the taillight illumination has changed, the bump recognition unit lowers the determination threshold to make it easier to identify the bump.

5. The environmental identification device as described in claim 4, characterized in that, It includes a map processing unit that configures candidate locations of the ridges estimated based on the feature map on a map and tracks them. The bump recognition unit tracks the behavior of other vehicles on the map obtained by the surrounding vehicle observation unit, and lowers the determination threshold to easily identify the bump if the behavior meets the preset conditions.

6. The environment recognition apparatus according to claim 1, wherein have: The communication unit is capable of communicating with at least one of other vehicles, road equipment, or servers that are different from the vehicle described above; as well as The environmental recording unit records at least elevation information including the location of the elevation. The communication unit will record the bulge information in the environmental recording unit and transmit and receive it with at least one of the other vehicles, road equipment, or servers.

7. The environmental identification device as described in claim 6, characterized in that, It has a self-position estimation unit that estimates the position of the vehicle. The bump identification unit identifies the bump based on the vehicle's self-position estimated by the self-position estimation unit and the bump information received by the communication unit from at least one of the other vehicles, road equipment, or map server.

Images