Image recognition device and image recognition method

Abstract

To prevent erroneous detection of a distance from the own vehicle to a pole by detecting approach to the pole with a camera image when the own vehicle approaches and moves to the pole.SOLUTION: An image recognition device 100 includes: an image storage unit 101; a cylindrical distortion correction unit 102; a vertical edge extraction unit 103; a column candidate extraction unit 104; a pole candidate determination unit 105; a pole footing position setting unit 106; a movement distance acquisition unit 107; a detected distance difference calculation unit 108; and a pole recognition unit 109. When the own vehicle approaches and moves to a pole candidate P', the movement distance acquisition unit 107 acquires a movement distance VL at which the own vehicle moves in a prescribed time period. The detected distance difference calculation unit 108 calculates a detected distance difference ΔL between a prescribed time period starting time detected distance Ls and an ending time detected distance Le. The pole recognition unit 109 recognizes that the pole candidate P' having an absolute value of difference between the movement distance VL and the detected distance difference ΔL |VL-ΔL| being less than a threshold is a pole P having a position to contact the ground being a pole footing position FP.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image recognition device and an image recognition method. [Background technology]

[0002] Conventionally, a pole detection device and a pole detection method are known that can identify the positional relationship between a pole and a vehicle even for a pole whose corners cannot be detected. The pole detection device includes a vertical edge detection unit that detects vertical edges in one target image captured at a different time in a time series, a tracking point setting unit that sets multiple tracking points aligned vertically on the vertical edges, a corresponding point detection unit that detects corresponding points corresponding to the tracking points in the other target image, a vertical edge identification unit that identifies vertical edges in the target image where the corresponding points are aligned vertically, and a pole position detection unit that detects the position of the pole on the vertical edge based on the coordinate positions of corresponding intersections in the two target images and the position of the vehicle when each target image was captured (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-197826 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional devices, when detecting the position of a vertical edge pole, the position of the pole is detected by image processing based on camera images. Therefore, when a pole (hereinafter referred to as a "pole") erected on the ground is present around the vehicle, if the pole is reflected on the road surface in the direction extending from the base of the pole, the conventional device will recognize the pole as being a sum of the actual pole image and the pole image reflected on the road surface in the camera image. Therefore, when the vehicle approaches the pole, if the distance between the vehicle and the pole is detected based on the camera image, the detected distance will be shorter than the actual distance (true value) from the vehicle to the pole, and the conventional device will erroneously detect the distance from the vehicle to the pole.

[0005] The present invention has been made with an eye on the above-mentioned problems, and aims to detect the approach to the pole using camera images when the vehicle approaches the pole, thereby preventing erroneous detection of the distance from the vehicle to the pole. [Means for solving the problem]

[0006] To achieve the above object, the image recognition device of the present invention includes a surrounding situation acquisition unit, a vertical edge extraction unit, a pillar candidate extraction unit, a pole candidate determination unit, a pole base position setting unit, a movement distance acquisition unit, a detection distance difference calculation unit, and a pole recognition unit. The surrounding situation acquisition unit acquires surrounding situations that change as a camera-equipped vehicle moves using time-series camera image information. The vertical edge extraction unit extracts, as vertical edges, portions where pixel brightness changes by a predetermined value or more when scanning the camera image information horizontally. The pillar candidate extraction unit, when searching for a pair of a positive edge peak and a negative edge peak among the vertical edges, extracts the found edge pair as a pillar candidate. The pole candidate determination unit determines whether a pole determination condition is met for the pillar candidate, and determines a pillar candidate for which the pole determination condition is met as a pole candidate erected on the ground. The pole base position setting unit sets the lowest edge position in the vertical edge image representing the pole candidate as the pole base position. The movement distance acquisition unit acquires the movement distance traveled by the host vehicle during a predetermined time period when the host vehicle approaches a pole candidate. The detection distance difference calculation unit detects the horizontal distance from the camera position to the base of the pole during the predetermined time period when the host vehicle approaches a pole candidate, and calculates the difference in detection distance between the start and end of the predetermined time period. The pole recognition unit recognizes a pole candidate for which the absolute difference between the movement distance and the detection distance difference is less than a threshold value as a pole with the base of the pole at a position where the pole touches the ground. [Effects of the Invention]

[0007] In the present invention, when the vehicle approaches a pole candidate, the distance traveled by the vehicle and the distance approaching the pole candidate are observed in time series. Then, for recognition and judgment, a condition is used in which, if the pole candidate is a pole that is not reflected on the road surface, the distance traveled by the vehicle and the distance approaching the pole candidate are equal. As a result, when the vehicle approaches a pole, the approach to the pole is detected from the camera image, and erroneous detection of the distance from the vehicle to the pole can be prevented. [Brief explanation of the drawings]

[0008] [Figure 1]1 is a block diagram showing the overall configuration of a collision damage mitigation brake system to which an image recognition device according to a first embodiment is applied. [Figure 2] FIG. 2 is a diagram showing an example of a fisheye image captured by a fisheye camera and stored in an image storage unit of the image recognition device. [Figure 3] 10 is a diagram showing an example of a cylindrically-distorted corrected image obtained by correcting a fisheye image by projecting it onto a cylinder perpendicular to the road surface in a cylindrical distortion correction unit of the image recognition device. FIG. [Figure 4] FIG. 10 is a diagram showing an example of a cylindrical distortion-corrected image including a pole. [Figure 5] 5 is a diagram showing a vertical edge image extracted from a cylindrical distortion-corrected image (FIG. 4) in a vertical edge extraction unit of the image recognition device, and vertical edge integrated value characteristics used for determining extraction of a pillar candidate in a pillar candidate extraction unit. FIG. [Figure 6] 10 is an explanatory diagram showing edge amount conditions for determining a pole candidate in a pole candidate determining unit of an image recognition device. FIG. [Figure 7] 10 is an explanatory diagram showing edge width conditions for determining pole candidates in a pole candidate determination unit of an image recognition device. FIG. [Figure 8] 10 is an explanatory diagram showing edge direction conditions for determining pole candidates in a pole candidate determination unit of an image recognition device. FIG. [Figure 9] FIG. 10 is an explanatory diagram showing the foot position set by the pole foot position setting unit of the image recognition device when the entire pole is reflected by the water film surface in a puddle at the pole foot. [Figure 10] FIG. 10 is an explanatory diagram showing a determination method when the foot position is set to the bottom end position of the water film surface reflection image of the pole in the pole recognition unit of the image recognition device. [Figure 11] 3 is a flowchart showing the flow of image recognition processing executed by the image recognition device of the first embodiment. [Figure 12] 10 is a diagram showing the relationship between the actual distance from the camera position to the pole position and the detected distance when the entire pole is reflected by the water film surface in a puddle at the pole's foot. FIG. [Figure 13]This is a time-series distance graph showing the true value and the detected distance to the pole in time when the entire pole is reflected by the water film surface in a puddle at the pole's foot. [Figure 14] 10 is an explanatory diagram illustrating a situation in which the vehicle moves backward and approaches a pole. FIG. [Figure 15] FIG. 10 is an explanatory diagram illustrating a situation in which the vehicle is stopped just before the pole due to automatic braking when the vehicle approaches the pole while traveling backward. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment for carrying out an image recognition device and an image recognition method according to the present invention will be described based on a first embodiment shown in the drawings. [Example]

[0010] The image recognition device 100 of the first embodiment is a device applied to a collision damage mitigation brake system A that avoids collision with an obstacle by automatically activating the brakes when the vehicle approaches an obstacle while traveling forward or backward.

[0011] First, a general configuration of the collision mitigation braking system A will be described with reference to Fig. 1. Here, the collision mitigation braking system is called "AEB (Abbreviation for Autonomous Emergency Braking)."

[0012] The collision damage mitigation braking system A comprises an image recognition device 100, a fisheye camera 200 (camera), a display device 300, an automatic brake controller 400, input sensors 500, a brake actuator 600, and an alarm 700, all of which are installed in the vehicle.

[0013] The image recognition device 100 processes the camera image information from the fisheye camera 200 and displays an image of the surroundings of the vehicle on the display screen 301 of the display device 300 as necessary. The image recognition device 100 is provided with an obstacle recognition function that outputs approach distance information to obstacles (including stationary and moving objects) around the vehicle recognized from the camera image to the automatic brake controller 400. Furthermore, a pole recognition function is provided that recognizes cylindrical poles P, which tend to be difficult to detect with a single sonar 504, and outputs approach distance information to the poles P to the automatic brake controller 400. The detailed configuration for realizing the pole recognition function in the image recognition device 100 will be described later.

[0014] The fisheye camera 200 is a camera using a fisheye lens with a short focal length that can capture a wide angle of view of approximately 180 degrees or more while distorting the subject. The fisheye camera 200 of the first embodiment serves, for example, as a front camera or a rear camera among four cameras installed on the front, rear, left, and right sides of the vehicle used in an omnidirectional monitoring system. The omnidirectional monitoring system is a system that supports smooth parking by allowing the driver to check at a glance the relationship between the vehicle and the parking position during parking, such as garage parking or parallel parking, using viewpoint-converted video displayed on the display screen 301 of the display device 300. The viewpoint-converted video refers to video from a bird's-eye view, as if looking down on the vehicle from directly above. The video signals obtained from the four cameras are viewpoint-converted in real time. The omnidirectional monitoring system then combines the four viewpoint-converted camera images into a single image centered on the vehicle.

[0015] The display device 300 has a display screen 301 that displays the camera image after image processing. For example, when the vehicle is traveling backward at a low speed, the camera image from the fish-eye camera 200 after image processing is displayed based on a display command from the image recognition device 100.

[0016] When the host vehicle approaches an obstacle, the automatic brake controller 400 performs brake control based on approach distance information from the obstacle to automatically activate the brakes of the host vehicle before the host vehicle comes into contact with the obstacle. The automatic brake controller 400 inputs the pole recognition result from the pole recognition unit 109 of the image recognition device 100. If the recognition result indicates that the pole P is from the ground, the automatic brake controller 400 uses the horizontal distance (=detection distance L') from the pole P to the pole foot position F calculated based on the camera image information as approach distance information. On the other hand, if the recognition result indicates that the pole P is not from the ground, the automatic brake controller 400 uses the distance detection value obtained by the sonar 504 (obstacle distance sensor) separate from the fisheye camera 200 as approach distance information. In other words, the automatic brake controller 400 acquires approach distance information through sensor fusion of the fisheye camera 200 and the sonar 504, and functions as a controller in which the fisheye camera 200 and the sonar 504 compensate for each other's weaknesses.

[0017] The input sensors 500 are a plurality of sensors and switches that provide necessary information to the automatic brake controller 400 and the image recognition device 100. The input sensors 500 include a range position sensor 501, a steering angle sensor 502, a vehicle speed sensor 503, and a sonar 504 (obstacle distance sensor).

[0018] The range position sensor 501 is a sensor that detects the range position selected in the on-board automatic transmission, and acquires information on the traveling direction of the vehicle (forward travel, backward travel, etc.). The steering wheel angle sensor 502 is a sensor that detects the steering wheel angle caused by the driver's steering of the steering wheel, and acquires steering wheel angle information (straight travel, turning travel, etc.).

[0019] The vehicle speed sensor 503 is a sensor that detects the vehicle speed of the host vehicle. When a predetermined time is set, the travel distance of the host vehicle in the predetermined time can be calculated by using the average vehicle speed and the predetermined time.

[0020] Sonar 504 is an example of an obstacle distance sensor that uses ultrasonic waves to detect the distance to an obstacle by utilizing a signal reflected from the obstacle. For example, when a front center sonar, a rear center sonar, and corner sonars at the four corners are installed to observe the surroundings of the vehicle, sonar 504 in the first embodiment serves as both the front center sonar and the rear center sonar. Note that although an example is shown in which sonar 504 is used as an obstacle distance sensor, other obstacle distance sensors such as laser radar, millimeter wave radar, and laser range finder can also be used as the obstacle distance sensor instead of sonar 504.

[0021] Brake actuator 600 is an actuator that automatically activates the brakes of the vehicle based on a brake activation command from automatic brake controller 400. Alarm 700 issues an alarm to the driver and passengers based on an alarm activation command from automatic brake controller 400. Automatic brake controller 400 uses approach distance information and time to collision TTC (abbreviation for "Time-To-Collision") to perform automatic brake control by combining an alarm and brake activation.

[0022] Next, a detailed configuration for realizing the pole recognition function in the image recognition device 100 will be described with reference to FIGS.

[0023] 1, the image recognition device 100 includes an image storage unit 101 (surrounding condition acquisition unit), a cylindrical distortion correction unit 102 (surrounding condition acquisition unit), a vertical edge extraction unit 103, a pillar candidate extraction unit 104, a pole candidate determination unit 105, and a pole base position setting unit 106. In addition, the image recognition device 100 includes a movement distance acquisition unit 107, a detection distance difference calculation unit 108, a pole recognition unit 109, and a distance conversion map setting unit 110. Each component will be described below.

[0024] The image storage unit 101 stores fisheye images FI acquired from the fisheye camera 200 in chronological order. When a pair of poles P, P are present on the left and right sides of the screen in a fisheye image FI, as shown in FIG. 2, the orientations of the pair of poles P, P in the image become misaligned due to the influence of lens distortion. The pole P on the left side has a shape that rises diagonally upward to the left from the foot position. The pole P on the right side has a shape that rises diagonally upward to the right from the foot position.

[0025] The cylindrical distortion correction unit 102 converts the fisheye image FI stored in the image storage unit 101 into a cylindrical distortion-corrected image CI. Here, as shown in the upper part of FIG. 3, the cylindrical distortion-corrected image CI can be converted into a distortion-corrected image that does not produce distortion in the vertical direction by performing image conversion to project the image onto a cylindrical screen CS perpendicular to the road surface. As shown in the lower part of FIG. 3, the pair of poles P, P in the cylindrical distortion-corrected image CI both have a shape that rises straight up vertically from the foot position. In the first embodiment, the image storage unit 101 and the cylindrical distortion correction unit 102 constitute a surrounding situation acquisition unit that acquires, from time-series camera image information, surrounding situations that change as the vehicle equipped with the fisheye camera 200 moves (see FIG. 1).

[0026] The vertical edge extraction unit 103 extracts, as vertical edges VE, portions where pixel brightness changes by a predetermined value or more when scanning the cylindrical distortion-corrected image CI (camera image information) multiple times at equal intervals in the horizontal direction. Here, vertical edges VE include both portions where pixel brightness changes from high to low by a predetermined value or more (hereinafter referred to as "positive edges") and portions where pixel brightness changes from low to high by a predetermined value or more (hereinafter referred to as "negative edges"). For example, assume that the cylindrical distortion-corrected image CI is an image with a single pole P in the center, as shown in FIG. 4. In this case, the vertical edge image obtained by extracting the vertical edges VE has a pair of positive edges +VE and negative edges -VE along both side edges of the pole P in the vertical direction, as shown in the upper part of FIG. 5.

[0027] When the pillar candidate extraction unit 104 finds a pair of a peak of the positive edge +VE and a peak of the negative edge -VE among the vertical edges VE, it extracts the found edge pair as a pillar candidate. Here, the pillar candidate extraction unit 104 extracts a pair that satisfies both an integrated value condition and a horizontal distance condition as a pillar candidate. The integrated value condition, as shown in the lower part of FIG. 5, is a condition where the vertical integrated value Σ+VE of the positive edge +VE is equal to or greater than the positive edge threshold +VEth, and the vertical integrated value Σ-VE of the negative edge -VE is equal to or less than the negative edge threshold -VEth. The horizontal distance condition, as shown in the lower part of FIG. 5, is a condition where the horizontal distance D between the peak position of the positive edge +VE and the peak position of the negative edge -VE is equal to or less than the distance threshold Dth. The distance threshold Dth is set to a value that includes cylindrical obstacles erected on the road surface that have a diameter smaller than that of a utility pole and are used, for example, to define no-entry areas.

[0028] The pole candidate determination unit 105 determines whether or not the pole determination conditions are met for the pillar candidates extracted by the pillar candidate extraction unit 104, and determines the pillar candidates for which the pole determination conditions are met as pole candidates P' erected on the ground. Here, the pole candidate determination unit 105 determines the edge amount condition, edge width condition, and edge direction condition as pole determination conditions in the image area downward from near the horizon in the cylindrical distortion-corrected image CI (camera image information). Then, the pole candidate determination unit 105 determines the pillar candidate for which all the pole determination conditions are met as pole candidate P' erected on the ground.

[0029] An image region precondition is given to define the image region for determining the pole candidate P'. The image region precondition is set by drawing a horizontal line HL near the horizon in the cylindrical distortion-corrected image CI based on the camera image, and defining the image region downward from the horizontal line HL, as shown in Figure 6. Therefore, pillar candidates that are located higher than the horizon and are not erected on the ground are excluded because the base of the pole is not on the ground and the image region precondition is not met.

[0030] The edge amount condition is satisfied when the edge amounts at the right and left ends of a pillar candidate are equal to or greater than a threshold value. Here, the "threshold value" is set to a value that satisfies the condition when vertically consecutive left and right edges are extracted from the plurality of equally spaced scanning lines SL that measure brightness changes, as indicated by the right arrows in Figure 6. Therefore, pillar candidates in which a pair of left and right vertical edges do not continue downward from the horizon line are excluded because the edge amount condition does not hold.

[0031] The edge width condition is determined to be true if the variation in the width of a pillar candidate is less than a predetermined variation range. Here, the "predetermined variation range" is set to a pixel-unit variation tolerance value that is determined to be true when a pair of vertical edge widths (the horizontal distance between the peak position of the positive edge and the peak position of the negative edge) can be considered to be approximately parallel. As shown in FIG. 7, the pole shadow PS, which extends directly from the bottom end of the pole P, has a pair of vertical edge widths that increase in the downward direction. Therefore, a pillar candidate formed by a pole that reflects the pole shadow PS is excluded because this edge width condition is not true. Furthermore, a pillar candidate that reflects road paint, such as a white line WL painted on the road surface, whose pair of vertical edge widths increase in the downward direction, can also be excluded because this edge width condition is not true.

[0032] The edge direction condition is satisfied when all edge peaks of a pillar candidate are in the vertical direction. Here, "vertical direction" refers to the direction that is satisfied when the fisheye image FI is converted into a cylindrical distortion-corrected image CI, allowing for conversion errors. Road paint such as white lines WL on the road surface appears diagonally in the cylindrical distortion-corrected image CI, as shown in Figure 8. Therefore, pillar candidates made of road paint are excluded because this edge direction condition does not hold. Note that pillar candidates in which a pole shadow PS is reflected, bending and extending from the bottom end of a pole P, can also be excluded because this edge direction condition does not hold.

[0033] The pole base position setting unit 106 sets the lowest edge position in the vertical edge image representing the pole candidate P' for which the pole determination condition is met as the pole base position FP. For example, as shown in FIG. 9, if a puddle spreads at the base of the pole P and the pole P is reflected in the camera image, extending from the base of the pole P in the longitudinal direction of the pole P, the camera image will recognize a pole with a length equal to the sum of the actual pole image PI and the pole image PR reflected on the road surface. In other words, the pole with a length equal to the sum of the actual pole image PI and the reflected pole image PR is determined to be the pole candidate P', and the position of the lowest point of the reflected pole image PR (the tip position of the pole P due to specular reflection) is set as the pole base position FP, as shown in FIG.

[0034] When the host vehicle moves toward the pole candidate P', the travel distance acquisition unit 107 acquires the travel distance VL traveled by the host vehicle during a predetermined time period (see FIG. 10). The travel distance acquisition unit 107 of the first embodiment reads the start vehicle speed from the vehicle speed sensor 503 at the start time of the predetermined time period, and reads the end vehicle speed from the vehicle speed sensor 503 at the end time of the predetermined time period. Then, the travel distance acquisition unit 107 calculates the average vehicle speed of the start vehicle speed and the end vehicle speed, and multiplies the average vehicle speed by the predetermined time period to acquire the travel distance VL traveled by the host vehicle during the predetermined time period. Note that if the position information of the host vehicle can be acquired with high accuracy using a GPS or the like, the travel distance VL traveled by the host vehicle may be acquired by other methods, such as calculating the travel distance from the host vehicle positions at the start and end of the predetermined time period.

[0035] When the vehicle approaches the pole candidate P', the detection distance difference calculation unit 108 detects the horizontal distance from the camera position to the pole base position FP, and calculates the detection distance difference ΔL between the start detection distance Ls and the end detection distance Le for a predetermined time period (see FIG. 10).

[0036] In the first embodiment, the start detection distance Ls and the end detection distance Le are calculated using a display device 300 having a display screen 301 that displays time-series camera image information, and a distance conversion map setting unit 110. The distance conversion map setting unit 110 sets a distance conversion map (such as a distance conversion formula or distance conversion table) that converts the vertical coordinate position of the pole foot position FP displayed on the display screen 301 into the horizontal distance from the installation position of the fisheye camera 200 to the pole foot position FP. Therefore, when the vertical coordinate position of the pole foot position FP on the display screen 301 is detected at the start of the predetermined time, the start detection distance Ls is calculated using the distance conversion map. When the vertical coordinate position of the pole foot position FP on the display screen 301 is detected at the end of the predetermined time, the end detection distance Le is calculated using the distance conversion map.

[0037] The pole recognition unit 109 recognizes a pole candidate P' for which the absolute difference |VL-ΔL| between the moving distance VL and the detected distance difference ΔL (=Ls-Le) is less than a threshold value as a pole P whose position in contact with the ground is the pole foot position FP. Here, the "threshold value" is set to a small value obtained by adding a predetermined margin to the calculation error of the moving distance VL and the detected distance difference ΔL, because in the case of a pole P whose position in contact with the ground is the pole foot position FP, the moving distance VL and the detected distance difference ΔL theoretically match and the difference becomes zero.

[0038] Next, with reference to the flowchart shown in FIG. 11, the image recognition processing operation executed by the image recognition device 100 to determine whether or not the pole candidate P' is the pole P having the pole foot position FP as the position where it contacts the ground will be described.

[0039] In step S1, when the vehicle equipped with the fisheye camera 200 moves, time-series fisheye images FI of the surrounding situation that changes as the vehicle moves are acquired.

[0040] In step S2, following acquisition of fisheye images FI in step S1, the fisheye images FI acquired in time series are converted into cylindrical distortion-corrected images CI (camera image information).

[0041] In step S3, following the image conversion to the cylindrical distortion-corrected image CI in step S2, when the cylindrical distortion-corrected image CI is scanned in the horizontal direction, a portion where the pixel brightness changes by more than a predetermined value is extracted as a vertical edge VE.

[0042] In step S4, following the extraction of vertical edges VE in step S3, pairs of peak positions of positive edges +VE and negative edges -VE among the vertical edges VE are searched for, and it is determined whether or not a pillar candidate has been extracted by the edge pair search. If the result is YES because a pillar candidate for which both the edge pair integrated value condition and the horizontal distance condition are met has been extracted, the process proceeds to step S5. On the other hand, if the result is NO because neither the edge pair integrated value condition nor the horizontal distance condition is met and no pillar candidate has been extracted, the process proceeds to RETURN.

[0043] In step S5, following the determination in step S4 that a pillar candidate has been extracted, it is determined whether the pillar candidate is a pole candidate P' by determining whether the pole determination conditions are met for the pillar candidate. If all of the pole determination conditions are met and the pillar candidate is a pole candidate P' erected on the ground, the result is YES, and the process proceeds to step S6. On the other hand, if none of the pole determination conditions are met and the pillar candidate is not a pole candidate P' erected on the ground, the result is NO, and the process proceeds to RETURN.

[0044] In step S6, following the determination in step S5 that the pillar candidate is pole candidate P', the lowest edge position in the vertical edge image representing pole candidate P' is set as pole base position FP. In this step S6, when the vehicle approaches and moves toward pole candidate P', the pole base position FP, which changes over time for the vertical edge images acquired in time series over a predetermined time, is set in time series.

[0045] In step S7, following the setting of the pole foot position FP in step S6, it is determined whether or not start information and end information for the predetermined time have been acquired. Here, start information for the predetermined time refers to information on the start vehicle speed Vs and start detection distance Ls. End information for the predetermined time refers to information on the end vehicle speed Ve and end detection distance Le. Note that vehicle speed information and detection distance information that change over time from the start to the end of the predetermined time may also be acquired in chronological order. If the start information and end information for the predetermined time have been acquired and the answer is YES, proceed to step S8. On the other hand, if the start information and end information for the predetermined time have not been acquired and the answer is NO, proceed to RETURN.

[0046] In step S8, following the determination in step S7 that the start information and end information for the predetermined time period have been acquired, the movement distance VL is acquired and the detection distance difference ΔL is calculated. Here, the movement distance VL refers to the distance traveled by the host vehicle during the predetermined time period when the host vehicle approaches the pole candidate P'. The detection distance difference ΔL refers to the difference between the start detection distance Ls and the end detection distance Le of the predetermined time period, which is calculated by detecting the horizontal distance from the camera position to the pole base position FP when the host vehicle approaches the pole candidate P'.

[0047] In step S9, following acquisition of the travel distance VL and calculation of the detected distance difference ΔL in step S8, it is determined whether the absolute difference (|VL - ΔL|) between the travel distance VL and the detected distance difference ΔL is less than a threshold. If the determination is YES because |VL - ΔL| is less than the threshold, the process proceeds to step S10 based on the recognition that the pole candidate P' is a pole P whose position where it contacts the ground is the pole foot position FP. On the other hand, if the determination is NO because |VL - ΔL| is greater than or equal to the threshold, the process proceeds to step S11 based on the recognition that the pole candidate P' is not a pole P whose position where it contacts the ground is the pole foot position FP.

[0048] In step S10, following the determination of YES in step S9, the automatic brake controller 400 is output with a recognition result that the pole P is in contact with the ground and the pole foot position FP is the pole P, and the output of approach distance information using the fisheye camera 200 is continued. From step S10, the process proceeds to RETURN. Therefore, the automatic brake controller 400 executes automatic brake control using the horizontal distance from the pole P to the pole foot position FP calculated based on the image information from the fisheye camera 200 as approach distance information.

[0049] In step S11, following the determination of NO in step S9, the system outputs to the automatic brake controller 400 the recognition result that the pole P is not the pole foot position FP, where the position is in contact with the ground, and stops outputting the approach distance information using the fisheye camera 200. From step S11, the system proceeds to return. Therefore, the automatic brake controller 400 executes automatic brake control using the distance detection value obtained by the sonar 504, which is separate from the fisheye camera 200, as approach distance information.

[0050] Next, referring to Figure 12, we will explain why the distance from the vehicle to pole P is incorrectly detected when the camera image of pole P is an image that combines the actual pole image PI with a pole image PR reflected on the road surface due to a puddle.

[0051] When detecting the distance from the vehicle to the pole P using a camera image, first, the pole foot position FP on the display screen is identified. Then, based on the pole foot position FP on the display screen, the distance from the vehicle to the pole P is detected using a distance conversion formula or the like that converts the ordinate position of the pole foot position FP on the display screen into the distance from the vehicle to the pole P. In other words, when detecting the distance from the vehicle to the pole P, the detected distance is acquired based on the pole foot position FP.

[0052] In contrast, if the entire pole is reflected by the water surface in a puddle at the pole's base, the camera image will recognize the pole as having a length equal to the sum of the actual pole image PI and the pole image PR reflected on the road surface, as shown in Figure 12.

[0053] The reason for this is that when a pole image due to water film surface reflection is reflected on the road surface in an actual pole image, a pair of vertical edges of the pole image due to water film surface reflection maintains parallelism from the foot position of the pole downward.Furthermore, when a pole image due to water film surface reflection is reflected on the road surface in an actual pole image, the actual pole image PI and the pole image PR reflected on the road surface due to water film surface reflection are connected in a straight line, regardless of whether the vehicle is approaching straight or turning (see Figure 9).

[0054] Therefore, as shown in Figure 12, when it is recognized that there is a pole with a length equal to the sum of the actual pole image PI and the pole image PR reflected on the road surface, the pole foot position FP is the lowest end position of the pole image PR reflected on the road surface. Therefore, since the two dotted triangles depicted in Figure 12 are similar in shape, the detected distance L' to the pole foot position FP is L'=L×Hc / (Hc+Hp) where L is the actual distance (true value), Hc is the height from the ground to the fish-eye camera 200, and Hp is the actual height of the pole P (= the height of the pole reflected on the road surface). As is clear from the above formula, if it is assumed that the height Hc from the ground to the fish-eye camera 200 and the actual height Hp of the pole P are equal, then the detection distance L' will be half the actual distance L (true value).

[0055] In this way, the detected distance L' is the distance to the intersection point between the line connecting the fisheye camera 200 and the lowest point of the pole image PR reflected on the road surface and the ground, so the distance from the vehicle to the actual pole P is erroneously detected as shorter than the true value (actual distance L).

[0056] Next, with reference to FIG. 13, the reason why it is possible to determine whether the camera image includes not only an actual pole image PI but also a pole image PR reflected on the road surface due to a puddle will be described.

[0057] When the vehicle approaches pole P, the relationship between the time until the vehicle interferes with pole P and the distance to pole P, and the relationship between the true value L and the detection distance L', is as shown in Figure 13. The true value L, which is based on a camera image that shows only the actual pole, decreases at a larger downward gradient angle over time than the detection distance L', as shown by the solid line characteristics in Figure 13. In contrast, when the camera image shows only the actual pole and an image of the pole reflected in the road surface is added, the detection distance L' decreases at a smaller downward gradient angle over time than the true value L, as shown by the dotted line characteristics in Figure 13.

[0058] Therefore, when a pole image PR reflected on the road surface is added, the distance change width of the true value L between the start time t1 and the end time t2 of the predetermined time becomes the travel distance VL of the vehicle. The difference between the detected distance L' at the start time t1 and the detected distance L' at the end time t2 of the predetermined time becomes the detected distance difference ΔL. Therefore, the relationship between the travel distance VL of the vehicle and the detected distance difference ΔL is VL > ΔL.

[0059] On the other hand, when the camera image only contains the actual pole image PI, the pole P is stationary, so the amount of change in the pole detection position (=detection distance difference ΔL) and the amount of change in the host vehicle position (=vehicle movement distance VL) are equal. Therefore, when the camera image only contains the actual pole image PI, the relationship between the host vehicle movement distance VL and the detection distance difference ΔL is VL = ΔL.

[0060] Therefore, the travel distance VL of the vehicle and the detected distance difference ΔL are observed in time series, and a condition (|VL - ΔL|<threshold) is set under which the detected distance difference ΔL and the travel distance VL of the vehicle can be considered equal. Then, by excluding results that deviate from this condition and retaining results that satisfy this condition, it is possible to determine whether the pole candidate recognized from the camera image is an image in which the pole image PR reflected on the road surface is added to the actual pole image PI.

[0061] Next, an example of automatic braking action when the vehicle is traveling backward will be described with reference to FIGS.

[0062] First, when a recognition result is obtained that the pole P has a pole foot position FP at its contact point with the ground, the process proceeds in the order of S1 → S2 → S3 → S4 → S5 → S6 → S7 → S8 → S9 → S10 in the flowchart shown in Fig. 11. Then, in step S10, the recognition result that the pole P has a pole foot position FP at its contact point with the ground is output to the automatic brake controller 400, and approach distance information using the fisheye camera 200 is also output. Therefore, the automatic brake controller 400 executes automatic brake control using the horizontal distance from the pole P to the pole foot position FP calculated based on the image information from the fisheye camera 200 as approach distance information.

[0063] For example, as shown in Fig. 14, it is assumed that the host vehicle OV is backing up at about 4 km / h toward a pole P that is about 5 m or more away. In this case, when the fisheye camera 200 detects the pole P and the automatic brake controller 400 acquires approach distance information from the camera image, the host vehicle OV automatically decelerates and comes to a complete stop. That is, as shown in Fig. 15, the host vehicle OV stops just before the pole P, and a collision between the host vehicle OV and the pole P is avoided.

[0064] As described above, the image recognition device 100 and the image recognition method according to the first embodiment provide the following effects.

[0065] (1) The image recognition device 100 includes a surrounding situation acquisition unit (image storage unit 101, cylindrical distortion correction unit 102), a vertical edge extraction unit 103, a pillar candidate extraction unit 104, a pole candidate determination unit 105, a pole base position setting unit 106, a movement distance acquisition unit 107, a detection distance difference calculation unit 108, and a pole recognition unit 109. The surrounding situation acquisition unit acquires surrounding situations that change as the vehicle equipped with a camera (fisheye camera 200) moves from time-series camera image information. The vertical edge extraction unit 103 extracts, as vertical edges, portions where pixel brightness changes by a predetermined value or more when scanning the camera image information in the horizontal direction. When the pillar candidate extraction unit 104 finds a pair of a positive edge +VE peak and a negative edge -VE peak among the vertical edges, it extracts the found edge pair as a pillar candidate. The pole candidate determination unit 105 determines whether the pole determination condition is satisfied for a pillar candidate, and determines the pillar candidate for which the pole determination condition is satisfied as a pole candidate P' erected on the ground. The pole base position setting unit 106 sets the lowest edge position in the vertical edge image representing the pole candidate P' as the pole base position FP. The movement distance acquisition unit 107 acquires the movement distance VL traveled by the host vehicle during a predetermined time as the host vehicle approaches and moves toward the pole candidate P'. The detection distance difference calculation unit 108 detects the horizontal distance from the camera position to the pole base position FP as the host vehicle approaches and moves toward the pole candidate P', and calculates the detection distance difference ΔL between the detection distance Ls at the start of the predetermined time and the detection distance Le at the end of the predetermined time. The pole recognition unit 109 recognizes the pole candidate P' for which the absolute difference |VL - ΔV| between the movement distance VL and the detection distance difference ΔL is less than a threshold as a pole P, with the position where the host vehicle contacts the ground being the pole base position FP.

[0066] That is, when the host vehicle moves toward the pole candidate P', the host vehicle's travel distance VL and the approach distance to the pole candidate P' (detection distance difference ΔL) are observed in time series. Then, if the pole candidate P' is a pole that is not reflected on the road surface, the host vehicle's travel distance VL and the approach distance to the pole candidate P' are equal. This condition is used for recognition and judgment, and the system recognizes whether the pole P has the pole foot position FP as its contact point with the ground. Therefore, it is possible to provide an image recognition device 100 that detects the host vehicle's approach to the pole P from the camera image when the host vehicle moves toward the pole P and prevents erroneous detection of the distance from the host vehicle to the pole P.

[0067] (2) The pillar candidate extraction unit 104 extracts as pillar candidates pairs that satisfy the following conditions: an integration value condition that the vertical integration value Σ+VE at the peak position of the positive edge +VE and the vertical integration value Σ-VE at the peak position of the negative edge -VE are each greater than or equal to a threshold value (+VEth, -VEth); and a horizontal distance condition that the horizontal distance D between the peak position of the positive edge +VE and the peak position of the negative edge -VE is less than or equal to a threshold value Dth.

[0068] That is, of the extracted edge pairs, those in which the plus edge +VE and the minus edge -VE are clearly visible and the vertical plus edge +VE and minus edge -VE are adjacent are extracted as pillar candidates. As a result, edge pairs in which at least one of the plus edge +VE and minus edge -VE is unclear or that are located far apart are excluded from the pillar candidates, making it possible to extract pillar candidates with high accuracy from the extracted edge pairs.

[0069] (3) The pole candidate determination unit 105 determines, as pole determination conditions, the edge amount condition that the edge amount of the right and left ends of the pillar candidate is equal to or greater than a threshold value in the image area downward from near the horizon in the camera image information, the edge width condition that the fluctuation in the width of the pillar candidate is equal to or less than a predetermined fluctuation width, and the edge direction condition that all edge peaks of the pillar candidate are in the vertical direction. A pillar candidate that satisfies all the pole determination conditions is determined to be a pole candidate P' erected on the ground.

[0070] That is, for the extracted pillar candidates, the image area prerequisites, edge amount conditions, edge width conditions, and edge direction conditions are determined as pole determination conditions, and a pillar candidate for which all of the pole determination conditions are met is determined to be a pole candidate P' standing on the ground. Therefore, pillar candidates with a pole shadow PS reflected therein and pillar candidates due to road paint such as white lines WL on the road surface are excluded, and it is possible to accurately determine a pole candidate P' from among the extracted pillar candidates.

[0071] (4) The device includes a display device 300 having a display screen 301 that displays time-series camera image information, and a distance conversion map setting unit 110 that sets a distance conversion map that converts the vertical coordinate position on the screen of the pole foot position FP displayed on the display screen 301 into the horizontal distance from the installation position of the fisheye camera 200 to the pole foot position FP. When the detection distance difference calculation unit 108 detects the vertical coordinate position of the pole foot position FP on the display screen 301 at the start of a predetermined time, it calculates the start detection distance Ls using the distance conversion map, and when the detection distance difference calculation unit 108 detects the vertical coordinate position of the pole foot position FP on the display screen 301 at the end of the predetermined time, it calculates the end detection distance Le using the distance conversion map.

[0072] That is, the horizontal distance from the installation position of the fisheye camera 200 to the pole base position FP is found based on the vertical coordinate position on the screen of the pole base position FP displayed on the display screen 301. Therefore, when calculating the detection distance difference ΔL between the detection distance Ls at the start and the detection distance Le at the end of a predetermined time, the detection distance difference ΔL can be easily calculated simply by detecting the vertical coordinate position of the pole base position FP on the display screen 301.

[0073] (5) The camera is a fisheye camera 200. The surrounding situation acquisition unit has an image storage unit 101 that stores fisheye images FI acquired from the fisheye camera 200 in chronological order, and a cylindrical distortion correction unit 102 that converts the fisheye images FI stored in the image storage unit 101 into cylindrical distortion-corrected images CI.

[0074] That is, the cylindrical distortion correction unit 102 converts the fisheye image FI of the pole P rising diagonally upward into a cylindrical distortion-corrected image CI of the pole P rising vertically, which is necessary for extracting vertical edges from the camera image. Therefore, even when using the fisheye camera 200, it is possible to obtain camera image information of the pole P rising vertically, without using a camera that can obtain distortion-free images. Note that if the fisheye camera 200 is an existing in-vehicle camera installed in the host vehicle, using it can also reduce costs.

[0075] (6) An automatic brake controller 400 is provided that automatically activates the brakes of the host vehicle when the host vehicle approaches an obstacle, based on information about the approach distance to the obstacle, before the host vehicle comes into contact with the obstacle. The pole recognition unit 109 outputs to the automatic brake controller 400 a pole recognition result indicating whether the pole P is a pole with the position where it contacts the ground as the pole foot position FP.

[0076] That is, if the automatic brake controller 400 can acquire approach distance information from a position far from the pole P, which is an obstacle, it can issue a warning to prompt the driver to apply the brakes at an early timing, thereby reducing the frequency of automatic brake activation. In contrast, approach distance information based on camera image information can be acquired from a time when the vehicle is located farther from the pole P than approach distance information acquired from, for example, a single sonar 504. Therefore, when the recognition result indicates that the pole P is from the ground, it is possible to meet the demand for acquiring approach distance information from a position far from the pole P in automatic brake control.

[0077] (7) An image recognition method for recognizing a pole erected on the ground based on camera image information, in which the surrounding conditions that change as the host vehicle equipped with a camera (fisheye camera 200) moves are acquired from time-series camera image information. When the camera image information is scanned horizontally, a portion where a change in pixel brightness of a predetermined value or more occurs is extracted as a vertical edge. When a pair of a positive edge +VE peak and a negative edge -VE peak is found among the vertical edges, the found edge pair is extracted as a pole candidate. Whether or not the pole determination condition is met for the pole candidate is determined, and the pole candidate for which the pole determination condition is met is determined to be a pole candidate P' erected on the ground. The edge position of the lowest edge in the vertical edge image representing the pole candidate P' is set as the pole base position FP. When the host vehicle moves toward the pole candidate P', the distance VL traveled by the host vehicle over a predetermined time is acquired. When the vehicle approaches a pole candidate P', the horizontal distance from the camera position to the pole foot position FP is detected, and the detected distance difference ΔL between the detected distance Ls at the start of a predetermined time period and the detected distance Le at the end is calculated. If the absolute value of the difference |VL-ΔV| between the moving distance VL and the detected distance difference ΔL is less than a threshold, the pole candidate P' is recognized as a pole P, with the pole foot position FP at the point where it touches the ground.

[0078] That is, when the host vehicle moves toward the pole candidate P', the host vehicle's moving distance VL and the approach distance to the pole candidate P' (detection distance difference ΔL) are observed in time series. Then, if the pole candidate P' is a pole that is not reflected on the road surface, the host vehicle's moving distance VL and the approach distance to the pole candidate P' are equal. This condition is used for recognition judgment, and it is recognized whether the pole P has the pole foot position FP as its contact point with the ground. Therefore, it is possible to provide an image recognition method that detects the host vehicle's approach to the pole P from the camera image when the host vehicle moves toward the pole P and prevents erroneous detection of the distance from the host vehicle to the pole P.

[0079] The image recognition device and image recognition method of the present invention have been described above based on Example 1, but the specific configuration is not limited to Example 1. Changes and additions to the design are permitted as long as they do not deviate from the gist of the invention according to each claim in the scope of the claims.

[0080] In the first embodiment, an example was shown in which a fisheye camera 200 was used as the camera, and the image storage unit 101 and the cylindrical distortion correction unit 102 were used as the surrounding situation acquisition unit. However, a camera with a standard lens or a wide-angle lens that does not cause distortion in the captured image may also be used as the camera, in which case cylindrical distortion correction processing is not required. Furthermore, it is sufficient for the camera and surrounding situation acquisition unit to be able to output at least a subject such as a vertically extending pole as a pair of edges extending vertically when the subject is captured as time-series camera images.

[0081] In the first embodiment, the detection distance difference calculation unit 108 detects the vertical coordinate position of the pole base position FP on the display screen 301 and then uses a distance conversion map to calculate the start detection distance Ls and the end detection distance Le for the predetermined time. However, the detection distance difference calculation unit is not limited to using a distance conversion map, as long as it calculates the detection distances based on the vertical coordinate position of the pole base position on the screen. For example, the detection distance difference calculation unit may calculate the start detection distance and the end detection distance for the predetermined time using the height dimension to the camera and the depression angle from the camera position toward the pole base position.

[0082] In the first embodiment, an example has been shown in which the image recognition device 100 is applied to a collision damage mitigation braking system A equipped with an automatic brake controller 400. However, the image recognition device of the present invention can also be applied to a parking assistance system or an obstacle avoidance assistance system that uses information on the distance to a pole set up on the ground as driving assistance information, in addition to the collision damage mitigation braking system.

[0083] In the first embodiment, an example in which the image recognition device 100 is applied to a passenger vehicle OV has been shown. However, the image recognition device of the present invention can be applied not only to passenger cars but also to large vehicles such as trucks and buses and various other vehicles. [Explanation of symbols]

[0084] A. Collision mitigation braking system 100 Image recognition device 101 Image memory unit (surrounding situation acquisition unit) 102 Cylindrical distortion correction unit (surrounding condition acquisition unit) 103 Vertical edge extraction unit 104 Pillar candidate extraction part 105 Pole candidate determination unit 106 Pole foot position setting section 107 Travel distance acquisition section 108 Detection distance difference calculation unit 109 Pole Recognition Unit 110 Distance conversion map setting section 200 Fisheye Camera (Camera) 300 display device 400 Automatic Brake Controller P pole P' Pole candidate FI Fisheye Image CI cylindrical distortion corrected image VL Travel distance Ls Initial detection distance Le End detection distance ΔL detection distance difference FP pole foot position

Claims

1. a surroundings situation acquisition unit that acquires, from time-series camera image information, surroundings situations that change as the vehicle moves; a vertical edge extraction unit that extracts, as a vertical edge, a portion where a change in pixel luminance of a predetermined value or more occurs when the camera image information is scanned in a horizontal direction; a pillar candidate extraction unit that, when a pair of a positive edge peak and a negative edge peak is found among the vertical edges, extracts the found edge pair as a pillar candidate; a pole candidate determination unit that determines whether or not a pole determination condition is satisfied for the pillar candidate, and determines the pillar candidate for which the pole determination condition is satisfied as a pole candidate erected on the ground; a pole base position setting unit that sets the edge position of the lowest end in the vertical edge image representing the pole candidate as the pole base position; a movement distance acquisition unit that acquires a movement distance traveled by the host vehicle during a predetermined time when the host vehicle moves toward the pole candidate; a detection distance difference calculation unit that detects the horizontal distance from a camera position to a base position of the pole when the host vehicle moves toward the pole candidate and calculates a detection distance difference between the detection distance at the start and the detection distance at the end of the predetermined time period; a pole recognition unit that recognizes the pole candidate for which the absolute value of the difference between the moving distance and the detected distance difference is less than a threshold as a pole having a position where the pole is in contact with the ground as the pole foot position; An image recognition device comprising:

2. 2. The image recognition device according to claim 1, The pillar candidate extraction unit extracts, as the pillar candidate, a pair of edges that satisfy an integration value condition that a vertical integration value of a peak position of the positive edge and a vertical integration value of a peak position of the negative edge are each equal to or greater than a threshold, and a horizontal distance condition that a horizontal distance between the peak position of the positive edge and the peak position of the negative edge is equal to or less than a threshold. put out An image recognition device characterized by:

3. 3. The image recognition device according to claim 1, the pole candidate determination unit determines, as the pole determination conditions, an edge amount condition that the edge amounts of the right and left ends of the pillar candidate are equal to or greater than a threshold value in an image region of the camera image information that is downward from near the horizon, an edge width condition that the fluctuation in width of the pillar candidate is equal to or less than a predetermined fluctuation width, and an edge direction condition that all edge peaks of the pillar candidate are in a vertical direction; The pillar candidate for which all the conditions of the pole determination conditions are satisfied is determined to be a pole candidate erected on the ground. An image recognition device characterized by:

4. The image recognition device according to any one of claims 1 to 3, a display device having a display screen for displaying the time-series camera image information; a distance conversion map setting unit that sets a distance conversion map that converts the vertical coordinate position of the pole foot position displayed on the display screen into a horizontal distance from the installation position of the camera to the pole foot position, When the detection distance difference calculation unit detects the vertical coordinate position of the foot of the pole on the display screen at the start of the predetermined time, it calculates the start detection distance using the distance conversion map, and when the detection distance difference calculation unit detects the vertical coordinate position of the foot of the pole on the display screen at the end of the predetermined time, it calculates the end detection distance using the distance conversion map. An image recognition device characterized by:

5. The image recognition device according to any one of claims 1 to 4, the camera is a fisheye camera, The surrounding situation acquisition unit has an image storage unit that stores fisheye images acquired from the fisheye camera in chronological order, and a cylindrical distortion correction unit that converts the fisheye images stored in the image storage unit into cylindrical distortion-corrected images that are free from distortion in the vertical direction by performing image conversion in which the fisheye images are projected onto a cylindrical screen that is perpendicular to a road surface, and outputs the cylindrical distortion-corrected images to the vertical edge extraction unit as the camera image information. An image recognition device characterized by:

6. The image recognition device according to any one of claims 1 to 5, an automatic brake controller that, when the host vehicle approaches an obstacle, automatically activates a brake of the host vehicle before the host vehicle comes into contact with the obstacle based on approach distance information to the obstacle, The pole recognition unit outputs a pole recognition result indicating whether the pole has a position where it contacts the ground as the pole foot position to the automatic brake controller. An image recognition device characterized by:

7. An image recognition method for recognizing a pole erected on the ground based on camera image information from a camera mounted on a vehicle, comprising: An image recognition device that processes the camera image information Obtaining a surrounding situation that changes as the vehicle on which the camera is mounted moves from the camera image information in a time series; extracting, as a vertical edge, a portion where a change in pixel luminance of a predetermined value or more occurs when the camera image information is scanned in a horizontal direction; When a pair of a positive edge peak and a negative edge peak is found among the vertical edges, the found edge pair is extracted as a pillar candidate; determining whether or not a pole determination condition is satisfied for the pillar candidate, and determining that the pillar candidate for which the pole determination condition is satisfied is a pole candidate erected on the ground; The position of the bottom edge in the vertical edge image representing the pole candidate is set as the pole base position; When the vehicle approaches the pole candidate, a travel distance traveled by the vehicle within a predetermined time is acquired; When the vehicle approaches the pole candidate, the horizontal distance from the camera position to the base position of the pole is detected, and a difference between the detected distance at the start and the detected distance at the end of the predetermined time period is calculated; The pole candidate for which the absolute difference between the movement distance and the detected distance difference is less than a threshold is recognized as a pole having a position where the pole is in contact with the ground as the pole foot position. An image recognition method comprising:

Images