Road marking recognition method and road marking recognition device

Abstract

To provide a road surface sign recognition method and a road surface sign recognition device which can accurately recognize a road surface sign.SOLUTION: A road surface sign recognition device 1 comprises: a camera 10 which images the periphery of the own vehicle at different timing plural times; and a controller 20 which processes the plurality of images captured at the different timing by the camera 10. The controller 20 calculates an optical flow using the plurality of images, selects, when the calculated optical flow is long, many images from the plurality of images in comparison to a case where the optical flow is short, selects a pixel having luminance equal to or greater than a prescribed level from the selected images, generates one composite image by using the selected pixel, and recognizes a road surface sign by using the generated composite image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a road marking recognition method and a road marking recognition device. [Background technology]

[0002] Conventionally, the invention described in Patent Document 1 has been known as an invention for recognizing road markings. The invention described in Patent Document 1 synthesizes images taken at different points in time based on the vehicle speed, generates an image in which lane markings (lane dividing lines) such as white lines are elongated, and detects the lane markings from the generated image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent No. 7,415,134 B2 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the invention described in Patent Document 1, even if the actual lane markings are dashed, the lane markings in the synthesized image may appear as solid lines, and when applied to road markings other than lane markings, the arrow lines may appear double.

[0005] The present invention has been made in view of the above problems, and an object of the present invention is to provide a road marking recognition method and a road marking recognition device that are capable of recognizing road markings with high accuracy. [Means for solving the problem]

[0006] A road sign recognition method according to one aspect of the present invention includes: calculating an optical flow using a plurality of images; and selecting a larger number of images from among the plurality of images when the calculated optical flow is long compared to when the calculated optical flow is short; For each of the small regions in the case where the image is decomposed into a plurality of small regions corresponding to each other among the plurality of images, an image in which the luminance of the pixels of the small region is equal to or higher than a predetermined value is selected from the selected images, and for each of the small regions, Selected imageA synthetic image is generated using the above method, and road markings are recognized using the generated synthetic image. Effect of the Invention

[0007] According to the present invention, it is possible to recognize road markings with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of a road marking recognition device 1 according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a stationary object region. [Diagram 3] FIG. 3 is a diagram for explaining an example of an image selection method. [Figure 4] FIG. 4 is a diagram for explaining an example of a method for calculating the luminance in the small region 41. In FIG. [Diagram 5] FIG. 5 is a diagram illustrating an example of an image with the highest brightness. [Figure 6] FIG. 6 is a diagram illustrating an example of a composite image. [Figure 7] FIG. 7 is a flowchart illustrating an example of the operation of the road marking recognition device 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and the description will be omitted.

[0010] An example of the configuration of a road sign recognition device 1 will be described with reference to Fig. 1. As shown in Fig. 1, the road sign recognition device 1 includes a camera 10, a yaw rate sensor 11, a controller 20, a steering actuator 12, an accelerator pedal actuator 13, and a brake actuator 14.

[0011] The road sign recognition device 1 may be mounted on a vehicle having an automatic driving function, or may be mounted on a vehicle not having an automatic driving function. The road sign recognition device 1 may also be mounted on a vehicle capable of switching between automatic driving and manual driving. The automatic driving function may be a driving assistance function that automatically controls only some of the vehicle control functions such as steering control, braking force control, and driving force control to assist the driver in driving. In this embodiment, the road sign recognition device 1 will be described as being mounted on a vehicle having an automatic driving function.

[0012] The camera 10 has an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The location where the camera 10 is installed is not particularly limited, but as an example, the camera 10 is installed in front, on the side, or behind the vehicle. The camera 10 continuously captures images of the surroundings of the vehicle at a predetermined cycle. In other words, the camera 10 acquires a plurality of images captured at different times. The camera 10 detects objects (pedestrians, bicycles, motorbikes, other vehicles, etc.) present around the vehicle, and information in front of the vehicle (division lines, traffic lights, signs, crosswalks, intersections, etc.). The images captured by the camera 10 are output to the controller 20.

[0013] The yaw rate sensor 11 detects the yaw rate of the host vehicle (the rate of change of the rotation angle in the turning direction). The yaw rate sensor 11 outputs the detected yaw rate to the controller 20.

[0014] The controller 20 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), a memory, and an input / output unit. A computer program for functioning as the road marking recognition device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as a plurality of information processing circuits equipped in the road marking recognition device 1. Note that here, an example is shown in which the plurality of information processing circuits equipped in the road marking recognition device 1 are realized by software, but it is of course possible to configure the information processing circuits by preparing dedicated hardware for executing each information processing shown below. In addition, the plurality of information processing circuits may be configured by individual hardware. The controller 20 includes an optical flow calculation unit 21, a distance calculation unit 22, a behavior calculation unit 23, a stationary object area setting unit 24, an image selection unit 25, a brightness calculation unit 26, an image synthesis unit 27, a road structure recognition unit 28, and a vehicle control unit 29 as an example of the plurality of information processing circuits.

[0015] The optical flow calculation unit 21 uses a past image and a current image to calculate how much an object in an image has moved. The optical flow is a vector that represents the movement of an object in an image. An example of a calculation method will be described. An image captured at time T1 is image I1, and an image captured at time T2 is image I2. Time T1<time T2. The optical flow calculation unit 21 divides the image I1 into arbitrary sizes. Although the arbitrary sizes are not particularly limited, the horizontal size of the image is N pixels, and the vertical size of the image is M pixels. The arbitrary sizes are N×M. The optical flow calculation unit 21 compares an area (N×M) of the image I1 with an area (N×M) of the image I2. As a comparison method, SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), ZNCC (Zero-mean Normalized Cross-Correlation), etc. are known. These methods are well-known techniques, so their explanations will be omitted. The optical flow calculation unit 21 detects, as related feature points, feature points on the current image that correspond to the same object in real space as an object corresponding to a feature point on a past image. The optical flow calculation unit 21 calculates a combination of the feature points on the past image and the feature points on the current image that are related to each other as an optical flow.

[0016] The distance calculation unit 22 calculates the distance from the camera 10 to an object corresponding to a feature point from the parallax (shift amount) between a feature point on a past image and a feature point on a current image that are related to each other. In other words, the distance calculation unit 22 calculates the distance from the vehicle to the object corresponding to the feature point. The distance calculation unit 22 calculates three-dimensional coordinates in real space corresponding to these feature points from the parallax between the feature points on a past image and a feature point on a current image that are related to each other. The distance calculation unit 22 may calculate the distance to an object by combining a radar or a lidar with the camera 10.

[0017] The behavior calculation unit 23 calculates the three-axis translational motion and three-axis rotational motion of the vehicle. The three-axis translational motion is the motion in the front-rear direction, the width direction, and the up-down direction of the vehicle. The three-axis rotational motion is the rotational motion around three axes including the roll axis, the pitch axis, and the yaw axis. These motions are calculated using a wheel speed sensor, an acceleration sensor, a rotation sensor, and the like. Alternatively, these motions are calculated using an image. As a calculation method using an image, Non-Patent Document 1 "KITT, Bernd; GEIGER, Andreas; LATEGAHN, Henning. Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In: 2010 ieee intelligent vehicles symposium. IEEE, 2010. p. 486-492." is known.

[0018] The stationary object area setting unit 24 combines the optical flow from the optical flow calculation unit 21, the calculation result from the distance calculation unit 22, and the calculation result from the behavior calculation unit 23 to calculate a three-dimensional movement amount (three-dimensional motion vector) excluding the behavior of the vehicle itself. The stationary object area setting unit 24 compares the calculated absolute movement amount with a threshold value. If the movement amount is greater than the threshold value, the area is determined to be an area where a moving object such as another vehicle exists. The stationary object area setting unit 24 excludes areas where moving objects exist from the image. In this way, a stationary object area where no moving objects exist is set.

[0019] The image selection unit 25 selects an image to be used for synthesis from among a plurality of images according to the magnitude of the optical flow calculated by the optical flow calculation unit 21 .

[0020] The luminance calculation unit 26 calculates the luminance in a direction perpendicular to the optical flow for a predetermined region.

[0021] The image synthesis unit 27 generates one image using the pixel having the highest brightness calculated by the brightness calculation unit 26. The image synthesis unit 27 outputs the generated image to the road structure recognition unit .

[0022] The road structure recognition unit 28 recognizes road structures using the images input from the image synthesis unit 27. In this embodiment, the road structures refer to road markings. Road markings are markings defined by laws, regulations, or rules, and representative examples include lane markings (so-called white lines), arrows indicating the direction of travel, and lines indicating that the vehicle is approaching a safety zone or an obstacle on the road.

[0023] The vehicle control unit 29 controls the steering actuator 12, the accelerator pedal actuator 13, and the brake actuator 14 based on the road markings recognized by the road structure recognition unit 28.

[0024] Next, an example of a method for setting a stationary object region will be described with reference to FIG.

[0025] Image 30 shown in Fig. 2 is an image captured by camera 10 while the host vehicle is traveling. In the scene shown in Fig. 2, the host vehicle is traveling in the right lane of a two-lane road. As seen from the host vehicle, there is another vehicle ahead on the lane in which the host vehicle is traveling. There is also another vehicle in the adjacent oncoming lane. Stationary object area setting unit 24 sets a stationary object area by excluding area 40 in which the other vehicle (moving object) exists from image 30 using the method described above.

[0026] Next, an example of an image selection method will be described with reference to FIG.

[0027] The image selection unit 25 selects an image to be used for synthesis from among the multiple images according to the magnitude of the optical flow. In the example of FIG. 3, the number of multiple images is 16, including images 30, 30a to 30o. As shown in image 30 of FIG. 3, the road markings around the vehicle are faded. Specifically, the lane dividing line separating the right lane from the left lane is faded, the arrow line on the lane in which the vehicle is traveling is also faded, and the line on the right side of the vehicle (the line indicating that the vehicle is approaching a safety zone or an obstacle on the road) is also faded. In this way, the road markings on the road may be faded. It goes without saying that road markings are important in autonomous driving, and therefore, when the road markings are faded, it is required that the controller 20 processes the image to such an extent that the road markings can be recognized.

[0028] When the same scenery is captured at different times at high speed while driving, the scenery flows in the opposite direction to the traveling direction of the vehicle, so that at the same observation point, the road markings may be blurred in one image while being clear in another image. Therefore, in this embodiment, an image in which the road markings are clearly visible is selected from among multiple images. Here, the road markings are usually clearly visible in a part of an image. Consider a case where the image 30 is finely decomposed into a grid pattern (for example, decomposed into each pixel or into an area of ​​several pixels vertically and horizontally). In this case, it is preferable to select an image in which the road markings are most clearly visible in each of the decomposed small areas. Therefore, in this embodiment, an image in which the road markings are most clearly visible in a certain small area is selected. However, what is used for synthesis is not the entire image, but only the pixel data of the part in which the road markings are most clearly visible. Therefore, selecting an image in which the road markings are most clearly visible means, in more detail, selecting an image in which the road markings are most clearly visible in a certain part from among 16 images. The method for dividing the road marking into small regions is not particularly limited, but it is sufficient to divide the road marking into small regions of a size that allows a part of the road marking to be recognized.

[0029] The optical flow is usually longer the closer to the vehicle and shorter the farther from the vehicle. This is because the scenery changes faster the closer to the vehicle and slower the farther from the vehicle. The image selection unit 25 selects more images as the optical flow is longer. On the other hand, the image selection unit 25 selects fewer images as the optical flow is shorter. As shown in FIG. 3, consider a case where an image 30 is divided into an upper half region R1 and a lower half region R2. In this case, the optical flow of region R1 is shorter than the optical flow of region R2. Therefore, as shown in FIG. 3, in region R1, images are selected by thinning out every other image. On the other hand, in region R2, all 16 images are selected. The reason for thinning out in region R1 is that the scenery changes slowly, making it difficult to detect parts where road markings are clearly reflected, and to reduce the computational processing load. On the other hand, the reason for selecting all 16 images in region R2 is that the scenery changes quickly, making it easy to detect parts where road markings are clearly reflected. In region R1, the image in which the road markings are most clearly visible in each small region is selected from the eight images. In region R2, the image in which the road markings are most clearly visible in each small region is selected from the 16 images. Note that eight and sixteen images are merely examples and are not limiting. It is sufficient to select more images for regions with longer optical flows.

[0030] Referring now to FIG. 4, an example of a method for selecting an image in which road markings are most clearly visible in a given portion will be described.

[0031] The small region 41 indicated by the reference numeral 41 in FIG. 4 corresponds to the small region described above. The luminance calculation unit 26 calculates the luminance of the small region 41 in the direction perpendicular to the optical flow. More specifically, the luminance calculation unit 26 calculates the luminance of the small region 41 in all 16 images of the images 30, 30a to 30o. The luminance increases as the road marking is more clearly captured. As shown in FIG. 5, it is assumed that the image with the highest luminance in the small region 41 among the images 30, 30a to 30o is the image 30e. In this case, the image synthesis unit 27 uses the pixel 50 of the image 30e for the small region 41. The same process is repeated for the other small regions. The image synthesis unit 27 selects the pixel with the highest luminance from the eight images for each small region in the region R1, and selects the pixel with the highest luminance from the 16 images for each small region in the region R2. The image synthesis unit 27 then generates one image using the pixels selected for each small region. The image generated in this manner is composed of the pixel with the highest luminance in each small region. The image from which the brightest pixels in each small region are synthesized is the image captured earliest among the 16 images, i.e., image 30. As shown in FIG. 6, the brightest pixels in each small region of image 30 are synthesized to generate a synthesized image 60. As can be seen from synthesized image 60, compared to image 30, the lane markings separating the right lane from the left lane, the arrow marking on the lane in which the vehicle is traveling, and the line on the right side of the vehicle (the line indicating that the vehicle is approaching a safety zone or an obstacle on the road) are clearer. This improves the recognition accuracy of road markings.

[0032] In the above example, it is assumed that the vehicle is traveling in a straight line. The vehicle may turn right or left while traveling. When the vehicle turns right or left, a large change occurs in the image before and after the turn. Therefore, when the present invention is applied to a scene in which the vehicle turns right or left, the image up to the time when the vehicle turns right or left may be used. For example, in FIG. 3, if the vehicle turns right at the timing when the image 30g is captured, the images 30, 30a to 30g may be used. Whether the vehicle turns right or left may be determined using the yaw rate acquired from the yaw rate sensor 11.

[0033] In addition, in the description, the image with the highest brightness is selected in each small region, but this is not limited to the above. An image with a brightness equal to or higher than a predetermined value may be selected. The brightness equal to or higher than a predetermined value may be any brightness that can be recognized by the controller 20.

[0034] Next, an example of the operation of the road marking recognition device 1 will be described with reference to the flowchart of FIG.

[0035] In step S101, the surroundings of the host vehicle are captured at different times by the camera 10 mounted on the host vehicle. The process proceeds to step S103, in which the controller 20 obtains the yaw rate of the host vehicle from the yaw rate sensor 11.

[0036] The process proceeds to step S105, where the stationary object region setting unit 24 excludes the region 40 in which the moving object exists from the image 30 (see FIG. 2), thereby setting a stationary object region in which no moving object exists.

[0037] The process proceeds to step S107, where the image selection unit 25 selects more images as the optical flow becomes longer. On the other hand, the image selection unit 25 selects fewer images as the optical flow becomes shorter. The process proceeds to step S109, where the luminance calculation unit 26 calculates the luminance in the direction perpendicular to the optical flow for this small region 41. The process proceeds to step S111, where the image synthesis unit 27 selects the pixel with the highest luminance from among the eight images for each small region in region R1, and selects the pixel with the highest luminance from among the 16 images for each small region in region R2 (see Figures 3 to 5). The image synthesis unit 27 then generates one image using the pixels selected for each small region (see Figure 6).

[0038] The process proceeds to step S113, where the road structure recognition unit 28 recognizes the road markings using the images input from the image synthesis unit 27. The process proceeds to step S115, where the vehicle control unit 29 controls the steering actuator 12, the accelerator pedal actuator 13, and the brake actuator 14 based on the road markings recognized by the road structure recognition unit 28.

[0039] (Action and effect) As described above, the road sign recognition device 1 according to this embodiment provides the following advantageous effects.

[0040] The road marking recognition device 1 includes a camera 10 that captures the surroundings of the vehicle multiple times at different times, and a controller 20 that processes the multiple images captured by the camera 10 at different times. The controller 20 calculates an optical flow using multiple images, and selects more images from the multiple images when the calculated optical flow is long compared to when the calculated optical flow is short. This means that, for example, as shown in FIG. 3, in an area R1 where the optical flow is short, images are selected in a thinned manner every other image, while in an area R2 where the optical flow is long, all 16 images are selected. The controller 20 selects pixels having a predetermined brightness or higher from the selected images, and generates one composite image using the selected pixels. The controller 20 then recognizes road markings using the generated composite image. As shown in FIG. 6, in the composite image 60, the lane markings separating the right lane from the left lane, the arrow line on the lane on which the vehicle is traveling, and the line on the right side of the vehicle (the line indicating that the vehicle is approaching a safety zone or an obstacle on the road) are clearer than in the image 30 before synthesis. In this way, according to this embodiment, the recognition accuracy of road markings is improved.

[0041] In addition, the controller 20 uses the optical flow to remove an area 40 where a moving object exists from the image (see FIG. 2), thereby eliminating the influence of the moving object.

[0042] In addition, the controller 20 calculates the luminance of the image in a direction perpendicular to the optical flow. This makes it possible to include road markings that affect the running of the host vehicle in the synthesis target.

[0043] The controller 20 may calculate the luminance of the image in a direction perpendicular to the traveling direction of the host vehicle. This makes it possible to include road markings that affect the traveling of the host vehicle in the synthesis target.

[0044] The controller 20 also selects the pixel with the highest brightness from the selected images. By using such a pixel with the highest brightness, the road markings in the combined image become clearer.

[0045] The controller 20 may select an image from among a plurality of images based on the yaw rate. Specifically, when the controller 20 determines that the host vehicle has turned right or left, it may select images up to the time when the host vehicle has turned right or left, and ignore images thereafter. This reduces the effect of image changes when the host vehicle turns right or left.

[0046] Each of the functions described in the above embodiments may be implemented by one or more processing circuits. A processing circuit includes a programmed processing device, such as a processor including electrical circuitry. A processing circuit also includes devices such as application specific integrated circuits (ASICs) or circuit components arranged to perform the described functions.

[0047] As described above, the embodiment of the present invention has been described, but the description and drawings forming a part of this disclosure should not be understood as limiting this invention. From this disclosure, various alternative embodiments, examples and operating techniques will become apparent to those skilled in the art. [Explanation of symbols]

[0048] 1 Road marking recognition device 10. Camera 11 Yaw rate sensor 12 Steering actuator 13 Accelerator pedal actuator 14 Brake Actuator 20 Controller 21 Optical flow calculation unit 22 Distance calculation unit 23 Behavior Calculation Unit 24 Stationary object area setting section 25 Image Selection Section 26 Luminance calculation section 27 Image synthesis section 28 Road structure recognition section 29 Vehicle control unit

Claims

1. A road sign recognition method for a road sign recognition device including a camera that captures an image of an area around a vehicle multiple times at different times, and a controller that processes the multiple images captured by the camera at different times, comprising: The controller: Calculating an optical flow using the plurality of images; selecting more images from the plurality of images when the calculated optical flow is long compared to when the calculated optical flow is short; selecting, for each of the small regions obtained by decomposing the image into a plurality of small regions corresponding to each other among the plurality of images, an image in which the luminance of pixels of the small region is equal to or greater than a predetermined value from among the selected images; generating a composite image using the images selected for each of the sub-regions; Recognizing road markings using the generated synthetic image A road marking recognition method comprising:

2. The controller uses the optical flow to exclude areas in which moving objects exist from the image.

2. The road marking recognition method according to claim 1,

3. The controller calculates the brightness of the image in a direction perpendicular to the optical flow.

3. The road marking recognition method according to claim 1 or 2.

4. The controller calculates the brightness of the image in a direction perpendicular to a traveling direction of the host vehicle.

3. The road marking recognition method according to claim 1 or 2.

5. The controller selects, for each of the small regions, from among the selected images, an image in which the pixel brightness of the small region is the highest.

5. The road marking recognition method according to claim 1,

6. The controller selects an image from the plurality of images based on a yaw rate acquired from a sensor that detects a yaw rate of the host vehicle.

6. The road marking recognition method according to claim 1,

7. A camera that captures images of the surroundings of the vehicle multiple times at different times; a controller for processing a plurality of images captured by the camera at different times; The controller: Calculating an optical flow using the plurality of images; selecting more images from the plurality of images when the calculated optical flow is long compared to when the calculated optical flow is short; selecting, for each of the small regions obtained by decomposing the image into a plurality of small regions corresponding to each other among the plurality of images, an image in which the luminance of pixels of the small region is equal to or greater than a predetermined value from among the selected images; generating a composite image using the images selected for each of the sub-regions; Recognizing road markings using the generated synthetic image A road marking recognition device comprising:

Images