Vehicle environment recognition method and vehicle environment recognition device

Abstract

To provide a vehicle environment recognition method and a vehicle environment recognition device capable of appropriately setting a region to which super-resolution processing is applied.SOLUTION: A vehicle environment recognition device 1 comprises: a camera 10 which images a periphery of a self vehicle a plurality of times in different timing; and a controller 20 for processing a plurality of images captured by the camera 10 in the different timing. The controller 20 estimates a travel direction of the self vehicle and sets a predetermined region on the image for each of the plurality of images on the basis of the estimated travel direction of the self vehicle. The controller 20 aligns the plurality of images on the basis of a moving amount on the image in the set region. The controller 20 performs super-resolution processing using the plurality of aligned images and generates a super-resolution image exceeding resolutions of the images captured by the camera 10. The controller 20 then recognizes a road structure on the basis of the super-resolution image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method and an apparatus for recognizing an environment for a vehicle. [Background technology]

[0002] Conventionally, there is known an invention that uses a camera to capture and recognize a vehicle license plate (Patent Document 1). The invention described in Patent Document 1 recognizes the license plate by performing super-resolution processing when it is determined that the area of ​​the identification medium is located within a preset area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2018 / 0189590 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in a method for setting a super-resolution region in a specific region, such as the invention described in Patent Document 1, there is a risk that an appropriate super-resolution region cannot be set.

[0005] The present invention has been made in consideration of the above problems, and its purpose is to provide a vehicle environment recognition method and a vehicle environment recognition device that make it possible to appropriately set the area in which super-resolution processing is performed. [Means for solving the problem]

[0006] A vehicle environment recognition method according to one embodiment of the present invention estimates the direction of travel of the vehicle, sets a predetermined area on each of a plurality of images based on the estimated direction of travel of the vehicle, aligns the plurality of images based on the amount of movement on the images in the set area, performs super-resolution processing using the aligned plurality of images, generates a super-resolution image that exceeds the resolution of an image captured by a camera, and recognizes road structures based on the super-resolution image. When it is determined that the vehicle is traveling through an intersection, the area is expanded in a direction corresponding to the left and right direction as seen from the vehicle so as to include the lane in which the vehicle is traveling, the oncoming lane, and the sidewalk, and the expanded area is divided into an area including the lane in which the vehicle is traveling, an area including the oncoming lane, and an area including the sidewalk, and each divided area is set. Effect of the Invention

[0007] According to the present invention, it is possible to appropriately set the area in which the super-resolution processing is performed. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of an environment recognition device 1 for a vehicle according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining an example of a method for setting an area where super-resolution processing is performed. [Diagram 3] FIG. 3 is a diagram for explaining another example of the method for setting the area where the super-resolution processing is performed. [Figure 4] FIG. 4 is a diagram for explaining still another example of the method for setting the area where the super-resolution processing is performed. [Diagram 5] FIG. 5 is a diagram for explaining still another example of the method for setting the area where the super-resolution processing is performed. [Figure 6] FIG. 6 is a diagram for explaining still another example of the method for setting the area where the super-resolution processing is performed. [Figure 7] FIG. 7 is a diagram for explaining still another example of the method for setting the area where the super-resolution processing is performed. [Figure 8] FIG. 8 is a flowchart illustrating an example of the operation of the vehicle environment recognition device 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and the description will be omitted.

[0010] An example of the configuration of a vehicle environment recognition device 1 will be described with reference to Fig. 1. As shown in Fig. 1, the vehicle environment recognition device 1 includes a camera 10, a yaw rate sensor 11, a GPS receiver 12, a navigation device 13, a controller 20, a steering actuator 14, an accelerator pedal actuator 15, and a brake actuator 16.

[0011] The vehicle environment recognition device 1 may be mounted on a vehicle having an automatic driving function, or may be mounted on a vehicle not having an automatic driving function. The vehicle environment recognition device 1 may also be mounted on a vehicle capable of switching between automatic driving and manual driving. The automatic driving function may be a driving assistance function that automatically controls only some of the vehicle control functions such as steering control, braking force control, and driving force control to assist the driver in driving. In this embodiment, the vehicle environment recognition device 1 will be described as being mounted on a vehicle having an automatic driving function.

[0012] The camera 10 has an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The location where the camera 10 is installed is not particularly limited, but as an example, the camera 10 is installed in front, on the side, or behind the vehicle. The camera 10 continuously captures images of the surroundings of the vehicle at a predetermined period (for example, a short period of about 10 msec). In other words, the camera 10 obtains a plurality of images captured at different timings. The camera 10 detects objects (pedestrians, bicycles, motorbikes, other vehicles, etc.) present around the vehicle, and information ahead of the vehicle (demarcation lines, traffic lights, signs, crosswalks, intersections, etc.). The images captured by the camera 10 are output to the controller 20.

[0013] The yaw rate sensor 11 detects the yaw rate of the host vehicle (the rate of change of the rotation angle in the turning direction). The yaw rate sensor 11 outputs the detected yaw rate to the controller 20.

[0014] The GPS receiver 12 detects the position information of the vehicle on the ground by receiving radio waves from artificial satellites. The position information of the vehicle detected by the GPS receiver 12 includes latitude information and longitude information. The method of detecting the position information of the vehicle is not limited to the GPS receiver 12. For example, the position may be estimated using a method called odometry. Odometry is a method of estimating the position of the vehicle by determining the amount of movement and the direction of movement of the vehicle according to the rotation angle and rotation angular velocity of the vehicle. The GPS receiver 12 outputs the detected position information to the controller 20.

[0015] The controller 20 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), a memory, and an input / output unit. A computer program for functioning as the vehicle environment recognition device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as a plurality of information processing circuits equipped in the vehicle environment recognition device 1. Note that here, an example is shown in which the plurality of information processing circuits equipped in the vehicle environment recognition device 1 are realized by software, but it is of course possible to configure the information processing circuits by preparing dedicated hardware for executing each information processing shown below. In addition, the plurality of information processing circuits may be configured by individual hardware. The controller 20 includes an area setting unit 21, an image shifting unit 22, a super-resolution processing unit 23, a road structure recognition unit 24, and a vehicle control unit 25 as an example of the plurality of information processing circuits.

[0016] The area setting unit 21 estimates the traveling direction of the host vehicle using the yaw rate acquired from the yaw rate sensor 11, and sets an area for performing super-resolution processing based on the estimated traveling direction of the host vehicle. In this embodiment, the area for performing super-resolution processing is a part of the image captured by the camera 10. Note that, as will be described later, the method for estimating the traveling direction of the host vehicle is not limited to the method using the yaw rate.

[0017] The image shifting unit 22 calculates the amount of movement between the images, and aligns the images by moving them by the calculated amount of movement. The amount of movement of the images is the amount of movement between two images. Such calculation of the amount of movement can be performed using SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), ZNCC (Zero-mean Normalized Cross-Correlation), or the like. Specifically, the image shifting unit 22 calculates the similarity by shifting one pixel at a time in the vertical and horizontal directions within the search range, and calculates the amount of movement to the location with the highest similarity as the amount of movement. A reference image is required to compare two images. As an example of a method of extracting a reference image, when the number of images required for super-resolution processing is N, the image captured earliest among the N images is extracted as the reference image.

[0018] The super-resolution processing unit 23 performs super-resolution processing using the image aligned by the image shifting unit 22 to generate an image with high resolution. Super-resolution processing is a technique for generating an image with high resolution by referring to multiple images. The super-resolution processing generates an image with a resolution that exceeds the resolution of the image captured by the camera 10. Note that the super-resolution processing is a well-known technique that is already publicly known, so it will not be described in detail here. The image generated by the super-resolution processing unit 23 is output to the road structure recognition unit 24.

[0019] The road structure recognition unit 24 recognizes road structures using the images input from the super-resolution processing unit 23. In this embodiment, road structures are defined as stationary objects. Specifically, road structures include structures such as lanes, stop lines, pedestrian crossings, road markings such as arrows, side strips, curbs, signs, and traffic lights. As an example of a method for recognizing road structures, segmentic segmentation is used, which recognizes what type of object each pixel is. The road structure recognition unit 24 outputs the recognition result to the vehicle control unit 25.

[0020] The vehicle control unit 25 controls the steering actuator 14, accelerator pedal actuator 15, and brake actuator 16 using the road structures recognized by the road structure recognition unit 24, and automatically drives the vehicle to the destination inputted into the navigation device 13.

[0021] In this embodiment, the GPS receiver 12 and the navigation device 13 do not have to be installed in the vehicle. If a mobile terminal (e.g., a smartphone) carried by a user can be linked to the functions of the vehicle, the mobile terminal functions as a substitute for the GPS receiver 12 and the navigation device 13.

[0022] Next, an example of a method for setting an area where super-resolution processing is performed will be described with reference to FIG.

[0023] The scene shown in FIG. 2 is a scene in which the host vehicle is traveling straight in the left lane on a three-lane road. An image 30 shown in FIG. 2 is a view in front of the host vehicle captured by the camera 10. In the scene shown in FIG. 2, the area setting unit 21 estimates the traveling direction of the host vehicle. As an example of a method for estimating the traveling direction of the host vehicle, the yaw rate is used. When the host vehicle is traveling straight as shown in FIG. 2, the yaw rate is zero or a value small enough to be considered as zero. Therefore, when it is determined that the yaw rate is zero or a value small enough to be considered as zero, the area setting unit 21 can estimate that the traveling direction of the host vehicle is a straight direction.

[0024] The method of estimating the traveling direction of the vehicle is not limited to the method using the yaw rate. For example, lane markings may be used. As shown in FIG. 2, when the vehicle is traveling straight, the lane markings are also straight. Therefore, when the lane markings are determined to be straight, the area setting unit 21 may estimate that the traveling direction of the vehicle is a straight direction. The lane markings are detected by well-known image processing.

[0025] Alternatively, the rotation angle of the steering wheel may be used. As shown in Fig. 2, when the vehicle is traveling straight, the rotation angle of the steering wheel is zero or a value small enough to be considered as zero. Therefore, when it is determined that the rotation angle of the steering wheel is zero or a value small enough to be considered as zero, the region setting unit 21 may estimate that the traveling direction of the vehicle is a straight direction.

[0026] In addition, in the case of an autonomous vehicle in which a future travel trajectory is set and the vehicle is controlled to travel along the set travel trajectory, the traveling direction of the vehicle can be estimated from the set travel trajectory. Alternatively, in the case where a travel route from the current position of the vehicle to the destination is searched for by a navigation device or the like, the traveling direction of the vehicle can be estimated based on the searched travel route.

[0027] After estimating the traveling direction of the host vehicle, the region setting unit 21 sets a region on the image where super-resolution processing is performed based on the estimated traveling direction of the host vehicle. The reference numeral 41 in FIG. 2 indicates the region where super-resolution processing is performed. As shown in FIG. 2, when the traveling direction of the host vehicle is estimated to be a straight direction, the region setting unit 21 sets a part of the traveling direction of the host vehicle in the image 30 as the region 41. The region 41 is a region on the image corresponding to a distant place as seen from the host vehicle, and is also a region on the image corresponding to the lane on which the host vehicle is traveling. However, as described later, the region set by the region setting unit 21 is not limited to only the region on the image corresponding to the lane on which the host vehicle is traveling. The region 41 may be set in the upper half of the image 30. In the following, unless otherwise specified, "region" means a region on the image, and for example, "region on the lane" means a region on the image corresponding to the lane.

[0028] After the region 41 is set, the image shifting unit 22 aligns the region 41 using multiple images captured at different times. The super-resolution processing unit 23 performs super-resolution processing using the images aligned by the image shifting unit 22 to generate an image with high resolution. The super-resolution processing makes the region 41 clear. As described above, according to this embodiment, the region 41 where super-resolution processing is performed is set based on the traveling direction of the host vehicle, so that it is possible to set the region where super-resolution processing is performed at a position appropriate for the traveling of the host vehicle.

[0029] Next, another example of the method for setting the area where the super-resolution processing is performed will be described with reference to FIG.

[0030] The scene shown in FIG. 3 is a continuation of the scene shown in FIG. 2, and is a scene in which the host vehicle changes lanes from the left lane to the center lane. In this case, the region setting unit 21 uses the yaw rate to estimate that the traveling direction of the host vehicle is from the left lane to the center lane. The yaw rate at the time of lane change is obtained in advance by experiments and simulations, and is stored in a storage device (not shown). The region setting unit 21 refers to the storage device and detects a yaw rate that matches or nearly matches the yaw rate detected during driving. Then, the region setting unit 21 estimates the traveling direction of the host vehicle based on the detection result.

[0031] The storage device stores not only the yaw rate when the vehicle changes lanes to the left, but also the yaw rate when the vehicle changes lanes to the right. Furthermore, the storage device also stores the yaw rate when the vehicle turns left and the yaw rate when the vehicle turns right.

[0032] In the scene shown in FIG. 3, when the traveling direction of the host vehicle is estimated to be from the left lane to the center lane, the region setting unit 21 sets the region 41 on the center lane.

[0033] Next, still another example of the method for setting the area where the super-resolution processing is performed will be described with reference to FIG.

[0034] The scene shown in FIG. 4 is a scene in which the host vehicle turns left. The image 31 shown in FIG. 4 is a scene in the left direction as seen from the host vehicle captured by the camera 10. In this case, the area setting unit 21 estimates that the traveling direction of the host vehicle is the left direction using the yaw rate. As described above, the yaw rate when the host vehicle turns left is also known in advance, so the area setting unit 21 can refer to this. Alternatively, the area setting unit 21 may estimate that the traveling direction of the host vehicle is the left direction using lane markings. A part of the lane markings shown in FIG. 4 is curved with a predetermined curvature. By using such lane markings, the area setting unit 21 can estimate that the traveling direction of the host vehicle is the left direction. When the traveling direction of the host vehicle is estimated to be the left direction, the area setting unit 21 sets an area 42 on the lane in which the host vehicle will travel in the future.

[0035] As shown in FIG. 4, when setting the region 42, the region setting unit 21 may also set regions 43-44. The regions 43-44 are regions set on the oncoming lane of the lane in which the host vehicle is traveling. The regions 43-44 are also regions on which super-resolution processing is performed, like the region 42. By expanding the region on which super-resolution processing is performed to the oncoming lane in this manner, the recognition accuracy of obstacles (e.g., oncoming vehicles) that may affect the traveling of the host vehicle is improved. Note that only the region 42 may be set.

[0036] Next, still another example of the method for setting the area where the super-resolution processing is performed will be described with reference to FIG.

[0037] The scene shown in FIG. 5 is a scene in which the host vehicle travels through an intersection. An image 32 shown in FIG. 5 is a view in front of the host vehicle captured by the camera 10. In the scene shown in FIG. 5, the host vehicle travels in a straight line. The area setting unit 21 estimates the travel direction of the host vehicle using the same method as described in FIG. 2. When the travel direction of the host vehicle is estimated to be a straight line, the area setting unit 21 sets an area 45 on the lane in which the host vehicle travels. The area 45 includes not only the lane in which the host vehicle travels beyond the intersection (the lane on the right side indicated by the area 45), but also the lane adjacent to the lane in which the host vehicle travels (the lane on the left side indicated by the area 45). However, the area 45 is not limited to this. The area 45 may be a range that includes only the lane in which the host vehicle travels beyond the intersection.

[0038] In the present embodiment, the area where the super-resolution process is performed is set based on the traveling direction of the vehicle. In addition, as shown in FIG. 5, when a pedestrian crossing is included in the image 32, the area setting unit 21 may set the pedestrian crossing in the image as the area 46, 47 where the super-resolution process is performed. Also, as shown in FIG. 6, when an intersection is included in the image 32, the area 45 may be expanded in a direction corresponding to the left and right direction as seen from the vehicle. The range of expansion is not particularly limited, but the area may be expanded to include the driving lane, the oncoming lane, and parts other than these lanes (for example, the sidewalk). Also, as shown in FIG. 7, the area expanded in the direction corresponding to the left and right direction may be divided into a plurality of areas. In the example shown in FIG. 7, the area is divided into four areas (areas 48 to 51). The area 48 includes the sidewalk, the area 49 includes the lane on which the vehicle is traveling, and the areas 50 to 51 include the oncoming lane. When performing the super-resolution process, it is preferable that the movements of objects in the target areas are the same or similar. Therefore, by setting the areas according to classifications such as the driving lane, the oncoming lane, and the sidewalk, highly accurate super-resolution process can be realized. The controller 20 can determine whether an intersection is included in the image 32, in other words, whether the vehicle is traveling through the intersection, by referring to the position information of the vehicle acquired from the GPS receiver 12 and a map database stored in the navigation device 13. Alternatively, the controller 20 may determine whether the vehicle is traveling through the intersection by image processing.

[0039] Next, an example of the operation of the vehicle environment recognition device 1 will be described with reference to the flowchart of FIG.

[0040] In step S101, the surroundings of the host vehicle are captured at different times by the camera 10 mounted on the host vehicle. The process proceeds to step S103, where the controller 20 acquires the yaw rate of the host vehicle from the yaw rate sensor 11. The controller 20 also acquires the position information of the host vehicle from the GPS receiver 12.

[0041] The process proceeds to step S105, where the region setting unit 21 estimates the traveling direction of the host vehicle mainly based on the yaw rate of the host vehicle from the yaw rate sensor 11. However, as described above, the method of estimating the traveling direction of the host vehicle is not limited to the method using the yaw rate. The region setting unit 21 sets a region for performing super-resolution processing for each of the multiple images based on the estimated traveling direction of the host vehicle.

[0042] The process proceeds to step S107, where the image shifter 22 aligns the region using a plurality of images captured at different times. The process proceeds to step S109, where the super-resolution processor 23 performs super-resolution processing using the images aligned by the image shifter 22 to generate an image with high resolution. The process proceeds to step S111, where the image generated by the super-resolution processor 23 is output to the road structure recognition unit 24.

[0043] The process proceeds to step S113, where the road structure recognition unit 24 recognizes road structures using the image input from the super-resolution processing unit 23. The process proceeds to step S115, where the vehicle control unit 25 controls the steering actuator 14, the accelerator pedal actuator 15, and the brake actuator 16 using the road structures recognized by the road structure recognition unit 24.

[0044] (Action and effect) As described above, the environment recognition device for a vehicle 1 according to this embodiment provides the following advantageous effects.

[0045] The vehicle environment recognition device 1 includes a camera 10 that captures images of the surroundings of the vehicle multiple times at different times, and a controller 20 that processes the multiple images captured by the camera 10 at different times. The controller 20 estimates the traveling direction of the vehicle and sets a predetermined area on each of the multiple images based on the estimated traveling direction of the vehicle. The controller 20 aligns the multiple images based on the amount of movement on the images in the set area. The controller 20 performs super-resolution processing using the aligned multiple images to generate a super-resolution image that exceeds the resolution of the image captured by the camera 10. The controller 20 then recognizes road structures based on the super-resolution image. According to this embodiment, the area for super-resolution processing is set based on the traveling direction of the vehicle, so that it is possible to set the area for super-resolution processing at a position appropriate for the running of the vehicle.

[0046] Furthermore, when it is determined that the host vehicle is traveling through an intersection, the controller 20 expands the area in which the super-resolution processing is performed in the direction corresponding to the left and right direction as seen from the host vehicle. This makes it possible to set the area in which the super-resolution processing is performed to include an obstacle (e.g., an oncoming vehicle) that may affect the traveling of the host vehicle.

[0047] Furthermore, the controller 20 may set an area for super-resolution processing at the position of the crosswalk on the image. This makes it possible to set an area for super-resolution processing that includes an obstacle (e.g., a pedestrian) that may affect the running of the host vehicle.

[0048] The controller 20 may also divide the expanded region into multiple regions. When performing super-resolution processing, it is preferable that the movements of objects in the target region are the same or similar. Therefore, by dividing the region so as to include objects with the same or similar movements, it is possible to realize highly accurate super-resolution processing.

[0049] The controller 20 estimates the traveling direction of the host vehicle based on the yaw rate acquired from the yaw rate sensor 11 that detects the yaw rate of the host vehicle, or on information about lane markings acquired from the image. Since the area where the super-resolution processing is performed is set based on the traveling direction of the host vehicle estimated in this way, it is possible to set the area where the super-resolution processing is performed at a position appropriate for the traveling of the host vehicle.

[0050] Each of the functions described in the above embodiments may be implemented by one or more processing circuits. A processing circuit includes a programmed processing device, such as a processor including electrical circuitry. A processing circuit also includes devices such as application specific integrated circuits (ASICs) or circuit components arranged to perform the described functions.

[0051] As described above, the embodiment of the present invention has been described, but the description and drawings forming a part of this disclosure should not be understood as limiting this invention. From this disclosure, various alternative embodiments, examples and operating techniques will become apparent to those skilled in the art. [Explanation of symbols]

[0052] 1 Vehicle environment recognition device 10. Camera 11 Yaw rate sensor 12 GPS receiver 13 Navigation devices 14 Steering actuator 15 Accelerator pedal actuator 16 Brake Actuator 20 Controller 21 Area setting section 22 Image shift section 23 Super-resolution processing section 24 Road structure recognition section 25 Vehicle control unit

Claims

1. An environment recognition method for a vehicle, comprising: an environment recognition device for a vehicle, the environment recognition device including a camera that captures an image of an environment around a vehicle multiple times at different times; and a controller that processes the multiple images captured by the camera at different times, The controller: Estimating a traveling direction of the host vehicle; setting a predetermined area on each of the plurality of images based on the estimated traveling direction of the host vehicle; performing position alignment of the plurality of images based on the amount of movement on the image in the set region; performing super-resolution processing using the aligned images to generate a super-resolution image having a resolution exceeding that of the image captured by the camera; Recognizing road structures based on the super-resolution image; When it is determined that the host vehicle is traveling through an intersection, the controller The area is expanded in a direction corresponding to a left-right direction as seen from the vehicle so as to include the lane on which the vehicle is traveling, the oncoming lane, and the sidewalk; The expanded area is divided into an area including a lane on which the vehicle is traveling, an area including the oncoming lane, and an area including the sidewalk, and each divided area is set. A method for recognizing an environment for a vehicle.

2. The controller sets the area at the position of the pedestrian crossing on the image.

2. The method for recognizing an environment for a vehicle according to claim 1.

3. The controller estimates a traveling direction of the host vehicle based on a yaw rate acquired from a sensor that detects a yaw rate of the host vehicle, or information on lane markings acquired from the image.

3. The method for recognizing an environment for a vehicle according to claim 1 or 2.

4. A camera that captures images of the surroundings of the vehicle multiple times at different times; a controller for processing a plurality of images captured by the camera at different times; The controller: Estimating a traveling direction of the host vehicle; setting a predetermined area on each of the plurality of images based on the estimated traveling direction of the host vehicle; performing position alignment of the plurality of images based on the amount of movement on the image in the set region; performing super-resolution processing using the aligned images to generate a super-resolution image having a resolution exceeding that of the image captured by the camera; Recognizing road structures based on the super-resolution image; When it is determined that the host vehicle is traveling through an intersection, the controller The area is expanded in a direction corresponding to a left-right direction as seen from the vehicle so as to include the lane on which the vehicle is traveling, an oncoming lane, and a sidewalk; The expanded area is divided into an area including a lane on which the vehicle is traveling, an area including the oncoming lane, and an area including the sidewalk, and each divided area is set. An environment recognition device for a vehicle.

Images