Image processing method and image processing device

Abstract

An image processing method according to the present invention is performed by a controller that generates a super-resolution image using images of a plurality of frames. The controller: sets, in the images of the plurality of frames, a target region in which the depth direction is parallel to the travelling direction of a vehicle and the width direction intersects the travelling direction of the vehicle; corrects a shift of the target region in the images of the plurality of frames on the basis of the amount of movement in the component of the travelling direction of the vehicle in the target region of the images of the plurality of frames; and generates a super-resolution image on the basis of the images of the plurality of frames in which the shift of the target region has been corrected.

Description

Image processing method and image processing device

[0001] The present invention relates to an image processing method and an image processing device.

[0002] Patent Document 1 discloses a method for extracting a target region that is a target of super-resolution processing from an environmental image acquired by an imaging device, and generating a super-resolution image by performing super-resolution processing on the target region.

[0003] International Publication No. 2022 / 209373

[0004] The technology described in Patent Document 1 has a problem in that it is not possible to extract an object that is difficult to recognize on an image as a target region, and therefore it is not possible to perform super-resolution processing.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing method and an image processing device that can perform super-resolution processing even in areas that are difficult to recognize on an image.

[0006] An image processing method according to one aspect of the present invention is executed by a controller that generates a super-resolution image using multiple frames of images captured in time series by an imaging device mounted on a vehicle. The controller sets a gaze area of ​​a predetermined shape in the multiple frames of images, corrects a shift of the gaze area in the multiple frames of images based on a movement amount of the gaze area in the multiple frames of images, and generates a super-resolution image based on the multiple frames of images in which the shift of the gaze area has been corrected.

[0007] According to the present invention, by focusing on an image in a fixation area that includes road structures, it is possible to refer to the amount of movement in the same direction as the road structures, thereby enabling super-resolution processing to be performed on the fixation area.

[0008] Fig. 1 is a block diagram showing the configuration of an image processing device according to this embodiment. Fig. 2 is a flowchart showing an image processing method according to this embodiment. Fig. 3 is a diagram showing an example of an image captured at a right-turn intersection. Fig. 4 is a diagram showing the relationship between a right-turn intersection and a vehicle. Fig. 5 is a diagram showing feature points and a gaze area.

[0009] 1 is a block diagram showing the configuration of an image processing device 1 according to this embodiment. The image processing device 1 generates a super-resolution image using a plurality of frames of images captured in time series. The image processing device 1 also detects road structures based on the super-resolution image. In this embodiment, the configuration and operation of the image processing device 1 will be described using an example situation in which a vehicle makes a right turn at an intersection ahead of the vehicle and detects a white line (an example of a road structure) on the road the vehicle will enter after turning right.

[0010] The image processing device 1 is mounted on a vehicle. The vehicle may have an automatic driving function or may not have an automatic driving function. The image processing device 1 may be a vehicle that can switch between automatic driving and manual driving. The automatic driving function may be a function that automatically controls only some of the vehicle control functions, such as steering control, braking force control, and driving force control, to assist the driver in driving. A control device (ECU: Electronic Control Unit) of a vehicle with an automatic driving function may control actuators that adjust the behavior of the vehicle, such as a steering actuator, an accelerator pedal actuator, and a brake actuator, based on road structures detected by the image processing device 1.

[0011] The image processing device 1 includes an imaging device 10 , a sensor 20 , a storage device 30 , and a controller 40 .

[0012] The imaging device 10 captures images of roads ahead and to the left and right of the vehicle. The imaging device 10 is a digital camera equipped with a solid-state imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The imaging device 10 is attached to the vehicle as an integral part of the vehicle. The imaging device 10 may be disposed so as to face the front of the vehicle, or may be disposed tilted from the front so as to face the left and right front of the vehicle. The images captured by the imaging device 10 may be color images or monochrome images.

[0013] The images captured by the imaging device 10 are output to the controller 40. The imaging device 10 captures images at a predetermined frame rate, and outputs the generated images of multiple frames to the controller 40 in chronological order.

[0014] The sensor 20 acquires location information indicating the location of the vehicle. The sensor 20 includes, for example, a GPS receiver that receives location information from a GPS system. The sensor 20 may also include a ranging device, such as a radar device, a camera, or a lidar device, that acquires ranging information including the distance and direction to objects around the vehicle. The information acquired by the sensor 20 is input to the controller 40.

[0015] The storage device 30 stores map information, which is information about roads on which the vehicle will travel. The map information includes information indicating the absolute positions of roads and road connections. The storage device 30 also stores route information indicating the vehicle's travel route. The route information includes the vehicle's future travel route, such as when the vehicle will turn right at an intersection ahead.

[0016] The controller 40 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memory, and input / output units. A computer program for causing the microcomputer to function as the image processing device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as multiple information processing circuits equipped in the image processing device 1.

[0017] In this embodiment, an example is shown in which the multiple information processing circuits provided in the image processing device 1 are realized by software. Of course, it is also possible to configure the multiple information processing circuits for executing the information processing described below using dedicated hardware. Furthermore, the multiple information processing circuits may be configured using individual hardware.

[0018] The controller 40 includes a plurality of information processing circuits (information processing functions), including a feature point detection unit 41, a feature point movement amount measurement unit 42, a direction determination unit 43, a gaze area setting unit 44, a super-resolution processing unit 45, and a structure detection unit 48.

[0019] The feature point detection unit 41 detects feature points on the image by processing the image acquired from the imaging device 10. A feature point is a group of one or more pixels. This feature point is a point that satisfies the following requirements: (1) the difference in brightness value from surrounding pixels is equal to or greater than a preset threshold, and (2) it exists on the road surface. Whether or not a point whose difference in brightness value from surrounding pixels is equal to or greater than a threshold exists on the road surface can be determined by determining the direction of movement in the image using well-known image processing techniques. The feature point detection unit 41 detects feature points in each of multiple frames of images.

[0020] The feature point movement amount measuring unit 42 receives information about feature points on the road surface from the feature point detecting unit 41. The feature point movement amount measuring unit measures the amount of movement of feature points in the images of multiple frames.

[0021] The direction determination unit 43 refers to map information and route information and measures the distance from the vehicle to a right-turn intersection. A right-turn intersection is an intersection on the vehicle's travel route where the vehicle plans to turn right. The direction determination unit 43 calculates the intersection angle, which is the angle between the road the vehicle is currently traveling on and the intersecting road, i.e., the road the vehicle will enter after turning right at the right-turn intersection.

[0022] The gaze area setting unit 44 sets a gaze area. The gaze area is an area to be observed on the image that is the target of super-resolution processing. In this embodiment, the shape of the gaze area is set to be rectangular because the white line on the road after a right turn is to be detected.

[0023] The super-resolution processing unit 45 performs super-resolution processing on the region of interest using images of multiple frames. By performing super-resolution processing, a super-resolution image with higher resolution than the image captured by the imaging device 10 is generated. The super-resolution processing unit 45 includes a correction unit 46 and an image integration unit 47.

[0024] The correction unit 46 corrects the image shift in the region of interest in the images of multiple frames. The image integration unit 47 generates a super-resolution image by combining the images of multiple frames that have been corrected.

[0025] The structure detection unit 48 detects white lines on the intersecting road ahead of the right turn based on the super-resolution image output for each region.

[0026] 2 is a flowchart showing an image processing method according to this embodiment. The process shown in this flowchart is executed by the controller 40.

[0027] 3, an image captured at a right-turn intersection includes a right-turn intersection 100 and a crossroad 110 that the vehicle enters after turning right at the right-turn intersection 100. The image also includes various white lines on the crossroad 110, specifically, an outer lane line 112, a lane boundary line 113, and a center line 115, as well as a crosswalk 120 for crossing the crossroad 110.

[0028] The direction determination unit 43 refers to the map information and route information to obtain the position of the right-turn intersection 100. The direction determination unit 43 detects its own position on the map based on the map information and information from the sensor 20. The direction determination unit 43 integrates this information and measures the distance from the vehicle to the right-turn intersection 100 (step S10).

[0029] If the vehicle is far from the right-turn intersection 100, the characteristic points will also be displayed small, and there is a possibility that the characteristic points will not be detected on the image. Therefore, in step S11, the direction determination unit 43 determines whether the distance D from the vehicle to the right-turn intersection 100 is within a determination distance, as shown in FIG. 4 . This determination distance is preset as a distance that allows the characteristic points on the right-turn intersection 100 and the characteristic points on the intersecting road 110 to be detected on the image, and is, for example, 30 m. If the distance D is greater than the determination distance, a negative determination is made in step S11, and the process of step S11 is performed again. If the distance D is within the determination distance, a positive determination is made in step S11, and the process proceeds to step S12.

[0030] The direction determination unit 43 measures the intersection angle (step S12). The direction determination unit 43 calculates the intersection angle as viewed from the imaging device 10, i.e., the intersection angle that appears on the image, taking into account the intersection angle θ of the road identified from the map information (see FIG. 4), the height (installation height) of the imaging device 10 from the ground, and the distance D.

[0031] The direction determination unit 43 determines whether the intersection angle is within a reference angle range (step S13). The reference angle range is an angle range for determining whether the road on which the vehicle travels before entering the right-turn intersection 100 is perpendicular to the intersecting road 110. The perpendicular state includes a state in which the roads are strictly perpendicular to each other and can be considered perpendicular. For example, the reference angle range is set to a range of 70° to 110°. The white line that is the subject of the super-resolution image extends along the intersecting road 110. This step S13 estimates whether the white line that is the subject of attention extends in a direction that intersects (is approximately perpendicular to) the vehicle's traveling direction. If the intersection angle is greater than the reference angle range, a negative determination is made in step S13, and the process ends. If the intersection angle is within the reference angle range, a positive determination is made in step S13, and the process proceeds to step S14.

[0032] The fixation area setting unit 44 sets fixation areas (step S14). In the example shown in Fig. 5, three fixation areas R1 to R3 are set.

[0033] Specifically, the gaze area setting unit 44 sets the size of the gaze area. The gaze area is a rectangular area with a depth direction parallel to the vehicle's traveling direction and a width direction perpendicular to the vehicle's traveling direction. Note that the width direction does not need to be strictly perpendicular to the vehicle's traveling direction, and may intersect within a range that can be considered perpendicular.

[0034] The depth direction size of the gaze area is set to a size such that the white lines in the image are included in one gaze area. The gaze area setting unit 44 estimates the thickness of the white lines in the image and sets the depth direction size of the gaze area based on the estimated white line thickness. For example, the thickness of the white lines in the image can be estimated from information such as the distance to the right-turn intersection 100 calculated by the direction determination unit 43, the installation height of the imaging device 10, and the standardized design value of the white line thickness. Then, the gaze area setting unit 44 sets the depth direction size of the gaze area to a value obtained by multiplying the estimated white line thickness by four.

[0035] The width direction size of the gaze area is set based on the intersection angle calculated by the direction determination unit 43. Specifically, if the intersection angle is a right angle (90°), the gaze area setting unit 44 sets the width direction size of the gaze area so that it is the same as the width of the image. Furthermore, the gaze area setting unit 44 sets the width direction size of the gaze area to be smaller than the width of the image the smaller the intersection angle is than 90°, and the larger the intersection angle is than 90°. In other words, the width direction size of the gaze area is set to be larger the closer the intersection angle is to a right angle.

[0036] The fixation area setting unit 44 sets the fixation area so that the center of the image in the horizontal direction becomes the center of the fixation area in the width direction.

[0037] It is not possible to know in advance which part of the image contains the white line of the intersecting road 110 that the vehicle will enter after turning right at the right-turn intersection 100. Therefore, the gaze area setting unit 44 sets multiple gaze areas so that the white line on the image is included in the gaze area. In this case, the gaze area setting unit 44 sets multiple gaze areas so that the entire road after the right turn is covered. The imaging device 10 is mounted facing forward without any depression angle. In the image obtained by the imaging device 10, the road surface appears below the vertical center of the image. Therefore, the gaze area setting unit 44 sets multiple gaze areas so that they are aligned vertically in the image within a range from the vertical center to the bottom of the image. In this case, the gaze area setting unit 44 may tilt the gaze areas according to the vertical position of the image in which each gaze area is set so that the gaze areas are parallel to the intersecting road 110.

[0038] The feature point movement amount measurement unit 42 measures the amount of movement of the feature points (step S15). Specifically, the feature point detection unit 41 detects feature points by processing the image. For example, edges of pedestrian crossings, edges of stop lines, etc. are detected as feature points. Six feature points P1 to P6 are illustrated in FIG. 5.

[0039] As the vehicle moves between frames, the feature points on the images also move from frame to frame. The feature point movement amount measurement unit 42 tracks the feature points detected by the feature point detection unit 41 in multiple frames of images and detects the amount of movement of the feature points in the multiple frames of images. When an image contains multiple feature points, the feature point movement amount measurement unit 42 detects the amount of movement for each feature point. By having the imaging device 10 output images at a high frame rate, the feature point movement amount measurement unit 42 can track the feature points with high accuracy.

[0040] As the vehicle moves from moment to moment, the position of the white line displayed in the images of the multiple frames also changes. Therefore, the correction unit 46 corrects the image shift of the gaze area in the images of the multiple frames (step S16).

[0041] Specifically, the correction unit 46 measures the amount of movement of the vehicle in the traveling direction component of the region of interest in the images of multiple frames, based on the amount of movement of the feature points in the images of multiple frames measured by the feature point movement amount measurement unit 42. The method for measuring the amount of movement performed by the correction unit 46 needs to be performed in units of sub-pixels, which is finer than in units of pixels. For example, the correction unit 46 measures the amount of movement of the image in the traveling direction component of the vehicle by a calculation method using normalized cross-correlation.

[0042] The correction unit 46 performs correction by shifting the image of the gaze area in a direction that cancels out the measured amount of movement of the gaze area in the vehicle's traveling direction component. Points on a white line extending perpendicular to the vehicle's traveling direction will have the same amount of movement of the image in the vehicle's traveling direction component. Therefore, the correction unit 46 performs correction based on feature points on an extension of the white line. In other words, the correction unit 46 performs correction based on feature points included in the gaze area among the feature points of the road surface extracted from the image.

[0043] As described above, since multiple gaze areas are set in the image, the correction unit 46 corrects the image shift of each gaze area for each gaze area. That is, the correction unit 46 detects the amount of image movement in the vehicle's traveling direction component in multiple frames of images for each gaze area to be corrected. Then, the correction unit 46 corrects the image shift of the gaze area based on feature points included in the gaze area to be corrected.

[0044] The image integration unit 47 generates a super-resolution image by integrating the corrected images of multiple frames (step S17). A super-resolution image is generated for each gaze area. This super-resolution process generates a super-resolution image in which the edges of the white lines at the intersection 110 are emphasized.

[0045] The structure detection unit 48 detects the positions and directions of white lines in the image based on the super-resolution image (step S18). The white line detection may be performed using a line detection method such as a Hough transform, or may be performed using segmentation based on machine learning.

[0046] As described above, according to the image processing method of this embodiment, a gaze area having a certain range is set in the image, so that a gaze area can be set even for white lines that are difficult to see in the distance. When creating a super-resolution image, it is necessary to refer to the extent to which the white line has moved on the image. At this time, by focusing on the image within the gaze area, it is possible to refer to the amount of movement in the same direction of travel as the white line. This makes it possible to perform super-resolution processing on the gaze area. As long as the gaze area includes the white line, super-resolution processing can be performed on the edge in the extension direction of the white line. This allows for accurate white line detection. Furthermore, the image processing device 1 according to this embodiment can also achieve the same effects as the image processing method.

[0047] According to the image processing method of this embodiment, the amount of movement of the vehicle in the traveling direction of the image in the gaze area is measured using characteristic points on the road surface that exist in the gaze area. With this method, by focusing on the characteristic points in the gaze area, it is possible to refer to the amount of movement in the traveling direction component, which is the same as that of the white line. This makes it possible to perform super-resolution processing on the gaze area.

[0048] According to the image processing method of this embodiment, multiple fixation areas are set so as to be aligned vertically across the image, which allows the fixation areas to be set so as to cover all the white lines on the image.

[0049] Furthermore, for each fixation area, the amount of image movement in the vehicle's traveling direction component in the fixation area is measured using feature points present in the fixation area. The amount of image movement in the fixation area varies depending on the vertical position of the image. By using feature points present in the fixation area to be processed, the amount of image movement in each fixation area can be measured with high accuracy.

[0050] In this embodiment, when the white line that is the subject of the super-resolution image is estimated to extend in a direction that intersects with the vehicle's traveling direction, a process for setting a gaze area is performed. This method allows for reference of the amount of movement in the same traveling direction component as the white line by focusing on a feature point within the gaze area. This allows for super-resolution processing to be performed on the gaze area.

[0051] In this embodiment, when a vehicle turns right at an intersection ahead of the vehicle, a process for setting a fixation area is performed before the vehicle turns right or left at the intersection. According to this method, by focusing on feature points within the fixation area, it is possible to refer to the amount of movement in the same traveling direction component as the white line. This allows super-resolution processing to be performed on the fixation area.

[0052] In this embodiment, when the distance from the vehicle to the intersection is within a predetermined distance, a process of setting a gaze area is performed. With this method, the vehicle approaches the intersection and the characteristic points on the road surface can be recognized on the image, so the amount of movement of the characteristic points can be accurately detected.

[0053] In this embodiment, the gaze area is set based on the distance from the vehicle to the intersection and the angle between the intersecting roads at the intersection. That is, the width of the gaze area is set to be larger as the angle between the intersecting roads at the intersection approaches a right angle. Similarly, the depth of the gaze area is set based on the distance from the vehicle to the intersection and the size of the road structure. This method allows the gaze area to be set so that the white lines are reliably included.

[0054] In this embodiment, the controller 40 may perform the process of setting the gaze area when it is estimated that a pedestrian crossing is present at the intersection. Alternatively, the controller 40 may perform the process of setting the gaze area when it is estimated that a stop line is present at the intersection. This method makes it possible to appropriately detect characteristic points on the road surface. This makes it possible to effectively refer to the amount of movement of the same traveling direction component as the white line.

[0055] Although the present embodiment is based on the assumption that a vehicle turns right at an intersection, the present embodiment may also be based on the assumption that a vehicle turns left at an intersection. Also, although the present embodiment is based on the assumption that a vehicle drives on the left side of the road, the present invention may also be based on the assumption that a vehicle drives on the right side of the road.

[0056] In addition, in the present embodiment, a white line is used as an example of a road structure, but the road structure may be a step, a curb, a median strip, or the like that indicates a road boundary other than a white line.

[0057] The contents of the present disclosure have been described above in accordance with the embodiments, but the present disclosure is not limited to these descriptions and various modifications and improvements are possible.

[0058] REFERENCE SIGNS LIST 1 Image processing device 10 Imaging device 20 Sensor 30 Storage device 40 Controller 41 Feature point detection unit 42 Feature point movement amount measurement unit 43 Direction determination unit 44 Gaze area setting unit 45 Super-resolution processing unit 46 Correction unit 47 Image integration unit 48 Structure detection unit

Claims

1. An image processing method executed by a controller that uses multiple frames of images captured in time series by an imaging device mounted on a vehicle to generate a super-resolution image with a higher resolution than the images generated by the imaging device, the image processing method comprising the steps of: setting a gaze area in the images of the multiple frames, the depth direction of which is parallel to the traveling direction of the vehicle and the width direction of which intersects with the traveling direction of the vehicle; correcting a shift of the gaze area in the images of the multiple frames based on an amount of movement of the gaze area in the traveling direction component of the vehicle in the images of the multiple frames; and generating the super-resolution image based on the images of the multiple frames in which the shift of the gaze area has been corrected.

2. The image processing method according to claim 1, further comprising the steps of: detecting characteristic points on the road surface in the images of the multiple frames; and measuring the amount of movement of the vehicle in the gaze area in the direction of travel of the image of the multiple frames using the characteristic points on the road surface that exist within the gaze area.

3. The image processing method according to claim 2, further comprising: setting a plurality of gaze areas aligned vertically across the image; and measuring, for each gaze area, the amount of image movement in the vehicle's travel direction component in the gaze area using characteristic points on the road surface that are present within the gaze area.

4. An image processing method as claimed in any one of claims 1 to 3, wherein the gaze area is set when a road structure that is the subject of the super-resolution image is estimated to extend in a direction that intersects with the vehicle's traveling direction.

5. The image processing method according to claim 4, wherein, when the vehicle turns right or left at an intersection ahead of the vehicle, the fixation area is set before the vehicle turns right or left at the intersection.

6. The image processing method according to claim 5, wherein the fixation area is set when the distance from the vehicle to the intersection is within a predetermined determination distance.

7. The image processing method according to claim 5, wherein the fixation area is set based on a distance from the vehicle to the intersection and an angle between the vehicle and a cross road at the intersection.

8. The image processing method according to claim 7, wherein the width of the fixation area is set to be larger as the angle between the intersecting roads at the intersection approaches a right angle.

9. The image processing method according to claim 7 or 8, wherein the depth direction size of the fixation area is set based on the distance from the vehicle to the intersection and the size of the road structure.

10. The image processing method according to claim 5, further comprising the step of setting the gaze area when it is estimated that there is a pedestrian crossing at the intersection.

11. The image processing method according to claim 5, further comprising the step of setting the gaze area when it is estimated that the intersection has a stop line.

12. An image processing device having a controller that uses multiple frames of images captured in time series by an imaging device mounted on a vehicle to generate a super-resolution image having a higher resolution than an image generated by the imaging device, wherein the controller: sets a gaze area in the multiple frame images whose depth direction is parallel to the vehicle's traveling direction and whose width direction intersects with the vehicle's traveling direction; corrects a shift of the gaze area in the multiple frame images based on an amount of movement of the gaze area in the vehicle's traveling direction component in the multiple frame images; and generates the super-resolution image based on the multiple frame images in which the shift of the gaze area has been corrected.

Images