Image processing apparatus, image processing program, and vehicle

Abstract

To provide an image processing apparatus, an image processing program, and a vehicle configured to improve the accuracy of estimating camera orientation.SOLUTION: An image processing apparatus includes a controller. The controller sets a plurality of regions of interest in an image captured by a camera mounted on a vehicle, and estimates camera orientation based on optical flow of feature points extracted from the regions of interest. The controller sets the regions of interest so that the lengths of the regions of interest in the vehicle-width direction may be shorter as the distance from the vehicle is longer when projecting the regions of interest on a road plane.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing program, and a vehicle. [Background technology]

[0002] Conventionally, a technique has been known in which multiple regions of interest (Regions of Interest) are set in an image captured by an in-vehicle camera, and the current posture (mounting position) of the in-vehicle camera is estimated based on the optical flow of feature points extracted from each region of interest (see, for example, Patent Document 1).

[0003] Furthermore, Patent Document 1 discloses a technology in which statistical information of optical flow is calculated for each of a plurality of regions of interest, and regions of interest whose statistical information does not satisfy certain conditions are excluded from the camera posture estimation process on the assumption that they contain unnecessary objects such as other vehicles. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-86257 Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional technologies have room for further improvement in terms of increasing the accuracy of camera posture estimation. For example, in the invention described in Patent Document 1, the further away from the host vehicle is, the more likely it is that unwanted objects, such as other vehicles, will be included in the region of interest, and the number of regions of interest that are excluded increases. In this case, the number of regions of interest used in the camera posture estimation process decreases, which could result in a decrease in the accuracy of camera posture estimation.

[0006] The present invention has been made in view of the above, and has an object to provide an image processing device, an image processing program, and a vehicle that can improve the accuracy of estimating the camera attitude. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, an image processing device according to the present invention includes a controller. The controller sets a plurality of attention areas in an image captured by a camera mounted on a vehicle, and estimates the posture of the camera based on the optical flow of feature points extracted from the plurality of attention areas. The controller sets the plurality of attention areas so that, when projected onto a road surface, the lengths of the attention areas in the vehicle width direction become shorter as they move away from the vehicle. [Effects of the Invention]

[0008] According to the present invention, the width of the attention area is narrowed as the distance from the vehicle increases, making it less likely that unnecessary objects will be included in the attention area. This reduces the number of attention areas excluded from the camera posture estimation process, thereby improving the accuracy of camera posture estimation. [Brief explanation of the drawings]

[0009] [Figure 1A] FIG. 1A is a diagram showing a posture estimation process of an image processing device according to an embodiment. [Figure 1B] FIG. 1B is a diagram for explaining the process of setting the region of interest. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of the image processing device. [Figure 3] FIG. 3 is a diagram for explaining the process of setting the attention area. [Figure 4] FIG. 4 is a flowchart showing the procedure of the process executed by the image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] An image processing device, an image processing program, and a vehicle according to embodiments will be described in detail below with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below. In addition, the term "predetermined" below can be read as "predetermined."

[0011] 1A is a diagram showing the attitude estimation process of the image processing device 1 according to the embodiment. The image processing device 1 according to the embodiment estimates the current attitude of a camera 10 (see FIG. 1B) mounted on a vehicle based on an image captured by the camera 10.

[0012] Specifically, when an image captured by the camera 10 is input, the image processing device 1 sequentially performs the following processes on the captured image: setting a region of interest, extracting feature points, calculating optical flow, excluding the region of interest, and estimating the camera posture.

[0013] The region of interest setting process is a process of setting a region of interest in a captured image, which will be an area from which feature points will be extracted in a later stage. In the present disclosure, multiple regions of interest are set in a captured image. The regions of interest are set mainly in areas where the road surface is reflected, such as the central area of ​​the captured image. In other words, the region of interest is set to an area excluding areas other than the road surface, such as the vehicle body and the sky.

[0014] The feature point extraction process is a process of extracting corner feature points from a set region of interest. Specifically, the image processing device 1 extracts pixels having an edge strength equal to or greater than a threshold as feature points based on the pixel values ​​of each pixel in the region of interest. The image processing device 1 also calculates a corner degree, which indicates the likelihood of the extracted feature points being corners, and extracts feature points whose corner degree is equal to or greater than the corner threshold as corner feature points that correspond to corners. In other words, corner feature points are points (pixels) that indicate the corners of lane markings such as white lines painted on the road surface. The corner degree can be calculated using a known detection method such as a Harris operator or a KLT (Kanade-Lucas-Tomasi) tracker.

[0015] The optical flow calculation process is a process of calculating an optical flow for each extracted corner feature point. Specifically, the image processing device 1 calculates, as the optical flow, a motion vector indicating the movement (position change) of the corner feature point between two captured images captured with a predetermined time interval between them. The predetermined time interval is, for example, the interval of the frame rate of the camera 10.

[0016] The region-of-interest exclusion process is a process of determining (excluding) regions of interest that will not be used in the subsequent camera posture estimation process, based on the extracted optical flow. Specifically, the image processing device 1 first generates an optical flow of road surface vectors by projecting the calculated optical flow onto a road surface plane corresponding to the road surface. Then, the image processing device 1 calculates the variance (an example of statistical information) of the lengths (magnitudes) of the road surface vectors for each region of interest, and excludes regions of interest where the variance is equal to or greater than a threshold. In other words, the image processing device 1 excludes regions of interest where there is a high probability that the optical flow of feature points extracted from objects other than lane markings (such as other vehicles) can be obtained. Note that the statistical information is not limited to the variance, and may be the average length (magnitude) of the road surface vectors. In this case, the image processing device 1 excludes regions of interest where the average value is greater than or equal to a threshold from a value corresponding to the travel distance.

[0017] The camera attitude estimation process is a process of estimating the current attitude of the camera 10 based on the calculated optical flow. Specifically, first, the image processing device 1 extracts a combination of two optical flows (road surface vectors) that are parallel to each other in the area of ​​interest that was not excluded in the previous exclusion process. More specifically, the image processing device 1 extracts a pair of optical flows from each of two different areas of interest. Then, the image processing device 1 estimates the attitude of the camera 10 based on the extracted pair of optical flows. Specifically, the image processing device 1 estimates the PAN, TILT, and ROLL angles of the camera 10 based on the two paired optical flows. Then, the image processing device 1 generates a histogram for each of the PAN, TILT, and ROLL angles, with the estimated values ​​(angle values) of the attitude estimated for each pair of optical flows as classes. Then, the image processing device 1 determines the estimated values ​​of the PAN, TILT, and ROLL, which are the camera attitudes, based on the histogram. For example, the image processing device 1 determines the median value in the histogram as the estimated value of PAN, TILT, and ROLL.

[0018] The image processing device 1 may use the determined estimated value of the camera attitude for further processing such as determining the positional deviation of the camera 10, or may output it to the outside.

[0019] In the present disclosure, in the above-described region-of-interest setting process, the size of the region of interest is devised to reduce the number of regions of interest excluded in the subsequent region-of-interest exclusion process. This point will be described with reference to FIG. 1B.

[0020] 1B is a diagram for explaining the process of setting an area of ​​interest. FIG. 1B shows a situation in which a host vehicle MC and another vehicle OC located in front of the host vehicle MC are traveling on a road sandwiched between dividing lines 200 such as dashed white lines. Also, in FIG. 1B, the camera 10 is shown as a camera that captures an image in front of the host vehicle MC, but it may also be a camera that captures an image behind the host vehicle MC.

[0021] Here, Fig. 1B shows a plurality of attention areas Ra, Rb seen from a bird's-eye view. That is, Fig. 1B shows a case where a plurality of attention areas Ra, Rb set in a captured image are projected onto a road surface plane. Note that, for convenience of explanation, Fig. 1B merely shows a diagram in which a plurality of attention areas Ra, Rb are projected onto a road surface plane. That is, the image processing device 1 sets a plurality of attention areas Ra, Rb in a captured image without performing the process of projecting the plurality of attention areas Ra, Rb onto a road surface plane in the process shown in Fig. 1A.

[0022] As shown in Figure 1B, the image processing device 1 sets multiple attention areas Ra, Rb so that when the multiple attention areas Ra, Rb are projected onto a road surface plane, the width (length in the vehicle width direction) of the attention areas Ra, Rb becomes shorter the farther away from the vehicle MC.

[0023] Specifically, the image processing device 1 sets the two attention areas Ra closer to the vehicle MC to a first width Wa, and sets the three attention areas Rb farther from the vehicle MC to a second width Wb that is shorter than the first width Wa.

[0024] 1A, the central attention region Rb that includes the corner feature point 100 corresponding to the other vehicle OC is excluded, while the attention regions Rb on both sides that include the corner feature point 100 of the lane marking 200 are not excluded. On the other hand, if the attention region Rb has the same width as the first width Wa of the attention region Ra (in this case, there will be two attention regions Rb, just like the attention region Ra), the corner feature point 100 corresponding to the other vehicle OC will be included in both of the two attention regions Rb and will be excluded.

[0025] In this way, according to the image processing device 1 of the embodiment, by narrowing the widths of the attention areas Ra, Rb as the distance from the host vehicle MC increases, it becomes less likely that unnecessary objects (other vehicles OC) will be included in the attention areas Ra, Rb. This allows the image processing device 1 to reduce the number of attention areas Ra, Rb excluded from the camera attitude estimation process, thereby improving the accuracy of camera attitude estimation.

[0026] 1B, the image processing device 1 sets multiple attention areas Ra, Rb so that the number of attention areas lined up in the vehicle width direction increases as the area is farther away from the host vehicle MC. In the example shown in FIG. 1B, two attention areas Ra are set closer to the host vehicle MC, and three attention areas Rb are set farther from the host vehicle MC. This ensures that a certain number or more of the remaining attention areas Rb remain, even if another vehicle is captured in one of the multiple attention areas Rb and is excluded from the estimation process.

[0027] The method of setting the attention regions Ra and Rb shown in FIG. 1B is an example, and a more detailed setting method will be described later with reference to FIG.

[0028] Next, an example of the configuration of the image processing device 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the image processing device 1. As shown in Fig. 2, the image processing device 1 is connected to a camera 10.

[0029] The camera 10 is the above-mentioned in-vehicle camera, and is fixedly disposed on the vehicle C. The camera 10 is connected to the image processing device 1 by wire or wirelessly, and outputs captured images to the image processing device 1. When the camera 10 is provided for the purpose of monitoring the situation around the vehicle C, it is preferably configured with four cameras 10: a front camera, a rear camera, a left side camera, and a right side camera. The front camera is a camera that captures images in front of the vehicle C. The rear camera is a camera that captures images behind the vehicle C. The left side camera is a camera that captures images to the left of the vehicle C. The right side camera is a camera that captures images to the right of the vehicle C. These four cameras 10 are configured using, for example, fisheye lenses, and have a horizontal angle of view of 180 degrees or more. This allows the four cameras 10 to capture images of the entire surroundings of the vehicle C in the horizontal direction.

[0030] As shown in FIG. 2, the image processing device 1 includes a controller 2 and a storage unit 3.

[0031] The controller 2 includes a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and various other circuits. The controller 2 controls the overall operation of the image processing device 1 by having the CPU execute a program stored in the ROM using the RAM as a working area. Note that the controller 2 may be partially or entirely configured with hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0032] The storage unit 3 is, for example, a RAM or a data flash. The storage unit 3 can store information about various programs, etc. The image processing device 1 may also acquire the above-mentioned programs and various information via another computer or portable recording medium connected via a wired or wireless network.

[0033] When the controller 2 receives an image captured by the camera 10, it sequentially performs the following processes on the captured image, as shown in Figure 1A: setting an area of ​​interest, extracting feature points, calculating optical flow, excluding the area of ​​interest, and estimating the camera attitude.

[0034] Here, the process of setting the attention area will be described in detail with reference to Fig. 3. Fig. 3 is a diagram for explaining the process of setting the attention area. In Fig. 3, the lower side of the paper is the side of the host vehicle MC. Also, Fig. 3 shows an example in which five attention areas Ra, Rb are set, but the number of attention areas may be less than four, or may be six or more.

[0035] As shown in Fig. 3, the multiple attention areas Ra, Rb are set as areas obtained by dividing a rectangular overall area FR on the road surface plane in the vehicle length direction and the vehicle width direction. In Fig. 3, the length H of the overall area FR in the vehicle length direction is referred to as height H, and the length W in the vehicle width direction is referred to as width W.

[0036] In Fig. 2, the controller 2 divides the height H of the entire region FR into a region of interest Ra with a height Ha and a region of interest Rb with a height Hb. Note that, although Fig. 2 shows the heights Ha and Hb as having the same length (Ha = Hb), they may have different lengths. Also, Fig. 2 shows an example in which the height H is divided into two, but it may be divided into three or more.

[0037] Furthermore, the controller 2 divides the width W of the entire region FR into attention regions Ra each having a width Wa. Note that while FIG. 2 shows an example in which the width W of the entire region FR is equally divided by the width Wa, the two attention regions Ra may be divided so that each has a different width. Furthermore, the controller 2 divides the width W of the entire region FR into attention regions Rb each having a width Wb. Note that while FIG. 2 shows an example in which the width W of the entire region FR is equally divided by the width Wb, the three attention regions Rb may be divided so that each has a different width.

[0038] 2, the width Wb is shorter than the width Wa. That is, the controller 2 sets the plurality of attention areas Ra, Rb so that the number of divisions in the vehicle width direction increases as the distance from the host vehicle MC in the vehicle length direction increases. As a result, even if, for example, another vehicle is captured in one of the plurality of attention areas Rb and is excluded from the estimation process, a certain number or more of the remaining attention areas Rb can be ensured.

[0039] Furthermore, it is preferable that the width W of the rectangular entire region FR is set to a value slightly longer than the general road width, for example. In this way, the controller 2 divides the rectangular entire region R to set multiple regions of interest, thereby enabling highly accurate extraction of corner feature points corresponding to the lane markings 200 (see FIG. 1B) on both ends of the road within the entire region R that is aligned with the shape of the road surface (road width).

[0040] Furthermore, it is preferable that the widths Wa and Wb of the attention regions Ra and Rb are longer than the width of the demarcation line 200. This allows one attention region to include corner feature points from both corners in the width direction of the demarcation line 200. As a result, more corner feature points are included per attention region, and the total number of attention regions is reduced, thereby reducing the number of attention regions to be processed and thereby reducing the processing load.

[0041] Although the rectangular overall region FR and region of interest are shown in FIG. 3, they are not limited to rectangles and may be circular, triangular, trapezoidal, or polygonal with five or more sides.

[0042] Next, the processing procedure of the image processing device 1 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the processing procedure executed by the image processing device 1. The processing shown in Fig. 5 is repeatedly executed at regular intervals from when the IG of the vehicle C is turned on until it is turned off.

[0043] As shown in FIG. 4, the controller 2 acquires a captured image from the camera 10 (step S101).

[0044] Next, the controller 2 sets a plurality of attention areas in the captured image (step S102). Specifically, the controller 2 sets a plurality of attention areas by dividing the entire rectangular area FR on the road surface plane in both the vehicle length direction and the vehicle width direction. More specifically, the controller 2 sets the plurality of attention areas so that, when the plurality of attention areas are projected onto the road surface plane, the length of the attention area in the vehicle width direction decreases with increasing distance from the host vehicle MC.

[0045] Next, the controller 2 extracts corner feature points from the set plurality of regions of interest (step S103). Specifically, the controller 2 extracts pixels having an edge strength equal to or greater than a threshold as feature points based on the pixel values ​​of each pixel in the regions of interest. The controller 2 also calculates a corner degree indicating the likelihood of the extracted feature points being corners, and extracts feature points whose corner degree is equal to or greater than the corner threshold as corner feature points that correspond to corners. The corner degree can be calculated using a known detection method such as the Harris operator or the KLT tracker.

[0046] Next, the controller 2 calculates an optical flow for each of the extracted corner feature points (step S104). Specifically, the controller 2 calculates, as the optical flow, a motion vector that indicates the movement (position change) of the corner feature point between two captured images captured at a predetermined time interval.

[0047] Next, the controller 2 calculates statistical information of the optical flow for each region of interest (step S105). Specifically, the controller 2 generates an optical flow of road surface vectors by projecting the calculated optical flow onto a road surface plane corresponding to the road surface. Then, the controller 2 calculates the variance (an example of statistical information) of the length (magnitude) of the road surface vector for each region of interest.

[0048] Next, the controller 2 selects an area of ​​interest whose calculated statistical information satisfies the condition (step S106). In other words, the controller 2 excludes areas of interest whose statistical information does not satisfy the condition. Specifically, the controller 2 selects areas of interest whose calculated variance as statistical information is less than a threshold, and excludes areas of interest whose variance is equal to or greater than the threshold.

[0049] Next, the controller 2 estimates the current attitude of the camera 10 based on the optical flow of the selected region of interest (step S107). Specifically, first, the controller 2 extracts a combination of two optical flows (road surface vectors) that are parallel to each other in the selected region of interest. More specifically, the controller 2 extracts a pair of optical flows for each of two different regions of interest. Then, the controller 2 estimates the attitude of the camera 10 based on the extracted pair of optical flows. Specifically, the controller 2 estimates the PAN, TILT, and ROLL angles of the camera 10 based on the pair of two optical flows. Then, the controller 2 generates a histogram for each of PAN, TILT, and ROLL, in which the estimated values ​​(angle values) of the attitude estimated for each pair of optical flows are classified. Then, the controller 2 determines the estimated values ​​of PAN, TILT, and ROLL, which are the camera attitudes, based on the histogram. For example, the controller 2 determines the median value in the histogram as the estimated value of PAN, TILT, and ROLL.

[0050] As described above, the image processing device 1 according to the embodiment includes the controller 2. The controller 2 sets a plurality of attention areas in a captured image captured by the camera 10 mounted on the host vehicle MC, and estimates the attitude of the camera 10 based on the optical flow of feature points extracted from the plurality of attention areas. The controller 2 sets the plurality of attention areas such that, when the plurality of attention areas are projected onto a road surface plane, the length of the attention area in the vehicle width direction decreases with increasing distance from the host vehicle MC.

[0051] According to the present invention, by narrowing the width of the attention area as it becomes farther away from the host vehicle MC, it becomes less likely that an unnecessary object (another vehicle OC) will be included in the attention area. This reduces the number of attention areas excluded from the camera posture estimation process, thereby improving the accuracy of camera posture estimation.

[0052] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0053] 1. Image processing device 2 Controller 3 Storage section 10 Camera 100 corner feature points 200 Lot Line C vehicle FR entire area MC Vehicle OC Other vehicles R whole area Ra, Rb attention area

Claims

1. a controller that sets a plurality of attention areas in an image captured by a camera mounted on the vehicle, and estimates the attitude of the camera based on the optical flow of feature points extracted from the plurality of attention areas; The controller The plurality of attention areas are set so that, when the plurality of attention areas are projected onto a road surface plane, the lengths of the attention areas in the vehicle width direction become shorter as the distance from the host vehicle increases. Image processing device.

2. The controller The plurality of attention areas are set so that, when the plurality of attention areas are projected onto a road surface plane, the number of the attention areas aligned in the vehicle width direction increases as the distance from the host vehicle increases. The image processing device according to claim 1 .

3. The plurality of regions of interest include: A region obtained by dividing the entire rectangular region on the road surface plane in each of the vehicle length direction and the vehicle width direction, The controller The plurality of attention areas are set so that the number of divisions in the vehicle width direction increases as the area of ​​interest is farther away from the host vehicle in the vehicle length direction. The image processing device according to claim 1 .

4. The length of the attention area in the vehicle width direction is Longer than the width of the lot line The image processing device according to claim 1 .

5. An image processing program that causes a computer to execute a procedure of setting a plurality of attention areas in an image captured by a camera mounted on a vehicle, and estimating an attitude of the camera based on an optical flow of feature points extracted from the plurality of attention areas, The procedure comprises: The plurality of attention areas are set so that, when the plurality of attention areas are projected onto a road surface plane, the lengths of the attention areas in the vehicle width direction become shorter as the distance from the host vehicle increases. Image processing program.

6. a camera that captures images of the surroundings of the vehicle; an image processing device that sets a plurality of regions of interest in an image captured by the camera and estimates the posture of the camera based on the optical flow of feature points extracted from the plurality of regions of interest; Equipped with The image processing device includes: The plurality of attention areas are set so that, when the plurality of attention areas are projected onto a road surface plane, the lengths of the attention areas in the vehicle width direction become shorter as the distance from the host vehicle increases. vehicle.

Images