Image processing device, image processing method, and program

Abstract

The goal is to improve the feature point matching process. [Solution] The system includes an image acquisition means for acquiring a first image and a second image captured by an imaging means having lens distortion; a lens distortion correction information storage means for storing lens distortion correction information for correcting the lens distortion of the imaging means; and a feature point matching means for performing feature point matching between the feature points of the first image and the feature points of the second image in a feature point matching region corresponding to the lens distortion correction information, using feature descriptors of the feature points of the first image and the second image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Techniques for estimating the self-position using a stationary region of an image in automatic driving, industrial robots, etc. are known. In performing self-position estimation, a matching technique between stationary regions in images acquired at different times is used to acquire the moving direction and moving amount of the stationary region.

[0003] At that time, a wide-angle lens is often used for the lens mounted on a vehicle or a robot in order to image a wide range. This wide-angle lens generally has a convex lens shape and is designed such that closer objects are imaged at a closer distance and farther objects are imaged at a farther distance. Therefore, in a wide-angle lens, the distortion aberration, in which the image is distorted in a barrel shape, becomes larger compared to a general single-focus lens.

[0004] When performing matching processing on an image with large distortion aberration, techniques for performing matching between frame images of the entire captured image and techniques for performing matching in a fixed region based on the maximum movement amount of the camera are known. In such matching processing including distortion aberration, it is conceivable to limit the matching region as described above to reduce the processing amount of matching while maintaining the matching accuracy.

[0005] For example, Patent Document 1 describes extracting feature amounts from an image including distortion aberration. Further, Patent Document 2 describes specifying an undistorted region based on distortion aberration information and determining a region for extracting feature amounts using the undistorted region.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] The method described in Patent Document 1 detects features from the entire captured image and performs recognition processing of a specific object after distortion correction of the detected features. Therefore, in systems requiring real-time performance, the load of distortion correction on the entire image becomes high. Furthermore, in the method described in Patent Document 2, as distortion aberration increases, the non-distorted region narrows, and the region from which features can be extracted narrows, resulting in a decrease in the accuracy of the recognition process. To improve the accuracy of self-localization estimation of vehicles and robots, it is necessary to extract features from the entire captured image including the distorted region and efficiently match features between consecutive frame images. Therefore, there has been room for improvement in the load and accuracy of feature point matching processing in the past.

[0008] This invention was made to solve the above-mentioned problems and aims to improve the feature point matching process. [Means for solving the problem]

[0009] An image processing apparatus according to one embodiment of the present invention comprises: an image acquisition means for acquiring a first image and a second image captured by an imaging means having lens distortion; a lens distortion correction information storage means for storing lens distortion correction information for correcting the lens distortion of the imaging means; a feature point extraction means for extracting one or more feature points from the first image and the second image; a feature descriptor generation means for generating feature descriptors that indicate the feature quantities of the feature points of the first image and the second image; a feature point matching region initialization means for calculating a feature point matching region in the second image corresponding to a feature point in the first image using the lens distortion correction information before imaging by the imaging means begins; a feature point matching region determination means for obtaining a feature point matching region to be used in correspondence with a feature point in the first image from the feature point matching region calculated by the feature point matching region initialization means; and a feature point matching means for performing feature point matching between a feature point in the first image and a feature point in the second image in the feature point matching region obtained by the feature point matching region determination means, using the feature descriptors of the feature points of the first image and the feature points of the second image. [Effects of the Invention]

[0010] According to the present invention, the feature point matching process can be improved. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing the overall configuration of a camera device according to Embodiment 1 of the present invention. [Figure 2] This is a diagram illustrating lens distortion. [Figure 3] This is a diagram illustrating lens distortion. [Figure 4] This figure shows the feature point matching region information for Embodiment 1. [Figure 5] This is a flowchart showing the initialization process of the feature point matching region in Embodiment 1. [Figure 6] This is a flowchart showing the process for determining the feature point matching region in Embodiment 1. [Figure 7] It is a block diagram showing the overall configuration of the camera device according to Embodiment 2 of the present invention. [Figure 8] It is a diagram showing the moving direction of feature points of a stationary object according to the steering angle. [Figure 9] It is an information structure diagram of the moving direction of feature points of a stationary object according to the steering angle. [Figure 10] It is a diagram showing the correction procedure of the motion vector of feature points based on the steering angle and vehicle speed. [Figure 11] It is a diagram showing the feature point matching area information based on the steering angle and vehicle speed in Embodiment 2. [Figure 12] It is a diagram showing the feature point matching area information based on the steering angle and vehicle speed in Embodiment 2. [Figure 13] It is a flowchart showing the initialization process of the feature point matching area in Embodiment 2. [Figure 14] It is a flowchart showing the determination process of the feature point matching area in Embodiment 2. [Figure 15] It is a flowchart for acquiring camera movement information in Embodiment 2. [Figure 16] It is a diagram showing the priority of the speed range of the feature point matching area in Embodiment 2. [Figure 17] It is a block diagram showing the overall configuration of the camera device according to Embodiment 3 of the present invention.

Modes for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. Also, the following embodiments do not limit the present invention according to the claims, and not all combinations of the features described in the present embodiments are essential for the solution means of the present invention. In each figure, the same members or elements are given the same reference numerals, and duplicate explanations are omitted or simplified.

[0013] <Embodiment 1> Embodiment 1 of the present invention will be described using a diagram. Figure 1 is a block diagram showing the overall configuration of a camera device 100 according to Embodiment 1 of the present invention. The camera device 100 is an example of an image processing device. The camera device 100 has an imaging unit 101 and a feature point pair generation unit 102. The imaging unit 101 has a lens 103, an image sensor 104, and an image transmission unit 105. The feature point pair generation unit 102 has an image receiving unit 106, an image correction unit 107, a feature point extraction unit 108, a feature descriptor generation unit 109, a feature point matching region determination unit 110, and a feature point matching unit 111. The feature point pair generation unit 102 further has an intermediate data storage unit 112, a feature point matching region storage unit 113, a lens distortion correction information storage unit 114, and a feature point matching region initialization unit 115. The camera device 100 is, for example, placed on a moving object such as a vehicle or a robot, moves with the moving object, and photographs the area around the moving object. Furthermore, the camera device 100 is positioned on a stationary object such as a building and photographs the area around the stationary object.

[0014] The imaging unit 101 captures images of the area in front and generates image data. The feature point pair generation unit 102 receives image data from the imaging unit 101, performs feature point matching processing between consecutive image data, and outputs feature point pair information.

[0015] First, let's explain the imaging unit 101. The lens 103 of the imaging unit 101 is an optical element that forms an image of the subject on the subsequent image sensor 104. The lens type of the lens 103 can be any type, such as a wide-angle lens, zoom lens, prime lens, or fisheye lens. Furthermore, the lens distortion (aberration) of the lens 103 is typically classified into two types: barrel distortion and pincushion distortion. However, other types of distortion are also acceptable, including, for example, a combination of barrel distortion and pincushion distortion such as a cone-shaped distortion.

[0016] The image sensor 104 is an image sensor composed of CMOS (complementary metal-oxide-semiconductor) or CCD (charge-coupled device). The subject image formed on the image sensor 104 via the lens 103 is converted into an electrical signal by the image sensor 104. The image transmission unit 105 transmits the electrical signal acquired from the image sensor 104 as image data to the feature point pair generation unit 102.

[0017] Next, the feature point pair generation unit 102 will be described. The feature point pair generation unit 102 is typically composed of an LSI, a CPU, a program executed by the CPU, a memory device, and various I / O components, but the configuration method is not particularly limited. LSI is an abbreviation for Large Scale Integration. CPU is an abbreviation for Central Processing Unit. I / O is an abbreviation for input-output.

[0018] The lens distortion correction information storage unit 114 is a non-volatile storage area that holds the lens distortion correction information used by the feature point matching area storage unit 113. Here, the lens distortion correction information will be explained using Figures 2, 3, and 4.

[0019] Figure 2(A) shows the image region divided into sections, assuming no lens distortion. The division size is irrelevant here. Figure 2(B) shows the divided region of Figure 2(A) distorted by the barrel-shaped lens distortion of lens 103. Figure 2(C) shows the movement of pixels in Figure 2(A) due to the barrel-shaped lens distortion in Figure 2(B) represented as vectors.

[0020] Figure 3(A) illustrates the movement of the pixel blocks in Figure 2(A) due to barrel-shaped lens distortion. Hereafter, a pixel block consisting of one or more pixels in a rectangular region will be referred to as a feature point region. Figure 3(A) shows that the upper left coordinates P1 and P2 of feature point regions 201 and 202, which are feature point regions in the absence of lens distortion in Figure 2(A), move to feature point coordinates Q1 and Q2 due to the lens distortion in Figure 2(B).

[0021] Here, the motion vector (start point P1, end point Q1) of the feature points in the feature point region 201 due to lens distortion represents the average motion vector of the feature points in the feature point region 201. In this embodiment, when using this motion vector, it is possible to move the start point P1 from the upper left coordinate of the feature point region 201 to any coordinate within the feature point region 201 and use it. The same applies to the motion vector (start point P2, end point Q2) of the feature points in the feature point region 202 due to lens distortion. In the following processing, all feature points in the feature point region are mapped to the left coordinate for processing. This is to reduce the data capacity stored in the feature point matching region storage unit 113 and the lens distortion correction information storage unit 114, which will be described later.

[0022] Figure 3(B) shows the lens distortion correction information stored in the lens distortion correction information storage unit 114. The area 203 stores numerical values ​​representing the width and height of the feature point area in units of pixels. The width and height of this feature point area may be fixed values, or they may be configured to be changeable during imaging by the imaging unit 101.

[0023] Region 204 stores the number of rows, columns, and total number of feature point regions. Specifically, the number of rows is the vertical length of the image data received by the image receiving unit 106 divided by the vertical width of region 203. The number of columns is the horizontal length of the image data received by the image receiving unit 106 divided by the horizontal width of region 203. The total number is the product of the number of rows and the number of columns.

[0024] Region 205 stores the feature point region coordinates P1(x,y) before correction and the feature point coordinates Q1(x,y) after correction due to lens distortion. Similarly, region 206 stores the feature point region coordinates P2(x,y) before correction and the feature point coordinates Q2(x,y) after correction due to lens distortion. In other words, regions 205 and beyond store the pre- and post-correction coordinates (x,y) for the total number of feature point regions in region 204. Here, if the size of the image data changes depending on the operating mode of the lens 103 and the image sensor 104, the lens distortion correction information shown in Figure 3(B) may be stored for each image data size. Alternatively, the lens distortion correction information shown in Figure 3(B) may be stored for the largest image data, and only the effective region of the lens distortion correction information may be used based on the size of the image data.

[0025] Next, the feature point matching region memory unit 113 will be explained using Figure 4. Figure 4(A) shows the feature point matching region before lens distortion correction. The coordinates of the feature point region 301 are represented by P1, and the coordinates of the four corners of the feature point matching region 303 relative to the feature point region 301 are represented by A1-TL, A1-TR, A1-BL, and A1-BR. Here, the coordinates of the four corners coincide with the coordinates of the top left, top right, bottom left, and bottom right of the feature point region, respectively. Also, the coordinates of the feature point region 302 are represented by P2, and the coordinates of the four corners of the feature point matching region 304 relative to the feature point region 302 are represented by A2-TL, A2-TR, A2-BL, and A2-BR.

[0026] The feature point matching region 303 is a square centered on the feature point region 301, and the feature point matching region 304 is only a part of the square region according to the image region. Here, except for cases where the square feature point matching region exceeds the image region, such as the feature point matching region 304, the feature point matching region in Embodiment 1 is fixed. Furthermore, the size of this fixed feature point matching region represents the maximum movement range in the next image data from within the central feature point region. When the camera device 100 is positioned on a moving object such as a vehicle and tracks the feature points of a stationary object in the captured image, the size of the feature point matching region is determined based on the maximum movement speed of the moving object. Also, when the camera device 100 is positioned on a stationary object such as a building and tracks the feature points of a moving object in the captured image, the size of the feature point matching region is determined based on the maximum movement speed of the moving object.

[0027] Figure 4(B) shows the feature point matching region after lens distortion correction. The feature point matching region 303 for the feature point region 301 is transformed using the lens distortion correction information in Figure 3(B) stored in the lens distortion correction information storage unit 114 to obtain the feature point matching region 305. Specifically, the coordinates A1-TL, A1-TR, A1-BL, and A1-BR of the feature point matching region 303 are transformed into the coordinates B1-TL, B1-TR, B1-BL, and B1-BR of the feature point matching region 305, respectively. Similarly, the coordinates A2-TL, A2-TR, A2-BL, and A2-BR of the feature point matching region 304 are transformed into the coordinates B2-TL, B2-TR, B2-BL, and B2-BR of the feature point matching region 306.

[0028] Figure 4(C) shows the feature point matching region information stored in the feature point matching region memory unit 113. Region 307 stores the coordinates P1 of feature point region 301. Region 308 stores the coordinates of the four points of feature point matching region 305. Region 309 stores the coordinates P2 of feature point region 302. Region 310 stores the coordinates of the four points of feature point matching region 306. Similarly, the coordinate information of the four corners of feature point matching regions related to other feature point regions is stored thereafter.

[0029] The feature point matching region information in Figure 4(C) is referenced by the feature point matching region determination unit 110. Specifically, the coordinates Pn of a feature point region are taken as input, the coordinates Pn of a matching feature point region are searched for from the feature point matching region information in Figure 4(C), and the coordinates of the four points of the feature point matching region recorded after the searched coordinates Pn are referenced.

[0030] Figure 5 illustrates the process by which the feature point matching area initialization unit 115 initializes the feature point matching area storage unit 113 before the start of imaging, when the camera device 100 is positioned on a moving object.

[0031] In step S401, the feature point matching area initialization unit 115 obtains the vertical and horizontal dimensions of the image data received by the image receiving unit 106. In step S402, the feature point matching area initialization unit 115 obtains the vertical and horizontal dimensions of the feature point area. In step S403, the feature point matching area initialization unit 115 obtains the maximum movement speed of the camera device 100.

[0032] In step S404, the feature point matching region initialization unit 115 uses the maximum movement speed obtained in step S403 to determine the length of one side of the feature point matching region before correction. Specifically, it determines the length of one side of the square (coordinates An-TL, An-TR, An-BL, An-BR) of the feature point matching region before correction. In step S405, the feature point matching region initialization unit 115 selects one of the feature point region coordinates Pn before correction from the lens distortion correction information in Figure 3(B).

[0033] In step S406, the feature point matching region initialization unit 115 calculates the coordinates of the four corners of the pre-correction feature point matching region using the feature point region coordinates Pn obtained in step S405 and the length of one side of the pre-correction feature point matching region obtained in step S404. The coordinates of the four corners of the pre-correction feature point matching region calculated here are An-TL, An-TR, An-BL, and An-BR.

[0034] In step S407, the feature point matching region initialization unit 115 corrects the coordinates of the four corners of the feature point matching region calculated in step S406 using the lens distortion correction information shown in Figure 3(B). Specifically, it converts the coordinates An-TL, An-TR, An-BL, and An-BR to coordinates Bn-TL, Bn-TR, Bn-BL, and Bn-BR.

[0035] In step S408, the feature point matching region initialization unit 115 stores the coordinates of the four corners of the feature point matching region corrected in step S406 in the feature point matching region storage unit 113. In step S409, the feature point matching region initialization unit 115 determines whether processing for all feature point region coordinates has been completed. If the feature point matching region initialization unit 115 determines that processing for all feature point region coordinates has been completed, the process shown in Figure 5 is terminated. If the feature point matching region initialization unit 115 determines that processing for all feature point region coordinates has not been completed, the process shown in step S405 is executed.

[0036] In cases where lens distortion changes with focal length, such as with a zoom lens, lens distortion correction information corresponding to the focal length is managed. Then, when the focal length is changed, the lens distortion correction information may be switched and the feature point matching area storage unit 113 may be updated.

[0037] Furthermore, if the configuration allows for changing the size of the image data output from the imaging unit 101, the feature point matching area initialization unit 115 may be used to update the feature point matching area storage unit 113 when changing the size of the image data.

[0038] Furthermore, the feature point matching region information calculated in advance may be stored in the feature point matching region storage unit 113 before the camera device 100 is started, and the feature point matching region initialization unit 115 may not be used.

[0039] The image receiving unit 106 receives image data transmitted from the imaging unit 101 and stores it in the intermediate data storage unit 112. The image receiving unit 106 is an example of an image acquisition means that acquires a first image and a second image captured by the imaging unit 101, which is an imaging means having lens distortion. The intermediate data storage unit 112 can store the three most recent consecutive image data in a ring buffer format.

[0040] The image correction unit 107 performs preprocessing necessary for feature point extraction on the image data transmitted from the image receiving unit 106. Typically, this includes demosaicing, which converts a mosaic-like color array, such as a Bayer array transmitted from the imaging unit 101, into an RGB image, and shading correction, which corrects brightness unevenness caused by peripheral light falloff due to the lens 103.

[0041] The feature point extraction unit 108 detects points in the image data output from the image correction unit 107 where the brightness or color changes abruptly, i.e., edges. The number of these edges is variable depending on the image data and is represented as pixel-level coordinate information. The feature point extraction unit 108 uses one of the following algorithms: FAST, AGAST, Harris Corner, etc. The coordinate information of the feature points extracted by the feature point extraction unit 108 is stored in the intermediate data storage unit 112 on an image data basis. This intermediate data storage unit 112 can store the coordinate information of feature points from at least the three most recent consecutive image data sets in a ring buffer format. The feature point pair generation unit 102 uses this ring buffer to execute the feature point extraction unit 108, the feature descriptor generation unit 109, the feature point matching region determination unit 110, and the feature point matching unit 111 in parallel.

[0042] The feature descriptor generation unit 109 calculates local features around feature points and describes the features of feature points as feature descriptors. The algorithm of the feature descriptor generation unit 109 is one of the following: BRIEF, ORB, BRISK, AKANE, etc. The feature descriptors generated by the feature descriptor generation unit 109 are stored in the intermediate data storage unit 112 on an image data basis. Similar to the feature point extraction unit 108, this intermediate data storage unit 112 can store feature descriptors of at least the three most recent consecutive image data in a ring buffer format.

[0043] When the image receiving unit 106 receives images in the order of the first image, the second image, and the third image, the feature point matching region determination unit 110 starts processing after the feature descriptor generation unit 109 stores the feature descriptors of the first and second images in the intermediate data storage unit 112. The feature point matching region determination unit 110 uses the coordinates of each feature point in the first image extracted by the feature point extraction unit 108 to obtain the feature point matching region in the second image from the feature point matching region storage unit 113.

[0044] The processing of the feature point matching region determination unit 110 will be explained using Figure 6. In step S501, the feature point matching region determination unit 110 obtains the width and height of the feature point region from the lens distortion correction information storage unit 114. In step S502, the feature point matching region determination unit 110 obtains the feature point coordinates Fn(x,y) of the first image from the intermediate data storage unit 112. In step S503, the feature point matching region determination unit 110 obtains the feature descriptor corresponding to the feature point coordinates Fn(x,y) of the first image from the intermediate data storage unit 112.

[0045] In step S504, the feature point matching region determination unit 110 calculates the feature point region coordinates Pn(x,y) to which the feature point coordinates Fn(x,y) of the first image belong. In this calculation, the feature point matching region determination unit 110 uses the width and height of the feature point region obtained in step S501 and the feature point coordinates Fn(x,y) obtained in step S502. Specifically, if the width of the feature point region is w, the height of the feature point region is h, and the feature point coordinates are Fn(x1,y1), then the feature point region coordinates Pn(x2,y2) are calculated using the following equations (Equation 1) and (Equation 2). In equations (1) and (2), mod is the modulo operator. x² = x¹ - (x¹ mod w) ... (Equation 1) y² = y¹ - (y¹ mod h) ... (Equation 2)

[0046] In step S505, the feature point matching region determination unit 110 obtains the coordinates Bn-TL, Bn-TR, Bn-BL, and Bn-BR of the feature point matching region of the second image. In obtaining these coordinates, the feature point matching region determination unit 110 uses the feature point region coordinates Fn(x,y) of the first image calculated in step S504 from the feature point matching region information in Figure 4(C).

[0047] In step S506, the feature point matching region determination unit 110 transmits the feature point coordinates Fn(x,y) of the first image acquired in step S504 to the feature point matching unit 111. Furthermore, in step S506, the feature point matching region determination unit 110 transmits the feature descriptor of the first image acquired in step S503 to the feature point matching unit 111. Furthermore, in step S506, the feature point matching region determination unit 110 transmits the coordinates of the four points of the feature point matching region of the second image calculated in step S505 to the feature point matching unit 111.

[0048] In step S507, the feature point matching region determination unit 110 determines whether processing has been completed for all feature point coordinates Fn(x,y) of the first image. If the feature point matching region determination unit 110 determines that processing of all feature point coordinates has been completed, the process shown in Figure 6 is terminated. If the feature point matching region determination unit 110 determines that processing of all feature point coordinates has not been completed, the process in step S502 is executed.

[0049] The feature point matching unit 111 receives the feature point coordinates Fn(x,y) of the first image from the feature point matching region determination unit 110. Furthermore, the feature point matching unit 111 receives the feature descriptor of the feature point coordinates Fn(x,y) of the first image from the feature point matching region determination unit 110. Furthermore, the feature point matching unit 111 receives the coordinates Bn-TL, Bn-TR, Bn-BL, and Bn-BR of the feature point matching region of the second image from the feature point matching region determination unit 110.

[0050] The feature point matching unit 111 sequentially acquires the feature point coordinates Fn(x,y) and feature descriptors within the feature point matching region of the second image, which are determined by the coordinates of four points, from the intermediate data storage unit 112. Furthermore, the feature point matching unit 111 compares the feature descriptors of the feature points of the first image with the feature descriptors of the feature points of the second image. Based on this, the feature point matching unit 111 finds the feature point coordinates Gn(x,y) of the feature descriptor with the highest similarity. Finally, the feature point matching unit 111 outputs the feature point pairs of the feature point coordinates Fn(x,y) of the first image and the feature point coordinates Gn(x,y) of the second image from the feature point pair generation unit 102.

[0051] As described above, in this embodiment, a camera device 100 placed on a moving or stationary object corrects the feature point matching area using lens distortion correction information before image capture begins. This allows for maintaining matching accuracy and reducing the matching area.

[0052] For example, when using a barrel-shaped lens that causes the image to expand significantly outwards, the accuracy of feature point matching can be maintained by correcting the feature point matching area to expand outwards. Similarly, when using a pincushion-shaped lens that causes the image to shrink significantly inwards, the feature point matching area can be reduced by shrinking it inwards. For instance, if the area before lens distortion correction in a given image region is reduced to half its vertical size due to lens distortion, the feature point matching area can be reduced by 50% compared to the area before lens distortion correction.

[0053] <Embodiment 2> Embodiment 2 of the present invention will be described using the figures. Figure 7 is a block diagram showing the overall configuration of the camera device 600 according to Embodiment 2 of the present invention. Configurations that perform the same processing as shown in Embodiment 1 are given the same reference numerals and their descriptions are omitted. The camera device 600 according to Embodiment 2 has an imaging unit 101 and a feature point pair generation unit 601. In addition to the configuration of the feature point pair generation unit 102 in Figure 1, the feature point pair generation unit 601 has a camera movement information acquisition unit 602 and a steering angle vector information storage unit 603.

[0054] The camera device 600 of Embodiment 2 is installed on a moving object such as a vehicle and is applied when generating feature point pairs related to stationary objects between consecutive image data. The following description will use the case where the camera device 600 is an in-vehicle camera as an example. In Embodiment 2, when the feature point matching area initialization unit 115 calculates the feature point matching area before the camera device 600 starts imaging, it uses the steering angle vector information of the camera device 600 in addition to the lens distortion correction information in Embodiment 1.

[0055] Figure 8 shows the direction of movement of feature points of stationary objects according to the steering angle. Figure 8(A) shows the direction in which feature points of stationary objects in image data move as vectors when the camera device 600 is moving in a straight line. When the camera device 600 is moving in a straight line, the vectors spread outwards radially from the center of the image.

[0056] Figure 8(B) shows a vector diagram indicating the movement of feature points of stationary objects in image data when the camera device 600 turns left. When the camera device 600 turns left, the vector moves from left to right, opposite to the steering angle. Figure 8(C) shows a vector diagram indicating the movement of feature points of stationary objects in image data when the camera device 600 turns right. When the camera device 600 turns right, the vector moves from right to left, opposite to the steering angle.

[0057] The steering angle vector information of the steering angle vector information storage unit 603 will be explained using Figure 9. Figure 9(A) is a diagram showing a movement vector that contains only information about the direction of movement of the feature point region coordinate vector during straight-line movement. Movement vector V1 is shown as the direction of movement of the feature point region coordinate P1 of the feature point region 801 in the next image data. Similarly, movement vector V2 is shown as the direction of movement of the feature point region coordinate P2 of the feature point region 802 in the next image data. It should be noted that movement vectors V1 and V2 in Figure 9(A) are part of Figure 8(A).

[0058] Figure 9(B) shows the movement vectors of the feature point region coordinates when turning right. The notation is the same as in Figure 9(A). Figure 9(B) shows that the movement vectors V1 and V2 are part of Figure 8(C).

[0059] Figure 9(C) shows the movement vector information for each steering angle in the steering angle vector information storage unit 603. The steering angle vector information storage unit 603 stores movement vector information indicating the destination in the second image, based on the coordinates of each feature point region in the first image, for each steering angle interval. Region 803 indicates the number N of steering angle intervals. In this embodiment, there are three steering angle intervals: straight, right turn, and left turn.

[0060] Region 804 represents the steering angle range A0 to A1 indicating straight-line movement and corresponds to the movement vector in Figure 9(A). Region 805 stores the coordinates P1 of feature point region 801 in Figure 9(A). Region 806 stores the movement vector V1 of feature point region 801 in Figure 9(A). Region 807 stores the coordinates P2 of feature point region 802 in Figure 9(A). Region 808 stores the movement vector V2 of feature point region 802 in Figure 9(A).

[0061] Region 809 represents the steering angle range A2 to A3 indicating a right turn and corresponds to the movement vector in Figure 9(B). Region 810 stores the coordinates P1 of feature point region 801 in Figure 9(B). Region 811 stores the movement vector V1 of feature point region 801 in Figure 9(B). Region 812 stores the coordinates P2 of feature point region 802 in Figure 9(B). Region 813 stores the movement vector V2 of feature point region 802 in Figure 9(B).

[0062] In this embodiment, the steering angle intervals are broadly divided into three categories: going straight, turning right, and turning left. However, the steering angle intervals for turning right and turning left may be further subdivided, for example.

[0063] Next, the procedure for correcting the motion vector of feature points based on steering angle and vehicle speed in Embodiment 2 will be explained using Figure 10. Here, the steering angle of the vehicle is assumed to be when moving in a straight line, and the motion vector of the feature points of a stationary object is as shown in Figure 8(A). Also, Figures 10(A), 10(B), and 10(C) show the upper right portion of the image data. Figure 10(A) shows the motion vector V of the feature points of a stationary object according to the steering angle of the feature point region coordinate P. The feature point region coordinate of feature point region 901 is P, and this indicates that the feature point region coordinate P moves in the direction of motion vector V in the next image data. The motion vector V can be obtained using the coordinate P of the feature point region from the motion vector information for each steering angle in Figure 9(C).

[0064] Figure 10(B) shows the motion vector (start point P, end point R1) when the current vehicle speed is S and the movement vector is V. This motion vector can be calculated by starting from the feature point region coordinate P, setting the direction of the vector to the direction of the movement vector V shown in Figure 10(A), and multiplying the length of the vector by the current vehicle speed S and the time interval between image data.

[0065] In other words, Figure 10(B) shows that a feature point in feature point region 901 with feature point region coordinates P in the first image moves into feature point region 902 with feature point coordinates R1 in the second image.

[0066] Figure 10(C) shows the motion vector of a feature point (start point P, end point R4) after correcting the motion vector (start point P, end point R1) of Figure 10(B) using the lens distortion correction information of Figure 3(B). The correction steps are explained below in (1) to (4). The method for calculating the feature point region coordinates is the same as in step S405. (1) Calculate the feature point region coordinate R2 of feature point region 902. (2) Refer to the lens distortion correction in Figure 3(B) to obtain the feature point coordinates Q by correcting the feature point region coordinates R2. (3) Find the feature point coordinates R3 that correspond to the endpoint when the starting point of the vector (start point R2, end point Q) is moved to R1. (4) Calculate the feature point region coordinates R4 of the feature point matching region 903 of the feature point coordinates R3.

[0067] By following the calculation procedures (1) to (4) above, it can be calculated that a feature point within the feature point region 901 at feature point region coordinates P in the first image moves into the feature point matching region 903 at feature point region coordinates R4 in the second image. In other words, it is possible to calculate the feature point matching region 903 in the second image that corresponds to the feature point region 901 in the first image when the vehicle speed is S and the movement vector corresponding to the steering angle is V. Therefore, the smallest unit of the feature point matching region in this embodiment is the feature point region.

[0068] The feature point matching region initialization unit 115 will be explained using Figures 11, 12, and 13. Figures 11 and 12 show feature point matching region information based on the vehicle's steering angle and vehicle speed, which is stored in the feature point matching region storage unit 113. Figure 13 is a flowchart showing the initialization process of the feature point matching region in Figure 12. Before imaging starts, the feature point matching region initialization unit 115 writes the information from region 1004 onwards of the feature point matching region information in Figure 12 to the feature point matching region storage unit 113.

[0069] In steps S1101 to S1102, the feature point matching region initialization unit 115 performs the same processing as in steps S401 to S402 in Figure 5. In step S1103, the feature point matching region initialization unit 115 obtains the number K of the speed intervals of the camera device 600 from region 1003 of the feature point matching region information in Figure 12.

[0070] Here, the speed range of the camera device 600 will be explained using Figure 12. The lowest speed in the speed range of the camera device 600 is the stopped state, which corresponds to S1. The highest speed in the speed range of the camera device 600 is the highest speed stored in area 1001. The steps in the speed range of the camera device 600 are stored in area 1002.

[0071] The number K of speed sections in region 1003 is the value obtained by dividing the speed sections S1:S6 of the camera device 600 by the speed range increment 1002 of the camera device 600. The sections obtained by dividing the speed sections S1:S6 of the camera device 600 by the speed range increment 1002 are called speed sections, and in this embodiment, the speed sections are divided into three sections S1:S2, S3:S4, and S5:S6. The number K of speed sections in region 1003 stores 3, which is the number of divisions of the speed section. In this embodiment, an example of dividing the speed sections equally is shown, but a configuration in which the number of each speed section and each speed section can be set is also possible. Speed ​​sections S5:S6 are speed sections where the vehicle speed is faster than speed sections S3:S4. Speed ​​sections S3:S4 are speed sections where the vehicle speed is faster than speed sections S1:S2.

[0072] The values ​​for regions 1001 to 1003 are stored before imaging, and regions 1004 and beyond, excluding the steering angle section including region 1004, are regions initialized by the feature point matching region initialization unit 115.

[0073] In step S1104, the feature point matching region initialization unit 115 obtains the number of steering angle intervals N from the movement vector information for each steering angle in Figure 9(C). In step S1105, the feature point matching region initialization unit 115 performs the same processing as in step S405 in Figure 5. In this embodiment, the feature point matching region is calculated only for the coordinate of the upper left of the feature point region, but the feature point matching region may also be calculated for, for example, the upper left, upper right, lower left, and lower right, depending on the region 203 (width and height of the feature point region) in Figure 3(B).

[0074] In step S1106, the feature point matching region initialization unit 115 selects steering angle intervals from the feature point matching region information in Figure 12. Specifically, it refers to the number N steering angle intervals in region 803 and obtains N steering angle intervals by adding an offset value to the address where the first steering angle interval, region 1004, is located.

[0075] In step S1107, the feature point matching area initialization unit 115 selects a speed interval. In this embodiment, speed intervals S1:S2, S3:S4, and S5:S6 are selected sequentially. In step S1108, the feature point matching area initialization unit 115 selects one speed within the speed interval selected in step S1107. In this embodiment, three more speeds are selected within each of the speed intervals S1:S2, S3:S4, and S5:S6. Here, a configuration may be provided in which the speed increment can be set for each speed interval. In that case, setting a finer speed increment improves the accuracy of the feature point matching area, but increases the processing load of the feature point matching area initialization unit 115 before imaging starts. Note that the memory size of the feature point matching area storage unit 113 does not change depending on the speed increment. Also, if the feature point matching area information is stored in advance and there is no feature point matching area initialization unit 115, the increase in processing load due to the speed increment will not affect the camera device 600.

[0076] In step S1109, the feature point matching region initialization unit 115 obtains a movement vector Vn from the steering angle range. That is, the feature point matching region initialization unit 115 obtains a movement vector Vn for the selected speed and the feature point region coordinate Pn(x,y) corresponding to the selected feature point region coordinate Pn, based on the movement vector information for each steering angle. The movement vector information for each steering angle used here is shown in Figure 9(C). The selected speed used here is the speed selected in step S1108. The selected feature point region coordinate Pn used here is the feature point region coordinate Pn selected in step S1105.

[0077] In step S1110, the feature point matching region initialization unit 115 calculates a motion vector using the velocity selected in step S1108 and the motion vector Vn obtained in step S1109, according to the method in Figure 10(B).

[0078] Here, we will explain the range of the endpoints of the motion vectors before lens distortion correction and the feature point matching regions in the speed intervals S1:S2, S3:S4, and S5:S6 using Figures 11(A) to 11(D). First, we will explain by referring to Figures 11(A) to 11(C). These figures show the motion vectors before lens distortion correction and the feature point matching regions for each speed interval in a specific motion vector V1 stored in the motion vector information for each steering angle in Figure 9(C), with respect to the feature point region coordinate P1 in the first image. The motion vector information for each steering angle used here is the one described in Figure 9(C).

[0079] Figure 11(A) shows the range of the endpoint of the motion vector and the feature point matching region before lens distortion correction in the velocity interval S1:S2. The feature point matching region of the velocity interval S1:S2 is the feature point region that includes the feature point region coordinate P1 and the feature point region that includes the range of the endpoint of the motion vector.

[0080] The feature point matching region in velocity interval S3:4 in Figure 11(B), and the feature point matching region in velocity interval step S5:S6 in Figure 11(C) are feature point regions that include the range of the end of the motion vector. Figure 11(D) shows the feature point matching regions of all velocity intervals S1:S6 before correction, and is a combined figure of the feature point matching regions from Figures 11(A) to 11(C).

[0081] In step S1111, the feature point matching region initialization unit 115 corrects the motion vector (start point P, end point R1) using the lens distortion correction information in Figure 3(B) according to the method in Figure 10(C), and calculates the feature point region coordinates R4 of the feature point matching region 903.

[0082] Here, the endpoints of the motion vectors after lens distortion correction and the feature point matching regions in the velocity intervals S1:S2, S3:S4, and S5:S6 will be explained using Figures 11(E) to 11(H).

[0083] Figure 11(E) shows the endpoints of the motion vectors after lens distortion correction and the feature point matching region for the velocity interval S1:S2. The feature point matching region for the velocity interval S1:S2 is the feature point region containing the feature point region coordinate P1 and the feature point region containing the endpoints of the corrected motion vectors corresponding to the three velocities within each velocity interval selected in step S1108.

[0084] The feature point matching region in velocity interval S3:S4 in Figure 11(F) is a feature point region that includes the endpoints of the corrected motion vectors corresponding to the three velocities within each velocity interval selected in step S1108. The feature point matching region in velocity interval steps S5:S6 in Figure 11(G) is a feature point region that includes the endpoints of the corrected motion vectors corresponding to the three velocities within each velocity interval selected in step S1108. Figure 11(H) shows the corrected feature point matching regions for all velocity intervals S1:S6, and is a combined figure of the feature point matching regions in Figures 11(E) to 11(H).

[0085] In step S1112, the feature point matching region initialization unit 115 stores the feature point region coordinates of the feature point matching region calculated in step S1111 in the intermediate data storage unit 112. As shown in the feature point matching region of velocity section S1:S2 in Figure 11(E), there may be multiple endpoints of the motion vector after lens distortion correction belonging to a particular feature point region. Therefore, if the feature point region coordinates calculated in step S1111 have already been stored in the intermediate data storage unit 112, no processing is performed in step S1112.

[0086] In step S1113, the feature point matching region initialization unit 115 determines whether all speeds have been selected in step S1108 within the speed range selected in step S1107. If the feature point matching region initialization unit 115 determines that all speeds have not been selected, the process in step S1108 is executed. If the feature point matching region initialization unit 115 determines that all speeds have been selected, the process in step S1114 is executed.

[0087] In step S1114, the feature point matching region initialization unit 115 corrects the feature point matching region into a rectangular region based on the feature point region coordinates of one or more feature point matching regions stored in the intermediate data storage unit 112 in step S1112.

[0088] The method for correcting the feature point matching region will be explained using Figure 11(I). Figure 11(I) shows the corrected feature point matching region of Figure 11(H), corrected so that the feature point matching region for each velocity interval becomes a rectangular region.

[0089] In the feature point matching region before correction in Figure 11(H), the velocity intervals S1:S2 and S3:S4 are not rectangular regions, and only step S5:S6 is a rectangular region. Therefore, as shown in Figure 11(I), the feature point matching regions of velocity intervals S1:S2 and S3:S4 are corrected to become rectangular regions. In this embodiment, the feature point matching region for each velocity interval is a rectangular region in which one or more feature point regions are adjacent.

[0090] In step S1115, the feature point matching region initialization unit 115 stores the coordinates of the four corners of the calculated rectangular feature point matching region of the second image relative to the selected feature region of the first image for the selected steering angle interval into the feature point matching region information. The selected steering angle interval used here is the steering angle interval selected in step S1106. The selected feature region of the first image used here is the feature region of the first image selected in step S1105. The calculated rectangular feature point matching region of the second image used here is the rectangular feature point matching region of the second image calculated in step S1114. The feature point matching region information used here is shown in Figure 12.

[0091] The method for storing the coordinates of the four corners of the feature point matching region will be explained using Figures 11(I) and 12. The feature point matching region information in Figure 12 stores the coordinates of the feature point matching region for each velocity interval for all feature point region coordinates Pn(x,y) in the first image for each steering angle interval, starting from region 1004. The number of feature point matching regions for a single feature point region coordinate Pn(x,y) is equal to the number of velocity intervals K.

[0092] The feature point matching region information in Figure 12 stores the coordinates of the four corners of the feature point matching regions for each rectangular region of the velocity intervals S1:S2, S3:S4, and S5:S6 in Figure 11(I). Specifically, region 1006 stores the coordinates of the four points S1-TL, S1-TR, S1-BL, and S1-BR of the feature point matching region for velocity interval S1:S2. Region 1007 stores the coordinates of the four points S3-TL, S3-TR, S3-BL, and S3-BR of the feature point matching region for velocity interval S3:S4. Region 1008 stores the coordinates of the four points step S5-TL, step S5-TR, step S5-BL, and step S5-BR of the feature point matching region for velocity interval step S5:S6.

[0093] In step S1116, the feature point matching region initialization unit 115 determines whether the selection of all velocity intervals was completed in step S1107. If the feature point matching region initialization unit 115 determines that the selection of all velocity intervals has not been completed, the process in step S1107 is executed. If the feature point matching region initialization unit 115 determines that the selection of all velocity intervals has been completed, the process in step S1117 is executed.

[0094] In step S1117, the feature point matching region initialization unit 115 determines whether the selection of all steering angle intervals was completed in step S1106. If the feature point matching region initialization unit 115 determines that the selection of all steering angle intervals has not been completed, the process in step S1106 is executed. If the feature point matching region initialization unit 115 determines that the selection of all steering angle intervals has been completed, the process in step S1118 is executed.

[0095] In step S1118, the feature point matching region initialization unit 115 determines whether the selection of all feature point regions was completed in step S1105. If the feature point matching region initialization unit 115 determines that the selection of all feature point regions has not been completed, the process in step S1105 is executed. If the feature point matching region initialization unit 115 determines that the selection of all feature point regions has been completed, the process in Figure 13 is terminated.

[0096] The feature point matching region determination unit 110 will be explained using Figures 14 to 16. In step S1201, the feature point matching region determination unit 110 acquires camera movement information from the camera movement information acquisition unit 602.

[0097] The camera movement information acquisition unit 602 will be explained using Figure 15. In step S1301, the camera movement information acquisition unit 602 acquires the current vehicle speed of the vehicle on which the camera device 600 is located. In step S1302, the camera movement information acquisition unit 602 determines the current acceleration / deceleration state of the vehicle on which the camera device 600 is located. Here, the camera movement information acquisition unit 602 determines whether the vehicle is in a constant speed state, a deceleration state, or an acceleration state based on the speed change over a certain period of time in the past. In step S1303, the camera movement information acquisition unit 602 acquires the current steering angle of the vehicle on which the camera device 600 is located. The camera movement information acquired by the feature point matching region determination unit 110 in step S1201 includes the vehicle speed acquired in step S1301, the acceleration / deceleration state determined in step S1302, and the steering angle acquired in step S1303.

[0098] Returning to the explanation of Figure 14, in step S1202, the feature point matching region determination unit 110 performs the same processing as in step S501 in Figure 6. In step S1203, the feature point matching region determination unit 110 uses the vehicle speed obtained in step S1301 and the acceleration / deceleration state determined in step S1302 to determine the processing order for each speed section of the feature point matching region.

[0099] The method for determining the processing order in the feature point matching region per velocity interval will be explained using Figure 16. The vertical axis of Figure 16 represents the velocity intervals S1:S2, S3:S4, and S5:S6 in Figure 11(I). The horizontal axis of Figure 16 represents the three types of acceleration / deceleration states determined in step S1302: constant speed state, deceleration state, and acceleration state. Each value in the table in Figure 16 indicates the priority when performing feature point matching processing in each acceleration / deceleration state per velocity interval; a smaller value indicates a higher priority.

[0100] If the current vehicle speed obtained in step S1301 is in speed section S3:S4, and the acceleration / deceleration state of the vehicle determined in step S1302 is in the acceleration state, then the speed section with the highest processing priority is the feature point matching region of speed section S3:S4 that includes the current vehicle speed.

[0101] In this embodiment, since the vehicle speed is acquired in step S1301 after image data is captured, if the vehicle is accelerating, the vehicle speed acquired in step S1301 may be faster than the vehicle speed at the time of capture. Therefore, the speed section with the second priority for processing is the feature point matching region of the slower speed section S1:S2 that precedes the speed section S3:S4 containing the current vehicle speed. Finally, the two adjacent speed sections S5:S6 of the speed section S3:S4 containing the current vehicle speed, which have not been selected, are given the third priority.

[0102] If the acceleration / deceleration state determined in step S1302 is a deceleration state, the second priority is set to the feature point matching region S5:S6, which is faster than the current vehicle speed, taking into account the delay amount in step S1301, similar to the case of an acceleration state.

[0103] If the acceleration / deceleration state determined in step S1303 is a constant speed state, the second priority can be either the low-speed section S1:S2 or the high-speed section step S5:S6, and in this embodiment, the low-speed section S1:S2 is set as the second priority.

[0104] The algorithm for determining such priorities may be incorporated into the feature point matching region determination unit 110, or a priority table as shown in Figure 16 may be stored in the feature point matching region storage unit 113 before the start of imaging. In addition, in order to reduce the delay amount in step S1301, for example, the image transmission unit 105 may be configured to acquire camera movement information using the camera movement information acquisition unit 602.

[0105] Returning to the explanation of Figure 14, steps S1204 to S1206 perform the same processing as steps S502 to S504 in Figure 6.

[0106] In step S1207, the feature point matching region determination unit 110 identifies a region where the coordinates of the feature point matching region for a steering angle interval are stored, based on the steering angle interval determined from the acquired steering angle and the calculated feature point region coordinates, using the feature point matching region information. The feature point matching region information used here is shown in Figure 12. The acquired steering angle used here is the steering angle acquired in step S1201. The calculated feature point region coordinates used here are the feature point region coordinates calculated in step S1206. The region identified here is one of regions 1006 to 1008 in Figure 12.

[0107] In step S1208, the feature point matching region determination unit 110 transmits the acquired processing order for each velocity interval, the acquired feature point coordinates, the acquired feature descriptors, and the address of the region where the coordinates of the identified feature point matching region are stored. The acquired processing order for each velocity interval used here is the processing order for each velocity interval acquired in step S1203. The acquired feature point coordinates used here are the feature point coordinates of the first image acquired in step S1204. The acquired feature descriptors used here are the feature descriptors of the feature point coordinates of the first image acquired in step S1205. The address of the region where the coordinates of the identified feature point matching region are stored is the address of the region where the coordinates of the feature point matching region for each velocity interval of the second image, identified in step S1207, are stored.

[0108] In step S1209, the feature point matching region determination unit 110 performs the same processing as in step S507 in Figure 6.

[0109] The feature point matching unit 111 receives the information transmitted by the feature point matching region determination unit 110 in step S1208. Specifically, the feature point matching unit 111 receives the processing order for each velocity interval, the coordinates of the feature points in the first image, the feature descriptors of those feature point coordinates, and the address of the region where the coordinates of the feature point matching region for each velocity interval of the second image are stored.

[0110] The feature point matching unit 111 compares the feature descriptors of the first image and the second image, similar to Embodiment 1. That is, based on the processing order in units of velocity intervals, for example, within velocity interval S3:S4 and in an accelerating state, the feature point matching unit 111 compares the feature descriptors in the feature point matching region of Figure 11(I) in the order of velocity intervals S3:S4, S1:S2, and S5:S6.

[0111] Here, the similarity threshold and the upper limit of the number of values ​​that exceed the threshold are determined before the start of imaging. When switching the feature point matching region for each speed interval, if the upper limit is exceeded, the comparison of feature quantities is terminated, and the feature descriptor with the highest similarity is set as the feature point pair. Here, in order to optimize the feature point matching process for the second and third images, a process to correct the similarity threshold and the upper limit of the number of values ​​that exceed the threshold may be added. Then, the feature point pairs of the feature point coordinates of the first image and the feature point coordinates of the second image are output from the feature point pair generation unit 102.

[0112] As described above, in this embodiment, in the camera device 600 placed on a moving object such as a vehicle, the feature point matching region is calculated using lens distortion correction information and camera movement information before the start of imaging. After the start of imaging, by using this feature point matching region, even if the lens distortion is large and the distortion affects the entire image data, the feature point matching process can be accelerated by limiting the feature point matching region while maintaining the accuracy of feature point matching.

[0113] <Embodiment 3> Embodiment 3 of the present invention will be described with reference to the figures. Figure 17 is a block diagram showing the overall configuration of the camera device 1500 according to Embodiment 3 of the present invention. Configurations that perform the same processing as shown in Embodiment 2 are given the same reference numerals and their descriptions are omitted. The camera device 1500 according to Embodiment 3 has an imaging unit 101 and a feature point pair generation unit 1501. In addition to the configuration of the feature point pair generation unit 601 in Figure 7, the feature point pair generation unit 1501 has a depth information storage unit 1502 and a scene determination unit 1503.

[0114] The camera device 1500 of Embodiment 3 is installed on a moving object such as a vehicle and is applied when generating feature point pairs relating to stationary objects between consecutive image data. The following explanation will use the case where the camera device 1500 is an in-vehicle camera as an example.

[0115] In Embodiment 3, when the feature point matching area initialization unit 115 calculates the feature point matching area before the camera device 1500 starts imaging, it uses depth information of the image data in addition to the information used in Embodiment 2. Depth information refers to information that is estimated and quantified by image processing of the distance from the camera device 1500 to an object in the image area, or information that is measured and quantified using ranging technology such as LiDAR. LiDAR is an abbreviation for Light Detection and Ranging.

[0116] The depth information storage unit 1502 stores depth information for each scene in front of the vehicle before imaging begins. For example, for scenes in urban areas, the depth information in the upper part of the image area is set to represent short distances, while for scenes in suburban areas, the depth information in the upper part of the image area is set to represent long distances. Depth information is information that indicates the distance from the camera device 1500 to an object in the image captured by the camera device 1500.

[0117] The method for calculating the motion vector in step S1110 by the feature point matching region initialization unit 115 will be explained using Figure 10(B). First, the feature point matching region initialization unit 115 calculates the length of the motion vector by multiplying the current vehicle speed S by the time interval between image data, similar to Embodiment 2. Then, it corrects the length of the motion vector for depth information in scenes with many stationary objects, such as buildings in urban areas, and depth information in scenes with few stationary objects, such as suburbs.

[0118] In scenes with many stationary objects, the distance from the camera device 1500 to the stationary objects is short, so the length of the motion vector in Figure 10(B) becomes longer when corrected with depth information. The length of the motion vector in scenes with many stationary objects is shorter than the motion vector in front of the camera. In scenes with few stationary objects, the distance from the camera device 1500 to the stationary objects is long, so the length of the motion vector in Figure 10(B) becomes shorter compared to scenes with many stationary objects when corrected with depth information.

[0119] In Embodiment 3, feature point matching information as shown in Figure 12 is generated for two scenes. The feature point matching region determination unit 110 uses the scene determination unit 1503 to determine the scene type for the second image data stored in the intermediate data storage unit 112. Next, the processing shown in Figure 13 is performed, similar to Embodiment 2, and in step S1207, the coordinates of the feature point matching region are obtained using the scene type, in addition to the feature point region coordinates and steering angle range in Embodiment 2.

[0120] Furthermore, a recognition process that generates depth information, or a LiDAR or the like that can acquire distance information, may be added to the configuration to acquire depth information in real time and update the feature point matching region in the feature point matching region initialization unit 115 during imaging.

[0121] As described above, in this embodiment, in the camera device 1500 mounted on a moving object such as a vehicle, depth information of the image data is used when calculating the feature point matching area in Embodiment 2, before the start of imaging. This makes it possible to limit the feature point matching area more than in Embodiment 2.

[0122] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0123] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence.

[0124] This embodiment includes the following configurations, methods, and programs. (Composition 1) Image acquisition means for acquiring a first image and a second image captured by an imaging means having lens distortion, A lens distortion correction information storage means for storing lens distortion correction information for correcting lens distortion present in the imaging means, A feature point extraction means for extracting one or more feature points from the first image and the second image, A feature descriptor generation means that generates feature descriptors that show the feature quantities of the feature points of the first image and the second image, A feature point matching region initialization means calculates a feature point matching region in a second image corresponding to a feature point in the first image using the lens distortion correction information before the start of imaging by the imaging means, A feature point matching region determination means obtains a feature point matching region to be used in correspondence with the feature points in the first image from the feature point matching region calculated by the feature point matching region initialization means, A feature point matching means that performs feature point matching between the feature points of the first image and the feature points of the second image in the feature point matching region acquired by the feature point matching region determination means, using feature descriptors of the feature points of the first image and the second image. An image processing apparatus characterized by having (Configuration 2) The aforementioned lens distortion correction information represents the direction of pixel movement when the lens is distorted compared to when the lens is not distorted. The image processing apparatus according to configuration 1, characterized in that... (Composition 3) The imaging means is positioned on a stationary object. An image processing apparatus according to configuration 1 or configuration 2, characterized by the above. (Composition 4) The imaging means is positioned on the moving body. An image processing apparatus according to configuration 1 or configuration 2, characterized by the above. (Composition 5) The feature point matching region initialization means further corrects the calculated feature point matching region using the speed range of the moving body on which the imaging means is located, and a movement vector indicating the direction of feature point movement of a stationary object according to the steering angle. The image processing apparatus according to configuration 4, characterized in that... (Composition 6) The feature point matching region determination means determines the feature point matching region to be acquired using the feature points of the first image and the steering angle of the moving body on which the imaging means is located. The image processing apparatus according to configuration 4 or configuration 5, characterized by the above. (Composition 7) The feature point matching means determines the processing order for feature point matching to the feature point matching region determined by the feature point matching region determination means for each speed section, based at least on the vehicle speed of the moving body. An image processing apparatus according to any one of configurations 4 to 6, characterized by the features described herein. (Composition 8) The feature point matching region initialization means further corrects the calculated feature point matching region using depth information, which is information indicating the distance from the imaging means to the object in the image captured by the imaging means. An image processing apparatus according to any one of configurations 4 to 7, characterized by the features described herein. (Composition 9) The lens distortion correction information and the depth information are stored in the memory area before the imaging by the imaging means starts. The image processing apparatus according to configuration 8, characterized by the above. (Method 1) An image acquisition step in which a first image and a second image are acquired by an imaging means having lens distortion, A lens distortion correction information storage step for storing lens distortion correction information that corrects the lens distortion of the imaging means, A feature point extraction step of extracting one or more feature points from the first image and the second image, A feature descriptor generation step that generates feature descriptors that show the feature quantities of the feature points of the first image and the second image, Before imaging is started by the imaging means, a feature point matching region initialization step is performed in which a feature point matching region in the second image corresponding to a feature point in the first image is calculated using the lens distortion correction information, A feature point matching region determination step is performed to obtain a feature point matching region to be used in correspondence with the feature points in the first image from the feature point matching region calculated in the feature point matching region initialization step, A feature point matching step is performed using feature descriptors of the feature points of the first image and the second image to perform feature point matching between the feature points of the first image and the feature points of the second image in the feature point matching region obtained in the feature point matching region determination step. An image processing method characterized by having the following features. (Program 1) Computers, Image acquisition means for acquiring a first image and a second image captured by an imaging means having lens distortion, Lens distortion correction information storage means for storing lens distortion correction information for correcting lens distortion present in the imaging means, Feature point extraction means for extracting one or more feature points from the first image and the second image, Feature descriptor generation means for generating feature descriptors that represent the feature quantities of the feature points of the first image and the second image, Before imaging is started by the imaging means, a feature point matching region initialization means calculates a feature point matching region in the second image corresponding to a feature point in the first image using the lens distortion correction information. A feature point matching region determination means that obtains a feature point matching region to be used in correspondence with the feature points in the first image from the feature point matching region calculated by the feature point matching region initialization means, and A feature point matching means that performs feature point matching between the feature points of the first image and the feature points of the second image in the feature point matching region obtained by the feature point matching region determination means, using feature descriptors of the feature points of the first image and the second image. A program characterized by being designed to function as such. [Explanation of Symbols]

[0125] 100 Camera equipment 101 Imaging Unit 102 Feature Point Pair Generation Unit 103 Lens 104 Image sensor 105 Image transmission unit 106 Image receiving unit 107 Image Correction Unit 108 Feature point extraction unit 109 Feature Descriptor Generation Unit 110 Feature Point Matching Region Determination Unit 111 Feature Point Matching Section 112 Intermediate data storage unit 113 Feature Point Matching Area Memory Unit 114 Lens distortion correction information storage unit 115 Feature Point Matching Region Initialization Unit

Claims

1. Image acquisition means for acquiring a first image and a second image captured by an imaging means having lens distortion, A lens distortion correction information storage means for storing lens distortion correction information for correcting lens distortion present in the imaging means, A feature point extraction means for extracting one or more feature points from the first image and the second image, A feature descriptor generation means that generates feature descriptors that show the feature quantities of the feature points of the first image and the second image, A feature point matching region initialization means calculates a feature point matching region in a second image corresponding to a feature point in the first image using the lens distortion correction information before the start of imaging by the imaging means, A feature point matching region determination means obtains a feature point matching region to be used in correspondence with the feature points in the first image from the feature point matching region calculated by the feature point matching region initialization means, A feature point matching means that performs feature point matching between the feature points of the first image and the feature points of the second image in the feature point matching region acquired by the feature point matching region determination means, using feature descriptors of the feature points of the first image and the second image. An image processing apparatus characterized by having

2. The aforementioned lens distortion correction information represents the direction of pixel movement when the lens is distorted compared to when the lens is not distorted. The image processing apparatus according to feature 1.

3. The imaging means is positioned on a stationary object. The image processing apparatus according to feature 1.

4. The imaging means is positioned on the moving body. The image processing apparatus according to feature 1.

5. The feature point matching region initialization means further corrects the calculated feature point matching region using the speed range of the moving body on which the imaging means is located, and a movement vector indicating the direction of feature point movement of a stationary object according to the steering angle. The image processing apparatus according to feature 4.

6. The feature point matching region determination means determines the feature point matching region to be acquired using the feature points of the first image and the steering angle of the moving body on which the imaging means is located. The image processing apparatus according to feature 4.

7. The feature point matching means determines the processing order for performing feature point matching on the feature point matching region determined by the feature point matching region determination means for each speed section, based at least on the vehicle speed of the moving body. The image processing apparatus according to feature 4.

8. The feature point matching region initialization means further corrects the calculated feature point matching region using depth information, which is information indicating the distance from the imaging means to the object in the image captured by the imaging means. The image processing apparatus according to feature 4.

9. The lens distortion correction information and the depth information are stored in the memory area before the imaging by the imaging means starts. The image processing apparatus according to feature 8.

10. An image acquisition step in which a first image and a second image are acquired by an imaging means having lens distortion, A lens distortion correction information storage step for storing lens distortion correction information that corrects the lens distortion of the imaging means, A feature point extraction step of extracting one or more feature points from the first image and the second image, A feature descriptor generation step that generates feature descriptors that show the feature quantities of the feature points of the first image and the second image, Before imaging is started by the imaging means, a feature point matching region initialization step is performed in which a feature point matching region in the second image corresponding to a feature point in the first image is calculated using the lens distortion correction information, A feature point matching region determination step is performed to obtain a feature point matching region to be used in correspondence with the feature points in the first image from the feature point matching region calculated in the feature point matching region initialization step, A feature point matching step is performed using feature descriptors of the feature points of the first image and the second image to perform feature point matching between the feature points of the first image and the feature points of the second image in the feature point matching region obtained in the feature point matching region determination step. An image processing method characterized by having the following features.

11. Computers, Image acquisition means for acquiring a first image and a second image captured by an imaging means having lens distortion, Lens distortion correction information storage means for storing lens distortion correction information for correcting lens distortion present in the imaging means, Feature point extraction means for extracting one or more feature points from the first image and the second image, Feature descriptor generation means for generating feature descriptors that represent the feature quantities of the feature points of the first image and the second image, Before imaging is started by the imaging means, a feature point matching region initialization means calculates a feature point matching region in the second image corresponding to a feature point in the first image using the lens distortion correction information. A feature point matching region determination means that obtains a feature point matching region to be used in correspondence with the feature points in the first image from the feature point matching region calculated by the feature point matching region initialization means, and A feature point matching means that performs feature point matching between the feature points of the first image and the feature points of the second image in the feature point matching region acquired by the feature point matching region determination means, using feature descriptors of the feature points of the first image and the second image. A program characterized by being designed to function as such.

Images