Image processing device, control method, and program

Abstract

To provide an image processing device that reduces the amount of processing required for feature point matching and enables faster processing. [Solution] The system includes: an imaging means positioned on a moving body and used to photograph the area around the moving body, from a plurality of time-series images acquired by an imaging means, the imaging means being used to photograph the area around the moving body, and a first image taken when the moving body is at a first position and a second image taken when the moving body is at a second position, and an imaging position movement direction limiting means that restricts the movement direction of the imaging position between the first position and the second position; a feature point matching means that selects a reference image and a search target image from the first image and the second image, respectively, and performs a correspondence between feature points; and a reference image switching means that switches the reference image based on information representing the movement direction of the moving body, wherein the feature point matching means searches for each feature point on the search target image within a limited search range based on the movement direction of the moving body.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method, and a program.

Background Art

[0002] There is a technique for matching points having the same feature between a plurality of images (hereinafter referred to as feature point matching). Feature points associated by feature point matching are used for SfM or vSLAM that reconstructs three-dimensional information from images, and are expected to be used for autonomous driving and robots. SfM is an abbreviation for Structure from Motion. vSLAM is an abbreviation for visual Simultaneous Localization and Mapping. When performing feature point matching, searching for corresponding points from the entire input image requires a huge amount of calculation and takes a long processing time. Therefore, various speed-up methods have been studied.

[0003] For example, in Patent Document 1, when searching for corresponding points of feature points on one image, the search range is limited based on the vehicle's self-position estimation information, thereby reducing the calculation amount of feature point matching.

[0004] Also, in Patent Document 2, the search range is limited based on the position of the vanishing point, thereby reducing the amount of calculation for detecting a moving object.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] Patent Document 1 reduces the computational load by limiting the search range. However, because processing is required to set the search range, there is a risk that the processing cannot be completed in real time.

[0007] Furthermore, Patent Document 2 limits the search range based on the position of the vanishing point. However, since the search range extends from the vanishing point to the edge of the image, the processing load may not be sufficiently reduced depending on the position of the vanishing point.

[0008] The present invention aims to provide an image processing apparatus that reduces the amount of processing required for feature point matching and enables faster processing. [Means for solving the problem]

[0009] An image processing apparatus according to one embodiment of the present invention comprises: an imaging means that selects a first image taken when the moving body is at a first position and a second image taken when the moving body is at a second position from a plurality of time-series images acquired from imaging means placed on a moving body and taking images of the area around the moving body, and an imaging position movement direction limiting means that restricts the movement direction of the imaging position between the first position and the second position; a feature point matching means that selects a reference image and a search target image from the first image and the second image, respectively, and performs a correspondence between feature points; and a reference image switching means that switches the reference image based on information representing the movement direction of the moving body, wherein the feature point matching means searches for each feature point on the search target image within a limited search range based on the movement direction of the moving body. [Effects of the Invention]

[0010] According to the present invention, it is possible to reduce the amount of processing involved in feature point matching and provide an image processing apparatus that can process at a higher speed. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram showing the configuration of a mobile body according to Embodiment 1 of the present invention. [Figure 2]It is a conceptual diagram showing the attachment position and shooting direction of the imaging unit according to Embodiment 1. [Figure 3] It is a conceptual diagram showing time-series images captured by the imaging unit according to Embodiment 1. [Figure 4] It is a conceptual diagram for extracting feature points from the captured image of the front camera according to Embodiment 1. [Figure 5] It is a conceptual diagram for switching the reference frame based on the traveling direction of the moving body according to Embodiment 1. [Figure 6] It is a conceptual diagram for performing corresponding point search within a pre-limited search range according to Embodiment 1. [Figure 7] It is an explanatory diagram regarding epipolar geometry according to Embodiment 1. [Figure 8] It is an operation flowchart of the integration processing unit according to Embodiment 1. [Figure 9] It is an operation flowchart of the reference frame switching unit according to Embodiment 1. [Figure 10] It is an operation flowchart of the feature point matching unit according to Embodiment 1. [Figure 11] It is an operation flowchart of the three-dimensional information generation unit according to Embodiment 1. [Figure 12] It is a configuration diagram of the moving body according to Embodiment 2 of the present invention. [Figure 13] It is an operation flowchart of the integration processing unit according to Embodiment 2. [Figure 14] It is an operation flowchart of the reference frame switching unit according to Embodiment 2. [Figure 15] It is a configuration diagram of the moving body according to Embodiment 3 of the present invention. [Figure 16] It is an explanatory diagram regarding epipolar geometry according to Embodiment 3. [Figure 17] It is an operation flowchart of the integration processing unit according to Embodiment 3. [Figure 18] It is an operation flowchart of the feature point matching unit according to Embodiment 3. [Figure 19] It is a configuration diagram of the moving body according to Embodiment 4 of the present invention. [Figure 20]It is a conceptual diagram showing the attachment position and shooting direction of the imaging unit according to Embodiment 4.

Embodiment for Implementing the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. Also, the following embodiments do not limit the present invention according to the claims, and not all combinations of the features described in this embodiment are essential for the solution means of the present invention. In each figure, the same member or element is assigned the same reference numeral, and duplicate explanations are omitted or simplified.

[0013] <Embodiment 1> Hereinafter, Embodiment 1 of the present invention will be described with reference to the drawings. In Embodiment 1, in a camera that shoots in the same or opposite direction as the traveling direction of a moving body, the moving direction of the shooting position is restricted by switching the reference frame for feature point matching based on the traveling direction of the moving body. Further, in Embodiment 1, a method for speeding up feature point matching, and thus SfM and vSLAM, by restricting the moving direction of the shooting position and limiting the search range for feature point matching will be described.

[0014] FIG. 1 is a configuration diagram of a moving body 1000 according to Embodiment 1. The moving body 1000 is, for example, a vehicle and includes an imaging unit 10, an integrated processing unit 200, and a traveling direction detection unit 30. The moving body 1000 is an example of an image processing apparatus.

[0015] The imaging unit 10 is an in-vehicle camera capable of capturing images in any direction around the moving object 1000, such as a front camera 10a, a rear camera 10b, a right side camera 10c, and a left side camera 10d. The imaging unit 10 may consist of a single camera or multiple cameras. The front camera 10a captures the front of the moving object 1000. The rear camera 10b captures the rear of the moving object 1000. The right side camera 10c and left side camera 10d are cameras used, for example, in a camera monitor system for an electronic side mirror, and capture images of the left and right rear of the moving object 1000. Each camera included in the imaging unit 10 is an example of an imaging means that is positioned on the moving object 1000 to capture images around the moving object 1000.

[0016] The mounting positions of the imaging unit 10 are shown in Figures 2(A) and 2(B). Figure 2(A) is a plan view of the mobile body 1000. Figure 2(B) is a side view of the mobile body 1000. Here, the plane passing through the camera and parallel to the ground is called the camera position. The camera position of the front camera 10a is called the front camera position 31a (see Figure 3). The camera position of the rear camera 10b is called the rear camera position 31b (see Figure 3). The camera position of the right side camera 10c is called the right side camera position 31c (see Figure 3). The camera position of the left side camera 10d is called the left side camera position 31d (see Figure 3).

[0017] Furthermore, the front camera 10a is mounted on the mobile body 1000 such that its shooting direction 21a is oriented to capture images in a direction parallel to the direction of travel of the mobile body 1000. The rear camera 10b is mounted on the mobile body 1000 such that its shooting direction 21b is oriented to capture images in a direction parallel to the opposite direction of travel of the mobile body 1000. The right side camera 10c and the left side camera 10d are mounted on the mobile body 1000 such that their shooting directions 21c and 21d are oriented to capture images in a direction parallel to the opposite direction of travel of the mobile body 1000.

[0018] Returning to the explanation of Figure 1, the imaging unit 10 acquires each imaging signal that captures an optical image at each time point, and outputs two time-series images, img(t) and img(t+1), obtained by RAW development of each imaging signal in chronological order, to the integrated processing unit 200.

[0019] The imaging unit 10 also has an image processing unit (not shown), and it is desirable that this image processing unit has a distortion correction function. This image processing unit performs distortion correction on the time-series images img(t) and img(t+1). As a result of this process, the front camera position 31a, rear camera position 31b, right side camera position 31c, and left side camera position 31d can be shown as straight lines on the image, as shown in Figure 3.

[0020] Figure 3 shows time-series images captured by each camera of the imaging unit 10 when the mobile vehicle 1000 is moving forward. The time-series image img(t) at time t captured by the front camera 10a is time-series image 32a, and the time-series image img(t+1) at time t+1 is time-series image 33a. The time-series image img(t) at time t captured by the rear camera 10b is time-series image 32b, and the time-series image img(t+1) at time t+1 is time-series image 33b. The time-series image img(t) at time t captured by the right side camera 10c is time-series image 32c, and the time-series image img(t+1) at time t+1 is time-series image 33c. The time-series image img(t) at time t captured by the left side camera 10d is time-series image 32d, and the time-series image img(t+1) at time t+1 is time-series image 33d.

[0021] Note that in Figure 3, the time-series images 32b, 32c, 32d, 33b, 33c, and 33d acquired from the rear camera 10b, the right side camera 10c, and the left side camera 10d, respectively, are horizontally flipped. However, when implementing the invention, it is not necessary to flip the images horizontally.

[0022] For simplification, Figure 3 places the vanishing point and the epipole (explained later in the feature point matching section 230) at the same coordinates on the image plane. However, in actual captured images, the coordinates of the vanishing point and the epipole do not necessarily coincide. Also, for simplification, Figure 3 shows the image taken with the bottom edge parallel to the ground. However, in actual captured images, it is not necessary to take the image with the bottom edge parallel to the ground.

[0023] Returning to the explanation of Figure 1, the integrated processing unit 200 includes a feature point extraction unit 210, a reference frame switching unit 220, a feature point matching unit 230, and a three-dimensional information generation unit 240. The integrated processing unit 200 also includes a CPU (not shown) that controls each of the units 210 to 240, and RAM as the main memory medium. CPU is an abbreviation for Central Processing Unit. RAM is an abbreviation for Random Access Memory.

[0024] The feature point extraction unit 210 is implemented, for example, as a function of the SoC / FPGA and has a feature point extraction function that extracts feature points from an image based on the input image. SoC is an abbreviation for System On Chip. FPGA is an abbreviation for Field Programmable Gate Array. The feature point extraction function can be implemented using common methods such as SIFT or ORB. SIFT is an abbreviation for Scale-Invariant Feature Transform. ORB is an abbreviation for Oriented FAST and Rotated BRIEF.

[0025] Figure 4 shows a conceptual diagram of extracting feature points from an image img(t+1)a captured by the front camera 10a. Figure 4(A) shows the input image, and Figure 4(B) shows the feature point cloud. When an image img(t+1)a like the one in Figure 4(A) is used as the input image for the feature point extraction function, a feature point cloud fp(t+1)a like the point cloud 41 shown in Figure 4(B) will be extracted.

[0026] Generally, feature points are extracted from edges in the input image, such as the contours of an object or the boundaries between multiple objects, using the algorithm employed for feature point extraction. Furthermore, feature points are more likely to be extracted from areas with maximum or minimum brightness, such as tiles or mosaic patterns. Therefore, as shown in Figures 4(A) and 4(B), feature points do not necessarily have to be extracted from the edges of the input image, and the algorithm used for feature point extraction is not limited to these specific areas.

[0027] Returning to the explanation of Figure 1, the feature point extraction unit 210 outputs the feature point groups fp(t) and fp(t+1), extracted from the time-series images img(t) and img(t+1), respectively, to the reference frame switching unit 220.

[0028] The reference frame switching unit 220 is implemented, for example, as a function of the SoC / FPGA. Here, among the feature point groups fp(t) and fp(t+1) extracted from the time series images img(t) and img(t+1), respectively, the feature point group that serves as the reference for the correspondence point search process 232 described later is called the reference feature point group fp_ref. The image at the time corresponding to the reference feature point group fp_ref is called the reference image img_ref. The dataset of the reference feature point group and the reference image is called the reference frame. The feature point group that is the target of the correspondence point search process 232 is called the search target feature point group fp_sub, and the image at the time corresponding to the search target feature point group is called the search target image img_sub. The dataset of the search target feature point group and the search target image is called the search target frame.

[0029] The reference frame switching unit 220 has control registers (not shown) that hold the shooting direction of each camera of the imaging unit 10, as well as the write destination pointers for each frame of the reference frame and the search target frame.

[0030] Furthermore, the reference frame switching unit 220 has a reference frame switching function that switches the write destination pointer of the reference frame in the control register and the write destination pointer of the frame to be searched. The reference frame switching function switches the write destination pointer of the reference frame in the control register and the write destination pointer of the frame to be searched based on the direction of travel signal of the moving object input from the direction of travel detection unit 30 and the shooting direction of each camera.

[0031] Furthermore, the reference frame switching unit 220 sets one of the time-series feature point groups fp(t) and fp(t+1) input from the feature point extraction unit 210 as the reference feature point group fp_ref, and the other as the search target feature point group fp_sub. The reference frame switching unit 220 outputs the reference feature point group fp_ref and the search target feature point group fp_sub to the feature point matching unit 230.

[0032] Here, the reference frame switching unit 220 assumes that the shooting position of each input frame transitions from the shooting position of the reference image to the shooting position of the search target image. The reference frame switching unit 220 switches the reference frame and the search target frame so that the direction of transition of the image shooting position is opposite to the direction of the camera's shooting direction. The reference frame switching unit 220 is an example of a reference image switching means that switches the reference image based on information representing the direction of movement of a moving object.

[0033] Here, we assume that the direction of movement of the moving object 1000 is forward. In this case, the reference frame switching unit 220 uses the time-series image img(t+1)a as the reference image and the time-series image img(t)a as the search target image when the time-series images img(t)a and img(t+1)a are captured by the front camera 10a. In this case, the reference frame switching unit 220 uses the time-series image img(t)b as the reference image and the time-series image img(t+1)b as the search target image when the time-series image is captured by the rear camera 10b. In this case, the reference frame switching unit 220 uses the time-series image img(t)c as the reference image and the time-series image img(t+1)c as the search target image when the time-series image is captured by the right side camera 10c. In this case, the reference frame switching unit 220 uses the time-series image img(t)d as the reference image and the time-series image img(t+1)d as the search target image for each time-series image captured by the left side camera 10d.

[0034] Here, we assume that the direction of movement of the moving object 1000 is backward. In this case, the reference frame switching unit 220 uses the time-series image img(t)a as the reference image and the time-series image img(t+1)a as the search target image when the time-series images img(t)a and img(t+1)a are captured by the front camera 10a. In this case, the reference frame switching unit 220 uses the time-series image img(t+1)b as the reference image and the time-series image img(t)b as the search target image when the time-series image is captured by the rear camera 10b. In this case, the reference frame switching unit 220 uses the time-series image img(t+1)c as the reference image and the time-series image img(t)c as the search target image when the time-series image is captured by the right side camera 10c. In this case, the reference frame switching unit 220 uses the time-series image img(t+1)d as the reference image and the time-series image img(t)d as the image to be searched, based on the time-series images captured by the left side camera 10d.

[0035] Figure 5 shows an example of switching between the reference image img_ref and the search target image img_sub from the time-series images img(t)a and img(t+1)a captured by the front camera 10a according to the direction of movement of the mobile body 1000. Figure 5(A) shows the case when the state of the shift lever 30a is Drive or Low and the mobile body 1000 is moving forward. Figure 5(B) shows the case when the state of the shift lever 30a is Reverse and the mobile body 1000 is moving backward.

[0036] The reference frame switching unit 220 outputs a reference image 51 and a search target image 52 when time-series images 32a and 33a are input, for example, when the direction of movement of the moving object 1000 is forward (see Figure 5(A)). Also, the reference frame switching unit 220 outputs a reference image 51' and a search target image 52' when time-series images 32a' and 33a' are input, for example, when the direction of movement of the moving object 1000 is backward (see Figure 5(B)).

[0037] Furthermore, if the image capture position is switched to transition in a direction opposite to the camera's shooting direction according to the direction of movement of the moving object 1000, the capture position will transition to move away from the captured stationary object, resulting in an appearance as shown in Figure 5.

[0038] Here, the direction in which the camera moves from the shooting position of the reference image to the shooting position of the image to be searched will be referred to as the direction of movement of the shooting position. In particular, in Embodiment 1, the direction of movement of the shooting position is defined as positive when it is opposite to the camera's shooting direction, and negative when it is in the direction that follows the camera's shooting direction.

[0039] As described above, by switching between the reference frame and the search target frame based on the direction of movement of the moving object, the direction of movement of the shooting position can be restricted to one direction regardless of changes in the direction of movement of the moving object. This means of restricting the direction of movement of the shooting position is an example of a means of restricting the direction of movement of the shooting position between the first position and the second position. Furthermore, by restricting the direction of movement of the shooting position, the search range in the corresponding point search process 232 described later can be limited in advance.

[0040] Furthermore, in Embodiment 1, the reference frame and the search target frame are switched so that the direction of movement of the shooting position is opposite to the direction of the camera's shooting direction. By doing so, the shooting position of the search target frame is taken from behind the shooting position of the reference frame, so that the feature points corresponding to the reference feature points do not fall outside the shooting range of the search target frame, and corresponding point search can be performed efficiently.

[0041] Returning to the explanation of Figure 1, the feature point matching unit 230 is composed of, for example, a CPU, RAM as the main memory medium, and a ROM 233 that stores the search range map 233a described later. ROM is an abbreviation for Read Only Memory. The feature point matching unit 230 performs cell division processing 231 and corresponding point search processing 232 when the CPU executes a program. The feature point matching unit 230 is an example of a feature point matching means that selects a reference image and a search target image from the first image and the second image, respectively, and associates feature points.

[0042] The cell splitting process 231 is a process that divides each frame of the reference frame and the search target frame input from the reference frame switching unit 220 into cells within a predetermined range. For example, if a 1920 x 1080 pixel frame is input, the cell splitting process 231 should divide it into 16 x 9 cells within a range of, for example, 120 x 120.

[0043] The corresponding point search process 232 is a process that selects a feature point from the reference feature point group fp_ref and then searches for a corresponding point in the target feature point group fp_sub to perform feature point mapping (hereinafter referred to as matching). The matching can be implemented using a general method such as a brute-force matcher.

[0044] However, the correspondence point search process 232 performs matching within a limited range based on the search range map 233a stored in the ROM 233. The limitation of the matching range based on the search range map 233a will be explained using Figure 6.

[0045] Figure 6 shows the reference frame and the search target frame formed by the image captured by the front camera 10a. Figure 6(A) shows the reference feature point cloud, and Figure 6(B) shows the search target feature point cloud.

[0046] In Figure 6(A), p1, p2, and p3 are the respective feature points included in the reference feature point group fp_ref{p1,p2,p3}. In Figure 6(B), p1', p2', and p3' are the respective feature points included in the search target feature point group fp_sub{p1',p2',p3'}.

[0047] Furthermore, in Figure 6, e and e' are the points where the baselines from each camera and the image plane coincide when each frame of the reference frame and the frame being searched is captured (hereinafter referred to as epipoles). Also, the lines connecting each feature point p1, p2, p3 to epipol e, or the lines connecting each feature point p1', p2', p3' to epipol e', are epipolar lines.

[0048] As shown in Figure 6, the search range map 233a is given with a unique size for each cell. For example, the search range maps 233a corresponding to cells 601, 602, and 603 can be given as search ranges 611(+1,+1), 612(+2,-1), and 613(-2,-2), respectively, starting from the position of the corresponding cell. In this case, as shown in Figure 6(B), in Embodiment 1, it is desirable that the direction in which the search range is limited is toward the epipole. The search range map 233a is an example of a search range based on predetermined range information.

[0049] The correspondence point search process 232 selects one feature point from the cell-divided reference frame and performs matching on the target frame within a limited range based on the search range map 233a that corresponds to the cell containing the selected feature point.

[0050] As described above, by restricting the direction of movement of the shooting position to one direction and performing the corresponding point search process 232 within a predetermined limited range based on the restricted direction of movement of the shooting position, the computational amount in the feature point matching process can be reduced.

[0051] Furthermore, by referring to a predetermined search range map 233a, it is not necessary to set the search range as an internal process, and the correspondence point search process 232 can be started early.

[0052] Furthermore, since the direction of movement of the shooting position is restricted to one direction, there is no need to change the search range when the direction of movement of the moving body 1000 changes. Therefore, since only one search range map 233a needs to be stored regardless of the direction of movement of the moving body 1000, the ROM capacity required to store the search range map 233a can be reduced. In addition, by performing cell division, the search range map 233a can be provided for each cell rather than each pixel, thus reducing the ROM capacity required to store the search range map 233a.

[0053] Subsequently, the corresponding point search process 232 repeatedly performs matching on all feature points included in the reference feature point group fp_ref.

[0054] Furthermore, when the mobile unit 1000 moves forward or backward parallel to the ground without rotating, ideally the coordinates of the epipole remain unchanged and are uniquely determined by the camera's mounting position and the angle it makes with the ground. Figure 7 shows the direction of movement of the shooting position, the position of the epipole, and the position of the feature point in this case.

[0055] Figure 7 illustrates the case where the mobile object 1000 is moving forward or backward. In Figure 7, the points e and e' where the baseline connecting the camera position C in the reference frame and the camera position C' in the search target frame coincides with the image plane of each frame are defined as epipoles 71 and 71'. In this case, since the mobile object 1000 moves forward or backward parallel to the ground, each epipole e and e' lies on the camera position. Furthermore, since the mobile object 1000 moves forward or backward without rotating, ideally the coordinates of epipoles e and e' do not change.

[0056] On the other hand, the points p and p' where the lines connecting the camera position C or C' and the real-space coordinate P of the feature point coincide with the image plane of each frame are defined as feature points 72 and 72' on the image. The lines connecting the epipoles e and e' and the feature points p and p' are defined as epipolar lines 73 and 73'. In this case, due to the epipolar constraint, the epipolar lines 73 and 73' lie on the same plane, and in Embodiment 1, since the moving body 1000 moves forward or backward parallel to the ground without rotating, the position of the epipolar lines on the image does not change.

[0057] Therefore, in Embodiment 1, by restricting the direction of movement of the shooting position to the positive direction, the search range when transitioning from feature point p to p' can be limited to the direction toward the epipole, thereby reducing the computational complexity of the corresponding point search process 232.

[0058] In reality, the positions of epipoles e and e' will vary due to vibrations, for example, when the mobile vehicle 1000 is in motion. If this variation needs to be considered, the amount of deviation from the uniquely determined epipol position, based on the camera's mounting position and the angle it makes with the ground, can be calculated, and if it falls within a certain threshold, a corresponding point search process 232 can be performed.

[0059] Returning to the explanation of Figure 1, the feature point matching unit 230 outputs pairs of feature points in the reference feature point group and feature points in the search target feature point group, which have been associated by the correspondence point search process 232, as matching feature point pairs to the three-dimensional information generation unit 240.

[0060] The three-dimensional information generation unit 240 is composed of, for example, a CPU, RAM as the main memory medium, and ROM (not shown) for storing internal parameters of the camera, etc. The three-dimensional information generation unit 240 performs camera pose estimation 241 and motion parallax distance measurement 242 when the CPU executes a program.

[0061] The camera pose estimation 241 calculates the camera's external parameters based on the matching feature point pairs input from the feature point matching unit 230 and the camera's internal parameters stored in ROM. The camera's external parameters are, for example, the amount of camera movement and rotation angle relative to the shooting position of the reference frame. The camera pose estimation 241 outputs the calculated camera's external parameters to the motion parallax distance meter 242. The camera pose estimation 241 can be implemented using a common method used in SfM, such as the 8-point method or the 5-point method. The camera pose estimation 241 is an example of a shooting position and pose calculation means that calculates information regarding the position and pose of the imaging means. The camera pose estimation 241 may also generate information representing the direction of movement of the moving body based on the information regarding the position and pose of the imaging means.

[0062] The motion parallax distance measuring system 242 generates three-dimensional coordinates of feature points relative to the shooting position of a reference frame, based on the camera's external parameters, the camera's internal parameters, and matching feature point pairs. Here, the camera's external parameters used by the motion parallax distance measuring system 242 are the camera's external parameters input from the camera pose estimation system 241. The camera's internal parameters used by the motion parallax distance measuring system 242 are the camera's internal parameters stored in ROM. The matching feature point pairs used by the motion parallax distance measuring system 242 are the matching feature point pairs input from the feature point matching unit 230. The motion parallax distance measuring system 242 can be implemented using general methods used in SfM, such as triangulation in stereo vision.

[0063] Although this explanation describes a case where the integrated processing unit 200, the feature point matching unit 230, and the three-dimensional information generation unit 240 each have their own CPUs, all of the processes described above may be performed using a single CPU.

[0064] Furthermore, this example describes an implementation where the feature point extraction unit 210 and the reference frame switching unit 220 are implemented using an SoC / FPGA, and the feature point matching unit 230 and the three-dimensional information generation unit 240 are implemented using a CPU. However, the present invention does not limit the implementation means and can be implemented using any device.

[0065] The direction of travel detection unit 30 has, for example, a shift lever 30a mounted on the vehicle. The shift lever 30a is a lever for shifting the gear of the transmission of the mobile body 1000. When the state of the shift lever 30a is Drive or Low, the mobile body 1000 moves forward. When the state of the shift lever 30a is Reverse, the mobile body 1000 moves backward. The direction of travel detection unit 30 detects the state of the shift lever 30a and can detect the direction of travel of the mobile body 1000 based on the detected state of the shift lever 30a. For example, the direction of travel detection unit 30 determines that the direction of travel of the mobile body 1000 is forward when the state of the shift lever 30a is Drive or Low, and determines that the direction of travel of the mobile body 1000 is backward when the state of the shift lever 30a is Reverse. The direction of travel detection unit 30 also outputs the detected direction of travel of the mobile body 1000 to the reference frame switching unit 220.

[0066] Figure 8 is a flowchart illustrating a series of operations of the integrated processing unit 200 in Embodiment 1. Each step shown in Figure 8 is performed sequentially by the CPU of the integrated processing unit 200 executing a computer program.

[0067] In step S101, the integrated processing unit 200 acquires time-series images img(t) and img(t+1) from the imaging unit 10. Then the process proceeds to step S102.

[0068] In step S102, the integration processing unit 200 extracts the time-series feature point clouds fp(t) and fp(t+1) from the time-series images img(t) and img(t+1). Then the process proceeds to step S103.

[0069] In step S103, the integrated processing unit 200 obtains the direction of movement of the moving body 1000 from the direction of movement detection unit 30. Then the process proceeds to step S110.

[0070] In step S110, the integrated processing unit 200 takes the direction of movement of the moving object 1000, the time-series images img(t), img(t+1), and the time-series feature point clouds fp(t), fp(t+1) as input and performs processing by the reference frame switching unit 220. The reference frame switching unit 220 outputs the reference feature point cloud fp_ref and the search target feature point cloud fp_sub as processing results. Then proceed to step S120.

[0071] In step S120, the integrated processing unit 200 takes the reference feature point cloud fp_ref, the search target feature point cloud fp_sub, and the search range map 233a as input and performs processing by the feature point matching unit 230. The feature point matching unit 230 outputs matching feature point pairs as processing results. The process then proceeds to step S130.

[0072] In step S130, the integrated processing unit 200 takes the matching feature point pair and the camera's internal parameters as input and performs processing by the three-dimensional information generation unit 240. The three-dimensional information generation unit 240 outputs, as processing results, the three-dimensional coordinates of each feature point relative to the shooting position C of the reference frame, and the camera's external parameters at the shooting position C' of the search target frame relative to the reference frame. After that, the integrated processing unit 200 completes the series of processes.

[0073] Figure 9 is a flowchart illustrating the series of operations of the reference frame switching unit 220 in Embodiment 1. The processing of each step shown in Figure 9 is performed sequentially by the SoC / FPGA of the reference frame switching unit 220.

[0074] In step S111, the reference frame switching unit 220 determines whether the camera's shooting direction includes the same direction as the moving object 1000's direction of travel, based on the direction of travel of the moving object 1000 and the camera's shooting direction acquired in step S103. If the reference frame switching unit 220 determines that the camera's shooting direction includes the same direction as the moving object 1000's direction of travel, the process in step S112 is executed. If the reference frame switching unit 220 determines that the camera's shooting direction does not include the same direction as the moving object 1000's direction of travel, the process in step S113 is executed. Note that in Embodiment 1, the camera captures the same or opposite direction as the moving object 1000's direction of travel. Therefore, if the camera's shooting direction does not include the same direction as the moving object 1000's direction of travel, the imaging unit 10 will be capturing the opposite direction to the moving object 1000's direction of travel.

[0075] In step S112, the reference frame switching unit 220 outputs the feature point cloud fp(t) from the time series feature point clouds fp(t) and fp(t+1) acquired in step S102 to the integrated processing unit 200 as the search target feature point cloud fp_sub. Also in step S112, the reference frame switching unit 220 outputs the feature point cloud fp(t+1) from the time series feature point clouds fp(t) and fp(t+1) acquired in step S102 to the integrated processing unit 200 as the reference feature point cloud fp_ref. After that, the processing ends.

[0076] In step S113, the reference frame switching unit 220 outputs the feature point cloud fp(t) from the time series feature point clouds fp(t) and fp(t+1) acquired in step S102 to the integrated processing unit 200 as the reference feature point cloud fp_ref. Also in step S113, the reference frame switching unit 220 outputs the feature point cloud fp(t+1) from the time series feature point clouds fp(t) and fp(t+1) acquired in step S102 to the integrated processing unit 200 as the search target feature point cloud fp_sub. After that, the processing ends.

[0077] Figure 10 is a flowchart illustrating a series of operations of the feature point matching unit 230 in Embodiment 1. Each step shown in Figure 10 is performed sequentially by the CPU of the feature point matching unit 230 executing a computer program.

[0078] In step S121, the feature point matching unit 230 divides the reference feature point group fp_ref and the search target feature point group fp_sub, respectively, obtained in step S110 into cells. Then proceed to step S122.

[0079] In step S122, the feature point matching unit 230 selects one arbitrary cell from the cell-divided reference feature point group fp_ref. Then proceed to step S123.

[0080] In step S123, the feature point matching unit 230 selects one arbitrary feature point from the cells selected in step S122. Then the process proceeds to step S124.

[0081] In step S124, the feature point matching unit 230 refers to the search range map 233a and searches for corresponding feature points from the search target feature point group fp_sub within the search range corresponding to the cell selected in step S122, to match the feature point selected in step S123. Then proceed to step S125.

[0082] In step S125, the feature point matching unit 230 determines whether the correspondence point search has been completed for each feature point included in the cell selected in step S122. If the feature point matching unit 230 determines that the correspondence point search has not been completed, the process in step S123 is executed. If the feature point matching unit 230 determines that the correspondence point search has been completed, the process in step S126 is executed.

[0083] In step S126, the feature point matching unit 230 determines whether each process from steps S122 to S125 has been completed for each cell divided in step S121. If the feature point matching unit 230 determines that the process is not complete, the process in step S122 is executed. If the feature point matching unit 230 determines that the process is complete, the process in step S127 is executed.

[0084] In step S127, the feature point matching unit 230 outputs the matching feature point pairs that have been matched by the correspondence point search to the integration processing unit 200. After that, the processing ends.

[0085] Figure 11 is a flowchart illustrating a series of operations of the three-dimensional information generation unit 240 in Embodiment 1. Each step shown in Figure 11 is performed sequentially by the CPU of the three-dimensional information generation unit 240 executing a computer program.

[0086] In step S131, the three-dimensional information generation unit 240 estimates the camera's orientation based on the matching feature point pair acquired in step S120 and the camera's internal parameters. Here, the three-dimensional information generation unit 240 acquires external camera parameters such as the amount of camera movement and rotation angle relative to the shooting position of the reference frame as estimation results. The process then proceeds to step S132.

[0087] In step S132, the three-dimensional information generation unit 240 calculates the three-dimensional coordinates of each feature point relative to the shooting position of the reference frame, based on the matching feature point pair, the camera's intrinsic parameters, and the camera's extrinsic parameters. The process then proceeds to step S133.

[0088] In step S133, the three-dimensional information generation unit 240 outputs the three-dimensional coordinates of each feature point and the camera's external parameters to the integrated processing unit 200. After that, the processing ends.

[0089] As described above, according to Embodiment 1, by switching between the reference frame and the search target frame based on the direction of movement of the moving object, the direction of movement of the shooting position is restricted to one direction regardless of changes in the direction of movement of the moving object. This makes it possible to pre-limit the search range in feature point matching.

[0090] Furthermore, by referring to a predetermined search range, it eliminates the need to set the search range as an internal process, allowing the search for corresponding points to begin early. In addition, performing the search for corresponding points within a limited search range reduces the computational complexity of the feature point matching process.

[0091] Therefore, by starting the feature point matching process early and reducing the computational load, the feature point matching process can be sped up. This, in turn, can speed up SfM and vSLAM processing.

[0092] Furthermore, according to Embodiment 1, since the direction of movement of the shooting position is restricted to one direction, there is no need to change the search range when the direction of movement of the moving body changes, and it is sufficient to store one search range map 233a regardless of the direction of movement of the moving body. Therefore, the ROM capacity required to hold the search range map 233a can be reduced.

[0093] Furthermore, by performing cell division, the search range map 233a can be provided for each cell rather than each pixel, thus reducing the ROM capacity required to store the search range map 233a.

[0094] Therefore, by minimizing the information used for the search range when searching for corresponding points, feature point matching can be implemented at a low cost. This, in turn, enables the implementation of SfM and vSLAM at a low cost.

[0095] Furthermore, according to Embodiment 1, the reference frame and the search target frame are switched so that the direction of movement of the shooting position is toward the camera. In this way, the shooting position of the search target frame is captured from behind the shooting position of the reference frame, so that the feature points corresponding to the reference feature points do not fall outside the shooting range of the search target frame, and correspondence point search can be performed efficiently.

[0096] <Embodiment 2> Next, Embodiment 2 of the present invention will be described. In Embodiment 2, the amount of camera movement relative to the shooting position of the reference frame calculated by the three-dimensional information generation unit 240 is fed back to the reference frame switching unit 220'. Embodiment 2 is configured to switch the reference frame for feature point matching in this way.

[0097] Figure 12 is a configuration diagram of the mobile body 1000' according to Embodiment 2. The mobile body 1000' is, for example, a vehicle. The imaging unit 10, feature point extraction unit 210, feature point matching unit 230, and three-dimensional information generation unit 240 in Figure 12 are the same as in Figure 1.

[0098] The integrated processing unit 200 has a reference frame switching unit 220' instead of the reference frame switching unit 220 in Figure 1. In addition to the configuration of the reference frame switching unit 220, the reference frame switching unit 220' has the following functions. Specifically, the reference frame switching unit 220' has a reference frame switching function that switches the write destination pointer of the reference frame and the write destination pointer of the search target frame based on external camera parameters input from the three-dimensional information generation unit 240. The external camera parameters are the amount of camera movement and rotation angle relative to the shooting position of the reference frame, as explained in Embodiment 1.

[0099] Here, the camera's movement is given such that the direction from the camera's position C, as shown in Figure 7, towards the image plane img_ref is considered positive. Note that, generally, in stereo vision, there are coordinate systems where the direction from the image plane towards the camera is considered positive; in that case, one should consider the positive and negative directions of the camera's movement as described in Embodiment 2 to be reversed.

[0100] Figure 13 is a flowchart illustrating a series of operations of the integrated processing unit 200 in Embodiment 2. Steps S101, S102, and S110-S130 in Figure 13 are the same as in Figure 8, so their explanation is omitted.

[0101] In step S103', the integrated processing unit 200 obtains the camera's movement amount from the three-dimensional information generation unit 240. Then the process proceeds to step S110.

[0102] Figure 14 is a flowchart illustrating the series of operations of the reference frame switching unit 220' in Embodiment 2. The processing of each step shown in Figure 14 is performed sequentially by the SoC / FPGA of the reference frame switching unit 220'.

[0103] In step S111', the reference frame switching unit 220' determines whether the camera movement is positive or negative based on the camera movement amount acquired in step S103'. If the reference frame switching unit 220' determines that the camera movement is positive, the process in step S114 is executed. If the reference frame switching unit 220' determines that the camera movement is not positive, i.e., the camera movement is negative, the process is terminated. In Embodiment 2, the camera movement is considered positive when the direction from the camera toward the image plane. Therefore, if the camera movement is positive, it means that the imaging unit 10 is capturing images in the same direction as the direction of travel of the moving body 1000, and if the camera movement is negative, it means that the imaging unit 10 is capturing images in the opposite direction to the direction of travel of the moving body 1000.

[0104] In step S114, the reference frame switching unit 220' switches the write destination pointers for the time-series feature point groups fp(t) and fp(t+1) acquired in step S102. Specifically, the reference frame switching unit 220' switches the write destination pointer of the feature point group currently being searched for, fp_sub, to the write destination pointer of the reference feature point group fp_ref. The reference frame switching unit 220' also switches the write destination pointer of the feature point group currently being searched for, fp_ref, to the write destination pointer of the feature point group being searched for, fp_sub. In this way, in step S114, the reference frame switching unit 220' switches the write destinations of each time-series feature point group fp(t) and fp(t+1) and outputs them to the integrated processing unit 200. After that, the reference frame switching unit 220' terminates processing.

[0105] Thus, in Embodiment 2, the amount of camera movement relative to the shooting position of the reference frame calculated by the three-dimensional information generation unit 240 is fed back to the reference frame switching unit 220'. When processing in the reference frame switching unit 220', the destination for writing the time-series feature point cloud is switched based on the sign of the camera's movement amount, thereby enabling the reference frame to be switched without using the direction of travel signal of the moving object 1000.

[0106] As described above, in Embodiment 2, camera external parameters calculated when generating three-dimensional information are fed back, and the order of time-series images and feature point clouds is rearranged based on the amount of camera movement. As a result, according to Embodiment 2, a reference frame can be determined without detecting the direction of travel signal of the moving object.

[0107] Therefore, since the direction of movement of the shooting position can be restricted to one direction based on the amount of camera movement without detecting the direction of movement signal of the moving object, the feature point matching process can be accelerated in the same way as in Embodiment 1. In turn, SfM and vSLAM processing can be accelerated.

[0108] <Embodiment 3> Next, Embodiment 3 of the present invention will be described. Embodiment 3 detects the turning direction of the mobile body 1000'' based on the steering angle of the handle 30b mounted on the mobile body 1000'' and inputs it to the corresponding point search process 232'''. Embodiment 3 is configured to perform a corresponding point search by referring to the search range map 233a corresponding to the turning state of the mobile body 1000''. The direction of travel detection unit 30'' which detects the steering angle of the handle 30b is an example of a turning direction detection means for detecting the turning direction of the mobile body. The feature point matching unit 230 searches for each feature point on the search target image within a limited search range based on the detection result of the direction of travel detection unit 30''.

[0109] Figure 15 is a configuration diagram of the mobile body 1000'' according to Embodiment 3. The mobile body 1000'' is, for example, a vehicle. The imaging unit 10, feature point extraction unit 210, reference frame switching unit 220, and three-dimensional information generation unit 240 in Figure 15 are the same as in Figure 1.

[0110] The feature point matching unit 230 has a correspondence point search process 232'' instead of the correspondence point search process 232 in Figure 1. The moving body 1000'' has a direction of travel detection unit 30'' instead of the direction of travel detection unit 30 in Figure 1. The direction of travel detection unit 30'' detects the state of the shift lever 30a as well as the steering angle of the steering wheel 30b. The steering wheel 30b determines the turning direction of the moving body 1000'' based on its steering angle. By detecting the steering angle of the steering wheel 30b, the direction of travel detection unit 30'' can output a turning direction signal for the moving body 1000''. The correspondence point search process 232'' performs a determination of the turning state of the moving body 1000'' in addition to the configuration of the correspondence point search process 232. The corresponding point search process 232'' determines the turning state of the moving body 1000'' based on the turning direction signal of the moving body 1000'' input from the direction of travel detection unit 30'', determining whether the moving body 1000'' is turning left, turning right, or not turning.

[0111] Furthermore, the corresponding point search process 232'' performs matching within a limited range by referring to the search range map 233a corresponding to the turning state of the mobile body 1000'' stored in the ROM 233, based on the determination result of the turning state. The search range map 233a has multiple search range maps with different search ranges, for example, a search range map for when turning right, a search range map for when turning left, and a search range map for when moving straight. The search range map 233a corresponding to the turning state of the mobile body 1000'' is, for example, one of the search range map for when turning right, a search range map for when turning left, and a search range map for when moving straight.

[0112] In Embodiment 3, the direction of movement for the moving body (forward / backward) is not distinguished, because, as explained in Embodiment 1, the direction of movement of the shooting position can be restricted to one direction regardless of the direction of movement for the moving body (forward / backward).

[0113] Ideally, when a moving object rotates and moves forward or backward parallel to the ground, the vertical component of the epipole's coordinates remains unchanged and is uniquely determined by factors such as the camera's mounting position and the angle it makes with the ground. On the other hand, when a moving object rotates and moves backward, for example, the horizontal component of the epipole's coordinates moves in the same direction as the rotation. Also, when a moving object rotates and moves forward, for example, the horizontal component of the epipole's coordinates moves in the opposite direction to the rotation.

[0114] For simplicity, Figure 16 shows the direction of movement of the shooting position, the position of the epipole, and the position of the feature point when the moving object 1000'' is, for example, moving backward while turning to the right.

[0115] Figure 16 illustrates the case where the moving object 1000'' is turning to the right. In Figure 16, the points e and e' where the baseline connecting the camera position C in the reference frame and the camera position C' in the search target frame coincides with the image plane of each frame are defined as epipoles 161 and 161'. At this time, since the moving object 1000'' is moving backward parallel to the ground, each epipole e and e' lies on the camera position. Also, since the moving object 1000'' is moving backward while turning to the right, the coordinates of epipoles e and e' in the direction parallel to the ground move to the right.

[0116] On the other hand, the points p and p' where the lines connecting the camera position C or C' and the real-space coordinate P of the feature point coincide with the image plane of each frame are defined as feature points 162 and 162' on the image. The lines connecting the epipoles e and e' and the feature points p and p' are defined as epipolar lines 163 and 163'. In this case, due to the epipolar constraint, the epipolar lines 163 and 163' lie on the same plane, but in Embodiment 3, since the moving body 1000'' rotates, the position and inclination of the epipolar lines on the image change.

[0117] Therefore, in Embodiment 3, with respect to the search range when transitioning from feature point p to p', the direction toward the epipole cannot be uniquely determined. However, when the moving object is moving backward while turning right, as shown in Figure 16, when the feature point transitions from p to p', the coordinates of the feature point parallel to the ground move to the right, and the coordinates of the feature point perpendicular to the ground move toward the camera position.

[0118] Even when a moving object is moving backward while turning left, we consider it symmetrically: the coordinates of the feature points parallel to the ground move to the left, and the coordinates of the feature points perpendicular to the ground move towards the camera position.

[0119] Therefore, the search range map is distinguished for right turns, left turns, and straight-ahead movements. For right turns, the vertical component of the search range is limited to the direction approaching the camera position, and the parallel component is limited to the direction approaching the right edge of the image. On the other hand, the search range for left turns is considered symmetrically, and the parallel component is limited to the direction approaching the left edge of the image. The search range for straight-ahead movements is limited in the same way as in Embodiment 1.

[0120] By defining the search range in this way, it is possible to perform a corresponding point search within an appropriately limited search range based on the rotation state of the moving object, thereby reducing the computational complexity of the corresponding point search process 232.

[0121] The direction of travel detection unit 30'' has, in addition to the configuration of the direction of travel detection unit 30 in Figure 1, a handle 30b mounted on the vehicle, for example. Furthermore, the direction of travel detection unit 30'' detects the turning direction of the moving body 1000'' based on the steering angle of the handle 30b. For example, if the steering angle of the handle 30b is tilted clockwise, the turning direction of the moving body 1000'' is determined to be a right turn. If the steering angle of the handle 30b is tilted counterclockwise, the turning direction of the moving body 1000'' is determined to be a left turn. If the steering angle of the handle 30b is not tilted, the turning direction of the moving body 1000'' is determined to be not turning.

[0122] Generally speaking, turning means moving in an arc, so even if the steering angle of the handle is clockwise, the generally accepted direction of turning of a moving object changes depending on whether the object is moving forward or backward. However, here we define turning as described above, regardless of whether the object is moving forward or backward.

[0123] The criteria for determining the steering angle of the steering wheel vary depending on the design, but for example, the angle when the steering wheel is rotated clockwise from 0° can be defined as positive, and the angle when it is rotated counterclockwise can be defined as negative. Furthermore, a certain threshold for rotation in the positive direction (right turn threshold) and a certain threshold for rotation in the negative direction (left turn threshold) can be established, and if the right turn threshold is exceeded, it can be determined as a right turn, and if the left turn threshold is exceeded, it can be determined as a left turn.

[0124] Furthermore, the direction of travel detection unit 30'' outputs the turning direction of the moving body 1000'' to the corresponding point search process 232''''.

[0125] Figure 17 is a flowchart illustrating a series of operations of the integrated processing unit 200 according to Embodiment 3. Steps S101, S102, and S110 to S130 in Figure 17 are the same as in Figure 8, so their explanation is omitted.

[0126] In step S103'', in addition to the processing in step S103 shown in Figure 8, the integrated processing unit 200 obtains the turning direction of the moving body 1000'' from the direction of travel detection unit 30''. Then the process proceeds to step S110.

[0127] Figure 18 is a flowchart illustrating a series of operations of the feature point matching unit 230 according to Embodiment 3. Steps S121 to S123 and S125 to S127 in Figure 18 are the same as in Figure 10, so their explanation is omitted.

[0128] The process in step S124'' involves referring to the search range map 233a corresponding to the rotation direction of the moving object 1000'' obtained in step S103'', and searching for corresponding points within the search range corresponding to the cell selected in step S122. The details of the process in step S124'' are explained below. The process in step S124'' includes the processes of steps S1241 to S1245.

[0129] In step S1241, the feature point matching unit 230 determines whether the rotation direction of the moving body 1000'' is a right turn. If the feature point matching unit 230 determines that it is a right turn, the process in step S1243 is executed. If the feature point matching unit 230 determines that it is not a right turn, the process in step S1242 is executed.

[0130] In step S1242, the feature point matching unit 230 determines whether the 1000'' rotation direction of the moving object is a left turn. If the feature point matching unit 230 determines that it is a left turn, the process in step S1244 is executed. If the feature point matching unit 230 determines that it is not a left turn, the process in step S1245 is executed.

[0131] In step S1243, the feature point matching unit 230 refers to the search range map during rightward rotation. Then, in step S1243, the feature point matching unit 230 searches for corresponding feature points from the search target feature point group fp_sub within the search range corresponding to the cell selected in step S122, and the feature points corresponding to the feature points selected in step S123. After that, the process proceeds to step S125.

[0132] In step S1244, the feature point matching unit 230 refers to the search range map during leftward rotation. Then, in step S1244, the feature point matching unit 230 searches for corresponding feature points from the search target feature point group fp_sub within the search range corresponding to the cell selected in step S122, and for the feature points corresponding to the feature points selected in step S123. After that, the process proceeds to step S125.

[0133] In step S1245, the feature point matching unit 230 refers to the search range map for straight-line movement. Then, in step S1245, the feature point matching unit 230 searches for corresponding feature points from the search target feature point group fp_sub within the search range corresponding to the cell selected in step S122, and the feature points corresponding to the feature points selected in step S123. After that, the process proceeds to step S125.

[0134] Thus, in the corresponding point search process 232'', the turning state of the moving object 1000'' is determined based on the turning direction of the moving object 1000'' detected by the direction of travel detection unit 30''. Then, by referring to the search range map corresponding to the turning state of the moving object 1000'', the range to be limited can be changed in response to changes in the turning direction of the moving object 1000'' and the corresponding point search can be performed.

[0135] As explained above, according to Embodiment 3, each search range corresponding to the rotation state of the moving body is defined, and the search range is changed in response to a change in the rotation direction of the moving body. In this way, according to Embodiment 3, even when the rotation direction of the moving body changes, the search for corresponding points can be started early.

[0136] Therefore, even when the rotation direction of the moving object changes, each search range corresponding to the rotation state of the moving object is predetermined, and the search range is changed in response to the change in the rotation direction of the moving object. This makes it possible to speed up the feature point matching process in the same way as in Embodiments 1 and 2. In turn, it is possible to speed up the processing of SfM and vSLAM.

[0137] <Embodiment 4> Next, Embodiment 4 of the present invention will be described. Embodiment 4 is a camera that captures images in a shooting direction that does not include the same or opposite direction as the direction of movement of the moving body 1000'''', and is configured to switch the reference frame for feature point matching based on the direction of movement of the moving body 1000''''.

[0138] Figure 19 is a configuration diagram of the mobile body 1000'''' according to Embodiment 4. The mobile body 1000'''' is, for example, a vehicle. The integrated processing unit 200 and the direction of travel detection unit 30 in Figure 19 are the same as in Figure 1.

[0139] The imaging unit 10'''' has the configuration of the imaging unit 10 in Figure 1, plus a right surround-view camera 10e and a left surround-view camera 10f. The right surround-view camera 10e and the left surround-view camera 10f are cameras used in the surround-view system (a system that allows the surroundings of a vehicle to be viewed with video) of a moving object 1000'''' which is a vehicle.

[0140] Figure 20 shows the mounting positions of the right surround-view camera 10e and the left surround-view camera 10f on the imaging unit 10''''. Note that the direction of movement of the moving body 1000'''' can be forward or backward, but Figure 20 is a representative example showing the case where the direction of movement of the moving body 1000'''' is forward. In Figure 20, the moving body 1000'''' at time t is shown by a dashed line, and the moving body 1000'''' at time t+1 is shown by a solid line.

[0141] The shooting directions of the right surround-view cameras 10e and 10e' are shooting directions 201e and 201e', respectively. The shooting directions of the left surround-view cameras 10f and 10f' are shooting directions 201f and 201f', respectively. The right surround-view camera 10e and the left surround-view camera 10f are mounted in such a way that they cannot capture images in the same direction as or opposite to the direction of movement of the moving object 1000''''.

[0142] Here, let 202e be the image plane when the image is taken from the position of the right surround-view camera 10e at time t+1, and let 202e' be the image plane when the image is taken from the position of the right surround-view camera 10e' at time t. In this case, the epipole where the baseline connecting the position of the right surround-view camera 10e at time t+1 and the position of the right surround-view camera 10e' at time t coincides with the respective image planes 202e and 202e' does not appear on the captured image. However, the epipole exists at either the point at infinity obtained by extending the right edge of the captured image, or the point at infinity obtained by extending the left edge of the captured image.

[0143] Furthermore, let 202f be the image plane when the image is taken from the position of the left surround-view camera 10f at time t+1, and let 202f' be the image plane when the image is taken from the position of the left surround-view camera 10f' at time t. In this case, the epipole where the baseline connecting the position of the right surround-view camera 10f at time t+1 and the position of the right surround-view camera 10f' at time t coincides with the respective image planes 202f and 202f' does not appear on the captured image. However, the epipole exists at either the point at infinity reached by extending the right edge of the captured image, or the point at infinity reached by extending the left edge of the captured image.

[0144] In either case, the epipolar line connecting the epipole and the feature point can be approximated as a line parallel to the ground or the camera position of the right or left surround-view camera. Therefore, the coordinates of the feature points on the image perpendicular to the ground ideally remain unchanged, and the coordinates of the feature points perpendicular to the ground transition in the opposite direction to the direction of movement of the shooting position, based on the direction of movement of the shooting position. For example, the group of feature points when transitioning from image plane 202e to image plane 202e' transitions to the left on the image, and the group of feature points when transitioning from image plane 202f to image plane 202f' transitions to the right on the image. Therefore, when performing a correspondence point search, it is sufficient to limit the search range to an appropriate one based on the direction of movement of the shooting position as described above.

[0145] In Embodiment 1 of Figure 1, as described above, the reference frame switching unit 220 switches the write destination pointer for the reference frame and the write destination pointer for the search target frame based on the direction of travel signal of the moving body 1000.

[0146] In Embodiment 4, when the direction of movement of the moving body 1000'''' is forward, the reference frame switching unit 220 sets, for example, the feature point group fp(t) as the search target feature point group fp_sub and the feature point group fp(t+1) as the reference feature point group fp_ref. Also, when the direction of movement of the moving body is backward, the reference frame switching unit 220 sets, for example, the feature point group fp(t+1) as the search target feature point group fp_sub and the feature point group fp(t) as the reference feature point group fp_ref.

[0147] In Embodiment 4, by doing so, even if the shooting direction does not include the same or opposite direction as the direction of movement of the moving body 1000'''', the direction of movement of the shooting position can be restricted by switching the reference frame.

[0148] As explained above, according to Embodiment 4, first, the time-series feature point cloud to be used as the reference feature point cloud is defined based on the direction of movement of the moving object. In this way, even if the shooting direction does not include the same or opposite direction as the direction of movement of the moving object, the direction of movement of the shooting position can be restricted by switching the reference frame.

[0149] Therefore, even with a configuration that captures images in a direction that does not include the same or opposite direction as the direction of movement of the moving object, the direction of movement of the shooting position can be restricted to one direction, thus enabling faster feature point matching processing, similar to Embodiment 1 and Embodiment 2. Consequently, SfM and vSLAM processing can be accelerated.

[0150] Although the present invention has been described in detail above based on its embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0151] Furthermore, the present invention also includes cases in which a software program that realizes the functions of the above-described embodiment is supplied directly from a recording medium or via wired / wireless communication to a system or device having a computer capable of executing the program, and the program is executed.

[0152] Therefore, in order to implement the functional processing of the present invention on a computer, the program code supplied to and installed on the computer itself also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0153] In this case, the form of the program is irrelevant as long as it possesses the functionality of a program, including object code, programs executed by an interpreter, and script data supplied to the OS. The recording medium for supplying the program may be, for example, a hard disk, magnetic tape or other magnetic recording media, optical / magneto-optical storage media, or non-volatile semiconductor memory.

[0154] Alternatively, the program may be supplied by storing the computer program forming the present invention on a server on a computer network, and a connected client computer downloading the computer program.

[0155] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0156] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence.

[0157] This embodiment includes the following configurations, methods, and programs. (Composition 1) From a plurality of time-series images acquired from imaging means positioned on a moving object and capturing images around the moving object, a first image captured when the moving object is at a first position and a second image captured when the moving object is at a second position are selected, and an imaging position movement direction limiting means restricts the movement direction of the imaging position between the first position and the second position. A feature point matching means that selects a reference image and a target image from the first image and the second image, respectively, and performs a correspondence between feature points, A reference image switching means that switches the reference image based on information representing the direction of movement of the moving body, Equipped with, The feature point matching means searches for each feature point on the target image within a limited search range based on the direction of movement of the moving object. An image processing apparatus characterized by the following: (Configuration 2) The feature point matching means determines the search range based on predetermined range information and searches for each feature point on the target image. The image processing apparatus according to configuration 1, characterized in that... (Composition 3) The system further comprises a direction of travel detection means for detecting the direction of travel of the moving body, Based on the detection result of the direction of movement detection means, information representing the direction of movement of the moving body is generated. An image processing apparatus according to configuration 1 or configuration 2, characterized by the above. (Composition 4) The system further comprises a shooting position and orientation calculation means for calculating information regarding the position and orientation of the imaging means, The shooting position and orientation calculation means generates information representing the direction of movement based on information regarding the position and orientation of the imaging means. An image processing apparatus according to any one of configurations 1 to 3, characterized by the above. (Composition 5) The system further includes a rotation direction detection means for detecting the rotation direction of the moving body, The feature point matching means searches for each feature point on the target image within a limited search range based on the detection result of the rotation direction detection means. An image processing apparatus characterized by any one of the configurations 1 to 4. (Composition 6) The imaging means is positioned on the moving body such that its imaging direction includes the same direction as or opposite to the direction of movement of the moving body. An image processing apparatus according to any one of configurations 1 to 5, characterized by the above. (Composition 7) The reference image switching means switches the reference image so that the direction of movement of the moving body is opposite to the direction of shooting of the imaging means. The image processing apparatus according to configuration 6, characterized by the features described therein. (Method 1) A process of selecting a first image taken when the moving body is at a first position and a second image taken when the moving body is at a second position from a plurality of time-series images acquired from an imaging means positioned on the moving body and taking images of the area around the moving body, and restricting the direction of movement of the imaging position between the first position and the second position, A feature point matching process is performed to select a reference image and a target image from the first image and the second image, respectively, and to associate the feature points. A reference image switching step, which switches the reference image based on information representing the direction of movement of the moving body, Equipped with, The feature point matching process searches for each feature point on the target image within a limited search range based on the direction of movement of the moving object. A control method for an image processing apparatus, characterized by the features described above. (Program 1) Computers, From a plurality of time-series images acquired from imaging means positioned on a moving object and capturing images of the area around the moving object, a first image captured when the moving object is at a first position and a second image captured when the moving object is at a second position are selected, and an imaging position movement direction limiting means restricts the direction of movement of the imaging position between the first position and the second position. A feature point matching means for selecting a reference image and a search target image from the first image and the second image, respectively, and for matching feature points, and A reference image switching means that switches the reference image based on information representing the direction of movement of the moving body. To make it function as, The feature point matching means searches for each feature point on the target image within a limited search range based on the direction of movement of the moving object. A program characterized by the following features. [Explanation of symbols]

[0158] 1000, 1000', 1000''', 1000'''': Mobile 10, 10'': Imaging Unit 10a: Front camera 10b: Rear camera 10c: Right side camera 10d: Left side camera 10e, 10e': Right Around View Camera 10f, 10f': Left Around View Camera 200: Integrated Processing Unit 210: Feature point extraction unit 220, 220': Reference frame switching section 230: Feature point matching section 231: Cell splitting process 232, 232'': Correspondence point search process 233:ROM 240: Three-dimensional information generation section 241: Camera pose estimation 242: Motion parallax ranging 30: Direction of travel detection unit

Claims

1. From a plurality of time-series images acquired from imaging means positioned on a moving object and capturing images around the moving object, a first image captured when the moving object is at a first position and a second image captured when the moving object is at a second position are selected, and an imaging position movement direction limiting means restricts the movement direction of the imaging position between the first position and the second position. A feature point matching means that selects a reference image and a target image from the first image and the second image, respectively, and performs a correspondence between feature points, A reference image switching means that switches the reference image based on information representing the direction of movement of the moving body, Equipped with, The feature point matching means searches for each feature point on the target image within a limited search range based on the direction of movement of the moving object. An image processing apparatus characterized by the following:

2. The feature point matching means determines the search range based on predetermined range information and searches for each feature point on the target image. The image processing apparatus according to feature 1.

3. The system further comprises a direction of travel detection means for detecting the direction of travel of the moving body, Based on the detection result of the direction of movement detection means, information representing the direction of movement of the moving body is generated. The image processing apparatus according to feature 1.

4. The system further comprises a shooting position and orientation calculation means for calculating information regarding the position and orientation of the imaging means, The shooting position and orientation calculation means generates information representing the direction of movement based on information regarding the position and orientation of the imaging means. The image processing apparatus according to feature 1.

5. The system further includes a rotation direction detection means for detecting the rotation direction of the moving body, The feature point matching means searches for each feature point on the target image within a limited search range based on the detection result of the rotation direction detection means. The image processing apparatus according to claim 1, characterized in that

6. The imaging means is positioned on the moving body such that its imaging direction includes the same direction as or opposite to the direction of movement of the moving body. The image processing apparatus according to feature 1.

7. The reference image switching means switches the reference image so that the direction of movement of the moving body is opposite to the direction of shooting of the imaging means. The image processing apparatus according to claim 6.

8. A process of selecting a first image taken when the moving object is at a first position and a second image taken when the moving object is at a second position from a plurality of time-series images acquired from an imaging means positioned on the moving object and photographing the area around the moving object, and restricting the direction of movement of the photographing position between the first position and the second position, A feature point matching process is performed to select a reference image and a target image from the first image and the second image, respectively, and to associate feature points. A reference image switching step, which switches the reference image based on information representing the direction of movement of the moving body, Equipped with, The feature point matching process searches for each feature point on the target image within a limited search range based on the direction of movement of the moving object. A control method for an image processing apparatus, characterized by the features described above.

9. Computers, From a plurality of time-series images acquired from imaging means positioned on a moving object and capturing images of the area around the moving object, a first image captured when the moving object is at a first position and a second image captured when the moving object is at a second position are selected, and an imaging position movement direction limiting means restricts the direction of movement of the imaging position between the first position and the second position. A feature point matching means for selecting a reference image and a search target image from the first image and the second image, respectively, and for matching feature points, and A reference image switching means that switches the reference image based on information representing the direction of movement of the moving body. To make it function as, The feature point matching means searches for each feature point on the target image within a limited search range based on the direction of movement of the moving object. A program characterized by the following features.

Images