Image processing apparatus, control method, storage medium, and computer program product

Abstract

An image processing apparatus, a control method, a storage medium, and a computer program product are provided. The image processing apparatus includes: an imaging position movement direction restricting section that selects a first image and a second image from a plurality of time-series images that have been acquired from an imaging section that captures an image around the mobile apparatus disposed on the mobile apparatus, and that controls a movement direction of an imaging position to a first position and a second position, the first image is an image taken when the position of the mobile device is a first position, and the second image is an image taken when the position of the mobile device is a second position; a feature point matching unit that selects a reference image and a search target image from the first image and the second image and associates feature points; and a reference image switching unit that switches the reference image on the basis of information indicating the direction of movement of the mobile device. The feature point matching unit searches for each feature point in the search target image within a search range that has been limited on the basis of the moving direction of the movable device.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image processing apparatus, a control method, and a storage medium, and the like. BACKGROUND

[0002] There is a technique of matching points having the same feature between a plurality of images (hereinafter referred to as feature point matching). Feature points that have been associated with each other by feature point matching are used in SfM and vSLAM that reconfigure three-dimensional information from images, and feature points are expected to contribute to autonomous driving and robotics. SfM is an abbreviation of Structure from Motion. vSLAM is an abbreviation of visual Simultaneous Localization and Mapping. When performing feature point matching, if a corresponding point is searched from all input images, it becomes necessary to have enormous computational complexity, and the processing time becomes long, and therefore, conventionally, various methods to improve the speed thereof have been considered.

[0003] For example, in Japanese Patent Application Laid-Open No. 2022-143412, when a corresponding point of a feature point in one image is being searched, a search range is limited based on self position estimation information of a vehicle, thereby reducing the computational complexity of feature point matching.

[0004] In addition, in Japanese Patent Application Laid-Open No. 2000-285245, a search range is limited based on the position of a vanishing point, thereby reducing the operation complexity for detecting a movable device.

[0005] In Japanese Patent Application Laid-Open No. 2022-143412, the computational complexity is reduced by limiting the search range. However, processing for setting the search range is necessary, and therefore, the processing can not be completed in real time.

[0006] In addition, in Japanese Patent Application Laid-Open No. 2000-285245, the search range is limited based on the position of a vanishing point. However, the search range is made a range from the vanishing point to the edge of the image, and therefore, there is a case where the amount of processing cannot be sufficiently reduced due to the position of the vanishing point. SUMMARY

[0007] An object of the present disclosure is to provide an image processing apparatus that reduces the amount of processing corresponding to feature point matching and that can improve the processing speed.

[0008] An image processing apparatus according to one embodiment of the present disclosure includes: a photographing position movement direction restriction section configured to select a first image and a second image from a plurality of time-series images that have been acquired from photographing sections arranged on a movable apparatus for capturing images of surroundings of the movable apparatus, and control a movement direction of the photographing position to a first position and a second position, wherein the first image is an image captured when the position of the movable apparatus is the first position, and the second image is an image captured when the position of the movable apparatus is the second position; a feature point matching section configured to select a reference image and a search target image from the first image and the second image, and perform feature point association; and a reference image switching section configured to switch the reference image based on information showing a movement direction of the movable apparatus. The feature point matching section searches for each feature point in the search target image within a search range that has been restricted based on the movement direction of the movable apparatus.

[0009] The features of the present disclosure will become apparent from the following description of the embodiments with reference to the attached drawings. The following description of embodiments is described by way of example with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a configuration diagram of a movable apparatus according to a first embodiment of the present disclosure.

[0011] Figure 2A and Figure 2B is a schematic diagram showing a mounting position and a photographing direction of a photographing unit according to the first embodiment.

[0012] Figure 3 is a schematic diagram showing time-series images captured by the photographing unit according to the first embodiment.

[0013] Figure 4A and Figure 4B is a schematic diagram showing extraction of feature points from a captured image from a front camera according to the first embodiment.

[0014] Figure 5A and Figure 5B is a schematic diagram showing switching of a reference frame based on a traveling direction of a movable apparatus according to the first embodiment.

[0015] Figure 6A and Figure 6B is a schematic diagram showing search for a corresponding point in a search range that has been predetermined according to the first embodiment.

[0016] Figure 7 is an explanatory diagram relating to epipolar geometry according to the first embodiment.

[0017] Figure 8 is an operation flowchart for the integration processing unit according to the first embodiment.

[0018] Figure 9 is an operation flowchart for the reference frame switching unit according to the first embodiment.

[0019] Figure 10 is an operation flowchart for the feature point matching unit according to the first embodiment.

[0020] Figure 11 is an operation flowchart for the three-dimensional information generation unit according to the first embodiment.

[0021] Figure 12 is a configuration diagram for a mobile device according to a second embodiment of the present disclosure.

[0022] Figure 13 is an operation flowchart for the integration processing unit according to the second embodiment.

[0023] Figure 14 is an operation flowchart for the reference frame switching unit according to the second embodiment.

[0024] Figure 15 is a configuration diagram for a mobile device according to a third embodiment of the present disclosure.

[0025] Figure 16 is an explanatory diagram relating to epipolar geometry according to the third embodiment.

[0026] Figure 17 is an operation flowchart for the integration processing unit according to the third embodiment.

[0027] Figure 18 is an operation flowchart for the feature point matching unit according to the third embodiment.

[0028] Figure 19 is a configuration diagram for a mobile device according to a fourth embodiment of the present disclosure.

[0029] Figure 20 is a schematic diagram showing mounting positions and imaging directions of imaging units according to the fourth embodiment. DETAILED DESCRIPTION

[0030] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. However, the present disclosure is not limited to the embodiment described below. In addition, the following embodiment does not limit the present disclosure according to the claims, and all combinations of features described in the present embodiment are not necessarily essential to the solution means of the present disclosure. Note that in each drawing, the same reference signs have been attached to the same parts and elements, and repeated explanation will be simplified or omitted.

[0031] <First Embodiment>

[0032] Hereinafter, a first embodiment of the present disclosure will be described with reference to the drawings. In the first embodiment, in a camera that captures an image from the same direction as the traveling direction of a mobile device or from the opposite direction to the traveling direction of the mobile device, the moving direction of the imaging position is limited by switching the reference frame for feature point matching based on the traveling direction of the mobile device. Further, in the first embodiment, a method is described in which the feature point matching is made faster and further the SfM and the vSLAM are made faster by limiting the search range of the feature point matching together with limiting the moving direction of the imaging position.

[0033] Figure 1 is a configuration diagram of a mobile device 1000 according to the first embodiment. The mobile device 1000 is, for example, a vehicle, and has an imaging unit 10, an integration processing unit 200, and a traveling direction detection unit 30. The mobile device 1000 is one example of an image processing device.

[0034] The imaging unit 10 is a vehicle-mounted camera capable of capturing an image from any direction around the mobile device 1000, as typified by, for example, a front camera 10a, a rear camera 10b, a right side camera 10c, and a left side camera 10d. The imaging unit 10 can be one camera, or it can also be composed of a plurality of cameras. The front camera 10a captures an image of the front of the mobile device 1000. The rear camera 10b captures an image of the rear of the mobile device 1000. The right side camera 10c and the left side camera 10d are, for example, cameras used in a camera monitoring system for electronic side mirrors, and the right side camera 10c captures an image of the right rear side, and the left side camera 10d captures an image of the left rear side of the mobile device 1000. Each camera included in the imaging unit 10 is one example of an imaging unit arranged on the mobile device 1000 and capturing an image of the surroundings of the mobile device 1000.

[0035] In Figure 2A and Figure 2B the mounting positions of the imaging unit 10 are shown. Figure 2A is a plan view of the mobile device 1000. Figure 2Bis a side view of the mobile device 1000. In the context, a plane parallel to the ground on which the camera travels is referred to as a camera position. In addition, the camera position of the front camera 10a is referred to as a front camera position 31a (refer to Figure 3 ). The camera position of the rear camera 10b is referred to as a rear camera position 31b (refer to Figure 3 ). The camera position of the right side camera 10c is referred to as a right side camera position 31c (refer to Figure 3 ). The camera position of the left side camera 10d is referred to as a left side camera position 31d (refer to Figure 3 ).

[0036] In addition, the front camera 10a is attached to the mobile device 1000 so that the imaging direction 21a of the front camera 10a becomes an orientation in which an image can be captured in a direction parallel to the direction of travel of the mobile device 1000. The rear camera 10b is attached to the mobile device 1000 so that the imaging direction 21b of the rear camera 10b becomes an orientation in which an image can be captured in a direction parallel to the opposite direction of the direction of travel of the mobile device 1000. The right side camera 10c and the left side camera 10d are attached to the mobile device 1000 so that the imaging direction 21c of the right side camera 10c and the imaging direction 21d of the left side camera 10d become orientations in which images can be captured in directions parallel to the opposite direction of the direction of travel of the mobile device 1000.

[0037] The explanation will return to Figure 1 . The imaging unit 10 acquires each of the imaging signals that have been captured as optical images for each of the times, and outputs two time-series images (time-series image img(t) and time-series image img(t+1)) that have been acquired by performing RAW development processing on each of the imaging signals in order of the sequence of the times to the integration processing unit 200.

[0038] Note that the imaging unit 10 has an image processing unit that is not shown, and the image processing unit can have a distortion aberration correction function. The imaging unit performs distortion aberration correction on the time-series image img(t) and the time-series image img(t+1). As a result of this processing, it becomes possible to show the front camera position 31a, the rear camera position 31b, the right side camera position 31c, and the left side camera position 31d on the image using straight lines.

[0039] Figure 3is a diagram showing time-series images taken by the respective cameras of the imaging unit 10 in a case where the movable device 1000 that is a vehicle is traveling forward. The time-series image img(t) from time t taken by the front camera 10a is a time-series image 32a, and the time-series image img(t+1) from time t+1 taken by the front camera 10a is a time-series image 33a. The time-series image img(t) from time t taken by the rear camera 10b is a time-series image 32b, and the time-series image img(t+1) from time t+1 taken by the rear camera 10b is a time-series image 33b. The time-series image img(t) from time t taken by the right camera 10c is a time-series image 32c, and the time-series image img(t+1) from time t+1 taken by the right camera is a time-series image 33c. The time-series image img(t) from time t taken by the left camera 10d is a time-series image 32d, and the time-series image img(t+1) from time t+1 taken by the left camera 10d is a time-series image 33d.

[0040] Note that the time-series image 32b and the time-series image 33b that have been acquired from the rear camera 10b, the time-series image 32c and the time-series image 33c that have been acquired from the right camera 10c, and the time-series image 32d and the time-series image 33d that have been acquired from the left camera 10d are all left-right reversed. However, right and left need not be reversed in the images during implementation of the present disclosure.

[0041] Note that although, for simplicity Figure 3 , the vanishing point and the kernel point to be explained below in the context of the feature point matching unit 230 are located on the same coordinates on the image plane, in real captured images, the coordinates of the vanishing point and the kernel point will not necessarily overlap. Also, although, for simplicity Figure 3 , the imaging has been performed so that the bottom edge of the image becomes parallel to the ground, in real captured images, the imaging need not necessarily be performed so that the bottom edge of the image becomes absolutely parallel to the ground.

[0042] The explanation will return to Figure 1 . The integration processing unit 200 has a feature point extraction unit 210, a reference frame switching unit 220, a feature point matching unit 230, and a three-dimensional information generation unit 240. Also, the integration processing unit 200 has a CPU that is not depicted for controlling each of the units 210 to 240, and a RAM that is also not depicted and functions as a main storage medium. The CPU is an abbreviation for Central Processing Unit. The RAM is an abbreviation for Random Access Memory.

[0043] The feature point extraction unit 210 has a feature point extraction function realized as, for example, an SoC / FPGA function for extracting a feature point in an image based on an input image. SoC is an abbreviation for System on Chip. FPGA is an abbreviation for Field Programmable Gate Array. Note that it is enough to realize the feature point extraction function using a general technique such as, for example, SIFT and ORB, and the like. SIFT is an abbreviation for Scale-Invariant Feature Transform. ORB is an abbreviation for Oriented FAST and Rotated BRIEF.

[0044] In Figure 4A and Figure 4B , a schematic view of extracting a feature point from an image img(t+1)a taken by the front camera 10a is shown. Figure 4A is a graph showing an input image, and Figure 4B is a graph showing a feature point group. In a case where an image img(t+1)a such as the image shown in Figure 4A is used as an input image of the feature point extraction function, a feature point group fp(t+1)a such as the point group 41 shown in Figure 4B is extracted.

[0045] Note that, generally, a feature point is extracted from, for example, an edge portion in an image such as an outline of a certain object and a boundary between objects in a plurality of objects using an algorithm used in feature point extraction. In addition, it is easy to extract a feature point from a portion having a maximum point and a minimum point in luminance with respect to a pattern such as a block and a mosaic, and there are various methods. Therefore, it is not necessarily absolutely necessary to extract a feature point from an edge portion of an input image as in Figure 4A and Figure 4B , and an algorithm used in feature point extraction is not limited.

[0046] The explanation will return to Figure 1 . The feature point extraction unit 210 outputs a feature point group fp(t) that has been extracted from the time-series image img(t) and a feature point group fp(t+1) that has been extracted from the time-series image img(t+1), and outputs it to the reference frame switching unit.

[0047] The reference frame switching unit 220 is realized as, for example, an SoC / FPGA function. In this context, from the feature point group fp(t) that has been extracted from the time-series image img(t) and the feature point group fp(t+1) that has been extracted from the time-series image img(t+1), the feature point group that is to become the reference for the corresponding point search processing 232, which will be described below, is made the feature point group fp_ref. In addition, the image at the time that corresponds to the reference feature point group fp_ref is made the reference image img_ref. In addition, the data set of the reference feature point group and the reference image is referred to as the reference frame. In addition, the feature point group that is to become the search target of the corresponding point search processing 232 is made the search target feature point group fp_sub, and the image at the time that corresponds to the search target feature point group is made the search target image img_sub. In addition, the data set of the search target feature point group and the search target image is referred to as the search target frame.

[0048] The reference frame switching unit 220 has a control register that maintains the imaging direction of each camera of the imaging unit 10 and a write destination pointer for the frames of both the reference frame and the search target frame, which are not shown.

[0049] In addition, the reference frame switching unit 220 has a reference frame switching function for switching the write destination pointer of the reference frame and the write destination pointer of the search target frame of the control register. The reference frame switching function switches the write destination pointer of the reference frame and the write destination pointer of the search target frame of the control register based on the travel direction signal of the mobile device that has been input from the travel direction detection unit 30 and the imaging direction of each camera.

[0050] In addition, the reference frame switching unit 220 makes one of the time-series feature point groups fp(t) and fp(t+1) that have been input from the feature point extraction unit 210 the reference feature point group fp_ref and the other the search target feature point group fp_sub. The reference frame switching unit 220 outputs the reference feature point group fp_ref and the search target feature point group fp_sub to the feature point matching unit 230.

[0051] In this context, the reference frame switching unit 220 assumes that the imaging position of each frame that has been input will be changed from the imaging position of the reference image to the imaging position of the search target image. The reference frame switching unit 220 switches the reference frame and the search target frame so that the direction in which the imaging position of the image is changed becomes the direction opposite to the imaging direction of the camera. The reference frame switching unit 220 is one example of a reference image switching unit that switches the reference image based on information that shows the direction of movement of the mobile device.

[0052] In this context, the direction of travel of the movable device 1000 is made to be forward travel. In this case, in the case of the time-series images img(t)a and img(t+1)a that have been photographed by the front camera 10a, the reference frame switching unit 220 makes the time-series image img(t+1)a the reference image, and makes the time-series image img(t)a the search target image. Also, in this case, in the case of the time-series images that have been photographed by the back camera 10b, the reference frame switching unit 220 makes the time-series image img(t)b the reference image, and makes the time-series image img(t+1)b the search target image. Also, in this case, in the case of the time-series images that have been photographed by the right camera 10c, the reference frame switching unit 220 makes the time-series image img(t)c the reference image, and makes the time-series image img(t+1)c the search target image. Also, in this case, in the case of the respective time-series images that have been photographed by the left camera 10d, the reference frame switching unit 220 makes the time-series image img(t)d the reference image, and makes the time-series image img(t+1)d the search target image.

[0053] In this context, the direction of travel of the movable device 1000 is made to be backward travel. In this case, in the case of the time-series images img(t)a and img(t+1)a that have been photographed by the front camera 10a, the reference frame switching unit 220 makes the time-series image img(t)a the reference image, and makes the time-series image img(t+1)a the search target image. Also, in this case, in the case of the time-series images that have been photographed by the back camera 10b, the reference frame switching unit 220 makes the time-series image img(t+1)b the reference image, and makes the time-series image img(t)b the search target image. Also, in this case, in the case of the time-series images that have been photographed by the right camera 10c, the reference frame switching unit 220 makes the time-series image img(t+1)c the reference image, and makes the time-series image img(t)c the search target image. Also, in this case, in the case of the time-series images that have been photographed by the left camera 10d, the reference frame switching unit 220 makes the time-series image img(t+1)d the reference image, and makes the time-series image img(t)d the search target image.

[0054] Figure 5A and Figure 5BAn example is shown where the reference image img_ref and the search object image img_sub are switched from the time-series image img(t)a and the time-series image img(t+1)a captured by the front camera 10a according to the direction of travel of the mobile device 1000. Figure 5A The diagram shows the state of the gear shift lever 30a when it is in drive or low gear, and the mobile device 1000 when it is moving forward. Figure 5B The diagram shows the gear shift lever 30a in reverse gear and the movable device 1000 moving backward.

[0055] For example, when the mobile device 1000 is moving forward and time series images 32a and 33a are input, the reference frame switching unit 220 outputs the reference frame image 51 and the search object image 52 (reference). Figure 5A Additionally, for example, when time series images 32a' and 33a' are input while the mobile device 1000 is moving backward, the reference frame switching unit 220 outputs a reference image 51' and a search object image 52' (see reference). Figure 5B ).

[0056] Note that, having already switched to change the image's camera position to the opposite direction of the camera's shooting direction according to the travel direction of the mobile device 1000, this switch causes the camera position to move away from the stationary object being photographed, and therefore, it appears as... Figure 5A and Figure 5B As shown.

[0057] Note that in this context, the direction in which the camera is moving from the camera position of the reference image to the camera position of the search object image is referred to as the camera position movement direction. Specifically, in the first embodiment, the direction of camera position movement is defined as the positive direction, which is opposite to the camera's shooting direction, and the direction aligned with the camera's shooting direction, which is the negative direction.

[0058] As described above, by switching the reference frame and the search object frame based on the travel direction of the mobile device, the movement direction of the camera position is restricted to one direction regardless of changes in the travel direction of the mobile device. The unit used to restrict this movement direction of the camera position is an example of a camera position movement direction limiting unit used to limit the movement direction of the camera position towards the first and second positions. Furthermore, the search range of the corresponding point search process 232, which will be described below, can be pre-limited by restricting the movement direction of the camera position.

[0059] Furthermore, in the first embodiment, the reference frame and the search target frame are switched such that the direction of movement of the camera position is reversed compared to the camera's shooting direction. By doing so, the camera position of the search target frame is captured from a more rearward position compared to the camera position of the reference frame, and therefore, there is no situation where the feature point corresponding to the reference feature point becomes outside the camera range of the search target frame, and the corresponding point search can be performed efficiently.

[0060] The instructions will return to Figure 1 The feature point matching unit 230 consists of, for example, RAM serving as a CPU and main storage medium, and ROM 233 storing the search range shown below in Figure 233a. ROM is an abbreviation for Read-Only Memory. The feature point matching unit 230 performs cell segmentation processing 231 and corresponding point search processing 232 by executing a program through its CPU. The feature point matching unit 230 is an example of a feature point matching unit that selects both the reference image and the search object image from the first image and the second image and performs feature point association.

[0061] Cell segmentation processing 231 is a process for segmenting each frame from the reference frame and the search target frame that have been input from the reference frame switching unit 220 into cells within a defined range. For example, if a frame of, for example, 1920×1080 pixels is input, cell segmentation processing 231 can segment it into, for example, 16×9 cells within a range of 120×120, which is sufficient.

[0062] Corresponding point search processing 232 is performed as follows: after selecting feature points from the reference feature point group fp_ref, it performs association (hereinafter referred to as matching) of feature points that correspond to the feature points already selected from the search object feature point group fp_sub. Note that matching can be achieved using a general method such as a brute-force matcher.

[0063] However, the corresponding point search process 232 performs matching within a range already limited based on the search range map 233a stored in ROM 233. In this context, it will use... Figure 6A and Figure 6B An explanation is given regarding the restrictions on the matching region based on the search range diagram 233a.

[0064] Figure 6A and Figure 6B This is a diagram showing the reference frame and the search target frame formed from images captured using the front-facing camera 10a. Figure 6A The reference feature point group is shown, and Figure 6B The set of feature points of the search object is shown.

[0065] Figure 6A p1, p2, and p3 shown are the individual feature points included in the baseline feature point group fp_ref{p1, p2, p3}. Figure 6B p1', p2', and p3' shown are the individual feature points included in the feature point group fp_sub{p1', p2', and p3'} of the search object.

[0066] in addition, Figure 6A and Figure 6B The points e and e' shown are the points (hereinafter referred to as kernel points) where the baselines of the respective cameras and the image plane overlap when each frame of the reference frame and the search target frame has been captured. Additionally, the lines connecting each feature point in feature points p1, p2, and p3 to kernel point e, and the lines connecting each feature point in feature points p1', p2', and p3' to kernel point e', are called kernel lines.

[0067] like Figure 6A and Figure 6B As shown, assign a unique size to each cell in the search range diagram 233a. For example, it is sufficient to use the position of the corresponding cell as the starting point and assign search range 611 (+1, +1) to the search range diagram 233a corresponding to cell 601, search range 612 (+2, -1) to the search range diagram 233a corresponding to cell 602, and search range 613 (-2, -2) to the search range diagram 233a corresponding to cell 603. At this time, as... Figure 6B As shown, in the first embodiment, the direction in which the search range is restricted can be the direction facing the core point. Figure 233a shows an example of a search range based on pre-determined range information.

[0068] Corresponding point search processing 232 selects a feature point from the reference frame segmented by cells, and matches the search object frame within a range limited by a search range map 233a corresponding to the cell containing the selected feature point.

[0069] As described above, the computational complexity during feature point matching can be reduced by performing corresponding point search processing 232 within a pre-limited range based on the restricted camera position's movement direction while limiting the camera position's movement direction to one direction.

[0070] In addition, by referring to the pre-determined search range diagram 233a, the corresponding point search process 232 can be started in advance without setting the search range as an internal process.

[0071] Furthermore, since the camera's movement direction is restricted to one direction, the search range does not need to be changed based on changes in the travel direction of the mobile device 1000. Therefore, it is sufficient to store only one search range map 233a regardless of the travel direction of the mobile device 1000, thus reducing the ROM capacity required to store the search range map 233a. This is further improved by providing the search range map 233a for each cell rather than each pixel through cell segmentation, thus further reducing the ROM capacity required to store the search range map 233a.

[0072] Subsequently, the corresponding point search process 232 repeatedly matches all feature points included in the baseline feature point group fp_ref.

[0073] Note that, ideally, when the mobile device 1000 is moving forward parallel to the ground without turning, or when it is moving backward parallel to the ground without turning, the coordinates of the kernel point remain unchanged and are uniquely determined based on the camera's attachment position and the angle formed between the camera's attachment position and the ground. For these situations, the direction of camera movement, the position of the kernel point, and the position of the feature point are determined in... Figure 7 As shown in the image.

[0074] Figure 7 This diagram illustrates a situation where a mobile device is in the middle of moving forward or backward. Figure 7 In this context, a kernel point 71 is defined as the point e where the baseline connecting the camera position C in the reference frame and the camera position C' in the search target frame overlaps with the image plane in each frame, and a kernel point 71' is defined as the point e' where the baseline connecting the camera position C in the reference frame and the camera position C' in the search target frame overlaps with the image plane in each frame. At this time, the mobile device 1000 is moving forward or backward parallel to the ground; therefore, both kernel point e and kernel point e' exist at the camera position. Furthermore, this is the case where the mobile device 1000 is moving forward or backward without turning; therefore, ideally, the coordinates of kernel point e and kernel point e' do not change.

[0075] Conversely, the point p where the line connecting the camera position C to the coordinates P in the real space of the feature point overlaps with the image plane of each frame is designated as feature point 72 in the image, and the point p' where the line connecting the camera position C' to the coordinates P in the real space of the feature point overlaps with the image plane of each frame is designated as feature point 72' in the image. Furthermore, the line connecting the kernel point e and the feature point p is designated as the epipolar line 73, and the line connecting the kernel point e' to the feature point p' is designated as the epipolar line 73'. At this time, due to the epipolar constraint, the epipolar line 73 and the epipolar line 73' are on the same plane, and in the first embodiment, the mobile device 1000 is moving forward or backward parallel to the ground without turning; therefore, the position of the epipolar line on the image does not change.

[0076] Therefore, in the first embodiment, the search range when transforming from feature point p to feature point p' can be limited to the direction facing the kernel point by restricting the movement direction of the camera position to the positive direction, and the computational complexity of the corresponding point search processing 232 can be reduced.

[0077] Note that, in reality, due to factors such as vibrations when the mobile device 1000, as a vehicle, is in motion, the positions of core point e and core point e' become separated from each other. In this context, taking this separation into account, this disclosure can also be implemented by calculating the deviation of the position of the core point, which has been uniquely determined based on the camera's mounting position and its angle with the ground, and performing a corresponding point search process 232 within a specific set threshold.

[0078] The instructions will return to Figure 1 The feature point matching unit 230 outputs the following pairs of feature points to the 3D information generation unit 240 for use as matching feature point pairs: feature points from the reference feature point group and feature points from the search object feature point group that have been associated by the corresponding point search process 232.

[0079] The 3D information generation unit 240 is composed, for example, of a CPU, RAM used as the main storage medium, and ROM (not depicted) storing internal parameters of the camera. The 3D information generation unit 240 performs camera pose estimation 241 and motion parallax measurement 242 by executing a program through its CPU.

[0080] Camera pose estimation 241 calculates the camera's extrinsic parameters based on the matched feature point pairs input from feature point matching unit 230 and the camera's intrinsic parameters stored in ROM. The camera's extrinsic parameters include, for example, the amount of movement and rotation angle of the camera relative to the reference frame's imaging position. Camera pose estimation 241 outputs the calculated camera extrinsic parameters to motion parallax measurement 242. Note that it is sufficient to implement camera pose estimation 241 using a general method utilized during SfM (e.g., an 8-point method or a 5-point method). Camera pose estimation 241 is an example of a camera position and pose calculation unit used to calculate information related to the position and pose of the imaging unit. Camera pose estimation 241 can also generate information indicating the direction of movement of the mobile device based on the information related to the position and pose of the imaging unit.

[0081] Motion parallax measurement 242 generates three-dimensional coordinates of the camera position of a feature point relative to a reference frame based on the camera's extrinsic parameters, intrinsic parameters, and matching feature point pairs. In this context, the camera's extrinsic parameters used by motion parallax measurement 242 are those input from camera pose estimation 241. Additionally, the camera's intrinsic parameters used by motion parallax measurement 242 are those stored in ROM. Furthermore, the matching feature point pairs used by motion parallax measurement 242 are those input from feature point matching unit 230. Note that motion parallax measurement 242 can also be implemented using common techniques used during SfM (e.g., triangulation used during stereo vision).

[0082] Note that although the context provides a case where the integrated processing unit 200, the feature point matching unit 230, and the 3D information generation unit 240 each have their own CPU, it is also possible to use a single CPU to perform the entire process described above.

[0083] Furthermore, in this context, examples are given of feature point extraction unit 210 and reference frame switching unit 220 being implemented using a SoC / FPGA, and feature point matching unit 230 and 3D information generation unit 240 being implemented using a CPU. However, this disclosure is not limited to its implementation methods, and it is sufficient to implement this disclosure using any means.

[0084] The travel direction detection unit 30 includes, for example, a gear shift lever 30a built into a vehicle. The gear shift lever 30a is a lever used to change the gear position of the transmission of the mobile device 1000. When the gear shift lever 30a is in drive or low gear, the mobile device 1000 travels forward. When the gear shift lever 30a is in reverse, the mobile device 1000 travels backward. In this context, the travel direction detection unit 30 detects the state of the gear shift lever 30a and can detect the travel direction of the mobile device 1000 based on the detected state of the gear shift lever 30a. For example, when the gear shift lever 30a is in drive or low gear, the travel direction detection unit 30 determines that the travel direction of the mobile device 1000 is forward, and when the gear shift lever 30a is in reverse, it determines that the travel direction of the mobile device 1000 is backward. In addition, the travel direction detection unit 30 outputs the detected travel direction of the mobile device 1000 to the reference frame switching unit 220.

[0085] Figure 8 This is a flowchart illustrating a series of operations of the integration processing unit 200 in the first embodiment. Figure 8 The processing of each step shown is performed sequentially by the CPU of the integrated processing unit 200 executing a computer program.

[0086] During step S101, the integration processing unit 200 acquires the time-series image img(t) and the time-series image img(t+1) from the imaging unit 10. Afterward, the processing proceeds to step S102.

[0087] During step S102, the integration processing unit 200 extracts the time series feature point group fp(t) from the time series image img(t) and the time series feature point group fp(t+1) from the time series image img(t+1). Afterward, the processing proceeds to step S103.

[0088] During step S103, the integration processing unit 200 uses the travel direction detection unit 30 to obtain the travel direction of the mobile device 1000. Afterwards, the processing proceeds to step S110.

[0089] In step S110, the integration processing unit 200 inputs the travel direction of the mobile device 1000, the time series image img(t), the time series image img(t+1), the time series feature point group fp(t), and the time series feature point group fp(t+1), and the processing is performed by the reference frame switching unit 220. The reference frame switching unit 220 outputs the reference feature point group fp_ref and the search object feature point group fp_sub as the processing result. Afterwards, the processing proceeds to step S120.

[0090] In step S120, the integration processing unit 200 inputs the reference feature point group fp_ref, the search object feature point group fp_sub, and the search range map 233a, and the feature point matching unit 230 performs the processing. The feature point matching unit 230 outputs the matched feature point pairs as the processing result. Afterwards, the processing proceeds to step S130.

[0091] During step S130, the integration processing unit 200 inputs matching feature point pairs and camera internal parameters, and the 3D information generation unit 240 performs the processing. The 3D information generation unit 240 outputs the 3D coordinates of each feature point corresponding to the camera position C of the reference frame, and the external parameters of the camera at the camera position C' of the search target frame relative to the reference frame as the processing result. After this, the series of processes by the integration processing unit 200 is completed.

[0092] Figure 9 This is a flowchart illustrating a series of processes of the reference frame switching unit 220 in the first embodiment. Figure 9 The processing of each step shown is performed sequentially by the SoC / FPGA of the reference frame switching unit 220.

[0093] During step S111, the reference frame switching unit 220 determines, based on the travel direction of the mobile device 1000 and the camera's shooting direction obtained during step S103, whether a direction identical to the travel direction of the mobile device 1000 is included in the camera's shooting direction. If the reference frame switching unit 220 has determined that a direction identical to the travel direction of the mobile device 1000 is included in the camera's shooting direction, the processing of step S112 is executed. If the reference frame switching unit 220 has determined that a direction identical to the travel direction of the mobile device 1000 is not included in the camera's shooting direction, the processing of step S113 is executed. Note that the first embodiment is a mode in which imaging is performed in the same or opposite direction as the travel direction of the mobile device 1000. Therefore, if a direction identical to the travel direction of the mobile device 1000 is not included in the camera's shooting direction, the imaging unit 10 is imaging in the opposite direction to the travel direction of the mobile device 1000.

[0094] During step S112, the reference frame switching unit 220 outputs feature point group fp(t) from the time series feature point group fp(t) and the time series feature point group fp(t+1) acquired during step S102 to the integration processing unit 200 for use as the search object feature point group fp_sub. Additionally, during step S112, the reference frame switching unit 220 outputs reference feature point group fp(t+1) from the time series feature point group fp(t) and the time series feature point group fp(t+1) acquired during step S102 to the integration processing unit 200 for use as the reference feature point group fp_ref. After this, the processing is complete.

[0095] During step S113, the reference frame switching unit 220 outputs feature point group fp(t) from the time series feature point group fp(t) and the time series feature point group fp(t+1) acquired during step S102 to the integration processing unit 200, to be used as the reference feature point group fp_ref. Additionally, during step S113, the reference frame switching unit 220 outputs feature point group fp(t+1) from the time series feature point group fp(t) and the time series feature point group fp(t+1) acquired during step S102 to the integration processing unit 200, to be used as the search object reference feature point group fp_sub. After this, the processing is complete.

[0096] Figure 10 This is a flowchart illustrating a series of processes performed by the feature point matching unit 230 in the first embodiment. Figure 10 The processing of each step shown is performed sequentially by the CPU of the feature point matching unit 230 executing a computer program.

[0097] During step S121, the feature point matching unit 230 performs cell segmentation on the two feature point groups (reference feature point group fp_ref and search object feature point group fp_sub) acquired during step S110. Afterward, the process proceeds to step S122.

[0098] During step S122, the feature point matching unit 230 selects an arbitrary cell from the reference feature point group fp_ref that has been segmented by cells. Afterward, the process proceeds to step S123.

[0099] During step S123, feature point matching unit 230 selects an arbitrary feature point from the cells already selected during step S122. The process then proceeds to step S124.

[0100] During step S124, feature point matching unit 230 refers to search range map 233a and performs a corresponding point search for feature points in the feature point group fp_sub of the search object that correspond to the feature points selected during step S123 within the search range corresponding to the cell selected during step S122. Afterwards, the process proceeds to step S125.

[0101] During step S125, the feature point matching unit 230 determines whether the corresponding point search has been completed for each feature point included in the cell selected during step S122. If the feature point matching unit 230 has determined that the corresponding point search has not yet been completed, the processing of step S123 is executed. If the feature point matching unit 230 has determined that the corresponding point search has been completed, the processing of step S126 is completed.

[0102] During step S126, the feature point matching unit 230 determines whether each cell segmented during step S121 has completed the processing from steps S122 to S125. If the feature point matching unit 230 determines that the processing is not yet complete, the processing of step S122 is executed. If the feature point matching unit 230 determines that the processing is complete, the processing of step S127 is completed.

[0103] During step S127, the feature point matching unit 230 outputs the matched feature point pairs that have been associated during the corresponding point search to the integration processing unit 200. After this, the processing is complete.

[0104] Figure 11 This is a flowchart illustrating a series of operations of the three-dimensional information generation unit 240 in the first embodiment. Figure 11 The processing of each step shown is performed sequentially by the CPU of the 3D information generation unit 240 executing a computer program.

[0105] During step S131, the 3D information generation unit 240 estimates the camera pose based on the matching feature point pairs acquired during step S120 and the camera's intrinsic parameters. In this context, the 3D information generation unit 240 acquires the camera's extrinsic parameters—the amount of camera movement and the rotation angle—corresponding to the camera position of the reference frame, to be used as the estimation result. The process then proceeds to step S132.

[0106] During step S132, the 3D information generation unit 240 calculates the 3D coordinates of each feature point relative to the camera position of the reference frame based on the matching feature point pairs, the camera's intrinsic parameters, and the camera's extrinsic parameters. Afterward, the process proceeds to step S133.

[0107] During step S133, the 3D information generation unit 240 outputs the 3D coordinates of each feature point and the camera's extrinsic parameters to the integration processing unit 200. After this, the processing is complete.

[0108] As described above, according to the first embodiment, by switching the reference frame and the search object frame based on the travel direction of the mobile device, the movement direction of the camera position is restricted to one direction regardless of how the travel direction of the mobile device changes. Therefore, the search range in which feature point matching occurs can be pre-limited.

[0109] Furthermore, by referencing a pre-determined search range, the search for corresponding points can begin in advance without needing to set the search range internally. Additionally, performing the search for corresponding points within a limited search range reduces the computational complexity during feature point matching.

[0110] Therefore, the speed of feature point matching can be improved by starting the process earlier and reducing its computational complexity. This, in turn, can also improve the speed of SfM and vSLAM processing.

[0111] Furthermore, according to the first embodiment, the movement direction of the camera position is limited to one direction. Therefore, it is not necessary to change the search range according to the change in the travel direction of the mobile device, and it is sufficient to store a search range diagram 233a regardless of the travel direction of the mobile device. Therefore, the ROM capacity required to store the search range diagram 233a can be suppressed.

[0112] In addition, by performing cell segmentation, it becomes sufficient to provide the search range map 233a for each cell rather than each pixel, thus reducing the ROM capacity required to store the search range map 233a.

[0113] Therefore, the information used in the search range during the corresponding point search can be suppressed at least, and feature point matching processing can be achieved at low cost. Furthermore, SfM and vSLAM can also be provided at low cost.

[0114] Furthermore, according to the first embodiment, the reference frame and the search target frame are switched so that the direction of movement of the camera position becomes the direction facing the camera. By doing so, the camera position of the search target frame is captured from a position further back than the camera position of the reference frame. Therefore, corresponding point search can be performed efficiently without the reference feature points and corresponding feature points leaving the camera range of the search target frame.

[0115] <Second Embodiment>

[0116] Next, a second embodiment of this disclosure will be described. In this second embodiment, the amount of camera movement calculated by the 3D information generation unit 240 in relation to the camera position of the reference frame is fed back to the reference frame switching unit 220. Thus, the second embodiment is configured to switch the reference frame for feature point matching.

[0117] Figure 12 This is a configuration diagram of a mobile device 1000' according to the second embodiment. The mobile device 1000' is, for example, a vehicle. Figure 12 The camera unit 10, feature point extraction unit 210, feature point matching unit 230, and 3D information generation unit 240 are included. Figure 1 The camera unit 10, feature point extraction unit 210, feature point matching unit 230 and 3D information generation unit 240 are the same.

[0118] The integrated processing unit 200 has a reference frame switching unit 220' instead. Figure 1 The reference frame switching unit 220. In addition to the configuration of the reference frame switching unit 220, the reference frame switching unit 220' also has functions that will be described below. That is, the reference frame switching unit 220' has a reference frame switching function that switches the write destination pointer of the reference frame and the write destination pointer of the search object frame based on the camera's external parameters that have been input from the 3D information generation unit 240. Note that, as described in the first embodiment, the camera's external parameters are the amount of movement and rotation angle relative to the camera's position on the reference frame.

[0119] In this context, the amount of camera movement is set such that, for example, from Figure 7 The camera position C shown is positive when facing the image plane img_ref. Note that although there is usually a coordinate system for the camera in stereo vision to make the orientation from the image plane towards the camera positive, it is sufficient to assume that the positive and negative orientations of the camera movement amount illustrated in the second embodiment will be reversed in this case.

[0120] Figure 13 This is a flowchart illustrating a series of processes performed by the integration processing unit 200 according to the second embodiment. Note that... Figure 13 Steps S101, S102, and S110 to S130 in the middle are Figure 8 Steps S101, S102, and S110 to S130 are the same, and their descriptions will be omitted.

[0121] During step S103', the integration processing unit 200 obtains the camera's movement amount from the 3D information generation unit 240. Afterward, the processing proceeds to step S110.

[0122] Figure 14 This is a flowchart illustrating a series of processes of the reference frame switching unit 220' in the second embodiment. Figure 14 The processing of each step shown is performed sequentially by the SoC / FPGA of the reference frame switching unit 220'.

[0123] During step S111', the reference frame switching unit 220' determines whether the camera movement is positive or negative based on the camera movement amount acquired during step S103'. If the reference frame switching unit 220' determines that the camera movement amount is positive, the processing of step S114 is executed. If the reference frame switching unit 220' determines that the camera movement amount is not positive (i.e., the camera movement amount is negative), the processing is completed. Note that, regarding the camera movement amount in the second embodiment, the orientation from the camera towards the image plane is positive. Therefore, if the camera movement amount is positive, it means that the imaging unit 10 is capturing images in the same direction as the travel direction of the mobile device 1000', and if the camera movement direction is negative, it means that the imaging unit 10 is capturing images in the opposite direction to the travel direction of the mobile device 1000'.

[0124] During step S114, the reference frame switching unit 220' switches the write destination pointers of the time series feature point group fp(t) and the time series feature point group fp(t+1) acquired during step S102. That is, the reference frame switching unit 220' switches the write destination pointers of the feature point group currently used as the search target feature point group fp_sub and the reference feature point group fp_ref. Additionally, the reference frame switching unit 220' switches the write destination pointer of the feature point group currently used as the reference feature point group fp_ref with the write destination pointer of the search target feature point group fp_sub. In this way, during step S114, the reference frame switching unit 220' outputs the write destinations of the switched time series feature point group fp(t) and the time series feature point group fp(t+1) to the integration processing unit 200. Afterward, the reference frame switching unit 220' completes the processing.

[0125] In this way, in the second embodiment, the amount of camera movement relative to the camera position of the reference frame, calculated by the 3D information generation unit 240, is fed back to the reference frame switching unit 220'. Furthermore, while processing is underway in the reference frame switching unit 220', the reference frame can be switched without using the travel direction signal of the movable device 1000' by switching the write destination of the time-series feature point group based on whether the amount of camera movement is positive or negative.

[0126] As explained above, in the second embodiment, the camera's external parameters calculated during the generation of 3D information are fed back, and the time-series images and feature point groups are ordered based on the amount of camera movement. Therefore, according to the second embodiment, a reference frame can be determined without detecting the travel direction signal of the mobile device.

[0127] Therefore, without detecting the direction of travel of the mobile device, the movement direction of the camera position can be restricted to one direction based on the amount of camera movement, thus improving the speed of feature point matching processing in the same manner as in the first embodiment. Furthermore, the speed of SfM and vSLAM processing can also be improved.

[0128] <Third Embodiment>

[0129] Next, a third embodiment of this disclosure will be described. In the third embodiment, the turning direction of the mobile device 1000'' is detected based on the steering angle of the steering wheel 30b built into the mobile device 1000'', and this is input into the corresponding point processing 232''. Therefore, the third embodiment is configured to perform corresponding point processing by referring to a search range map 233a corresponding to the turning state of the mobile device 1000''. The travel direction detection unit 30'' that detects the steering angle of the steering wheel 30b is an example of a turning direction detection unit that detects the turning direction of the mobile device. Based on the detection results of the travel direction detection unit 30'', the feature point matching unit 230 searches for each feature point in the search object image within a limited search range.

[0130] Figure 15 This is a configuration diagram of a mobile device 1000'' according to a third embodiment. The mobile device 1000'' is, for example, a vehicle. Figure 15 The camera unit 10, feature point extraction unit 210, reference frame switching unit 220, and 3D information generation unit 240 are included. Figure 1 The camera unit 10, feature point extraction unit 210, reference frame switching unit 220 and three-dimensional information generation unit 240 are the same.

[0131] Feature point matching unit 230 has corresponding point search processing 232'' to replace Figure 1 The corresponding point search processing 232. Additionally, the mobile device 1000'' has a travel direction detection unit 30'' to replace... Figure 1The direction of travel detection unit 30'' detects the state of the gear shift lever 30a and the steering angle of the steering wheel 30b. The steering wheel 30b determines the turning direction of the mobile device 1000'' based on its steering angle. The direction of travel detection unit 30'' can output the turning direction signal of the mobile device 1000'' by detecting the steering angle of the steering wheel 30b. In addition to the configuration of the corresponding point search processing 232, the corresponding point search processing 232'' also determines the turning state of the mobile device 1000''. Based on the turning direction signal of the mobile device 1000'' input from the direction of travel detection unit 30'', the corresponding point search processing 232'' determines whether the mobile device 1000'' is turning left, turning right, or not turning at all.

[0132] Furthermore, the corresponding point search process 232'' performs matching within a restricted area based on the turning state determination result, by referring to the search range map 233a corresponding to the turning state of the mobile device 1000'' stored in ROM 233. The search range map 233a includes multiple search range maps with different search ranges, and for example, includes a search range map when the mobile device is turning right, a search range map when the mobile device is turning left, and a search range map when the mobile device is going straight. This ensures that the search range map 233a corresponding to the turning state of the mobile device 1000'' is, for example, one of the search range maps when the mobile device is turning right, when the mobile device is turning left, and when the mobile device is going straight.

[0133] Note that although in the third embodiment, no distinction is made based on the forward / backward direction of the mobile device, as explained in the first embodiment, this is because the movement direction of the camera position can be restricted to one direction regardless of the forward / backward direction of the mobile device.

[0134] Note that when the mobile device is moving forward (parallel to the ground) while turning, or when it is moving backward, ideally, the component of the core point's coordinates in the direction perpendicular to the ground will remain unchanged and will be uniquely determined based on the camera's mounting position and the angle formed between the camera's mounting position and the ground. Conversely, when the mobile device is moving backward, for example, while turning, the component of the core point's coordinates in the direction parallel to the ground will move in the same direction as the turning direction. Furthermore, when the mobile device is moving forward, for example, while turning, the component of the core point's coordinates in the direction parallel to the ground will move in the opposite direction to the turning direction.

[0135] For the sake of simplicity, in Figure 16 The diagram shows, for example, the direction of movement of the camera position, the position of the core point, and the position of the feature point when the mobile device 1000'' is restricted to, for example, moving backward during a right turn.

[0136] Figure 16 This diagram illustrates the situation where the movable device 1000'' is turning right. Figure 16 In this context, points e and e' that overlap with the baseline connecting the camera position C in the reference frame and the camera position C' in the search target frame are designated as kernel points 161 and 161'. At this time, the mobile device 1000'' is moving backward parallel to the ground; therefore, kernel points e and e' exist at the camera position. Furthermore, this is the case where the mobile device 1000'' is moving backward while turning right; therefore, the coordinates of kernel points e and e' in the direction parallel to the ground shift to the right.

[0137] In contrast, the point p where the line connecting the camera position C and the coordinates P in the real space of the feature point overlaps with the image plane of each frame is designated as feature point 162 in the image, and the point p' where the line connecting the camera position C' and the coordinates P in the real space of the feature point overlaps with the image plane of each frame is designated as feature point 162' in the image. Furthermore, the line connecting the epipolar point e and the feature point p is designated as epipolar line 163, and the line connecting the epipolar point e' and the feature point p' is designated as epipolar line 163'. Although epipolar line 163 and epipolar line 163' are on the same plane due to epipolar constraints, the third embodiment involves the mobile device 1000'' turning, thus changing the position and slope of the epipolar line in the image.

[0138] Therefore, in the third embodiment, the search range for the feature point transitioning from p to p' cannot uniquely determine the direction facing the kernel point. However, in the case where the mobile device is turning right while moving backward, such as Figure 16As shown, when the feature point changes from p to p', the coordinates of the feature point in the direction parallel to the ground move to the right, and the coordinates of the feature point in the direction perpendicular to the ground move in the direction closer to the camera position.

[0139] In the case where the mobile device is moving backward while turning left, symmetry is also assumed, and the coordinates of the feature point in the direction parallel to the ground move to the left, while the coordinates of the feature point in the direction perpendicular to the ground move in the direction closer to the camera position.

[0140] Therefore, the search range map is differentiated for right turns, left turns, and straight-ahead movements. The component of the search range perpendicular to the ground during a right turn is restricted to a direction close to the camera position, and the component parallel to the ground during a right turn is restricted to a direction close to the right edge of the image. Conversely, symmetry is also assumed for the search range during a left turn, and the component parallel to the ground during a left turn is restricted to a direction close to the left edge of the image. The search range during straight-ahead movements is restricted in the same manner as in the first embodiment.

[0141] By determining the search range in this way, corresponding point searches can be performed within a search range that has been appropriately limited based on the turning state of the mobile device, and the computational complexity of corresponding point search processing 232 can be reduced.

[0142] Apart from Figure 1 In addition to the configuration of the direction of travel detection unit 30, the direction of travel detection unit 30'' also includes, for example, a steering wheel 30b built into the vehicle. Furthermore, the direction of travel detection unit 30'' detects the turning direction of the mobile device 1000'' based on the steering angle of the steering wheel 30b. For example, if the steering angle of the steering wheel 30b is tilted clockwise, it is determined that the turning direction of the mobile device 1000'' is right. If the steering angle of the steering wheel 30b is tilted counterclockwise, it is determined that the turning direction of the mobile device 1000'' is left. If the steering angle of the steering wheel 30b is not tilted, it is determined that the mobile device 1000'' is not turning.

[0143] Note that, generally, turning means rotating as if drawing an arc, and therefore, even if the handle's steering angle is clockwise, the direction commonly referred to as the steering direction of the mobile device will change based on whether the mobile device is traveling forward or backward. However, in this context, the steering direction is defined as described above, regardless of whether the mobile device is traveling forward or backward.

[0144] Note that although there are various criteria for judging the steering angle of the controller according to the design, this makes it possible to define the angle as positive when the starting point of the controller's steering angle is 0° and the controller has already rotated clockwise, and the angle as negative when the starting point of the controller's steering angle is 0° and the controller has already rotated counterclockwise. Furthermore, this can be achieved by preparing two fixed thresholds: a right-turn threshold for when the controller is already rotating in the positive direction and a left-turn threshold for when the controller is already rotating in the negative direction. If the right-turn threshold is exceeded, it is determined that a right turn is in progress; if the left-turn threshold is exceeded, it is determined that a left turn is in progress.

[0145] In addition, the travel direction detection unit 30'' outputs the turning direction of the mobile device 1000'' to the corresponding point search processing 232''.

[0146] Figure 17 This is a flowchart illustrating a series of operations of the integration processing unit 200 according to the third embodiment. Note that... Figure 17 Each step in steps S101, S102, and S110 to S130 is related to... Figure 8 The steps S101, S102 and S110 to S130 are the same, so their descriptions will be omitted.

[0147] During step S103'', except Figure 8 In addition to the processing in step S103, the integration processing unit 200 also obtains the turning direction of the mobile device 1000 from the travel direction detection unit 30''. Afterwards, the processing proceeds to step S110.

[0148] Figure 18 This is a flowchart illustrating a series of processes performed by the feature point matching unit 230 according to the third embodiment. Note that... Figure 18 Each step in steps S121 to S123 and steps S125 to S127 is related to Figure 10 The steps S121 to S123 and S125 to S127 are the same, and their descriptions will be omitted.

[0149] The processing of step S124'' is based on the turning direction of the mobile device 1000'' obtained during step S103'', referring to the search range map 233a corresponding to the turning direction of the mobile device 1000'', and performing a corresponding point search within the search range corresponding to the cell selected during step S122. The details of the processing of step S124'' will be explained below. The processing of step S124'' includes the processing of steps S1241 to S1245.

[0150] During step S1241, the feature point matching unit 230 determines whether the turning direction of the mobile device 1000'' is a right turn. If the feature point matching unit 230 has determined that the turning direction is a right turn, the process of step S1243 is executed. If the feature point matching unit 230 has determined that the turning direction is not a right turn, the process of step S1242 is executed.

[0151] During step S1242, the feature point matching unit 230 determines whether the turning direction of the mobile device 1000'' is a left turn. If the feature point matching unit 230 has determined that the turning direction is a left turn, the process of step S1244 is executed. If the feature point matching unit 230 has determined that the turning direction is not a left turn, the process of step S1245 is executed.

[0152] During step S1243, feature point matching unit 230 refers to the search range map during right turn. Additionally, during step S1243, feature point matching unit 230 searches for corresponding points from the search object feature point group fp_sub within the search range corresponding to the cell selected during step S122, targeting the feature points corresponding to the feature points selected during step S123. Afterward, the process proceeds to step S125.

[0153] During step S1244, feature point matching unit 230 refers to the search range map during the left turn. Additionally, during step S1244, feature point matching unit 230 searches for corresponding points from the search object feature point group fp_sub within the search range corresponding to the cell selected during step S122, targeting the feature points corresponding to the feature points selected during step S123. Afterward, the process proceeds to step S125.

[0154] During step S1245, the feature point matching unit 230 refers to the search range map during straight-line movement. Additionally, during step S1245, the feature point matching unit 230 searches for corresponding points from the search object feature point group fp_sub within the search range corresponding to the cell selected during step S122, targeting the feature points corresponding to the feature points selected during step S123. Afterward, the process proceeds to step S125.

[0155] In this way, during the corresponding point search processing 232'', the turning state of the mobile device 1000'' is determined based on the turning direction of the mobile device 1000'' detected by the travel direction detection unit 30''. Furthermore, by referring to a search range map corresponding to the turning state of the mobile device 1000'', the corresponding point search can be performed by changing the range limited in response to the change in the turning direction of the mobile device 1000''.

[0156] As described above, according to the third embodiment, each search range corresponding to the turning state of the mobile device is predetermined, and the search range is updated in response to a change in the turning direction of the mobile device. Therefore, according to the third embodiment, even if the turning direction of the mobile device changes, the search for the corresponding point can begin in advance.

[0157] Therefore, even when the turning direction of the mobile device changes, each search range corresponding to the turning state of the mobile device is predetermined, and the search range is updated in response to the change in the turning direction of the mobile device. Thus, the speed of feature point matching processing can be improved in the same manner as in the first and second embodiments. Furthermore, the processing speed of SfM and vSLAM can also be improved.

[0158] <Fourth Embodiment>

[0159] Next, a fourth embodiment of the present disclosure will be described. The fourth embodiment is configured to switch the reference frame for feature point matching in a camera that captures images in the imaging direction based on the travel direction of the mobile device 1000''', so as not to include directions that are the same as or opposite to the travel direction of the mobile device 1000'''.

[0160] Figure 19 This is a configuration diagram of the mobile device 1000''' according to the fourth embodiment. The mobile device 1000''' is, for example, a vehicle. Figure 19 The integrated processing unit 200 and the travel direction detection unit 30 in the middle Figure 1 The integrated processing unit 200 and the travel direction detection unit 30 are the same.

[0161] Apart from Figure 1 In addition to the camera unit 10, the camera unit 10''' also has a right surround-view camera 10e and a left surround-view camera 10f. The right surround-view camera 10e and the left surround-view camera 10f are cameras used in the surround-view system (a system capable of confirming video images around the vehicle) of the vehicle's mobile device 1000'''.

[0162] exist Figure 20The mounting positions of the right surround-view camera 10e and the left surround-view camera 10f of the camera unit 10''' are shown. Note that although there are cases where the movable device 1000''' travels in the forward direction and cases where the movable device 1000''' travels in the backward direction, in Figure 20 The image shows a representative example of the mobile device 1000''' moving in a forward direction. Figure 20 In the diagram, the movable device 1000''' at time t is shown using a dashed line, and the movable device 1000''' at time t+1 is shown using a solid line.

[0163] The right panoramic camera 10e's camera direction is camera direction 201e, and the right panoramic camera 10e''s camera direction is camera direction 201e'. The left panoramic camera 10f's camera direction is camera direction 201f, and the left panoramic camera 10f''s camera direction is camera direction 201f'. The right panoramic camera 10e and the left panoramic camera 10f are attached with orientations that prevent them from capturing images in the same direction as the travel direction of the mobile device 1000''' and in the opposite direction to the travel direction of the mobile device 1000'''.

[0164] In this context, the image plane when the image is captured from the position of the right panoramic camera 10e at time t+1 is 202e, and the image plane when the image is captured from the position of the right panoramic camera 10e' at time t is 202e'. In this case, the kernel point where the baseline connecting the position of the right panoramic camera 10e from time t+1 and the position of the right panoramic camera 10e' at time t overlaps with the image plane 202e, and the kernel point where the baseline connecting the position of the right panoramic camera 10e from time t+1 and the position of the right panoramic camera 10e' at time t overlaps with the image plane 202e', do not appear in the captured image. However, the kernel point exists at infinity extending from the right edge of the captured image, and the kernel point exists at infinity extending from the left edge of the captured image.

[0165] Furthermore, the image plane when the image is captured from the position of the left panoramic camera 10f at time t+1 is 202f, and the image plane when the image is captured from the position of the left panoramic camera 10f' at time t is 202f'. In this case, the kernel point where the baseline connecting the position of the left panoramic camera 10f from time t+1 and the position of the left panoramic camera 10f' at time t overlaps with the image plane 202f, and the kernel point where the baseline connecting the position of the left panoramic camera 10f from time t+1 and the position of the left panoramic camera 10f' at time t overlaps with the image plane 202f', do not appear in the captured image. However, the kernel point exists at the point of infinity extending from the right edge of the captured image, and the kernel point exists at the point of infinity extending from the left edge of the captured image.

[0166] In both cases, the epipolar line connecting the kernel point and the feature point can be approximated as a line parallel to the ground and the camera position of the right panoramic camera, or a line parallel to the ground and the camera position of the left panoramic camera. Therefore, the coordinates of the feature points in the image in the direction perpendicular to the ground ideally remain unchanged, and the coordinates of the feature points in the direction perpendicular to the ground are transformed to the opposite direction of the camera position's movement based on the direction of camera position movement. For example, when there is a transformation from image plane 202e to image plane 202e', the feature point group transforms to the left in the image, and when there is a transformation from image plane 202f to image plane 202f', the feature point group transforms to the right in the image. Therefore, when performing a corresponding point search, it is sufficient to appropriately limit the search range based on the camera position movement direction as described above.

[0167] exist Figure 1 In the first embodiment, as described above, the reference frame switching unit 220 switches the write destination pointer of the reference frame and the write destination pointer of the search object frame based on the travel direction signal of the mobile device 1000.

[0168] In the fourth embodiment, when the travel direction of the mobile device 1000''' is forward, the reference frame switching unit 220, for example, sets the feature point group fp(t) to the search target feature point group fp_sub, and sets the feature point group fp(t+1) to the reference feature point group fp_ref. Conversely, when the travel direction of the mobile device is backward, for example, the reference frame switching unit 220 sets the feature point group fp(t+1) to the search target feature point group fp_sub, and sets the feature point group fp(t) to the reference feature point group fp_ref.

[0169] In the fourth embodiment, by doing so, even if there is a configuration that captures images in a direction that does not include the same direction as the travel direction of the mobile device 1000''' and the opposite direction to the travel direction of the mobile device 1000''', the movement direction of the camera position can be limited by switching the reference frame.

[0170] As described above, according to the fourth embodiment, firstly, a time-series feature point group is defined as a reference feature point group based on the travel direction of the mobile device. By doing so, even if there is a configuration for shooting in a direction that does not include the same direction as the travel direction of the mobile device 1000''' and the opposite direction to the travel direction of the mobile device 1000''', the movement direction of the camera position can be limited by switching the reference frame.

[0171] Therefore, even if there is a camera configuration that captures images in directions that do not include directions in the same direction as or opposite to the direction of travel of the mobile device, the movement direction of the camera position can be restricted to one direction. Thus, the speed of feature point matching processing can be improved in the same manner as in the first and second embodiments. Furthermore, the processing speed of SfM and vSLAM can also be improved.

[0172] Although a detailed description of the present disclosure has been given above based on embodiments thereof, the present disclosure is not limited to these specific embodiments, and the present disclosure also includes various modes without departing from the spirit of the present disclosure. Some of the above embodiments may also be suitably combined.

[0173] Additionally, this disclosure also includes cases where a software program that implements the functions of the above embodiments is provided to a system or apparatus having a computer capable of executing the program directly from a storage medium or by using wired / wireless communication, and the program is executed.

[0174] Therefore, the program code provided to and installed on a computer for using the computer to implement the functional processing of this disclosure is itself included in this disclosure. That is, the computer program used to implement the functional processing of this disclosure is itself included in this disclosure.

[0175] In this case, the form of the program, such as the object code, the program executed by the interpreter, and the script data supplied to the OS, is irrelevant, as long as it has the functionality of a program. As the storage medium for providing the program, for example, magnetic recording media such as hard disks and magnetic tapes, optical / magneto-optical storage media, and non-volatile semiconductor memories can also be used.

[0176] Alternatively, as a method of providing the program, the computer program forming this disclosure is stored on a server on a computer network, and a client computer with a connection downloads the computer program.

[0177] Other embodiments

[0178] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs / instructions) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU), microprocessor unit (MPU)) of the system or device reads out and executes the computer program / instructions.

[0179] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

[0180] According to this disclosure, an image processing apparatus can be provided that reduces the amount of processing associated with feature point matching and is capable of faster processing.

[0181] This application claims the benefit of Japanese Patent Application 2024-150437, filed on September 2, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing apparatus comprising: a photographing position movement direction limiting section configured to select a first image and a second image from a plurality of time-series images that have been acquired from a photographing section arranged on a movable apparatus for capturing an image of surroundings of the movable apparatus, and control a movement direction of the photographing position to a first position and a second position, wherein the first image is an image captured when the position of the movable apparatus is the first position, and the second image is an image captured when the position of the movable apparatus is the second position; a feature point matching section configured to select a reference image and a search target image from the first image and the second image, and perform feature point association; a reference image switching section configured to switch the reference image based on information showing a movement direction of the movable apparatus, wherein the feature point matching section searches each feature point in the search target image within a search range that has been limited based on the movement direction of the movable apparatus. The feature point matching section determines the search range based on range information that has been determined in advance, and searches each feature point in the search target image.

2. The image processing device according to claim 1, wherein 3. The image processing apparatus according to claim 1, further comprising a travel direction detection section configured to detect a travel direction of the movable apparatus, the information showing the movement direction of the movable apparatus is generated based on a detection result of the travel direction detection section. wherein 4. The image processing apparatus according to claim 1, further comprising a photographing position posture calculation section configured to calculate information related to a position and a posture of the photographing section, the photographing position posture calculation section generates the information showing the movement direction based on the information related to the position and the posture of the photographing section. wherein, 5. The image processing apparatus according to claim 1, further comprising a turning direction detection section configured to detect a turning direction of the movable apparatus, the feature point matching section searches each feature point in the search target image within a search range that has been limited based on a detection result of the turning direction detection section. wherein The photographing section is arranged on the movable apparatus by causing a photographing direction of the photographing section to include a direction identical to or opposite to a travel direction of the movable apparatus.

6. The image processing device according to claim 1, wherein The reference image switching section switches the reference image so that the movement direction of the movable apparatus becomes a direction opposite to a photographing direction of the photographing section.

7. The image processing device according to claim 6, wherein 8. A control method of an image processing apparatus, the control method comprising: ​ a camera position movement direction restriction step for selecting a first image and a second image from a plurality of time-series images that have been acquired from an image capturing section arranged on a movable device for capturing images of the surroundings of the movable device, and controlling a movement direction of the camera position to a first position and a second position, wherein the first image is an image captured when the position of the movable device is the first position, and the second image is an image captured when the position of the movable device is the second position; a feature point matching step for selecting a reference image and a search target image from the first image and the second image, and performing feature point association; and a reference image switching step for switching the reference image based on information showing a movement direction of the movable device, wherein in the feature point matching step, each feature point in the search target image is searched for within a search range that has been restricted based on the movement direction of the movable device.

9. A non-transitory storage medium storing a program of an image processing device, the program causing a computer to perform each step of a method for the image processing device, the method comprising: a camera position movement direction restriction step for selecting a first image and a second image from a plurality of time-series images that have been acquired from an image capturing section arranged on a movable device for capturing images of the surroundings of the movable device, and controlling a movement direction of the camera position to a first position and a second position, wherein the first image is an image captured when the position of the movable device is the first position, and the second image is an image captured when the position of the movable device is the second position; a feature point matching step for selecting a reference image and a search target image from the first image and the second image, and performing feature point association; and a reference image switching step for switching the reference image based on information showing a movement direction of the movable device, wherein in the feature point matching step, each feature point in the search target image is searched for within a search range that has been restricted based on the movement direction of the movable device.

10. A computer program product comprising a program of an image processing device, the program causing a computer to perform each step of a method for the image processing device, the method comprising: a camera position movement direction restriction step for selecting a first image and a second image from a plurality of time-series images that have been acquired from an image capturing section arranged on a movable device for capturing images of the surroundings of the movable device, and controlling a movement direction of the camera position to a first position and a second position, wherein the first image is an image captured when the position of the movable device is the first position, and the second image is an image captured when the position of the movable device is the second position; a feature point matching step for selecting a reference image and a search target image from the first image and the second image, and performing feature point association; and a reference image switching step for switching the reference image based on information showing a moving direction of the mobile device, wherein in the feature point matching step, each feature point in the search target image is searched within a search range that has been limited based on the moving direction of the mobile device.

Images