Image processing device, image processing method, and program

Abstract

To improve alignment accuracy between images photographed by an imaging apparatus when a constant posture change occurs in the imaging apparatus.SOLUTION: Information on a feature point included in a first image obtained by the imaging apparatus is acquired. The feature point included in a second image obtained by the imaging apparatus, corresponding to the feature point included in the first image is searched from a search range set in the second image. A positional deviation between the feature point included in the first image and the feature point included in the second image corresponding to the feature point included in the first image is determined. A new search range set to the second image is decided on the basis of the information indicating a history of the positional deviation determined for each of a plurality of second images sequentially obtained. The search range is updated on the basis of the decision.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program, and in particular to a technique for aligning images. [Background technology]

[0002] While an imaging device captures multiple images of the same subject, the orientation of the imaging device may change due to factors such as vibration. In this case, the images of the subject in each captured image do not necessarily match. Therefore, techniques for aligning images captured at different times are known. Such techniques can be used as preprocessing for processes that assume that the images of the same subject match between images. For example, in a process of extracting a foreground using background subtraction, the foreground can be extracted with high accuracy by using multiple images that are aligned so that the positions of the same subject match. As another example, a video can be generated by combining images captured by multiple cameras. In this case, the orientation of each camera can be determined by calibration. Then, based on the orientation of each camera, correction processing for alignment is performed on the images captured by each camera, and the images can be combined.

[0003] Such alignment can be performed by searching for features in one image that correspond to features in the other image. Such correspondence can be performed based on the similarity of the image around the feature or its feature amount. Patent Document 1 discloses a technique for aligning a first image (hereinafter referred to as a reference image) obtained by capturing images during calibration with multiple second images (hereinafter referred to as correction target images) used for video generation. In Patent Document 1, shifts between the reference image and the correction target image are detected using patch data extracted from the reference image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 10,121,262 Summary of the Invention [Problem to be solved by the invention]

[0005] The orientation of an imaging device can change not only due to vibration, but also due to temperature changes caused by sunlight or the passage of time. Attitude changes due to vibration generally occur over a short period of time. On the other hand, attitude changes due to temperature changes or the passage of time can occur over a long period of time. Long-term attitude changes can lead to a large image misalignment between the reference image and the correction target image. In this case, it becomes more likely that features in the correction target image that correspond to features in the reference image cannot be found. Expanding the search range for features in the correction target image to address this issue increases processing time. Furthermore, matching between incorrect corresponding points can result in reduced alignment accuracy.

[0006] The present disclosure aims to improve the accuracy of alignment between images captured by an imaging device when the imaging device is constantly changing in posture. [Means for solving the problem]

[0007] An image processing apparatus according to an embodiment has the following configuration: The imaging system includes an acquisition means for acquiring information on feature points included in a first image obtained by an imaging device, a search means for searching for feature points included in a second image obtained by the imaging device that correspond to the feature points included in the first image within a search range set in the second image, a determination means for determining a positional deviation between the feature points included in the first image and feature points included in the second image that correspond to the feature points included in the first image, a determination means for determining a new search range to be set in the second image based on information indicating a history of the positional deviation determined for each of a plurality of second images obtained sequentially, and an update means for updating the search range based on the determination of the determination means. [Effects of the Invention]

[0008] It is possible to improve the accuracy of alignment between images captured by the imaging device when the imaging device is constantly changing in posture. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system according to an embodiment. [Figure 2] FIG. 10 is a diagram showing an example of a format of feature point data. [Figure 3] FIG. 1 is a diagram showing an example of the hardware configuration of an image processing apparatus. [Figure 4] FIG. 10 is a diagram showing an example of a search range for feature points. [Figure 5] 10 is a flowchart showing an example of a method for calculating a feature point correction amount. [Figure 6] 1 is a flowchart showing a processing example of an image processing method according to an embodiment. [Figure 7] 10 is a flowchart of an example of processing for determining whether or not to apply a feature point correction amount. [Figure 8] 10 is a flowchart of an example process for applying a feature point correction amount. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] 1 shows an example of the configuration of an image processing device according to an embodiment. The image processing device 100 includes a feature selection unit 101, a data storage unit 102, a feature search unit 103, a correction amount calculation unit 104, a deformation amount calculation unit 105, and a control unit 106. The image processing device 100 is also connected to a deformation unit 107, an image input unit 108, and an imaging unit 109. An image processing system according to an embodiment includes the image processing device 100, the deformation unit 107, the image input unit 108, and the imaging unit 109.

[0012] The imaging unit 109 generates a reference image (first image) and a correction target image (second image) by capturing an image of a subject. The imaging unit 109 can also sequentially generate multiple correction target images by repeatedly capturing an image of the subject. The reference image and the correction target image contain the same subject. However, the reference image and the correction target image may contain a changing subject in addition to a stationary subject. For example, the imaging unit 109 may generate the reference image and the correction target image by capturing an image of a field at different times. Here, additional subjects, such as players, may be moving on the field, which is the subject. Even in such a case, the reference image and the correction target image can be aligned based on feature points corresponding to a stationary subject, such as the field. The feature points will be described later.

[0013] The image input unit 108 stores the reference image and the correction target image generated by the imaging unit 109 in the data storage unit 102 .

[0014] The deformation unit 107 performs a correction process on the reference image and / or the correction target image to align the reference image and the correction target image. This correction process can be performed based on the positional deviation between the feature points included in the reference image and the feature points included in the correction target image, which is determined by the deformation amount calculation unit 105. In this embodiment, the deformation unit 107 performs a correction process on the correction target image based on the deformation amount calculated by the deformation amount calculation unit 105. The deformation unit 107 can generate a corrected image by applying the deformation amount calculated by the deformation amount calculation unit 105 to the correction target image.

[0015] The correction processing method is not particularly limited. As will be described later, the deformation amount calculated by the deformation amount calculation unit 105 indicates the positional deviation between feature points included in the reference image and corresponding feature points included in the correction target image. In other words, this deformation amount indicates the correspondence between feature points included in the reference image and feature points included in the correction target image. The deformation amount calculation unit 105 can calculate correction parameters between the reference image and the correction target image based on such correspondence for each of a plurality of feature points. For example, this correction parameter can be expressed as an affine transformation or a homography transformation. The deformation unit 107 can then generate a corrected image by applying a correction processing using such correction parameters to the correction target image. Note that the deformation unit 107 may use a deformation amount (i.e., a rotation amount) in a direction θ (described later) when calculating the correction parameters. On the other hand, the deformation unit 107 may perform a correction processing on the reference image for alignment.

[0016] The corrected image can be used for various purposes. For example, an image processing unit (not shown) included in the image processing device 100 can use the reference image and the corrected image to perform processing to separate the foreground from the background in the corrected image. In this way, the foreground area can be extracted from the corrected image. Furthermore, the image processing unit (not shown) included in the image processing device 100 can generate a three-dimensional model of the subject using the reference image and the corrected image, for example, according to the volume intersection method.

[0017] The imaging unit 109 can be realized by an imaging device such as a camera. At least one of the transformation unit 107 and the image input unit 108 can be realized by an information processing device such as a server or a computer. Alternatively, the image processing device 100 may have at least one of the transformation unit 107 and the image input unit 108. Alternatively, the imaging unit 109 may have the image input unit 108.

[0018] The feature selection unit 101 acquires information about feature points included in the reference image. For example, the feature selection unit 101 can store data of a first feature point group including a plurality of feature points selected from the reference image in the data storage unit 102. In this specification, each feature point included in the first feature point group may be referred to as a first feature point. The feature point data may include, for example, the coordinates of the feature point and the feature amount of the feature point.

[0019] The method for determining feature points is not particularly limited. For example, feature points may be lattice points of the reference image, or points selected from the lattice points of the reference image. Furthermore, feature points may be points indicating edges in the reference image. In the following description, the feature selection unit 101 extracts a first group of feature points using the method described in Patent Document 1. The method for calculating feature quantities is also not particularly limited. For example, feature quantities can be calculated based on pixel values ​​in the surrounding areas of feature points. As a specific example, feature quantities may be calculated by inputting a partial image around the feature points into a neural network. Furthermore, the partial image around the feature points may itself be used as the feature quantity. In the following description, feature quantities are a set of values ​​calculated according to the algorithm described in Patent Document 1. An example format of feature point data 200 used in the following description is shown in FIG. 2. The feature point data 200 includes coordinates 201, a direction 202, and a feature quantity 203. The direction 202 is calculated according to the algorithm described in Patent Document 1.

[0020] However, it is not necessary for the feature selection unit 101 to extract feature points. For example, feature points and their feature amounts in a reference image may be determined in advance. In this case, the feature selection unit 101 can acquire information about feature points that is stored in advance in a memory such as the auxiliary storage device 314.

[0021] The feature search unit 103 searches for feature points in the correction target image that correspond to feature points in the reference image. In this embodiment, the feature search unit 103 searches for feature points within a search range set in the correction target image. Here, as will be described later, the search range for the coordinates of the feature points in the reference image is set in advance. The feature search unit 103 can search for corresponding feature points in the correction target image for each of the first feature point groups. For example, the feature search unit 103 can evaluate the differences between the feature amounts of the feature points in the reference image and the feature amounts of each point in the search range of the correction target image. Then, based on the difference evaluation results, it can determine feature points in the correction target image that correspond to the feature points in the reference image. In this example, the difference between the feature amounts of the feature points in the reference image and the corresponding feature amounts is equal to or less than a threshold. Furthermore, the difference between the feature amounts of the feature points in the reference image and the corresponding feature amounts is smaller than the difference between the feature amounts of the feature points in the reference image and the feature amounts of the other points in the search range.

[0022] In this specification, feature points included in the image to be corrected that correspond to first feature points included in the reference image may be referred to as second feature points. In the following description, the feature search unit 103 searches for coordinates and directions in the image to be corrected that provide feature amounts closest to those of each feature point in the first feature point group. When the feature search unit 103 has finished searching for corresponding feature points for all first feature points included in the first feature point group, it notifies the deformation amount calculation unit 105 of the search results.

[0023] As described above, in this embodiment, the feature searching unit 103 searches within a search range set in the image to be corrected for feature points included in the image to be corrected that correspond to feature points included in the reference image. On the other hand, the feature searching unit 103 may search within a search range set in the reference image for feature points included in the image to be corrected that correspond to feature points included in the image to be corrected.

[0024] The deformation amount calculation unit 105 determines the positional deviation between a feature point included in the reference image and a feature point included in the correction target image that corresponds to the feature point included in the reference image. This positional deviation can be expressed, for example, by the amount of movement in the X and Y directions. In this specification, information indicating this positional deviation may be referred to as the deformation amount. Furthermore, the feature search unit 103 can determine the positional deviation for each of the first feature points as described above. In the following description, the feature search unit 103 calculates the amount of change in coordinates and direction between a first feature point in the reference image and a second feature point searched for in the correction target image as the deformation amount corresponding to the first feature point.

[0025] The correction amount calculation unit 104 determines a new search range for the correction target image based on information indicating the history of positional deviations determined for each of the multiple correction target images obtained sequentially. In this embodiment, the correction amount calculation unit 104 calculates feature point correction amounts used to determine the search range for the correction target image, as will be described later. The correction amount calculation unit 104 can also request the control unit 106 to update the setting of the feature point correction amounts.

[0026] The control unit 106 updates the setting of the search range of the correction target image used by the feature search unit 103 to search for feature points, based on the new search range determined by the correction amount calculation unit 104. By performing processing described below, the control unit 106 can update the setting of the feature point correction amount when the deviation between the reference image and the correction target image consistently exceeds a predetermined value. The control unit 106 can also control each unit of the image processing device 100.

[0027] The image processing device according to this embodiment can be realized by a computer including a processor and a memory. Fig. 3 is a block diagram showing an example of the hardware configuration of the image processing device 100. As shown in Fig. 3, the image processing device 100 has a CPU 311, a ROM 312, a RAM 313, an auxiliary storage device 314, and a bus 318. The image processing device 100 is also connected to a display unit 315, a communication unit 316, and an operation unit 317.

[0028] The CPU 311 controls the entire image processing device 100 using computer programs or data stored in the ROM 312 or the RAM 313. The ROM 312 stores programs or data that do not require modification. The RAM 313 temporarily stores programs or data supplied from the auxiliary storage device 314, or data such as image data supplied from the outside via the communication unit 316. The auxiliary storage device 314 is, for example, a hard disk or a solid state drive. The auxiliary storage device 314 can accumulate images input from the communication unit 316. In this way, a processor such as the CPU 311 can realize the functions of each unit shown in FIG. 1 by executing programs stored in a memory such as the ROM 312, the RAM 313, or the auxiliary storage device 314.

[0029] The display unit 315 is a display device such as a liquid crystal display. The display unit 315 can display a GUI (Graphical User Interface) that a user uses to use the image processing device 100, or a corrected image. The operation unit 317 is a device that accepts user input, such as a keyboard or a mouse. The operation unit 317 can input various instructions to the CPU 311 in accordance with user operations. The communication unit 316 is a communication device for communicating with an external device. For example, if the image processing device 100 is connected to an external device via a wired connection, a communication path such as a LAN cable is connected to the communication unit 316. Furthermore, if the image processing device 100 has a function for wirelessly communicating with an external device, the communication unit 316 has an antenna. The bus 318 connects the various components of the image processing device 100, the display unit 315, the communication unit 316, and the operation unit 317. The bus 318 transmits information among these components.

[0030] Next, an example of a search range that is the target of feature point search by the feature search unit 103 will be described. The following describes a case where the feature search unit 103 searches the correction target image for a feature point that corresponds to feature point A included in the first feature point group extracted from the reference image. The feature search unit 103 calculates feature amounts for points within the search range that are in the correction target image and that are near the coordinates of feature point A. The feature amount for a point can be calculated based on pixel values ​​of a partial area centered on the point. The feature search unit 103 then determines that the point having the feature amount closest to the feature amount of feature point A is the feature point that corresponds to feature point A. Hereinafter, the feature point that is detected from the correction target image and corresponds to feature point A may be referred to as feature point B.

[0031] In this embodiment, the feature search unit 103 searches for coordinates and directions that provide feature amounts closest to those of feature point A. Here, a case will be described in which the two-dimensional coordinates of feature point A are (x', y'). In this example, the search range is M×N pixels. In the following description, coordinates represent lattice points between pixels. However, the coordinates may represent the center coordinates of pixels.

[0032] The feature search unit 103 scans each point in a search range represented by coordinates (x'-M / 2, y'-N / 2) to (x'+M / 2, y'+N / 2) in the correction target image, and calculates the feature amount for each direction for each point. In this example, the feature amount for each point is calculated based on pixel values ​​in a partial area of ​​M × N pixels. If the direction of feature point A is θ°, the feature search unit 103 scans each point in the coordinates (x'-M / 2, y'-N / 2) to (x'+M / 2, y'+N / 2) of the correction target image rotated by -θ°, and calculates the feature amount for each point from the M × N partial area in each direction. Note that the size of the search range does not need to match the size of the partial area referenced to calculate the feature amount.

[0033] In the above example, the coordinates of the feature points in the reference image and the correction target image are the same. On the other hand, correction is possible even when there is a constant misalignment (described later) between the coordinates of the feature points in the reference image and the coordinates of the feature points in the correction target image. In this embodiment, the search range in the correction target image corresponding to the coordinates of a feature point (feature point A) included in the reference image can be corrected. This corrected search range is set using the feature point correction amount. The feature point correction amount has values ​​in the X and Y directions. The feature point correction amount also indicates the relative position of the coordinates of the reference point of the search range with respect to the coordinates of feature point A. To reiterate, the search range is an area of ​​a predetermined size (e.g., an area of ​​M×N pixels) centered on the reference point. For example, when the feature point correction amount = (0,0), i.e., when there is no constant misalignment between the coordinates of the feature points in the reference image and the coordinates of the feature points in the correction target image, the search range is an area of ​​M×N pixels centered on the coordinates (x', y'). On the other hand, for example, when the feature point correction amount is (-3, -4), that is, when there is a constant deviation between the coordinates of the feature point in the reference image and the coordinates of the feature point in the image to be corrected, the search range is an M x N pixel area centered at coordinates (x'-3, y'-4). In this example, the size of the search range is constant (e.g., M x N pixels) before and after correction, that is, the search range shifts according to the feature point correction amount.

[0034] Here, the deformation amount calculation unit 105 may determine the positional deviation of feature point B relative to the reference point. In this specification, information indicating the positional deviation of feature point B relative to the reference point is called the displacement amount. In this case, the deformation amount calculation unit 105 can calculate the positional deviation of feature point B relative to feature point A, i.e., the deformation amount, based on the displacement amount and the feature point correction amount. Specifically, the deformation amount calculation unit 105 can calculate the deformation amount by adding the feature point correction amount to the coordinate value indicated by the displacement amount.

[0035] The processing performed by the feature searching unit 103 will be further described with reference to Fig. 4. In this example, a reference image and an image to be corrected are obtained by capturing an image of an unchanging object 405. In this example, the feature searching unit 103 calculates feature amounts for a partial region of 4 x 4 pixels (M = 4 and N = 4) centered on a feature point.

[0036] Fig. 4(A) shows an example in which the camera posture is the same when capturing the reference image and the correction target image. Note that the feature point correction amount is (0,0) in Figs. 4(A) to 4(C). When the coordinates of feature point A (feature point 400) are (a,b), the feature search unit 103 searches for a feature point that can obtain a feature amount corresponding to feature point A while scanning each point within the coordinates (a-2,b-2) to (a+2,b+2) indicated by a search range 401 in the correction target image.

[0037] Region 402 of the correction target image is a region in the correction target image that is referenced when searching for a feature point corresponding to feature point A. Regions 403 and 404 indicate regions that are referenced to calculate feature amounts corresponding to directions of 0° and 45° at coordinates (a-2, b+2) in region 402, respectively. If the camera orientations when the reference image and the correction target image were captured are the same, the feature amount calculated from the region in the correction target image centered at coordinates (a, b) and facing a direction of 0° (which corresponds to search range 401 in FIG. 4A) will be closest to the feature amount of feature point A. In this case, the coordinates and direction of feature point A are the same as those of the corresponding feature point B in the correction target image. Therefore, the displacement amount of the detected feature point is (X, Y, θ) = (0, 0, 0). Furthermore, in this example, the feature point correction amount is (0, 0), so the deformation amount between feature point A and feature point B is (X, Y, θ) = (0, 0, 0).

[0038] FIG. 4(B) shows an example in which, due to a change in camera posture, feature point A (feature point 400), which was at coordinates (a, b) in the reference image, moves to coordinates (a+2, b+2) in the correction target image, and the image around the feature point is further rotated by -30°. In this example, the feature amount calculated from area 406 obtained by rotating a 4x4 pixel area centered at coordinates (a+2, b+2) in the correction target image by -30° is closest to the feature amount of feature point A. In this case, the displacement amount of the detected feature point is (X, Y, θ) = (2, 2, -30°). Furthermore, since the feature point correction amount = (0, 0), the deformation amount between feature point A and feature point B is (X, Y, θ) = (2, 2, -30°).

[0039] FIG. 4C shows a case where, due to a change in camera posture, feature point A (feature point 400) located at coordinates (a, b) in the reference image moves to coordinates (a-4, b-4) in the correction target image. In this example, region 408, which corresponds to coordinates (a-6, b-6) to (a-2, b-2) in the correction target image, is the region corresponding to feature point A. However, the search range of feature search unit 103 extends to coordinates (a-2, b-2), and therefore the referenced region extends to region 407 (in the case of a direction of 0°). In this example, the subject captured in region 407 is different from the subject captured in region 408, so a feature amount corresponding to (for example, the same as) the feature amount of feature point A cannot be obtained from the correction target image.

[0040] In such a case, the feature searching unit 103 determines that there is no feature point corresponding to feature point A in the correction target image. Furthermore, the amount of deformation for feature point A is not calculated. In other words, feature point A is not used for alignment. Note that if no corresponding feature point is found for all feature points included in the first feature point group, the feature searching unit 103 can notify the deformation amount calculation unit 105 of an error. If the number of first feature points for which corresponding feature points have been found is equal to or less than a threshold, the feature searching unit 103 may also notify the deformation amount calculation unit 105 of an error. In this case, processing on the correction target image being processed is terminated, and correction processing on this correction target image is not performed.

[0041] On the other hand, in this embodiment, the search range is corrected based on the feature point correction amount. FIG. 4(D) shows a case where the camera posture changes similar to that shown in FIG. 4(C). In this example, (X, Y) = (-3, -4) is set as the feature point correction amount. In this case, the feature search unit 103 searches the search range indicated by region 409, that is, the range of coordinates (a-2-3, b-2-4) to (a+2-3, b+2-4). Through this process, the feature search unit 103 can detect region 408 where a feature amount corresponding to feature point A (feature point 400) can be obtained. In this case, the displacement amount of the detected feature point B is (X, Y, θ) = (-2, -2, 0°). Furthermore, since the feature point correction amount = (-3, -4), the deformation amount between feature point A and feature point B is (X, Y, θ) = (-5, -6, 0°).

[0042] Fig. 5 is a flowchart of a process in which the correction amount calculation unit 104 calculates feature point correction amounts and updates correction values ​​used by the image processing device 100. As will be described later with reference to Fig. 6, the control unit 106 controls the correction amount calculation unit 104 to perform processing for each frame (S608). That is, the processing of the correction amount calculation unit 104 is performed for each of a plurality of correction target images. Note that before the processing shown in Fig. 5 starts, the deformation amount calculation unit 105 calculates the deformation amount (S606).

[0043] In this embodiment, the correction amount calculation unit 104 calculates feature point correction amounts based on the history of deformation amounts evaluated for each of a plurality of correction target images. These feature point correction amounts are used to update the feature point search range, as will be described later. In this way, by using the history of information indicating posture changes, the correction amount calculation unit 104 can calculate feature point correction amounts based on constant posture changes of the camera.

[0044] In the following embodiment, the correction amount calculation unit 104 determines a new search range based on the deformation amounts evaluated for one or more second images selected according to predetermined conditions, among the deformation amounts evaluated for each of the multiple correction target images. That is, the correction amount calculation unit 104 can calculate feature point correction amounts based on the deformation amounts selected according to the predetermined conditions. On the other hand, the correction amount calculation unit 104 can exclude deformation amounts that were not selected from the basis for calculating the feature point correction amounts (e.g., deformation amounts stored in the data storage unit 102). For example, if the correction amount calculation unit 104 estimates that the deformation amounts are affected by a temporary change in the camera's posture due to vibration or the like, it can exclude such deformation amounts. With this configuration, the correction amount calculation unit 104 can calculate the feature point correction amounts while suppressing the effects of temporary changes in the camera's posture.

[0045] In S501 to S503, the correction amount calculation unit 104 determines whether to use a deformation amount to calculate the feature point correction amount based on a predetermined index. In one embodiment, the predetermined condition includes a condition related to a change in the camera's posture when the correction target image was captured. For example, in S501, the degree of change in the camera's posture between frames is used as an index. Specifically, the correction amount calculation unit 104 determines whether the change in the camera's posture when the correction target image was captured is equal to or less than a predetermined standard. Then, based on the result of this determination, it can determine whether to select a correction target. If the change in the camera's posture between frames is large, it is estimated that the calculated deformation amount is affected by a temporary change in the camera's posture. In this case, the deformation amount is not used to calculate the feature point correction amount.

[0046] The correction amount calculation unit 104 may determine the degree of change in camera attitude based on information indicating the attitude of the camera when the correction target image was captured. For example, the correction amount calculation unit 104 can determine the change in camera attitude when the correction target image of the current frame was captured based on the attitude of the camera when the correction target image of the current frame was captured and the attitude of the camera when the correction target image of the previous frame was captured. The correction target image of the current frame is the correction target image captured at a first time point. The correction target image of the previous frame is the correction target image captured at a second time point before the first time point. The information indicating the camera attitude can be expressed by components in the x direction, y direction, and z direction (e.g., pan direction, tilt direction, and roll direction). The attitude of the camera when the correction target image was captured can be determined based on the amount of deformation. For example, the relative change in camera attitude between when the reference image was captured and when the correction target image was captured can be determined based on the correspondence between feature points included in the reference image and feature points included in the correction target image. As described above, such correspondence is indicated by the amount of deformation calculated by the deformation amount calculation unit 105. Then, the correction amount calculation unit 104 may compare the above-mentioned relative change in camera posture between the previous frame and the current frame.

[0047] In this case, the correction amount calculation unit 104 compares the camera orientation (x, y, z) when the correction target image of the current frame was captured with the camera orientation (x, y, z) when the correction target image of the previous frame was captured. Then, if the differences in the x, y, and z directions are each equal to or less than a predetermined value, the correction amount calculation unit 104 determines that the change in camera orientation is small. In this case, the process proceeds to S502. If not, the process of FIG. 5 ends.

[0048] On the other hand, the correction amount calculation unit 104 may determine the degree of change in the camera's posture based on the correction parameters calculated by the deformation unit 107. For example, the correction amount calculation unit 104 may compare the correction parameters calculated by the deformation unit 107 between the previous frame and the current frame.

[0049] Furthermore, the degree of change in camera posture may be evaluated based on the difference in the amount of deformation between the previous frame and the current frame. Specifically, the correction amount calculation unit 104 compares the amount of deformation (X, Y, θ) determined in the previous frame with the amount of deformation (X, Y, θ) determined in the current frame. In this embodiment, the amount of deformation is calculated for each feature point included in the first feature point group. The correction amount calculation unit 104 may calculate statistical values ​​(e.g., average values) of each component of the amount of deformation for each feature point between the previous frame and the current frame. Then, the correction amount calculation unit 104 can determine that the change in camera posture is small when the differences in the statistical values ​​of the components in the X, Y, and θ directions are each equal to or less than a predetermined value. Note that the correction amount calculation unit 104 may compare only the X and Y components of the determined amount of deformation between the previous frame and the current frame.

[0050] In this way, the degree of change in camera pose may be evaluated based on the estimated result of the camera pose, or may be evaluated based on other information affected by the camera pose (e.g., deformation amount or correction parameter). In one embodiment, the degree of change in camera position between frames is used as the index. That is, the degree of change in camera position and / or pose between frames can be used as the index.

[0051] In S502, the following indicators are used: whether or not there is a change in the characteristic subject, whether or not it is obscured by an obstacle, the result of feature point matching, or the number of matched feature points. If feature points selected from the background are occasionally obscured by the foreground, there may be frames in which the number of feature points decreases. Furthermore, if the shape of the subject used as a feature point changes due to weather conditions or the passage of time, differences in feature values ​​may occur between the reference image and the correction target image. In these cases, the accuracy of determining the amount of deformation may decrease. For example, if the number of matched feature points, i.e., the number of feature points in the correction target image detected as corresponding to feature points in the reference image, is small, the error in the amount of deformation calculated based on the matching may be large.

[0052] In one embodiment, the predetermined condition includes a condition regarding the number of feature points searched for in the correction target image that correspond to feature points included in the reference image. In this case, the correction amount calculation unit 104 determines whether the number of second feature points searched for in the correction target image that correspond to each of the first feature points is equal to or greater than a threshold. The correction amount calculation unit 104 can then select the correction target image based on the result of this determination. That is, if the correction amount calculation unit 104 determines that the number of associated feature points is equal to or greater than a predetermined number, it can determine that a deformation amount is used to calculate the feature point correction amount. In this case, the process proceeds to S503. Otherwise, the process of FIG. 5 ends. With this configuration, it is expected that a deformation amount with a small error will be used to calculate the feature point correction amount. Note that the specific index is not particularly limited. For example, the number of feature points for which an outlier deformation amount was obtained when the feature search unit 103 performed feature point association may be used. Specifically, the process may proceed to S503 if the ratio of feature points for which an outlier deformation amount was obtained to the total number of feature points for which a valid deformation amount was obtained is equal to or less than a predetermined value.

[0053] In S503, the difference between the reference image and the correction target image is used as an index. In one embodiment, the predetermined condition includes a condition regarding the difference between the correction target image and the reference image. For example, the correction amount calculation unit 104 can determine whether the difference between information (e.g., feature amount) of a first feature point included in the reference image and information (e.g., feature amount) of a second feature point searched for in the correction target image is equal to or less than a predetermined standard. Then, the correction amount calculation unit 104 can select the correction target image based on the determination result. For example, if the correction amount calculation unit 104 determines that the difference between the feature amount is equal to or less than the predetermined standard, the process proceeds to S504. If not, the process of FIG. 5 ends.

[0054] If the difference in such feature amounts is large, the accuracy of the correspondence may be low, and therefore the error in the amount of deformation may be large. Therefore, when comparing the feature amounts of the reference image and the image to be corrected, the feature search unit 103 can calculate the difference between the feature amount for the first feature point and the feature amount for the second feature point for each of the first feature point group. The feature search unit 103 can then store the feature amount difference calculated for each of the first feature points. At this time, the correction amount calculation unit 104 can use the sum of the feature amount difference values ​​as the difference in the feature amounts. That is, the correction amount calculation unit 104 determines whether the sum of the differences calculated for the feature points included in the first feature point group is equal to or less than a predetermined threshold, and can select the second image based on the determination that the sum of the differences is equal to or less than the predetermined threshold.

[0055] As another example, the comparison result between the reference image and the corrected image can be used as an index. That is, if the correction amount calculation unit 104 determines that the difference between the reference image and the corrected image is equal to or less than a predetermined standard, the process proceeds to S504. Otherwise, the process of FIG. 5 ends. With this configuration, it is possible to calculate the feature point correction amount based on a frame with a small difference, i.e., a highly reliable amount of deformation. As a specific example, the correction amount calculation unit 104 can compare surrounding pixel information for one or more corresponding feature points between the corrected image and the reference image. Then, the correction amount calculation unit 104 can use the similarity of such surrounding pixel information as information indicating the difference between the reference image and the corrected image. The similarity can be obtained, for example, by calculating the normalized cross-correlation for a predetermined region around the coordinates of the feature point.

[0056] Up to this point, various indices that can be used to determine whether or not to use a deformation amount to calculate a feature point correction amount have been described in S501 to S503. However, it is not necessary to use all of the indices described above, and it is not essential to use three types of indices. For example, only one of S502 and S503 may be performed. The order of S501 to S503 is also not limited. Furthermore, instead of S501 to S503, the correction amount calculation unit 104 can determine to use a deformation amount to calculate a feature point correction amount using one or more of the indices described so far. In this case, the process proceeds to S504. On the other hand, if the correction amount calculation unit 104 determines that a predetermined processing termination condition is met based on the indices described above, the process of FIG. 5 ends.

[0057] In another embodiment, the correction amount calculation unit 104 omits the process of determining whether to use deformation amounts to calculate feature point correction amounts based on a predetermined index. Even in this case, the correction amount calculation unit 104 can calculate feature point correction amounts based on a history of deformation amounts for multiple past frames in the process described below. This makes it possible to calculate feature point correction amounts that reflect constant changes in the camera's posture while suppressing the effects of temporary changes in the camera's posture.

[0058] In S504, the correction amount calculation unit 104 stores the determined amount of deformation for the frame currently being processed in the data storage unit 102.

[0059] In S505, the correction amount calculation unit 104 determines whether there is a sufficient history of deformation amounts for calculating the feature point correction amounts. In this embodiment, the history of deformation amounts for past frames is used to calculate the feature point correction amounts. Therefore, the correction amount calculation unit 104 can determine whether the deformation amounts determined for a predetermined number of frames are stored in the data storage unit 102. If there is a sufficient amount of deformation, the process proceeds to S506. If not, the process of FIG. 5 ends.

[0060] In S506, the correction amount calculation unit 104 calculates the feature point correction amount. The correction amount calculation unit 104 calculates the feature point correction amount based on the deformation amount information stored in the data storage unit 102, i.e., information indicating the history of positional deviations between the first feature point and the second feature point determined for each of the multiple correction target images obtained sequentially. As described above, the data storage unit 102 stores information on the deformation amounts for the correction target images selected in accordance with predetermined conditions in S501 to S503.

[0061] In this embodiment, the feature point correction amount is represented by (X, Y). The correction amount calculation unit 104 can determine a new search range based on statistical values ​​of the deformation amounts evaluated for each of the selected correction target images. For example, the correction amount calculation unit 104 can calculate the feature point correction amounts in the X and Y directions based on the deformation amounts in the X and Y directions for multiple frames. For example, the correction amount calculation unit 104 can use statistical values ​​such as the average or median of the deformation amounts determined in the past as the feature point correction amounts.

[0062] In this embodiment, the deformation amount is calculated for each feature point included in the first feature point group. The correction amount calculation unit 104 can calculate the feature point correction amount based on the deformation amount for each frame and each feature point. For example, the correction amount calculation unit 104 can calculate, for each feature point, statistical values ​​(e.g., average values) of the deformation amounts in the X and Y directions for multiple frames. Furthermore, the correction amount calculation unit 104 can calculate, as the feature point correction amount (X, Y), the statistical value (e.g., average value) of the statistical values ​​of the deformation amounts in the X and Y directions calculated for each feature point. The feature point correction amount calculated in this way can be commonly used to set search ranges for all feature points included in the first feature point group.

[0063] Alternatively, a feature point correction amount may be calculated independently for each feature point included in the first feature point group. In this case, a statistical value (e.g., an average value) of the deformation amounts in the X and Y directions for multiple frames calculated for each feature point can be used as the feature point correction amount for each feature point. That is, for each first feature point, a search range can be set according to the corresponding feature point correction amount. Then, the feature search unit 103 can search for a second feature point within the corresponding search range for each of the first feature points.

[0064] The correction amount calculation unit 104 can calculate the feature point correction amount based on the deformation amount for a predetermined number of frames. This predetermined number is not particularly limited. For example, when there is little constant posture deviation of the camera or when there is little vibration and sufficient deformation amount data available for calculating the feature point correction amount can be obtained in a short time, the correction amount calculation unit 104 can use the average of the deformation amounts as the feature point correction amount. On the other hand, when there is a lot of vibration and the posture of the camera is difficult to stabilize, the correction amount calculation unit 104 may use the latest deformation amount or the median or average of the deformation amounts for the most recent several frames as the feature point correction amount. Furthermore, when the deformation amount tends to change in a specific direction, the correction amount calculation unit 104 can use the latest deformation amount or the median or average of the deformation amounts for the most recent several frames as the feature point correction amount to minimize the influence of this tendency.

[0065] Furthermore, the method for calculating the feature point correction amount is not limited to the above method. For example, the correction amount calculation unit 104 may calculate the feature point correction amount according to the correction parameter calculated by the deformation unit 107 for each frame. Such a correction parameter is calculated based on the deformation amount. Therefore, the correction parameter indicates the positional deviation between the first feature point and the second feature point. Therefore, even with such a method, the feature point correction amount can be calculated based on information indicating the history of the deformation amount or information indicating the history of the positional deviation between the first feature point and the second feature point.

[0066] In S507, the correction amount calculation unit 104 notifies the control unit 106 of the completion of calculation of the feature point correction amount in order to reflect the feature point correction amount calculated in S506 in the feature point search range.

[0067] Next, an image processing method performed by the image processing device 100 will be described with reference to Fig. 6. This method includes a process of applying a change in the search range for feature points based on the feature point correction amount.

[0068] In S601, the image input unit 108 stores the reference image in the data storage unit 102. The image input unit 108 notifies the control unit 106 that the reference image input has been completed.

[0069] In S602, the control unit 106 instructs the feature selection unit 101 to extract feature points from the reference image. In this example, the control unit 106 extracts a first group of feature points from the reference image.

[0070] In S603, the feature selection unit 101 notifies the control unit 106 that extraction of the first feature point group from the reference image has been completed.

[0071] In the subsequent processing, correction target images corresponding to each of the multiple frames are input to the image processing device 100 from the image input unit 108. The correction target images are stored in the data storage unit 102. The control unit 106 performs control so as to generate corrected images by processing each correction target image.

[0072] In S604, the image input unit 108 notifies the control unit 106 that input of the correction target image has been completed. This notification may be made in units of one frame. Alternatively, this notification may be made in units of multiple frames (for example, in units of a video sequence). The processes of S605 to S610 are performed for each input frame of the correction target image.

[0073] In S605, the control unit 106 instructs the feature searching unit 103 to search for feature points included in the correction target image that correspond to the feature points extracted from the reference image as described above. In this example, the feature searching unit 103 searches for feature points corresponding to each of the first feature point group in the correction target image.

[0074] When the feature search unit 103 completes the search for all of the first feature points, it outputs the search results to the deformation amount calculation unit 105 in S606. Furthermore, the feature search unit 103 instructs the deformation amount calculation unit 105 to calculate the amount of deformation. Then, the deformation amount calculation unit 105 can calculate the amount of deformation for each feature point as described above. As described above, the correction amount calculation unit 104 may use a feature amount (for example, a difference between feature amounts) as the index described above. In this case, the deformation amount calculation unit 105 can store, in the data holding unit 102, the feature amount for the corresponding feature point included in the correction target image, which was calculated during the feature point search. When the calculation of the amount of deformation is completed, the deformation amount calculation unit 105 notifies the control unit 106 of the completion of processing in S607.

[0075] In S608, the control unit 106 instructs the correction amount calculation unit 104 to calculate the feature point correction amount. Then, the correction amount calculation unit 104 calculates the feature point correction amount by the method shown in FIG.

[0076] In S609, the control unit 106 instructs the deformation unit 107 to perform correction processing on the correction target image. The deformation unit 107 performs correction processing on the correction target image acquired from the data storage unit 102 by the method already described, and outputs a corrected image.

[0077] When the correction process is completed, in S610, the deformation unit 107 notifies the control unit 106 of the completion of the process. The image processing device 100 generates a corrected image corresponding to each correction target image by repeating the above process for each correction target image. As described above, the correction amount calculation unit 104 may use the comparison result between the corrected image and the reference image as the above-mentioned index. In this case, in order to make the determination in S502, after the process of S610 is completed, the control unit 106 may notify the correction amount calculation unit 104 of the completion of generation of the corrected image.

[0078] When the calculation process of the feature point correction amount is completed, in S611 the correction amount calculation unit 104 notifies the control unit 106 of the completion of the process. In S612, a process to be described later with reference to Figs. 7 and 8 is performed to apply the feature point correction amount to the feature search unit 103. When this process is completed, in S613 the feature search unit 103 notifies the control unit 106 of the completion of the process.

[0079] Thereafter, when a correction target image corresponding to a new frame is input, the processing from S604 onwards is repeated. At this time, the feature search unit 103 sets the feature point search range in accordance with the feature point correction amount applied in S612. This allows for more accurate feature point search, taking into account constant camera posture shifts and feature point position shifts.

[0080] The process performed in S612 for applying the feature point correction amount calculated by the correction amount calculation unit 104 in S608 to the feature search unit 103 will be described below.

[0081] In this embodiment, the control unit 106 instructs the feature search unit 103 to apply a feature point correction amount when a constant camera posture deviation occurs. Specifically, the control unit 106 can determine whether to update the search range setting based on the difference between a reference search range and a new search range determined by the correction amount calculation unit 104. That is, the control unit 106 can determine whether to apply a feature point correction amount based on the difference between the reference feature point correction amount and the feature point correction amount calculated by the correction amount calculation unit 104. This reference feature point correction amount may be an initial value, for example, (0,0). In this case, if the feature point correction amount calculated by the correction amount calculation unit 104 is greater than a predetermined threshold, the control unit 106 can determine to apply the feature point correction amount. The threshold is not particularly limited. For example, the thresholds for the feature point correction amounts in the X and Y directions may be half the search ranges in the X and Y directions by the feature search unit 103, respectively. In this case, the control unit 106 can determine to apply the feature point correction amount when the feature point correction amount in the X direction or Y direction exceeds the corresponding threshold. Camera posture deviation includes not only constant deviation but also deviation due to vibration. Therefore, the difference between the search range and the amount of feature point displacement that may occur due to camera vibration can be used as the threshold. With this configuration, it becomes easier to find corresponding feature points even when vibration is added in addition to constant posture deviation.

[0082] The reference feature point correction amount may be the currently set feature point correction amount. In other words, if a feature point correction amount has already been applied, and the difference between the feature point correction amount calculated by the correction amount calculation unit 104 and the currently applied feature point correction amount is greater than a predetermined threshold, the control unit 106 can determine to apply a new feature point correction amount.

[0083] Furthermore, it is not necessary to update the feature point correction amount for every frame. For example, the correction amount calculation unit 104 may instruct the feature search unit 103 to apply the feature point correction amount when a predetermined time (update interval) has elapsed since the previous update of the feature point correction amount. In this case, the update interval can be set to a time that is sufficiently shorter than the time it takes for constant changes in the camera's posture to occur.

[0084] 7 is a flowchart showing an example of processing performed by the control unit 106 to determine whether to apply a feature point correction amount. In S701, the control unit 106 waits for a time period set as an update interval to elapse since the previous application of a feature point correction amount. If the control unit 106 determines that the update interval has elapsed, the process proceeds to S702.

[0085] In S702, the control unit 106 determines whether the feature point correction amount calculated by the correction amount calculation unit 104 is greater than the threshold value. If the control unit 106 determines that the feature point correction amount is greater than the threshold value, the process proceeds to S703. If not, the process returns to S701.

[0086] In S703, the control unit 106 instructs the feature searching unit 103 to perform processing for applying the feature point correction amount. Fig. 8 is a flowchart showing an example of processing for applying the feature point correction amount, which is performed by the feature searching unit 103.

[0087] In S801, the feature search unit 103 determines whether the feature point correction amount calculated by the correction amount calculation unit 104 is a correction value that can be set in the feature search unit 103. For example, the feature search unit 103 can determine whether the feature point correction amount is within a predetermined upper limit. As described above, the feature point correction amount applied to the feature search unit 103 indicates values ​​to be added to the X and Y coordinate values ​​of the first feature point, respectively, in order to calculate a reference point for setting a search range for feature points in the correction target image. Furthermore, when a non-zero value is applied as the feature point correction amount, the feature search unit 103 calculates the deformation amount for the feature point by adding the X and Y direction values ​​indicated by the feature point correction amount to the calculated displacement amounts in the X and Y directions of the feature point, respectively.

[0088] If the feature search unit 103 determines that the feature point correction amount is equal to or less than the upper limit, the process proceeds to S803. Otherwise, the process proceeds to S802. In S802, the feature search unit 103 sets the upper limit of the applicable feature point correction amount as the feature point correction amount. Thereafter, the process proceeds to S803. Note that the processes of S801 to S802 can be performed independently for each of the X direction and the Y direction.

[0089] In S803, if a feature point search process is currently being performed, the feature search unit 103 waits for the feature point search process to finish. In this embodiment, the feature point correction amount application process may be performed in parallel with the feature point search process. Therefore, the feature search unit 103 can wait for the timing when the feature search process for one frame is completed. In this case, the correction amount calculation unit 104 calculates the feature point correction amount in S608 based on the feature point search process in S605, and then the feature search unit 103 can apply the feature point correction amount. In this way, the feature point correction value is updated in S804 when the feature search process for one frame is completed.

[0090] In S804, the feature search unit 103 applies the feature point correction amount by updating the feature point correction amount used when searching for feature points. In this way, both the correction value added to the reference point in the search range and the correction value added when calculating the deformation amount are switched to the updated value at the same time. In step S705, the control unit 106 is notified that the feature point correction amount has been updated.

[0091] With this configuration, the correction amount calculation unit 104 can calculate a feature point correction amount indicating a constant positional deviation between the reference image and the correction target image. Then, based on the feature point correction amount, the control unit 106 can set a feature point search range for the feature search unit 103 so as to include an area where a corresponding feature point is likely to exist. Note that the processing of this embodiment is equivalent to performing a correction process (e.g., translation) on one of the reference image and the correction target image according to the feature point correction amount, and then searching for a corresponding feature point. This corresponds to removing a constant positional deviation component from the correction target image and then searching for a corresponding feature point. In this case, the deformation amount of the feature point before removing the constant positional deviation component can be calculated based on the displacement amount calculated for the searched feature point and the feature point correction amount. This equivalent configuration is substantially the same as and is included in the above-described embodiment.

[0092] In S501, the correction amount calculation unit 104 determined whether to use a deformation amount to calculate the feature point correction amount based on the degree of change in the camera's posture. At this time, the correction amount calculation unit 104 evaluated the degree of change in the camera's posture based on the difference in the deformation amount between the previous frame and the current frame. However, the method for determining the change in the camera's posture is not limited to this method. For example, the correction amount calculation unit 104 can determine the change in the camera's posture when the correction target image was captured based on the output from a sensor provided in the camera.

[0093] For example, the camera may include a gyro sensor. The gyro sensor can measure changes in the camera's attitude (angular velocity). Such a sensor can output information indicating the camera's attitude or attitude change for each frame. In this case, the reference image and the correction target image generated by the imaging unit 109 can include, in addition to pixel information, information indicating the camera's attitude or information indicating changes in the camera's attitude (e.g., the speed of the attitude change in each direction). The information indicating the camera's attitude or attitude change can be expressed by components in the x, y, and z directions (e.g., pan, tilt, and roll directions). In this case, the image input unit 108 can associate the correction target image with the information indicating the attitude or attitude change using information identifying the image frame, such as a time code. The image input unit 108 can store this information in the data storage unit 102.

[0094] In this way, when information indicating the camera's attitude or attitude change is available, in S608 the control unit 106 instructs the correction amount calculation unit 104 to use this information. The correction amount calculation unit 104 acquires, from the data storage unit 102, information indicating the camera's attitude or attitude change for the frame corresponding to the correction target image used by the deformation amount calculation unit 105 to calculate the deformation amount. Then, the correction amount calculation unit 104 can determine the degree of the change in the camera's attitude based on this information.

[0095] As a specific example, the correction amount calculation unit 104 may determine the degree of change in camera posture based on information indicating the change in camera posture for the current frame. Alternatively, the correction amount calculation unit 104 may determine the degree of change in camera posture by comparing information indicating the change in camera posture for the current frame with information indicating the change in camera posture for the previous frame. For example, the correction amount calculation unit 104 may calculate the amount of change in the value indicating the camera posture between the previous frame and the current frame for each of the x, y, and z directions. If the amount of change in any of the x, y, and z directions exceeds a threshold, it can be determined that a change in camera posture has occurred. In this case, the processing of FIG. 5 ends. Otherwise, the processing proceeds to S502. This threshold can be set so that the camera is considered to be stationary if the difference is equal to or less than the threshold.

[0096] In this way, the correction amount calculation unit 104 can determine whether or not the camera's posture has changed based on the history of information indicating the posture or posture change of the camera. The correction amount calculation unit 104 may detect a period during which the camera's posture does not change between consecutive frames, and calculate the feature point correction amount using the deformation amount calculated by the deformation amount calculation unit 105 for the frames during which the camera's posture does not change.

[0097] In another embodiment, the deformation amount calculation unit 105 can determine whether the change in posture of the camera when capturing the correction target image is equal to or less than a predetermined standard. This determination can be made in the same manner as in S501. Then, in response to determining that the change in posture is equal to or less than the predetermined standard, the deformation amount calculation unit 105 can determine a new search range based on information (e.g., the amount of deformation) indicating the positional deviation evaluated for the correction target image. With this configuration, the feature point correction amount can be calculated based on the amount of deformation that is not significantly affected by temporary changes in posture of the camera due to vibrations, etc. Therefore, the correction amount calculation unit 104 can calculate the feature point correction amount based on constant changes in posture of the camera. In this embodiment, it is not essential to correct the feature point correction amount based on information indicating the history of the amount of deformation.

[0098] (Other Examples) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit (e.g., ASIC) that realizes one or more functions.

[0099] The disclosure of this specification includes the following imaging device, imaging system, method, and program. (Item 1) an acquisition means for acquiring information on feature points included in a first image acquired by an imaging device; a search means for searching for feature points included in a second image acquired by the imaging device, the feature points corresponding to the feature points included in the first image, within a search range set in the second image; a determining means for determining a positional deviation between a feature point included in the first image and a feature point included in the second image corresponding to the feature point included in the first image; a determination means for determining a new search range to be set in the second image based on information indicating a history of the positional deviation determined for each of the plurality of second images obtained sequentially; an update means for updating the search range based on the determination of the determination means; An image processing device comprising: (Item 2) Item 1. The image processing device according to item 1, characterized in that the determination means determines the new search range based on information indicating the positional deviation evaluated for each of one or more of the second images selected according to predetermined conditions from information indicating the positional deviation evaluated for each of the plurality of second images. (Item 3) 3. The image processing device according to item 2, wherein the predetermined conditions include a condition regarding a change in the posture of the imaging device when the second image is captured. (Item 4) The image processing device described in any one of items 2 to 3, characterized in that the determination means determines whether a change in posture of the imaging device when capturing the second image is less than a predetermined standard, and determines to select the second image based on the result of the determination. (Item 5) 5. The image processing device according to any one of items 3 to 4, wherein the determination means determines a change in attitude of the imaging device when the second image is captured at a first time based on an attitude of the imaging device when the second image is captured at the first time and an attitude of the imaging device when the second image is captured at a second time prior to the first time. (Item 6) 6. The image processing device according to any one of items 3 to 5, wherein the posture of the imaging device when capturing the second image is determined based on a positional shift between a feature point included in the first image and a feature point included in the second image. (Item 7) 6. The image processing device according to any one of items 3 to 5, characterized in that the change in posture of the imaging device when capturing the second image is determined based on an output from a sensor provided in the imaging device. (Item 8) 8. The image processing device according to item 7, wherein the sensor is a gyro sensor. (Item 9) 9. The image processing device according to any one of items 2 to 8, wherein the predetermined condition includes a condition regarding the number of feature points searched for in the second image that correspond to feature points included in the first image. (Item 10) 10. The image processing device according to any one of items 2 to 9, characterized in that the determination means determines whether the number of feature points searched from the second image that correspond to feature points included in the first image is equal to or greater than a threshold, and selects the second image based on the result of the determination. (Item 11) 11. The image processing device according to any one of items 2 to 10, wherein the predetermined conditions include a condition regarding a difference between the second image and the first image. (Item 12) 12. The image processing device according to any one of items 2 to 11, characterized in that the determination means determines whether a difference between information on feature points included in the first image and information on feature points searched for in the second image and corresponding to feature points included in the first image is equal to or less than a predetermined standard, and selects the second image based on a result of the determination. (Item 13) the determining means calculates, for each of a plurality of feature points included in the first image, a difference between a feature amount for the feature point included in the first image and a feature amount for a feature point searched for in the second image and corresponding to the feature point included in the first image; 13. The image processing device according to any one of items 2 to 12, characterized in that the determination means determines whether the sum of the differences calculated for a plurality of feature points included in the first image is equal to or less than a predetermined threshold, and selects the second image based on the determination that the sum of the differences is equal to or less than the predetermined threshold. (Item 14) 14. The image processing device according to any one of items 2 to 13, characterized in that the determination means determines the new search range based on statistical values ​​of the displacement evaluated for each of the one or more second images. (Item 15) 15. The image processing device according to any one of items 1 to 14, wherein the update means determines whether to update the search range used by the search means to search for feature points based on a difference between the reference search range and the search range determined by the determination means. (Item 16) 16. The image processing device according to any one of items 1 to 15, wherein the search means searches for feature points included in the second image that correspond to feature points included in the first image by evaluating differences between feature amounts of feature points included in the first image and feature amounts of each point within the search range of the second image. (Item 17) 17. The image processing device according to any one of items 1 to 16, further comprising a correction unit that performs a correction process on the first image and / or the second image to align the first image and the second image based on the positional deviation between the feature points included in the first image and the feature points included in the second image determined by the determination unit. (Item 18) an acquisition means for acquiring information on feature points included in a first image acquired by an imaging device; a search means for searching a search range set in the second image for feature points included in the second image obtained by the imaging device that correspond to feature points included in the first image, and determining a positional deviation between the feature points included in the first image and the feature points included in the second image; a determination means for determining whether a change in posture of the imaging device when capturing the second image is equal to or less than a predetermined standard, and determining a new search range to be set for the second image based on information indicating the positional deviation evaluated for the second image in response to determining that the change in posture is equal to or less than the predetermined standard; an update means for updating the search range based on the determination of the determination means; An image processing device comprising: (Item 19) An image processing method performed by an image processing device, acquiring information about feature points included in a first image acquired by an imaging device; searching for feature points included in a second image acquired by the imaging device that correspond to feature points included in the first image within a search range set in the second image; determining a positional deviation between a feature point included in the first image and a feature point included in the second image that corresponds to the feature point included in the first image; determining a new search range to be set in the second image based on information indicating a history of the positional deviation determined for each of the plurality of second images obtained sequentially; updating the search range based on the determination in the determining step; An image processing method comprising: (Item 20) 19. A program for causing a computer to function as the image processing device according to any one of items 1 to 18.

[0100] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0101] 100: Image processing device, 101: Feature selection unit, 102: Data storage unit, 103: Feature search unit, 104: Correction amount calculation unit, 105: Deformation amount calculation unit, 106: Control unit, 107: Deformation unit, 108: Image input unit, 109: Imaging unit

Claims

1. an acquisition means for acquiring information on feature points included in a first image acquired by an imaging device; a search means for searching for feature points included in a second image acquired by the imaging device, the feature points corresponding to the feature points included in the first image, within a search range set in the second image; a determining means for determining a positional deviation between a feature point included in the first image and a feature point included in the second image corresponding to the feature point included in the first image; a determination means for determining a new search range to be set in the second image based on information indicating a history of the positional deviation determined for each of the plurality of second images obtained sequentially; an update means for updating the search range based on the determination of the determination means; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the determining means determines the new search range based on information indicating the positional deviation evaluated for each of one or more of the second images selected in accordance with predetermined conditions from information indicating the positional deviation evaluated for each of the plurality of second images.

3. The image processing device according to claim 2 , wherein the predetermined condition includes a condition related to a change in the posture of the image capturing device when the second image is captured.

4. 3. The image processing device according to claim 2, wherein the determining means determines whether a change in posture of the imaging device when capturing the second image is equal to or less than a predetermined standard, and determines to select the second image based on the result of the determination.

5. 4. The image processing device according to claim 3, wherein the determination means determines a change in attitude of the imaging device when the second image is captured at a first time based on the attitude of the imaging device when the second image is captured at the first time and the attitude of the imaging device when the second image is captured at a second time prior to the first time.

6. 6. The image processing device according to claim 5, wherein the orientation of the imaging device when capturing the second image is determined based on a positional deviation between a feature point included in the first image and a feature point included in the second image.

7. The image processing device according to claim 3 , wherein a change in the posture of the imaging device when the second image is captured is determined based on an output from a sensor provided in the imaging device.

8. 8. The image processing device according to claim 7, wherein the sensor is a gyro sensor.

9. The image processing device according to claim 2 , wherein the predetermined condition includes a condition regarding the number of feature points found in the second image that correspond to feature points contained in the first image.

10. 3. The image processing device according to claim 2, wherein the determining means determines whether a number of feature points searched from the second image that correspond to feature points contained in the first image is equal to or greater than a threshold, and selects the second image based on a result of the determination.

11. The image processing device according to claim 2 , wherein the predetermined condition includes a condition regarding a difference between the second image and the first image.

12. 3. The image processing device according to claim 2, wherein the determining means determines whether a difference between information on feature points included in the first image and information on feature points searched for in the second image and corresponding to the feature points included in the first image is equal to or less than a predetermined standard, and selects the second image based on a result of the determination.

13. the determining means calculates, for each of a plurality of feature points included in the first image, a difference between a feature amount for the feature point included in the first image and a feature amount for a feature point searched for in the second image and corresponding to the feature point included in the first image; 3. The image processing device according to claim 2, wherein the determining means determines whether a sum of the differences calculated for a plurality of feature points included in the first image is equal to or less than a predetermined threshold, and selects the second image based on the determination that the sum of the differences is equal to or less than the predetermined threshold.

14. 3. The image processing apparatus according to claim 2, wherein the determining means determines the new search range based on statistics of the displacement evaluated for each of the one or more second images.

15. 2. The image processing device according to claim 1, wherein the updating means determines whether to update the search range used by the searching means to search for feature points based on a difference between the reference search range and the search range determined by the determining means.

16. 2. The image processing device according to claim 1, wherein the search means searches for feature points included in the second image that correspond to the feature points included in the first image by evaluating differences between feature amounts of the feature points included in the first image and feature amounts of each point within the search range of the second image.

17. 2. The image processing device according to claim 1, further comprising a correction unit that performs a correction process on the first image and / or the second image to align the first image and the second image based on the positional deviation between the feature points included in the first image and the feature points included in the second image determined by the determination unit.

18. an acquisition means for acquiring information on feature points included in a first image acquired by an imaging device; a search means for searching a search range set in the second image for feature points included in the second image obtained by the imaging device that correspond to feature points included in the first image, and determining a positional deviation between the feature points included in the first image and the feature points included in the second image; a determination means for determining whether a change in posture of the imaging device when capturing the second image is equal to or less than a predetermined standard, and determining a new search range to be set for the second image based on information indicating the positional deviation evaluated for the second image in response to determining that the change in posture is equal to or less than the predetermined standard; an update means for updating the search range based on the determination of the determination means; An image processing device comprising:

19. An image processing method performed by an image processing device, acquiring information about feature points included in a first image acquired by an imaging device; searching for feature points included in a second image acquired by the imaging device that correspond to feature points included in the first image within a search range set in the second image; determining a positional deviation between a feature point included in the first image and a feature point included in the second image that corresponds to the feature point included in the first image; determining a new search range to be set in the second image based on information indicating a history of the positional deviation determined for each of the plurality of second images obtained sequentially; updating the search range based on the determination in the determining step; An image processing method comprising:

20. A program for causing a computer to function as the image processing device according to any one of claims 1 to 18.

Images