Movement measurement system

Abstract

To reduce a processing amount of movement amount calculation.SOLUTION: A movement amount measurement system sets a tracking object region A(t=i) in NH rows × NV columns in a photographed image P(t=i) at each time point of t=i, searches a region in a photographed image P(t=i+1) matching the tracking object region A(t=i) in an image P(t=i+1) at each time point of t=i+1, and calculates a movement amount from a movement vector to the searched region from the tracking object region A(t=i). In searching of the region, the system obtains the sum of luminance values of NV pixels in the row for each of the NH rows of the tracking object region A(t=i), and sets a searching object row array HA(t=i) in NH rows × 1 column in which the obtained sum is arranged in the row arrangement direction. Also, the system generates an image in which a value of each pixel of the image P(t=i+1) is replaced with the sum of the luminance values of the pixels in the region in 1 row × NV columns including the pixel as a pixel in the prescribed numbered column, and performs one-dimensional template matching of the searching object row array HA(t=i) to the generated image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a technique for measuring the amount of relative movement between a camera and an object by using an image of the object captured by a camera. [Background technology]

[0002] A known technique for measuring the amount of relative movement between a camera and an object using an image of the object captured by a camera is to use template matching to search for an area in an image captured at a time t2 after time t1 that is similar to a characteristic area in an image captured at a first time t1, and convert the amount of movement from the characteristic area of ​​the searched area into the amount of movement in real space to calculate the amount of relative movement between the camera and the object (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-190458 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to the technology for measuring the relative amount of movement between the camera and the target object using the above-mentioned template matching, if the feature region is considered to be a one-dimensional region, it cannot be said with certainty that the region searched by template matching is a region that contains the same part of the target object that is captured in the feature region, and therefore measurements cannot be made with high reliability.

[0005] On the other hand, if the feature region is a two-dimensional region, two-dimensional template matching, which requires a large amount of processing, is required, resulting in an increase in the processing load and a decrease in processing speed. Therefore, an object of the present invention is to reliably measure the amount of relative movement between a camera and an object using an image of the object captured by a camera, without excessively increasing the processing load. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention provides a movement amount measuring system that repeatedly measures the relative movement amount of a camera with respect to an object from a captured image that is an image of the object captured by a camera, the system comprising: a tracked area setting means that sets a tracked area of ​​H rows by V columns (where H>2, V>2) in the current captured image in each measurement; and a searched row array that sets a searched row array of H rows by 1 column in which, for each of H rows of the tracked area set by the tracked area setting means, the sum of luminance values ​​of V pixels in the row is calculated and values ​​determined by the calculated sum are arranged in the row arrangement direction. The method includes a column setting means, a search means for searching, in each measurement, for a target searched row array, which is a searched row array set by the searched row array setting means for the tracked area set by the tracked area setting means last time, and an area in the current captured image of H rows by V columns in which the sum of the luminance values ​​of V pixels in each row matches, as a tracking result area, and a movement amount calculation means for converting, in each measurement, the amount of movement from the tracked area set by the tracked area setting means last time to the tracking result area searched by the current search means into the amount of movement of the camera relative to the target object.

[0007] This movement amount measurement system may generate a row array search image in which the value of each pixel in the current captured image is replaced with a value determined by the sum of the brightness values ​​of pixels in a 1 row by V column area that includes the pixel in question as the pixel of a specified column during each measurement using the search means, and search for the tracking result area by matching the target searched row array to the generated row array search image.

[0008] In order to achieve the above object, the present invention provides a movement amount measuring system that repeatedly measures the relative movement amount of a camera with respect to an object from a captured image that is an image of the object captured by a camera, the system comprising: a tracked area setting means that sets a tracked area of ​​H rows by V columns (where H>2, V>2) in the captured image at each measurement; and a searched column array that sets a 1 row by V column searched column array in which, for the tracked area set by the tracked area setting means, a sum of luminance values ​​of H pixels in each of V columns of the tracking area is calculated and values ​​determined by the calculated sum are arranged in the direction of arrangement of the columns. The device is equipped with an array setting means, a search means which, in each measurement, searches for a target searched column array, which is a searched column array set by the searched column array setting means for the tracked area set by the tracked area setting means last time, and an area in the current captured image of H rows by V columns in which the sum of the luminance values ​​of H pixels in each column matches, as a tracking result area, and a movement amount calculation means which, in each measurement, converts the movement amount from the tracked area set by the tracked area setting means last time to the tracking result area searched by the search means this time into a relative movement amount of the camera with respect to the target object.

[0009] This movement amount measurement system may generate a column array search image in which the value of each pixel in the current captured image is replaced with a value determined by the sum of the luminance values ​​of pixels in an H row x 1 column area including the pixel in question as a pixel in a predetermined row, during each measurement by the search means, and search for the tracking result area by matching the target searched column array to the generated column array search image.

[0010] Further, in order to achieve the above object, the present invention provides a movement amount measuring system which repeatedly measures the amount of movement of a camera relative to an object from a captured image which is an image of the object captured by a camera, the system comprising: a tracked area setting means which sets a tracked area of ​​H rows by V columns in the current captured image in each measurement; and for the tracked area set by the tracked area setting means, a searched row array of H rows by 1 column is set in which, for each of H rows of the tracking area, a sum of luminance values ​​of V pixels in the row is calculated and values ​​determined by the calculated sums are arranged in the row direction, and a searched row array of 1 row by V columns is set in which, for each of V columns of the tracking area, a sum of luminance values ​​of H pixels in the column is calculated and values ​​determined by the calculated sums are arranged in the column direction. The apparatus includes a searched array setting means for setting a searched column array, a searching means for searching, in each measurement, an H row by V column area in the current captured image where a sum of luminance values ​​of V pixels in each row and a sum of luminance values ​​of H pixels in each column match a target searched row array and a target searched column array, which are the searched row array and the searched column array set by the searched array setting means for the tracked area set by the tracked area setting means last time, as a tracking result area, and a movement amount calculation means for converting, in each measurement, an amount of movement from the tracked area set by the tracked area setting means last time to the tracking result area searched by the search means this time into an amount of movement of the camera relative to the target object.

[0011] This movement amount measurement system may use the search means to generate, for each measurement, a row array search image in which the value of each pixel of the current captured image is replaced with a value determined by the sum of the brightness values ​​of pixels in a 1 row by V column area which includes the pixel in question as a pixel in a predetermined column, and a column array search image in which the value of each pixel of the current captured image is replaced with a value determined by the sum of the brightness values ​​of pixels in a H row by 1 column area which includes the pixel in question as a pixel in a predetermined row, and search for the tracking result area by matching the target searched row array to the generated row array search image and matching the target searched column array to the generated column array search image.

[0012] The above-mentioned movement amount measuring system may be provided with a rotation detection means for detecting relative rotation of the camera with respect to the target object, with a rotation axis parallel to the optical axis of the camera, and when the rotation detection means detects the rotation, the searching means may search for the tracking result area by using, instead of the captured image, an image output by the camera rotated so that the rotation detected by the rotation detection means is cancelled out.

[0013] Alternatively, the movement amount measurement system may be provided with a rotation detection means for detecting relative rotation of the camera with respect to the target object, with a direction parallel to the optical axis of the camera as a rotation axis, and in the search means, when the rotation detection means detects the rotation, instead of the captured image, an image in which each column of the image output by the camera is sequentially shifted in the row arrangement direction so that the shift of the image in the row arrangement direction due to the rotation detected by the rotation detection means is offset is used as the captured image, and a search may be performed for an H row x V column area in which the sum of the brightness values ​​of V pixels in each row matches the target searched row array.

[0014] Furthermore, the movement amount measurement system may include a rotation detection means for detecting a relative rotation of the camera with respect to the target object, the rotation axis being a direction parallel to the optical axis of the camera, and in the searching means, when the rotation detection means detects the rotation, a search may be performed for an H row by V column area in which the sum of the luminance values ​​of H pixels in each column matches the target searched column array when the rotation detection means detects the rotation, using, instead of the captured image, an image in which each row of the image output by the camera is sequentially shifted in the column arrangement direction so that the shift of the image in the column arrangement direction due to the rotation detected by the rotation detection means is offset.

[0015] The movement amount measuring system may also be configured such that the camera is mounted on a moving object and captures an image of a surface on which the moving object travels as the target object. According to such a movement measurement system, the tracked area is tracked by matching with the searched row array and searched column array, which are one-dimensional arrays, so that the tracked area can be tracked and the movement amount can be measured with less processing power than when performing two-dimensional template matching.

[0016] In addition, since the searched row array and searched column array have two-dimensional array information of the brightness values ​​of the pixels in the tracked area convolved therein, it is possible to track the tracked area and measure the amount of movement with a higher reliability than when simply performing template matching on the tracked area as a one-dimensional area. Effect of the Invention

[0017] According to the present invention, the amount of relative movement between a camera and an object can be measured with high reliability using an image of the object captured by a camera, without excessively increasing the processing load. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing a configuration of a movement amount measuring system according to a first embodiment of the present invention. [Diagram 2] 1 is a diagram showing an application example of a movement measuring system according to a first embodiment of the present invention; [Diagram 3] 5A to 5C are diagrams illustrating an example of a tracking operation of the measurement apparatus according to the first embodiment of the present invention. [Figure 4] FIG. 11 is a diagram illustrating an example of a calculation operation of an actual distance conversion coefficient of the measurement apparatus according to the first embodiment of the present invention. [Diagram 5] FIG. 4 is a diagram illustrating an example of an operation of template matching according to the first embodiment of the present invention. [Figure 6] FIG. 4 is a diagram illustrating an example of an operation of template matching according to the first embodiment of the present invention. [Figure 7] 11A to 11C are diagrams illustrating an example of a calculation operation of a turning amount of the measurement instrument according to the second embodiment of the present invention. [Figure 8] 13A to 13C are diagrams illustrating an example of the template matching operation of the measurement apparatus according to the second embodiment of the present invention. [Figure 9] 13A to 13C are diagrams illustrating another example of the operation of template matching in the measurement apparatus according to the second embodiment of the present invention. [Figure 10] 13A to 13C are diagrams illustrating another example of the operation of template matching in the measurement apparatus according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] A first embodiment of the present invention will be described. FIG. 1 shows the configuration of a movement amount measuring system according to the first embodiment. As shown in the figure, the movement measurement system includes a measurement device 1 and a stereo camera 2. The measurement device 1 includes a tracked region setting unit 11 , a tracking processing unit 12 , a separation distance measuring unit 13 , a movement vector calculation unit 14 , an actual distance conversion coefficient calculation unit 15 , and a movement state calculation unit 16 . Moreover, the stereo camera 2 includes two cameras, a first camera 21 and a second camera 22. For simplicity, the stereo camera 2 is assumed to be a parallel stereo camera in which the optical axes of the first camera 21 and the second camera 22 are parallel.

[0020] The movement measurement system is a system that measures the relative movement between the object photographed by the stereo camera 2 and the stereo camera 2, and can be applied to measuring the movement as shown in Figures 2a and 2b, for example.

[0021] 2a shows an application example in which the stereo camera 2 is fixed to a moving body (an automobile in the figure) and the amount of movement of the moving body relative to the road surface is measured with the road surface as the target object. In this case, the stereo camera 2 is arranged so that the first camera 21 and the second camera 22 capture the image of the road surface from directly above. The first camera 21 and the second camera 22 are arranged at a distance from each other in a direction perpendicular to the main movement direction MD (forward direction of the moving body) of the stereo camera 2 relative to the target object (road surface) and perpendicular to the optical axes of the first camera 21 and the second camera 22. The first camera 21 and the second camera 22 are arranged so that the upward direction of the captured image is the movement direction MD.

[0022] 2b shows an application example in which the stereo camera 2 is fixed to a reference coordinate system, and the amount of movement of an object surface (in the figure, the surface on which the conveyor belt carries the transported object) moving relative to the reference coordinate system is measured. In this case, the stereo camera 2 is arranged so that the first camera 21 and the second camera 22 capture the object surface from directly above. The first camera 21 and the second camera 22 are arranged at a distance from each other in a direction perpendicular to the main movement direction MD (the movement direction of the object surface relative to the reference coordinate system) of the object (object surface) relative to the stereo camera 2 and perpendicular to the optical axis of the first camera 21 and the second camera 22. The first camera 21 and the second camera 22 are arranged so that the upward direction of the captured image is the movement direction MD.

[0023] Therefore, the first camera 21 and the second camera 22 are arranged side by side in a direction that corresponds to the column arrangement direction (left-right direction of the image) in the captured image, the row arrangement direction (up-down direction of the image) in the captured image is parallel to the movement direction MD, and the column arrangement direction in the captured image is perpendicular to the movement direction MD on the target object.

[0024] Next, the tracking operation performed by the tracked area setting unit 11 and the tracking processing unit 12 of the measurement device 1 will be described. After the measurement operation of the measurement device 1 is started, the measurement operation is repeatedly executed intermittently. Here, as shown in FIG. 3a, the start time of the measurement operation is represented as t=1, and the execution time of the n-th measurement operation from the start time of the process is represented as t=n. Moreover, the image captured by first camera 21 at time t=n is represented as P(t=n). The tracked region setting unit 11 sets a tracked region A(t=i) in an image P(t=i) at each time point t=i (i is an arbitrary integer equal to or greater than 1). At each point in time t=i+1 (i is any integer equal to or greater than 1), the tracking processing unit 12 searches for an area in image P(t=i+1) that contains an image most similar to the image of tracked area A(t=i) in image P(t=i), as the tracking result area B(t=i+1), using a template matching process described below.

[0025] The tracked area A(t=i) set by the tracked area setting unit 11 has a fixed size of a plurality of pixels by a plurality of pixels (3 pixels by 3 pixels in the illustrated example). Furthermore, for the initial tracked area A(t=1) at t=1, the tracked area setting unit 11 sets it so that the reference point Ag(t=1), which is the pixel position at the bottom left of the tracked area A(t=1), is included in the reference point allowable area Q, which is a fixed area in the image captured by the first camera 21, as shown in FIG. 3b. The position and size of the reference point allowable area Q may be updated based on the relative state quantity between the target object and the stereo camera 2. Furthermore, the tracked area A(t=1) is set so that the tracked area A(t=1) does not become an area that includes only featureless images as much as possible.

[0026] Next, as shown in Figures 3c1 and c2, at each point in time t=i+1, the tracking processing unit 12 searches for an area in image P(t=i+1) that contains an image most similar to the image of tracked area A(t=i) in image P(t=i) as a tracking result area B(t=i+1), and calculates a movement vector VP(t=i+1) in image space that has its starting point as reference point Ag(t=i) of tracked area A(t=i) and its ending point as reference point Bg(t=i+1), which is the bottom left pixel position of the searched tracking result area B(t=i+1).

[0027] For the second or subsequent tracked area A(t=i+1), if the reference point Bg(t=i+1) of the tracking result area B(t=i+1) searched at t=i+1 is within the reference point allowable area Q as shown in FIG. 3d1, the tracked area setting unit 11 sets the tracking result area B(t=i+1) as it is as the tracked area A(t=i+1). On the other hand, if the reference point Bg(t=i+1) of the tracking result area B(t=i+1) is not within the reference point allowable area Q as shown in FIG. 3d2, the tracked area A(t=i+1) is set so that the reference point Ag(t=i+1) is included within the reference point allowable area Q, similar to the setting of the first tracked area A(t=1) at t=1. However, even if the reference point Bg(t=i+1) is within the reference point allowable region Q, the tracked region A(t=i+1) may be set again so that the reference point Ag(t=i+1) is included within the reference point allowable region Q, as in the case of the initial setting of the tracked region A(t=1) at t=1.

[0028] Next, the operation of calculating the actual distance conversion coefficient performed by the separation distance measuring unit 13 and the actual distance conversion coefficient calculating unit 15 of the measuring device 1 will be described. At time t=1, the separation distance measuring unit 13 calculates the distance ZA(t=1) in the optical axis direction from the first camera 21 and the second camera 22 constituting the stereo camera 2 to the position on the target object reflected in the center of the tracked area A(t=1). Also, at each time t=i+1, the separation distance measuring unit 13 calculates the distance ZA(t=i+1) in the optical axis direction from the first camera 21 and the second camera 22 to the position on the target object reflected in the center of the tracked area A(t=i+1) and the distance ZB(t=i+1) in the optical axis direction from the first camera 21 and the second camera 22 to the position on the target object reflected in the center of the tracking result area B(t=i+1).

[0029] The distance Z in the optical axis direction from the first camera 21 and the second camera 22 to a position on the target object can be calculated as follows. 4, F [mm] is the focal length between the first camera 21 and the second camera 22 of the stereo camera 2, and M [mm] is the base length, which is the distance in real space between the first camera 21 and the second camera 22. Also, Tg is the position on the object where the distance Z [mm] is measured, c1 is the left-right position in the image space where the position Tg is photographed by the first camera 21, and c2 is the left-right position in the image space where the position Tg is photographed by the second camera 22. Also, c11 is the position corresponding to c1 on the surface SF that is a focal length F away from the first camera 21 in the optical axis direction, and c21 is the position corresponding to c2 on the surface SF that is a focal length F away from the second camera 22 in the optical axis direction.

[0030] Further, xl [mm] is the distance in the left-right direction from the center position of the first camera 21 to c11, and xr [mm] is the distance in the left-right direction from the center position of the second camera 22 to c22. And if p=xl-xr, according to the principle of triangulation, the distance Z of the position Tg is, Z = (M × F) / p It is calculated by:

[0031] If the size of the area on a surface projected onto one pixel of the first camera 21 at a focal distance F from the first camera 21 is S [mm / pixel], then when the coordinate in the image space of the first camera 21 changes by 1, the position of Tg changes by (Z / F) × S in the direction perpendicular to the distance Z.

[0032] Therefore, using this relationship, the actual distance conversion coefficient calculation unit 15 calculates the actual distance conversion coefficient K(t=i+1) from ZA(t=i) and ZB(t=i+1) measured by the separation distance measurement unit 13 at each point in time t=i+1.

[0033] The actual distance conversion coefficient K(t=i+1) is calculated by taking the approximate Z(t=i+1) as the average value of Z(t=i) and ZB(t=i+1), {Z(t=i)+ZB(t=i+1)} / 2, K(t=i+1)={Z(t=i+1) / F}×S It is calculated by:

[0034] Next, the movement vector calculation unit 14 of the measurement device 1 calculates, at each time point t=i+1, a relative movement vector V(t=i+1) of the target object with respect to the stereo camera 2, using the movement vector VP(t=i+1) in the image space and the real distance conversion coefficient K(t=i+1) calculated by the real distance conversion coefficient calculation unit 15, as follows: Calculate V(t=i+1)=K(t=i+1)×VP(t=i+1).

[0035] The movement state calculation unit 16 of the measurement device 1 calculates and outputs various movement states such as the relative movement speed, acceleration, and movement direction of the target object from the relative movement vector V(t=i+1) of the target object with respect to the stereo camera 2 calculated by the movement vector calculation unit 14 at each time point t=i+1. The movement state calculation unit 16 may further calculate and output the relative angle of the target object, etc., as a movement state from multiple distances in the image calculated by the separation distance measurement unit 13.

[0036] Next, we will explain the template matching process performed by the tracking processing unit 12 to search for the area of ​​image P(t=i+1) that contains the image most similar to the image of tracked area A(t=i) in image P(t=i) as the tracking result area B(t=i+1).

[0037] In the template matching process, the tracking processing unit 12 calculates the sum of the luminance values ​​of each pixel in each of the NH rows of the tracked area A(t=i) in the image P(t=i) at each time point t=i, as shown in Fig. 5, where the number of rows and the number of columns in the tracked area A(t=i) are NH and NV, respectively. Then, the tracking processing unit 12 sets a one-dimensional array of NH rows x 1 column in which the sums of the luminance values ​​calculated for each row are arranged in the row arrangement direction according to the row arrangement order as the searched row array HA(t=i).

[0038] Also, for each of NV columns of the tracked region A(t=i) in the image P(t=i), the sum of the luminance values ​​of each pixel in the column is calculated. Then, the sums of the luminance values ​​calculated for each column are arranged in the column arrangement direction according to the column arrangement order to create a one-dimensional array of 1 row by NV columns, which is set as the searched column array VA(t=i).

[0039] In the illustrated example, the tracked area A(t=i) is 3 pixels x 3 pixels, so the tracked area A(t=i) is an area of ​​3 rows x 3 columns, and the searched row array HA(t=i) is a one-dimensional array of 3 rows x 1 column in which the sum of the luminance values ​​of the three pixels in the first row of the tracked area A(t=i), the sum of the luminance values ​​of the three pixels in the second row of the tracked area A(t=i), and the sum of the luminance values ​​of the three pixels in the third row of the tracked area A(t=i) are arranged in the row direction in that order. Also, the searched column array VA(t=i) is a one-dimensional array of 1 row x 3 columns in which the sum of the luminance values ​​of the three pixels in the first column of the tracked area A(t=i), the sum of the luminance values ​​of the three pixels in the second column of the tracked area A(t=i), and the sum of the luminance values ​​of the three pixels in the third column of the tracked area A(t=i) are arranged in the column direction in that order. Next, in the template matching process, at each time point t=i+1, the tracking processing unit 12 generates a row array search image PH(t=i+1) and a column array search image PV(t=i+1) from an image P(t=i+1), as shown in FIGS. 6a and 6b.

[0040] The number of rows in the tracked area A(t=i) is NH and the number of columns is NV. The row array search image PH(t=i+1) is obtained by replacing the value of each pixel in the image P(t=i+1) with the sum of the brightness values ​​of the pixels in a 1 row × NV column area that is set to include the pixel in the specified column.

[0041] In the illustrated example, the tracked area A(t=i) is an area of ​​3 rows and 3 columns, and the row array search image PH(t=i+1) is obtained by replacing the value of each pixel of the image P(t=i+1) with the sum of the brightness values ​​of the pixels in a 1 row by 3 column area that is set to include the pixel in question as the pixel in the second column.

[0042] For example, the value of the pixel in the 3rd row and 6th column of the row array search image PH(t=i+1) is the sum of the brightness values ​​of the pixel in the 3rd row and 5th column, the pixel in the 3rd row and 6th column, and the pixel in the 3rd row and 7th column of image P(t=i+1); the value of the pixel in the 5th row and 4th column of the row array search image PH(t=i+1) is the sum of the brightness values ​​of the pixel in the 5th row and 3rd column, the pixel in the 5th row and 4th column, and the pixel in the 5th row and 5th column of image P(t=i+1); and so on. The value of the pixel in the Ath row and Bth column of the row array search image PH(t=i+1) is the sum of the brightness values ​​of the pixel in the Ath row and B-1st column, the pixel in the Ath row and Bth column, and the pixel in the Ath row and B+1th column of image P(t=i+1).

[0043] However, for the pixels in the first and last columns of image P(t=i+1), a 1 row x 3 column area cannot be set to include those pixels as pixels in the second column, so the sum resulting from replacing those pixels is not included in row array search image PH(t=i+1).

[0044] Such a row array search image PH(t=i+1) can be generated, for example, by calculating a moving sum of the brightness values ​​of NV pixels for each row of image P(t=i+1) while moving in the column direction.

[0045] In addition, the column array search image PV(t=i+1) is obtained by replacing the value of each pixel of the image P(t=i+1) with the sum of the brightness values ​​of the pixels in an area of ​​NH rows x 1 column that is set to include the pixel in question as the pixel of a specified row.

[0046] In the illustrated example, the tracked area A(t=i) is an area of ​​3 rows and 3 columns, and the column array search image PV(t=i+1) is obtained by replacing the value of each pixel of the image P(t=i+1) with the sum of the brightness values ​​of the pixels in a 3 row by 1 column area that is set to include the pixel in question as the pixel in the second row.

[0047] For example, the value of the pixel in the third row and sixth column of the column array search image PV(t=i+1) is the sum of the brightness values ​​of the pixel in the second row and sixth column, the pixel in the third row and sixth column, and the pixel in the fourth row and sixth column of image P(t=i+1); the value of the pixel in the fifth row and fourth column of the column array search image PV(t=i+1) is the sum of the brightness values ​​of the pixel in the fourth row and fourth column, the pixel in the fifth row and fourth column, and the pixel in the sixth row and fourth column of image P(t=i+1); and so on. The value of the pixel in the A row and B column of the column array search image PV(t=i+1) is the sum of the brightness values ​​of the pixel in the A-1 row and B column, the pixel in the A row and B column, and the pixel in the A+1 row and B column of image P(t=i+1).

[0048] However, for the pixels in the first and last rows of the image P(t=i+1), a 3 row x 1 column area cannot be set to include those pixels as pixels in the second row, so the sum resulting from replacing those pixels is not included in the column array search image PV(t=i+1).

[0049] Such a column array search image PV(t=i+1) can be generated, for example, by calculating a moving sum of the brightness values ​​of N-th pixels for each column of image P(t=i+1) while moving in the row arrangement direction.

[0050] It should be noted that the row array search image PH(t=i+1) and the column array search image PV(t=i+1) do not necessarily need to be generated from the entire image P(t=i+1). If the range of the searched column array VA(t=i) in image P(t=i+1) can be limited by the previously calculated movement vector V(t=i) or the like, they may be generated only from that range of image P(t=i+1).

[0051] Then, in the template matching process, at each time point t=i+1, the tracking processing unit 12 performs either or both of template matching between the row array search image PH(t=i+1) generated as shown in FIGS. 6a and 6b and the searched row array HA(t=i) and template matching between the column array search image PV(t=i+1) and the searched column array VA(t=i) to search for the tracking result area B(t=i+1).

[0052] Assuming that the number of rows in the tracked area A(t=i) is NH, the number of columns is NV, the searched row array HA(t=i) is NH rows × 1 column, and the searched column array VA(t=i) is 1 row × NV columns, when searching for the tracking result area B(t=i+1) by performing only template matching between the row array search image PH(t=i+1) and the searched row array HA(t=i), an area of ​​NH rows × 1 column that is most similar to the searched row array HA(t=i) in the row array search image PH(t=i+1) is searched for. Then, the area of ​​NH rows × NV columns in the image P(t=i+1) where each row of the searched area represents the sum of brightness values ​​is set as the tracking result area B(t=i+1). Then, when the tracking result region B(t=i+1) is used as the tracked region A(t=i+1) as is, the region of NH rows x 1 column that is most similar to the searched row array HA(t=i) in the searched row array search image PH(t=i+1) is designated as the searched row array HA(t=i+1).

[0053] Furthermore, when searching for the tracking result region B(t=i+1) by performing only template matching between the column array search image PV(t=i+1) and the searched column array VA(t=i), a region of 1 row × NV columns that is most similar to the searched row array VA(t=i) in the column array search image PV(t=i+1) is searched for. Then, a region of NH rows × NV columns in the image P(t=i+1) in which each column of the searched region represents the sum of brightness values ​​is set as the tracking result region B(t=i+1). Then, when the tracking result region B(t=i+1) is used as it is as the tracked region A(t=i+1), a region of NH rows × 1 column that is most similar to the searched row array VA(t=i) in the searched column array search image PV(t=i+1) is set as the searched row array VA(t=i+1).

[0054] The tracked area A(t=i) in template matching of the searched row array HA(t=i) and the tracked area A(t=i) in template matching of the searched column array VA(t=i) may be different areas, and each may adopt a tracked area A(t=i) consisting of a set of unequal numbers of rows NH and columns NV.

[0055] As shown in Figures 5 and 6, when the tracked area A(t=i) is 3 rows x 3 columns, the searched row array HA(t=i) is 3 rows x 1 column, and the searched row array VA(t=i) is 1 row x 3 columns, when searching for the tracking result area B(t=i+1) by performing only template matching between the row array search image PH(t=i+1) and the searched row array HA(t=i), a 3 row x 1 column area that is most similar to the searched row array HA(t=i) in the row array search image PH(t=i+1) is searched. Then, the 3 row x 3 column area of ​​the image P(t=i+1) where the three rows of the searched area represent the sum of the respective brightness values ​​is set as the tracking result area B(t=i+1).

[0056] Furthermore, when searching for the tracking result region B(t=i+1) by performing only template matching between the column array search image PV(t=i+1) and the searched column array VA(t=i), a 1 row x 3 column region that is most similar to the searched row array VA(t=i) in the column array search image PV(t=i+1) is searched for. Then, the 3 row x 3 column region of the image P(t=i+1) where each column of the searched region represents the sum of brightness values ​​is determined as the tracking result region B(t=i+1).

[0057] Note that when searching for tracking result region B(t=i+1) by performing only template matching between the row array search image PH(t=i+1) and the searched row array HA(t=i), there is no need to generate the column array search image PV(t=i+1) and the searched column array VA(t=i); and when searching for tracking result region B(t=i+1) by performing only template matching between the column array search image PV(t=i+1) and the searched column array VA(t=i), there is no need to generate the row array search image PH(t=i+1) and the searched row array HA(t=i).

[0058] When searching for the tracking result region B(t=i+1) by performing both template matching between the row array search image PH(t=i+1) and the searched row array HA(t=i) and template matching between the column array search image PV(t=i+1) and the searched column array VA(t=i), the search is performed as follows.

[0059] First, a template matching is performed between the row array search image PH(t=i+1) and the searched row array HA(t=i) to search for an NH row by 1 column region in the row array search image PH(t=i+1) that is similar to the searched row array HA(t=i) at a predetermined level or more. Then, for each searched region, an NH row by NV column region in the image P(t=i+1) is set as a tracking result region candidate corresponding to the region, and the similarity between the region and the searched row array HA(t=i) is set as the first similarity of the tracking result region candidate corresponding to the region. Here, the similarity between the searched row array HA(t=i) and each region is calculated so that it is larger as the sum of differences between values ​​of rows at the same position in the region is smaller.

[0060] Next, for each 1 row x NV columns region in the column array search image PV(t=i+1) that represents the sum of the luminance values ​​of each tracking result region candidate, template matching is performed between the 1 row x NV columns region and the searched column array VA(t=i) to calculate the similarity with the searched column array VA(t=i), and the 1 row x NV columns region is set as the second similarity of the tracking result region candidate that represents the sum of the luminance values. Here, the similarity between the searched column array VA(t=i) and each region is calculated so that it is larger, for example, as the sum of the differences between the values ​​of columns at the same position in the region is smaller.

[0061] Then, the tracking result area candidate having the highest similarity to the tracked area A(t=i) indicated by the first similarity and the second similarity is set as the tracking result area B(t=i+1). For example, the tracking result area candidate having the largest sum of the first similarity and the second similarity or the tracking result area candidate having the largest product of the first similarity and the second similarity can be set as the tracking result area B(t=i+1) as the tracking result area candidate having the highest similarity to the tracked area A(t=i).

[0062] Then, when the tracking result region B(t=i+1) is used as the tracked region A(t=i+1) as is, the NH row by 1 column region in the row array search image PH(t=i+1) representing the sum of the brightness values ​​of the tracking result region B(t=i+1) is set as the searched row array HA(t=i+1), and the 1 row by NV column region in the column array search image PV(t=i+1) representing the sum of the brightness values ​​of the tracking result region B(t=i+1) is set as the searched row array VA(t=i+1).

[0063] In addition, the order of template matching between the row array search image PH(t=i+1) and the searched row array HA(t=i) and template matching between the column array search image PV(t=i+1) and the searched column array VA(t=i) may be reversed, and a first similarity may be calculated between the tracking result area candidate and template matching between the column array search image PV(t=i+1) and the searched column array VA(t=i), and a second similarity may be calculated between an area in the row array search image PH(t=i+1) corresponding to the tracking result area candidate and the searched row array HA(t=i).

[0064] When performing template matching of the searched row array HA(t=i) and template matching of the searched column array VA(t=i) for different tracked areas A(t=i), vectors VP(t=i+1) are calculated from different tracking result areas B(t=i+1), so the amount of movement as image P(t=i+1) may be the row component of vector VP(t=i+1), which is the template matching result of the searched row array HA(t=i), and the column component of vector VP(t=i+1), which is the template matching result of the searched column array VA(t=i).

[0065] As described above, according to the first embodiment, the tracked region A(t=i) is tracked by matching with the searched row array HA(t=i) and the searched column array VA(t=i), which are one-dimensional arrays, so that the tracked region A(t=i) can be tracked and the amount of movement measured with less processing power than when performing two-dimensional template matching.

[0066] Furthermore, since the searched row array HA(t=i) and the searched column array VA(t=i) are convolved with information on the two-dimensional array of brightness values ​​of pixels in the tracked area A(t=i), it is possible to track the tracked area A(t=i) and measure the amount of movement with a higher reliability than when simply performing template matching on the tracked area A(t=i) as a one-dimensional area.

[0067] Next, a second embodiment of the present invention will be described. In the second embodiment, in the tracking operation of the movement measurement system according to the first embodiment, template matching is performed taking into account the relative rotation between the stereo camera 2 and the target object around the optical axis direction of the stereo camera 2.

[0068] Below, the differences between the second embodiment and the first embodiment will be described. In the second embodiment, the tracked area setting unit 11 sets two tracked areas, tracked area A1(t=i) and tracked area A2(t=i), in an image P(t=i) at each time point t=i (i is any integer equal to or greater than 1), as shown in Fig. 7a. The tracked area A1(t=i) and tracked area A2(t=i) are set at positions spaced apart in the arrangement direction of the columns.

[0069] At each point in time t=i+1 (i is any integer equal to or greater than 1), the tracking processing unit 12 uses a template matching process to search for an area in image P(t=i+1) that contains an image most similar to the image of tracked area A1(t=i) in image P(t=i) as a tracking result area B1(t=i+1), as shown in FIG. 7b, and searches for an area in image P(t=i+1) that contains an image most similar to the image of tracked area A2(t=i) in image P(t=i) as a tracking result area B2(t=i+1).

[0070] In addition, the tracking processing unit 12 calculates a movement vector VP1(t=i+1) in the image space having its starting point at the reference point Ag1(t=1) in the tracked area A1(t=1) and its ending point at the reference point Bg1(t=i+1) in the searched tracking result area B1(t=i+1), and a movement vector VP2(t=i+1) in the image space having its starting point at the reference point Ag2(t=1) in the tracked area A2(t=1) and its ending point at the reference point Bg2(t=i+1) in the searched tracking result area B2(t=i+1).

[0071] Then, the tracking processing unit 12 calculates the relative rotation angle θ of the stereo camera 2 with respect to the target object from the calculated movement vector VP1(t=i+1) and movement vector VP2(t=i+1) as follows: When the movement vector VP1(t=i+1) and the movement vector VP2(t=i+1) are calculated as shown in Figure 7c, the intersection of a line perpendicular to the movement vector VP1(t=i+1) that passes through the midpoint of the movement vector VP1(t=i+1) and a line perpendicular to the movement vector VP2(t=i+1) that passes through the midpoint of the movement vector VP2(t=i+1) is calculated as the turning center point C.

[0072] Then, the distance from the turning center point C to the reference point Bg1(t=i+1) of the tracking result area B1(t=i+1) and the distance from the turning center point C to the reference point Bg2(t=i+1) of the tracking result area B2(t=i+1) are calculated. If the distance to the reference point Bg1(t=i+1) is R, the distance to the reference point Bg2(t=i+1) is r, d=Rr, and the turning angle at time t=i+1 is θ(t=i+1), then |VP1(t=i+1)|=2Rtan{θ(t=i+1) / 2}=2(r+d)tan{θ(t=i+1) / 2} |VP2(t=i+1)|=2r{θ(t=i+1) / 2} and |VP1(t=i+1)|-|VP2(t=i+1)|=2dtan{θ(t=i+1) / 2} Since the above equation holds, the turning angle θ(t=i+1) is calculated from the magnitudes of VP1(t=i+1) and VP2(t=i+1) according to the equation.

[0073] In addition, when the turning center point C can be considered to be far away and in a direction corresponding to the arrangement direction of the rows of the images captured by the first camera 21, the distance dx in the arrangement direction of the rows between the reference point Bg1(t=i+1) and the reference point Bg2(t=i+1) may be used as d in the above equation. Furthermore, when the turning center point C is sufficiently far away and the turning radius is large, and VP1(t=i+1) and VP2(t=i+1) can be considered as arcs of a circle centered on the turning center point C, the distance dx in the arrangement direction of the rows between the reference point Bg1(t=i+1) and the reference point Bg2(t=i+1) may be considered as the difference in the turning radius, and the difference dy in the size of the rows of the images VP1(t=i+1) and VP2(t=i+1) may be considered as the difference in the length of the turned arc, and θ(t=i+1) may be calculated from dy=dx×θ(t=i+1).

[0074] When the stereo camera 2 rotates by a rotation angle θ(t=i+1) around the optical axis direction relative to the target object between times t=i and t=i+1, as shown in FIG. 8a, the image in image P(t=i) taken at time t=i will be tilted by -θ(t=i+1) in the opposite direction to the rotation direction relative to the image in P(t=i) in image P(t=i+1) taken at time t=i+1, as shown in FIG. 8b.

[0075] Therefore, at each time point t=i+1 (t≧3), the tracking processing unit 12 regards the rotation angle θ(t=i) determined at the previous time point t=i as the rotation angle θ(t=i+1) at the current time point, and if the absolute value of the rotation angle θ(t=i) is greater than a predetermined threshold value, as shown in FIG. 8c, generates an image RP(t=i+1) by rotating the captured image P(t=i+1) by the rotation angle θ(t=i), and performs the above-mentioned template matching process by using the image RP(t=i+1) instead of image P(t=i+1) to generate an image PH(t=i+1) for row array search and an image PV(t=i+1) for column array search.

[0076] However, since the generation of the rotated image has a relatively large processing load, instead of generating the row array search image PH(t = i + 1) and the column array search image PV(t = i + 1) from the rotated image RP(t = i + 1), the row array search image PH(t = i + 1) and the column array search image PV(t = i + 1) may be generated as follows.

[0077] That is, as shown in Fig. 9a, an image SPH(t = i + 1) is generated by shifting the pixels of the captured image P(t = i + 1) in column units so that the overall inclination of the upper and lower sides is θ(t = i), and the row array search image PH(t = i + 1) may be generated from the image SPH(t = i + 1). According to such an image SPH(t = i + 1), the deviation of the image in the row arrangement direction due to turning is canceled out, so that better template matching can be performed than when generating the row array search image PH(t = i + 1) from the image P(t = i + 1).

[0078] In this case, if NC is the order of the columns counted from the left in Fig. 9a, the row array search image PH(t = i + 1) can be generated by shifting each column upward by J rows, where J is a number that satisfies J ≦ NCtanθ(t = i) < J + 1 (where J is an integer of 0 or more) when θ(t = i) is positive. When θ(t = i) is negative, each column can be generated by shifting downward by J rows, where J is a number that satisfies -(J + 1) < NCtanθ(t = i) ≦ -J (where J is an integer of 0 or more).

[0079] Also, as shown in Fig. 9b, an image SPV(t = i + 1) is generated by shifting the pixels of the captured image P(t = i + 1) in row units so that the overall inclination of the left and right sides is θ(t = i), and the column array search image PV(t = i + 1) may be generated from the image SPV(t = i + 1). According to such an image SPV(t = i + 1), the deviation of the image in the column arrangement direction due to turning is canceled out, so that better template matching can be performed than when generating the column array search image PV(t = i + 1) from the image P(t = i + 1).

[0080] In this case, assuming that the order of the lines counted from the bottom of FIG. 9b is NL, when θ(t = i) is positive, the column-array search image PV(t = i + 1) can be generated by shifting J columns to the left, where J is an integer greater than or equal to 0 and satisfies J ≦ NLtanθ(t = i) < J + 1. When θ(t = i) is negative, each row can be generated by shifting J columns to the right, where J is an integer greater than or equal to 0 and satisfies -(J + 1) < NLtanθ(t = i) ≦ -J.

[0081] Even in this way, when the resolution of the image P(t = i + 1) is high and the turning angle θ(t = i) is relatively small, good template matching can be performed. Here, instead of using the image RP(t = i + 1) generated as shown in FIG. 8b for generating the row-array search image PH(t = i + 1) and the column-array search image PV(t = i + 1) to perform template matching processing, or using the images SPH(t = i + 1) and SPV(t = i + 1) generated as shown in FIGS. 9a and 9b for generating the row-array search image PH(t = i + 1) and the column-array search image PV(t = i + 1) to perform template matching processing, the template matching processing considering turning can be performed as follows.

[0082] Assuming that the tracked area A1(t=i+1) and the tracked area A2(t=i+1) are represented by the tracked area A#(t=i+1), when the absolute value of the calculated rotation angle θ(t=i+1) at each point in time t=i+1 is greater than a predetermined threshold value, the tracking processing unit 12 generates a rotated partial image RPP by rotating the image portion PP including the set tracked area A#(t=i+1) and its surrounding area in the image P(t=i+1) captured as shown in FIG. 10a by -θ(t=i+1) as shown in FIG. 10b using the calculated rotation angle θ(t=i+1). 10c, an area of ​​the same size as the tracked area A#(t=i+1) in the center of the turning partial image RPP is set as the post-turning tracked area RA#(t=i+1), and a searched row array HA(t=i+1) and a searched column array VA(t=i+1) are generated from the post-turning tracked area RA#(t=i+1) in place of the tracked area A#(t=i+1). Note that at the time t=1, tracking processing unit 12 generates the searched row array HA(t=1) and the searched column array VA(t=1) from the tracked area A(t=1) as described above.

[0083] Next, at each point in time t=i+1, the movement vector calculation unit 14 of the measurement device 1 multiplies the movement vector VP1(t=i+1) or the movement vector VP2(t=i+1) in the image space calculated by the tracking processing unit 12, or the average vector of the movement vector VP1(t=i+1) and the movement vector VP2(t=i+1), by the actual distance conversion coefficient K(t=i+1) calculated by the actual distance conversion coefficient calculation unit 15, to calculate the relative movement vector V(t=i+1) of the target object with respect to the stereo camera 2.

[0084] Furthermore, in addition to the operation of the first embodiment, the movement state calculation unit 16 of the measurement device 1 also calculates and outputs the relative rotation of the target object as one of the movement states from the rotation angle θ calculated by the tracking processing unit 12.

[0085] The embodiment of the present invention has been described above. In the embodiment, the distance ZA(t=i) and the distance ZB(t=i+1) calculated by the separation distance measuring unit 13 using the stereo camera 2 may be calculated using other devices such as a laser range finder. Also, the turning angle θ calculated by the tracking processing unit 12 in the embodiment may be calculated using other devices such as a gyro sensor.

[0086] Moreover, the roles of the first camera 21 and the second camera 22 in the embodiment may be interchanged. In addition, in the embodiment, the sum of brightness values ​​is used as the value of each position of the searched row array HA(t=i), the searched column array VA(t=i), the row array search image PH(t=i+1), and the column array search image PV(t=i+1). However, other values ​​such as the average of brightness values ​​may be used as long as they are values ​​that can represent the sum of brightness values.

[0087] When the average brightness value is used, a low-pass filter can be applied to generate the row array search image PH(t=i+1) and the column array search image PV(t=i+1). [Explanation of symbols]

[0088] 1...measuring device, 2...stereo camera, 11...tracked area setting unit, 12...tracking processing unit, 13...separation distance measuring unit, 14...movement vector calculation unit, 15...actual distance conversion coefficient calculation unit, 16...movement state calculation unit, 21...first camera, 22...second camera.

Claims

1. A movement amount measurement system that repeatedly measures a relative movement amount of a camera with respect to an object from a captured image that is an image of the object captured by a camera, a tracking area setting means for setting a tracking area of ​​H rows by V columns (where H>2, V>2) in the captured image in each measurement; a searched row array setting means for calculating a sum of luminance values ​​of V pixels in each of H rows of the tracking area set by the tracking area setting means, and setting a searched row array of H rows x 1 column in which values ​​determined by the calculated sums are arranged in the row arrangement direction; a searching means for searching, in each measurement, an area in the current captured image of H rows by V columns in which the sum of luminance values ​​of V pixels in each row matches a target searched row array, which is a searched row array set by the searched row array setting means for the tracked area set by the tracked area setting means in the previous measurement, as a tracking result area; a movement amount calculation means for converting, in each measurement, a movement amount from the tracked area previously set by the tracked area setting means to the tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; a rotation detection means for detecting a relative rotation of the camera with respect to the object, the rotation axis being a direction parallel to an optical axis of the camera; a search means for searching the tracking result area by using, instead of the captured image, an image output by the camera that has been rotated so as to cancel out the rotation detected by the rotation detection means, when the rotation detection means detects the rotation, in place of the captured image.

2. A movement amount measurement system that repeatedly measures a relative movement amount of a camera with respect to an object from a captured image that is an image of the object captured by a camera, comprising: a tracking area setting means for setting a tracking area of ​​H rows by V columns (where H>2, V>2) in the captured image in each measurement; a searched column array setting means for calculating a sum of luminance values ​​of H pixels in each of V columns of the tracking area set by the tracking area setting means, and setting a searched column array of 1 row by V columns in which values ​​determined by the calculated sums are arranged in the column arrangement direction; a searching means for searching, in each measurement, an area in the current captured image of H rows by V columns in which the sum of luminance values ​​of H pixels in each column matches a target searched column array, which is a searched column array set by the searched column array setting means for the tracked area set by the tracked area setting means in the previous measurement, as a tracking result area; a movement amount calculation means for converting, in each measurement, a movement amount from the tracked area previously set by the tracked area setting means to the tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; a rotation detection means for detecting a relative rotation of the camera with respect to the object, the rotation axis being a direction parallel to an optical axis of the camera; a search means for searching the tracking result area by using, instead of the captured image, an image output by the camera that has been rotated so as to cancel out the rotation detected by the rotation detection means, when the rotation detection means detects the rotation, in place of the captured image.

3. A movement amount measurement system that repeatedly measures a relative movement amount of a camera with respect to a target object from a captured image that is an image of the target object captured by a camera, a tracking area setting means for setting a tracking area of ​​H rows by V columns in a captured image in each measurement; a searched array setting means for setting a searched row array of H rows by 1 column by calculating the sum of luminance values ​​of V pixels in each of H rows of the tracking area set by the tracked area setting means, and for setting a searched column array of 1 row by V columns by calculating the sum of luminance values ​​of H pixels in each of V columns of the tracking area, and for setting a searched column array of 1 row by V columns by calculating the sum of luminance values ​​of H pixels in each of V columns of the tracking area, and for setting a searched column array of 1 row by V columns by calculating the sum of luminance values ​​of a searching means for searching, in each measurement, as a tracking result area, an area of ​​H rows by V columns in the current captured image where a sum of luminance values ​​of V pixels in each row and a sum of luminance values ​​of H pixels in each column match a target searched row array and a target searched column array, which are the searched row array and the searched column array set by the searched array setting means for the tracked area set by the tracked area setting means in the previous measurement; a movement amount calculation means for converting, in each measurement, a movement amount from the tracked area previously set by the tracked area setting means to the tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; a rotation detection means for detecting a relative rotation of the camera with respect to the object, the rotation axis being a direction parallel to an optical axis of the camera; a search means for searching the tracking result area by using, instead of the captured image, an image output by the camera that has been rotated so as to cancel out the rotation detected by the rotation detection means, when the rotation detection means detects the rotation, in place of the captured image.

4. A movement amount measurement system that repeatedly measures a relative movement amount of a camera with respect to an object from a captured image that is an image of the object captured by a camera, a tracking area setting means for setting a tracking area of ​​H rows by V columns (where H>2, V>2) in the captured image in each measurement; a searched row array setting means for calculating a sum of luminance values ​​of V pixels in each of H rows of the tracking area set by the tracking area setting means, and setting a searched row array of H rows x 1 column in which values ​​determined by the calculated sums are arranged in the row arrangement direction; a searching means for searching, in each measurement, an area in the current captured image of H rows by V columns in which the sum of luminance values ​​of V pixels in each row matches a target searched row array, which is a searched row array set by the searched row array setting means for the tracked area set by the tracked area setting means in the previous measurement, as a tracking result area; a movement amount calculation means for converting, in each measurement, a movement amount from the tracked area previously set by the tracked area setting means to the tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; a rotation detection means for detecting a relative rotation of the camera with respect to the object, the rotation axis being a direction parallel to an optical axis of the camera; a search means for searching an H row by V column region in which a sum of luminance values ​​of V pixels in each row matches the target searched row array when the rotation detection means detects the rotation, using as the captured image, instead of the captured image, an image in which each column of the image output by the camera is sequentially shifted in the row arrangement direction so that the shift of the image in the row arrangement direction due to the rotation detected by the rotation detection means is offset,

5. A movement amount measurement system that repeatedly measures a relative movement amount of a camera with respect to a target object from a captured image that is an image of the target object captured by a camera, a tracking area setting means for setting a tracking area of ​​H rows by V columns in a captured image in each measurement; a searched array setting means for setting a searched row array of H rows by 1 column by calculating the sum of luminance values ​​of V pixels in each of H rows of the tracking area set by the tracked area setting means, and for setting a searched column array of 1 row by V columns by calculating the sum of luminance values ​​of H pixels in each of V columns of the tracking area, and for setting a searched column array of 1 row by V columns by calculating the sum of luminance values ​​of H pixels in each of V columns of the tracking area, and for setting a searched column array of 1 row by V columns by calculating the sum of luminance values ​​of a searching means for searching, in each measurement, as a tracking result area, an area of ​​H rows by V columns in the current captured image where a sum of luminance values ​​of V pixels in each row and a sum of luminance values ​​of H pixels in each column match a target searched row array and a target searched column array, which are the searched row array and the searched column array set by the searched array setting means for the tracked area set by the tracked area setting means in the previous measurement; a movement amount calculation means for converting, in each measurement, a movement amount from the tracked area previously set by the tracked area setting means to the tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; a rotation detection means for detecting a relative rotation of the camera with respect to the object, the rotation axis being a direction parallel to an optical axis of the camera; a search means for searching an H row by V column region in which a sum of luminance values ​​of V pixels in each row matches the target searched row array when the rotation detection means detects the rotation, using as the captured image, instead of the captured image, an image in which each column of the image output by the camera is sequentially shifted in the row arrangement direction so that the shift of the image in the row arrangement direction due to the rotation detected by the rotation detection means is offset,

6. A movement amount measurement system that repeatedly measures a relative movement amount of a camera with respect to a target object from a captured image that is an image of the target object captured by a camera, a tracking area setting means for setting a tracking area of ​​H rows by V columns (where H>2, V>2) in the captured image in each measurement; a searched column array setting means for calculating a sum of luminance values ​​of H pixels in each of V columns of the tracking area set by the tracking area setting means, and setting a searched column array of 1 row by V columns in which values ​​determined by the calculated sums are arranged in the column arrangement direction; a searching means for searching, in each measurement, an area in the current captured image of H rows by V columns in which the sum of luminance values ​​of H pixels in each column matches a target searched column array, which is a searched column array set by the searched column array setting means for the tracked area set by the tracked area setting means in the previous measurement, as a tracking result area; a movement amount calculation means for converting, in each measurement, a movement amount from the tracked area previously set by the tracked area setting means to the tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; a rotation detection means for detecting a relative rotation of the camera with respect to the object, the rotation axis being a direction parallel to an optical axis of the camera; the searching means, when the rotation detection means detects the rotation, uses, instead of the captured image, an image in which each row of the image output by the camera is sequentially shifted in the column arrangement direction so that a shift of the image in the column arrangement direction due to the rotation detected by the rotation detection means is offset, and searches for an area of ​​H rows by V columns in which a sum of luminance values ​​of H pixels in each column matches the target searched column array, the movement amount measurement system characterized in that

7. A movement amount measurement system that repeatedly measures a relative movement amount of a camera with respect to a target object from a captured image that is an image of the target object captured by a camera, a tracking area setting means for setting a tracking area of ​​H rows by V columns in a captured image in each measurement; a searched array setting means for setting a searched row array of H rows by 1 column by calculating the sum of luminance values ​​of V pixels in each of H rows of the tracking area set by the tracked area setting means, and for setting a searched column array of 1 row by V columns by calculating the sum of luminance values ​​of H pixels in each of V columns of the tracking area, and for setting a searched column array of 1 row by V columns by calculating the sum of luminance values ​​of H pixels in each of V columns of the tracking area, and for setting a searched column array of 1 row by V columns by calculating the sum of luminance values ​​of a searching means for searching, in each measurement, as a tracking result area, an area of ​​H rows by V columns in the current captured image where a sum of luminance values ​​of V pixels in each row and a sum of luminance values ​​of H pixels in each column match a target searched row array and a target searched column array, which are the searched row array and the searched column array set by the searched array setting means for the tracked area set by the tracked area setting means in the previous measurement; a movement amount calculation means for converting, in each measurement, a movement amount from the tracked area previously set by the tracked area setting means to the tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; a rotation detection means for detecting a relative rotation of the camera with respect to the object, the rotation axis being a direction parallel to an optical axis of the camera; the searching means, when the rotation detection means detects the rotation, uses, instead of the captured image, an image in which each row of the image output by the camera is sequentially shifted in the column arrangement direction so that a shift of the image in the column arrangement direction due to the rotation detected by the rotation detection means is offset, and searches for an area of ​​H rows by V columns in which a sum of luminance values ​​of H pixels in each column matches the target searched column array, the movement amount measurement system characterized in that

8. A movement amount measurement system according to claim 1, 2, 3, 4, 5, 6 or 7, The movement amount measuring system is characterized in that the camera is mounted on a moving object and captures an image of a surface on which the moving object moves as the target object.

Images