Movement measurement system

Abstract

To quickly search for a region similar to a specific region in a preceding image, out of images captured by a camera which moves relative to a target body.SOLUTION: A first camera 21 is provided for sequentially outputting pixel data of each row of captured images of a target body, in the order in the row. The first camera 21 is arranged in a direction in which a part of the target body represented by a row output earlier by the first camera 21 regarding each of the captured images, may be an end part of the target body in a counter-relative moving direction, the counter-relative moving direction being a direction opposite a standard relative moving direction of the first camera 21 with respect to the target body. Regions of the same size as a tracked region A in an image P are set to a correlation calculation regions CA as soon as image data in the regions are output from the first camera 21. The correlation with the tracked region A is calculated. A region corresponding to the tracked region A is calculated as a tracking result region B, on the basis of the calculated correlation.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for measuring the amount of relative movement between a camera and an object using an image of the object captured by a camera. [Background technology]

[0002] A known technique for measuring the amount of relative movement between a camera and an object using an image of the object captured by a camera is to search for an area in a second image captured at a second time later than the first time that is similar to a specific area in a first image captured at a first time, convert the amount of movement of the searched area from the specific area into the amount of movement in real space, and calculate the amount of relative movement between the camera and the object (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-190458 Summary of the Invention [Problem to be solved by the invention]

[0004] When measuring the relative amount of movement between the camera and the target object using the above-mentioned technology, it is preferable to complete the search for areas similar to a specific area quickly with a small amount of processing, from the perspective of suppressing delays in measuring the amount of movement, reducing the amount of processing, and ensuring the stability of tracking feedback. Therefore, an object of the present invention is to quickly search for an area in an image of an object moving relative to a camera at a first time, the area being similar to a specific area in the image of the object photographed at a second time after the first time, with a small amount of processing effort. [Means for solving the problem]

[0005] To achieve the above object, the present invention provides a movement measurement system that measures the relative movement of a camera with respect to an object from captured images that are images of the object captured with a camera that outputs pixel data for each row in the order of the rows of the image, wherein a direction opposite to a standard relative movement direction of the camera with respect to the object is defined as an anti-relative movement direction, and the camera is positioned such that, for each of the captured images, a portion of the object represented by an earlier row output by the camera is a portion of the object closer to the anti-relative movement direction.The movement measurement system also includes: a tracked area setting means that, for each measurement, sets a tracked area in the current captured image; a search means that, for each measurement, searches, as a tracking result area, for an area in the current captured image that matches the object tracked area that is the tracked area set by the tracked area setting means last time; and an on-image movement amount calculation means that, for each measurement, calculates an on-image movement amount that is the amount of movement from the object tracked area to the tracking result area searched for this time by the search means. The searching means receives pixel data of the captured image in the order of output from the camera, and sets an area in the currently captured image having the same size as the target tracked area as soon as input of pixel data of all pixels included in the area is completed, calculates the correlation between the correlation calculation area and the target tracked area, and calculates the tracking result area based on the correlation calculated for one or more of the correlation calculation areas.

[0006] It is also preferable that this movement amount measurement system is configured such that the tracked area setting means predicts the amount of movement on the image to be calculated next time based on the amount of movement on the image calculated up to this point, and sets the tracked area so that the area to which the tracked area has moved by the predicted amount of movement on the image becomes an area near the row of the captured image that is first output from the camera.

[0007] In this case, the tracked area setting means may set the tracked area so that the area moved by the predicted amount of movement on the image of the tracked area becomes an area near the column of the captured image that is first output from the camera. Furthermore, the movement amount measurement system may be configured such that the search means calculates the correlation as the sum of the differences in pixel data between pixels at corresponding positions in the correlation calculation area and the target tracked area, and that before the target tracked area is set, the differences in pixel data between pixels in the target tracked area other than the pixel last output from the camera in the target tracked area and pixels in the target tracked area at positions corresponding to those pixels are calculated in advance when the pixel data of the pixels in the target tracked area is output from the camera.

[0008] Alternatively, the movement amount measurement system may, in the search means, calculate the correlation between the correlation calculation area and the object tracked area from the correlation between rows at corresponding positions in the correlation calculation area and the object tracked area, and before the correlation calculation area is set, calculate in advance the correlation between rows in the correlation calculation area other than the row last output from the camera of the correlation calculation area and rows of the object tracked area at positions corresponding to the rows, when all pixel data in the rows in the correlation calculation area have been output from the camera.

[0009] In addition, the movement measurement system may be provided with a second camera that captures a second captured image, which is an image of the target object, and outputs pixel data for each row in the order in which the rows of the image are arranged, and the second camera may be positioned in an orientation such that, for each second captured image, the part of the target object represented by the row that the second camera outputs earlier is the part of the target object that is closer to the more standard relative movement direction. In this case, the movement amount measurement system is provided with a second tracked area setting means for setting a second tracked area in the current second captured image at each measurement; a second search means for searching, at each measurement, an area in the current second captured image that matches the second target tracked area, which is the second tracked area set by the second tracked area setting means the previous time, as a second tracking result area; a second image movement amount calculation means for calculating, at each measurement, a second image movement amount, which is the movement amount from the second target tracked area to the second tracking result area searched by the second search means at this time; and a movement amount calculation means for calculating, at each measurement, the image movement amount calculated by the image movement amount calculation means or the second image movement amount calculated by the second image movement amount calculation means, selectively using one of the image movement amount calculated by the image movement amount calculation means and the second image movement amount calculated by the second image movement amount calculation means, to calculate the relative movement amount of the camera with respect to the target object. Then, the pixel data of the second captured image is input to the second search means in the order of output from the second camera, and the second search means sets an area in the current second captured image of the same size as the second target tracked area as a second correlation calculation area as soon as the input of pixel data for all pixels contained in that area is completed, calculates the correlation between the second correlation calculation area and the second target tracked area, and calculates the second tracking result area based on the correlation calculated for one or more of the second correlation calculation areas.

[0010] In the movement amount measurement system, the camera may be mounted on a moving body and may capture an image of a surface on which the moving body travels as the target object. According to such a movement amount measuring system, the earlier pixel data is output from an area in a captured image, the more quickly the correlation with the tracked area can be calculated. Furthermore, when the camera is moving in the standard relative movement direction relative to the target object, the camera is positioned in an orientation that captures a moving image in which the image of the target object flows from the row in which pixel data is output later to the row in which pixel data is output earlier, and therefore the tracking result area to be searched is highly likely to be located closer to the row in the image captured by the camera in which pixel data is output first.

[0011] Therefore, it is expected that the tracking result area can be searched for more quickly with a smaller amount of processing. Furthermore, if the tracking area is set so that the area obtained by moving the tracking area by the predicted amount of movement on the image is an area close to the row of the captured image that is first output from the camera, it is possible to further increase the likelihood that the position of the tracking result area to be searched for will be sufficiently close to the row of the image from which pixel data is first output, thereby further ensuring that the search for the tracking result area can be carried out quickly and with little processing effort. [Effects of the Invention]

[0012] As described above, according to the present invention, it is possible to quickly search for an area in an image of an object moving relative to the camera at a first time, the area being similar to a specific area in the image of the object photographed at a second time after the first time, with a small amount of processing effort. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a configuration of a movement amount measuring system according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an application example of a movement amount measuring system according to an embodiment of the present invention. [Figure 3] 1 is a diagram illustrating a relationship between a standard movement direction of a stereo camera relative to a target object and the orientation of an image captured by the stereo camera according to an embodiment of the present invention. [Figure 4] 10A and 10B are diagrams illustrating an example of a tracking operation of the measurement apparatus according to the embodiment of the present invention. [Figure 5] 10A and 10B are diagrams illustrating transition of a correlation calculation region according to an embodiment of the present invention. [Figure 6] 10A and 10B are diagrams illustrating an example of an operation for calculating the correlation between a correlation calculation region and a tracked region according to an embodiment of the present invention. [Figure 7] 10A and 10B are diagrams illustrating another example of the operation of calculating the correlation between the correlation calculation region and the tracked region according to the embodiment of the present invention. [Figure 8] 10A and 10B are diagrams illustrating an example of a calculation operation of an actual distance conversion coefficient according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] An embodiment of the present invention will be described. FIG. 1 shows the configuration of a movement amount measuring system according to an embodiment. As shown in the figure, the movement amount measurement system includes a measurement device 1 and a stereo camera 2. The measurement device 1 includes a tracked area setting unit 11, a tracking processing unit 12, a separation distance measuring unit 13, a movement vector calculating unit 14, an actual distance conversion coefficient calculating unit 15, and a movement state calculating unit 16. The stereo camera 2 includes two cameras, a first camera 21 and a second camera 22. For ease of explanation, the stereo camera 2 is assumed to be a parallel stereo camera in which the optical axes of the first camera 21 and the second camera 22 are parallel. The movement measurement system is a system that measures the relative movement between the object photographed by the stereo camera 2 and the stereo camera 2, and can be applied to measuring the movement as shown in Figures 2a and 2b. 2a shows an application example in which the stereo camera 2 is fixed to a moving body (an automobile in the figure) and the amount of movement of the moving body relative to the road surface is measured with the road surface as the target object. In this case, the stereo camera 2 is positioned so that the first camera 21 and the second camera 22 capture images of the road surface from directly above. The first camera 21 and the second camera 22 are also positioned at a distance in a direction perpendicular to the standard movement direction MD (forward direction of the moving body) of the stereo camera 2 relative to the target object (road surface) and perpendicular to the optical axes of the first camera 21 and the second camera 22.

[0015] 2b shows an application example in which the stereo camera 2 is fixed to a reference coordinate system, and the amount of movement of an object surface (in the figure, the surface on which an object is placed on a belt conveyor) that moves relative to the reference coordinate system is measured. In this case, the stereo camera 2 is positioned so that the first camera 21 and the second camera 22 capture the image of the object surface from directly above. The first camera 21 and the second camera 22 are also positioned at a distance from each other in a direction perpendicular to the standard movement direction MD (the movement direction relative to the object surface in the reference coordinate system) of the stereo camera 2 relative to the object (object surface) and perpendicular to the optical axes of the first camera 21 and the second camera 22.

[0016] The first camera 21 and the second camera 22 output each row of the image in the order of the rows in the image, and output the pixel data within each row in the order of the columns in the image. That is, the first camera 21 and the second camera 22 output each pixel data of the image in a raster scan manner.

[0017] 2a and 2b, the first camera 21 and the second camera 22 are arranged in an orientation such that the portion of the object represented by the row output earlier by the first camera 21 and the second camera 22 in each captured image is the portion of the object closer to the opposite relative movement direction. That is, the row output first by the first camera 21 and the second camera 22 is the row in the captured image that represents the portion of the object closest to the opposite relative movement direction, and the row output last by the first camera 21 and the second camera 22 is the row in the captured image that represents the portion of the object closest to the movement direction MD.

[0018] As an example of such first camera 21 and second camera 22, the following description will be given assuming that first camera 21 and second camera 22 sequentially output pixel data in each row from left to right of an image in which the vertical direction is the row arrangement direction, the horizontal direction is the column arrangement direction, the movement direction MD is the upward direction, and the opposite relative movement direction is the downward direction, as shown in Figure 3, and sequentially output each row from bottom to top, thereby outputting each pixel data of an image captured in a raster scan manner from the bottom left to the top right.

[0019] By positioning the first camera 21 and the second camera 22 in this orientation, when the stereo camera 2 moves in the movement direction MD relative to the target object, the video captured by the first camera 21 and the second camera 22 is a video in which the image of the target object flows from top to bottom. Next, the tracking operation performed by the tracked area setting unit 11 and the tracking processing unit 12 of the measurement device 1 will be described. After the measurement operation of the measurement device 1 is started, the measurement operation is repeatedly executed intermittently. As shown in FIG. 4a, the start time of the measurement operation is represented by t=1, and the time when the nth measurement operation is performed from the start time of the process is represented by t=n. Also, the image captured by the first camera 21 at t=n is represented by P(t=n). 4b, the tracking area setting unit 11 sets a tracking area A(t=i) in the image P(t=i) at time t=i, and sets the pixel position at the bottom left of the tracking area A(t=i) as the reference point Ag(t=i). The setting operation of this tracking area A will be described later. The tracked area A(t=i) set by the tracked area setting unit 11 has a fixed size of a plurality of pixels by a plurality of pixels (3 pixels by 3 pixels in the illustrated example). Next, as shown in Figure 4c, the tracking processing unit 12 searches for an area in image P(t=i+1) that contains an image similar to the image of tracked area A(t=i) in image P(t=i) at time t=i+1, as tracking result area B(t=i+1), and calculates a movement vector VP(t=i+1) in image space that starts from reference point Ag(t=i), which is the pixel position at the bottom left of tracked area A(t=i), and ends at reference point Bg(t=i+1), which is the pixel position at the bottom left of the searched tracking result area B(t=i+1).

[0020] Furthermore, at time t=i+1, the tracked region setting unit 11 sets a tracked region A(t=i+1) in the image P(t=i+1), as shown in FIG. 4d. Next, as shown in Figure 4e, the tracking processing unit 12 searches for an area in image P(t=i+2) that contains an image similar to the image of tracked area A(t=i+1) in image P(t=i+1) at time t=i+2 as tracking result area B(t=i+2), and calculates a movement vector VP(t=i+2) in image space that starts from reference point Ag(t=i+1) in tracked area A(t=i+1) and ends at reference point Bg(t=i+2), which is the pixel position at the bottom left of the searched tracking result area B(t=i+2).

[0021] Similarly, at each subsequent time point, the search for the tracking result area B at that time point, the calculation of the movement vector VP at that time point, and the setting of the tracked area A at that time point are repeated. In addition, when i is not 1, the tracking result area B(t=i) and the movement vector VP(t=i) are calculated using the tracked area A(t=i-1) even at the time t=i shown in Figure 4b. Here, the setting operation of the tracked area A will be described. For the first tracked area A(t=1) at t=1, the tracked area setting unit 11 sets the tracked area A(t=1) so that a predetermined position in the image captured by the first camera 21 becomes the reference point Ag(t=1). Next, the tracked area A(t=j) at the second or subsequent time point t=j will be described using j=i+1, ie, the case of t=i+1 shown in FIG. 4d, as an example. At time t=i+1, the tracked area A(t=i+1) is set so that the predicted position of the tracking result area B(t=i+2) to be searched at the next time t=i+2 is near the bottom edge of the image P(t=i+2). Specifically, first, a motion vector VP(t=i+2) determined at time t=i+2 is predicted, and this vector is set as a predicted motion vector EVP(t=i+2). The predicted motion vector EVP(t=i+2) may be obtained by moving only the starting point of the motion vector VP(t=i+1) calculated at time t=i+1 while keeping the direction and magnitude the same. Alternatively, the acceleration vector of motion vector VP(t=i+1) calculated at time t=i+1 may be calculated from the motion vector VP(t=i+1) calculated at time t=i+1 and the previous motion vector VP(t=i), and the calculated acceleration vector may be applied to motion vector VP(t=i+1) to obtain the predicted motion vector EVP(t=i+2).

[0022] Then, the amount of movement represented by the predicted movement vector EVP(t=i+2) and the position of the reference point Ag after the movement are taken as the predicted position EBg(t=i+2) of the reference point Ag, and a tracked area A(t=i+1) is set so that the predicted position EBg(t=i+2) of the reference point Ag is located within an area Q that is set in advance near the bottom edge of the image of the first camera 21 shown in FIG. 4f1. If multiple such tracked areas A(t=i+1) can be set, the tracked area A(t=i+1) is set so that the predicted position of the reference point Ag is located at the lowest position.

[0023] If there is no area within area Q where the amount of movement represented by the predicted movement vector EVP(t=i+2) and the predicted position EBg(t=i+2) of the reference point Ag after movement are located, the tracked area A(t=i+1) is set so that the amount of movement represented by the predicted movement vector EVP(t=i+2) and the predicted position EBg(t=i+2) of the reference point Ag after movement are located as low as possible.

[0024] Here, at time t=i+1, the tracked area A(t=i+1) may be set so that the predicted position of the tracking result area B(t=i+2) searched for at time t=i+2 is near the bottom and left edge in the image P(t=i+2).In this case, area Q is an area near the bottom and left edge of the image of the first camera 21, as shown in Figure 4f2.

[0025] Next, the search operation of the tracking processing unit 12 for the tracking result region B(t=j) at the second or subsequent time point t=j will be described. The search for the tracking result region B(t=j) is performed by sequentially selecting different regions in the image P(t=j) that are the same size as the tracked region A(t=j-1) as correlation calculation regions CA, and repeating the process of calculating the correlation between the image of the correlation calculation region CA and the image of the tracked region A(t=j-1) until the calculation of the tracking result region B(t=j) is successful. The calculation of the tracking result region B(t=j) is performed according to a predetermined algorithm using the correlations calculated for one or more correlation calculation regions CA.

[0026] The algorithm used to calculate the tracking result region B(t=j) using correlation can be any known algorithm such as SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), NCC (Normalized Cross Correlation), ZNCC (Zero Means Normalized Cross Correlation), or POC (Phase-Only Correlation).

[0027] Any area in the image P(t=j) that is the same size as the tracked area A(t=j-1) is set as a correlation calculation area CA as soon as all of the pixel data within that area is output from the first camera 21, and the correlation is calculated. Therefore, since the first camera 21 is a camera that outputs each pixel data of the image in a raster scan manner from the bottom left to the top right as shown in Figure 3, the area set as the correlation calculation area CA in the image P(t=j) transitions as shown in Figure 5. That is, when the first camera 21 outputs pixel data for the pixel at the upper right corner of the area in the lower left corner of image P(t=j) of the same size as the tracked area A(t=j-1) shown in Figure 5a, the area at the lower left corner of image P(t=j) becomes the initial correlation calculation area CA. Thereafter, each time pixel data of one pixel is output from first camera 21, the area that will become the correlation calculation area CA moves one pixel (one column) to the right, as shown in FIGS. 5b and 5c. Then, as shown in Fig. 5c, when the correlation calculation area CA reaches the right edge of image P(t=j) and pixel data for the number of columns of correlation calculation area CA is thereafter output from first camera 21, the area that becomes correlation calculation area CA becomes the leftmost area in image P(t=j), one pixel above (one row above) the area that became correlation calculation area CA the previous time, as shown in Fig. 5d. Thereafter, each time pixel data for one pixel is output from first camera 21, the area that becomes correlation calculation area CA moves one pixel (one column) to the right, as shown in Figs. 5e and 5f.

[0028] Thereafter, the area that becomes the correlation calculation area CA moves in the same manner, and as a result, as shown in FIG. 5, the area that becomes the correlation calculation area CA moves in a raster scan manner from the lower left to the upper right within the image P(t=j). Therefore, the correlation with the tracked area A(t=j-1) is calculated more quickly for a region further downward in the image P(t=j). Also, if the vertical position is the same, the correlation with the tracked area A(t=j-1) is calculated more quickly for a region further to the left. Furthermore, since the first camera 21 is positioned in an orientation that captures a video in which the image of the target object flows from top to bottom when the first camera 21 is moving in the standard movement direction MD relative to the target object, there is a high probability that the tracking result area B(t=j) to be searched will be located in a lower position in the image captured by the first camera 21.

[0029] Therefore, it is expected that the search for the tracking result area B(t=j) can be performed more quickly with a smaller amount of processing. Furthermore, in the tracked area setting unit 11, the tracked area A(t=j-1) is set so that the predicted position of the tracking result area B(t=j) to be searched at time t=j is near the bottom edge of the image P(t=j). This further increases the probability that the position of the tracking result area B(t=j) to be searched will be sufficiently low in the image captured by the first camera 21, thereby increasing the certainty that the search for the tracking result area B(t=j) can be performed quickly with a small amount of processing.

[0030] Furthermore, if the tracked area setting unit 11 sets the tracked area A(t=j-1) so that the predicted position of the tracking result area B(t=j) to be searched at time t=j is near the bottom and left edge in the image P(t=j), the probability that the position of the tracking result area B(t=j) to be searched will be located to the left of the image captured by the first camera 21 can be further increased, and it is expected that the search for the tracking result area B(t=j) can be performed more quickly and with less processing effort.

[0031] When the SAD method is used as a method for calculating the tracking result area B(t=j) using correlation, the correlation between the image of the correlation calculation area CA in the image P(t=j) and the image of the tracked area A(t=j-1) is calculated, for example, as the sum of the absolute values ​​of the differences between the pixel data of pixels located at the same position in the tracked area A(t=j-1) and the correlation calculation area CA, calculated for each position of the tracked area A(t=j-1), and the correlation calculation area CA where the calculated sum is smaller than a predetermined level is determined to be the tracking result area B(t=j).

[0032] When calculating the correlation as the sum of the differences in pixel data in this manner, the pixel data of the image P(t=j) output from the first camera 21 can be stored, and the absolute value of the difference between the pixel data of each pixel in each correlation calculation area CA and the pixel data of the pixels in the tracked area A(t=j-1) and the sum thereof can be calculated when the correlation calculation area CA is set (when all of the pixel data in the correlation calculation area CA is output from the first camera 21). However, the absolute value of the difference between the pixel data of each pixel other than the pixel in the upper right corner of the correlation calculation area CA and the pixel data of each pixel other than the pixel in the upper right corner of the tracked area A(t=j-1) can also be calculated in advance before the correlation calculation area CA is set. Then, when the correlation calculation area CA is set, the absolute value of the difference between the pixel data of the pixel in the upper right corner of the correlation calculation area CA and the pixel data of the pixel in the upper right corner of the tracked area A (t=j-1) is calculated, and the correlation is calculated as the sum of the absolute value of the calculated difference and the absolute values ​​of the differences that have been calculated in advance for each of the other pixels.

[0033] Such a pre-calculation of the absolute value of the difference can be performed as follows for each output of pixel data from the first camera 21. That is, if each of the areas that can be set as a correlation calculation area CA including pixel px of image P(t=j) shown in Figure 6a as a pixel other than the pixel in the upper right corner is defined as a virtual correlation calculation area VCA for pixel px, the virtual correlation calculation area VCA for pixel px will be the eight virtual correlation calculation areas VCA shown in Figure 6b1 to b8.

[0034] Therefore, when pixel data of pixel px is output from the first camera 21, the absolute value dpx of the difference between the pixel data of pixel px and the pixel data of the tracked area A(t=j-1) whose position in the virtual correlation calculation area VCA is the same as that of pixel px in the tracked area A(t=j-1) is calculated for each of the eight virtual correlation calculation areas VCA.

[0035] Then, when the correlation calculation area CA is set, the absolute value dpx of the difference calculated for the virtual correlation calculation area VCA set in that correlation calculation area CA may be used as the absolute value of the difference between the pixel data of pixel px in the correlation calculation area CA and the pixel data of a pixel in the tracked area A (t=j-1) that is located at the same position in the area as pixel px, to calculate the correlation.

[0036] Note that dpx may represent the difference without being converted to an absolute value, and may be converted to an absolute value when calculating the sum, or conversely, the calculation of the sum of absolute values ​​dpx may proceed to the extent possible. Next, as the correlation between the correlation calculation area CA and the tracked area A(t=j-1), the correlation function between each row of the tracked area A(t=j-1) and the row located at the same position in the correlation calculation area CA is calculated using the POC method or the like, and when calculating the tracking result area B(t=j) taking into account the correlation function of each row calculated for one or more tracked areas A(t=j-1), the correlation function between each row other than the topmost row of each correlation calculation area CA and each row other than the topmost row of the tracked area A(t=j-1) may be calculated in advance before the correlation calculation area CA is set. Then, when the correlation calculation area CA is set, the correlation function between the topmost row of the correlation calculation area CA and the topmost row of the tracked area A(t=j-1) is calculated, and the correlation between the correlation calculation area CA and the tracked area A(t=j-1) may be calculated using the calculated correlation function and the correlation functions previously calculated for each of the other rows.

[0037] Such a correlation function can be calculated in advance for each pixel data output from the first camera 21 as follows. If we define the virtual correlation calculation areas LVCA for each of the areas in the image P(t=j) shown in Figure 7a that can be set as correlation calculation areas CA, including row pl of the same size as the rows in the tracked area A as a row other than the topmost row, then the virtual correlation calculation areas LVCA for row pl will be the two virtual correlation calculation areas LVCA shown in Figures 7b1 and 7b2.

[0038] Therefore, once all pixel data of row pl has been output from the first camera 21, for each of the two virtual correlation calculation areas LVCA, the correlation function cpl between row pl and the row in the tracked area A (t=j-1) whose position in the area is the same as that of row pl in the virtual correlation calculation area LVCA is calculated. Then, when the correlation calculation area CA is set, the correlation function cpl calculated for the virtual correlation calculation area LVCA set in that correlation calculation area CA is used to calculate the correlation, along with the correlation function between the topmost row pl of the correlation calculation area CA and the topmost row of the tracked area A (t=j-1) which is located at the same position in the area as row pl.

[0039] Next, the calculation operation of the actual distance conversion coefficient performed by the separation distance measurement unit 13 and the actual distance conversion coefficient calculation unit 15 of the measurement device 1 will be described. At time t=1, the separation distance measuring unit 13 calculates the distance ZA(t=1) in the optical axis direction from the first camera 21 and the second camera 22 constituting the stereo camera 2 to the position on the target object reflected in the center of the tracked area A(t=1). Also, at each time t=i+1, the separation distance measuring unit 13 calculates the distance ZA(t=i+1) in the optical axis direction from the first camera 21 and the second camera 22 to the position on the target object reflected in the center of the tracked area A(t=i+1), and the distance ZB(t=i+1) in the optical axis direction from the first camera 21 and the second camera 22 to the position on the target object reflected in the center of the tracking result area B(t=i+1).

[0040] The distance Z in the optical axis direction from the first camera 21 and the second camera 22 to the position on the target object can be calculated as follows. 8, F [mm] is the focal length between the first camera 21 and the second camera 22 of the stereo camera 2, and M [mm] is the base length, which is the distance in real space between the first camera 21 and the second camera 22. Furthermore, Tg is the position on the object where distance Z is measured, c1 is the left-right position in the image space where position Tg is photographed by the first camera 21, and c2 is the left-right position in the image space where position Tg is photographed by the second camera 22. Furthermore, c11 is the position corresponding to c1 on the surface SF, which is a focal length F away from the first camera 21 in the optical axis direction, and c21 is the position corresponding to c2 on the surface SF, which is a focal length F away from the second camera 22 in the optical axis direction.

[0041] Also, let xl [mm] be the distance in the left-right direction from the center position of the first camera 21 to c11, and let xr [mm] be the distance in the left-right direction from the center position of the second camera 22 to c21. Then, if p=xl-xr, according to the principle of triangulation, the distance Z [mm] at position Tg is Z=(M×F) / p It is calculated by If the size of the area on the surface projected onto one pixel of the first camera 21 and located at a focal distance F from the first camera 21 is S (mm / pixel), then when the coordinate in the image space of the first camera 21 changes by 1, the position of Tg changes by (Z / F) × S in the direction perpendicular to the distance Z. Therefore, using this relationship, the actual distance conversion coefficient calculation unit 15 calculates the actual distance conversion coefficient K(t=i+1) from ZA(t=i) and ZB(t=i+1) measured by the separation distance measurement unit 13 at each time point t=i+1. The actual distance conversion coefficient K(t=i+1) is calculated by taking the approximate Z(t=i+1) as the average value of Z(t=i) and Z(t=i+1), {Z(t=i)+Z(t=i+1)} / 2, K(t=i+1)={Z(t=i+1) / F}×S It is calculated by:

[0042] Next, at each time point t=i+1, the movement vector calculation unit 14 of the measurement device 1 calculates the relative movement vector V(t=i+1) of the target object with respect to the stereo camera 2 using the movement vector VP(t=i+1) in the image space and the actual distance conversion coefficient K(t=i+1) calculated by the actual distance conversion coefficient calculation unit 15, as follows: Calculate V(t=i+1)=K(t=i+1)×VP(t=i+1).

[0043] The movement state calculation unit 16 of the measurement device 1 calculates and outputs various movement states such as the relative movement speed, acceleration, and movement direction of the target object from the relative movement vector V(t=i+1) of the target object with respect to the stereo camera 2 calculated by the movement vector calculation unit 14 at each time point t=i+1. The movement state calculation unit 16 may also calculate and output the relative angle of the target object as a movement state from multiple distances in the image calculated by the separation distance measurement unit 13.

[0044] The embodiments of the present invention have been described above. In the above embodiment, the movement vector VP is calculated using only the images captured by the first camera 21, but the movement vector VP may also be calculated by performing a similar tracking process using images captured by the second camera 22. In this case, the second camera 22 is positioned in an orientation such that the portion of the object represented by the row output earlier is the portion of the object closer to the movement direction MD, and when the stereo camera 2 is moving in the anti-relative movement direction opposite the movement direction MD with respect to the object, the video captured by the second camera 22 is a video in which the image of the object flows from top to bottom.

[0045] In this case, the movement vector calculation unit 14 of the measurement device 1 calculates the relative movement vector V of the target object with respect to the stereo camera 2 from the movement vector VP calculated earlier at each measurement point, either the movement vector VP calculated from the image captured by the first camera 21 or the movement vector VP calculated from the image captured by the second camera 22.

[0046] In addition, the actual distance conversion coefficient calculation unit 15 performs coordinate conversion of the image captured by the second camera 22 into an image that would be captured if the second camera 22 were facing in the same direction as the first camera, and then calculates the actual distance conversion coefficient K using the above-mentioned processing. By doing so, it is expected that the calculation of the movement vector V can be performed more quickly with a smaller amount of processing, even when the stereo camera 2 is moving relatively to the target object in the direction opposite to the movement direction MD. In the embodiment, the distance ZA(t=i) and the distance ZB(t=i+1) calculated by the separation distance measuring unit 13 using the stereo camera 2 may be calculated using another device such as a laser rangefinder. Furthermore, the pixel data output by the first camera 21 and the second camera 22 may be input to the measuring device 1 via a buffer memory in a FIFO (First In First Out) format. Furthermore, the roles of the first camera 21 and the second camera 22 in the embodiment may be interchanged. [Explanation of symbols]

[0047] 1...measuring device, 2...stereo camera, 11...tracked area setting unit, 12...tracking processing unit, 13...separation distance measurement unit, 14...movement vector calculation unit, 15...actual distance conversion coefficient calculation unit, 16...movement state calculation unit, 21...first camera, 22...second camera.

Claims

1. A movement amount measurement system for measuring a relative movement amount of a camera with respect to an object from a captured image, which is an image of the object captured by a camera that outputs pixel data of each row in order of the rows of the image, a direction opposite to a standard relative movement direction of the camera with respect to the object is defined as an anti-relative movement direction, and the camera is disposed in an orientation such that, for each of the captured images, a portion of the object represented by a row output earlier by the camera is a portion of the object closer to the anti-relative movement direction, The movement amount measurement system includes: a tracking area setting means for setting a tracking area in the captured image at each measurement; a search means for searching, in each measurement, an area in the captured image at this time that matches the target tracked area, which is the tracked area set by the tracked area setting means at the previous time, as a tracking result area; and an on-image movement amount calculation means for calculating an on-image movement amount, which is a movement amount from the target tracked area to the tracking result area searched by the search means at each measurement, The search means receives pixel data of the captured image in the order of output from the camera, and the search means sets an area in the currently captured image of the same size as the target tracked area as soon as input of pixel data for all pixels contained in that area is completed, calculates the correlation between that correlation calculation area and the target tracked area, and calculates the tracking result area based on the correlation calculated for one or more of the correlation calculation areas.

2. 2. The movement amount measurement system according to claim 1, The movement measurement system is characterized in that the tracked area setting means predicts the amount of movement on the image to be calculated next time based on the amount of movement on the image calculated up to this point, and sets the tracked area so that the area moved by the predicted amount of movement on the image of the tracked area becomes an area near the first row output from the camera in the captured image.

3. 3. The movement amount measurement system according to claim 2, The movement measurement system is characterized in that the tracking area setting means sets the tracking area so that the area obtained by moving the tracking area by the predicted amount of movement on the image becomes an area near the column of the captured image that is first output from the camera.

4. 4. The movement amount measurement system according to claim 1, 2 or 3, The search means calculates the correlation as the sum of absolute values ​​of the differences in pixel data between pixels located at corresponding positions in the correlation calculation area and the object tracked area, and calculates, before the correlation calculation area is set, the differences in pixel data between pixels in the correlation calculation area other than the pixel last output from the camera in the correlation calculation area and pixels in the object tracked area located at corresponding positions to the pixels in the correlation calculation area, or the absolute values ​​of the differences, in advance when the pixel data of the pixels in the correlation calculation area is output from the camera.

5. 4. The movement amount measurement system according to claim 1, 2 or 3, The search means calculates the correlation between the correlation calculation area and the object tracked area from the correlation between rows at corresponding positions in the correlation calculation area and the object tracked area, and before setting the correlation calculation area, calculates in advance the correlation between rows in the correlation calculation area other than the row last output from the camera of the correlation calculation area and rows of the object tracked area at positions corresponding to the rows, when all pixel data in the rows in the correlation calculation area are output from the camera.

6. 4. The movement amount measurement system according to claim 1, 2 or 3, a second camera that captures a second captured image that is an image of the target object and outputs pixel data of each row in the order in which the rows of the image are arranged; the second camera is disposed in an orientation such that, for each second captured image, a portion of the object represented by a row output earlier by the second camera is a portion of the object on a more standard relative movement direction side; The movement amount measurement system includes: a second tracking area setting means for setting a second tracking area in the second captured image at each measurement; a second search means for searching, in each measurement, for an area in the second captured image at this time that matches the second target tracked area, which is the second tracked area set by the second tracked area setting means at the previous measurement, as a second tracking result area; a second on-image movement amount calculation means for calculating a second on-image movement amount, which is a movement amount from the second target tracked area to the second tracking result area searched by the second search means at each measurement; a movement amount calculation means for calculating a relative movement amount of the camera with respect to the object by selectively using either the on-image movement amount calculated by the on-image movement amount calculation means or the second on-image movement amount calculated by the second on-image movement amount calculation means in each measurement; The second search means receives pixel data of the second captured image in the order of output from the second camera, and the second search means sets an area in the current second captured image of the same size as the second target tracked area as soon as the input of pixel data for all pixels included in that area is completed, calculates the correlation between the second correlation calculation area and the second target tracked area, and calculates the second tracking result area based on the correlation calculated for one or more second correlation calculation areas.

7. 4. The movement amount measurement system according to claim 1, 2 or 3, The movement amount measurement system is characterized in that the camera is mounted on a moving body and captures an image of the surface on which the moving body travels as the target object.

Images