Movement measurement system

Abstract

To acquire the latest movement amount by processing the calculation of the movement amount of a target body in a short period of time.SOLUTION: An area A to be tracked is set in an image P, each movement vector within the range of a movement vector between images of the next time predicted from the movement vector between images calculated up to the present is set as a search target movement vector. If an area matching the area to be tracked exists within the area where the area A to be tracked is matched with an area to which each search target movement vector moves in the image at the next time, the search target movement vector corresponding to the existing area is made to be a searched movement vector between images. If such an area does not exist (a2), division search operation is repeated until search of the movement vector between images becomes successful. The position of the area A to be tracked in the image P is fixed in the division search operation, each movement vector within the range is set as the search target movement vector while sequentially switching the range of the movement vector in each division search operation, and the movement vector between images is searched (c1 to c7).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for measuring the amount of relative movement between a camera and an object using an image of the object captured by a camera. [Background technology]

[0002] A known technique for measuring the relative amount of movement between a camera and an object using an image of the object captured by a camera involves searching, by template matching, for an area in an image captured at the next time that is similar to a characteristic area in the captured image, and repeatedly calculating the amount of movement of the searched area from the characteristic area (for example, Patent Document 1).

[0003] There is also known a technique for reducing the processing time required for a search by predicting the position of an image of a characteristic region in an image to be captured at the next time based on the amount of movement found up to the current time, and then performing a search using template matching only within the predicted range, which is the area surrounding the predicted position (for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-190458 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-112210 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the above-described technology that performs a search using template matching only on the prediction range, if a search for a region similar to the feature region within the prediction range fails, the amount of movement this time cannot be calculated, it becomes impossible to set the next prediction range, and subsequent searches become impossible. Therefore, if a search within the predicted range fails, it is possible to perform a search using template matching on the entire image, but a search that targets the entire image requires more processing time than a search that targets only the predicted range. If a search that covers the entire area takes longer than the cycle of repeating a search that covers only the prediction range, the information obtained by the search will be older than the information obtained by a search that covers only the prediction range by the time required for processing.If this technology is applied to calculating the amount of movement, the obtained amount of movement will be an amount of movement from the past by the time required for processing, which will reduce the accuracy of the next prediction, and there is a high possibility that the search will fail again because the target object is not included in the prediction range.

[0006] An object of the present invention is to obtain the latest movement amount by quickly calculating the movement amount of a target object that moves relative to a camera. [Means for solving the problem]

[0007] To achieve the above object, the present invention provides a movement measurement system that measures the amount of relative movement between an object and a camera from captured images of the object captured by a camera, and includes: a basic search means that performs a basic search operation to search for an inter-image movement vector; a partition search means that performs a partition search operation to search for the inter-image movement vector; and a movement calculation means that calculates the amount of relative movement between the object and the camera from the searched inter-image movement vector. The movement measurement system has search execution states including a basic search iteration state in which the basic search means repeatedly performs the basic search operation, and a partition search iteration state in which the partition search means repeatedly performs the partition search operation. The movement measurement system also includes search execution state switching means that switches the search execution state to a partition search iteration state if the basic search means fails to search for an inter-image movement vector using the basic search operation in the basic search iteration state, and that switches the search execution state to the basic search iteration state if the partition search means succeeds in searching for an inter-image movement vector using the partition search operation in the partition search iteration state. In each basic search operation, the basic search means sets a tracked area in the current captured image, predicts multiple motion vectors that are likely to be the inter-image motion vectors to be searched for next time based on the inter-image motion vectors searched so far, sets each predicted motion vector as the next search target motion vector, and performs a search operation for the current inter-image motion vector using the current search target motion vectors set as the next search target motion vectors in the previous basic search operation as search targets. In each split search operation, the division search means sets a tracked area in the current captured image, sets multiple motion vectors as the current search target motion vectors, and performs a search operation for the current inter-image motion vector using the set current search target motion vectors as search targets. The split search means switches the multiple motion vectors set as search target motion vectors for each split search operation so that all motion vectors within a predetermined range are set as search target motion vectors during repeated execution of the split search operation.Then, in the search operation for the current inter-image movement vector, which is performed in the basic search operation and the divided search operation and which uses each of the current search target movement vectors as the search target, the area in the current captured image corresponding to each of the current search target movement vectors is taken as the area into which the previously set tracked area has been moved by the corresponding search target movement vector, and if an area that matches the previously set tracked area exists in the search area combined with the areas corresponding to each search target movement vector, the search for the inter-image movement vector is deemed successful, and the search target movement vector corresponding to the existing area is deemed to be the inter-image movement vector searched for this time; if no area exists, the search for the current inter-image movement vector is deemed to be unsuccessful.

[0008] In this movement amount measurement system, the division search means may switch the multiple movement vectors to be set as search target movement vectors for each division search operation so that once all of the movement vectors within a predetermined range have been set as search target movement vectors, all of the movement vectors within the predetermined range are set as search target movement vectors while the division search operation is repeatedly performed again.

[0009] In this movement amount measurement system, the predetermined range may be a range of a movement vector that, when set as the search target movement vector, causes the search region to be an area within the captured image. In this movement amount measuring system, the division search means may set the tracked region at the same position in the captured image in each division search operation. In this case, the division search means may set a search target movement vector for each time so that the search area moves from a position closer to the tracked area to a position farther away while repeatedly performing the division search operation. Alternatively, in this movement amount measurement system, the division search means may set the tracked area at a different position in the captured image in each division search operation so that the search area in each division search operation is at the same position in the captured image. In this case, the split search means may set the tracked area and the search target movement vector in each split search operation so that the tracked area moves from a position closer to the search area to a position farther away while repeatedly executing the split search operation. Alternatively, in this movement amount measuring system, the division search means may change the position of the tracked region in the captured image and the position of the search region in the captured image while repeatedly performing the division search operation. In addition, in this movement amount measurement system, the division search means may set a different set of movement vectors in each division search operation as the search target movement vector in order of the smallest absolute value of the difference from the inter-image movement vector that was last successfully searched in the basic search operation.

[0010] In addition, in this movement amount measurement system, the divided search means may set the tracked area in which the inter-image movement vector was last successfully searched for in the basic search operation as the tracked area in each divided search operation, and may set the search target movement vector in each divided search operation so that the search area moves from a position close to the search area in which the inter-image movement vector was last successfully searched for in the basic search operation to a position farther away.

[0011] In addition, in this movement amount measurement system, the divided search means may set the tracked area and the search target movement vector in each divided search operation so that the search area in which the inter-image movement vector was last successfully searched for in the basic search operation becomes the search area in each divided search operation, and the tracked area moves from a position close to the tracked area in which the inter-image movement vector was last successfully searched for in the basic search operation to a position farther away.

[0012] Furthermore, the movement amount measuring system may be one in which the camera is mounted on a moving body and captures an image of a surface on which the moving body travels as the target object. This motion measurement system uses a search target motion vector, which is predicted from the inter-image motion vectors calculated up to that point and is expected to be calculated as the next inter-image motion vector. If the basic search operation fails to find an inter-image motion vector from among the search target motion vectors, a split search operation is performed. The split search operation then searches for an inter-image motion vector from among the search target motion vectors while comprehensively changing the search target motion vector sequentially within a predetermined range. Therefore, regardless of the motion vector between the actual captured images, it is expected that the split search operation will eventually succeed in finding the inter-image motion vector.

[0013] Furthermore, since the split search operation sets a tracked area on the most recent captured image for each split search operation, the inter-image movement vector searched for by the split search operation is always the movement vector between captured images that are adjacent in time, and the most recent movement vector between captured images can be calculated as the inter-image movement vector.

[0014] Furthermore, since the split search operation sets a tracked area on the most recent captured image for each split search operation, even if an image matching the tracked area no longer exists in the captured image because the part of the object corresponding to the image of the tracked area at the time when the basic search operation failed to search for the inter-image movement vector is outside the shooting range or the orientation of that part has changed, the inter-image movement vector can be found without any problems.

[0015] Furthermore, the standard state is the basic search repetition state in which the basic search operation of searching for inter-image motion vectors is repeatedly performed using only motion vectors predicted from inter-image motion vectors searched up to now, so the amount of processing required for inter-image motion vector search in the standard state does not increase. [Effects of the Invention]

[0016] As described above, according to the present invention, the amount of movement of an object that moves relative to the camera can be calculated in a short time, thereby making it possible to acquire the most recent amount of movement. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a configuration of a movement amount measuring system according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an application example of a movement amount measuring system according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram illustrating a basic search operation of a motion vector according to an embodiment of the present invention. [Figure 4] FIG. 10 is a diagram illustrating a basic search operation of a motion vector according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram illustrating a division search operation of a motion vector according to an embodiment of the present invention. [Figure 6] 10A and 10B are diagrams illustrating another example of a division search operation for a motion vector according to an embodiment of the present invention. [Figure 7] 10A and 10B are diagrams illustrating another example of a division search operation for a motion vector according to an embodiment of the present invention. [Figure 8] 10A and 10B are diagrams illustrating another example of a division search operation for a motion vector according to an embodiment of the present invention. [Figure 9] 10A and 10B are diagrams illustrating an example of a calculation operation of an actual distance conversion coefficient according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] A first embodiment of the present invention will be described. FIG. 1 shows the configuration of a movement amount measuring system according to the first embodiment. As shown in the figure, the movement amount measurement system includes a measurement device 1 and a stereo camera 2. The measurement device 1 includes a movement vector search unit 11, a separation distance measurement unit 12, an actual distance conversion coefficient calculation unit 13, and a movement state calculation unit . The stereo camera 2 includes two cameras: a first camera 21 and a second camera 22. The movement measurement system is a system that measures the relative movement between the object photographed by the stereo camera 2 and the stereo camera 2, and can be applied to measuring the movement as shown in Figures 2a and 2b, for example. 2a shows an application example in which the stereo camera 2 is fixed to a moving body (a car in the figure) and the movement amount of the moving body relative to the road surface is measured with the road surface as the target object. In this case, the stereo camera 2 is arranged so that the first camera 21 and the second camera 22 capture images of the road surface from directly above. The first camera 21 and the second camera 22 are also arranged at a distance in a direction perpendicular to the standard movement direction MD (forward direction of the moving body) of the stereo camera 2 relative to the target object (road surface) and perpendicular to the optical axes of the first camera 21 and the second camera 22. The first camera 21 and the second camera 22 are also arranged so that the upward direction of the captured images corresponds to the movement direction MD.

[0019] FIG. 2b shows an application example in which the stereo camera 2 is fixed to a reference coordinate system, and the object surface (in the figure, the surface on which the transported object of a belt conveyor is placed) that moves relative to the reference coordinate system is used to measure the amount of movement of the object relative to the reference coordinate system. In this case, the stereo camera 2 is arranged so that the first camera 21 and the second camera 22 capture images of the object surface from directly above. The first camera 21 and the second camera 22 are also arranged at a distance from each other in a direction perpendicular to the relative movement direction MD of the stereo camera 2 with respect to the object (object surface) (the opposite direction to the movement direction of the object surface with respect to the reference coordinate system) and perpendicular to the optical axes of the first camera 21 and the second camera 22. The first camera 21 and the second camera 22 are also arranged so that the upward direction of the captured images corresponds to the movement direction MD.

[0020] Next, the basic search operation performed by the movement vector search unit 11 of the measurement device 1 will be described. The basic search operation is a process of searching for the movement vector of the captured image between adjacent measurement operation execution points in time, and is performed at each measurement operation execution point, which consists of a fixed cycle, after the measurement operation of the measurement device 1 starts, until the search for the movement vector fails. As shown in FIG. 3a, the start time of the measurement operation is represented by t=1, and the time when the nth measurement operation is performed from the start time of the process is represented by t=n. Also, the image captured by the first camera 21 at t=n is represented by P(t=n). 3b, in the basic search operation, at time t=i, a tracked area A(t=i) is set in an image P(t=i), and a next search area NS(t=i) is also set. The setting operations of the tracked area A(t=i) and the next search area NS(t=i) will be described later. The tracked area A(t=i) has a fixed size of multiple pixels by multiple pixels (3 pixels by 3 pixels in the illustrated example). Next, as shown in Figure 3c, at time t=i+1, each motion vector toward each area of ​​the same size as the tracked area A(t=i) in the next search area NS(t=i) previously set for image P(t=i+1) is set as the search target motion vector corresponding to that area, and an inter-image motion vector VP(t=i+1) is searched for from among the search target motion vectors corresponding to each area.

[0021] The inter-image motion vector VP(t=i+1) is searched for from among these search target motion vectors as follows. That is, the area corresponding to each search target motion vector is sequentially set as a similarity calculation area, and the similarity between the similarity calculation area and the tracked area A(t=i) is calculated. If a similarity calculation area with a calculated similarity above a predetermined threshold appears, the search for the inter-image motion vector VP is deemed successful, and the resulting similarity calculation area is designated as the tracking result area B(t=i+1). Then, the motion vector in image space (the search target motion vector corresponding to the tracking result area B(t=i+1)) with its starting point at the reference point Ag(t=i) of the tracked area A(t=i) and its ending point at the reference point Bg(t=i+1), which is the pixel position at the bottom left of the tracking result area B(t=i+1) is designated as the searched inter-image motion vector VP(t=i+1). On the other hand, if no similarity calculation area with a similarity above the threshold appears, the search for the inter-image motion vector VP is deemed unsuccessful. Here, similarity is an index between 0 and 1 calculated based on the image information of two regions, and the more similar the two regions are, the closer the value is to 1, while if the image information is exactly the same, the value is 1.

[0022] In addition, the tracked area setting unit sets a tracked area A(t=i+1) in the image P(t=i+1) and also sets a next search area NS(t=i+1), as shown in FIG. 3d. Next, as shown in Figure 3e, at time t=i+2, each motion vector toward each area of ​​the same size as the tracked area A(t=i+1) in the next search area NS(t=i+1) previously set for image P(t=i+2) is set as the search target motion vector corresponding to that area, and an inter-image motion vector VP(t=i+2) is searched for from among the search target motion vectors corresponding to each area.

[0023] In searching for the inter-image motion vector VP(t=i+2), the regions corresponding to each search target motion vector are sequentially set as similarity calculation regions, the similarity between the similarity calculation region and the tracked region A(t=i+1) is calculated, and if a similarity calculation region with a calculated similarity above a predetermined threshold appears, the search for the inter-image motion vector VP is deemed successful, and the resulting similarity calculation region is designated as the tracking result region B(t=i+2).Then, the motion vector in image space (the search target motion vector corresponding to the tracking result region B(t=i+2)) starting from the reference point Ag(t=i+1) of the tracked region A(t=i+1) and ending at the reference point Bg(t=i+2), which is the pixel position at the bottom left of the searched tracking result region B(t=i+2) is designated as the searched inter-image motion vector VP(t=i+2). On the other hand, if no similarity calculation region with a similarity equal to or greater than the threshold value appears, the search for the inter-image motion vector VP is deemed to have failed.

[0024] Similarly, the search for the inter-image motion vector VP from the current search target motion vector and the setting of the current tracked area A and the next search area NS are repeated until the search for the inter-image motion vector VP fails. Note that if the similarity calculated for multiple search target motion vectors each time exceeds a predetermined threshold, the inter-image motion vector VP can be the motion vector with the highest similarity, the motion vector closest in magnitude to the previous inter-image motion vector VP, or the motion vector closest in direction to the previous inter-image motion vector VP, or one motion vector selected by a composite evaluation.

[0025] Here, the setting operation of the tracked area A and the next search area NS will be described. In this setting operation, the motion vector search unit 11 sets a plurality of search candidate motion vectors as a search candidate motion vector set. Each search candidate motion vector, where the current time is t=i, is either the currently calculated inter-image motion vector VP(t=i) or a predetermined number of motion vectors predicted from the history of inter-image motion vectors VP calculated up to the current time and with the highest probability of being calculated as the next inter-image motion vector VP(t=i+1). The predetermined number of motion vectors with the highest probability of becoming the next inter-image motion vector VP(t=i+1), calculated using a predetermined probability calculation method based on the history of inter-image motion vectors VP calculated up to the current time, can be used as the predetermined number of motion vectors with the highest probability of becoming the next inter-image motion vector VP(t=i+1). Alternatively, a predetermined number of motion vectors with similar orientations and magnitudes to the currently calculated inter-image motion vector VP(t=i) or the next inter-image motion vector VP(t=i+1) predicted from the history of inter-image motion vectors VP calculated up to the current time, can be used as the predetermined number of motion vectors with the highest probability of being calculated. However, at the initial time t=1, there is no inter-image motion vector VP calculated up to that point, so a plurality of predetermined search candidate motion vectors are set as a search candidate motion vector set.

[0026] Then, according to the search candidate motion vector set, the relative positional relationship between the tracked area A(t=i) and the next search area NS(t=i) and the shape of the next search area NS(t=i) are determined. The next search area NS(t=i) is determined as the sum of the areas obtained by moving the tracked area A(t=i) by the search candidate motion vector for each search candidate motion vector in the search candidate motion vector set, and its shape and relative position to the tracked area A(t=i) are determined. For example, if the search candidate motion vector set includes 16 search candidate motion vectors ev1-ev16 shown in Figure 4a, for each of the search candidate motion vectors ev1-ev16, the next search area NS(t=i) shown in Figure 4b2 is determined as the sum of the areas obtained by moving the tracked area A(t=i) by the search candidate motion vector as shown in Figure 4b1.

[0027] Then, according to the relative positional relationship between the shape of the next search area NS(t=i) and the tracked area A(t=i), the tracked area A(t=i) is set in the image P(t=i) and the next search area NS(t=i) is set so that both the tracked area A(t=i) and the next search area NS(t=i) are areas within the captured image. At this time, the tracked area A(t=i) and the next search area NS(t=i) may be set so that the tracked area A(t=i) in the image P(t=i) is an area that includes as many distinctive images as possible.

[0028] Therefore, when the next search area NS(t=i) is determined as shown in Figure 4b2, the tracked area A(t=i) and the next search area NS(t=i) are set so that both the tracked area A(t=i) and the next search area NS(t=i) are areas within the captured image, as shown in Figure 4c, for example. By setting the next search area NS(t=i) based on the search candidate motion vector set in this manner, in the basic search operation at t=i+1, the search candidate motion vector set used to set the next search area NS(t=i) becomes the set of search target motion vectors, and a search is performed for the inter-image motion vector VP(t=i+1) using each of the vectors included in the search candidate motion vector set as the search target motion vector.

[0029] Here, the number and range of vectors included in the search candidate motion vector set are set so that the search for the inter-image motion vector VP can be completed in the basic search operation at each measurement execution time point before the next measurement execution time point arrives. Furthermore, if the motion vector search unit 11 fails to search for the inter-image motion vector VP in the basic search operation, it stops the basic search operation and starts a divided search operation. That is, when a tracked area A(t=j) and a next search area NS(t=j) are set at t=j as shown in Fig. 5a1, if there is no area in the next search area NS(t=j) of image P(t=j+1) at t=j+1 that has a similarity to the tracked area A(t=j) that is equal to or greater than the threshold as shown in Fig. 5a2, the search for the inter-image motion vector VP at t=j+1 fails, the basic search operation is stopped, and the split search operation is started. The split search operation starts at t=j+1, when the search for the inter-image motion vector VP failed, and image P(t=j+1) becomes the target of the initial split search operation.

[0030] The division search operation is also a process of searching for the motion vector of the captured images between the time points of adjacent measurement operations as the inter-image motion vector VP, and is performed at each measurement operation time point until the search for the inter-image motion vector VP is successful. Then, if the search for the inter-image motion vector VP is successful, the motion vector search unit 11 stops the division search operation and returns to the basic search operation.

[0031] In the division search operation, at each time point, an inter-image motion vector VP is searched for from among the search target motion vectors at that time point, and the tracked area A and the next search area NS at that time point are set. However, in the first division search operation at t=j+1, only the tracked area A and the next search area NS are set.

[0032] The difference between the split search operation and the basic search operation is that the position of the tracked area A in the image P is fixed, and the set of motion vectors used as the set of search target motion vectors in each split search operation is switched in sequence. The position in image P where tracked area A is fixed is, for example, the center position of image P shown in Fig. 5b1. Furthermore, switching of the set of movement vectors used as the set of search target movement vectors is performed by setting the next search area NS for each iteration so that the similarity calculation area for calculating the similarity with tracked area A can be changed from the inside to the outside of the spiral centered on tracked area A in a series of divided search operations, as shown in Fig. 5b2. The next search area NS for each iteration is set so that all areas in image P that are the same size as tracked area A are sequentially set as similarity calculation areas.

[0033] However, the size and shape of the next search area NS are set so that the search for the inter-image motion vector VP can be completed before the next measurement execution time arrives in the division search operation performed at each measurement execution time. 5c1-c7 show the transition of the next search area NS set in this way. By setting the next search area NS for each iteration as shown in the figures, it is possible to change the similarity calculation area for calculating the similarity with the tracked area A along the spiral from the inside to the outside of the spiral centered on the tracked area A in a series of divided search operations.

[0034] Then, in a series of division search operations, motion vector search unit 11 calculates the similarity between the previously set tracked area A and the similarity calculation area while changing the similarity calculation area within the next search area NS set previously, moving from the inside to the outside of the spiral centered on tracked area A and proceeding along the spiral, and if a similarity calculation area appears for which a similarity calculation is calculated that is equal to or greater than a predetermined threshold, it determines that the search for the inter-image motion vector VP for that division search operation is successful, and the similarity calculation area that appears is designated as tracking result area B. The motion vector (search target candidate vector corresponding to tracking result area B) that starts at reference point Ag of the previous tracked area A and ends at reference point Bg, the pixel position at the bottom left of current tracking result area B, is designated as the searched inter-image motion vector VP. On the other hand, if a similarity calculation area with a similarity equal to or greater than the threshold does not appear, the search for the inter-image motion vector VP is determined to be unsuccessful.

[0035] Furthermore, if the search for the inter-image motion vector VP is not successful even for the last next search area NS shown in Figure 5c7, the next search area NS will be set again, sequentially, from the first next search area NS in Figure 5c1, as shown in Figures 5c1-c7. The division search operation has been described above. In the above, the set of motion vectors used as the set of search target motion vectors in each divided search operation is switched in sequence by changing the next search area NS, but this switching can also be performed without using the next search area NS. In this case, in each division search operation, the inter-image movement vector VP is searched for while moving the similarity calculation area from the inside to the outside of the spiral, starting from the similarity calculation area whose position along the spiral centered on the tracked area A is the next position of the similarity calculation area last set in the previous division search operation, and the current division search operation is terminated before the next measurement execution time arrives, and the search is resumed in the next division search operation.

[0036] According to this embodiment, a predetermined number of motion vectors with the highest expected probability of being calculated as the next inter-image motion vector VP, predicted from the inter-image motion vectors VP calculated up to this point, are selected as search target motion vectors. If the basic search operation to search for the inter-image motion vector VP among the search target motion vectors fails, the basic search operation is stopped and a split search operation is performed. Then, in the split search operation, the search target motion vector is sequentially and comprehensively changed to search for the inter-image motion vector VP among the search target motion vectors. Therefore, regardless of the actual inter-image motion vector, it is expected that the split search operation will eventually succeed in searching for the inter-image motion vector VP.

[0037] Furthermore, since the division search operation sets a tracked area A on the latest image P for each division search operation, the inter-image motion vector VP searched by the division search operation is always the motion vector between images P that are adjacent in time, and the latest motion vector between images P can be calculated as the inter-image motion vector VP.

[0038] Furthermore, since the split search operation sets a tracked area A on the latest image P for each split search operation, even if an image matching the tracked area A no longer exists in the image P because the part of the object corresponding to the image of the tracked area A at the time when the basic search operation failed to search for the inter-image movement vector VP has gone outside the shooting range or the orientation of that part has changed, the inter-image movement vector VP can be found without any problems.

[0039] Furthermore, if the standard state is to repeatedly perform a basic search operation that searches for an inter-image motion vector VP using only search candidate motion vectors predicted from the inter-image motion vectors VP that have been searched so far, then even in the divided search operation, the amount of processing performed at the time of execution of each measurement operation, which consists of a fixed cycle, does not increase.

[0040] In the above division search operation, as shown in Fig. 5b2, in each division search operation, the similarity calculation area is changed so as to proceed along the spiral from the inside to the outside of the spiral centered on the tracked area A. By changing the similarity calculation area in this way, the search for the inter-image movement vector VP, which is the search target for a smaller movement vector, is performed earlier, so it can be expected that the inter-image movement vector VP can be searched for more quickly when the relative movement speed between the target object and the stereo camera 2 is low.

[0041] Here, in the above-described division search operation, the next search area NS for each round is set so that in a series of division search operations, the similarity calculation area changes from the inside to the outside of the spiral centered on the tracked area A, whose position is fixed. However, the next search area NS for each round may also be set so that the similarity calculation area changes from the inside to the outside of the spiral centered on the area to which the tracked area A has moved by the inter-image movement vector VP that was last successfully searched.

[0042] By doing this, the search for the inter-image motion vector VP is performed first, with the motion vector to be searched being the motion vector with the smaller absolute value of the difference from the last successfully searched inter-image motion vector VP. Therefore, when the change in the relative motion speed between the target object and the stereo camera 2 is small, it is expected that the inter-image motion vector VP can be searched for more quickly.

[0043] In addition, the above-mentioned split search operation may be configured such that the tracked area A in which the inter-image motion vector VP was last successfully searched for in the basic search operation is set as the tracked area in each split search operation, and the next search area NS in each split search operation moves from a position closer to the next search area NS in which the inter-image motion vector was last successfully searched for in the basic search operation toward a position farther away.

[0044] Alternatively, the above split search operation may be performed by setting the next search area NS in each split search operation to the next search area NS in the basic search operation, and by setting the tracked area A in each split search operation so that it moves from a position closer to the tracked area A in the basic search operation in which the inter-image motion vector VP was last successfully searched for, toward a position further away.

[0045] Even if we do this, the search for the inter-image motion vector VP will be performed first, with the motion vector to be searched being the motion vector with the smaller absolute value of the difference from the last successfully searched inter-image motion vector VP. Therefore, if the change in the relative motion speed between the target object and the stereo camera 2 is small, it can be expected that the inter-image motion vector VP can be searched for more quickly.

[0046] In addition, each division search operation may be performed in the order of a raster scan (a raster scan from the top left to the bottom right) in which the operation proceeds from left to right, as shown in Figure 6a, and the similarity calculation area is changed by moving from top to bottom in the vertical direction from left to right. In this case, by setting the next search area NS for each round as shown in FIGS. 6b1 to 6b7, the similarity calculation area can be changed in the raster scan order in each divided search operation. Furthermore, if the search for the inter-image motion vector VP is not successful even in the last next search area NS shown in Figure 6b7, the next search area NS will be set again, starting from the first next search area NS in Figure 6b1, in sequence as shown in Figures 6b1-b7. This division search operation may also be performed without using the next search area NS. In this case, in each division search operation, the inter-image movement vector VP is searched for while advancing the similarity calculation area in raster scan order from the similarity calculation area whose position in raster scan order is the next position of the similarity calculation area last set in the previous division search operation, and the current division search operation is terminated before the next measurement execution time arrives, and the search is resumed in the next division search operation.

[0047] In the above, in the split search operation, the position of the tracked area A in the image P is fixed and the next search area NS set in each split search operation is changed. However, by fixing the position of the next search area NS in the image P and changing the position of the tracked area A set in each split search operation, the set of motion vectors used as the set of search target motion vectors in each split search operation can be switched in sequence.

[0048] In this case, for example, the position of the next search region NS in the image P is fixed at the center of the image P, as shown in FIG. 7a. Then, as shown in Figures 7b1-b15, the position of the tracked area A is changed so that each search target movement vector of the set of search target movement vectors determined by the positional relationship between the tracked area A and the next search area NS does not overlap in each divided search operation. Preferably, the tracked area A is set each time so that it moves from a position closer to the next search area NS to a position farther away. By moving the tracked area A in this way, a search for an inter-image motion vector VP that targets a smaller motion vector is performed earlier, so that when the standard relative motion speed between the target object and the stereo camera 2 is low, it can be expected that the inter-image motion vector VP can be searched for more quickly.

[0049] If the search is unsuccessful even after the setting of the tracked area A at the final position, the search is started again from the setting of the tracked area A at the initial position. In this way, even if the position of the next search area NS is fixed and the position of the tracked area A is changed, the set of motion vectors used as the set of search target motion vectors in each divided search operation can be switched in sequence. In addition, the above split search operation may be performed while changing both the position of the next search area NS and the position of the tracked area A, thereby switching the set of search candidate motion vectors used as search target motion vectors in sequence for each split search operation. For example, as shown in Fig. 8a1, tracked region A is fixed to the center position in the horizontal direction of the top edge of image P, and in each divided search operation, the next search region NS is set as shown in Figs. 8c1-c7, and the similarity calculation region is changed in raster scan order from the upper left to the lower right as shown in Fig. 8a2 while repeating the divided search operation. Then, when the next search region NS has advanced to the position of the lower right corner of image P, tracked region A is fixed to the center position in the horizontal direction of the bottom edge of image P as shown in Fig. 8b1, and in each divided search operation, the next search region NS is set as shown in Figs. 8c8-c14, and the similarity calculation region is changed in raster scan order from the lower right to the upper left as shown in Fig. 8b2 while repeating the divided search operation.

[0050] If the search for the inter-image motion vector VP is not successful even after proceeding to the setting in FIG. 8c14, the search returns to the setting in FIG. 8c1 again. In this way, by changing both the position of the next search area NS and the position of the tracked area A in a series of divided search operations, it is possible to set a larger motion vector as the search target motion vector and search for the inter-image motion vector VP than when changing only one of the positions. In the case of Figure 8, it is possible to set a larger motion vector as the search target motion vector in the main motion direction MD and in the direction opposite to the motion direction MD than when changing only one of the positions.

[0051] This division and search operation may also be performed without using the next search area NS. In this case, in each division and search operation, first, the tracked area A is fixed to the center position in the horizontal direction of the upper edge of image P as shown in Fig. 8a1, and the inter-image motion vector VP is searched for while proceeding through the similarity calculation areas in the raster scan order shown in Fig. 8a2, starting from the similarity calculation area next to the similarity calculation area last set in the previous division and search operation; next, the tracked area A is fixed to the center position in the horizontal direction of the lower edge of image P as shown in Fig. 8b1, and the inter-image motion vector VP is searched for while proceeding through the similarity calculation areas in the raster scan order shown in Fig. 8b2, starting from the similarity calculation area next to the similarity calculation area last set in the previous division and search operation. Each division and search operation is terminated before the next measurement execution time arrives, and the search is resumed in the next division and search operation.

[0052] Next, the calculation operation of the actual distance conversion coefficient performed by the separation distance measurement unit 12 and the actual distance conversion coefficient calculation unit 13 of the measurement device 1 will be described. At time t=1, separation distance measurement unit 12 calculates the distance ZA(t=1) in the optical axis direction from first camera 21 and second camera 22 constituting stereo camera 2 to a position on the target object reflected in the center of tracked area A(t=1). Also, at each time t=i+1, it calculates the distance ZA(t=i+1) in the optical axis direction from first camera 21 and second camera 22 to a position on the target object reflected in the center of tracked area A(t=i+1), and the distance ZB(t=i+1) in the optical axis direction from first camera 21 and second camera 22 to a position on the target object reflected in the center of tracking result area B(t=i+1).

[0053] The distance Z in the optical axis direction from the first camera 21 and the second camera 22 to the position on the target object can be calculated as follows. 9, let F [mm] be the focal length between the first camera 21 and the second camera 22 of the stereo camera 2, and M [mm] be the base length, which is the distance in real space between the first camera 21 and the second camera 22. Also, let Tg be the position on the object where distance Z [mm] is measured, let c1 be the left-right position in the image space where position Tg is photographed by the first camera 21, and let c2 be the left-right position in the image space where position Tg is photographed by the second camera 22. Also, let c11 be the position corresponding to c1 on the surface SF that is a focal length F away from the first camera 21 in the optical axis direction, and let c21 be the position corresponding to c2 on the surface SF that is a focal length F away from the second camera 22 in the optical axis direction.

[0054] Also, let xl [mm] be the distance in the left-right direction from the center position of the first camera 21 to c11, and let xr [mm] be the distance in the left-right direction from the center position of the second camera 22 to c21. Then, if p=xl-xr, according to the principle of triangulation, the distance Z of the position Tg is Z=(M×F) / p It is calculated by If the size of the area on the surface projected onto one pixel of the first camera 21 and located at a focal distance F from the first camera 21 is S [mm / pixel], when the coordinate in the image space of the first camera 21 changes by 1, the position of Tg changes by (Z / F) × S in the direction perpendicular to the distance Z. Therefore, using this relationship, the actual distance conversion coefficient calculation unit 13 calculates the actual distance conversion coefficient K(t=i+1) from ZA(t=i) and ZB(t=i+1) measured by the separation distance measurement unit 12 at each point in time t=i+1. The actual distance conversion coefficient K(t=i+1) is calculated by taking the approximate Z(t=i+1) as the average value of Z(t=i) and Z(t=i+1), {Z(t=i)+Z(t=i+1)} / 2, K(t=i+1)={Z(t=i+1) / F}×S It is calculated by:

[0055] Next, at each time point t=i+1, the movement state calculation unit 14 of the measurement device 1 calculates the relative movement vector V(t=i+1) of the target object with respect to the stereo camera 2 using the inter-image movement vector VP(t=i+1) in the image space searched by the movement vector search unit 11 and the actual distance conversion coefficient K(t=i+1) calculated by the actual distance conversion coefficient calculation unit 13, as follows: Calculate V(t=i+1)=K(t=i+1)×VP(t=i+1).

[0056] Then, from the calculated relative movement vector V(t=i+1) of the object with respect to the stereo camera 2, various movement states such as the movement speed, acceleration, and movement direction of the object with respect to the stereo camera 2, or of the stereo camera 2 with respect to the object, are calculated and output.Which of the movement state to calculate, the relative movement state of the object with respect to the stereo camera 2 or the relative movement state of the stereo camera 2 with respect to the object, is set according to the application of the movement amount measurement system.

[0057] Furthermore, the movement state calculation unit 14 may also calculate and output a relative angle or the like as a movement state from a plurality of distances in the image calculated by the separation distance measurement unit 12. The embodiments of the present invention have been described above. In the embodiment, the distance ZA(t=i) and the distance ZB(t=i+1) calculated by the separation distance measuring unit 12 using the stereo camera 2 may be calculated using another device such as a laser range finder. Furthermore, the roles of the first camera 21 and the second camera 22 in the embodiment may be interchanged. [Explanation of symbols]

[0058] 1...measuring device, 2...stereo camera, 11...movement vector search unit, 12...separation distance measurement unit, 13...actual distance conversion coefficient calculation unit, 14...movement state calculation unit, 21...first camera, 22...second camera.

Claims

1. A movement amount measurement system that measures a relative movement amount between an object and a camera from a captured image that is an image of the object captured by the camera, a basic search means for performing a basic search operation to search for an inter-image motion vector; a division search means for performing a division search operation to search for an inter-image motion vector; Search execution state switching means; a movement amount calculation means for calculating a relative movement amount between the object and the camera from the inter-image movement vector found; The movement amount measurement system has, as search execution states, a basic search repetition state in which the basic search means repeatedly executes the basic search operation, and a divided search repetition state in which the divided search means repeatedly executes the divided search operation, the movement amount measurement system comprises a search execution state switching means for switching a search execution state to the divided search iteration state if the basic search means fails to search for an inter-image movement vector by the basic search operation in the basic search iteration state, and for switching a search execution state to the basic search iteration state if the divided search means succeeds in searching for an inter-image movement vector by the divided search operation in the divided search iteration state; In each of the basic search operations, the basic search means sets a tracked area in the current captured image, predicts a plurality of motion vectors that are likely to become the inter-image motion vectors to be searched for next from the inter-image motion vectors searched for up to now, sets each predicted motion vector as a next search target motion vector, and performs a search operation for the current inter-image motion vector, with each current search target motion vector set as the next search target motion vector in the previous basic search operation as a search target; the division search means, in each division search operation, sets a tracked region in the current captured image, sets a plurality of movement vectors as current search target movement vectors, and performs a search operation for a current inter-image movement vector using the set current search target movement vectors as search targets; the division search means switches the plurality of movement vectors to be set as search target movement vectors for each division search operation so that all movement vectors within a predetermined range are set as search target movement vectors while repeatedly executing the division search operation; A movement amount measurement system characterized in that, in the search operation for the current inter-image movement vector, which is performed in the basic search operation and the divided search operation and which uses each of the current search target movement vectors as the search target, the area in the current captured image corresponding to each of the current search target movement vectors is taken as the area into which the previously set tracked area is moved by the corresponding search target movement vector, and if an area that matches the previously set tracked area exists in the search area which is the combined area corresponding to each search target movement vector, the search for the inter-image movement vector is deemed to be successful, and the search target movement vector corresponding to the existing area is deemed to be the inter-image movement vector searched for this time, and if no area exists, the search for the current inter-image movement vector is deemed to be unsuccessful.

2. 2. The movement amount measurement system according to claim 1, A movement amount measurement system characterized in that the division search means switches the multiple movement vectors to be set as search target movement vectors for each division search operation so that once all movement vectors within a predetermined range have been set as search target movement vectors, all movement vectors within the predetermined range are set as search target movement vectors while repeatedly executing the division search operation.

3. 3. The movement amount measurement system according to claim 1, A movement amount measurement system, wherein the predetermined range is a range of movement vectors that, when set to the search target movement vector, makes the search area an area within the captured image.

4. 3. The movement amount measurement system according to claim 1, The movement amount measuring system is characterized in that the division search means sets the tracked region at the same position in the captured image in each division search operation.

5. 3. The movement amount measurement system according to claim 1, A movement measurement system characterized in that the divided search means sets the tracked area to a different position in the captured image in each divided search operation so that the search area in each divided search operation is at the same position in the captured image.

6. 3. The movement amount measurement system according to claim 1, A movement amount measuring system, characterized in that the position of the tracked region in the photographed image and the position of the search region in the photographed image change while the division search means repeatedly executes the division search operation.

7. 5. The movement amount measuring system according to claim 4, A movement measurement system characterized in that the division search means sets a search target movement vector in each division search operation so that the search area moves from a position closer to the tracked area to a position farther away while repeatedly executing the division search operation.

8. The movement amount measurement system according to claim 5, A movement measurement system characterized in that the division search means sets a tracked area and a search target movement vector in each division search operation so that the tracked area moves from a position closer to the search area to a position farther away while repeatedly executing the division search operation.

9. 3. The movement amount measurement system according to claim 1, The motion measurement system is characterized in that the division search means sets a different set of motion vectors in each division search operation as the search target motion vector in order of the smallest absolute value of the difference from the inter-image motion vector that was last successfully searched in the basic search operation.

10. 3. The movement amount measurement system according to claim 1, The movement amount measurement system is characterized in that the division search means sets the tracked area in which the inter-image movement vector was last successfully searched for in the basic search operation as the tracked area in each division search operation, and sets the search target movement vector in each division search operation so that the search area moves from a position close to the search area in which the inter-image movement vector was last successfully searched for in the basic search operation to a position farther away.

11. 3. The movement amount measurement system according to claim 1, The movement amount measurement system is characterized in that the division search means sets the tracked area and the search target movement vector in each division search operation so that the search area in which the inter-image movement vector was last successfully searched for in the basic search operation becomes the search area in each division search operation, and the tracked area moves from a position close to the tracked area in which the inter-image movement vector was last successfully searched for in the basic search operation to a position farther away.

12. 3. The movement amount measurement system according to claim 1, The movement amount measurement system is characterized in that the camera is mounted on a moving body and captures an image of the surface on which the moving body travels as the target object.

Images