Movement measurement system
The system addresses the challenge of measuring relative movement with long monochromatic objects by defining pixel directions and using brightness change criteria to set tracked areas, ensuring accurate and efficient movement measurement.
Patent Information
- Application Number
- JP2021195427
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-12-01
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-12-01
AI Technical Summary
Existing methods for measuring relative movement between a camera and an object using template matching struggle when the object has long monochromatic parts in the direction of movement, leading to incorrect area searches and inaccurate movement measurements.
A movement measurement system that defines pixel directions on an image, sets tracked areas using brightness change non-flatness criteria, and calculates relative movement by converting tracked area movements into actual distances, excluding areas with flat brightness changes in the direction of movement.
Enables accurate measurement of relative movement between a camera and an object, even with long monochromatic parts, by setting appropriate tracked areas and reducing processing load and delay.
Smart Images

Figure 0007588575000001 
Figure 0007588575000002 
Figure 0007588575000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for measuring the amount of relative movement between a camera and an object by using an image of the object captured by a camera. [Background technology]
[0002] A known technique for measuring the amount of relative movement between a camera and an object using an image of the object captured by a camera involves setting a specific area in an image of the road surface captured by a camera mounted on a moving object at a first time t1 as a feature area, searching for an area in an image of the road surface captured by the camera at a time t2 after time t1 that is similar to the feature area using template matching, regarding the searched area as an area that contains the same part of the object captured in the feature area, converting the amount of movement of the searched area from the feature area into the amount of movement in real space, and calculating the amount of movement of the camera relative to the road surface (for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2020-190458 A Summary of the Invention [Problem to be solved by the invention]
[0004] According to the technology for measuring the relative amount of movement between a camera and an object using template matching described above, if an area in which a long, monochromatic part in the direction of movement of the object, such as a white line on the road, is reflected is set as a feature area at a first time t1, a large number of areas similar to the feature area will appear in the image captured at time t2, and it may become impossible to correctly search for an area in the image captured at time t2 that contains the same part of the object as the part reflected in the feature area.
[0005] Therefore, an objective of the present invention is to properly measure the amount of relative movement between a camera and an object using an image of the object captured by a camera, even if the object has a long, monochromatic part in the direction of movement. [Means for solving the problem]
[0006] In order to achieve the above object, the present invention provides a movement amount measurement system that measures the relative movement amount of a camera with respect to an object from a captured image, which is an image of the object captured by a camera, and is equipped with a tracked area setting means that sets a tracked area of N rows by M columns (where N>1, M>1) in the current captured image, where a direction in the captured image corresponding to the main relative movement direction of the camera with respect to the object is defined as the direction in which pixels are arranged in rows and the up-down direction, and a direction perpendicular to the direction in which the rows are arranged in the captured image is defined as the direction in which pixels are arranged in columns and the left-right direction, a search means that searches for an area in the current captured image that matches the tracked area set by the tracked area setting means last time, as a tracking result area, and a movement amount calculation means that converts the movement amount from the tracked area set by the tracked area setting means last time to the tracking result area searched by the search means this time, into the relative movement amount of the camera with respect to the object, during each measurement. The tracked area setting means sequentially calculates the degree to which the luminance change in the evaluation area is not flat as a luminance change non-flatness degree for each evaluation set for a plurality of small areas of N rows x M columns obtained by dividing a predetermined range in the current captured image, and sets the small area in which the evaluation area in which the luminance change non-flatness degree indicating that the luminance change is not flat by a predetermined level or more is set as the tracked area. The evaluation area set for the small area is made up of less than N rows including at least one of some rows in the small area and an area of 1 row x M columns adjacent to the small area in the vertical direction.
[0007] In addition, in order to achieve the above-mentioned object, the present invention provides a movement amount measurement system that measures the relative movement amount of a camera with respect to an object from a captured image, which is an image of the object captured by a camera, and is equipped with a tracked area setting means that sets a tracked area of N rows by M columns (where N>1, M>1) in the current captured image, where a direction in the captured image corresponding to a main relative movement direction of the camera with respect to the object is defined as the direction of arrangement of pixel rows and the up-down direction, and a direction perpendicular to the direction of arrangement of rows in the captured image is defined as the direction of arrangement of pixel columns and the left-right direction, a search means that searches for an area in the current captured image that matches the tracked area set by the tracked area setting means last time, as a tracking result area, and a movement amount calculation means that converts the movement amount from the tracked area set by the tracked area setting means last time to the tracking result area searched by the search means this time, into the relative movement amount of the camera with respect to the object, in each measurement. The tracked area setting means sets, as an evaluation area, an area corresponding to the bottommost row of a plurality of N rows x M columns small areas obtained by dividing a predetermined range in the current captured image, and sequentially calculates the degree to which the luminance change of each evaluation area is not flat as a luminance change non-flatness, and sets, as the tracked area, a small area having, as the bottommost row, an evaluation area for which a luminance change non-flatness indicating that the luminance change is not flat by a predetermined level or more, or an area shifted one row up from a small area having, as the bottommost row, an evaluation area for which the luminance change non-flatness was calculated,
[0008] In addition, in order to achieve the above-mentioned object, the present invention provides a movement amount measurement system that measures the relative movement amount of a camera with respect to an object from a captured image, which is an image of the object captured by a camera, and is equipped with a tracked area setting means that sets a tracked area of N rows by M columns (where N>1, M>1) in the current captured image, where a direction in the captured image corresponding to a main relative movement direction of the camera with respect to the object is defined as the direction of arrangement of pixel rows and the up-down direction, and a direction perpendicular to the direction of arrangement of rows in the captured image is defined as the direction of arrangement of pixel columns and the left-right direction, a search means that searches for an area in the current captured image that matches the tracked area set by the tracked area setting means last time, as a tracking result area, and a movement amount calculation means that converts the movement amount from the tracked area set by the tracked area setting means last time to the tracking result area searched by the search means this time, into the relative movement amount of the camera with respect to the object, in each measurement. The tracked area setting means moves an evaluation area, which is an area of 1 row by M columns, within a predetermined range in the current captured image, and calculates the degree to which the luminance change in the evaluation area is not flat as a luminance change non-flatness, and sets as the tracked area an N row by M column area having as the bottom row the evaluation area for which a luminance change non-flatness indicating that the luminance change is not flat by a predetermined level or more, or an N row by M column area adjacent to and above the evaluation area for which the luminance change non-flatness was calculated.
[0009] In this movement amount measurement system, the camera is configured to sequentially output pixels of each row of the captured image from left to right, and sequentially output each row of the captured image from bottom to top, and the tracked area setting means moves from the leftmost area to the rightmost area within each row of the specified range so that the area adjacent to the right of the current evaluation area becomes the next evaluation area, and calculates a luminance change non-flatness of the evaluation area while moving the evaluation area so that the row through which the evaluation area moves within the row changes sequentially from the bottommost row to the topmost row of the specified range, and when a luminance change non-flatness indicating that the luminance change is not flat by more than a predetermined level appears, an N row x M column area with the evaluation area in which the luminance change non-flatness was calculated as the lowest row, or an N row x M column area adjacent above the evaluation area in which the luminance change non-flatness was calculated, may be set as the tracked area.
[0010] In this movement amount measurement system, the camera is configured to sequentially output pixels of each row of the captured image from left to right, and sequentially output each row of the captured image from bottom to top, and the tracked area setting means moves from the leftmost area to the rightmost area within each row of the specified range so that the area where the current evaluation area is shifted one column to the right becomes the next evaluation area, and calculates a luminance change non-flatness of the evaluation area while moving the evaluation area so that the row through which the evaluation area moves within the row changes sequentially from the bottommost row to the topmost row of the specified range, and when a luminance change non-flatness indicating that the luminance change is not flat by more than a predetermined level appears, an N row x M column area with the evaluation area in which the luminance change non-flatness was calculated as the lowest row, or an N row x M column area adjacent to the evaluation area in which the luminance change non-flatness was calculated, may be set as the tracked area.
[0011] Here, in the movement amount measurement system, the tracked area setting means may calculate the sum of absolute values of the luminance differences between each pixel in the evaluation area and the pixels in the evaluation area adjacent to the pixel in the column arrangement direction as the luminance change non-flatness of the evaluation area. In the above movement amount measuring system, the tracked area setting means may determine a luminance change non-flatness degree exceeding a predetermined threshold value as a luminance change non-flatness degree indicating that the luminance change is not flat by a predetermined level or more. In the above-described movement amount measurement system, the tracked area setting means may determine that a luminance change non-flatness value that exceeds a predetermined threshold value, obtained by multiplying the luminance change non-flatness value by a weight that is set to be larger the closer the evaluation area is to the center of the captured image, is the luminance change non-flatness value that indicates that the luminance change is not flat by a predetermined level or more.
[0012] In the above-described movement amount measurement system, the tracked area setting means may set the luminance change non-flatness or a value obtained by multiplying the luminance change non-flatness by a weight that is set so that the weight increases the closer the evaluation area in which the luminance change non-flatness is calculated, as an index value of the luminance change non-flatness, and may set a luminance change non-flatness whose index value exceeds a predetermined threshold value and is greater than the index value of the next calculated luminance change non-flatness as the luminance change non-flatness that indicates that the luminance change is not flat by a predetermined level or more.
[0013] The camera may be mounted on a moving body and capture an image of a surface on which the moving body travels as the target object. According to the above-mentioned movement amount measurement system, only the area including the evaluation area or adjacent to the evaluation area, which is located in the vertical direction of the evaluation area where the change in luminance is not flat by a predetermined level or more, is set as the tracked area. And, since the vertical direction is the main relative movement direction of the camera with respect to the target object, by setting the tracked area in this way, it is possible to prevent the area of the captured image corresponding to the image of the monochromatic part of the target object that is long in the movement direction, that is, the part where the change in luminance is flat, from being set as the tracked area. Therefore, it is possible to properly track the tracked area and properly measure the relative movement amount between the camera and the target object.
[0014] In addition, an evaluation area smaller than the size of the tracked area is used, and whether or not the area can be set as a tracked area is determined without checking the flatness of the overall brightness change of the N row x M column area that is a candidate for the tracked area, so that the tracked area can be set with little processing load or processing delay. Effect of the Invention
[0015] As described above, according to the present invention, even if the target object has a long, monochromatic part in the direction of movement, the relative amount of movement between the camera and the target object can be properly measured using an image of the target object captured by a camera. [Brief description of the drawings]
[0016] [Figure 1] 1 is a block diagram showing a configuration of a movement amount measuring system according to a first embodiment of the present invention. [Diagram 2] 1 is a diagram showing an application example of a movement measuring system according to a first embodiment of the present invention; [Diagram 3] 5A to 5C are diagrams illustrating an example of a tracking operation of the measurement apparatus according to the first embodiment of the present invention. [Figure 4] 5A to 5C are diagrams illustrating an example of an operation for calculating an actual distance conversion coefficient of the measurement apparatus according to the first embodiment of the present invention. [Diagram 5] 5A to 5C are diagrams illustrating a setting operation of a tracked area according to the first embodiment of the present invention. [Figure 6] 5A to 5C are diagrams illustrating a setting operation of a tracked area according to the first embodiment of the present invention. [Figure 7] 4 is a diagram illustrating an example of the configuration of a calculator of luminance transition non-flatness according to the first embodiment of the present invention. FIG. [Figure 8] 5A to 5C are diagrams illustrating a setting operation of a tracked area according to the first embodiment of the present invention. [Figure 9] 11 is a diagram showing a setting operation of a tracked area according to the second embodiment of the present invention. FIG. [Figure 10] 13A to 13C are diagrams illustrating a setting operation of a tracked area according to the third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] A first embodiment of the present invention will be described. FIG. 1 shows the configuration of a movement amount measuring system according to the first embodiment. As shown in the figure, the movement measurement system includes a measurement device 1 and a stereo camera 2. The measurement device 1 includes a tracked region setting unit 11 , a tracking processing unit 12 , a separation distance measuring unit 13 , a movement vector calculation unit 14 , an actual distance conversion coefficient calculation unit 15 , and a movement state calculation unit 16 . The stereo camera 2 includes two cameras: a first camera 21 and a second camera 22. The movement measurement system is a system that measures the relative movement between the object photographed by the stereo camera 2 and the stereo camera 2, and can be applied to measuring the movement as shown in Figures 2a and 2b, for example. 2a shows an application example in which the stereo camera 2 is fixed to a moving body (an automobile in the figure) and the amount of movement of the moving body relative to the road surface is measured with the road surface as the target object. In this case, the stereo camera 2 is arranged so that the first camera 21 and the second camera 22 capture the image of the road surface from directly above. The first camera 21 and the second camera 22 are arranged at a distance from each other in a direction perpendicular to the main movement direction MD (forward direction of the moving body) of the stereo camera 2 relative to the target object (road surface) and perpendicular to the optical axes of the first camera 21 and the second camera 22. The first camera 21 and the second camera 22 are arranged so that the upward direction of the captured image is the movement direction MD.
[0018] 2b shows an application example in which the stereo camera 2 is fixed to a reference coordinate system, and the amount of movement of an object surface (in the figure, the surface on which the conveyor belt carries the transported object) moving relative to the reference coordinate system is measured as the object. In this case, the stereo camera 2 is arranged so that the first camera 21 and the second camera 22 capture the object surface from directly above. The first camera 21 and the second camera 22 are arranged at a distance from each other in a direction perpendicular to the main movement direction MD (the direction opposite to the movement direction of the object surface relative to the reference coordinate system) of the stereo camera 2 relative to the object (object surface) and perpendicular to the optical axis of the first camera 21 and the second camera 22. The first camera 21 and the second camera 22 are arranged so that the upward direction of the captured image is the movement direction MD.
[0019] Therefore, the first camera 21 and the second camera 22 are arranged side by side in the direction that corresponds to the left-right direction of the captured image, the up-down direction of the captured image (the direction in which the rows are arranged) is parallel to the movement direction MD, and the left-right direction of the captured image (the direction in which the columns are arranged) is perpendicular to the movement direction MD on the target object.
[0020] Next, the tracking operation performed by the tracked area setting unit 11 and the tracking processing unit 12 of the measurement device 1 will be described. After the measurement operation of the measurement device 1 is started, the measurement operation is repeatedly executed intermittently. Here, as shown in FIG. 3a, the start time of the measurement operation is represented as t=1, and the execution time of the n-th measurement operation from the start time of the process is represented as t=n. Moreover, the image captured by first camera 21 at time t=n is represented as P(t=n). The tracked region setting unit 11 sets a tracked region A(t=i) of a fixed size in an image P(t=i) at each time point t=i (i is an arbitrary integer equal to or greater than 1). At each point in time t=i+1 (i is any integer equal to or greater than 1), the tracking processing unit 12 searches for an area in image P(t=i+1) that contains an image most similar to the image of tracked area A(t=i) in image P(t=i) by template matching processing, and sets this area as the tracking result area B(t=i+1). Furthermore, the tracked area setting unit 11 sets the initial tracked area A(t=1) at t=1 by a new tracked area setting operation. In the new tracked area setting operation at t=1, as shown in Fig. 3b, the tracked area A(t=1) is set so that the reference point Ag(t=1), which is the pixel position at the lower left of the tracked area A(t=1), is included in the reference point allowable area Q(t=1), which is a preset area in the image P(t=1) captured by the first camera 21.
[0021] Next, as shown in Figures 3c1 and c2, at each point in time t=i+1, the tracking processing unit 12 searches for an area in image P(t=i+1) that contains an image most similar to the image of tracked area A(t=i) in image P(t=i) as a tracking result area B(t=i+1), and calculates a movement vector VP(t=i+1) in image space that has its starting point as reference point Ag(t=i) of tracked area A(t=i) and its ending point as reference point Bg(t=i+1), which is the bottom left pixel position of the searched tracking result area B(t=i+1).
[0022] For the second or subsequent tracked area A(t=i+1), if the reference point Bg(t=i+1) of the tracked result area B(t=i+1) searched at t=i+1 is within the reference point allowable area Q(t=i+1) as shown in FIG. 3d1, the tracked area setting unit 11 sets the tracked area B(t=i+1) as it is as the tracked area A(t=i+1). On the other hand, if the reference point Bg(t=i+1) of the tracked result area B(t=i+1) is not within the reference point allowable area Q(t=i+1) as shown in FIG. 3d2, the tracked area A(t=i+1) is set by a new tracked area setting operation. In the new tracked area setting operation at t=i+1, the tracked area A(t=i+1) is set so that the reference point Ag(t=i+1) is included in the reference point allowable area Q(t=i+1). However, even if the reference point Bg(t=i+1) of the tracking result area B(t=i+1) is within the reference point allowable area Q(t=i+1), the tracked area A(t=i+1) may be set by the new tracked area setting operation.
[0023] Here, the reference point acceptable area Q(t=i+1) is set according to the movement vector VP(t=i+1) and the acceleration vector determined from the history of the movement vector VP so as to ensure that, at the next time t=i+2, the image of the part of the target object captured in the tracked area A(t=i+1), which is set so that the reference point Ag(t=i+1) is included in the reference point acceptable area Q(t=i+1), is included in the image P(t=i+2).
[0024] Next, the operation of calculating the actual distance conversion coefficient performed by the separation distance measuring unit 13 and the actual distance conversion coefficient calculating unit 15 of the measuring device 1 will be described. At time t=1, the separation distance measuring unit 13 calculates the distance ZA(t=1) in the optical axis direction from the first camera 21 and the second camera 22 constituting the stereo camera 2 to the position on the target object reflected in the center of the tracked area A(t=1). Also, at each time t=i+1, the separation distance measuring unit 13 calculates the distance ZA(t=i+1) in the optical axis direction from the first camera 21 and the second camera 22 to the position on the target object reflected in the center of the tracked area A(t=i+1) and the distance ZB(t=i+1) in the optical axis direction from the first camera 21 and the second camera 22 to the position on the target object reflected in the center of the tracking result area B(t=i+1).
[0025] The distance Z in the optical axis direction from the first camera 21 and the second camera 22 to a position on the target object can be calculated as follows. 4, F [mm] is the focal length between the first camera 21 and the second camera 22 of the stereo camera 2, and M [mm] is the base length, which is the distance in real space between the first camera 21 and the second camera 22. Also, Tg is the position on the object where the distance Z [mm] is measured, c1 is the left-right position in the image space where the position Tg is photographed by the first camera 21, and c2 is the left-right position in the image space where the position Tg is photographed by the second camera 22. Also, c11 is the position corresponding to c1 on the surface SF that is a focal length F away from the first camera 21 in the optical axis direction, and c21 is the position corresponding to c2 on the surface SF that is a focal length F away from the second camera 22 in the optical axis direction.
[0026] Further, xl [mm] is the distance in the left-right direction from the center position of the first camera 21 to c11, and xr [mm] is the distance in the left-right direction from the center position of the second camera 22 to c22. And if p=xl-xr, according to the principle of triangulation, the distance Z of the position Tg is, Z = (M × F) / p It is calculated by: If the size of the area on a surface projected onto one pixel of the first camera 21 at a focal distance F from the first camera 21 is S [mm / pixel], then when the coordinate in the image space of the first camera 21 changes by 1, the position of Tg changes by (Z / F) × S in the direction perpendicular to the distance Z. Therefore, using this relationship, the actual distance conversion coefficient calculation unit 15 calculates the actual distance conversion coefficient K(t=i+1) from ZA(t=i) and ZB(t=i+1) measured by the separation distance measurement unit 13 at each point in time t=i+1. The actual distance conversion coefficient K(t=i+1) is calculated by taking the approximate Z(t=i+1) as the average value of Z(t=i) and ZB(t=i+1), {Z(t=i)+ZB(t=i+1)} / 2, K(t=i+1)={Z(t=i+1) / F}×S It is calculated by:
[0027] Next, the movement vector calculation unit 14 of the measurement device 1 calculates, at each time point t=i+1, a relative movement vector V(t=i+1) of the target object with respect to the stereo camera 2, using the movement vector VP(t=i+1) in the image space and the real distance conversion coefficient K(t=i+1) calculated by the real distance conversion coefficient calculation unit 15, as follows: Calculate V(t=i+1)=K(t=i+1)×VP(t=i+1).
[0028] The movement state calculation unit 16 of the measurement device 1 calculates and outputs various movement states such as the relative movement speed, acceleration, and movement direction of the target object from the relative movement vector V(t=i+1) of the target object with respect to the stereo camera 2 calculated by the movement vector calculation unit 14 at each time point t=i+1. The movement state calculation unit 16 may further calculate and output the relative angle of the target object, etc., as a movement state from multiple distances in the image calculated by the separation distance measurement unit 13.
[0029] The following will explain in detail the new tracked area setting operation performed by the tracked area setting unit 11 as described above. As shown in FIG. 5a, the first camera 21 and the second camera 22 output the pixel data within each row sequentially from left to right and each row sequentially from bottom to top, thereby outputting each pixel data of the image in a raster scan pattern from the bottom left to the top right. Next, assuming that the time when the new tracked area setting operation is performed is t=j, when an image P(t=j) taken at t=j has a long monochrome area in the main movement direction MD, as shown in Figure 5b1, for each row of image P(t=j), the absolute value of the difference in brightness between each pixel in that row and the pixel to the right of that pixel is plotted in the order of the pixels arranged left and right, and the range of pixels in the long monochrome area in the movement direction MD becomes flat with values close to 0, as shown in Figure 5b2.
[0030] The range of pixels in each graph that is flat at values close to 0 is connected in the vertical direction to form a long, monochrome region NG in the direction of movement MD in image P(t=j) shown in Figure 5c. If tracked region A(t=j) is set to overlap with this region NG, many regions similar to tracking region A(t=j) will appear in image (t=j+1), increasing the possibility that the region in image P(t=j+1) that contains the same part of the target object as that captured in tracked region A(t=j) will not be searched correctly.
[0031] Therefore, in the new tracked area setting operation, in order to set the tracked area A(t=j) so as to overlap with area NG as little as possible while minimizing the processing volume and processing delay, the tracked area A(t=j) is set as follows. In the new tracked area setting operation performed by the tracked area setting unit 11, first, as shown in Fig. 6a, a reference point allowable area Q(t=j) is set on the image P(t=j), and then, as shown in Fig. 6b, a placement possible area EQ(t=j) is set, which is a range that can include the tracked area A(t=j) with a reference point Ag(t=j) located within the reference point allowable area Q(t=j). Then, as shown in Fig. 6c, the placement possible area EQ(t=j) is divided into a number of small areas SS1-SS8. Each small area is the same size as the tracked area A(t=j).
[0032] Then, the degree of luminance change non-flatness of the evaluation area DA set for each of the small areas SS1 to SS8 is calculated in sequence. The evaluation area DA of each small area is the bottom row within the small area, and the luminance transition non-flatness of the evaluation area DA is the sum of the absolute values of the luminance differences between each pixel in the evaluation area DA and the pixel to the right of that pixel. However, since the rightmost pixel of the evaluation area DA has no adjacent pixel to the right of that pixel, the absolute value of the difference is not calculated, and it is not included in the sum that is the luminance transition non-flatness.
[0033] Here, the degree of luminance change non-flatness of each evaluation area DA represents the degree to which the luminance change within that evaluation area DA is not flat. Moreover, the degree of luminance change non-flatness of each evaluation area DA is calculated as soon as the pixel data of that evaluation area DA is output from the first camera 21. That is, the luminance transition non-flatness of the evaluation area DA is calculated using, for example, a calculator shown in FIG. As shown in the figure, this calculator includes a pixel delay unit 71 that delays the pixel data output from the first camera 21 by one pixel and outputs it as left pixel data, a calculator 72 that treats the pixel data output from the first camera 21 as right pixel data and outputs the absolute value of the difference between the luminance value represented by the left pixel data and the luminance value represented by the right pixel data, a plurality of sequential delay elements 73 that sequentially delay the output of the calculator 72 by one pixel, and a sum calculation unit 74 that calculates and outputs the sum of the output of the calculator 72 and the output of each sequential delay element 73.
[0034] The number of sequential delay elements 73 is the number of pixels in the evaluation area DA minus two. Then, for each evaluation area DA, the sum calculated and output by sum calculation section 74 immediately after the pixel data of the rightmost pixel in that evaluation area DA is output becomes the luminance transition non-flatness of that evaluation area DA. Then, when an evaluation area DA appears in which the degree of luminance change non-flatness meets a predetermined pass criterion, the small area in which the evaluation area DA is set is set as the tracked area A(t=j). The passing criteria will be one of the following A, B, C, or D. A: The evaluation value is the degree of unevenness of the luminance change, and if the evaluation value is greater than a predetermined threshold value Th, it is considered to be a pass. B: The weight W is set so that the closer the evaluation area DA is to the center of the image P(t=j), the larger the value is. The evaluation value is multiplied by the degree of unevenness of the luminance change, and the evaluation value is considered to be a pass if it is greater than a predetermined threshold value Th. C: The evaluation value (brightness change non-flatness) calculated based on the criteria in A is acceptable, and the next calculated evaluation value is smaller than the current calculated evaluation value. D: The evaluation value calculated based on the criteria in B (luminance change non-flatness multiplied by weight W) is considered to be acceptable, and the next calculated evaluation value is considered to be acceptable if it is smaller than the evaluation value calculated this time. Therefore, when the luminance change non-flatness of the evaluation area DA set for the small areas SS1-SS4 is obtained as shown in Figure 8a, when the pass criterion of A is applied, the evaluation value (luminance change non-flatness) that first appears and exceeds the threshold value Th is the evaluation value of the evaluation area DA of the small area SS1, so as soon as the pixel data of the evaluation area DA of the small area SS1 is output, the small area SS1 is set as the tracked area A (t = j).
[0035] On the other hand, when the weight W is set so that the value is larger for evaluation area DA closer to the center of image P(t=j) as shown in Figure 8b, and the pass criterion B is applied, the evaluation value (the value obtained by multiplying the weight W by the brightness change non-flatness) of evaluation area DA of small areas SS1-SS4 is obtained as shown in Figure 8c, and the evaluation value of evaluation area DA of small area SS1 does not exceed threshold value Th, and the first evaluation value that exceeds threshold value Th becomes the evaluation value of evaluation area DA of small area SS2, so as soon as the pixel data of evaluation area DA of small area SS2 is output, small area SS2 is set as the tracked area A(t=j).
[0036] The weight W is set to a larger value for the evaluation area DA that is closer to the center of the image P(t=j) in order to preferentially set the central part of the image P(t=j), where image distortion due to lenses, etc. is small, as the tracked area A(t=j). Furthermore, when the pass criteria of C and D are applied, as shown in Figs. 8a and 8c, the evaluation area DA that first appears and whose evaluation value exceeds the threshold value Th and whose evaluation value of the evaluation area DA calculated next time is smaller is the evaluation area DA of the small area SS3. This is found out when the evaluation value of the evaluation area DA of the small area SS4 to the right of the evaluation area DA of the small area SS3 is calculated, so as soon as the pixel data of the small area SS4 is output, the small area SS3 is set as the tracked area A(t=j). The new tracked area setting operation has been described above.
[0037] In addition, in the new tracked area setting operation, an area having the same horizontal extent as the small area in the row one line below the small area SS1-SS8 may be set as the evaluation area DA for each small area, as shown in Fig. 6d. Setting an area in this manner is equivalent to setting the bottommost row of the small areas SS1-SS8 obtained by dividing the range obtained by shifting the arrangement possible range EQ(t=j) one line down as the evaluation area DA instead of the arrangement possible range EQ(t=j), and setting the area obtained by shifting the small area for which the evaluation area DA that satisfies the pass criterion is calculated one line up as the tracked area A(t=j).
[0038] As a result, after the generation of an evaluation area DA in which a brightness change non-flatness that satisfies the pass criterion has been calculated, the pixels output from the first camera 21 can be used as pixels of the tracked area A(t=j), thereby eliminating the need to retain the pixels of the evaluation area DA for use as pixels of the tracked area A(t=j).
[0039] In addition, multiple consecutive or separated rows within each small region may be set as the evaluation area DA for each small region. In this case, the evaluation area DA may include an area with the same horizontal extent as a small region in the row immediately below the small region. In these cases, the luminance transition non-flatness of the evaluation area DA is the sum of the absolute values of the differences between the luminance of each pixel and the luminance of the pixel adjacent to the right of the pixel in question, calculated for each row in the evaluation area DA.
[0040] As described above, according to the first embodiment, only the area including the evaluation area DA or adjacent to the evaluation area above the evaluation area DA, in which the change in luminance is not flat by a predetermined level or more, and having the same range in the left-right direction as the evaluation area DA, is set as the tracked area A(t=j). Since the up-down direction is the main relative movement direction MD of the first camera 21 with respect to the target object, by setting the tracked area A(t=j) in this manner, a monochromatic part in which the target object is long in the movement direction MD, that is, an area in the captured image P(t=j) where the change in luminance is flat, is excluded from being set as the tracked area A(t=j). Therefore, it is guaranteed that the tracking of the tracked area P(t=j) can be properly performed and the relative movement amount between the stereo camera 2 and the target object can be properly measured.
[0041] In addition, an evaluation area DA smaller than the tracked area A(t=j) is used to determine whether or not a candidate area for the tracked area can be set as the tracked area A(t=j) without checking the flatness of the overall brightness change of the area, so that the tracked area A(t=j) can be set with little processing load or processing delay.
[0042] Next, a second embodiment will be described. The second embodiment differs from the first embodiment only in the new tracked area setting operation performed by the tracked area setting unit 11. In the new tracked area setting operation of the second embodiment, as shown in Figure 9a, the coordinate of the left end of the evaluation area DA is moved in a raster scan manner from the bottom left to the top right within the reference point allowable area Q (t = j) while obtaining the non-flatness of the brightness change of the evaluation area DA. The evaluation area DA is an area of one row and has the same number of columns as the tracked area A(t=j). Furthermore, within each row, the coordinate of the left end of the evaluation area DA moves from the leftmost area to the rightmost area of the reference point allowable area Q(t=j) so that the area adjacent to the right of the current evaluation area DA becomes the next evaluation area DA. Furthermore, the row to which the coordinate of the left end of the evaluation area DA moves within the row changes sequentially from the bottommost row to the topmost row of the reference point allowable area Q(t=j). Furthermore, the evaluation area DA moves each time pixel data for the number of columns of the evaluation area DA is output from the first camera 21.
[0043] In the new tracked area setting operation, the luminance change non-flatness of each evaluation area DA is calculated by a calculator or the like shown in Fig. 7 as soon as the pixel data of the evaluation area DA is output from the first camera 21. If a luminance change non-flatness that satisfies the pass criterion appears, as shown in Fig. 9b, an area having the same size in the vertical and horizontal directions as the area to be set as the tracked area A(t=j) is set as the tracked area A(t=j), with the evaluation area DA for which the luminance change non-flatness was calculated as the bottom row.
[0044] As the pass criterion, the AD criterion shown in the first embodiment can be applied. In addition, in the new tracked area setting operation, the area to which the evaluation area DA is moved is the reference point acceptance area Q(t=j) shifted one row downward, and if a brightness change non-flatness that satisfies the pass criterion is calculated, an area of the same size in the vertical and horizontal directions as the area to be set in the tracked area A(t=j), adjacent to the upper side of the evaluation area DA where the brightness change non-flatness was calculated, may be set as the tracked area A(t=j), as shown in Figure 9c.
[0045] According to the new tracked area setting operation of the second embodiment, the tracked area A(t=j) can be set with a higher degree of freedom in the vertical position than when the tracked area A(t=j) is set by the new tracked area setting operation of the first embodiment. Next, a third embodiment will be described. The third embodiment differs from the second embodiment only in the way in which the tracked area setting unit 11 moves the evaluation area DA in the new tracked area setting operation. That is, in the third embodiment, as in the second embodiment, the luminance change non-flatness of the evaluation area DA is obtained while moving the coordinate of the left end of the evaluation area DA in a raster scan manner from the lower left to the upper right in the reference point allowable area Q(t=j). However, in the third embodiment, as shown in FIG. 10, each time one pixel data is output from the first camera 21, the coordinate of the left end of the evaluation area DA is sequentially moved to the right by one pixel (one column) in each row of the reference point allowable area Q(t=j) from the leftmost area to the rightmost area of the tracked area A(t=j). Note that the coordinate of the left end of the evaluation area DA does not move for the output of pixel data between the pixel data at the left end of the captured image P(t=j) and the second pixel data from the right of the leftmost evaluation area DA. The coordinate of the left end of the evaluation area DA does not move for the output of pixel data to the right of the rightmost evaluation area DA.
[0046] In the new tracked area setting operation, the luminance change non-flatness of each evaluation area DA is calculated by a calculator or the like shown in Fig. 7 as soon as the pixel data of the evaluation area DA is output from the first camera 21. When a luminance change non-flatness that satisfies the pass criterion appears, as shown in Fig. 9b, an area having the same size in the vertical and horizontal directions as the area to be set as the tracked area A(t=j) is set as the tracked area A(t=j), with the evaluation area DA for which the luminance change non-flatness was calculated being the lowest row.
[0047] In the third embodiment, too, the area in which the coordinate of the left end of the evaluation area DA is moved is defined as the area shifted one row downward from the reference point acceptance area Q(t=j), and if a brightness change non-flatness that satisfies the pass criterion is calculated, then as shown in Figure 9c, an area adjacent to the upper side of the evaluation area DA in which the brightness change non-flatness was calculated and having the same vertical, horizontal and lateral sizes as the area set in the tracked area A(t=j) may be set as the tracked area A(t=j).
[0048] According to the new tracked area setting operation of the third embodiment, the tracked area A(t=j) can be set with a higher degree of freedom in the left-right position than when the tracked area A(t=j) is set by the new tracked area setting operation of the second embodiment. The embodiment of the present invention has been described above. In the above embodiment, the distance ZA(t=i) and the distance ZB(t=i+1) calculated by the separation distance measuring unit 13 using the stereo camera 2 may be calculated using another device such as a laser range finder. Also, the roles of the first camera 21 and the second camera 22 in the above embodiment may be interchanged. [Explanation of symbols]
[0049] 1...measuring device, 2...stereo camera, 11...tracked area setting unit, 12...tracking processing unit, 13...separation distance measurement unit, 14...movement vector calculation unit, 15...actual distance conversion coefficient calculation unit, 16...movement state calculation unit, 21...first camera, 22...second camera, 71...pixel delay unit, 72...calculator, 73...delay element, 74...sum calculation unit.
Claims
1. A movement amount measurement system that measures a relative movement amount of a camera with respect to a target object from a captured image that is an image of the target object captured by a camera, a tracking area setting means for setting a tracking area of N rows by M columns (where N>1, M>1) in the current captured image, with a direction in the captured image corresponding to a main relative movement direction of the camera with respect to the target being a pixel row arrangement direction and a vertical direction, and with a direction perpendicular to the row arrangement direction in the captured image being a pixel column arrangement direction and a horizontal direction; a search means for searching, as a tracking result area, an area in a current captured image that matches the tracked area previously set by the tracked area setting means; a movement amount calculation means for converting, in each measurement, a movement amount from a tracked area previously set by the tracked area setting means to a tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; the tracked region setting means sequentially calculates, for each evaluation region set in a plurality of small regions of N rows by M columns obtained by dividing a predetermined range in the current captured image, a degree to which the change in luminance of the evaluation region is not flat as a luminance change non-flatness degree, and sets, as the tracked region, the small region set in which the evaluation region has been set with a luminance change non-flatness degree that indicates that the change in luminance is not flat by a predetermined level or more; A movement measurement system characterized in that an evaluation area set for the small area consists of less than N rows, including at least one of a portion of rows within the small area and an area of 1 row by M columns adjacent to the small area in the vertical direction.
2. A movement amount measurement system that measures a relative movement amount of a camera with respect to a target object from a captured image that is an image of the target object captured by a camera, a tracking area setting means for setting a tracking area of N rows by M columns (where N>1, M>1) in the current captured image, with a direction in the captured image corresponding to a main relative movement direction of the camera with respect to the target being a pixel row arrangement direction and a vertical direction, and with a direction perpendicular to the row arrangement direction in the captured image being a pixel column arrangement direction and a horizontal direction; a search means for searching, as a tracking result area, an area in a current captured image that matches the tracked area previously set by the tracked area setting means; a movement amount calculation means for converting, in each measurement, a movement amount from a tracked area previously set by the tracked area setting means to a tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; The movement amount measurement system is characterized in that the tracked area setting means sets, as the evaluation area, an area corresponding to the bottommost row of a plurality of small areas of N rows x M columns obtained by dividing a predetermined range in the currently captured image, and sequentially calculates the degree to which the luminance change of each evaluation area is not flat as a luminance change non-flatness, and sets, as the tracked area, a small area having the evaluation area in the bottommost row for which a luminance change non-flatness indicating that the luminance change is not flat by a predetermined level or more, or an area shifted one row up from the small area having the evaluation area in the bottommost row for which the luminance change non-flatness was calculated, as the tracked area.
3. A movement amount measurement system that measures a relative movement amount of a camera with respect to a target object from a captured image that is an image of the target object captured by a camera, a tracking area setting means for setting a tracking area of N rows by M columns (where N>1, M>1) in the current captured image, with a direction in the captured image corresponding to a main relative movement direction of the camera with respect to the target being a pixel row arrangement direction and a vertical direction, and with a direction perpendicular to the row arrangement direction in the captured image being a pixel column arrangement direction and a horizontal direction; a search means for searching, as a tracking result area, an area in a current captured image that matches the tracked area previously set by the tracked area setting means; a movement amount calculation means for converting, in each measurement, a movement amount from a tracked area previously set by the tracked area setting means to a tracked result area currently searched by the search means into a relative movement amount of the camera with respect to the object; The tracked area setting means moves through an evaluation area, which is an area of 1 row by M columns, within a predetermined range in the current captured image, and calculates the degree to which the change in luminance of the evaluation area is not flat as a luminance change non-flatness degree, and sets as the tracked area an N row by M column area having as the bottom row the evaluation area for which a luminance change non-flatness degree indicating that the change in luminance is not flat by a predetermined level or more, or an N row by M column area adjacent to and above the evaluation area for which the luminance change non-flatness degree was calculated.
4. The movement amount measuring system according to claim 3, the camera sequentially outputs pixels of each row of the captured image from left to right and sequentially outputs each row of the captured image from bottom to top; The tracked area setting means moves from the leftmost area to the rightmost area within each row of the specified range so that the area adjacent to the right of the current evaluation area becomes the next evaluation area, and calculates a luminance change non-flatness of the evaluation area while moving the evaluation area so that the row through which the evaluation area moves changes sequentially from the bottommost row to the topmost row in the specified range, and when a luminance change non-flatness indicating that the luminance change is not flat by a predetermined level or more appears, sets as the tracked area an N row x M column area with the evaluation area in which the luminance change non-flatness was calculated as the bottommost row, or an N row x M column area adjacent above the evaluation area in which the luminance change non-flatness was calculated.
5. The movement amount measuring system according to claim 3, the camera sequentially outputs pixels of each row of the captured image from left to right and sequentially outputs each row of the captured image from bottom to top; The tracked area setting means moves from the leftmost area to the rightmost area within each row of the specified range so that the area shifted one column to the right of the current evaluation area becomes the next evaluation area, and calculates a luminance change non-flatness of the evaluation area while moving the evaluation area so that the row through which the evaluation area moves changes sequentially from the bottommost row to the topmost row in the specified range, and when a luminance change non-flatness indicating that the luminance change is not flat by a predetermined level or more appears, sets as the tracked area an N row x M column area with the evaluation area in which the luminance change non-flatness was calculated as the bottommost row, or an N row x M column area adjacent to the evaluation area in which the luminance change non-flatness was calculated.
6. A movement amount measuring system according to claim 1, 2, 3, 4 or 5, The tracking area setting means calculates the sum of absolute values of the luminance differences between each pixel in the evaluation area and the pixels in the evaluation area adjacent to the pixel in the column arrangement direction as the luminance change non-flatness of the evaluation area.
7. A movement amount measuring system according to claim 1, 2, 3, 4, 5 or 6, The movement amount measuring system is characterized in that the tracked area setting means sets a luminance change non-flatness degree exceeding a predetermined threshold value as a luminance change non-flatness degree indicating that the luminance change is not flat by more than a predetermined level.
8. A movement amount measuring system according to claim 1, 2, 3, 4, 5 or 6, The movement amount measurement system is characterized in that the tracking area setting means sets a luminance change non-flatness degree that exceeds a predetermined threshold value, the luminance change non-flatness degree being a luminance change non-flatness degree that indicates that the luminance change is not flat by a predetermined level or more, when the luminance change non-flatness degree is multiplied by a weight that is set to be larger the closer the evaluation area is to the center of the captured image.
9. A movement amount measuring system according to claim 1, 2, 3, 4, 5 or 6, the tracked area setting means sets, as an index value of the luminance change non-flatness, the luminance change non-flatness or a value obtained by multiplying the luminance change non-flatness by a weight that is set so that the weight increases as the evaluation area in which the luminance change non-flatness is calculated is closer to the center of the captured image, and sets, as a luminance change non-flatness indicating that the change in luminance is not flat by a predetermined level or more, a luminance change non-flatness in which the index value exceeds a predetermined threshold value and is greater than the index value of the next calculated luminance change non-flatness.
10. A movement amount measuring system according to claim 1, 2, 3, 4, 5, 6, 7, 8 or 9, The movement amount measuring system is characterized in that the camera is mounted on a moving object and captures an image of a surface on which the moving object moves as the target object.
Citation Information
Patent Citations
Traveling object sensing device
JP1999284997A
Car outside monitoring device
JP2001088636A
Moving object detector
JP2006047091A
Moving object tracking apparatus
JP2006050489A
Moving picture conversion apparatus and moving picture conversion method, and computer program
JP2006345445A
