Object tracking system

Abstract

To improve accuracy of object tracking in a continuous image.SOLUTION: In a continuous image etc., obtained by photographing a sign by a camera mounted on a vehicle or other movable bodies, processing of tracking the sign or other target objects photographed in each of constitutional images constituting the continuous image is executed by a following method. Initially, the method extracts a candidate of an object which is a tracking target by image processing etc., while the method estimates a position in the constitutional image in the next time by using a movement model, on the basis of the position of the object in the constitutional image in the preceding time. At this time, the method reflects a preset parameter in the movement model, according to the position in the preceding time. Then, the method identifies which of the extracted candidates is the same as the object in the preceding time, by matching the candidate with the estimated position. Thus, the method can improve accuracy of object tracking.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an apparatus using image data.

Background Art

[0002] There is a technique for predicting the position of an object in the next frame by means such as a Kalman filter or a particle filter based on the position of the object in the previous frame and its motion model (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to more effectively utilize image data.

Means for Solving the Problems

[0005] The present invention is an object tracking system for tracking an object in a continuous image composed of a plurality of constituent images, an object extraction unit that extracts candidates for the object in the constituent image, based on the position of the object in the constituent image at a previous time, by a predetermined movement model At the following time a position estimation unit that estimates the position of the object in the constituent image, A storage unit for storing parameters used in the aforementioned movement model, based on the candidate extracted by the object extraction unit and the position estimated by the position estimation unit At the following time within the constituent image of and a specifying unit that specifies the object, The storage unit stores the parameters for each cell formed by dividing the constituent image into a plurality of cells. The position estimation unit sets the parameter corresponding to the cell to which the position of the object in the constituent image at the previous time point belongs in the movement model, and estimates the position of the object in the constituent image at the next time point.It can be configured as an object tracking system.

[0006] In other embodiments, the present invention is: In a sequence of images composed of multiple constituent images, in the constituent image at the previous time point things Body position based on, In a given mobility model Therefore The position of the object in the constituent image at the next time point is estimated. In this regard, the parameters of the aforementioned movement model A parameter setting system for setting parameters, The aforementioned constituent image is divided into multiple cells, The average value of the amount of movement of the object within the constituent image from the previous time to the next time is Each cell Calculate and based on this ,before The parameters used in the movement model are Each of the aforementioned cells It can be configured as a parameter setting system equipped with a parameter setting unit for configuring parameters.

[0007] In one embodiment of the present invention, in addition to the above-described embodiment as an object tracking system or parameter setting system, it may also be configured as a method for tracking an object or setting parameters using a computer. Furthermore, it may be configured as a computer program that implements these processes, and a computer-readable recording medium on which the computer program is stored. [Brief explanation of the drawing]

[0008] [Figure 1] This is an explanatory diagram illustrating the overview of object tracking. [Figure 2] This is an explanatory diagram showing the configuration of the object tracking system and the parameter setting system. [Figure 3] This is an explanatory diagram showing an object tracking method. [Figure 4] This is a flowchart for object tracking. [Figure 5] This is a flowchart for the movement model setup process. [Figure 6] This is a flowchart of the grid parameter creation process. [Figure 7]It is an explanatory diagram showing an example of grid division. [Figure 8] It is an explanatory diagram showing an example of setting grid parameters. [Figure 9] It is an explanatory diagram showing an example of object tracking.

Embodiment for Carrying Out the Invention

[0009] As an embodiment of the present invention, taking an object tracking system that tracks road signs, signals, and other features as objects in continuous images taken by a camera fixed to a vehicle as an example, it will be described in the following order. A. Overview of Object Tracking: B. System Configuration: C. Object Tracking Process: D. Grid Parameter Creation Process: E. Effects and Variation Examples:

[0010] A. Overview of Object Tracking: FIG. 1 is an explanatory diagram showing the overview of object tracking. First, in this embodiment, it will be described what continuous images are targeted for object tracking.

[0011] FIG. 1(a) shows the situation of taking continuous images. A camera is mounted on the roof of the vehicle, and images are taken while the vehicle is moving. The images to be taken may be a video, or may be still images taken intermittently like the imaging positions P1 to P4 shown in the figure. In the figure, it is assumed that the intervals D between the imaging positions P1 to P4 are the same, but the intervals may be different. Also, in addition to taking images at a predetermined interval, it may be a moving image taken at a predetermined time interval. In the case of taking a moving image, frame images at a predetermined interval or time interval may be extracted for object tracking.

[0012] In this embodiment, a sequence of multiple images arranged in chronological order will be referred to as a continuous image, and each individual still image or frame image constituting the continuous image will be referred to as a constituent image. The time interval between constituent images can be determined arbitrarily, but the present invention is useful when the amount of movement of an object between constituent images is relatively large. For example, assuming the case of taking images while moving in a vehicle, it is useful for analyzing images taken at a period of approximately 1 to 2 images per second or more, although this depends on the vehicle's speed, the relative distance to the object, etc.

[0013] Objects (OBJ) that are the subject of filming and tracking include geographical features such as signs, signals, and billboards. Objects may also be things drawn on the ground, such as stop lines and pedestrian crossings. Other objects may include structures such as bridges and buildings, and trees. Furthermore, objects are not necessarily limited to fixed objects; moving objects such as people, animals, and other vehicles may also be considered objects. Furthermore, the object OBJ being tracked is not limited to one; multiple objects can be targeted.

[0014] In this embodiment, we will explain using an image of an object (OBJ) captured by a camera fixed to a vehicle as an example, but instead of a vehicle, a railway, drone, ship, aircraft, or various other mobile objects can be used. Furthermore, while the embodiment shows an example where the vehicle moves in a straight line along the road, it may also move along curves or make right or left turns. In addition, although the embodiment targets an object OBJ that is fixed, the present invention can also be used when the moving body is fixed and the object OBJ is moving, or when both are moving. However, this embodiment assumes that the relative movements of the vehicle and the object OBJ are qualitatively known within a predetermined error range.

[0015] Figure 1(b) shows the movement of an object OBJ within a series of images superimposed on four images that make up a sequence of images taken by a vehicle. At the time of shooting at position P1, the object OBJ is far away, so it is captured in the image at the position and size of object OBJ1. Subsequently, as the vehicle moves to shooting positions P2 to P4, the relative distance between the vehicle and the object OBJ decreases, causing the object OBJ to move in position and increase in size within the image, as shown by objects OBJ2 to OBJ4. The movement speeds V12, V23, and V34 from object OBJ1 to object OBJ4 also increase.

[0016] Object tracking refers to identifying that objects OBJ1 to OBJ4 are identical across the four diagrammed images. The results of object tracking can be used to analyze the object's ground position and 3D shape. When dealing with moving objects, the results can also be used to analyze their movement trajectory.

[0017] B. System Configuration: Figure 2 is an explanatory diagram showing the configuration of the object tracking system and the parameter setting system. The object tracking system 10 is a system that reads continuous images and performs the object tracking described above. The parameter setting system 20 is a system for setting the parameters used for object tracking. The object tracking system 10 and the parameter setting system 20 are connected to the network N. Parameters set in the parameter setting system 20 are transmitted to the object tracking system 10 via the network N and used. Furthermore, the object tracking system 10 receives image data from client 1 connected to the same network and returns the results of the object tracking analysis. This configuration is merely an example, and the object tracking system 10 and the parameter setting system 20 may be configured as a single unit. Furthermore, the object tracking system 10 may directly input image data and output analysis results without going through the network N.

[0018] In this embodiment, the object tracking system 10 and the parameter setting system 20 are configured in software by installing computer programs that implement the various functions shown in the diagram on servers connected to the network N and equipped with a CPU and memory. Some or all of these functions may be configured in hardware. The following describes each function.

[0019] The functions provided by the object tracking system 10 are as follows: The input / output unit 11 performs various inputs and outputs necessary for object tracking. Examples include reading sequential image data, specifying the object to be tracked, and outputting the analysis results of object tracking. The position estimation unit 12 is one of the processes for object tracking, and estimates the position of an object in the constituent image at the next time step based on the position of the object in the constituent image at the previous time step. The object extraction unit 13 extracts candidate objects to be tracked by image processing of the constituent images. The identification unit 14 performs a process to determine which of the extracted candidates corresponds to the object, based on the estimation result of the object's position. The movement model setting unit 15 sets the movement model that the position estimation unit 12 uses to estimate the position of an object. The grid parameter storage unit 16 is a database that stores the parameters used in the movement model described above. In this embodiment, as will be described later, the constituent image is divided into multiple cells, and parameters are set for each cell. The movement model setting unit 15 determines which cell an object belongs to in the constituent image, obtains the parameters, and sets the movement model based on these parameters.

[0020] The parameter setting system 20 is a system for setting parameters stored in the grid parameter storage unit 16 of the object tracking system 10. The functions provided by the parameter setting system 20 are as follows: The input / output unit 21 performs various inputs and outputs necessary for setting parameters. Examples include reading sequential image data, specifying the object to be tracked, and outputting the parameter setting results. The preprocessing unit 22 performs preprocessing to set parameters. For example, this could involve removing constituent images from the input sequence of images that are unsuitable for setting the parameters. The parameter setting unit 24 sets the parameters. The grid parameter storage unit 23 stores the set parameters.

[0021] The illustrated configurations for both the object tracking system 10 and the parameter setting system 20 are merely examples; other functions may be added, and some functions may be omitted as needed.

[0022] C. Object tracking process: Figure 3 is an explanatory diagram illustrating the object tracking method. In this embodiment, object tracking is performed based on the extraction of candidate objects within the constituent image and the estimation of the object's position. The example in Figure 3 shows the case where four objects are tracked. In Figure 3, the squares (□) represent candidates A to E extracted from the constituent image through image processing. Although there are four objects to be tracked, five candidates have been extracted. It is acceptable for the number of objects to be tracked and the number of candidates to differ, including cases where the number of candidates is less than the number of objects to be tracked. In Figure 3, the dashed rectangles represent the positions a0 to d0 of the object within the compositional image at the previous time point. The rectangles marked with an "x" represent the estimated positions a1 to d1 of the object based on positions a0 to d0. In this way, matching is performed between candidates A to E and the estimated positions a1 to d1. In this embodiment, matching is performed so that the overall distance between the candidates and the estimated positions is minimized. For example, the Hungarian method can be used for matching. In other words, it is a method of selecting the one among various correspondences between candidates and estimated positions that minimizes the overall distance between them. In the situation shown in the figure, when position a1 is matched with candidate A, position b1 with candidate B, position c1 with candidate C, and position d1 with candidate D, the overall distance is La + Lb + Lc + Ld, which is the minimum of the other matches, so this matching is selected. As a result, candidates A to D are identified as the objects to be tracked, and candidate E is determined not to be the object to be tracked.

[0023] Next, we will explain the process for realizing object tracking using the method described in Figure 3. Figure 4 is a flowchart of the object tracking process. It shows the coordinated execution of each function of the object tracking system (see Figure 2). When processing begins, the object tracking system reads a sequence of images (step S10). Here, image data from multiple constituent images that make up the sequence are read. The object tracking system then extracts candidate objects from the initial compositional image and identifies the target object to be tracked (step S11). In this embodiment, since road signs, traffic lights, etc., are to be tracked, it is sufficient to extract parts of the compositional image that are recognized as circular, rectangular, or other shapes using pattern recognition or the like as candidates. The color and patterns of the image may also be taken into consideration. Various image processing techniques can be applied to extract candidate objects, and artificial intelligence-based image recognition may also be used. Furthermore, the identification of the target object may be performed automatically by the object tracking system using pattern recognition or artificial intelligence-based image recognition, or it may be specified manually by an operator. The target object can be one or multiple objects.

[0024] Once the object to be tracked is determined, the object tracking system extracts candidate objects from the following constituent image (step S12). Similar to the process in step S11, candidate extraction can be performed using various image processing techniques.

[0025] The object tracking system sets up a movement model for estimating the position of a target object in a constituent image through a movement model setting process (step S13), and uses this to estimate the position of the target object in the next constituent image (step S14). The details of the movement model setting process will be described later. If there are multiple target objects, the position estimation is performed for all of them. Position estimation can be performed in various ways, including methods that calculate based on a set movement model formula, and methods that use techniques such as Kalman filters and particle filters. In this example, a Kalman filter is used.

[0026] Thus, as explained in Figure 3, candidates extracted from the constituent image and the position estimation results are obtained, and the object tracking system matches these (step S15) to track the target object. Matching can be performed using any method, but for example, as explained in Figure 3, the Hungarian method can be used.

[0027] The object tracking system repeats the above process for all constituent images (step S16) until it has finished, and then terminates the object tracking process.

[0028] Figure 5 is a flowchart of the movement model setting process. This corresponds to the process in step S13 of the object tracking process (Figure 4), and is the process of setting up a movement model for estimating the position of the target object. In this embodiment, the movements of the vehicle and object OBJ are qualitatively known within a predetermined error range. For example, as shown earlier in Figure 1(a), in a series of images taken while the vehicle approaches the object in a straight line, it can be seen that the object moves approximately linearly from the center to the edge of the constituent image, as shown in Figure 1(b). Therefore, assuming such movement, the following equation (1) can be set as the movement model for each component of the 2D coordinate system. Next center coordinates = Previous center coordinates + Movement speed × Time ... (1) Similarly, when a vehicle makes other movements, such as turning a corner, a movement model can be set up to correspond to those movements.

[0029] When this process is initiated, the object tracking system obtains the position coordinates of the target object in the compositional image at the previous time (step S20). In this embodiment, parameters are pre-set for each cell into which the constituent image is divided, and stored in the grid parameter storage unit (see Figure 2). Therefore, the object tracking system determines which cell the target object belongs to based on the position coordinates obtained in step S20, and retrieves the parameters set for that cell (step S21). In the example shown in the figure, parameters 1.3, which are assigned to the cell where the target object P is located, will be set.

[0030] The object tracking system then reflects the set parameters in the object's movement model (step S22), and terminates the movement model setting process. In this embodiment, the parameters are reflected by multiplying the movement speed by the set parameters in equation (1) above. Various methods are possible for reflecting the set parameters in the movement model, depending on the content of the movement model and parameters. As explained earlier, the resulting movement model is used to estimate the position of the target object.

[0031] Here, we will explain the significance of the movement model and grid parameters. As in this embodiment, when taking a photograph while approaching an object, the object moves towards the edge of the compositional image as the approach increases, and its speed of movement also increases. Therefore, in order to reflect this change in speed of movement in the movement model, a method can be considered in which the speed of movement is linearly increased over time in the form of speed × coefficient. However, as the shooting period for consecutive images lengthens, the amount of object movement between constituent images increases, and therefore, changes in movement speed within the constituent images cannot be adequately reflected by a linear increase. Consequently, in a movement model that linearly increases the movement speed, the error in position estimation increased as the object approached the edge of the constituent image. In this embodiment, object tracking is performed by matching candidates extracted from the constituent images with the position estimation results. Therefore, if the error in the position estimation results becomes large, errors occur in the matching, leading to incorrect object tracking results. This problem could not be adequately resolved even by changing the change in movement speed using a mathematical formula (e.g., a function) that represents a non-linear object movement model so that it changes with elapsed time.

[0032] In this embodiment, to solve the aforementioned problem, the movement speed is changed according to the position of the object in the constituent image. In other words, while conventional methods changed the movement speed according to the elapsed time, this method focuses on the position of the object in the constituent image and changes the movement speed according to the position. This is based on the idea that if the movement of the object in the constituent image is known within a predetermined error range, the change in movement speed depends more on the position in the constituent image than on the elapsed time since the start of the analysis. As a simpler method to implement this approach, in this embodiment, the constituent image is divided into cells, parameters are set for each cell, and the movement speed is changed based on these parameters.

[0033] As described later, the parameters can be set to values ​​that allow for an appropriate movement model to be obtained based on images taken while approaching the object. Therefore, if the images are taken under the same conditions, it is possible to improve the accuracy of position estimation by using the parameters set in this way.

[0034] Furthermore, in this embodiment, the above method is useful precisely because it is sufficient to estimate the object's position with enough accuracy to avoid errors in matching it with candidates extracted from the constituent image. If it were necessary to estimate the object's position precisely using a method other than this, the object would not necessarily move under exactly the same conditions as the sequential images used to pre-set the grid parameters, and therefore, even using grid parameters would not necessarily eliminate the errors completely. However, in this embodiment, the method involves tracking the object by matching it with candidates, and the requirement for accuracy in position estimation is not very high, making the grid parameter method simple and highly effective.

[0035] In the object tracking process described above (Figure 4), the target object was identified by first extracting candidates from the constituent image (step S12), then estimating their positions (step S14), and finally matching the two (step S15). Alternatively, the object tracking process can be performed by changing the order of each step. For example, first, the position is estimated (steps S13, S14), and then candidates are extracted using the estimated position (step S12). In this case, the candidate extraction can be limited to the vicinity (within a predetermined range) of the estimated position. The extraction range can be predetermined, or it can be set based on the size of the target object, for example, by doubling the size of the target object in both length and width at the previous time point. Then, objects superimposed on the estimated position are identified as matched (step S15). This method simplifies the processes of candidate extraction (step S12) and matching (step S15). In other words, the object extraction unit 13 may extract candidate objects within the constituent image before the position estimation unit 12 estimates the position of the object, or it may extract candidate objects within a predetermined range of the estimated position of the object after the position estimation unit 12 has estimated the position of the object.

[0036] D. Grid parameter creation process: Figure 6 is a flowchart of the grid parameter creation process. This process is performed by the parameter setting system. When processing begins, the parameter setting system reads the image data of the sequence of images (step S30). The image data read here is data that has already undergone object tracking. The results of object tracking are represented for each constituent image by a combination of an ID representing the target object and the 2D coordinate values ​​within that constituent image. The data representing the results may be integrated with the image data or read as a separate list. Object tracking may be performed by the operator specifying an object in each constituent image, or by using a sequence of images from which the object tracking system has obtained results without error.

[0037] Furthermore, it is preferable that the sequential images used be taken under the same shooting conditions as the sequential images that are the target of object tracking. For example, when performing object tracking using sequential images taken while moving in a straight line along a road, it is preferable to use sequential images taken while moving in a straight line in the parameter settings as well. Similar shooting conditions mean that the conditions match to the extent that the movement of the object in the constituent images is qualitatively approximated within a predetermined error range, and it is not necessary for the movement speed or the positional relationship between the object and the vehicle to match precisely. It is preferable to ensure that the camera orientation is consistent during shooting. To verify that these conditions are met, it is also preferable to record the camera orientation when capturing a series of images. Furthermore, in order to set the parameters accurately, it is preferable to use a series of images taken at equal intervals, with the vehicle's movement speed remaining nearly constant during shooting. However, this does not mean that the parameters cannot be set if the vehicle's movement speed or shooting interval changes.

[0038] Next, the parameter setting system extracts target images (step S31). Target images refer to a sequence of images suitable for setting parameters. For example, images where the shooting interval has changed significantly from the previous time frame, images where the camera direction has changed abruptly due to sharp curves or steep slopes, and blurry images will be excluded. The diagram illustrates the processing steps. Suppose that, after reading constituent images F1 through F8, constituent images F2 and F7 are determined to be inappropriate for the reasons mentioned above. In this case, constituent images F1 and F8 will be missing the constituent image for the next or previous time point, leaving only constituent images F3 through F6 as a continuous image. Step S31 extracts a continuous image suitable for parameter setting in this manner.

[0039] Next, the parameter setting system extracts target objects from each component image (step S32). A target object refers to an object in a sequence of images suitable for parameter setting. For example, objects that are partially missing outside the component image or objects that are partially obscured by obstacles will be excluded. Objects that are clearly visible but for which object tracking has not been performed for any reason will also be excluded.

[0040] Next, an initial grid is set (step S33), and the constituent image is divided into multiple cells. The initial grid can be set to any shape depending on the shooting interval of the constituent image, the size of the target object, etc. In this embodiment, as will be described later, the constituent image is divided vertically and horizontally into four sections to form rectangular cells. It may also be divided into different numbers vertically and horizontally. Furthermore, the shape of the cells can also be arbitrarily determined, and they may be divided into triangles, etc.

[0041] Next, the parameter setting system calculates the average and variance of the movement of the target object for each cell (step S34). In this embodiment, since a series of images in which object tracking has been performed in advance are used, the movement of the object between the previous and next constituent images can be easily determined. This movement is then stored as the movement of the cell where the object in the previous constituent image is located. If this process is performed for all target objects and all constituent images, multiple calculation results for movement amounts will be stored for each cell. After that, the average and variance of these movement amounts can be calculated.

[0042] If the obtained variance exceeds a predetermined threshold (step S35), the parameter setting system divides the cell (step S36). The method of division is arbitrary. This process is performed for each cell. That is, cells whose variance is below the threshold do not need to be divided. For the divided cells, the average and variance of the movement amount are calculated for each divided cell (step S34). In this way, the cell division is repeated until the variance within the cell falls below a threshold (step S35). By doing this, the variation in the average amount of movement for all cells can be reduced, and the error in the set parameters can be suppressed. Furthermore, this process may result in uneven cell shapes. However, if the entire cell structure is to be uniform, it becomes necessary to divide cells even if the variance of the average movement amount is below a threshold, unnecessarily increasing the overall number of cells and parameters. This embodiment has the advantage of suppressing such waste by allowing uneven cells.

[0043] After the above process, the average value of the movement amount for each cell is determined, and the parameter setting system sets the parameters for each cell based on the result (step S37), and then terminates the grid parameter setting process.

[0044] There are various ways to set the parameters in step S37. For example, the average value of the cell movement amount represents the movement speed of the object within the constituent image for each cell. Therefore, when setting up a movement model that directly utilizes this movement speed, the average value of the cell movement amount can be used directly as a parameter. Alternatively, the average movement of each cell may be normalized by the average movement of any of the cells, and this normalized value may be set as a parameter. In this way, the movement model during object tracking can be used to obtain a new movement speed by multiplying the movement speed at the previous time by this parameter, for example. Furthermore, assuming the use of a movement model that changes the movement speed linearly, the movement speed may be corrected by multiplying it by an additional parameter. In addition, the parameters can be set in a manner appropriate to the content of the motion model used for position estimation. After setting the parameters, the object tracking process can be repeated on the sequential images read in step S30, and the parameters can be set through trial and error to ensure that the object tracking is performed correctly.

[0045] Figure 7 is an explanatory diagram showing an example of grid division. It corresponds to an example of the process in step S36 of the grid parameter setting process (Figure 6). Figure 7(a) shows an example of an initial grid. In this example, a grid divided into four sections vertically and horizontally is set up. Assume that the variance of the movement amount exceeds a threshold in the hatched cells. Figure 7(b) shows examples of how these cells can be divided. As shown in the upper right cell (4 columns, 1 row), a single cell can be divided vertically and horizontally into two parts, resulting in four cells a through d. Alternatively, as shown in the left cell (1 column, 2 rows), it can be divided diagonally into two cells e and f. Qualitatively, an object moves in the direction of arrow A in the figure, and the speed of movement also increases or decreases in this direction. Therefore, when dividing diagonally, it is preferable to divide along a diagonal line that intersects arrow A. As such, various methods of cell division are possible, but either of the two methods described above may be used, or both may be combined.

[0046] Figure 7(c) shows an example of parameter settings. In this example, the bottom row (row 4) is set to 1.0. When using a parameter multiplied by the speed of movement, this setting means that the parameter is set not to affect the speed of movement. In cases like this embodiment, where a vehicle is approaching an object and capturing continuous images, depending on the target object, the bottom row may be an area that is not important for object tracking, such as an area where the object is unlikely to come. In such cases, it is acceptable to set a parameter that does not affect the speed of movement in this way. Allowing such a setting can reduce the load on the parameter setting process.

[0047] Figure 8 is an explanatory diagram showing an example of grid parameter settings. Figure 8(a) is an example where the lower half is treated as a single cell, similar to the bottom row of Figure 7(c), and a parameter of 1.0 is set as a parameter that does not affect the movement speed. In the upper half, as shown in the figure, the parameter is set as a large cell near the center. This is because the movement speed of the object is relatively small near the center of the compositional image, and therefore errors are less likely to occur. Figure 8(b) is similar to Figure 7(c) in that some parameters, including the bottom row, are set to 1.0. Also, in the top row, the leftmost parameter is 1.3, while the rightmost is 1.2. Thus, grid parameters can be set asymmetrically. Figure 8(c) shows that the parameters set on the left are generally large, while the parameters set on the right are small. When a camera is mounted on a vehicle and taking pictures, if traffic is on the left, objects coming from the left appear to be closer to the camera and moving faster than objects coming from the right. Therefore, by setting the parameters to prioritize the left side in this way, it is possible to create a setting suitable for object tracking. Figure 8(d), like Figure 8(c), sets a large value on the left side, but also sets a larger value on the right side than in Figure 8(d). When a vehicle is traveling on a two-lane road, the possibility of an object appearing on the right side is low, so a setting like that in Figure 8(c) is acceptable. However, when traveling on a narrow road such as a one-lane road, there is a possibility of an object appearing on the right side, so the setting is adjusted accordingly.

[0048] In this way, grid parameters may be set considering the surrounding environment at the time of shooting. In addition, parameters may be set according to (1) the vehicle's heading angle, pitch angle, and roll angle, (2) the curvature and gradient of the road, (3) the image resolution and frame interval of the constituent images, (4) the vehicle's speed, (5) the camera mounting position and the vehicle's tire pressure, and (6) the type of target object. Furthermore, to accommodate situations such as the vehicle moving in reverse or tracking image frames in reverse order, separate parameters may be provided in addition to the parameters for forward movement. The parameters may be prepared by separating them into components, such as a parameter for the velocity in the width direction (X direction) of the image and a parameter for the velocity in the height direction (Y direction).

[0049] In this way, parameters set based on numerous conditions can be selectively used depending on the object tracking process, based on the conditions under which the sequential images were captured. This makes it possible to perform object tracking while suppressing the effects of errors caused by shooting conditions and other factors. Furthermore, multiple parameters may be used in combination. For example, when multiple elements coexist in shooting conditions, such as vehicle speed and road curvature, a parameter that takes both into account may be set by multiplying the parameters set for each element by weights and adding them together, and this parameter may be used.

[0050] E. Effects and variations: Based on the object tracking system and parameter setting system described above, it becomes possible to track objects with high accuracy, as shown below.

[0051] Figure 9 is an explanatory diagram illustrating an example of object tracking. Figure 9(b) in the center shows an example of the constituent image at the previous time point. It shows an example of tracking three target objects P, Q, and R. In the figure, the markers enclosed in squares represent candidates extracted from the constituent image. The positions p0, q0, and r0, marked with an "x" in a square, represent the estimated positions of the objects. At the point shown in Figure 9(b), the candidates and the estimated positions are appropriately matched, and object tracking is performed.

[0052] Next, Figure 9(a) shows the tracking results using the conventional movement model. This is the construct image at the next time step after Figure 9(b). In the construct image, only one target object P from Figure 9(a) is extracted as a candidate. This is because the other two target objects Q and R have moved outside the construct image. However, there are three estimated object positions p1, q1, and r1, which are marked with an "x" in a square. That is, two of the positions estimated by applying the movement model to the object's position at the previous time step should have been outside the construct image, but due to errors, they remain within the construct image. As a result, the estimated result q1 and object P are incorrectly matched, and the object tracking also yields incorrect results.

[0053] Figure 9(c) shows an example where the method of this embodiment is applied to the same constituent image as in Figure 9(a). Similar to Figure 9(a), only one marker is extracted as a candidate. However, in Figure 9(c), as already explained, applying the parameters modifies the object's movement model, resulting in improved accuracy of the position estimation result p2. Therefore, in Figure 9(c), only one position estimation result p2 from the previous time point is obtained, and for markers outside the constituent image, the position estimation result is also outside the constituent image. As a result, the candidate P and the position estimation result p2 are properly matched, and object tracking is performed without error.

[0054] The various features described in the examples do not necessarily need to be present in their entirety; some can be omitted or combined as appropriate. Furthermore, the present invention is not limited to the examples and can be configured in various modified forms as described below.

[0055] (1) Variations of the object tracking system: As described above, the object tracking system of the present invention is An object tracking system that tracks an object within a sequence of images composed of multiple constituent images, An object extraction unit that extracts candidate objects from the aforementioned configuration image, A position estimation unit that estimates the position of the object in the constituent image at the next time point based on the position of the object in the constituent image at the previous time point using a predetermined movement model, The system includes an identification unit that identifies the object in the constituent image at the next time based on the candidate extracted by the object extraction unit and the position estimated by the position estimation unit, The position estimation unit can be configured as an object tracking system that changes the parameters used in the movement model according to the position of the object.

[0056] By estimating the position using a movement model that reflects parameters set according to the position of objects within the constituent image, the accuracy of the position estimation can be improved. The parameters may be provided, for example, in the form of a function that depends on the location, or in the form of a database that pre-stores parameters for each location. Alternatively, a combination of both may be used. The parameters do not necessarily need to be set for all positions in the constituent image; they may be set for only some positions.

[0057] In the above embodiment, object extraction may be performed on the entire constituent image or on a part thereof. For example, as described in the embodiment, an object may be identified by extracting objects from the entire constituent image and then associating them with the estimated position. Alternatively, the object may be identified by first estimating its position and then extracting it within a predetermined range that includes that position.

[0058] In the above embodiment, The aforementioned sequence of images may be taken while relatively reducing the distance to the object.

[0059] In such a sequence of images, the movement speed within the constituent image increases as an object approaches the edge of the constituent image, thus the present invention can be effectively utilized.

[0060] Furthermore, in the above embodiment, The aforementioned sequence of images may be captured by a camera fixed to a moving object.

[0061] In this invention, it is preferable that the relative movement between the camera and the object is known within a predetermined error range. According to the above embodiment, since the movement of the camera depends on the movement of the moving object, there is an advantage in that it is easier to reproduce known movements and improve accuracy.

[0062] In the above embodiment, The position estimation unit may perform the position estimation using a Kalman filter.

[0063] In the above embodiment, When matching the positions of candidates extracted from the constituent image with those of the target object, analysis of feature quantities such as color and shape of the extracted candidates may be used in conjunction. In addition to this, or as an alternative, analysis of the similarity between the aspect ratio of the extracted candidates and the aspect ratio of the target object may also be used.

[0064] The present invention is particularly useful in such cases.

[0065] In the above embodiment, It has a storage unit that pre-stores the parameters according to their positions within the aforementioned configuration image, The position estimation unit may set the parameters by referring to the storage unit.

[0066] This approach allows for greater flexibility in setting parameters compared to providing a mathematical formula (such as a function) that represents the object's movement model, thus enabling improved accuracy.

[0067] In the above embodiment, The storage unit divides the constituent image into multiple cells and stores the parameters in each cell. The position estimation unit may set the parameters based on the cell to which the object's position belongs.

[0068] By dividing the data into cells in this way, the amount of data stored for each parameter can be reduced.

[0069] In the above embodiment, The aforementioned cells may be allowed to have an irregular shape.

[0070] By doing this, it becomes possible to finely adjust parameters in certain areas while suppressing the overall amount of memory required for the parameters.

[0071] In the above embodiment, Multiple types of the aforementioned parameters are stored according to the shooting conditions of the continuous images. The position estimation unit may change the parameters not only according to the position of the object but also according to the shooting conditions of the continuous images.

[0072] By using different parameters depending on the shooting conditions, it is possible to further improve accuracy. In such cases, multiple parameters may be combined or modified depending on the shooting conditions.

[0073] In the above embodiment, The object extraction unit, the position estimation unit, and the identification unit may each perform processing on multiple objects.

[0074] When tracking multiple objects, the likelihood of mismatches between candidates and estimated positions increases, thus further enhancing the usefulness of the present invention.

[0075] (2) Variations of the parameter setting system: As described above, the parameter setting system of the present invention is A parameter setting system for setting parameters of a predetermined movement model in which, in a sequence of images composed of multiple constituent images, the position of an object in a constituent image at a given time is estimated based on the position of the object in the constituent image at a given time, and the parameters of the movement model are set accordingly. The parameter setting system can be configured to include a parameter setting unit that calculates the average value of the amount of movement of the object within the constituent image from the previous time to the next time, according to its position within the constituent image, and sets the parameters to be used in the movement model based on this value.

[0076] In the above-described procedure, The aforementioned image may be divided into multiple cells, and the parameters may be set for each cell.

[0077] Since errors are also included in the calculation of the amount of movement, by dividing the data into cells and processing them separately, these errors can be suppressed through statistical processing, and as a result, the accuracy of the parameters can be improved.

[0078] In the above-described procedure, If the variation in the amount of movement within the cell exceeds a predetermined value, the cell may be divided.

[0079] By doing so, the amount of movement within the cell can be smoothed out, and the accuracy of the parameters in each cell can be made more uniform.

[0080] The embodiments described in whole or in part above solve one of the following problems: appropriate processing of image data, map data or feature data; improvement of processing speed; improvement of processing accuracy; improvement of usability; improvement of functions using data or provision of appropriate functions; provision of appropriate data, programs, recording media, devices and / or systems, such as reduction of data and / or program capacity and miniaturization of devices and / or systems; and appropriate production and manufacturing of data, programs, recording media, devices and / or systems, such as reduction of production and manufacturing costs, simplification of production and manufacturing, and reduction of production and manufacturing time. [Explanation of symbols]

[0081] 1 Client 10 Object Tracking Systems 11 Input / output section 12 Position estimation part 13 Object extraction part 14 Specific section 15. Movement Model Setting Section 16 Grid parameter storage unit 20 Parameter Setting System 21 Input / output section 22 Pre-processing section 23 Grid parameter storage unit 24 Parameter setting section

Claims

1. An object tracking system that tracks an object within a sequence of images composed of multiple constituent images, An object extraction unit that extracts candidate objects from the aforementioned configuration image, A position estimation unit that estimates the position of the object in the constituent image at the next time point based on the position of the object in the constituent image at the previous time point using a predetermined movement model, A storage unit for storing parameters used in the aforementioned movement model, The system includes an identification unit that identifies the object in the constituent image at the next time based on the candidate extracted by the object extraction unit and the position estimated by the position estimation unit, The storage unit stores the parameters for each cell formed by dividing the constituent image into a plurality of cells. The position estimation unit is an object tracking system that estimates the position of the object in the constituent image at the next time step by setting the parameter corresponding to the cell to which the position of the object in the constituent image at the previous time step belongs in the movement model.

2. An object tracking system according to claim 1, wherein the object extraction unit extracts candidate objects within the constituent image before the position estimation unit estimates the position of the object.

3. An object tracking system according to claim 1, wherein the object extraction unit extracts candidate objects within a predetermined range of the estimated position of the object after the position estimation unit has estimated the position of the object.

4. An object tracking system according to any one of claims 1 to 3, The position estimation unit is an object tracking system that estimates the position using a Kalman filter.

5. An object tracking system according to any one of claims 1 to 4, The aforementioned parameters are asymmetrical values ​​in this object tracking system.

6. An object tracking system according to any one of claims 1 to 5, The aforementioned cell is an object tracking system in which a non-uniform shape is permitted.

7. A parameter setting system for setting parameters of a predetermined movement model in which, in a sequence of images composed of multiple constituent images, the position of an object in a constituent image at a given time is estimated at a given time based on the position of the object in the constituent image at a given time, and the parameters of the movement model are set accordingly. A parameter setting system comprising a parameter setting unit that divides the aforementioned constituent image into a plurality of cells, calculates the average amount of movement of the object within the constituent image from the previous time to the next time for each cell, and sets parameters to be used in the movement model for each cell based on this.

8. A parameter setting system according to claim 7, The parameter setting unit is a parameter setting system that further divides the cell when the variation in the amount of movement within the cell exceeds a predetermined value.

9. A program for enabling a computer to perform the function of tracking an object within a sequence of images composed of multiple constituent images, A function to store parameters used for a predetermined movement model in each cell formed by dividing the aforementioned constituent image into multiple cells, A function for extracting candidate objects within the aforementioned constituent image. A function that sets the parameter corresponding to the cell to which the position of the object in the constituent image at the previous time belongs in the movement model, and estimates the position of the object in the constituent image at the next time using the movement model to which the parameter corresponding to the cell to which the position of the object belongs has been set, based on the position of the object in the constituent image at the previous time. A program that enables a computer to perform the function of identifying the object in the constituent image at the next time, based on the extracted candidates and the estimated position of the object.

10. A program for setting parameters of a predetermined movement model when estimating the position of an object in a sequential image composed of multiple constituent images at a next time step, based on the position of the object in the constituent image at a previous time step, wherein the movement model is used to estimate the position of the object in the constituent image at a next time step, A function that divides the aforementioned constituent image into multiple cells and calculates the average value of the amount of movement of the object within the constituent image from the previous time to the next time for each of the cells. A program that enables a computer to implement a function to set parameters used in the movement model for each cell, based on this.

Images