Method for tracking object in flow channel
By receiving frame time series in a high object throughput channel, forming a trajectory set and detecting trajectory errors, the problems of inaccurate tracking and insufficient computing resources in the prior art are solved, and efficient and accurate real-time tracking effect is achieved.
Patent Information
- Application Number
- CN202380052446.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-07-11
- Filing Date
- 2023-07-04
- Publication Date
- 2025-03-18
AI Technical Summary
The prior art is difficult to efficiently track objects in flow channels with high object throughput, especially when the object density is high and the flow rate is fast, and the problem of misidentification of objects or insufficient computing resources is likely to occur, resulting in difficulty in real-time tracking.
By receiving the time series of frames, a collection of trajectories is formed, and track errors are detected when expanding the trajectory, and trajectory modifications are found and applied to maintain the high accuracy and computational efficiency of the trajectory. The method can make trajectory modifications based on the observed object locations of at least three frames, ensuring accurate tracking at high throughput conditions.
It realizes accurate tracking of objects in high object throughput flow channels, improves computing efficiency, and supports real-time tracking, especially suitable for tasks such as cell classification in microfluidic applications.
Smart Images
Figure CN119654657A_ABST
Abstract
Description
Technical Field
[0001] The present inventive concept generally relates to a method for tracking an object in a flow channel. Background Art
[0002] In microfluidic applications, objects in a flow channel are typically imaged by acquiring a time series of images. Based on such a time series of images, tracking algorithms can be used to extract the trajectories of individual objects. Objects can then be classified based on the movements derived from the trajectories. For example, two different cell types may move in different ways, for example, because one cell type is heavier or larger than the other. There is still room for improvement in tracking algorithms. Summary of the invention
[0003] It is an object of the inventive concept to facilitate tracking of objects in a flow channel with a high object throughput. Another object is to facilitate accurate tracking of objects in a flow channel with a high object throughput. Yet another object is to facilitate computationally efficient tracking of objects in a flow channel with a high object throughput. Yet another object is to facilitate fast tracking of objects in a flow channel with a high object throughput, in particular to facilitate real-time tracking of objects in a flow channel with a high object throughput. These and other objects of the inventive concept are at least partly met by the invention as defined in the independent claims. Preferred embodiments are listed in the dependent claims.
[0004] According to a first aspect, there is provided a method for tracking an object in a flow channel, the method comprising:
[0005] receiving a temporal sequence of frames, each frame being associated with a time and comprising an observed object position, the observed object position being a position of the object observed in the flow channel at the time, the temporal sequence of frames comprising a first set of frames and a first frame that is subsequent to an ending frame of the first set of frames;
[0006] forming a first set of trajectories, each trajectory in the first set of trajectories comprising observed object positions from the first set of frames, each trajectory representing a path through space and time of a single object in the flow channel;
[0007] extending the first set of trajectories by adding the observed object position of the first frame to the first set of trajectories;
[0008] detecting a trajectory error, the trajectory error being a portion of a trajectory in the extended first set of trajectories that indicates an impossible extension of the first set of trajectories;
[0009] finding a trajectory modification, the trajectory modification being a modification to the expanded first set of trajectories that eliminates trajectory errors, wherein finding the trajectory modification is based on observed object positions in at least three frames from the temporal sequence of frames; and
[0010] The extended first set of trajectories is modified by the trajectory modification.
[0011] The object may be a living thing, such as a cell or a bacterium. Alternatively, the object may be a non-living thing, such as a particle, a fluorescent marker, a bead, etc. The flow channel may be a microfluidic flow channel or any other type of flow channel. The flow channel may, but need not, include a column or pillar that alters the movement of the object, such as when the object hits a column while passing through the flow channel. Such a column may allow objects of similar types to be classified. Two cell types that move in a similar manner may move very differently with a column than without the column. The object may flow in a liquid or a gas in the flow channel.
[0012] To help understand the present invention, the concept of a frame will be briefly discussed below. A frame will be described as representing an observed object position, which is observed in an image of an object in a flow channel. However, it should be understood that the observed object position of a frame may also be derived by other means.
[0013] FIG. 1 shows how a time sequence 20 of frames 21 is derived from a time sequence 10 of images 11 . Figure 1a shows a time series 10 of images 11, and Figure 1b A time sequence 20 of frames 21 derived from the time sequence 10 of images 11 is shown. Each frame 21 may correspond to an image 11 of the flow channel 2, and the time associated with the frame 21 may be a time representing the acquisition time of the corresponding image. An object 1 in the flow channel 2 may be imaged by acquiring the time sequence 10 of images 11, shown here as images 11a, 11b, 11c, and 11d. The time sequence 10 of images 11 may be converted into a time sequence 20 of frames 21, wherein each frame 21 (shown here as frames 21a, 21b, 21c, and 21d) includes an observed object position 4 derived from the image 11 corresponding to the frame 21. For example, the observed object position included in frame 21d may be the object position observed in image 11d.
[0014] A frame may, for example, include a list of observed object locations and the image from which the frame was derived.
[0015] Alternatively, a frame may include a list of observed object locations and portions of the image from which the frame was derived, such as portions of the image surrounding the observed object locations.
[0016] Alternatively, a frame may, for example, include a list of observed object locations rather than the image from which the frame was derived.
[0017] The observed object positions may be represented in the frame in other ways than in a list. The observed object positions may be represented in a matrix, where each matrix element corresponds to a position in the flow path, such as a pixel in the image from which the frame was converted. The object positions may be marked in the matrix of the frame by a number in the matrix element, such as "1".
[0018] In addition to the observed object locations, frames may also include other information. A frame may include information about objects in the image corresponding to the frame. Such information may be, for example,
[0019] The size of the object; and / or
[0020] object brightness; and / or
[0021] Object shape; and / or
[0022] The object z position; and / or
[0023] The average refractive index of the object; and / or
[0024] Object absorbance; and / or
[0025] Object boundary uniformity; and / or
[0026] Object orbital movement.
[0027] FIG. 2 shows how a trajectory 41 is formed from a collection 30 of frames 21 . Figure 2a A set 30 of frames 21 is shown, including frames 21a-c, which represent the movement of an object 1' and an object 1". Frames 21a-c include observed object positions 4a', 4b', 4c' and 4a", 4b", 4c". Figure 2a In , frames 21a-c also include images of objects 1' and 1". Figure 2a , objects 1', 1" move from left to right, i.e., from the left edge of frame 21 (which can be interpreted as the inlet 6 of flow channel 2) to the right edge of frame 21 (which can be interpreted as the outlet 8 of flow channel 2). The tracking algorithm can infer that the observed object positions 4a', 4b', 4c' represent the movement of object 1', while the observed object positions 4a", 4b", 4c" represent the movement of object 1". Therefore, the tracking algorithm can form a set 40 of trajectories 41, including trajectories 41' and 41", as shown. Figure 2b It should be noted that Figure 2b The trajectory 41 can be represented in many different ways, for example as a list of observed object positions.
[0028] Prior art tracking methods typically process frames sequentially and extend the trajectory for each new frame. Thus, in prior art tracking methods, a trajectory set may be formed based on frames 1 to i, where i is an integer. The trajectory set may then be extended with the observed object position obtained from frame i+1. The extended trajectory set may then be further extended with the observed object position obtained from frame i+2, and so on.
[0029] The inventors have recognized that tracking objects in high-throughput flow channels (i.e., flow channels through which many objects pass per time unit) is desirable but difficult to accomplish using prior art tracking methods. For example, cell classification based on cell trajectories in microfluidic channels is useful in lab-on-a-chip applications. Classification can be based on, for example:
[0030] - the first cell type moves slower than the second cell type; and / or
[0031] - the first cell type takes a longer path than the second cell type; and / or
[0032] -The first cell type speeds up or slows down faster than the second cell type.
[0033] A high throughput of cells allows for fast analysis. High throughput may require a high density of objects in the flow channel and a fast flow rate, which in turn results in large object displacements per frame. In this case, prior art tracking methods may stop working, for example because objects are wrongly identified so that when extending the trajectory, the observed object position is added to the wrong trajectory. Alternatively, prior art tracking methods may require too many computing resources to operate in real time. This in turn means that even if the throughput of the flow channel is high, the analysis time is still long.
[0034] According to the present invention, forming the first set of trajectories can be accomplished by any type of tracking method, such as any type of prior art tracking method. For example, the first set of frames can be frames 1 to i of a time sequence of frames, such that the first set of trajectories includes the observed object positions from frames 1 to i.
[0035] Recognize that extending the first set of trajectories by adding the observed object position of the first frame to the first set of trajectories may be a quick way to construct the trajectories. Continuing with the above example, the ending frame of the first set of frames may be frame i in the temporal sequence of frames, and thus, the first frame may be frame i+1 in the temporal sequence of frames.
[0036] It is further recognized that extending a set of trajectories may sometimes introduce errors, for example, if the observed object position is added to an erroneous trajectory. In addition, if only subsequent frames are used to extend the trajectory, the extended trajectory may produce a chain reaction of errors in the subsequent extension of the trajectory. Therefore, it is recognized that by detecting trajectory errors, finding trajectory modifications and modifying the extended first set of trajectories, high accuracy of the trajectory can be maintained; even in the case of high object throughput and real-time tracking. For example, each frame can be processed sequentially, and the trajectory can be extended for each new frame. The trajectory error can be checked after each extension or after multiple unchecked extensions, such as after at least 3 times, at least 5 times, or at least 10 times of unchecked extensions. If a trajectory error is detected, a search for trajectory modifications can be performed, so that the trajectory modification can be found. Finding the trajectory modification is based on the observed object position in at least three frames of the time sequence of frames. For example, if the first frame is frame i+1 of the time sequence of frames, and the extension of the first set of trajectories results in a trajectory error, the trajectory modification can be found based on frames i-2, i-1, and i. As another example, finding the trajectory modification can be based on frames i-3, i-2, i-1, and i. As another example, finding the trajectory modification can be based on frames i-1, i and i+1. Finding the trajectory modification can be based on at least three consecutive frames. By performing trajectory modification based on at least three frames, high accuracy of the trajectory is achieved. The accuracy may come at the expense of computational cost, because processing at least three frames simultaneously may be computationally more difficult than processing only two frames at a time. However, it may not be necessary to process at least three frames simultaneously frequently, for example, it is not necessary to do so when the trajectory is extended for each frame, only when a trajectory error is detected. Therefore, the method can facilitate tracking of objects in a high-throughput flow channel. Tracking can be accurate and computationally efficient. Therefore, fast tracking of objects in a high-throughput flow channel, particularly real-time tracking, is facilitated. It should be understood that the method may be particularly advantageous when used on frames derived from images captured by holographic imaging (also known as lensless imaging), because these images may have a large field of view, which in turn causes large displacements of objects per frame.
[0037] When extending the first set of trajectories, the observed object position of the first frame may be added to the first set of trajectories as part of an existing trajectory in the first set of trajectories or as a new trajectory. When extending the first set of trajectories, the observed object position of the first frame may be added to the first set of trajectories without changing the trajectory in the portion associated with the frames before the first frame. Thus, the observed object position of the first frame may be concatenated to an existing trajectory in the first set of trajectories.
[0038] The method may be performed by one or more processing units. For example, the method may be performed by a processing unit of a computer.
[0039] The processing unit may receive a time sequence of frames from an imager (e.g., from an imager connected to a computer) and then process the frames. Alternatively, the processing unit may receive a time sequence of images from an imager, e.g., from an imager connected to a computer. The processing unit may identify object locations in the images and form frames. Here, each frame may include, for example, an image and a list of object locations. Thus, the frames may be received by the processing unit. The processing unit may then form a first set of trajectories, extend the first set of trajectories, detect trajectory errors, find trajectory modifications, and modify the extended first set of trajectories. Alternatively, some of these tasks may be passed to another processing unit.
[0040] The first set of extended trajectories may be based on only two frames in the temporal sequence of frames, the only two frames being the end frame of the first set of frames and the first frame. This may enable computationally efficient and / or fast extension of the first set of trajectories.
[0041] Detecting trajectory errors can be done in various ways.
[0042] A trajectory error may include a portion of a trajectory in the extended first set of trajectories that shares an observed object position with another trajectory in the extended first set of trajectories. Thus, two trajectories in the extended first set of trajectories that include the same observed object position in the same time domain may be trajectory errors. For example, if the observed object position in frame A has been input into two different trajectories in the time domain of the trajectory corresponding to the time associated with frame A, this may constitute a trajectory error. Such a tracking error may indicate that two objects may be misidentified. For example, objects may cross paths, which may result in an erroneous extension of the trajectory. Alternatively, an object may not be identified at all, which may result in the tracking algorithm using the same observed object position twice. The simultaneous presence of two objects in the same place may be considered an impossible event. Thus, this may be considered an impossible extension of the trajectory.
[0043] A trajectory error may include a trajectory start point in the expanded first set of trajectories that is not at the flow channel entrance. The start point of the trajectory may be the first observed object position of the trajectory. The flow channel entrance may correspond to an edge of the frame, such as at Figure 2a is the left edge of the frame. Being at the entrance of the flow channel can be interpreted as being within a threshold distance from the flow channel entrance, for example within a distance corresponding to 10% of the frame width. The sudden appearance of an object can be considered an impossible event. Therefore, this can be regarded as an impossible extension of the trajectory.
[0044] A trajectory error may include a trajectory endpoint in the expanded first set of trajectories that is not at the flow channel exit. The trajectory start point may be the first observed object position of the trajectory. The flow channel exit may correspond to an edge of the frame, such as at Figure 2a is the right edge of the frame. Being at the exit of the flow channel can be interpreted as being within a threshold distance from the flow channel exit, for example within a distance corresponding to 10% of the frame width. The object disappearing far from the exit can be considered an impossible event. Therefore, this can be regarded as an impossible extension of the trajectory.
[0045] The trajectory error may include a portion of the trajectory in the extended first set of trajectories within a forbidden region, which is a region of the flow channel that the object cannot enter. The forbidden region may be a wall of the flow channel or a column or pillar in the flow channel. Object movement through the forbidden region may be considered an impossible event. Therefore, this may be considered an impossible extension of the trajectory.
[0046] A trajectory error may include a portion of a trajectory in the extended first set of trajectories merging with another trajectory in the extended first set of trajectories. An object merging with another object may be considered an impossible event. Therefore, this may be considered an impossible extension of the trajectory.
[0047] A trajectory error may include a portion of a trajectory in the extended first set of trajectories being separated from another trajectory in the extended first set of trajectories. The separation of an object into two objects may be considered an impossible event. Therefore, this may be considered an impossible extension of the trajectory.
[0048] It should be understood that a trajectory error may be detected in the last time entry in the extended first set of trajectories, i.e., in the time entry associated with the first frame (frame i+1 in the above example). Alternatively, a trajectory error may be detected in an earlier time entry in the extended first set of trajectories. Some trajectory errors may not be apparent until after several extensions of the first set of trajectories.
[0049] It should be appreciated that finding trajectory modifications may be based on a subset of trajectories within the extended first set of trajectories. Thus, calculations for finding trajectory modifications may not necessarily involve all trajectories in the extended first set of trajectories. This facilitates computationally efficient and / or fast finding of trajectory modifications.
[0050] Finding trajectory modifications may include:
[0051] selecting a subset of trajectories within the expanded first set of trajectories, the selected subset of trajectories being trajectories adjacent to the trajectory error;
[0052] Wherein the trajectory modification only involves trajectories in the extended first set of trajectories within the selected trajectory subset. Trajectories adjacent to a trajectory error may be trajectories whose portion of the trajectory is within a threshold range of the trajectory error both in space and time. The selected trajectory subset may be a plurality of trajectories closest to the trajectory error in space and time, such as 1 trajectory, 2 trajectories, or 5 trajectories. The trajectory subset may include trajectories with trajectory errors.
[0053] Finding trajectory modifications can be based on a region of interest, which represents an observed object position in a portion of the flow channel. Therefore, the calculations used to find trajectory modifications can be based on portions of frames rather than entire frames. Therefore, finding trajectory modifications can be based on portions of at least three frames in a time sequence of frames. This facilitates computationally efficient and / or fast finding of trajectory modifications. For example, if a trajectory error corresponding to an observed object position at image coordinates [a, b] in frame i-2 is detected, the region of interest can correspond to image coordinates [aN,…,a+N; bM,…,b+M], where N and M are integers. The region of interest can be applied to the at least three frames of the time sequence of frames used in finding trajectory modifications, such as frames i-3, i-2, i-1, and i.
[0054] N may correspond, for example, to 10 times the size of the object, or 20 times the size of the object, which may be, for example, the radius of the object. Alternatively, N may correspond, for example, to 10% or 20% of the width or height of the frame. Alternatively, N may correspond to twice or four times the distance that the object travels in the flow channel between two consecutive frames.
[0055] M may correspond, for example, to 10 times the size of the object, or 20 times the size of the object, which may be, for example, the radius of the object. Alternatively, M may correspond, for example, to 10% or 20% of the frame width or height. Alternatively, M may correspond to twice or four times the distance that the object travels in the flow channel between two consecutive frames.
[0056] The trajectory modification may include modification of a portion of the trajectory that represents a time before the time associated with the first frame. Some trajectory errors may not be easily detected until after several expansions of the first set of trajectories. Modifying a portion of the trajectory that represents a time before the time associated with the first frame makes it possible to correct such trajectory errors. Finding the trajectory modification may include:
[0057] forming a plurality of candidate trajectory modifications;
[0058] comparing each of the plurality of candidate trajectory modifications to a criterion indicating a possible extension of the first set of trajectories; and
[0059] One of the candidate trajectory modifications that meets the criteria is selected as a trajectory modification to modify the expanded first set of trajectories.
[0060] The plurality of candidate trajectory modifications may include a trajectory modification that represents all or some of the different permutations of ways to combine spatially and temporally proximate observed object locations of trajectory errors into a trajectory.
[0061] The criterion indicating the possible extension of the first set of trajectories may be a criterion indicating possible and / or physically reasonable movement of the object. For example, the criterion may penalize impossible and / or unreasonable movement of the object.
[0062] Finding the trajectory modification in this way can provide an accurate trajectory modification. Several different alternatives for the object's movement can be examined and the most likely one or one of the most likely ones can be selected.
[0063] This standard can be designed in many different ways.
[0064] The criterion may include that the variance of the velocity of the object at the observed object position of the candidate trajectory modification is below a threshold. In some cases (e.g., the object moves in laminar flow or in a flow channel area with few obstacles, pillars or columns), small changes in velocity may indicate possible movement. In this case, the object may be unlikely to accelerate and decelerate. Therefore, candidate trajectory modifications indicating such movement may be impossible. The threshold value of the velocity variance may be, for example, 10% of the average velocity of the object represented by the trajectory. As an alternative, the criterion may be that the variance of the velocity of the object at the observed object position of the candidate trajectory modification is minimized.
[0065] The criterion may include that the variance of the brightness of the object at the observed object position modified by the candidate trajectory is below a threshold. The object may be unlikely to change brightness, i.e. flicker. Therefore, candidate trajectory modifications indicating such brightness changes may be unlikely. The threshold for the brightness variance may be, for example, 10% of the average brightness of the object represented by the trajectory. As an alternative, the criterion may be that the variance of the brightness of the object at the observed object position modified by the candidate trajectory is minimized.
[0066] The criterion may include that the variance of the size of the object at the observed object position modified by the candidate trajectory is below a threshold. The object is unlikely to change size. However, when imaging the object, the size may be perceived differently depending on the viewing angle. However, large changes in perceived size are unlikely. Therefore, candidate trajectory modifications indicating such size changes may be impossible. The threshold for the size variance may be, for example, 10% of the average size of the objects represented by the trajectory. As an alternative, the criterion may be that the variance of the size of the object at the observed object position modified by the candidate trajectory is minimized.
[0067] The criterion may include that the variance of the roundness of the object at the observed object position modified by the candidate trajectory is below a threshold. The object is unlikely to change roundness. However, when imaging the object, the roundness may be perceived differently depending on the viewing angle. However, a large change in perceived roundness is unlikely. Therefore, a candidate trajectory modification indicating such a change in roundness may be impossible. The threshold for the roundness variance may be, for example, 10% of the average roundness of the object represented by the trajectory. As an alternative, the criterion may be that the variance of the roundness of the object at the observed object position modified by the candidate trajectory is minimized.
[0068] The criterion may include that the z-direction variance of the object at the observed object position of the candidate trajectory modification is below a threshold. In the case of lensless imaging, such as interference-based imaging, the position of the object in the z-direction (i.e., the distance to the imaging device) can be derived. The position of the object in the z-direction can correspond to the depth at which the object moves in the flow channel. It is unlikely that the object moves up and down in the flow channel, i.e., changes depth. Therefore, candidate trajectory modifications indicating such depth changes may be impossible. The threshold value for the z-direction variance can be, for example, 10% of the average z-direction of the object represented by the trajectory. As an alternative, the criterion may be that the z-direction variance of the object at the observed object position of the candidate trajectory modification is minimized.
[0069] The criterion may include that the total path length of the candidate trajectory modification is below a threshold. The object is likely to take the shortest path. Therefore, a candidate trajectory modification indicating a long path length may be unlikely. The threshold path length may be, for example, 150% of the frame width. Alternatively, the criterion may be that the total path length of the object at the observed object position of the candidate trajectory modification is minimized.
[0070] Criteria can include variances for:
[0071] The size of the object; and / or
[0072] object brightness; and / or
[0073] Object shape; and / or
[0074] The object z position; and / or
[0075] The average refractive index of the object; and / or
[0076] Object absorbance; and / or
[0077] Object boundary uniformity; and / or
[0078] Object orbital movement.
[0079] Extending the first set of trajectories includes applying a local tracking algorithm that operates on the end frame and the first frame of the first set of frames. The local tracking algorithm can be an algorithm that determines the extension of the trajectory based on local information around the trajectory (e.g., around the latest entry of the trajectory). The local information can be local temporal information (i.e., two consecutive frames) and / or local spatial information (i.e., a limited area around the trajectory). The local tracking algorithm can be a greedy tracking algorithm, a nearest neighbor tracking algorithm, or a Kalman filter tracking algorithm. Extending the first set of trajectories in this way facilitates computationally efficient and / or fast extension of the trajectory.
[0080] Finding the trajectory modification may include applying a global tracking algorithm to at least three frames in the time sequence of frames. The global tracking algorithm may be an algorithm that is not limited to local temporal or spatial information. The global tracking algorithm may be a tracking algorithm based on dynamic programming, a multiple hypothesis algorithm, or a global optimization problem algorithm. Finding the trajectory modification in this way may provide accurate trajectory modifications.
[0081] Finding trajectory modifications may include calculating a velocity based on at least one trajectory in the expanded first set of trajectories. Using velocity may be a very efficient method of finding possible trajectory modifications. Velocity may be calculated in many different ways. For example, consider a situation where the first and second observed object positions are part of the same trajectory. The velocity may then be calculated as the difference between the distance traveled between the first and second observed object positions divided by the time associated with the first and second observed object positions. The calculated velocity may represent the velocity of the object at the first observed object position, at the second observed object position, or during travel between the first and second observed object positions.
[0082] The at least three frames of the temporal sequence of frames may include at least one frame after the first frame. When a trajectory error is detected at a certain point in time, it may be useful to look at a future point in time to determine how to resolve the error. In this way, accurate trajectory modification is provided.
[0083] According to a second aspect of the present invention, there is provided a computer program comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to the first aspect of the present invention.
[0084] According to a third aspect of the present invention, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method.
[0085] The effects and features of the second and third aspects are generally similar to those described above in conjunction with the first aspect. The embodiments mentioned in relation to the first aspect are generally compatible with the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0086] The above and other objects, features and advantages of the inventive concept will be better understood through the following detailed description which is illustrative and non-restrictive with reference to the accompanying drawings. Unless otherwise specified, the same reference numerals will be used for the same elements in the accompanying drawings.
[0087] Figure 1a A time sequence of images is shown.
[0088] Figure 1b A time sequence of frames is shown.
[0089] Figure 2a A collection of frames is shown.
[0090] Figure 2b A collection of trajectories is shown.
[0091] Figure 3 A flow chart of the method is shown. DETAILED DESCRIPTION
[0092] In conjunction with the accompanying drawings, the technical content and detailed description of the present invention are described below according to preferred embodiments, which are not intended to limit the scope of the protection claimed. However, the present invention can be implemented in many different forms and should not be interpreted as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided for thoroughness and completeness, and to fully convey the scope of the present invention to those skilled in the art.
[0093] Figure 3 A method 100 for tracking an object 1 in a flow channel 2 is shown.
[0094] According to the method 100, a time sequence 20 of frames 21 is received at S102. Each frame 21 in the time sequence 20 of frames 21 is associated with a time and includes an observed object position 4, which is an object position observed in the flow channel 2 at the time. The frame 21 may be a combination of Figure 1a -b describes frame 21.
[0095] For example, the frames 21 may be processed sequentially and the trajectory 41 may be extended for each new frame 21. Each extension of the trajectory 41 may be done based on only two frames 21.
[0096] To illustrate the method 100, consider a point during the processing of frames 21 where the first set 30 of frames 21 has been processed. A first set 40 of trajectories 41 is formed S104, each trajectory 41 in the first set 40 of trajectories 41 comprising an observed object position 4 from the first set 30 of frames 21.
[0097] Then, the first set 40 of trajectories 41 is extended S106 by adding the observed object position 4 of the first frame 31 to the first set 40 of trajectories 41. Here, the first frame 31 is a frame 21 after the end frame 30 of the first set 30 of frames 21. For example, the end frame of the first set 20 of frames 21 may be frame i in the temporal sequence of frames, and the first frame 31 may be frame i+1 in the temporal sequence 30 of frames 21. Then, the observed object position 4 of frame i+1 may be added to trajectory 41 in the first set 40 of trajectories 41, after the last entry in the trajectory 41, the last entry of the trajectory 41 being the entry corresponding to frame i.
[0098] A check for trajectory errors may then be performed, and a trajectory error may be detected in S108 .
[0099] The method may then search for trajectory modifications and when a trajectory modification is found in S110, modify S120 the extended first set 40 of trajectories 41 by the trajectory modification. Finding S110 trajectory modifications is here based on observed object positions 4 in at least three frames 21 of the temporal sequence 20 of frames 21 .
[0100] The method 100 may further include an optional step S112 of selecting a subset of trajectories 41 in the expanded first set of trajectories 41, the selected subset of trajectories 41 being trajectories 41 adjacent to the trajectory error. Therefore, in S110, the trajectory modification may be based on the subset of trajectories 41 in the expanded first set of trajectories 41.
[0101] Additionally or alternatively, the method 100 may further include the following steps:
[0102] Forming S114 multiple candidate trajectory modifications;
[0103] comparing S116 each of the plurality of candidate trajectory modifications to a criterion indicating a possible extension of the first set 40 of trajectories 41 ; and
[0104] One of the candidate trajectory modifications meeting the criteria is selected S118 as a trajectory modification to modify the extended first set 40 of trajectories 41 .
[0105] As previously mentioned, the method 100 can be performed in many different ways. For example, the method 100 can be applicable to many different types of trajectory errors. Trajectory errors can include:
[0106] a portion of a trajectory 41 in the extended first set 40 of trajectories 41 that shares an observed object position 4 with a portion of another trajectory 41 in the extended first set 40 of trajectories 41;
[0107] A starting point of a trajectory 41 in the extended first set 40 of trajectories 41 that is not at the entrance of the flow channel 2;
[0108] an end point of the trajectory 41 in the extended first set 40 of trajectories 41 that is not at the outlet of the flow channel 2;
[0109] a portion of a trajectory 41 of the extended first set 40 of trajectories 41 within a forbidden region, the forbidden region being a region of the flow channel 2 into which the object 1 cannot enter;
[0110] a portion of a trajectory 41 in the extended first set 40 of trajectories 41 , the portion being merged with another trajectory 41 in the extended first set 40 of trajectories 41 ;
[0111] a portion of a trajectory 41 in the extended first set 40 of trajectories 41 , the portion being separated from another trajectory 41 in the extended first set 40 of trajectories 41 ;
[0112] As another example, if a trajectory modification is found in S110 by using a candidate trajectory modification, the candidate trajectory modification may be selected S118 to be used to modify S120 the extended first set 40 of trajectories 41 according to any one of the following criteria:
[0113] The velocity variance of object 1 at the observed object position 4 modified by the candidate trajectory is below a threshold;
[0114] The brightness variance of object 1 at the observed object position 4 modified by the candidate trajectory is lower than the threshold; the size variance of object 1 at the observed object position 4 modified by the candidate trajectory is lower than the threshold; the roundness variance of object 1 at the observed object position 4 modified by the candidate trajectory is lower than the threshold; the z-direction variance of object 1 at the observed object position 4 modified by the candidate trajectory is lower than the threshold;
[0115] The total path length modified by the candidate trajectory is below the threshold.
[0116] In particular, it should be understood that the method may include any combination of the detection S108 of one of the above-given trajectory error examples and the finding S110 of trajectory modification by using one of the above-given standard examples.
[0117] However, in brief, method 100 is hereinafter illustrated by detecting S108 a portion of a trajectory 41 in the extended first set of trajectories 41 that shares an observed object position 4 with a portion of another trajectory 41 in the extended first set of trajectories 41, and then finding S110 a trajectory modification using a criterion related to a velocity variance of object 1 at the observed object position 4 of the candidate trajectory modification being below a threshold.
[0118] In this example, four consecutive frames 21 corresponding to consecutive times t1, t2, t3 and t4 are considered, where the frames 21 represent the movement of objects A and B and the movement of other objects. In this example, the observed object positions A1, A2, A3 and A4 represented in the four consecutive frames are the object positions of object A at times t1, t2, t3 and t4. In addition, in this example, the observed object positions B1, B2, B3 and B4 represented in the four consecutive frames are the object positions of object B at times t1, t2, t3 and t4. A first set 40 of trajectories 41 of S104 is formed based on the frames corresponding to times t1-t3, in which trajectory A includes A1, A2, A3 and trajectory B includes B1, B2, B3. Then, the first set 40 of trajectories 41 of S106 is extended by adding the observed object position 4 of the first frame 31 to the first set 40 of trajectories 41. In this example, trajectory 41 is extended S106 by applying a local tracking algorithm to frames corresponding to times t3 and t4. Here, the local tracking algorithm may search for the observed object position in the first frame 31 (here associated with time t4), near the observed object position 4 of the ending frame 30 of the first set of frames 21 (here associated with time t3). The local tracking algorithm may, for example, find A4 near A3, and may correctly form the extended trajectory A as A1, A2, A3, A4. Furthermore, for the purposes of this example, the local tracking algorithm may also find A4 near B3, and may incorrectly form the extended trajectory B as B1, B2, B3, A4. Furthermore, for the purposes of this example, the local tracking algorithm may also create a new trajectory - trajectory C, and assign B4 to trajectory C.
[0119] According to the method 100, a trajectory error is then detected in S108. In this example, a trajectory error may be detected S108 as a portion of a trajectory 41 in the extended first set 40 of trajectories 41 shares an observed object position 4 with a portion of another trajectory 41 in the extended first set 40 of trajectories 41. Therefore, since A4 is also included in the extended trajectory A, the extended trajectory B including A4 may be detected as a trajectory error.
[0120] According to the method 100, a trajectory modification is then found in S110, wherein the trajectory modification eliminates the trajectory error. Finding S110 the trajectory modification is based on observed object positions 4 in at least three frames 21 of the temporal sequence 20 of frames 21. In this example, finding S110 the trajectory modification is based on observed object positions 4 from frames associated with times t2, t3, and t4. For example, a plurality of candidate trajectory modifications may be formed S114 and compared to a criterion S116, after which one candidate trajectory modification that meets the criterion may be selected S118. Finding S110 the trajectory modification may be accomplished using a global tracking algorithm.
[0121] All observed object positions 4 in the frame 21 used to find S110 trajectory modification may not necessarily be used. For example, a subset of S112 trajectory 41 can be selected, wherein the subset of trajectory 41 is the trajectory 41 adjacent to the trajectory error. In this example, the subset of trajectory 41 can be an extended trajectory A, an extended trajectory B, and a trajectory C. In addition or alternatively, finding S110 trajectory modification can be based on a region of interest. In this example, finding S110 trajectory modification is based only on observed object positions A2, A3, A4 and B2, B3, B4 from frames associated with time t2, t3, and t4, even though other observed object positions 4 (e.g., object positions associated with objects D, E, and F, etc.) may exist in these frames. Different arrangements of these observed object positions 4 can represent different candidate trajectory modifications. Examples of candidate trajectory modifications may be:
[0122] A2, B3, A4;
[0123] B2, B3, A4;
[0124] B2, B3, B4;
[0125] A2, A3, A4;
[0126] etc.
[0127] When these trajectory modifications are compared to a criterion such as the speed variance of object 1 at the observed object position 4 of the candidate trajectory modification being below a threshold, it is clear that B2, B3, B4 should be selected S118 as the trajectory modification to replace the last three entries of the extended trajectory B (which is B1, B2, B3, A4). Therefore, the extended first set of trajectories 41 is modified S120 accordingly. The selection S118 of B2, B3, B4 as the trajectory modification can here be based on, for example, that B2, B3, B4 have the smallest speed variance among the candidate trajectory modifications.
[0128] In the above, the inventive concept has mainly been described with reference to a limited number of examples. However, as readily appreciated by a person skilled in the art, other examples than the ones disclosed above are equally possible within the scope of the inventive concept as defined by the appended claims.
Claims
1. A method (100) for tracking an object (1) in a flow channel (2), the method (100) comprising: Receiving (S102) a temporal sequence (20) of frames (21), each frame (21) being associated with a time and comprising an observed object position (4), the observed object position (4) being an object position observed in the flow channel (2) at the time, the temporal sequence (20) of frames (21) comprising a first set (30) of frames (21) and a first frame (31), the first frame (31) being subsequent to an end frame of the first set (30) of frames (21); forming (S104) a first set (40) of trajectories (41), each trajectory (41) of the first set (40) of trajectories (41) comprising observed object positions (4) from the first set (30) of frames (21), each trajectory representing a path in space and time of a single object (1) in the flow channel (2); extending (S106) said first set (40) of trajectories (41) by adding the observed object position of said first frame (31) to said first set (40) of trajectories (41); detecting (S108) a trajectory error, the trajectory error being a portion of a trajectory in the extended first set (40) of trajectories (41), the trajectory error indicating an impossible extension of the first set (40) of trajectories (41); finding (S110) a trajectory modification, the trajectory modification being a modification of the extended first set (40) of trajectories (41) that eliminates the trajectory error, wherein finding the trajectory modification is based on observed object positions (4) in at least three frames (21) from the time sequence (20) of frames (21); and The extended first set (40) of trajectories (41) is modified (S120) by the trajectory modification.
2. The method (100) according to claim 1, characterized in that: The first set (40) of extended (S106) trajectories (41) is based on only two frames (21) in the time series (20) of frames (21), the only two frames (21) being the end frame of the first set (30) of frames (21) and the first frame (31).
3. The method (100) according to claim 1 or 2, characterized in that: The trajectory error includes at least one of the following: a portion of a trajectory (41) in the extended first set (40) of trajectories (41) that shares an observed object position (4) with a portion of another trajectory (41) in the extended first set (40) of trajectories (41); a starting point of a trajectory (41) in the expanded first set (40) of trajectories (41), the starting point not being at the entrance of the flow channel (2); an end point of a trajectory (41) in the expanded first set (40) of trajectories (41), the end point not being at an exit of the flow channel (2); a portion of a trajectory (41) in the extended first set (40) of trajectories (41) within a forbidden region, the forbidden region being a region of the flow channel (2) into which the object (1) cannot enter; A portion of a trajectory (41) in the extended first set (40) of trajectories (41), the portion being merged with another trajectory (41) in the extended first set (40) of trajectories (41); A portion of a trajectory (41) in the extended first set (40) of trajectories (41) that is separated from another trajectory (41) in the extended first set (40) of trajectories (41).
4. The method (100) according to any one of the preceding claims, characterized in that Finding that the trajectory modification is based on a subset of trajectories (41) within the extended first set (40) of trajectories (41).
5. The method (100) according to claim 4, characterized in that: Finding (S110) the trajectory modification includes: Selecting (S112) a subset of the trajectories (41) within the expanded first set (40) of trajectories (41), the selected subset of trajectories (41) being trajectories (41) that are adjacent to the trajectories (41); The trajectory modification involves only trajectories (41) within a selected subset of trajectories (41) of the expanded first set (40) of trajectories (41).
6. The method (100) according to any one of the preceding claims, characterized in that It is found (S110) that the trajectory modification is based on a region of interest, the region of interest representing an observed object position (4) in a portion of the flow channel (2).
7. The method (100) according to any one of the preceding claims, characterized in that The trajectory modification comprises a modification of a portion of a trajectory (41), the modification representing a time prior to a time associated with the first frame (31).
8. The method (100) according to any one of the preceding claims, characterized in that Finding (S110) the trajectory modification includes: forming ( S114 ) a plurality of candidate trajectory modifications; comparing each of the plurality of candidate trajectory modifications to a criterion (S116), the criterion indicating a possible extension of the first set (40) of trajectories (41); and One of the candidate trajectory modifications that meets the criteria is selected (S118) as a trajectory modification to modify the extended first set (40) of trajectories (41).
9. The method (100) according to claim 8, characterized in that: The criteria include at least one of the following: The velocity variance of the object (1) at the observed object position (4) modified by the candidate trajectory is below a threshold; The brightness variance of the object (1) at the observed object position (4) modified by the candidate trajectory is below a threshold; The variance of the size of the object (1) at the observed object position (4) modified by the candidate trajectory is below a threshold; The circularity variance of the object (1) at the observed object position (4) modified by the candidate trajectory is below a threshold; The z-direction variance of the object (1) at the candidate trajectory modified observed object position (4) is below a threshold; The total path length modified by the candidate trajectory is below the threshold.
10. The method (100) according to any one of the preceding claims, characterized in that: Expanding (S106) said first set (40) of trajectories (41) comprises applying a local tracking algorithm operating on an end frame of said first set (30) of frames (21) and said first frame (31).
11. The method (100) according to any one of the preceding claims, characterized in that: Finding (S110) the trajectory modification comprises applying a global tracking algorithm to the at least three frames (21) of the time sequence (20) of frames (21).
12. The method (100) according to any one of the preceding claims, characterized in that: Finding (S110) the trajectory modification comprises calculating a velocity from at least one trajectory (41) of the extended first set (40) of trajectories (41).
13. The method (100) according to any one of the preceding claims, characterized in that The at least three frames (21) in the time sequence (20) of frames (21) include at least one frame subsequent to the first frame (31).
14. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to claims 1-13.
15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to claims 1-13.