Feature tracking system and method
Patent Information
- Application Number
- JP2022193462
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-16
- Filing Date
- 2022-12-02
- Publication Date
- 2025-12-03
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a facial tracking system and method. [Background technology]
[0002] Facial tracking is often used in motion capture (e.g., performance capture), which converts an actor's facial expressions from video footage into a 3D mesh of a virtual character. Summary of the Invention [Problem to be solved by the invention]
[0003] Current face tracking has a number of drawbacks.
[0004] First, good results require consistently high-quality makeup markers, which require specialist clothing and headgear, and the application of dots and lines on the actor's face, which is often uncomfortable and distracts from the actor's performance or that of their co-stars.
[0005] Second, the tracked points are very sparse, making it difficult to track the fine details of the performance in the final animation, and the markers themselves can obscure subtleties in the actor's face.
[0006] Third, issues such as marker occlusion during performance (i.e., the actor touching their face) often cause tracking loss, leading to critical failure.
[0007] Fourth, tracking techniques are often used as black boxes, making them difficult to apply to specific use cases (e.g., when using camera assets such as stereo footage).
[0008] Finally, the quality of the source footage (e.g., lighting fluctuations, video resolution, etc.) often has a significant impact on the performance of a tracking system.
[0009] The present invention aims to mitigate or alleviate some or all of the above problems. [Means for solving the problem]
[0010] Various aspects and features of the present invention are defined in terms of the claims and the detailed description.
[0011] In a first aspect, in claim 1, a point tracking method is provided.
[0012] In a second aspect, in claim 12, a point tracking system is provided. [Brief explanation of the drawings]
[0013] A more complete understanding of the present disclosure and its advantages will be obtained from the following detailed description read in conjunction with the accompanying drawings, in which: [Figure 1] 1 is a schematic diagram of a point tracking system according to a first embodiment of the present application. [Figure 2] 1 is a flowchart of a point tracking method according to a first embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION
[0014] This specification discloses a facial tracking system and method. In the following description, some specific details are set forth for the purpose of providing a thorough understanding of embodiments of the present invention. However, it will be apparent to those skilled in the art that the use of these specific details is not necessary to practice the present invention. Conversely, for the purposes of clarity, certain details known to those skilled in the art may be omitted where appropriate.
[0015] Embodiments of the present disclosure may be applicable to entertainment systems such as computers or video game consoles, development kits for such systems, or motion capture systems using dedicated hardware or computers, or suitable camera systems, and in this application the terms entertainment system and motion capture system may be considered equivalent to any suitable such equipment device.
[0016] For ease of description and reference, like or corresponding parts are designated with like reference numerals in the following drawings. Figure 1 illustrates an example entertainment system 10, such as the Sony PlayStation 5 (PS5)®.
[0017] The entertainment system 10 includes a central processor 20. This may be a single or multi-core processor, and may have, for example, eight cores, as in the PS5. The entertainment system 10 also includes a graphics processing unit (GPU) 30. The GPU 30 may be physically separate from the CPU, or may be integrated into the CPU as a system-on-chip (SoC), as in the PS5.
[0018] The entertainment system 10 may also include RAM 40, which may be separate from each CPU and GPU, or may include shared RAM as in the PS5. Each RAM may be separate, or may be integrated as part of the SoC as in the PS5. Further storage is provided as a disk 50, which may be an external or internal hard drive, an external solid-state drive, or an internal solid-state drive as in the PS5.
[0019] The entertainment system 10 may transmit or receive data via one or more data ports 60. Such ports may be USB ports, Ethernet ports, WiFi ports, etc. It may also optionally receive data via an optical drive 70.
[0020] Interaction with the system is typically provided using one or more handheld controllers 80 (e.g., DualSense® in the PS5).
[0021] Typically, audio / visual output from entertainment system 10 is provided via one or more A / V ports 90 or one or more wireless data ports 60 .
[0022] Where components are integrated, they may be connected using dedicated data links or buses 100 as needed.
[0023] An example of a device that displays images output from the entertainment system 10 is a head-mounted display "HMD" 802 worn by a user 800.
[0024] Such an entertainment system 10 may be used to consume motion capture-generated content or to generate motion capture data for, for example, driving a user avatar in a game or visual social environment. In addition to such "live" capture scenarios, such motion capture performances may be used by game developers or film directors to capture an actor's performance for a game character or to transfer an actor or performer into a virtual environment. Again, the terms "user," "actor," "performer," etc., used herein may be used interchangeably unless otherwise specified.
[0025] 2, some embodiments of the present disclosure propose a dense markerless tracking scheme to reduce or prevent tracking drift of these points relative to a 3D morphable model (e.g., face or full body) while typically tracking 100 or more points, which may be selectively tailored to the specific face / body of the currently captured performer.
[0026] This scheme involves several steps.
[0027] In a first optional initialization step 200, a 3D morphable model (3DMM) is trained.
[0028] A 3DMM is a mathematical 3D model of a face that is used to keep tracked points in a location that anatomically matches a person's facial expression. Similarly, the 3DMM may be (or comprise) a model of a human body. Similarly, if an animal is captured, the animal's face and / or body may be used.
[0029] The 3DMM may fit facial expressions and (optionally) facial geometry.
[0030] Facial expressions are modeled using a combination of blendshapes, while facial shape is optionally modeled using eigenvectors obtained after performing principal component analysis (PCA) on a training data set (a set of face meshes with neutral expressions).
[0031] The 3DMM is then optionally tuned to the facial shape of a particular actor as follows: To tune the 3DMM in step s210, a neutral image of the actor (e.g., a neutral face or simply an upright body) is used. As mentioned above, a PCA-based model is previously trained using a facial dataset (e.g., a neutral synthetic face) to obtain these eigenvectors (so-called eigenfaces). PCA parameters ("tuning parameters" 202) are then determined, representing the combination of eigenvectors that best approximates the actor's face.
[0032] During initialization, the set of weights that deform the base mesh used in the 3DMM is modified based on the PCA parameters, resulting in a mesh that better fits the actor's initial face. The modified mesh is then saved as the base mesh onto which facial deformations (blendshapes) for this particular actor's video sequence are fitted.
[0033] The facial expression fitting process, which includes a deep facial feature detector that detects special points 204 for 3DMM fitting in step s220, is described below.
[0034] As mentioned above, the initialization step may also use a deep face detector in step s220 to detect special points 204 for 3DMM fitting.
[0035] The deep appearance detector may be a dedicated hardware module or may be the CPU and / or GPU of the entertainment device operating under suitable software instructions.
[0036] A deep face detector detects specific keypoints in an input image. This module is typically a collection of detectors (one for detecting keypoints around the eyes, one for detecting keypoints around the chin, one for detecting keypoints around the lips, and other detectors for detecting salient facial features such as eyebrows (if trained independently of the eyes), nose, ears, lip segments (for lip sync), the entire face, gaze, etc. Similarly, for body tracking, the detection may be for specific limbs, hands, legs, torso, etc. More generally, a deep face detector detects salient visual features of the tracked target.
[0037] These detectors can be provided by template matching, but are preferably deep learning models trained to extract visual features (keypoints) from the respective parts of the face or body. Typically, these keypoints cover part or all of the face (and / or body) and form special points 204.
[0038] Typically, the deep face detector generates a special point 204 between 100 and 500 on the face without the need for make-up markers or the like.
[0039] Preferably, each detector uses its own deep learning model, but alternatively, these models may be trained on multiple parts of the face or body, or on the entire face or body.
[0040] In many cases, after selective initialization, motion capture may begin. An input image frame 203 is provided to a deep facial expression detector to identify special points 204. This may be performed in the same manner as the selective neutral image 201 described above.
[0041] Optionally, the input image frame may be a stereoscopic image. In this case, a depth image may be generated by applying a stereoscopic algorithm to the left / right images in optional step s230. Then, to enable 3D tracking of specific points on the face or body surface, the specific points may be raised according to the depth values of corresponding points in the depth image. This improves the fitting result of the 3DMM and outputs 3D tracking data.
[0042] In step s240, the optical flow module calculates the trajectory of the optical flow over successive input image frames 203. The optical flow is initialized to track a particular location on the face (or body), for example, from the first input frame 203 in a given input sequence.
[0043] Typically, the specific locations are at least a subset of the special points 204 identified by the deep face detector. Thus, typically, in step s240, the optical flow module tracks some or all of the special points in successive input frames. However, alternatively or additionally, the optical flow module can track points independent of the special points 240. Thus, more generally, such points can be referred to as "flow points." They may or may not exactly coincide with the special points 240.
[0044] The output is a dense set of trajectories 206 (eg, tracked flow points), typically on the order of 100-500 trajectories.
[0045] The optical flow module may be a dedicated hardware module or may be the CPU and / or GPU of the entertainment device operating under suitable software instructions.
[0046] The problem with optical flow tracking is that the tracking can drift. To mitigate this, the optical flow is checked / corrected using a 3D morphable model.
[0047] The 3D morphable models themselves are typically linear blendshape-based models of the human face, such as those used in animation art. These blendshapes are usually hand-crafted by artists as a library of 3D offsets that are applied on top of a neutral-face base mesh and may be individually adjusted to fit the look of a particular actor.
[0048] A blendshape model contains multiple facial expressions, or blendshape targets. A facial expression is a linear combination of these. This is therefore similar to constructing a given face from so-called eigenfaces. In fact, blendshape targets can be chosen in a similar way from a principal component analysis of facial expression training images.
[0049] The 3D morphable model can be maintained, fitted, and selectively adjusted / tuned using a 3DMM module, which may be a dedicated hardware module or the CPU and / or GPU of the entertainment device operating under suitable software instructions.
[0050] In step s250, the 3DMM is fitted to the visual features of the current facial expression extracted by the deep face detector (e.g., some or all of the special points 204). The fit is optimized using a nonlinear least squares method, which minimizes the projection error of a subset of the model's 3D facial vertices to the specific feature locations computed from the input image using the deep face detector.
[0051] The algorithm can optimize two types of parameters. a. Facial Expression: A set of blendshape weights that determine the actor's facial expression at a given frame. The blendshape weights are a set of scalars that scale the offset (i.e., the amount of deformation applied to the base mesh). This deformation moves keypoints on the mesh. The correspondence between these keypoints and detected special points 204 is used to determine when the resulting facial expression best matches the actor's facial expression in the image. b. Camera Pose: A 6DOF transformation that models the orientation of the camera relative to the actor. These camera parameters allow the image projection to be calculated and can optionally be used to model camera vibrations or movements (which can adversely affect tracking results).
[0052] Thus, the 3DMM fitting step determines a 3D morphable model that fits the actor's facial expressions and relative poses (so as to determine some or all of the special points).
[0053] As mentioned above, the base mesh modified by the blendshape may optionally use adjustment parameters that morph the model to the anatomical proportions of the actor's face (if this was determined in the initialization step).
[0054] In particular, fitting a 3DMM to some or all of the special points on an actor's face can produce a model that closely approximates the actor's current, real-world expression, while the key points are uncorrupted by noise, classification errors, and other data outliers that may arise in detecting the special points.
[0055] Thus, fitting the 3DMM to some or all of the special points 204 on the actor's face generates a normalized version of the actor's facial expression that best matches the collective representation of the special points while removing irregularities.
[0056] Therefore, if some of the special points are misclassified / misplaced such that they are placed in a location that does not match the face that can be expressed using the 3DMM's blendshapes (e.g. some lips take on an unexpected shape), these outliers will not show up in the 3DMM because they are constrained by the allowed range of expressions.
[0057] Similar principles apply to whole-body 3DMM.
[0058] Optionally, parameters of the model (e.g., representations of selected blendshapes or their relative contributions) may be output separately to drive animation or other processes, if desired.
[0059] The fitted 3DMM indicates that the identified special points according to the blendshapes best match possible or legitimate facial expressions. Conversely, this facilitates identifying identified special points when they represent facial parts in locations that are considered impossible or invalid (e.g., if the nose shadow is partially identified as being part of the nose, the nose will immediately change size).
[0060] Similarly, the 3D morphable model may be used as a normalized representation of the special points to correct for optical flow drift as follows:
[0061] Once the 3DMM has been fitted and the optical flow calculated, the drift correction module can operate in step s260.
[0062] The drift correction module may be a dedicated hardware module or may be the CPU and / or GPU of the entertainment device operating under suitable software instructions.
[0063] In step s260, the optical flow trajectory is checked and / or corrected using one or more of the following heuristics: i. Flow points cannot be placed very far from the projected 3DMM points. This is directly true for flow points that are special points 204. On the other hand, for flow points that are tracked separately from the special points, the spatial relationship between such flow points and their nearby 3DMM points and / or special points can optionally be maintained to satisfy a similar condition. -This constraint prevents the tracked flow points from drifting to positions that are not aligned with possible / legitimate facial expressions formed by the 3DMM model, keeping the optical flow tracker correct. If a flow point is determined to drift more than a predetermined distance from the projected 3D MMM point (or from its predicted spatial relationship to such a point), the flow point can be returned to the projected / predicted point. Alternatively, the flow point can be returned to an interpolated point between adjacent flow points (if this interpolated point satisfies this heuristic). Alternatively, the flow point can be returned to an interpolated point between the current flow and the projected 3D MMM point (if this interpolated point satisfies this heuristic). Such interpolation modifies the flow point, but reduces any apparent visual discontinuity caused by this modification. ii. Flow points within a contour do not intersect each other (e.g., outer / inner lip contour, outer / inner eye contour, etc.). Heuristic i) helps to keep flow points within a given distance from the projected 3DMM points. However, if the given distance is large enough, visually destructive errors can occur. In particular, lips and eyes are physically very close and may contain subjectively important features. These flow points may be placed close to the projected 3DMM points, but optionally, it is desirable for them to satisfy the additional constraint that they do not intersect with each other as described above (in other words, the lines or contours they represent do not intersect with each other except within a given small point tolerance defined by the 3DMM). - A similar correction approach as in heuristic i) may be used. iii. Flow points within a contour must maintain consistent distances and directions. Like heuristic ii), heuristic i) helps to keep the flow points within a certain distance from the projected 3D MMM points, but apart from this, there are certain relationships between facial features that must be kept consistent. For example, typically the middles of the upper and lower eyelids must be parallel. On the other hand, with regard to the body, typically the limbs must maintain their visible length and be the same length as each other. - A similar correction approach as in heuristic i) may be used.
[0064] When we say changing the location of a flow point, this typically also means changing the trajectory of the flow point, either directly (by changing the track value) or indirectly (by modifying the position of the flow point before calculating updated tracking, or optionally by retrying the tracking process with the modified position information).
[0065] Thus, by using special points on the face or body (typically identified by one or more deep learning facial feature detectors), a 3D morphable model of the actor's face can be generated that corresponds to the facial expressions defined by these special points. This 3DMM can be used to correct for optical flow tracking drift of the points (typically, but not limited to, some or all of the special points), thereby keeping these points consistent with possible or valid facial expressions defined by the 3DMM. Optionally, more subtle effects of tracking drift between facial or body features can also be corrected.
[0066] The resulting checked / corrected trajectory (herein referred to as the "Mirror Trajectory" 208) is output to drive performance capture of the selected behavior. This may be driving a live avatar in a game or social virtual environment, inputting physical movements into a dance game or the like (playing or recording a reference performance for later comparison), capturing a replay of a performance by a video game character, capturing a performance used in a movie or TV show, or similar use of optical flow trajectories 208, special points 204, and / or 3D MMM facial parameters 207.
[0067] Referring to FIG. 2, in a summary embodiment of the present disclosure, the point tracking method includes the following steps. In a first step, successive input image frames are received from a sequence of input image frames 203 that contain a tracked object (eg, an actor). In a second step s220, for each of the input image frames, a number of facial features are detected (for example, using one or more deep learning systems). In a third step s230, a 3D morphable model is mapped to the plurality of appearance points. - In a fourth step s240, optical flow tracking of the flow points (which typically includes at least a subset of the face points 204) between successive input image frames is performed. In a fifth step s250, the optical flow tracking is modified with respect to at least the first flow point position responsive to the mapped 3D morphable model.
[0068] It will be appreciated that at least the third and fourth steps may be performed in reverse order or in parallel.
[0069] Those skilled in the art will appreciate that various modifications of the above method (which corresponds to the method of operating the device described herein) are possible without departing from the scope of the present invention, including, but not limited to, the following: In one example of a summary embodiment, the flow points include some or all of the detected facial features points. In one example of a summary embodiment, facial features are detected using one or more machine learning models. In this example, optionally, each group of feature points is detected using a respective machine learning model trained to detect each visual feature of the tracked object. In some examples of the summary embodiment, the tracked object includes a face or a body. In this example, optionally, the 3D morphable model is a linear blendshape based model. Also in this example, optionally, the point tracking method includes adjusting the 3D morphable model to the anatomical proportions of the person depicted in the input image frame. -In one example of an embodiment, the successive input image frames include a pair of stereo images, and the point tracking method includes the steps of generating a depth map from the pair of stereo images, mapping the detected facial points to corresponding depth positions, and mapping a 3D morphable model to the plurality of facial points at the mapped depth positions. In one example of the summary embodiment, the step of correcting the optical flow tracking comprises: i. modifying the position of flow points that are located more than a predetermined distance from a corresponding point of the 3D morphable model in order to reduce that distance; or ii. modifying the position of a flow point if a corresponding point of the 3D morphable model matches the predetermined feature and the position of the flow point causes the predetermined feature to intersect with another predetermined feature; or iii. modifying the position of a flow point when a corresponding point of the 3D morphable model matches a predetermined feature and the position of the flow point causes the predetermined feature to have a positional or directional relationship with other predetermined features that is inconsistent with the predetermined relationship. In one example of a summary embodiment, for a current input image frame: i. Corrected flow tracking data; or ii. facial expression parameters corresponding to the 3D morphable model; or iii. Appearance point data outputting the
[0070] Those skilled in the art will appreciate that the above methods may be performed by conventional hardware (eg, entertainment device 10) or dedicated hardware with suitable software instructions applied.
[0071] Accordingly, necessary adaptations to existing components of conventional equivalent devices may be realized in the form of computer program products. These computer program products comprise processor-executable instructions stored on a non-transitory, machine-readable medium (e.g., floppy disk, optical disk, hard disk, solid-state disk, PROM, RAM, flash memory, or any combination of these or other storage media). Alternatively, they may be implemented in hardware, such as an ASIC (application-specific integrated circuit), FPGA (field-programmable gate array), or other configurable circuitry suitable for use in adapting conventional equivalent devices. Alternatively, such computer programs may be transmitted via data signals over a network (e.g., Ethernet, wireless network, the Internet, or any combination of these or other networks).
[0072] Thus, in one example of a summary embodiment, a point tracking system comprises:
[0073] First, a video input module (e.g., by suitable software instructions) configured to receive successive input image frames 203 from a sequence of input image frames containing a tracked object (e.g., from the data port 60 of the entertainment device 10, or a pre-recorded audio source such as on an optical drive 70 or data drive 50 using the A / V port 90 and / or GPU 30).
[0074] Second, a face detection module (eg, CPU 20 and / or GPU 30) configured (eg, by suitable software instructions) to detect a plurality of face points for each of the input image frames.
[0075] Third, a 3D morphable model module (eg, CPU 20 and / or GPU 30) configured to map (eg, by suitable software instructions) a 3D morphable model to a plurality of facial points.
[0076] Fourth, an optical flow module (eg, CPU 20 and / or GPU 30) configured (eg, by suitable software instructions) to perform optical flow tracking of flow points between successive input image frames.
[0077] Fifth, a drift correction module (e.g., CPU 20 and / or GPU 30) configured to correct (e.g., by suitable software instructions) the optical flow tracking with respect to at least the first flow point position responsive to the mapped 3D morphable model.
[0078] Those skilled in the art will appreciate that various modifications of the above system (equivalent to the method described herein) are possible without departing from the scope of the present invention, including, but not limited to: In one example of a summary embodiment, the flow points include some or all of the detected facial features points. Summary In one example embodiment, respective groups of feature points are detected using respective machine learning models trained to detect respective visual features of the tracked object. In one example of a summary embodiment, the drift correction module comprises: i. modifying the position of flow points that are located more than a predetermined distance away from corresponding points of the 3D morphable model in order to reduce the distance; or ii. modifying the position of a flow point when a corresponding point of the 3D morphable model matches the predetermined feature and the position of the flow point causes the predetermined feature to intersect with another predetermined feature; or iii. modifying the position of a flow point when a corresponding point of the 3D morphable model matches the predetermined feature and the position of the flow point causes the predetermined feature to have a positional or orientation relationship with another predetermined feature that is inconsistent with the predetermined relationship; The optical flow tracking is adapted to be corrected using In one example of a summary embodiment, the method comprises, for a current input image frame: i. Corrected flow tracking data; or ii. facial expression parameters corresponding to said 3D morphable model; or iii. Appearance point data and outputting the result.
[0079] The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. Those skilled in the art will recognize that the present invention can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative and not limiting of the scope of the present invention and the claims that follow. The present disclosure, including any identifiable variations of the above teachings, defines in part the scope of the claim terms. The subject matter of the invention is not dedicated to the public.
Claims
1. receiving successive input image frames from a sequence of input image frames including a tracked object; detecting a plurality of facial features for each of the input image frames; mapping a 3D morphable model to the plurality of facial points; performing optical flow tracking of flow points between successive input image frames; modifying the optical flow tracking with respect to at least the first flow point positions responsive to the mapped 3D morphable model; A point tracking method comprising:
2. The point tracking method of claim 1 , wherein the flow points include some or all of the detected facial features.
3. The point tracking method of claim 2 , wherein the facial points are detected using one or more machine learning models.
4. 4. The point tracking method of claim 3, wherein each group of feature points is detected using a respective machine learning model trained to detect each visual feature of the tracked object.
5. The tracking target is i. face, or ii. body 5. The point tracking method according to claim 1, further comprising:
6. The point tracking method of claim 5 , wherein the 3D morphable model is a linear blendshape-based model.
7. The point tracking method of claim 5, further comprising adjusting the 3D morphable model to the anatomical proportions of a person depicted in the input image frame.
8. the successive input image frames include a stereo image pair; This point tracking method: generating a depth map from the stereo image pair; mapping the detected feature points to corresponding depth positions; mapping the 3D morphable model to a plurality of facial points at the mapped depth locations; The point tracking method of claim 1, further comprising:
9. The step of correcting the optical flow tracking includes: i. modifying the positions of flow points that are located more than a predetermined distance from a corresponding point of the 3D morphable model in order to reduce that distance; or ii. modifying the position of a flow point if the corresponding point of the 3D morphable model matches a predetermined feature and the position of the flow point causes the predetermined feature to intersect with another predetermined feature; or iii. modifying the position of a flow point when a corresponding point of the 3D morphable model matches a predetermined feature and the position of the flow point causes the predetermined feature to have a positional or orientation relationship with another predetermined feature that is inconsistent with a predetermined relationship; The point tracking method of claim 1, further comprising:
10. For the current input image frame, i. Corrected flow tracking data, or ii. facial expression parameters corresponding to said 3D morphable model; or iii. Appearance point data 2. The point tracking method according to claim 1, further comprising the step of outputting:
11. A computer program comprising computer-executable instructions for carrying out the point tracking method of claim 1.
12. a video input module configured to receive successive input image frames from a sequence of input image frames including a tracked object; a face detection module configured to detect a plurality of face points for each of the input image frames; a 3D morphable model module configured to map a 3D morphable model to the plurality of facial points; an optical flow module configured to perform optical flow tracking of flow points between successive input image frames; a drift correction module configured to correct the optical flow tracking with respect to at least the first flow point positions responsive to the mapped 3D morphable model; A point tracking system comprising:
13. The point tracking system of claim 12, wherein the flow points include some or all of the detected facial features.
14. 14. The point tracking system of claim 12 or 13, wherein each group of feature points is detected using a respective machine learning model trained to detect each visual feature of the tracked object.
15. The drift correction module: i. modifying the positions of flow points that are located more than a predetermined distance from a corresponding point of the 3D morphable model in order to reduce that distance; or ii. modifying the position of a flow point if the corresponding point of the 3D morphable model matches a predetermined feature and the position of the flow point causes the predetermined feature to intersect with another predetermined feature; or iii. modifying the position of a flow point when a corresponding point of the 3D morphable model matches a predetermined feature and the position of the flow point causes the predetermined feature to have a positional or orientation relationship with another predetermined feature that is inconsistent with a predetermined relationship; 13. The point tracking system of claim 12, configured to correct optical flow tracking using