System and method for tracking multiple deformable objects in self-cented video

Abstract

Systems and methods for tracking a plurality of variable objects in a self-centric video are disclosed. A computer-implemented method may include receiving a video stream having a plurality of frames; detecting an object within a selected frame of the video stream; decomposing the object in the selected frame into a plurality of blocks; associating a subset of the plurality of tiles with candidate tiles within a subsequent frame of the video stream; and determining a position of the object within a subsequent frame of the video stream based at least in part on the position of the candidate block within the subsequent frame. Various other methods, systems, and computer readable media are also disclosed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. non-provisional patent application No. 18 / 353,237, filed on July 17, 2023. Technical Field

[0003] The present disclosure relates generally to systems and methods for object tracking in video, including systems and methods suitable for tracking multiple deformable objects in egocentric video. Background Art

[0004] Efficiently and reliably performing basic computer vision processing tasks (e.g., object detection, tracking, and segmentation) using on-device processors can be difficult for egocentric videos from wearable cameras. Summary of the invention

[0005] The disclosed systems and methods can handle fast object deformation and occlusion as well as ego-motion in a reliable and computationally efficient manner. The systems and methods described herein can include a motion disentanglement network (e.g., which distinguishes global camera motion from local motion of a tracked object), a tile association network (which divides a tracked object into multiple tiles to be matched individually between frames), and / or a tile storage network (for saving features of tiles over many frames, which may be helpful in situations where a portion of a tracked object is temporarily occluded). BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings illustrate several exemplary embodiments and are a part of the specification. Together with the following description, these drawings illustrate and explain various principles of the present disclosure.

[0007] Figure 1A , Figure 1B and Figure 1C is an illustration of examples of deformation, occlusion, and ego-motion in a video.

[0008] Figure 2 is a diagram showing an example object being decomposed into multiple patches.

[0009] Figure 3 is an illustration of an example system for tracking multiple deformable objects in egocentric videos.

[0010] Figure 4 is a diagram of an example motion disentanglement system.

[0011] Figure 5 is an illustration of example blocks propagating across video frames.

[0012] Figure 6is an illustration of an example system for tracking object regions across video frames.

[0013] Figure 7 is an illustration of an example method for tracking multiple deformable objects in egocentric videos.

[0014] Figure 8 is an illustration of exemplary augmented reality glasses that may be used in conjunction with embodiments of the present disclosure.

[0015] Fig. 9 is an illustration of an exemplary virtual reality headset that can be used in conjunction with embodiments of the present disclosure.

[0016] In all figures, the same reference numerals and descriptions refer to similar but not necessarily identical elements. Although the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown in the drawings by way of example and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the specific forms disclosed. Instead, the present disclosure covers all modifications, equivalents, and alternatives that fall within the scope of the appended claims. DETAILED DESCRIPTION

[0017] Wearable cameras have become a rapidly growing trend in personal technology, changing the way users record their lives and interact with the world. These unobtrusive devices provide users with the ability to capture hands-free, point-of-view footage in a variety of scenarios, including augmented reality, adventure experiences, law enforcement duties, and content creation. Using wearable devices, complex tasks can potentially be performed through the analysis of visual signals. For example, smart glasses can alert users if a child runs out of view in a supermarket, or record the time and place a pet was last seen.

[0018] Wearable cameras have become more powerful with increasing battery capacity, sensor size, onboard memory capacity, and more sophisticated in-device processors, and they can provide large amounts of high-quality video every day. However, these cameras can produce highly unstable streams due to their egocentric field of view. Especially in the case of wearable cameras worn on the user's head (e.g., in the case of smart glasses), large ego-motions caused by the wearer's head movements can often be dramatic, unpredictable, and largely uncorrelated with the object motion.

[0019] Using on-device processors to efficiently and reliably perform basic computer vision processing tasks (e.g., object detection, tracking, and segmentation) can be difficult for egocentric videos from wearable cameras. Multiple object tracking algorithms can be stressed by the changing egocentric viewpoint in addition to being challenged by occlusions, deforming shapes, and multiple visually similar objects.

[0020] The present disclosure generally relates to systems and methods for object tracking in video, including systems and methods suitable for tracking multiple deformable objects in egocentric video. Thus, for example, the disclosed systems and methods can handle rapid object deformations and occlusions as well as ego-motion in a reliable and computationally efficient manner. The systems and methods described herein can include a motion disentanglement network (e.g., which distinguishes global camera motion from local motion of a tracked object), a tile association network (which divides a tracked object into multiple tiles for separate matching between frames), and / or a tile storage network (for saving features of tiles over many frames, which may be helpful in situations where a portion of a tracked object is temporarily occluded).

[0021] As will be explained in more detail below, the various embodiments of the present disclosure can improve the functionality of a computer by increasing the computer's computer vision processing capabilities. In addition, these embodiments can represent an advancement in the field of computer vision processing.

[0022] According to the general principles described herein, the features of any of the embodiments described herein can be used in combination with each other. These and other embodiments, features and advantages will be more fully understood after reading the following detailed description in conjunction with the accompanying drawings and claims.

[0023] In some examples, the systems and methods described herein may include and / or implement an end-to-end trainable method for tracking multiple deformable objects in egocentric videos. In some examples, the systems and methods may represent the deformable objects as a set of patches, which the systems and methods may then detect and track individually and then reassemble at updated positions in subsequent frames.

[0024] In some examples, the systems and methods described herein may include and / or implement a block association network that is capable of finding different local regions (e.g., a dog's head and feet) to locate and associate objects in later frames. In addition, in some examples, the systems and methods may save in memory deformed and / or occluded blocks that cannot be matched in the current frame for future association when the areas of the object corresponding to these blocks appear clearly again. Thus, the systems and methods may successfully track even severely deformed objects (e.g., a dog that is deformed by turning quickly while playing). In addition, in some examples, the systems and methods described herein may include and / or implement a motion disentanglement network that compensates for object and camera motion, narrows the search range, and further improves the performance of the systems and methods.

[0025] The systems and methods described herein may include and / or implement any of a variety of components, modules, and / or subsystems for tracking objects across video frames. In one example, these systems and methods may include a motion disentanglement module (e.g., a motion disentanglement network (motion disentanglement network, MDN)), a patch association module (e.g., a patch association network (patch association network, PAN)) and / or a patch storage component (e.g., a patch memory network (patch memory network, PMN)).

[0026] As used herein, the term "object" may refer to any complete entity that can be tracked within a video. In some examples, the term "object" may refer to an entity that has a coherent form over time. Furthermore, in some examples, the term "object" may refer to an entity that can be deformed (e.g., changed shape) over time while maintaining its coherent form. Additionally or alternatively, the term "object" may refer to an entity that can move within an environment. In some examples, the term "object" may refer to an entity that deforms and / or moves itself within an environment (e.g., an animal). In some examples, the term "object" may refer to the entirety of an entity. In some examples, the term "object" may refer to only a portion of an entity (e.g., it may refer to only a person's face, only a person's head, etc.).

[0027] A motion disentanglement module (e.g., MDN) can estimate the motion flow between two frames (e.g., two consecutive frames). For example, the motion disentanglement module can be used to control global camera motion before estimating local object motion (thereby making the systems and methods described herein more robust and effective in tracking objects when the video includes a large amount of self-motion). As used herein, the term "self-motion" generally refers to changes in video perspective (e.g., caused by movement of the camera) and / or apparent movement of objects within a video frame due to such changes in perspective. In some examples, the self-motion described herein can be associated with movement of a user-mounted camera (e.g., a head-mounted camera). As will be explained in more detail below, in some examples, the motion disentanglement module can isolate the local motion of an object from the egocentric global motion of the video at least in part by analyzing the differences between frames to determine the egocentric global motion. Additionally or alternatively, in some examples, the motion disentanglement module can isolate the object's local motion from the egocentric global motion at least in part by receiving data from a motion sensor (e.g., an accelerometer and / or gyroscope) that detects egocentric global motion.

[0028] A block association module (e.g., PAN) can track objects even if they are deformed or partially obscured by dividing them into multiple blocks and locating each block by finding matching blocks in an upcoming frame. As used herein, the term "block" can refer to any image (and / or a set of features extracted therefrom) obtained from the location of an object in a frame (e.g., within a bounding box of an object in the frame) - or from a potential location of an object in a frame. In some examples, a block can be a closed shape. In some examples, a block can be a mosaic block (i.e., a shape that is mosaicked and / or can be mosaicked). In some examples, all blocks generated, compared, matched, and / or combined by the systems and methods described herein can have the same size, shape, size, and / or orientation. In one example, a block can be a rectangle. In some examples, the term "block" can refer to an image having a certain minimum size (e.g., such that one or more complex features can be identified within the block and / or extracted from the block). Thus, for example, the term "block" can refer to an image that is larger than one pixel in size. In various examples, the term "block" may refer to a rectangular image having no side shorter than 3 pixels, no side shorter than 5 pixels, no side shorter than 10 pixels, or no side shorter than 20 pixels.

[0029] A block storage module (e.g., PMN) can save (and in some examples, update) blocks of tracked objects and / or feature embeddings of blocks over a long time window (e.g., at least one frame in which a given block does not appear). In some examples, the block storage module can include a transformer network through which the systems and methods described herein can use historical block features for long-term block association.

[0030] As will be explained in more detail below, the systems and methods described herein can efficiently and reliably track objects within a video under a variety of conditions. As an example, Figure 1A , Figure 1B and Figure 1C Provides example illustrations of deformation, occlusion, and ego-motion in videos.

[0031] So, for example, Figure 1A An example of tracking a dog 112 across video frames 110(a), 110(b), and 110(c) is shown. Figure 1A As shown, the image of dog 112 experiences deformation across frames 110(a)-110(c) due to, for example, movement of various parts of dog 112 relative to each other and due to the projection of the image of dog 112 on frames 110(a)-110(c) changing as the position and angle of dog 112 relative to the camera changes. As used herein, the term "deformation" may refer to any change in the image of an object across two or more frames of a video (e.g., due to changes in the shape of the object and / or changes in the perspective from which the image of the object is projected onto the frames).

[0032] also, Figure 1B An example of tracking dogs 122, 124, and 126 across video frames 120(a), 120(b), and 120(c) is shown. Figure 1B As shown, although dogs 122, 124, and 126 are all fully visible in frame 120(a), in subsequent frames, dogs 124 and 126 are at least partially occluded (e.g., by dog ​​122). As used herein, the term "occlusion" may refer to at least a portion of an object disappearing from view (e.g., including but not limited to situations where an intervening object at least partially blocks the view of the occluded object).

[0033] also, Figure 1C An example of tracking dogs 132 and 134 across video frames 130(a), 130(b), and 130(c) is shown. Figure 1CAs shown, although dogs 132 and 134 may be relatively stationary in their environment across video frames 130(a) to 130(c), their relative positions within video frames 130(a) to 130(c) may change rapidly and erratically due to self-motion. As used herein, the term "self-motion" may refer to any movement of an image within a video that is caused by movement (e.g., translation and / or rotation) of a camera (rather than movement due to, for example, movement of an object within the environment). In some examples, the term "self-motion" may refer to the apparent motion of an object caused by camera movement associated with a person-worn camera. In some examples, the term "self-motion" may refer to such camera movement associated with a head-mounted camera. In some examples, the term "global motion" may refer to motion within a video frame of an image and / or an element of an image that is caused by and / or is related to self-motion, while "local motion" may refer to motion within a video frame of an image and / or an element of an image that excludes and / or controls global motion and / or is not caused by self-motion.

[0034] As will be described in more detail below, the systems and methods described herein can successfully track objects such as dogs 112, 122, 124, 126, 132, and 134 (e.g., Figure 1A , Figure 1B ,and Figure 1C shown).

[0035] Figure 2 2 is a diagram of an example object 202 being decomposed into a set of blocks 200. Figure 2 As shown, one or more of the systems and methods described herein may divide an object 202 into a set of tiles 200. As will be explained in more detail below, the systems and methods may then track the object 202 across frames at least in part by detecting features of the tiles across frames.

[0036] Figure 3 is an illustration of an example system 300 for tracking multiple deformable objects in an egocentric video. Figure 3 As shown, system 300 may take as input a video stream including, for example, frame 302 (e.g., at time t-1) and frame 304 (e.g., at time t). In one example, the systems and methods described herein may detect objects (e.g., two dogs) within frame 304 that have previously been detected in frame 302. Feature extraction module 306 may have previously extracted feature 308 from frame 302 and may extract feature 309 from frame 304.

[0037] The systems and methods described herein may extract features in any suitable manner and in any suitable format. As an example, the feature extraction module 306 may convert the input image into a feature map pyramid (e.g., a feature map pyramid Among them I (t) is a frame at time t, and l is a level of the pyramid). Thus, for example, features 308 and 309 may each include a hierarchical representation of frames 302 and 304, representing frames 302 and 304 at different levels of detail and / or abstraction, respectively. In one example, feature extraction module 306 may include a deep layer aggregation (DLA) network. For example, feature extraction module 306 may include a DLA-34 backbone network (e.g., a DLA network with 34 layers). In one example, feature extraction module 306 may include a weight sharing backbone network.

[0038] The motion disentanglement module 310 (eg, MDN) may take features 308 and 309 as input to estimate the global motion between frames 302 and 304. As shown below (eg, with respect to Figure 4 ) will be explained in more detail, in some examples, the motion disentanglement module 310 can estimate the pixel-level motion flow between the frames 302 and 304. For example, the motion disentanglement module 310 can estimate the pixel-level motion flow between the frames 302 and 304. In one example, as previously discussed, the features 308 and 309 as inputs to the motion disentanglement module 310 may be pyramid feature maps (e.g., and ).

[0039] After estimating the global motion between frames 302 and 304, the systems and methods described herein may then propagate the position of the object to frame 304 (e.g., such that bounding boxes 322 and 324 are applied to frame 304), based at least in part on the estimated global motion and at least in part on the position of the previously identified object in frame 302 (e.g., bounding boxes 318 and 320). It will be appreciated that these systems and methods may include an error tolerance in the initial propagation of the position of the object to frame 304 (e.g., such that the initial bounding box is larger than the bounding box used for frame 302).

[0040] The systems and methods described herein may decompose the image within bounding boxes 322 and 324 into a plurality of tiles after initially having propagated bounding boxes 318-320 from frame 302 to bounding boxes 322 and 324 of frame 304. For example, the image within bounding box 322 may be decomposed into a plurality of tiles 312. The systems and methods may have previously decomposed the image within bounding boxes 318 and 320 into a plurality of tiles. For example, the image within bounding box 318 may have been decomposed into a plurality of tiles 316.

[0041] Continuing with the example of the dog on the left side of frames 302 and 304, the systems and methods described herein can provide multiple tiles 312 and multiple tiles 316 as inputs to a tile association network (PAN) 330. As will be described in more detail below, PAN 330 can detect and associate objects by matching subdivided tiles between frames 302 and 304. In some examples, detection and association can be performed by a single network to improve both positioning accuracy and identity association accuracy. PAN 330 can generate a tile map 326 as an output. The systems and methods described herein can determine bounding boxes 332 and 334 of the two dogs (which may be more precise and accurate than bounding boxes 322 and 324) based at least in part on tile map 326.

[0042] The systems and methods described herein may also identify unmatched tiles (e.g., from the output of PAN 330) and save them for later use. For example, PAN 330 may save unmatched tiles 314 in a memory buffer (e.g., a fixed-length annular tile memory buffer) to track objects based on changing deformation and / or occlusion conditions in future frames. Thus, the systems and methods described herein may include buffered tiles and new tiles from newly seen frames when PAN 330 performs tile matching in subsequent time steps.

[0043] Figure 4 is a diagram of an example motion disentanglement system 400. Figure 4 As shown, the motion disentanglement system 400 may include a motion disentanglement network 410. In some examples, the motion disentanglement network 410 may correspond to Figure 3 The systems and methods described herein may provide a pyramid feature map 402 and pyramid feature map 404 As input to the motion disentanglement network 410. The motion disentanglement network 410 can then estimate the optical flow between the two frames based on pyramid processing, warping and / or the use of cost volumes. In some examples, the cost volume search range can include the entire frame (e.g., to better handle large camera motions for egocentric videos).

[0044] In one example, The estimated optical flow at pyramid level l can be represented as: and To construct a cost volume, the cost volume captures the cost of associating each pixel with all pixels at the next frame, as shown in equation (1):

[0045]

[0046] Where T is the transpose operator and is a one-dimensional feature vector extracted from the pixel. The cost volume CV0 is fed to the optical flow estimator and the context network and produces It is mainly responsible for large camera motions and is self-supervised, as described in more detail below.

[0047] In some examples, rather than performing a global search over the entire image for any level l>=0, the motion disentanglement system 400 can use a different approach by constructing a cost volume that captures the cost of associating each pixel only with neighboring pixels of the next frame, as shown in equation (2):

[0048]

[0049] in, yes In the upsampling flow Similar to level 0, CV l Feed to the optical flow estimator and context network and produce This is continued until layer L-1 and produces the full-resolution stream The recursive process.

[0050] The systems and methods described herein may determine a feature map (eg, according to one or more of the steps described above) and Estimated flow and the subject being tracked in I (t-1) These systems and methods can use feature maps of two frames, the flow between frames, and the bounding boxes of previous frames to extend the track to I by simultaneously detecting and tracking one or more objects. (t) .

[0051] In some examples, the systems and methods described herein may first use a region of interest (ROI) alignment operation to obtain each tracked object in the I (t-1) Features in Where n is a hyperparameter that defines the feature resolution, and d is the number of channels of the ROI feature embedding. By doing so, these systems and methods can obtain features that are not only translation invariant but also have uniform feature size. Therefore, in some examples, these systems can perform the ROI alignment operation by dividing the region of interest into n×n superpixels, where each superpixel is a small image patch. Furthermore, in some examples, the systems and methods can convert each block into a d-dimensional feature embedding, which can be represented as Where i is the block index.

[0052] Figure 5 is a diagram of an example block propagating across a video frame. For example, the systems and methods described herein can track (t-1) (e.g., frame 502) to I (t) (e.g., frame 504) (and therefore track objects composed of these blocks). Figure 5 As shown, an object decomposed into a plurality of blocks may be located within a bounding box 510 (denoted by denoted). The systems and methods described herein can locate in I (t) and obtain the location on For example, these systems and methods can establish potential search area 512 within frame 504. In one example, these systems and methods can be performed by using stream The center point of the bounding box 510 is propagated to establish a potential search area 512. The systems and methods can then enlarge the size of the bounding box 510 by a predetermined ratio α>1 to produce a potential search area 512 that is large enough to capture the expected object even when it is enlarged or stretched between frames. The systems and methods can then subdivide the potential search area 512 into αn×αn superpixels, which can produce feature

[0053] As will be explained in more detail below, the systems and methods described herein can detect and track objects from frame 502 to frame 504 by associating each block in frame 502 with a candidate block in frame 504 (e.g., within potential search area 512). Once the systems have located matching blocks within frame 504, the systems and methods can associate a minimum bounding box 514 (defined by In some examples, these systems and methods can refine the initial tracked object position to generate a bounding box 516 (represented by ) of the object in frame 504. express).

[0054] Figure 6 is a diagram of an example system 600 for tracking object blocks across video frames. Figure 6 As shown, system 600 may include a block association network (PAN) 608 and a block storage network (PMN) 610. In some examples, PAN 608 may correspond to Figure 3 The PAN 330 and PMN 610 may correspond to Figure 3 PMN 319. In some examples, PAN 608 and PMN 610 may represent and / or include a converter network. System 600 may also include a frame i (T) Block 604 (denoted as p (T) ), from frame i (t-1) Block 606 (denoted as P (t-1) ) and from the memory block 606 (denoted as P mem ). Some of the blocks 606 may match one or more of the blocks 604 (e.g., become matching blocks 613), while some of the blocks 606 may become unmatched blocks 614. Figure 6 As shown, 'Q', 'K', and 'V' may represent query, type, and value inputs to the PAN 608 and PMN 610 (eg, an attention mechanism to a transformer network).

[0055] PAN 608 can send the (T-1) The block from frame I (t) In some examples, PAN 608 can associate blocks in a manner that takes into account cross-block relationships (rather than, for example, processing each individual block independently). Thus, PAN 608 can implement a method that takes into account blocks of an object that belong to the same object. For example, for each object, PAN 608 can use a transformer encoder-decoder network frame g(·,·) to calculate (e.g., from frame I (T-1) and / or blocks from memory) and each query block (e.g., from frame I (t) A set of correlation scores between all candidate blocks in the search area This generates a score Then, PAN 608 can process the frame I (t-1) The query block and frame I (t)The correlation scores of all candidate blocks in Normalization: For example, PAN 608 may use a softmax operation to normalize each group of association scores.

[0056] After calculating the association scores for all tiles, the system 600 can select matches between tiles based at least in part on these association scores. In one example, the system 600 can use a sub-grid search to select matches between tiles. For example, the system 600 can consider all query tiles with a maximum association score less than a threshold θ as unmatched (e.g., because these tiles may be severely occluded or disappear from view). The system 600 can then mark the remaining tiles as matches, where, for each matched tile, the score Representation block Corresponding to frame I (t) Then, the system 600 can use the normalized correlation score as a weight to perform a weighted sum of all possible grid positions on the frame I. (t) Each matching block in is located.

[0057] Once the system 600 is in frame I (t) After locating the blocks of the object, the system 600 can determine the minimum bounding rectangle of these located blocks as the initial tracking object position. System 600 can The bounding box is considered as a class-specific proposal and further refined based on objectness using a region-based convolutional neural network (e.g., Faster-RCNN). The system 600 can use the regression branch to refine the bounding box and obtain and generate a tracking confidence score The tracking confidence score Both detection confidence and consistency association confidence can be combined. Therefore, the confidence score can carry information about appearance similarity in the temporal domain and help minimize false negatives.

[0058] System 600 can compute the confidence score of a trajectory T of length t as a sequence of boxes detected in past frames such that T t = {B (0) ,B (1) ,…,B (t)}. The system 600 can recursively (e.g., using the chain rule) calculate the confidence of the trajectory T, as shown in equation (3):

[0059]

[0060] in, has two components, one is the regression confidence obtained during box refinement, and the other is the ROI-level association score The system 600 can be implemented by normalizing the correlation scores of all matching blocks within the object. Take the average to calculate the relevance score at the ROI level Therefore, the system 600 can calculate the tracking confidence score as shown in equation (4):

[0061]

[0062] As previously described, the system 600 may include a block storage network (PMN) 610. The PMN 610 may collect a set of blocks P mem to describe the long-term appearance of each tracked object. As described above, PAN 608 may include P in the block association process mem and frame I (t-1) Once PAN 608 has performed the tile association process, tiles 606 may include matching tiles 613 and unmatched tiles 614. Some unmatched tiles 614 may have new appearances (e.g., introduced by deformation or occlusion). Thus, system 600 may use unmatched tiles 614 as queries in PMN 610 to explore the unmatched tiles with all other tiles (including those inherited from previous frames). mem ). System 600 can obtain updated unmatched block features from the output of PMN 610 and include them in P mem In addition, the system 600 can be divided into those features of the partition. mem In some examples, the system 600 can manage P in a first-in-first-out manner. mem , where old blocks can be updated according to a fixed time window Automatically removed.

[0063] In some examples, the systems and methods described herein can track objects, identify when an object has not been successfully tracked, and / or track new objects. For example, the systems and methods can include and / or receive input from an object detector and begin tracking an object detected by the object detector. In some examples, the systems and methods can apply one or more requirements before tracking an object detected by the object detector. For example, the systems and methods can only track newly detected objects whose detection confidence scores are above a predetermined threshold. Additionally or alternatively, the systems and methods can only track newly detected objects whose intersection on a joint metric is below a predetermined threshold of existing tracked objects.

[0064] Additionally, these systems and methods can track all active objects from previous frames. After the tracking step, these systems and methods can mark all objects with a confidence score above a predetermined threshold. All remaining objects are marked as mistracked. In some examples, these systems and methods may not immediately discard mistracked objects, but instead retain these objects in memory for a predetermined number of frames so that tracking may be resumed after a brief occlusion.

[0065] In some examples, the systems and methods described herein can train one or more of the machine learning components described herein. In some examples, the systems and methods can train a motion detection network (MDN). In one example, during training, the systems can provide the MDN with two frames randomly sampled from the same video or static images that are randomly moved and rotated as input. The systems can train a camera flow estimator in a supervised manner with static images, where random transformations are converted into dense flow vector fields. Thus, the camera loss can be formulated as the standard endpoint error between the predicted camera stream and the generated dense camera stream. When the input is a sequence of video images, these systems can be set Because the ground truth camera stream may not be available in the video sequence.

[0066] In some examples, the systems described herein can generate sparse pseudo flow for training instead of acquiring ground truth dense motion flow annotations for real-world video acquisition. Given an annotated object appearing in two different frames, these systems can compute the bounding box center point p (t) and p (t-1) The displacement is calculated and expanded to a small neighborhood of radius r, as shown in equation (5):

[0067]

[0068] These systems can then set the overall motion flow loss as the endpoint errors accumulated between layers l in the feature pyramid, as shown in Equation (6):

[0069]

[0070] in, It is only computed at pixels with generated ground truth flow.

[0071] As another example, the system described herein can train a patch association network (PAN). PAN training can be performed by using a patch association loss To supervise, as shown in equation (7):

[0072]

[0073] where i is the index of the query block, and represents the matching score between the query block and the jth candidate block. overlaps with the ground truth bounding box, then Otherwise σ is set to zero. α is the scale adaptation ratio described previously.

[0074] To train all modules in the network, the system described in this paper can use the combined loss function shown in equation (8):

[0075]

[0076] in, is the standard detection loss.

[0077] Figure 7 is an illustration of an example computer-implemented method 700 for tracking multiple deformable objects in an egocentric video. Figure 7 The steps shown in can be performed by any suitable computer executable code and / or computing system. In one example, Figure 7 Each of the steps shown in may represent an algorithm whose structure includes and / or is represented by a plurality of sub-steps, examples of which are provided in more detail below.

[0078] like Figure 7 As shown, at step 710, one or more of the systems described herein may receive a video stream having a plurality of frames. The systems may receive the video stream in any suitable context. For example, the systems may receive a live video stream in real time. In some examples, the systems may receive a video stream from a user-installed camera. For example, the systems may receive a video stream from a head-mounted camera. In some examples, one or more of the systems described may operate as part of a broader user-installed system that includes a user-installed camera. The systems may receive the video stream in any suitable format. For example, the systems may receive the video stream as a series of decoded frames.

[0079] At step 720, one or more of the systems described herein may detect at least one object within a selected frame of a video stream. The systems may detect the object in any of a variety of ways. In some examples, the systems may detect new objects (e.g., objects that were not detected in a previous frame and / or any previous frame of the video stream). For example, the systems may receive input from an object detection module that identifies objects within a frame by appearance. In some examples, the systems may detect objects within a frame by identifying tracked objects. For example, the systems may identify objects that were tracked across one or more previous frames. In some examples, the systems may identify an object that was previously tracked incorrectly, which did not appear in the immediately previous frame but matched one or more features stored in a memory in a previous tracking operation performed on one or more previous frames in the video stream.

[0080] At step 730, one or more of the systems described herein may decompose the object within the selected frame into a plurality of tiles. The systems may decompose the object into a plurality of tiles in any suitable manner. For example, the systems may divide the image defining the location of the object within the bounding box into smaller boxes.

[0081] At step 740, one or more of the systems described herein may associate a subset of the plurality of blocks with one or more candidate blocks in a subsequent frame of the video stream. The systems may associate the subset of the plurality of blocks with the candidate blocks in any suitable manner. In some examples, the systems may associate the subset of the plurality of blocks with the candidate blocks based at least in part on similarities between features of the blocks in the subset of the plurality of blocks and features of the blocks in the candidate blocks. Additionally or alternatively, the systems may associate the subset of the plurality of blocks with the candidate blocks based at least in part on relationships between the subset of the plurality of blocks (e.g., relationships with objects) and estimated relationships between the candidate blocks. In some examples, the systems may associate the subset of the plurality of blocks with the candidate blocks based at least in part on the location of the candidate blocks in the subsequent frame relative to the location of the subset of the plurality of blocks in the selected frame (e.g., after controlling the overall flow determined between the selected frame and the subsequent frame).

[0082] At step 750, one or more of the systems described herein may determine the location of the object in a subsequent frame of the video stream based at least in part on the location of the candidate block in the subsequent frame. For example, the systems may determine the location of the object in the subsequent frame by determining a bounding box that contains the candidate block.

[0083] Example Embodiments

[0084] [Inventor(s): The following section is a restatement of the claims for legal purposes. Feel free to skip this section and focus your review on the claims]

[0085] Example 1: A computer-implemented method may include: receiving a video stream having multiple frames; detecting at least one object within a selected frame of the video stream; decomposing the at least one object within the selected frame into a plurality of blocks; associating a subset of the plurality of blocks with at least one candidate block within a subsequent frame of the video stream; and determining a position of an object within the subsequent frame of the video stream based at least in part on a position of the at least one candidate block within the subsequent frame.

[0086] Example 2: The computer-implemented method of Example 1, wherein decomposing at least one object into a plurality of blocks comprises decomposing the at least one object into a plurality of blocks arranged in a checkerboard pattern.

[0087] Example 3: A computer-implemented method according to any of Examples 1 and 2, wherein determining the position of the object includes: determining a bounding box of the object based at least in part on determining a bounding box containing at least one candidate block.

[0088] Example 4: The computer-implemented method according to any one of Examples 1 to 3 further includes: retrieving at least one previous block of the object from a previous frame of the video stream; wherein associating a subset of multiple blocks with at least one candidate block in a subsequent frame further includes: associating the at least one previous block of the object with at least one candidate block in the subsequent frame.

[0089] Example 5: The computer-implemented method of any one of Examples 1 to 4 further includes storing at least one block of the subset of the plurality of blocks in association with the object.

[0090] Example 6: A computer-implemented method according to any of Examples 1 to 5, wherein associating a subset of multiple blocks with at least one candidate block in a subsequent frame includes: estimating the position of the at least one candidate block in the subsequent frame based in part on at least one of the following items: the position of one or more blocks in the subset of the multiple blocks in the selected frame; or the trajectory of the object at the time of the selected frame.

[0091] Example 7: A computer-implemented method according to any of Examples 1 to 6, wherein associating a subset of multiple blocks with at least one candidate block in a subsequent frame includes: estimating the position of the at least one candidate block in the subsequent frame based at least in part on isolating the local motion of the object from the egocentric global motion of the video stream between the selected frame and the subsequent frame.

[0092] Example 8: A computer-implemented method according to any of Examples 1 to 7, wherein isolating the local motion of an object from the egocentric global motion of a video stream includes analyzing the difference between a selected frame and subsequent frames to estimate the egocentric global motion of the video stream between the selected frame and the subsequent frames.

[0093] Example 9: A computer-implemented method according to any of Examples 1 to 8, wherein isolating the local motion of an object from the egocentric global motion of a video includes estimating the egocentric global motion of a video stream based at least in part on a motion sensor that detects motion of a device capturing the video stream.

[0094] Example 10: A computer-implemented method according to any one of Examples 1 to 9, wherein detecting at least one object within a selected frame includes detecting multiple objects; the method further includes: separately tracking the multiple objects based on separate sets of blocks associated with each of the multiple objects.

[0095] The various embodiments of the present disclosure may include various types of artificial reality systems or be implemented in combination with various types of artificial reality systems. Artificial reality is a form of reality that has been adjusted in some way before being presented to the user. Artificial reality may include, for example, virtual reality, augmented reality, mixed reality, hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include completely computer-generated content or computer-generated content combined with collected (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that brings a three-dimensional (3D) effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in artificial reality in other ways (e.g., to perform activities in artificial reality).

[0096] Artificial reality systems can be implemented in a variety of different form factors and configurations. Some artificial reality systems can be designed to work without a near-eye display (NED). Other artificial reality systems can include an NED that also provides visibility into the real world (e.g., Figure 8 ) or visually immersing a user in an artificial reality (e.g., Fig. 9900 in the virtual reality system 900). Although some artificial reality devices may be stand-alone systems, other artificial reality devices may communicate with and / or cooperate with external devices to provide an artificial reality experience to a user. Examples of such external devices include handheld controllers, mobile devices, desktop computers, devices worn by a user, devices worn by one or more other users, and / or any other suitable external system.

[0097] Go to Figure 8 , the augmented reality system 800 may include an eyewear device 802 having a frame 810 configured to hold a left display device 815(A) and a right display device 815(B) in front of the user's eyes. The left display device 815(A) and the right display device 815(B) may act together or independently to present an image or series of images to the user. Although the augmented reality system 800 includes two displays, embodiments of the present disclosure may be implemented in an augmented reality system having a single NED or more than two NEDs.

[0098] In some embodiments, the augmented reality system 800 may include one or more sensors, such as sensor 840. Sensor 840 may generate measurement signals in response to the movement of the augmented reality system 800 and may be located substantially on any portion of the frame 810. Sensor 840 may represent one or more of a variety of different sensing mechanisms, such as a position sensor, an inertial measurement unit (IMU), a depth camera assembly, a structured light emitter and / or detector, or any combination thereof. In some embodiments, the augmented reality system 800 may or may not include sensor 840, or may include more than one sensor. In embodiments where sensor 840 includes an IMU, the IMU may generate calibration data based on the measurement signals from sensor 840. Examples of sensor 840 may include, but are not limited to, accelerometers, gyroscopes, magnetometers, other suitable types of sensors that detect movement, sensors for error correction of IMUs, or some combination thereof.

[0099] In some examples, the augmented reality system 800 may also include a microphone array having a plurality of acoustic transducers 820(A) to 820(J), which are collectively referred to as acoustic transducers 820. Acoustic transducers 820 may represent transducers that detect changes in air pressure caused by sound waves. Each acoustic transducer 820 may be configured to detect sound and convert the detected sound into an electronic format (e.g., an analog format or a digital format). Figure 8The microphone array in may, for example, include ten acoustic transducers: acoustic transducers 820(A) and 820(B) that may be designed to be placed within respective ears of a user, acoustic transducers 820(C), 820(D), 820(E), 820(F), 820(G) and 820(H) that may be positioned at various locations on the frame 810, and / or acoustic transducers 820(I) and 820(J) that may be positioned on corresponding neckbands 805.

[0100] In some embodiments, one or more of the acoustic transducers 820(A) to 820(J) can be used as output transducers (e.g., speakers). For example, the acoustic transducers 820(A) and / or 820(B) can be earbuds, or any other suitable type of headphones or speakers.

[0101] The configuration of each acoustic transducer 820 in the microphone array can be varied. Figure 8 820, but the number of acoustic transducers 820 can be more or less than ten. In some embodiments, using a larger number of acoustic transducers 820 can increase the amount of audio information collected and / or improve the sensitivity and accuracy of the audio information. In contrast, using a smaller number of acoustic transducers 820 can reduce the computing power required for the associated controller 850 to process the collected audio information. In addition, the position of each acoustic transducer 820 in the microphone array can be changed. For example, the position of the acoustic transducer 820 can include a limited position on the user, a limited coordinate on the frame 810, an orientation associated with each acoustic transducer 820, or some combination thereof.

[0102] The acoustic transducers 820(A) and 820(B) can be positioned at different locations on the user's ear, such as behind the pinna, behind the tragus, and / or within the auricle or ear cavity. Alternatively, in addition to the acoustic transducer 820 within the ear canal, additional acoustic transducers 820 can be present on or around the ear. Positioning the acoustic transducer 820 near the user's ear canal can enable the microphone array to collect information about how sound reaches the ear canal. By positioning at least two of the acoustic transducers 820 on both sides of the user's head (e.g., as binaural microphones), the augmented reality system 800 can simulate binaural hearing and capture a 3D stereo sound field around the user's head. In some examples, acoustic transducers 820(A) and 820(B) may be connected to augmented reality system 800 via wired connection 830, while in other embodiments, acoustic transducers 820(A) and 820(B) may be connected to augmented reality system 800 via a wireless connection (e.g., a BLUETOOTH connection). In other embodiments, acoustic transducers 820(A) and 820(B) may not be used in conjunction with augmented reality system 800 at all.

[0103] The acoustic transducers 820 on the frame 810 can be positioned in a variety of different ways, including along the length of the temples, across the bridge of the nose, above or below the left display device 815 (A) and the right display device 815 (B), or some combination thereof. The acoustic transducers 820 can also be oriented so that the microphone array can detect sounds in a wide range of directions around the user who is wearing the augmented reality system 800. In some embodiments, an optimization process can be performed during the manufacture of the augmented reality system 800 to determine the relative positioning of the acoustic transducers 820 in the microphone array.

[0104] In some examples, the augmented reality system 800 may include or be connected to an external device (e.g., a paired device), such as a neckband 805. The neckband 805 generally represents any type or form of paired device. Therefore, the following discussion of the neckband 805 may also be applied to various other paired devices, such as charging boxes, smart watches, smart phones, wristbands, other wearable devices, handheld controllers, tablet computers, laptop computers, other external computing devices, etc.

[0105] As shown, the neckband 805 can be coupled to the eyeglass device 802 via one or more connectors. These connectors can be wired or wireless and can include electronic components and / or non-electronic components (e.g., structural components). In some cases, the eyeglass device 802 and the neckband 805 can operate independently without any wired or wireless connection between them. Although Figure 8The components of the eyeglass device 802 and the neckband 805 are shown as being located at example locations on the eyeglass device 802 and the neckband 805, but the components may be located at other locations on the eyeglass device 802 and / or the neckband 805 and / or distributed in different ways on the eyeglass device 802 and / or the neckband 805. In some embodiments, the components of the eyeglass device 802 and the neckband 805 may be located on one or more additional peripheral devices that are paired with the eyeglass device 802, the neckband 805, or some combination thereof.

[0106] Pairing an external device (e.g., a neckband 805) with an augmented reality eyewear device can enable the eyewear device to achieve the form factor of a pair of glasses while still providing sufficient battery power and computing power for the expanded capabilities. Some or all of the battery power, computing resources, and / or additional features of the augmented reality system 800 can be provided by a paired device, or shared between a paired device and the eyewear device, thereby reducing the weight, heat distribution, and form factor of the eyewear device as a whole while still maintaining the desired functionality. For example, the neckband 805 can allow components that were originally included on the eyewear device to be included in the neckband 805 because the user can bear a heavier weight load on their shoulders than the weight load they bear on their head. The neckband 805 can also have a larger surface area through which heat is diffused and dissipated to the surrounding environment. Therefore, the neckband 805 can allow for greater battery capacity and computing power than would otherwise be feasible on a standalone eyewear device. Because the weight carried in the neckband 805 can be less intrusive to the user than the weight carried in the eyeglass device 802, the user can tolerate wearing a lighter eyeglass device and carrying or wearing a paired device for longer periods of time than the user can tolerate wearing a heavy stand-alone eyeglass device, thereby enabling the user to more fully integrate the artificial reality environment into their daily activities.

[0107] The neckband 805 can be communicatively coupled to the eyewear device 802 and / or communicatively coupled to other devices. These other devices can provide certain functions (e.g., tracking, positioning, depth map construction, processing, storage, etc.) to the augmented reality system 800. Figure 8 In an embodiment of the present invention, the neckband 805 may include two acoustic transducers (e.g., acoustic transducers 820 (I) and 820 (J)), which are part of a microphone array (or potentially form their own microphone subarray). The neckband 805 may also include a controller 825 and a power supply 835.

[0108] The acoustic transducers 820(I) and 820(J) in the neckband 805 may be configured to detect sound and convert the detected sound into an electronic format (analog or digital). Figure 8 In an embodiment of the present invention, acoustic transducers 820(I) and 820(J) can be positioned on the neckband 805, thereby increasing the distance between the acoustic transducers 820(I) and 820(J) of the neckband and other acoustic transducers 820 positioned on the eyeglass device 802. In some cases, increasing the distance between the acoustic transducers 820 in the microphone array can improve the accuracy of beamforming performed via the microphone array. For example, if acoustic transducers 820(C) and 820(D) detect a sound, and the distance between acoustic transducers 820(C) and 820(D) is greater than the distance between acoustic transducers 820(D) and 820(E), the determined source location of the detected sound can be more accurate than when the sound is detected by acoustic transducers 820(D) and 820(E).

[0109] The controller 825 in the neckband 805 can process information generated by various sensors on the neckband 805 and / or the augmented reality system 800. For example, the controller 825 can process information from the microphone array that describes the sound detected by the microphone array. For each detected sound, the controller 825 can perform a direction-of-arrival (DOA) estimate to estimate the direction in which the detected sound arrives at the microphone array. When the microphone array detects a sound, the controller 825 can fill the audio data set with this information. In an embodiment where the augmented reality system 800 includes an inertial measurement unit, the controller 825 can calculate all inertial and spatial calculations based on the IMU located on the eyeglass device 802. The connector can transmit information between the augmented reality system 800 and the neckband 805, and between the augmented reality system 800 and the controller 825. This information can be in the form of optical data, electronic data, wireless data, or any other transmittable data form. Moving the processing of information generated by the augmented reality system 800 to the neckband 805 can reduce the weight and heat of the eyewear device 802, making the eyewear device more comfortable for the user.

[0110] The power supply 835 in the neckband 805 can provide power to the eyeglass device 802 and / or the neckband 805. The power supply 835 can include, but is not limited to, lithium-ion batteries, lithium-polymer batteries, disposable lithium batteries, alkaline batteries, or any other form of power storage. In some cases, the power supply 835 can be a wired power supply. Including the power supply 835 on the neckband 805 instead of the eyeglass device 802 can help to better disperse the weight and heat generated by the power supply 835.

[0111] As mentioned, some artificial reality systems may generally use a virtual experience to replace one or more of a user's multiple sensory perceptions of the real world, rather than mixing the artificial reality with the actual reality. An example of this type of system is a head-mounted display system that covers most or all of the user's field of view, such as Fig. 9 Virtual reality system 900 in FIG. Virtual reality system 900 may include a front rigid body 902 and a band 904 shaped to fit around a user's head. Virtual reality system 900 may also include output audio transducers 906 (A) and 906 (B). In addition, although Fig. 9 Not shown, but front rigid body 902 may include one or more electronic components including one or more electronic displays, one or more inertial measurement units (IMUs), one or more tracking transmitters or detectors, and / or any other suitable device or system for creating an artificial reality experience.

[0112] Artificial reality systems may include various types of visual feedback mechanisms. For example, the display devices in the augmented reality system 800 and / or the virtual reality system 900 may include one or more liquid crystal displays (LCD), one or more light emitting diode (LED) displays, one or more micro LED displays, one or more organic LED (OLED) displays, one or more digital light project (DLP) micro displays, one or more liquid crystal on silicon (LCoS) micro displays, and / or any other suitable type of display screen. These artificial reality systems may include a single display screen for both eyes, or a display screen may be provided for each eye, which may provide additional flexibility for zoom adjustment or correction of the user's refractive error. Some of these artificial reality systems may also include multiple optical subsystems having one or more lenses (e.g., concave or convex lenses, Fresnel lenses, adjustable liquid lenses, etc.), through which the user can view the display screen. These optical subsystems can be used for a variety of purposes, including collimating light (e.g., making an object appear to be at a greater distance than it is physically at), magnifying light (e.g., making an object appear larger than its actual size), and / or delivering light (e.g., delivering light to a viewer's eye). These optical subsystems can be used in a direct-view architecture (e.g., a single lens configuration that directly collimates light but produces so-called pincushion distortion) and / or a non-direct-view architecture (e.g., a multi-lens configuration that produces so-called barrel distortion to eliminate pincushion distortion).

[0113] In addition to or instead of using a display screen, some of the artificial reality systems described herein may include one or more projection systems. For example, a display device in the augmented reality system 800 and / or in the virtual reality system 900 may include a micro LED projector (e.g., using a waveguide) that projects light into a display device, such as a transparent combination lens that allows ambient light to pass through. The display device may refract the projected light toward the user's pupil, and may enable the user to simultaneously view both the artificial reality content and the real world. The display device may achieve this using any of a variety of different optical components, including waveguide components (e.g., holographic waveguide elements, planar waveguide elements, diffractive waveguide elements, polarization waveguide elements, and / or reflective waveguide elements), light manipulation surfaces and elements (e.g., diffractive elements and gratings, reflective elements and gratings, and refractive elements and gratings), coupling elements, etc. The artificial reality system may also be configured with any other suitable type or form of image projection system, such as a retinal projector for a virtual retinal display.

[0114] The artificial reality systems described herein may also include various types of computer vision components and subsystems. For example, the augmented reality system 800 and / or the virtual reality system 900 may include one or more optical sensors, such as a two-dimensional (2D) camera or a 3D camera, a structured light emitter and detector, a time-of-flight depth sensor, a single beam rangefinder or a scanning laser rangefinder, a 3D light detection and ranging (LiDAR) sensor, and / or any other suitable type or form of optical sensor. The artificial reality system may process data from one or more of these sensors to identify the user's location, map the real world, provide the user with context related to the real-world surroundings, and / or perform various other functions.

[0115] The artificial reality system described herein may also include one or more input audio transducers and / or output audio transducers. The output audio transducer may include a voice coil speaker, a ribbon speaker, an electrostatic speaker, a piezoelectric speaker, a bone conduction transducer, a cartilage conduction transducer, an ear tragus vibration transducer, and / or any other suitable type or form of audio transducer. Similarly, the input audio transducer may include a condenser microphone, a dynamic microphone, a ribbon microphone, and / or any other type or form of input transducer. In some embodiments, a single transducer may be used for both audio input and audio output.

[0116] In some embodiments, the artificial reality system described herein may also include a tactile (i.e., haptic) feedback system that can be incorporated into headwear, gloves, clothing, handheld controllers, environmental equipment (e.g., chairs, floor mats, etc.), and / or any other type of device or system. The tactile feedback system can provide various types of skin feedback, including vibration, thrust, traction, texture, and / or temperature. The tactile feedback system can also provide various types of kinesthetic feedback, such as motion and compliance. Motors, piezoelectric actuators, fluid systems, and / or various other types of feedback mechanisms can be used to implement tactile feedback. The tactile feedback system can be implemented independently of other artificial reality devices, implemented within other artificial reality devices, and / or implemented in combination with other artificial reality devices.

[0117] By providing tactile perception, auditory content, and / or visual content, artificial reality systems can create complete virtual experiences or enhance users' real-world experiences in various contexts and environments. For example, artificial reality systems can assist or expand users' perception, memory, or cognition in a specific environment. Some systems can enhance users' interactions with other people in the real world, or can enable more immersive interactions with other people in the virtual world. Artificial reality systems can also be used for educational purposes (e.g., for teaching or training in schools, hospitals, government agencies, military agencies, enterprises, etc.), entertainment purposes (e.g., for playing video games, listening to music, watching video content, etc.), and / or for accessibility purposes (e.g., as hearing aids, visual aids, etc.). The various embodiments disclosed herein can implement or enhance users' artificial reality experiences in one or more of these contexts and environments, and / or in other contexts and environments.

[0118] As detailed above, the computing devices and systems described and / or shown herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions (e.g., those instructions contained in the modules described herein). In the most basic configuration of one or more computing devices, the one or more computing devices may each include at least one storage device and at least one physical processor.

[0119] In some examples, the term "storage device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a storage device can store, load and / or maintain one or more modules in each module described herein. Examples of storage devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache memory, a variation or combination of one or more of the above, or any other suitable memory.

[0120] In some examples, the term "physical processor" generally refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, the physical processor can access and / or modify one or more modules stored in the above-mentioned storage device. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field programmable gate arrays (FPGAs) that implement soft-core processors, application-specific integrated circuits (ASICs), parts of one or more of the above, variations or combinations of one or more of the above, or any other suitable physical processors.

[0121] Although each module described and / or shown in this article is shown as a separate element, these modules can represent a part in a single module or application. In addition, in some embodiments, one or more modules in these modules can represent one or more software applications or programs, which can make the computing device perform one or more tasks when executed by a computing device. For example, one or more modules in each module described and / or shown in this article can represent such a module: the module is stored in one or more of each computing device or system described and / or shown in this article, and is configured to run on one or more of each computing device or system described and / or shown in this article. One or more modules in these modules can also represent all or part of one or more special-purpose computers configured to perform one or more tasks.

[0122] In addition, one or more of the modules described herein may convert data, physical devices, and / or representations of physical devices from one form to another. Additionally or alternatively, one or more of the modules described herein may transform a processor, volatile memory, non-volatile memory, and / or any other portion of the physical computing device from one form to another by executing on a computing device, storing data on a computing device, and / or otherwise interacting with the computing device.

[0123] In some embodiments, the term "computer-readable medium" generally refers to any form of device, carrier or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmission-type media and non-transitory media, such as carrier waves, such as magnetic storage media (e.g., hard disk drives, tape drives, and floppy disks), optical storage media (e.g., compact disks (CDs), digital video disks (DVDs), and Blu-ray disks), electronic storage media (e.g., solid-state drives and flash memory media), and other distribution systems.

[0124] The process parameters and step sequences described and / or shown herein are provided by way of example only and may be changed as desired. For example, although the steps shown and / or described herein may be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order shown or discussed. The various exemplary methods described and / or shown herein may also omit one or more of the steps described or shown herein, or may include additional steps in addition to those disclosed.

[0125] The previous description has been provided to enable others skilled in the art to best utilize the various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the present disclosure. The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. In determining the scope of the present disclosure, reference should be made to the attached claims and their equivalents.

[0126] Unless otherwise indicated, the terms "connected to" and "coupled to" (and their derivatives) as used in the specification and claims should be interpreted as allowing both direct and indirect (i.e., via other elements or components) connections. In addition, the terms "a" or "an" as used in the specification and claims are to be interpreted as meaning "at least one of..." Finally, for ease of use, the terms "including" and "having" (and their derivatives) as used in the specification and claims are interchangeable with the word "comprising" and have the same meaning as the word "comprising".

Claims

1. A computer-implemented method comprising: receiving a video stream having a plurality of frames; detecting at least one object within a selected frame of the video stream; decomposing the at least one object within the selected frame into a plurality of blocks; associating a subset of the plurality of blocks with at least one candidate block in a subsequent frame of the video stream; as well as The position of the at least one object within the subsequent frame of the video stream is determined based at least in part on the position of the at least one candidate block within the subsequent frame.

2. The computer-implemented method of claim 1 , wherein: Decomposing the at least one object into the plurality of blocks includes decomposing the at least one object into a plurality of blocks arranged in a checkerboard pattern.

3. The computer-implemented method of claim 1 , wherein: Determining the position of the at least one object includes determining a bounding box of the at least one object based at least in part on determining a bounding box that includes the at least one candidate block.

4. The computer-implemented method of claim 1, Also includes: retrieving at least one previous block of the at least one object from a previous frame of the video stream; Wherein, associating the subset of the plurality of blocks with the at least one candidate block in the subsequent frame further comprises: associating the at least one previous block of the at least one object with the at least one candidate block in the subsequent frame.

5. The computer-implemented method of claim 1 , further comprising: At least one chunk of the subset of the plurality of chunks is stored in association with the at least one object.

6. The computer-implemented method of claim 1 , wherein: Associating the subset of the plurality of blocks with the at least one candidate block within the subsequent frame includes estimating a position of the at least one candidate block within the subsequent frame based at least in part on at least one of: the location of one or more blocks in the subset of the plurality of blocks within the selected frame; or The trajectory of the at least one object at the time of the selected frame.

7. The computer-implemented method of claim 1 , wherein: Associating a subset of the plurality of blocks with the at least one candidate block within the subsequent frame includes estimating a position of the at least one candidate block within the subsequent frame based at least in part on isolating local motion of the at least one object from egocentric global motion of the video stream between the selected frame and the subsequent frame.

8. The computer-implemented method of claim 7, wherein: Isolating the local motion of the at least one object from the egocentric global motion of the video stream includes analyzing differences between the selected frame and the subsequent frame to estimate the egocentric global motion of the video stream between the selected frame and the subsequent frame.

9. The computer-implemented method of claim 7, wherein: Isolating the local motion of the at least one object from the egocentric global motion of the video stream includes estimating the egocentric global motion of the video stream based at least in part on a motion sensor that detects motion of a device capturing the video stream.

10. The computer-implemented method of claim 1, in, detecting the at least one object within the selected frame comprises detecting a plurality of objects; The method also includes separately tracking each of the plurality of objects based on a separate set of tiles associated with each of the plurality of objects.

11. A system comprising: at least one physical processor; a physical memory comprising computer executable instructions that, when executed by the physical processor, cause the physical processor to: receiving a video stream having a plurality of frames; detecting at least one object within a selected frame of the video stream; decomposing the at least one object within the selected frame into a plurality of blocks; associating a subset of the plurality of blocks with at least one candidate block in a subsequent frame of the video stream; as well as The position of the at least one object within the subsequent frame of the video stream is determined based at least in part on the position of the at least one candidate block within the subsequent frame.

12. The system according to claim 11, wherein: Decomposing the at least one object into the plurality of blocks includes decomposing the at least one object into a plurality of blocks arranged in a checkerboard pattern.

13. The system according to claim 11, wherein: Determining the position of the at least one object includes determining a bounding box of the at least one object based at least in part on determining a bounding box that includes the at least one candidate block.

14. The system according to claim 11, Also includes: retrieving at least one previous block of the at least one object from a previous frame of the video stream; Wherein, associating the subset of the plurality of blocks with the at least one candidate block in the subsequent frame further comprises: associating the at least one previous block of the at least one object with the at least one candidate block in the subsequent frame.

15. The system of claim 11, further comprising: At least one chunk of the subset of the plurality of chunks is stored in association with the at least one object.

16. The system according to claim 11, wherein: Associating the subset of the plurality of blocks with the at least one candidate block within the subsequent frame includes estimating a position of the at least one candidate block within the subsequent frame based at least in part on at least one of: the location of one or more blocks in the subset of the plurality of blocks within the selected frame; or The trajectory of the at least one object at the time of the selected frame.

17. The system of claim 11, wherein: Associating a subset of the plurality of blocks with the at least one candidate block within the subsequent frame includes estimating a position of the at least one candidate block within the subsequent frame based at least in part on isolating local motion of the at least one object from egocentric global motion of the video stream between the selected frame and the subsequent frame.

18. The system of claim 17, wherein: Isolating the local motion of the at least one object from the egocentric global motion of the video stream includes analyzing differences between the selected frame and the subsequent frame to estimate the egocentric global motion of the video stream between the selected frame and the subsequent frame.

19. The system of claim 17, wherein: Isolating the local motion of the at least one object from the egocentric global motion of the video stream includes estimating the egocentric global motion of the video stream based at least in part on a motion sensor that detects motion of a device capturing the video stream.

20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to: receiving a video stream having a plurality of frames; detecting at least one object within a selected frame of the video stream; decomposing the at least one object within the selected frame into a plurality of blocks; associating a subset of the plurality of blocks with at least one candidate block in a subsequent frame of the video stream; as well as The position of the at least one object within the subsequent frame of the video stream is determined based at least in part on the position of the at least one candidate block within the subsequent frame.

Images