Multi-camera person association via pairwise matching in consecutive frames for immersive video
The method addresses unstable bounding box detection in multi-camera systems by using pairwise matching and tracklet fusion to achieve accurate and real-time multi-camera person or object association, enhancing immersive video experiences.
Patent Information
- Application Number
- JP2022577486
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-07-30
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-07-30
AI Technical Summary
Existing techniques for multi-camera person or object association in immersive video applications face challenges such as unstable and inaccurate bounding box detection, especially in crowded scenes with occlusions, leading to non-contiguous associations and high computational complexity, making real-time processing difficult.
A method involving pairwise matching and optimization of distance matrices with hard constraints to reduce search space, followed by tracklet generation and fusion to stabilize player positions and numbers across consecutive frames, using a multi-camera association module to refine associations in real-time.
Achieves stable and accurate multi-camera person or object association in consecutive frames, providing robust player tracking and immersive experiences by reducing computational complexity and enabling real-time processing.
Smart Images

Figure 0007581584000084 
Figure 0007581584000085 
Figure 0007581584000086
Abstract
Description
[Background technology]
[0001] In immersive video and other situations such as computer vision applications, multiple cameras are placed around a scene of interest. For example, cameras may be placed in a stadium around a playing field. Using the video acquired from the cameras, a 3D model (e.g., via a point cloud volumetric model or other representation) is generated of the scene for each time instance of the video. Photorealistic views from virtual views within the scene may then be generated to provide an immersive experience to the user. In particular, such techniques enable the virtual camera to navigate in 3D space and replay actions from the scene at any position and from any viewpoint or angle within the scene.
[0002] While correlating the same person from different camera views together is a complex problem in multiple player tracking algorithms, tracking people (e.g., players) in a scene (e.g., a field or court) is a key component for automatically controlling a virtual camera. Multiple camera player correlation integrates the detection and tracking of bounding boxes of the same person from different cameras at the same timestamp together.
[0003] There are several challenges in providing stable and accurate association from multiple cameras. First, the detection results from each camera are not completely reliable. In detection, bounding box misses are common, especially when players crowd together and occlusion occurs. For example, one bounding box may cover more than one person (e.g., more than three people), which reduces the number of people that can be associated. Second, the detected bounding boxes are unstable in consecutive frames. In particular, two adjacent frames may have different numbers of bounding boxes, which results in non-contiguous association positions in adjacent frames. Third, images of the same person in different cameras may have large variations. Different viewpoints and overlapping people make matching difficult. Furthermore, there are strong limitations on the execution time for real-time processing.
[0004] Current techniques for association from multiple cameras include geometric constraint-based techniques and trajectory fusion-based techniques. Geometric constraint techniques are typically used for single-frame association, while trajectory fusion techniques are usually used in consecutive frame association. Furthermore, current techniques may perform multi-view matching by combining geometric constraints, human keypoints, and player re-identification features together to build a similarity matrix from the geometric constraints and the re-identification, and use convex optimization to optimize the similarity matrix to obtain the best association. Tracking the association may also be modeled as an optimization task by collecting appearance, motion, and geometry information. However, such techniques have limitations. For geometric constraint techniques, only single-frame information is used, which results in jitter in consecutive frames. Some techniques use global optimization strategies to optimize the association results, but such techniques are time-consuming and cannot be performed in real time. Current tracking fusion methods rely on single-camera tracking, which inconveniently causes frequent identification switching in crowded cases such as sports applications. Moreover, such techniques are complex and, again, not available for real-time applications.
[0005] It is desirable to provide multi-camera person or object association in consecutive frames in real time with high accuracy and stability. In relation to these and other considerations, this improvement has been needed. Such improvements can be crucial as the desire to provide new, immersive user experiences from multi-view video becomes more widespread. [Brief description of the drawings]
[0006] Materials described herein are illustrated by way of example, and not by way of limitation, in the accompanying figures. For simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, dimensions of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals have been repeated among the figures to indicate corresponding or analogous elements. The figures are as follows: [Figure 1] FIG. 1 illustrates an exemplary system for tracking players in a multi-camera environment. [Diagram 2] FIG. 1 illustrates an example camera array trained on an example scene. [Diagram 3] FIG. 2 illustrates an example multi-camera association module for generating object associations using pair-wise matching in consecutive frames. [Figure 4] FIG. 13 illustrates an exemplary determination of a bounding box and corresponding player position using estimated footprints. [Diagram 5] FIG. 2 illustrates an example distance matrix. [Figure 6] FIG. 1 illustrates an exemplary bounding box pairing using two-way minimum distance. [Figure 7] FIG. 13 illustrates another exemplary bounding box pairing using two-way minimum distance. [Figure 8] FIG. 2 illustrates an example bidirectional minimum distance flag matrix and row-wise matrix transformation. [Figure 9] FIG. 2 illustrates the matrix resulting from a row-wise transformation. [Figure 10] FIG. 2 illustrates the matrix resulting from a column-wise transformation. [Figure 11] FIG. 13 illustrates a resultant matrix and example sub-matrices after subsequent row-wise and column-wise transformations. [Figure 12] FIG. 2 illustrates an exemplary tracklet generation where generation begins at each time frame. [Figure 13] FIG. 1 illustrates an exemplary tracklet fusion. [Figure 14] FIG. 2 illustrates an example process for performing tracklet fusion. [Figure 15] FIG. 1 illustrates an exemplary player reconfiguration or reallocation process. [Figure 16] FIG. 13 illustrates an exemplary large distance player removal process. [Figure 17] 1 is a flow diagram illustrating an example process for performing object association in multiview video. [Figure 18] 1 is an illustration of an example system for performing object association in multiview video. [Figure 19] FIG. 1 is an illustration of an exemplary system. [Figure 20] 1A-1D illustrate example devices, all arranged in accordance with at least some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0007] One or more embodiments or examples will now be described with reference to the enclosed figures. While specific configurations and arrangements are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements may be employed without departing from the spirit and scope of the present specification. Those skilled in the art will recognize that the techniques and / or arrangements described herein may also be employed in a variety of other systems and applications other than those described herein.
[0008] While the following description describes various embodiments that may be manifested within an architecture, such as, for example, a system-on-chip (SoC) architecture, embodiments of the techniques and / or arrangements described herein are not limited to a particular architecture and / or computing system, but may be implemented by any architecture and / or computing system for the same purpose. For example, various architectures employing multiple integrated circuit (IC) chips and / or packages, and / or various computing devices, and / or consumer electronics (CE) devices, such as set-top boxes, smartphones, etc., may implement the techniques and / or arrangements described herein. Furthermore, although the following description may set forth numerous specific details, such as implementations of logic, types and interrelationships of system components, logic partitioning / integration choices, etc., the claimed subject matter may be practiced without such specific details. In other instances, some material, such as, for example, control structures and complete software instruction sequences, may not be shown in detail so as not to obscure the material disclosed herein.
[0009] The material disclosed herein may be implemented in hardware, firmware, software, or any combination thereof. The material disclosed herein may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any medium and / or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, electrical, optical, acoustic or other types of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others.
[0010] References herein to "one embodiment," "one example," "one exemplary embodiment," and the like indicate that the embodiment being described may include a particular feature, structure, or characteristic, but that not every embodiment necessarily includes that particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with one embodiment, it is stated that it is within the knowledge of one of ordinary skill in the art to effect such feature, structure, or characteristic in connection with other embodiments, whether or not expressly described herein.
[0011] The terms "substantially," "close," "approximately," "near," and "approximately" generally refer to being within + / - 10% of a target value. For example, unless otherwise specified in the express context of their use, the terms "substantially equal," "approximately equal," and "nearly equal" mean not to exceed incidental variations among those so described. In the art, such variations are typically no more than + / - 10% of a predefined target value. Unless otherwise specified, the use of ordinal numbers such as "first," "second," and "third" to describe a common object is merely intended to indicate reference to different instances of a similar object and is not intended to imply that the objects so described must be in a given order, either in time, space, order, or in any other manner.
[0012] Methods, devices, apparatus, computing platforms, and articles are described herein for providing multi-camera or multi-view person or object association in successive frames.
[0013] As mentioned above, it is an advantage to provide multi-camera person or object association in consecutive frames in real time with high accuracy and stable execution. The technique discussed herein performs multi-camera association via pair-wise matching through consecutive frames. Such technique first performs association on a frame-by-frame basis, and then implements multi-frame position tracking results to refine the association. For example, a distance matrix across all camera views for each time instance is generated in a pair-wise manner. Then, player association is determined by optimization of the distance matrix. To perform in real time, hard constraints are provided prior to such optimization in order to substantially reduce the search space. After such association is provided for several frames (e.g., association results are generated for several consecutive frames), position tracking is performed based on the current association based on tracking, interpolation, and merging to generate continuous player positions and stable player numbers across multiple time instances. Based on the aforementioned player numbers and positions, bounding box reconstruction and removal is then performed to further refine error matching across time instances.
[0014] FIG. 1 illustrates an example system 100 for tracking players in a multi-camera environment, arranged in accordance with at least some embodiments of the present disclosure. As shown in FIG. 1, the system 100 includes a camera array 101 having any number N of cameras, including camera 102 and camera 103. Each camera generates a corresponding video, including video 104 and video 105. Player detection modules, including player detection modules 106, 107, detect players in the captured video, and player tracking modules, including player tracking modules 108, 109, track such detected players to generate bounding boxes, including bounding boxes 120, 121. For clarity of presentation, the techniques are discussed herein in terms of player detection, tracking, association, etc. However, the techniques may be applied to any people, animals, objects, or combinations thereof. In particular, in the context of sporting events, such player tracking includes tracking referees.
[0015] FIG. 2 is a diagram illustrating an example camera array 101 trained on an example scene 210, arranged in accordance with at least some embodiments of the present disclosure. In the illustrated embodiment, the camera array 101 includes 36 cameras trained on a sports field. However, the camera array 101 may include any suitable number of cameras, such as 15 or more cameras, trained on the scene 210. For example, the camera array 101 may be trained on the scene 210 to generate a 3D model of the scene 210, where fewer cameras may not provide enough information to generate the 3D model. The camera array 101 may be mounted, calibrated, and trained on the scene 210 to capture simultaneous video frames of the scene 210, along with a ground surrounding the sports field, and in a stadium (not shown) surrounding the sports field of the scene 210. As shown, each camera of the camera array 101 has a particular view of the scene 210. For example, camera 102 has a first view of the scene 210, camera 103 has a second view of the scene, etc. As used herein, the term view refers to the image content of an image plane of a particular camera of camera array 101, or the image content of any view from a virtual camera located in scene 210. In particular, a view may be a photographed view (e.g., a view acquired using image capture in a camera) such that multiple views contain representations of the same person, object, entity, etc.
[0016] Returning to FIG. 1 , as shown, the per-camera or view processing generates bounding boxes, including bounding boxes 120, 121. Bounding boxes 120 may include any data structure that represents or indicates bounding boxes for detected and tracked players in some or all frames of video 105. For example, bounding boxes 120 may include data that represents the position and size of each bounding box in each frame, every other frame, every third frame, or the like. Bounding boxes 121 include similar data for video 105. Any number of per-view or per-camera bounding boxes or bounding box data are provided to multi-camera association module 114 for processing as discussed herein to generate association list 122 indicating matching bounding boxes deemed to contain the same player. Association list 122 may include one or more lists having any suitable data structure for indicating such association or correspondence. For example, each bounding box may be indexed within each frame from each view at a particular time instance (e.g., occurrence of 36 indexed bounding boxes), and the association list 122 may list the indexes of each bounding box from which a particular player corresponds for a time instance. In some embodiments, the multi-camera association module 114 further receives jersey recognition data from a jersey recognition module, including jersey recognition modules 110, 111, and team classification data from a team classification module, including team classification modules 112, 113, and each player may be further identified by a team (e.g., home or away) and a player number.For example, association list 122 may include an entry for home player number 5 showing a respective bounding box for that player in each view available from camera array 101, and an entry for away player number 12 showing a respective bounding box for that player in each view available from camera array 101, and so on for all identified players or persons. Further, similar lists may be generated at each time instance or timestamp of a captured frame, or for every other time instance, etc.
[0017] As shown, the association list 122 is provided to an optional multi-camera tracking module 115 and a player trajectory optimization module 116, which combine player information in the temporal and spatial domains to provide player trajectories, and further track and optimize the player trajectories to provide outputs for use by the smooth and robust system 100. In particular, with reference to FIG. 2, desired virtual cameras within the scene 210 may be managed to follow key players to generate a final volumetric video (e.g., a point cloud representation or other data structure). As shown in FIG. 1, such a player tracking pipeline begins with single camera player detection and tracking. Then, team classification and jersey recognition are provided for each detected player. Such data is provided to a multi-camera association module 114 to associate or provide correspondence for players from different views of the scene. Multi-camera tracking is applied to fuse together all player information (e.g., team, jersey) in the temporal and spatial domains and generate player trajectories, followed by player trajectory optimization (PTO) to provide a smooth and accurate output for use by the system 100. The focus of the techniques discussed herein is to provide accurate, real-time multi-camera correlation results. As discussed, inputs including tracking bounding boxes generated by all camera views are provided and outputs including correlation lists or lists that provide identical player bounding boxes in all camera views are generated.
[0018] 3 is a diagram illustrating an example multi-camera association module 114 for generating object associations using pair-wise matching in successive frames, arranged in accordance with at least some embodiments of the present disclosure. As discussed, the multi-camera association module 114 receives bounding box data, such as bounding box data 120, 121, from player tracking modules 108, 109, respectively. As shown, in some embodiments, the multi-camera association module 114 includes a bounding box correspondence construction module 301, a position tracking module in successive frames 302, and a post-processing module 303. With respect to the position tracking module in successive frames 302, positions are tracked over multiple time instances, which may also be referred to as successive time frames or simply successive frames. However, such time frames are not to be confused with video frames or video images generated by each camera at each time instance.
[0019] In some embodiments, the multi-camera association problem is addressed by using a continuous timestamp buffer. In some embodiments, pair-wise matching with hard constraints is used to build correspondences between bounding box pairs. Such a technique significantly reduces the association complexity at each time frame so that the technique can operate in real time. Furthermore, a position tracking and merging technique based on the association results in the 2D ground plane is employed to maintain stable player positions and player numbers. Such a technique provides stable and accurate positions and tracking. Also, the association results are refined based on the tracking results for association error correction. The systems and techniques discussed herein generate stable and accurate association results in real time.
[0020] In some embodiments, the adopted pipeline is split into two parts: association and post-processing. Association matches identical players from different camera views. Post-processing refines the association results through heuristic techniques for increased stability and accuracy. For example, the association part may be provided by the bounding box correspondence construction module 301 and the position tracking in consecutive frames module 302, and the post-processing may be provided by the post-processing module 303, which provides association result reconstruction and large distance removal as shown. As discussed, the input to the multi-camera association module 114 is the tracked bounding boxes generated from all camera views, and the output is an association list 122 containing identical player bounding boxes in all camera views.
[0021] The bounding box correspondence construction module 301 receives bounding box data 120, 121, which indicates, for each corresponding video frame at a particular time instance, the number of bounding boxes, the position of each bounding box in the video frame, and the size of each bounding box. That is, the number of bounding boxes, the position of each bounding box in the video frame, and the size of each bounding box are provided for each view of the scene 210. In particular, a player within each bounding box may be visible from many or all views of the scene 210, and thus bounding boxes correspond when they contain the same player. The techniques discussed herein attempt to associate such bounding boxes (e.g., within video frames) across all such views.
[0022] In some embodiments, the position of each such bounding box on a ground plane in 3D space is determined and used for correspondence. The position of each bounding box on the ground plane may be generated using any suitable technique or techniques. In some embodiments, the distances of the footprints for different bounding boxes are used as constraints to build the bounding box correspondence.
[0023] 4 is a diagram illustrating an example determination of a bounding box position and corresponding player with estimated foot points positioned in accordance with at least some embodiments of the present disclosure. As shown, for a bounding box 411 with a corresponding player 412, a foot point 406 (which may also be characterized as a bounding box position or position, a player position or position, a foot position or position, or the like) may be determined based on ankle points 401, 402 and a bounding box bottom line 405. In some embodiments, the foot point 406 is generated as the intersection of the bounding box bottom line 405 and a perpendicular line 404 that is perpendicular to the bounding box bottom line 405 and extends to a center position 403 (or intermediate position) between the ankle points 401, 402. For example, the central center point characterized as the foot point 406 may be used as a representative position of the bounding box 411 and the player 412 in a 3D global coordinate system applied to the scene 210 (not shown). In particular, foot points of the same player from different camera views will overlap in the global coordinate system. As shown, the intersection point of the bounding box bottom line 405 with a vertical line 404 extending through the center position 403 of the ankle points 401, 402 of the player 412 is used as a proxy for the bounding box bottom line center for improved accuracy. The ankle points 401, 402 may be determined using any suitable technique or techniques. In some embodiments, a deep high-resolution representation network for human pose estimation is employed to determine the ankle points 401, 402.
[0024] Each footprint 406 (e.g., for each bounding box in each video frame corresponding to each view) is then projected, and a projection matrix for the camera view (e.g., a projection matrix for transforming from a video frame to a 3D coordinate system) is used to project all footprints from each camera view onto the ground plane of the 3D coordinate system. These projected points represent the player positions on the court from different views.
[0025] 3, based on such techniques, the bounding box correspondence construction module 301 generates positions on the ground plane of a 3D coordinate system for each bounding box in each video frame at each time instance. The bounding box correspondence construction module 301 then determines correspondences between the bounding boxes using such positions and distances between them (at each time instance or time frame) as follows:
[0026] The camera array 101 includes M cameras, and each video frame (for each camera view) is m Assuming that we have bounding boxes, the toe distances between each combination of bounding boxes between every pair (e.g., each available pairing) of video frames (or camera views) are determined and provided in a distance matrix. For example, for each two camera view pairs i and j, i ×N j Distance matrix
number
number
number
[0027] For all M cameras in the camera array 101,
number
number
number
number
number
[0028] As discussed, the camera array 101 may include any number of cameras, such as 15 or more cameras. For example, in a soccer match employing 18 cameras, each view is assumed to include 22 players and 3 referees. In such a situation, determining the correspondence matrix would be excessively time consuming to perform in real time. In some embodiments, to reduce computational complexity and enable real-time applications, hard constraints are employed to split the matrix into sub-matrices that can be optimized separately. For example, N i ×N j For a distance matrix (and correspondence matrix) with dimensions j, a correspondence is constructed (e.g., evaluated) only if bounding box j has the smallest distance from bounding box i, and at the same time i has the smallest distance from j. That is, for an example bounding box i in a first video frame (or first view), the bounding box in the second video frame (or second view) with the smallest distance is determined (from all available bounding boxes in the second video frame). Assuming that the smallest distance bounding box in the second video frame (or second view) is j, a correspondence is declared between i and j only if the bounding box in the first video frame (or first view) with the smallest distance from j is also i.
[0029] After generating such hard-constrained correspondences, the distance matrix is calculated as follows:
number
number
number
[0030] FIG. 5 illustrates an example distance matrix 500 arranged in accordance with at least some embodiments of the present disclosure. In the example, a situation including three cameras (or video frames or views) and three bounding boxes per camera is presented for clarity of illustration. As discussed herein, any number of cameras (e.g., approximately 15 to 40 cameras) and any number of detected bounding boxes per camera (e.g., approximately 15 to 25) may be employed. Furthermore, the distance matrix 500 is generated by pairing each combination of video frames to determine the distance between the bounding boxes of the video frames, as discussed herein, and the distance matrix 500 may be described as an overall or aggregate distance matrix for a particular time instance. In particular, the distance matrix 500 may be generated for any or all time instances of the video frames.
[0031] In the example of Figure 5, three cameras are employed: A, B, and C. Furthermore, each camera includes three bounding boxes: 1, 2, 3. Thus, camera A has three bounding boxes A1, A2, A3, camera B has three bounding boxes B1, B2, B3, and so on. As discussed, in general, the distance matrix 500
number
[0032] As discussed, a correspondence matrix similar to distance matrix 500 is generated to determine correspondence between bounding boxes, such that for each position in distance matrix 500, a value of 1 (indicating a correspondence) or 0 (indicating no correspondence) is provided in the correspondence matrix. Furthermore, as discussed, optimizing the correspondence matrix based on application of equation (4) with the entire distance matrix 500 is excessively time consuming and computationally intensive for real-time applications. Instead, a hard constraint is applied to split distance matrix 500 (and the correspondence matrix) into smaller sub-matrices, which are solved separately to determine correspondence.
[0033] In some embodiments, employing the hard constraints includes building a correspondence only when a bidirectional minimum distance is determined, i.e., for every camera pair, a bounding box pair flag is set between the bounding boxes of the different camera views in response to a correspondence in which the bounding box in the first view has the smallest distance from the bounding box in the second view, and vice versa.
[0034] FIG. 6 illustrates an example bounding box pairing 600 using two-way minimum distance arranged in accordance with at least some embodiments of the present disclosure. In the example of FIG. 6, the bounding box pairing between camera A and camera B is shown for clarity of view. In the example, each grid box includes the distance along the ground plane in the 3D coordinate system between the footprints of the bounding boxes. As shown, the distance between a first bounding box in camera view A and a second bounding box in camera view B is 2 units (e.g., the distance from A1 to B2 is 2 units), and so on. To determine the pair of bounding boxes with two-way minimum distance, starting with bounding box A1, we find that the minimum distance bounding box is in view B2, which is at a distance of 2 units. For A1 and B2 to be paired using two-way minimum distance, in this case, bounding box A1 also needs to be the minimum distance bounding box in view A for bounding box B2. Such a process continues for each bounding box to determine two-way minimum distance flags 601, 602, 603 pairing bounding boxes A1 and B2, A2 and B1, and A3 and B3.
[0035] FIG. 7 is another diagram illustrating an example bounding box pairing 700 using two-way minimum distance arranged in accordance with at least some embodiments of the present disclosure. In the example of FIG. 7, two camera views A and B are used again, but now the number of bounding boxes in each view is six, such that camera view A has bounding boxes A1, A2, A3, A4, A5, and A6, and camera view B has bounding boxes B1, B2, B3, B4, B5, and B6. Note that for clarity of presentation, not all two minimum distance pairs are shown in FIG. 7. In this example, the process continues as discussed with respect to FIG. 6. In particular, with respect to bounding box A4, the minimum distance bounding box in view A is determined to be bounding box B3 (with a distance of 2.2 units) as it is the smallest distance in column A4. For the bidirectional pair to be established, bounding box A4 also needs to be the minimum distance bounding box in view B for bounding box A4 as it is the smallest distance in row B3. Continuing with the processing of bounding box A6, bounding box B4 is the smallest distance bounding box in view B (with a distance of 7.77 units) as it is the smallest distance in column A6. However, since bounding box A6 is not the smallest distance bounding box for bounding box B4 in view A, the bidirectional pairing is not established. Instead, the smallest distance bounding box for bounding box B4 in view A is bounding box A2 (with a distance of 6.73 units) as it is the smallest distance in row B4. Thus, a bidirectional minimum distance flag 701 is provided for pairing bounding boxes A4 and B3, but no pairing flag is provided for grid box 702.
[0036] In particular, the dimension of each camera pair is N i ×N jFor the distance matrix j, a correspondence is established only if j (e.g., a bounding box index in camera view B) is the minimum distance to i (e.g., a bounding box index in camera view A) and at the same time i has the minimum distance to j. As shown in Figure 7, A4 and B3 are a corresponding pair, while A6 and B4 are not. With the two-way distance flags discussed (e.g., corresponding pairs), the distance matrix 500 is then converted and partitioned into sub-matrices to determine the final correspondences.
[0037] 8 illustrates an example bidirectional minimum distance flag matrix 800 and row-wise matrix transformations 811, 812 arranged in accordance with at least some embodiments of the present disclosure. In FIG. 8, grid boxes with shading indicate bidirectional minimum distance correspondence, as shown for grid box 801 (e.g., bounding boxes A2 and C2 are a corresponding pair indicated by flags in grid box 801), while grid boxes without shading do not indicate such a flag, as shown for grid box 802 (e.g., bounding boxes B1 and C3 are not paired). As shown in FIG. 8, bounding box A1 is paired with bounding boxes B2 and C1. In response, the matrices (e.g., distance matrix 500 and correspondence matrix) are transformed in accordance with base matrix transformations 811, 812 (each labeled swap) to move the row corresponding to bounding boxes B2 and C1 directly below the row corresponding to bounding box A1.
[0038] Fig. 9 illustrates a matrix 900 resulting from a row-wise transformation arranged in accordance with at least some embodiments of the present disclosure. As shown in Fig. 9, rows B2 and A2 have been swapped, as have rows C1 and A3, to merge correspondences toward the upper left corner of the matrix. In addition, Fig. 9 illustrates column-wise matrix transformations 911, 912 similar to row-wise matrix transformations 811, 812, to move columns corresponding to bounding boxes B2 and C1 directly adjacent to the column corresponding to bounding box A1. Such an elementary transformation again merges correspondences toward the upper left corner of the matrix.
[0039] FIG. 10 illustrates a matrix 1000 resulting from a column-wise transformation arranged in accordance with at least some embodiments of the present disclosure. As shown in FIG. 10, columns B2 and A2 are swapped, as are columns C1 and A3, to merge the correspondences toward the upper-left corner of the matrix. Furthermore, FIG. 10 illustrates that not only does bounding box B2 have a correspondence with bounding box A1, but bounding box B2 also has a correspondence with bounding box C2. Furthermore, bounding box C2 has a correspondence with bounding box A2, as shown with respect to boxes 1002, 1003. In response to such correspondences, the column corresponding to bounding box C2 and the row corresponding to bounding box C2 are moved, using a continuing elementary transformation, to a position after the correspondences in box 1001 (e.g., the correspondences of bounding boxes B1 and C1 and bounding boxes A2 and B1).
[0040] FIG. 11 illustrates a resultant matrix 1100 after continuing row-wise and column-wise transformations and example sub-matrices arranged in accordance with at least some embodiments of the present disclosure. As shown in FIG. 11, column C2 is swapped with column B3, and similarly row C2 is swapped with column B3 to further merge correspondences toward the upper left corner of the matrix. After the formation of the resultant matrix 1100, the resultant sub-matrices 1101, 1102 of the resultant matrix 1100 are solved separately via separate applications of equation (4) discussed above. In the example of FIG. 11, the resultant matrix 1100 is separated into two sub-matrices 1101, 1102. However, the resultant matrix 1100 may be separated into sub-matrices having any number of corresponding groupings, such as 3, 4, 5 or more sub-matrices.
[0041] Returning to Figure 3, the bounding box correspondence construction module 301 solves for the correspondence sub-matrices separately to generate correspondences at each time instance. The correspondence data indicates the bounding boxes between video frames at a time instance that correspond such that they are considered to contain the same player.
[0042] Such correspondence data is provided to the position tracking module 302 in successive frames. As discussed, player detection in each camera view may not be stable, which results in inconsistent detection results in adjacent frames (e.g., detection bounding box numbers are not the same even in the absence of player movement). For example, the associated bounding boxes may change in temporally adjacent frames, which results in unstable player positions, player misses, or player overlaps. To overcome such limitations, a multi-frame buffer (e.g., a buffer of multiple time frames) is employed to predict player positions and numbers. In some embodiments, two phases are utilized: player tracklet generation and player tracklet fusion. In particular, tracklet generation involves linking together the same players from different time frames, while tracklet fusion involves merging redundant tracklets and deleting erroneous tracklets.
[0043] FIG. 12 illustrates an example tracklet generation 1200, in which generation begins at each time frame, arranged in accordance with at least some embodiments of the present disclosure. As shown in FIG. 12, a buffer of tracking frames may be maintained, including frames 1201, 1202, 1203, 1204, 1205. The frame buffer may be of any size, such as 15 frames or the like. In the example of FIG. 12, each frame indicates the location of a corresponding bounding box at a particular time instance. For example, each player dot may be an average (or other aggregation) of the player's location from each of the camera views at the time instance based on the bounding boxes considered to be corresponding as discussed above. That is, each dot may be the average location on the ground plane in a 3D coordinate system of the locations of the corresponding bounding boxes. Furthermore, with respect to tracklets, a start frame is used to indicate the frame at which the tracklet was generated. Thus, the frame may be a frame after the start of the tracklet or the start frame of the tracklet.
[0044] In the situation of FIG. 12, a group of tracklets is generated using a number of frames in a frame buffer as a starting frame. The frame used as a starting point may be, for example, any frame. As a result, several groups of tracklets are generated and may be merged later as the groups of tracklets are compared, or some tracklets may be discarded. That is, the tracklets generated starting at frame 1 may be used as an initial group of tracklets. Then, the tracklets generated starting at frame 2 are compared to those generated starting at frame 1, and some tracklets may be discarded, added, or merged with respect to the initial group of tracklets. The tracklets generated starting at frame 3 are compared to these preceding tracklets (as tracked using the correspondence list), and some tracklets may again be discarded, added, or merged, etc. Such techniques are discussed further herein below.
[0045] Thus, a group of tracklets is generated from each frame as the starting frame. i Given, position correspondences in subsequent frames may be generated using any suitable technique or techniques, such as application of combinatorial optimization algorithms, including the Hungarian algorithm. For example, the position of player x in a starting frame s is given by:
number
number
number
number
[0046] The predicted position at frame t+1 is then
number
number
[0047] Referring to FIG. 12, starting from the start frame 1201, x,11 Shown is a player position 1211 denoted by (position of player x in frame 1 based on starting frame 1). Note that the processing discussed with respect to player position 1211 is performed in parallel for the other four player positions in starting frame 1201. A predicted position 1212 of the player is generated in frame 1202 as discussed with respect to equation (6), L' x,12(Predicted position of player x based on starting frame 1 in frame 2). In this example, the actual position does not correspond to the predicted position 1212, and the position may be added to the tracklet. The predicted position 1213 of the player is then generated as discussed with respect to equation (6), L' x,13 (Predicted player x position based on starting frame 1 at frame 3). Based on application of the Hungarian algorithm and a distance threshold, an actual player position 1214 is selected and added to the tracklet for player position 1211. The process continues by generating a predicted player position 1215, as discussed with respect to equation (6) and L' x,14 (Predicted player x position based on start frame 1 at frame 4). Based on application of the Hungarian algorithm and a distance threshold, an actual player position 1216 is selected and added to the tracklet for player position 1211. Such processing continues up to frame n, such that tracklet 1241 is generated for player position 1211. As discussed, beginning at start frame 1201, five tracklets are generated, including tracklet 1241.
[0048] Similarly, starting from start frame 1202, x,22 12. As shown, a player position 1221 (the position of player y in frame 2 based on the starting frame 2) is denoted by L'. As mentioned above, the processing discussed with respect to player position 1221 is performed in parallel for the other three player positions in starting frame 1202. In particular, as a player position corresponding to player position 1211 is not seen in starting frame 1202, a corresponding tracklet will not be generated. A predicted position 1222 of the player is generated in frame 1203, L' y,23(Predicted player y position based on starting frame 2 in frame 3). Based on the application of the Hungarian algorithm and a distance threshold, the actual player position 1223 (L y,23 ) is selected and added to the tracklet for player position 1222. The process continues with L' y,24 We continue by generating a predicted position 1224 of the player as indicated by (Predicted player y position based on starting frame 2 at frame 4). Based on the application of the Hungarian algorithm and a distance threshold, we generate an actual player position 1225 (L y,24 ) is selected and added to the tracklet for player position 1211. Such processing continues up to frame n, such that tracklet 1242 is generated for player position 1221. For example, beginning at start frame 1202, four tracklets are generated, including tracklet 1242.
[0049] Such processing is z,33 This can continue for any number of starting frames, as shown with respect to starting frame 1203, where player position 1231 is indicated by (position of player z based on starting frame 3 in frame 3). The process as discussed above can continue for any number of starting frames, as shown with respect to starting frame 1203, where player position 1234 (L z,34 ) for player position 1231. For example, starting at start frame 1203, six tracklets are generated, including tracklet 1243.
[0050] 3, after such tracklet generation, the position tracking in successive frames module 302 performs a tracklet fusion operation to provide robust player tracking. For example, after position correspondence construction, each player in frame i is generated a correspondence list
number
number
[0051] In some embodiments, for each player in frame i,
number
number
number
number
number
number
number
number
number
number
number
number
number
number
[0052] 13 is a diagram illustrating an example tracklet fusion 1300, arranged in accordance with at least some embodiments of the present disclosure. As shown in FIG. 13, for each of frames 1301, 1302, 1303, 1304, 1305, a tracklet set has been generated starting from each such frame. For example, starting from frame 1301, tracklet 1313 is generated for player 1311 (indicating player 1 with a starting frame of 1).
number
number
number
number
number
[0053] 14 illustrates an example process 1400 for performing tracklet fusion, arranged in accordance with at least some embodiments of the present disclosure. The process 1400 may include one or more operations 1401-1411, as shown in FIG 14. For example, the operations 1401-1411 may be performed by the position tracking module 302 in successive frames.
[0054] The process starts with an empty player set (C a The process begins with operation 1401 where a set of player sets (or player sets) are generated. The resulting filled player set will show player and bounding box correspondence across consecutive frames or time instances, as discussed herein. Processing continues at operation 1402, where a frame is selected (in temporal order) and a tracklet or player list generated using the frame as a starting frame is obtained. For example, for frame 1301 as the starting frame, tracklets 1313, 1314 are obtained, for frame 1302 as the starting frame, tracklets 1324, 1325, 1327 are obtained, for frame 1303 as the starting frame, tracklets 1331, 1332, 1333, 1334 are obtained, and so on.
[0055] Processing continues at operation 1403 where the tracklets for the frame are compared to the players in the player set. Processing continues at decision operation 1404 where for each of the tracklets a determination is made as to whether the player position in the current frame is in any of the tracklets (player positions) already in the player set. Here, an intersection indicates that two tracklets share the same position in the same frame. If so, the tracklet or player position is not added to the overall player set; instead, the tracklet is discarded as redundant at operation 1405. If no intersection is found, processing continues at decision operation 1406 where, for each tracklet that does not have an intersection, the tracklet (
number
number
[0056] If not, processing continues at decision operation 1408, where a determination is made as to whether the length of the tracklet that does not have an intersection (and therefore does not have an intersection with any player in the overall player set) and is not merged has a length greater than a threshold TH2 (e.g., one-third of the buffer size). If so, then at operation 1409, the tracklet is added to the overall player set as a new player. If not, then the tracklet is not added (i.e., discarded) and processing continues at operation 1410, where a determination is made as to whether the last frame for processing has been selected. If so, processing ends at operation 1411. If not, processing continues at operation.
[0057] The operation of the process 1400 will now be illustrated with respect to FIG. 13. The process begins with an empty player set (C a ) As discussed, the process begins with each player (
number
number
number
number
number
number
number
number
[0058] For example, the process starts with an empty player set (C a ) and then tracklet 1313 (
number
number
number
number
[0059] Similarly, tracklet 1314 (
number
number
number
number
number
[0060] Now, move to frame 1302 and add tracklet 1324 (
number
number
number
number
number
number
number
[0061] Next, tracklet 1327(
number
number
number
number
number
number
number
number
[0062] Moving now to frame 1303, tracklets 1331, 1332, and 1333 intersect with tracklets 1313, 1314, and 1327, as discussed, and are therefore discarded.
number
number
number
number
number
number
number
number
number
number
[0063] Returning to FIG. 3, the bounding box correspondence construction module 301 and the position tracking in consecutive frames module 302 generate bounding box association results and player positions in each frame for each frame from the multi-camera views. As discussed, in order to obtain stable player positions and player numbers, player merging, interpolation, and deletion are performed to refine the associated player positions. Therefore, due to association errors and new position interpolation, the current player positions may not match the association results. To address such issues, large distance elimination and adjacent player bounding box reconstruction are used to refine the association results via the post-processing module 303. In particular, the post-processing module 303 performs reconstruction and distance elimination to provide the final association list 122.
[0064] FIG. 15 illustrates an example player reconstruction or reallocation process 1500, arranged in accordance with at least some embodiments of the present disclosure. In FIG. 15, in a particular frame 1501, bounding boxes or player positions from multiple views are shown such that corresponding players have matching dot colors or patterns. For example, player positions 1502 are all from the same player and are grouped in a tight pattern indicating accurate player association results. Similar results are seen for other player positions outside of the region of interest 1503, which are also shown in a zoomed-in view for clarity of view. As shown, a first player position 1504 (indicated by darker shaded dots) blends in with a second player position 1504 (indicated by white dots).
[0065] Additionally, a hatched circle 1507 is shown at the center of a first player position 1504 after bounding box correspondence construction (and before any adjustments due to multi-frame position tracking), and a solid circle 1506 shows the center for the first player after multi-frame position tracking. Similarly, a hatched circle 1508 is shown at the center of a second player position 1505 after bounding box correspondence construction (and before any adjustments due to multi-frame position tracking), and a solid circle 1509 shows the center for the second player after multi-frame position tracking.
[0066] In FIG. 15, inconsistencies in player positions, possibly due to tracklet fusion, are evident for a first player position 1504 and a second player position 1505. In particular, association errors are common when players are very close to each other. To eliminate such problems, a distance threshold is employed to select players in a crowd of players and provide association refinement. In some embodiments, given a distance threshold δ, for frame 1501, pairwise distances are determined between all player positions in the frame. The distance d between positions i and j is ij is smaller than a threshold (e.g., d ij < δ), then the pair is added to a candidate list. The association bounding box list is then reordered for such candidate list positions. For each footpoint, the distances to the new association are centered (e.g., taking the average of each candidate player's positions). The player position (or footpoint) is then added to or belongs to the association center with the smallest distance value. Note that for each association center, one camera view has only one footpoint. If two footpoints have the smallest distance to one association center, the footpoint with the larger distance (comparing between the two footpoint-to-association center distances) is moved to the other association center, etc.
[0067] For example, as shown with respect to the reconfiguration operation 1530 of FIG. 15, player position 1502 is likely to be added to the same candidate list and have the smallest distance to the corresponding association center (not shown), and therefore no reconfiguration occurs for player position 1502. In the region of interest 1503, a first player position 1504 and a second player position 1505 are added to the same candidate list and then repositioned to be closest to the corresponding association center to generate a first player position 1514 and a second player position 1515 with different player assignments for such positions. As shown, such a reconfiguration operation 1530 swaps the player identities of player position 1521 (changing from second player to first player) and player position 1522 (changing from first player to second player), as well as others. Such a technique provides a more robust bounding box correspondence for the video frames or frames 1501 of a time instance.
[0068] FIG. 16 illustrates an example large distance player removal process 1600 arranged in accordance with at least some embodiments of the present disclosure. In FIG. 16, in a particular frame 1601, bounding boxes or player positions from multiple views are shown, again with corresponding players having matching dot colors or patterns. For example, player positions 1602 are all from the same player and are grouped in a tight pattern indicating accurate player association results. In the region of interest 1603 shown in the zoomed-in view, the first player position 1504 is also tightly grouped. However, a second player position 1605 includes a player position 1610, which is dissociated from the others in the group. In particular, the hatched circle 1607 showing the center of the first player position 1604 after bounding box correspondence construction (and before any adjustments due to multi-frame position tracking) and the solid circle 1606 showing the center for the first player after multi-frame position tracking have similar or identical positions. However, the hatched circle 1608 showing the center of the second player position 1605 after bounding box correspondence construction (and before any adjustments due to multi-frame position tracking) and the solid circle 1609 showing the center for the second player after multi-frame position tracking are spaced apart due to the outlier player position 1610.
[0069] In some embodiments, for each player position in the grouping, the distance from the player position to the association center (e.g., the average position of all corresponding bounding boxes after correspondence building and before adjustments due to position tracking in multi-frames) is compared to a threshold. If the distance is less than the threshold, the player position is kept in the grouping. If the distance is greater than the threshold, the player position is discarded as a false identification, as shown with respect to the removal operation 1630. In some embodiments, footpoints with large distances to their associated centers are discarded. In some embodiments, footpoints with large distances to all association centers are discarded. For example, the footpoint may be moved to another association center.
[0070] For example, to keep player positions stable and accurate, footpoints that are far away from other associated footpoints (e.g., measured using the association center) are removed. Footpoints with large distances relative to other footpoints are typically isolated points caused by incorrect associations or player misdetections (e.g., bounding boxes containing several players). Such points, if not discarded, will cause jitter in player positions.
[0071] 17 is a flow diagram illustrating an example process 1700 for performing object association in a multi-view video, arranged in accordance with at least some embodiments of the present disclosure. The process 1700 may include one or more operations 1701-1705 as shown in FIG. 17. The process 1700 may form at least a portion of an object association process for a multi-view video, for example. As a non-limiting example, the process 1700 may form at least a portion of a process as performed by the system 100 as discussed herein. Furthermore, the process 1700 will be described herein with reference to the system 1800 of FIG. 18.
[0072] 18 is an illustration of an example system 1800 for performing object association in multi-view video, arranged in accordance with at least some embodiments of the present disclosure. As shown in FIG. 18, the system 1800 includes a central processor 1801, an image processor 1802, a memory 1803, and a camera array 101. As also shown, the image processor 1802 may include or implement a multi-camera association module 114 including a bounding box correspondence construction module 301, a position tracking module 302 in successive frames, and a post-processing module 303. In the example of the system 1800, the memory 1803 may store video frames, bounding box data, object or player position data, association list data, or any other data discussed herein.
[0073] As shown, in some examples, one or more or parts of the multi-camera association module 114 including the bounding box correspondence construction module 301, the position tracking module in successive frames 302, and the post-processing module 303 are implemented via the image processor 1802. In other examples, one or more or parts of the multi-camera association module 114 including the bounding box correspondence construction module 301, the position tracking module in successive frames 302, and the post-processing module 303 are implemented via the central processor 1801, an image processing unit, an image processing pipeline, an image signal processor, or the like. In some examples, one or more or parts of the multi-camera association module 114 including the bounding box correspondence construction module 301, the position tracking module in successive frames 302, and the post-processing module 303 are implemented in hardware as a system on chip (SoC). In some examples, one or more or portions of the multi-camera association module 114, including the bounding box correspondence construction module 301, the position tracking in successive frames module 302, and the post-processing module 303, are implemented in hardware via an FPGA.
[0074] The image processor 1802 may include any number and type of image or graphics processor or processing unit that may provide operations as discussed herein. Such operations may be implemented via software or hardware, or a combination thereof. For example, the image processor 1802 may include specialized circuitry for manipulating and / or analyzing frames or frame data obtained from the memory 1803. The central processor 1101 may include any number and type of processors, processing units, or modules that may provide control and other high-level functions for the system 1800 and / or provide any operations as discussed herein. The memory 1803 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, the memory 1803 may be implemented by a cache memory. In one embodiment, one or more or parts of the multi-camera association module 114 including the bounding box correspondence construction module 301, the position tracking module in successive frames 302, and the post-processing module 303 are implemented via an execution unit (EU) of the image processor 1802. The EU may include, for example, programmable logic, such as a logic core or cores, or circuitry that may provide a wide range of programmable logic functions. In one embodiment, one or more or parts of the multi-camera association module 114 including the bounding box correspondence construction module 301, the position tracking module in successive frames 302, and the post-processing module 303 are implemented via dedicated hardware, such as a fixed function circuit or the like. A fixed function circuit may include dedicated logic or circuitry and may provide a set of fixed function input points that may be mapped to dedicated logic for a fixed purpose or function.
[0075] Returning to the discussion of FIG. 17, process 1700 begins at operation 1701 where bounding box data for each of a plurality of video frames is received, such that each of the plurality of video frames corresponds to a particular view of a scene at the same time instance. For example, the video frames may be synchronized video frames, such that they capture simultaneous or near-simultaneous video frames or video images of a scene, such as a sporting event. The techniques discussed herein provide correspondence across video frames at the same time instance, as well as across multiple time instances for a set of video frames. Such time instances are also referred to herein as time frames or simply frames.
[0076] Processing continues at operation 1702, where a distance matrix is generated based on the bounding box data such that the distance matrix includes distances between all available combinations of bounding boxes between pairs of video frames. For example, the distances may be generated by pairing the video frames in all pair-wise combinations, inputting a distance value for each combination of bounding boxes between the frames, and repeating the distance values for all such pairs of video frames.
[0077] Processing continues at operation 1703, where a distance submatrix is determined from the distance matrix based on two-way minimum distance corresponding bounding box pairs in the distance matrix. For example, such two-way minimum distance correspondence may indicate that for a first bounding box in a first frame, a second bounding box in a second frame is a minimum distance bounding box, and for the second bounding box, the first bounding box is also a minimum distance bounding box. In some embodiments, a first bidirectional minimum distance correspondence bounding box pair of the bidirectional minimum distance correspondence bounding box pair includes a correspondence between a first bounding box in the first video frame and a second bounding box in the second video frame, such that the first bidirectional minimum distance correspondence bounding box pair is responsive to the second bounding box having a minimum distance from the first bounding box from all available distances from the first bounding box to all candidate bounding boxes in the second video frame and the first bounding box having a minimum distance from the second bounding box from all available distances from the second bounding box to all candidate bounding boxes in the first video frame.
[0078] Further, the distance submatrix is smaller than the original distance matrix and is generated by rearranging the rows and columns of the original distance matrix and then segmenting it into submatrices. In some embodiments, determining the submatrix includes transforming the distance matrix based on the two-way minimum distance corresponding bounding box pairs via row and column basic transformations to merge the two-way minimum distance corresponding bounding box pairs into the submatrix.
[0079] Processing continues at operation 1704, where a corresponding submatrix is determined for each of the distance submatrices separately based on optimizing an objective function that includes each corresponding submatrix and each distance submatrix. For example, the objective function may include a sum of inner products of the distance submatrix and the corresponding submatrix summed with a product of a low rank constraint and the rank of the corresponding submatrix.
[0080] Processing continues at operation 1705, where an association list is provided for the bounding box data of the video frames based on the correspondence sub-matrices. The association list may include any suitable data structure indicating correspondence between bounding boxes across the video frames. For example, a first index may be provided for a first object or player, and the association list may include a frame number and a bounding box number for each bounding box that corresponds to the first index, the same may be provided for a second index, and so on. Alternatively, an index may be attached to each bounding box indicating the object or player that the bounding box corresponds to. Other techniques for indicating such associations or correspondences are available.
[0081] Further, the association list for a set of temporally adjacent video frames (e.g., time instances) may be used to generate the object or player tracking. Further, the object or player tracking may be used to update or modify the association list via the set of tracklets. In some embodiments, the process 1700 further includes generating a first set of tracklets using the same time instance as a starting time frame, generating one or more additional sets of tracklets using additional time instances subsequent to the same time instance as starting time frames, and adjusting the initial association list for one of the time instances using the first set of tracklets and the additional set of tracklets. In some embodiments, a first tracklet of the first set of tracklets is generated by projecting a position of a bounding box for the same time instance onto a second position of a bounding box for a second time instance subsequent to the same time instance, and adding an object position at the second time instance to the first tracklet in response to applying a combinatorial optimization algorithm and the object position at the second time instance being within a threshold distance of the object position at the same time instance.
[0082] In some embodiments, the process 1700 further includes blending the first set of tracklets with one or more sets of additional tracklets to generate a resultant set of tracklets. In some embodiments, blending the first set of tracklets with the one or more sets of additional tracklets includes determining that a second tracklet in the second set of the one or more sets of additional tracklets does not intersect with any tracklets in the first set of tracklets, and adding the second tracklet to the resultant set of tracklets in response to the second tracklet having a length that exceeds a threshold. In some embodiments, the step of blending the first set of tracklets with the one or more additional sets of tracklets includes determining that a second tracklet in the second set of the one or more additional sets of tracklets does not intersect with any tracklet in the first set of tracklets, and merging the second tracklet with a third tracklet in the resulting set of tracklets in response to a sum of the lengths of the second and third tracklets being smaller than a buffer size corresponding to the same and additional time instances.
[0083] Further, after such tracklet generation and fusion, the (adjusted) one or more association lists may again be modified to correct outliers and provide better stability. In some embodiments, providing the association list includes adjusting one or more objects in the association list based on a comparison of the positions of the objects in the association list after position tracking of the bounding box data. In some embodiments, adjusting the one or more objects includes changing an association for a first object bounding box in the first video frame from a first object association to a second object association in response to a first distance of the first object bounding box to a center position of the first object association exceeding a second distance of the first object bounding box to a center position of the second object association. In some embodiments, adjusting the one or more objects includes discarding a first object bounding box in the first video frame in response to the first object bounding box being farther than a threshold distance from the first object association position.
[0084] Various components of the systems described herein may be implemented in software, firmware, and / or hardware and / or any combination thereof. For example, various components of the devices or systems discussed herein may be provided at least in part by hardware in a computing system-on-chip (SoC), such as may be found in a computing system such as a smartphone. Those skilled in the art may recognize that the systems described herein may include additional components not shown in the corresponding figures. For example, the systems discussed herein may include additional components not shown for clarity.
[0085] While examples of example processes discussed herein may include performing all of the operations shown in the order illustrated, the disclosure is not limited in this respect and in various examples, examples of example processes herein may include only a subset of the operations shown, may include operations performed in an order different from that shown, or may include additional operations.
[0086] Furthermore, any one or more of the operations discussed herein may be performed in response to instructions provided by one or more computer program products. Such program products may include, for example, a signal-bearing medium providing instructions that, when executed by a processor, may provide the functionality described herein. A computer program product may be provided in any form of one or more machine-readable media. Thus, for example, a processor including one or more graphics processing units or processor cores may perform one or more of the blocks of the example processes herein in response to program code and / or instructions or instruction sets being conveyed to the processor by one or more machine-readable media. In general, a machine-readable medium may convey software in the form of program code and / or instructions or instruction sets that may cause any device and / or system described herein to implement at least a portion of the device or system or any other component or module as discussed herein.
[0087] As used in any embodiment described herein, the term "module" refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. Software may be embodied as a software package, code, and / or instruction set or instructions, and "hardware" as used in any embodiment described herein may include, for example, hardwired circuitry, programmable circuitry, state machine circuitry, fixed function circuitry, execution unit circuitry, and / or firmware that stores instructions executed by programmable circuitry, alone or in any combination. These modules may be embodied collectively or individually as circuitry that forms part of a larger system, such as, for example, an integrated circuit (IC) system-on-chip (SoC).
[0088] 19 is an illustration of an example system 1900 arranged in accordance with at least some embodiments of the present disclosure. In various embodiments, the system 1900 can be a mobile device system, although the system 1900 is not limited in this context. For example, the system 1900 can be incorporated into a personal computer (PC), a laptop computer, an ultra laptop computer, a tablet, a touchpad, a portable computer, a handheld computer, a palmtop computer, a personal digital assistant (PDA), a mobile phone, a mobile phone / PDA combination, a television, a smart device (e.g., a smartphone, a smart tablet, or a smart television), a mobile internet device (MID), a messaging device, a data communication device, a camera (e.g., a point-and-shoot camera, a super zoom camera, a digital single-lens reflex (DSLR) camera), a surveillance camera, a surveillance system including a camera, and the like.
[0089] In various embodiments, system 1900 includes a platform 1902 coupled to a display 1920. Platform 1902 may receive content from a content device, such as a content services device 1930 or a content delivery device 1940, or other content source, such as an image sensor 1919. For example, platform 1902 may receive image data as discussed herein from image sensor 1919 or any other content source. A navigation controller 1950 including one or more navigation features may be used, for example, to interact with platform 1902 and / or display 1920. Each of these components is described in more detail below.
[0090] In various embodiments, platform 1902 may include any combination of chipset 1905, processor 1910, memory 1912, antenna 1913, storage 1914, graphics subsystem 1915, applications 1916, image signal processor 1917, and / or radio 1918. Chipset 1905 may provide intercommunication between processor 1910, memory 1912, storage 1914, graphics subsystem 1915, applications 1916, image signal processor 1917, and / or radio 1918. For example, chipset 1905 may include a storage adapter (not shown) capable of providing intercommunication with storage 1914.
[0091] The processor 1910 may be implemented as a complex instruction set computer (CISC) processor or a reduced instruction set computer (RISC) processor, an x86 instruction set compatible processor, a multi-core, or any other microprocessor or central processing unit (CPU). In various embodiments, the processor 1910 may be a dual-core processor, a dual-core mobile processor, or the like.
[0092] The memory 1912 may be implemented as a volatile memory device, such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), or static RAM (SRAM).
[0093] Storage 1914 may be implemented as a non-volatile storage device, such as, but not limited to, a magnetic disk drive, an optical disk drive, a tape drive, an internal storage device, an attached storage device, flash memory, battery-backed SDRAM (synchronous DRAM), and / or a network-accessible storage device. In various embodiments, storage 1914 may include technology to promote enhanced protection of storage performance for expensive digital media, for example, when multiple hard drives are included.
[0094] The image signal processor 1917 may be implemented as a dedicated digital signal processor or the like used for image processing. In some examples, the image signal processor 1917 may be implemented based on a single instruction multiple data or multiple instruction multiple data architecture or the like. In some examples, the image signal processor 1917 may be characterized as a media processor. As discussed herein, the image signal processor 1917 may be implemented based on a system-on-chip architecture and / or based on a multi-core architecture.
[0095] The graphics subsystem 1915 may perform processing of images, such as still images or video, for display. The graphics subsystem 1915 may be, for example, a graphics processing unit (GPU) or a visual processing unit (VPU). An analog or digital interface may be used to communicatively couple the graphics subsystem 1915 and the display 1920. For example, the interface may be any of High Definition Multimedia Interface, DisplayPort, Wireless HDMI, and / or Wireless HD compliant technologies. The graphics subsystem 1915 may be integrated into the processor 1910 or the chipset 1905. In some embodiments, the graphics subsystem 1915 may be a separate device communicatively coupled to the chipset 1905.
[0096] The graphics and / or video processing techniques described herein may be implemented in a variety of hardware architectures. For example, the graphics and / or video capabilities may be integrated within a chipset. Alternatively, a separate graphics and / or video processor may be used. As yet another example, the graphics and / or video capabilities may be provided by a general purpose processor, including a multi-core processor. In a further embodiment, the capabilities may be implemented in a consumer electronic device.
[0097] Radio 1918 may include one or more radios capable of transmitting and receiving signals using a variety of suitable wireless communication technologies. Such technologies may involve communication over one or more wireless networks. Examples of wireless networks include, but are not limited to, wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area networks (WMANs), cellular networks, and satellite networks. In communicating across such networks, the radio 1918 may operate according to one or more applicable standards in any version.
[0098] In various embodiments, display 1920 may include any television-type monitor or display. Display 1920 may include, for example, a computer display screen, a touch screen display, a video monitor, a television-like device, and / or a television. Display 1920 may be digital and / or analog, in various embodiments, display 1920 may be a holographic display. Additionally, the display 1920 may be a transparent surface capable of receiving a visual projection. Such projections may convey various forms of information, images and / or objects, for example, such projections may be visual overlays for mobile augmented reality (MAR) applications. Under the control of one or more software applications 1916 , the platform 1902 may display a user interface 1922 on the display 1920 .
[0099] In various embodiments, content services device(s) 1930 may be hosted by any national, international, and / or independent service and thus accessible to platform 1902, for example, via the Internet. Content services device(s) 1930 may be coupled to platform 1902 and / or to display 1920. Platform 1902 and / or content services device(s) 1930 may be coupled to network 1960 to communicate (e.g., send and / or receive) media information to and from network 1960. Content delivery device(s) 1940 may also be coupled to the platform 1902 and / or the display 1920.
[0100] Image sensor 1919 may include any suitable image sensor that may provide image data based on a scene. For example, image sensor 1919 may include a semiconductor charge-coupled device (CCD) based sensor, a complementary metal-oxide semiconductor (CMOS) based sensor, an N-type metal-oxide semiconductor (NMOS) based sensor, or the like. For example, image sensor 1919 may include any device that may detect information of a scene to generate image data.
[0101] In various embodiments, content services device(s) 1930 may include a cable television box, a personal computer, a network, a telephone, an Internet-enabled device or appliance capable of delivering digital information and / or content, and any other similar device capable of uni- or bi-directionally communicating content between a content provider and platform 1902 and / or display 1920 over network 1960 or directly. It will be understood that content may be communicated uni- and / or bi-directionally from and to any one of the components in system 1900 and the content provider over network 1960. Examples of content may include any media information including, for example, video, music, medical and gaming information, and the like.
[0102] Content services device 1930 may receive content such as cable television programming, including media information, digital information, and / or other content. Examples of content providers may include any cable or satellite television, or over-the-air or Internet content provider. The examples provided are not meant to limit the embodiments of the present disclosure in any way.
[0103] In various examples, the platform 1902 may receive control signals from a navigation controller 1950 having one or more navigation functions. The navigation functions of the navigation controller 1950 may be used, for example, to interact with the user interface 1922. In various embodiments, the navigation controller 1950 may be a pointing device, which may be a computer hardware component (particularly a human interface device) that allows a user to input spatial (e.g., continuous and multi-dimensional) data into a computer. Many systems, such as graphical user interfaces (GUIs) and televisions and monitors, allow a user to control and provide data to a computer or television using physical gestures.
[0104] Movement of navigation features of navigation controller 1950 may be replicated on a display (e.g., display 1920) by movement of a pointer, cursor, focus ring, or other visual indicator displayed on the display. For example, under control of software application 1916, navigation features located on navigation controller 1950 may be mapped to virtual navigation features displayed, for example, on user interface 1922. In various embodiments, navigation controller 1950 may not be a separate component, but may be integrated into platform 1902 and / or display 1920. However, this disclosure is not limited to the components or aspects shown or described herein.
[0105] In various examples, the drivers (not shown) may include technology that, when enabled, allows a user to quickly turn the platform 1902 on and off after initial boot-up, for example, with the touch of a button, similar to a television. Even when the platform is "off," program logic may enable the platform 1902 to stream content to a media adapter or other content service device 1930 or content delivery device 1940, and further, the chipset 1905 may include hardware and / or software support for, for example, 5.1 surround sound audio and / or high-definition 7.1 surround sound audio. The drivers may include graphics drivers for an integrated graphics platform. In various embodiments, the graphics drivers may include a peripheral component interconnect (PCI) express graphics card.
[0106] In various embodiments, any one or more of the components illustrated in system 1900 may be integrated. For example, platform 1902 and content services device(s) 1930 may be integrated, or platform 1902 and content delivery device(s) 1940 may be integrated, or, for example, platform 1902, content services device(s) 1930, and content delivery device(s) 1940 may be integrated. In various embodiments, platform 1902 and display 1920 may be an integrated unit. For example, display 1920 and content services device(s) 1930 may be integrated, or display 1920 and content delivery device(s) 1940 may be integrated. These examples are not meant to limit the disclosure.
[0107] In various embodiments, the system 1900 may be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, the system 1900 may include components and interfaces suitable for communicating over a wireless shared medium, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc. Examples of wireless shared media may include portions of a wireless spectrum, such as the RF spectrum. When implemented as a wired system, the system 1900 may include components and interfaces suitable for communicating over a wired communication medium, such as input / output (I / O) adapters, physical connectors for connecting the I / O adapters with corresponding wired communication media, network interface cards (NICs), disk controllers, video controllers, and audio controllers. Examples of wired communication media may include wires, cables, metal leads, printed circuit boards (PCBs), backplanes, switch fabrics, semiconductor materials, twisted pair wires, coaxial cables, optical fibers, etc.
[0108] The platform 1902 may establish one or more logical or physical channels to communicate information. The information may include media information and control information. Media information may refer to any data representing content for a user. Examples of content may include, for example, data from a voice conversation, a video conference, streaming video, electronic mail ("email") messages, voicemail messages, alphanumeric symbols, graphics, images, video, text, etc. Data from a voice conversation may be, for example, speech information, silence periods, background noise, comfort noise, tones, etc. Control information may refer to any data representing commands, instructions, or control words for an automated system. For example, control information may be used to route media information through a system or to instruct a node to process the media information in a predetermined manner. However, embodiments are not limited to the elements or contexts shown or described in FIG. 19 .
[0109] As mentioned above, the system 1900 may be embodied in a variety of physical styles or form factors. FIG. 20 illustrates an example small form factor device 2000 arranged in accordance with at least some embodiments of the present disclosure. In some examples, the system 1900 may be implemented via the device 2000. In other examples, other systems, components, or modules discussed herein, or portions thereof, may be implemented via the device 2000. For example, in various embodiments, the device 2000 may be implemented as a mobile computing device with wireless capabilities. For example, a mobile computing device may refer to any device having a processing system and a mobile power source or supply, such as one or more batteries.
[0110] Examples of mobile computing devices may include personal computers (PCs), laptop computers, ultra laptop computers, tablets, touchpads, portable computers, handheld computers, palmtop computers, personal digital assistants (PDAs), mobile phones, mobile phone / PDA combinations, televisions, smart devices (e.g., smartphones, smart tablets, or smart mobile televisions), mobile internet devices (MIDs), messaging devices, data communications devices, cameras (e.g., point-and-shoot cameras, super zoom cameras, and digital single-lens reflex (DSLR) cameras), and the like.
[0111] Examples of mobile computing devices may also include computers arranged to be implemented by automobiles or robots or worn by people, such as wrist computers, finger computers, ring computers, eyeglass computers, belt clip computers, armband computers, shoe computers, clothing computers, and other wearable computers. In various embodiments, for example, a mobile computing device may be implemented as a smartphone capable of executing computer applications and voice and / or data communications. Some embodiments may be described using a mobile computing device implemented as a smartphone as an example, but it may be understood that other embodiments may be implemented using other wireless mobile computing devices as well. The embodiments are not limited in this context.
[0112] As shown in FIG. 20, device 2000 may include a housing having a front surface 2001 and a back surface 2002. Device 2000 includes a display 2004, an input / output (I / O) device 2006, a color camera 2021, a color camera 2022, and an integrated antenna 2008. In some embodiments, color camera 2021 and color camera 2022 acquire planar images as discussed herein. In some embodiments, device 2000 does not include color cameras 2021 and 2022, and device 2000 acquires input image data (e.g., any input image data discussed herein) from another device. Device 2000 may also include navigation functionality 2012. I / O device 2006 may include any suitable I / O device for inputting information into a mobile computing device. Examples of I / O devices 2006 may include an alphanumeric keyboard, numeric keypad, touchpad, input keys, buttons, switches, microphones, speakers, voice recognition devices, software, and the like. Information may also be entered into device 2000 by way of a microphone (not shown) or digitized by a voice recognition device. As shown, device 2000 may include color cameras 2021, 2022 and a flash 2010 integrated into the back 2002 (or elsewhere) of device 2000. In other examples, color cameras 2021, 2022 and a flash 2010 may be integrated into the front 2001 of device 2000 or provided on both the front and back sets of cameras. The color cameras 2021, 2022 and flash 2010 may be components of a camera module that produces color image data with IR texture correction that may be output to the display 2004 and / or processed into images or streaming video that may be communicated remotely from the device 2000, for example, via the antenna 2008.
[0113] Various embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (APIs), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. The decision whether an embodiment is implemented using hardware and / or software elements may vary according to any number of factors, such as desired computation rates, power levels, thermal tolerances, processing cycle budgets, input data rates, output data rates, memory resources, (data bus speeds, and other design or performance constraints.
[0114] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium that represent various logic within a processor that, when read by a machine, causes the machine to create logic that causes the machine to perform the techniques described herein. Such representations, known as IP cores, may be stored on tangible machine-readable media and provided to various customers or manufacturing facilities for incorporation into manufacturing machines that actually produce the logic or processor.
[0115] While certain features described herein have been described with reference to various embodiments, this description is not intended to be construed in a limiting sense. Thus, various modifications of the embodiments described herein, as well as other embodiments, that are apparent to those skilled in the art to which the disclosure pertains, are deemed to be within the spirit and scope of the disclosure.
[0116] In one or more first embodiments, a method for performing object association in multiview video includes receiving bounding box data for each of a plurality of video frames, each of the plurality of video frames corresponding to a particular view of a scene at the same time instance; generating a distance matrix based on the bounding box data, the distance matrix having distances between combinations of available bounding boxes between all pairs of video frames; determining a plurality of distance submatrices from the distance matrix based on a two-way minimum distance corresponding bounding box pair in the distance matrix; determining a corresponding submatrix separately for each of the distance submatrices based on optimizing an objective function with each corresponding submatrix and each distance submatrix; and providing an association list for the bounding box data of the video frames based on the corresponding submatrix.
[0117] In one or more second embodiments, further to the first embodiment, in some embodiments, a first bi-directional minimum distance corresponding bounding box pair of the bi-directional minimum distance corresponding bounding box pairs includes a correspondence between a first bounding box in the first video frame and a second bounding box in the second video frame, and the first bi-directional minimum distance corresponding bounding box pair is responsive to the second bounding box having a minimum distance from the first bounding box from all available distances from the first bounding box to all candidate bounding boxes in the second video frame and the first bounding box having a minimum distance from the second bounding box from all available distances from the second bounding box to all candidate bounding boxes in the first video frame.
[0118] In one or more third embodiments, further to the first or second embodiment, the step of determining the multiple sub-matrices includes transforming a distance matrix based on the two-way minimum distance corresponding bounding box pairs via row and column basic transformations to merge the two-way minimum distance corresponding bounding box pairs within the sub-matrices.
[0119] In one or more fourth embodiments, further to any of the first to third embodiments, the method further comprises generating a first set of tracklets using the same time instance as a starting time frame, generating one or more additional sets of tracklets using additional time instances subsequent to the same time instance as a starting time frame, and adjusting an initial association list for one of the time instances using the first set of tracklets and the additional set of tracklets.
[0120] In one or more fifth embodiments, further to any of the first to fourth embodiments, a first tracklet of the first set of tracklets is generated by projecting a position of a bounding box for a same time instance onto a second position of a bounding box for a second time instance subsequent to the same time instance, and adding an object position at the second time instance to the first tracklet in response to applying a combinatorial optimization algorithm and the object position at the second time instance being within a threshold distance of the object position at the same time instance.
[0121] In one or more sixth embodiments, further to any of the first to fifth embodiments, the method further comprises a step of fusing the first set of tracklets with one or more additional sets of tracklets to generate a resulting set of tracklets.
[0122] In one or more seventh embodiments, further to any of the first to sixth embodiments, the step of blending the first set of tracklets with the one or more additional sets of tracklets includes determining that a second tracklet in the second set of tracklets of the one or more additional sets of tracklets does not intersect with any tracklet in the first set of tracklets, and in response to the second tracklet having a length that exceeds a threshold, adding the second tracklet to the resulting set of tracklets.
[0123] In one or more eighth embodiments, further to any of the first to seventh embodiments, the step of blending the first set of tracklets with the one or more additional sets of tracklets comprises determining that a second tracklet in the second set of the one or more additional sets of tracklets does not intersect with any tracklet in the first set of tracklets, and merging the second tracklet with a third tracklet in the resulting set of tracklets in response to a sum of the lengths of the second and third tracklets being smaller than a buffer size corresponding to the same and additional time instances.
[0124] In one or more ninth embodiments, further to any of the first to eighth embodiments, providing the association list includes adjusting one or more objects in the association list based on a comparison of the positions of the objects in the association list after position tracking of the bounding box data.
[0125] In one or more tenth embodiments, further to any of the first to ninth embodiments, adjusting the one or more objects includes changing an association for a first object bounding box in the first video frame from a first object association to a second object association in response to a first distance of the first object bounding box to a center position of the first object association exceeding a second distance of the first object bounding box to a center position of the second object association.
[0126] In one or more eleventh embodiments, further to any of the first to tenth embodiments, adjusting the one or more objects includes discarding a first object bounding box in the first video frame in response to the first object bounding box being farther than a threshold distance from the first object association position.
[0127] In one or more twelfth embodiments, a device or system includes a memory and one or more processors for executing a method according to any one of the preceding embodiments.
[0128] In one or more thirteenth embodiments, at least one machine-readable medium includes a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform a method according to any one of the preceding embodiments.
[0129] In one or more of the fourteenth embodiments, an apparatus comprises means for carrying out the method according to any one of the preceding embodiments.
[0130] It will be recognized that the embodiments are not limited to the embodiments described hereinabove, but can be practiced with modification and alteration without departing from the scope of the appended claims. For example, the above-described embodiments may include specific combinations of features. However, the above-described embodiments are not limited in this respect, and in various implementations, the above-described embodiments may include implementing only a subset of such features, implementing such features in a different order, implementing such features in different combinations, and / or implementing additional features beyond those features explicitly recited. The scope of these embodiments should therefore be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. [Other possible items] [Item 1] 1. A system for performing object association in multi-view video, comprising: a memory for storing bounding box data for each of a plurality of video frames, each of the plurality of video frames corresponding to a particular view of a scene at a same time instance; one or more processors coupled to the memory, the one or more processors comprising: receiving bounding box data for each of a plurality of video frames, each of the plurality of video frames corresponding to a particular view of a scene at a same time instance; generating a distance matrix based on the bounding box data, the distance matrix comprising distances between combinations of available bounding boxes between all pairs of the video frames; determining a plurality of distance submatrices from the distance matrix based on two-way minimum distance corresponding bounding box pairs in the distance matrix; determining a corresponding submatrix separately for each of the distance submatrices based on optimizing an objective function having each corresponding submatrix and each distance submatrix; providing an association list for the bounding box data of the video frames based on the corresponding sub-matrices; and and one or more processors executing A system comprising: [Item 2] 2. The system of claim 1, wherein a first one of the bidirectional minimum distance correspondence bounding box pairs has a correspondence between a first bounding box in a first video frame and a second bounding box in a second video frame, the first bidirectional minimum distance correspondence bounding box pair being responsive to the second bounding box having a minimum distance from the first bounding box out of all available distances from the first bounding box to all candidate bounding boxes in the second video frame and the first bounding box having a minimum distance from the second bounding box out of all available distances from the second bounding box to all candidate bounding boxes in the first video frame. [Item 3] 2. The system of claim 1, wherein the one or more processors that perform determining the plurality of sub-matrices include the one or more processors that perform transforming the distance matrix based on the two-way minimum distance corresponding bounding box pairs via row and column basic transformations to merge the two-way minimum distance corresponding bounding box pairs within the sub-matrices. [Item 4] The one or more processors: generating a first set of tracklets using the same time instance as a starting time frame; generating one or more additional sets of tracklets using additional time instances subsequent to the same time instance as a starting time frame; adjusting an initial association list for one of the time instances using the first set of tracklets and the additional set of tracklets; 4. The system according to any one of items 1 to 3, [Item 5] A first tracklet of the first set of tracklets is processed by the one or more processors, projecting a position of the bounding box for the same time instance onto a second position of the bounding box for a second time instance that follows the same time instance; adding the object position at a second time instance to the first tracklet in response to applying a combinatorial optimization algorithm and the object position at the second time instance being within a threshold distance of the object position at the same time instance; 5. The system of claim 4, wherein the system is generated by the one or more processors executing the [Item 6] The one or more processors: aggregating the first set of tracklets with the one or more additional sets of tracklets to generate a resultant set of tracklets. 5. The system according to item 4, [Item 7] The one or more processors performing the fusing of the first set of tracklets and the one or more additional sets of tracklets may further comprise: determining that a second tracklet in a second set of tracklets of the one or more additional sets of tracklets does not intersect with any tracklet in the first set of tracklets; adding the second tracklet to the set of resulting tracklets in response to the second tracklet having a length exceeding a threshold; 7. The system of claim 6, further comprising the one or more processors executing [Item 8] The one or more processors performing the fusing of the first set of tracklets and the one or more additional sets of tracklets may further comprise: determining that a second tracklet in a second set of tracklets of the one or more additional sets of tracklets does not intersect with any tracklet in the first set of tracklets; merging the second tracklet with a third tracklet in the resulting set of tracklets, in response to the sum of the lengths of the second and third tracklets being less than a buffer size corresponding to the same and additional time instances; 7. The system of claim 6, further comprising the one or more processors executing [Item 9] 2. The system of claim 1, wherein the one or more processors that perform providing the association list include the one or more processors that perform adjusting one or more objects in the association list based on a comparison of positions of objects in the association list after position tracking of the bounding box data. [Item 10] 10. The system of claim 9, wherein the one or more processors that perform adjusting the one or more objects include the one or more processors that perform changing an association for a first object bounding box in a first video frame from a first object association to a second object association in response to a first distance of the first object bounding box to a center position of the first object association exceeding a second distance of the first object bounding box to a center position of the second object association. [Item 11] 10. The system of claim 9, wherein the one or more processors that perform adjusting the one or more objects include the one or more processors that perform discarding a first object bounding box in a first video frame in response to the first object bounding box being farther than a threshold distance from a first object association position. [Item 12] 1. A method for performing object association in multi-view video, comprising: receiving bounding box data for each of a plurality of video frames, each of the plurality of video frames corresponding to a particular view of a scene at a same time instance; generating a distance matrix based on the bounding box data, the distance matrix comprising distances between combinations of available bounding boxes between all pairs of the video frames; determining a plurality of distance submatrices from the distance matrix based on two-way minimum distance corresponding bounding box pairs in the distance matrix; determining a corresponding submatrix separately for each of the distance submatrices based on optimizing an objective function having each corresponding submatrix and each distance submatrix; providing an association list for the bounding box data of the video frames based on the corresponding sub-matrix; A method for providing the above. [Item 13] Item 13. The method of item 12, wherein the step of determining the plurality of sub-matrices includes transforming the distance matrix based on the two-way minimum distance corresponding bounding box pairs via row and column basic transformations to merge the two-way minimum distance corresponding bounding box pairs within the sub-matrices. [Item 14] generating a first set of tracklets using said same time instance as a starting time frame; generating one or more additional sets of tracklets using additional time instances subsequent to the same time instance as a starting time frame; adjusting an initial association list for one of the time instances using the first set of tracklets and the additional set of tracklets; 14. The method according to item 12 or 13, further comprising: [Item 15] amalgamating the first set of tracklets with the one or more additional sets of tracklets to generate a resultant set of tracklets. 15. The method of claim 14, further comprising: [Item 16] Item 13. The method of item 12, wherein providing the association list further comprises adjusting one or more objects in the association list based on a comparison of positions of objects in the association list after position tracking of the bounding box data. [Item 17] At least one machine-readable medium, which in response to being executed on a computing device, causes the computing device to: receiving bounding box data for each of a plurality of video frames, each of the plurality of video frames corresponding to a particular view of a scene at a same time instance; generating a distance matrix based on the bounding box data, the distance matrix comprising distances between combinations of available bounding boxes between all pairs of the video frames; determining a plurality of distance submatrices from the distance matrix based on two-way minimum distance corresponding bounding box pairs in the distance matrix; determining a corresponding submatrix separately for each of the distance submatrices based on optimizing an objective function having each corresponding submatrix and each distance submatrix; providing an association list for the bounding box data of the video frames based on the corresponding sub-matrix; At least one machine-readable medium comprising a plurality of instructions for performing object association in multi-view video. [Item 18] Item 18. The machine-readable medium of item 17, wherein determining the plurality of sub-matrices includes transforming the distance matrix based on the two-way minimum distance corresponding bounding box pairs via row and column basic transformations to merge the two-way minimum distance corresponding bounding box pairs within the sub-matrices. [Item 19] In response to being executed on a computing device, the computing device generating a first set of tracklets using said same time instance as a starting time frame; generating one or more additional sets of tracklets using additional time instances subsequent to the same time instance as a starting time frame; adjusting an initial association list for one of the time instances using the first set of tracklets and the additional set of tracklets; 19. The machine-readable medium of claim 17 or 18, further comprising a plurality of instructions for performing object association in multi-view video. [Item 20] The method comprises: amalgamating the first set of tracklets with the one or more additional sets of tracklets to generate a resultant set of tracklets. 20. The machine-readable medium of claim 19, further comprising: [Item 21] 20. The machine-readable medium of claim 17, wherein providing the association list further comprises adjusting one or more objects in the association list based on a comparison of positions of objects in the association list after position tracking of the bounding box data. [Item 22] 1. A system comprising: means for receiving bounding box data for each of a plurality of video frames, each of the plurality of video frames corresponding to a particular view of a scene at a same time instance; means for generating a distance matrix based on the bounding box data, the distance matrix comprising distances between combinations of available bounding boxes between all pairs of the video frames; means for determining a plurality of distance submatrices from the distance matrix based on two-way minimum distance corresponding bounding box pairs within the distance matrix; means for separately determining, for each of the distance submatrices, a corresponding submatrix based on optimizing an objective function having each corresponding submatrix and each distance submatrix; means for providing an association list for the bounding box data of the video frames based on the corresponding sub-matrix; A system comprising: [Item 23] 23. The system of claim 22, wherein the means for determining the plurality of sub-matrices comprises means for transforming the distance matrix based on the two-way minimum distance corresponding bounding box pairs via row and column basic transformations to merge the two-way minimum distance corresponding bounding box pairs within the sub-matrices. [Item 24] means for generating a first set of tracklets using said same time instance as a starting time frame; means for generating one or more additional sets of tracklets using additional time instances subsequent to the same time instance as a starting time frame; means for adjusting an initial association list for one of the time instances using the first set of tracklets and the additional set of tracklets; 24. The system according to item 22 or 23, further comprising: [Item 25] means for amalgamating the first set of tracklets with the one or more additional sets of tracklets to generate a resultant set of tracklets. 25. The system of claim 24, further comprising:
Claims
1. 1. A system for performing object association in multi-view video, comprising: a memory for storing bounding box data for each of a plurality of video frames, each of the plurality of video frames corresponding to a particular view of a scene at a same time instance; one or more processors coupled to the memory, the one or more processors comprising: generating a distance matrix based on the bounding box data, the distance matrix comprising distances between combinations of available bounding boxes between all pairs of the plurality of video frames; determining a plurality of distance submatrices from the distance matrix based on two-way minimum distance corresponding bounding box pairs in the distance matrix; determining a corresponding submatrix separately for each of the plurality of distance submatrices based on optimizing an objective function having each corresponding submatrix and each distance submatrix; providing an association list for the bounding box data of the plurality of video frames based on the corresponding sub-matrices; and and one or more processors executing A system comprising:
2. 2. The system of claim 1, wherein a first one of the bi-directional minimum distance correspondence bounding box pairs has a correspondence between a first bounding box in a first video frame and a second bounding box in a second video frame, the first bi-directional minimum distance correspondence bounding box pair being responsive to the second bounding box having a minimum distance from the first bounding box out of all available distances from the first bounding box to all candidate bounding boxes in the second video frame and the first bounding box having a minimum distance from the second bounding box out of all available distances from the second bounding box to all candidate bounding boxes in the first video frame.
3. 3. The system of claim 1 or 2, wherein the one or more processors that perform determining the plurality of distance submatrices include the one or more processors that perform transforming the distance matrices based on the two-way minimum distance corresponding bounding box pairs via row and column basic transformations to merge the two-way minimum distance corresponding bounding box pairs within the plurality of distance submatrices.
4. The one or more processors: generating a first set of tracklets using said same time instance as a starting time frame; generating one or more additional sets of tracklets using additional time instances subsequent to the same time instance as a starting time frame; adjusting an initial association list for one of the time instances using the first set of tracklets and the additional set of tracklets; The system according to claim 1 , further comprising:
5. A first tracklet of the first set of tracklets is processed by the one or more processors, projecting a position of the bounding box for the same time instance onto a second position of the bounding box for a second time instance that follows the same time instance; adding the object position at a second time instance to the first tracklet in response to applying a combinatorial optimization algorithm and the object position at the second time instance being within a threshold distance of the object position at the same time instance; The system of claim 4 , wherein the system is generated by the one or more processors executing:
6. The one or more processors: aggregating the first set of tracklets with the one or more additional sets of tracklets to generate a resultant set of tracklets. The system of claim 4 , further comprising:
7. The one or more processors performing the fusing of the first set of tracklets and the one or more additional sets of tracklets may further comprise: determining that a second tracklet in a second set of tracklets of the one or more additional sets of tracklets does not intersect with any tracklet in the first set of tracklets; adding the second tracklet to the set of resulting tracklets in response to the second tracklet having a length exceeding a threshold.
7. The system of claim 6, further comprising the one or more processors executing:
8. The one or more processors performing the fusing of the first set of tracklets and the one or more additional sets of tracklets may further comprise: determining that a second tracklet in a second set of tracklets of the one or more additional sets of tracklets does not intersect with any tracklet in the first set of tracklets; merging the second tracklet with a third tracklet in the resulting set of tracklets, in response to a sum of lengths of the second tracklet and the third tracklet being less than a buffer size corresponding to the same time instance and the additional time instance; 7. The system of claim 6, further comprising the one or more processors executing:
9. 9. The system of claim 1 , wherein the one or more processors that perform providing the association list comprise the one or more processors that perform adjusting one or more objects in the association list based on a comparison of positions of objects in the association list after position tracking of the bounding box data.
10. 10. The system of claim 9, wherein the one or more processors that perform adjusting the one or more objects further comprise the one or more processors that perform changing an association for a first object bounding box in a first video frame from a first object association to a second object association in response to a first distance of the first object bounding box to a center position of the first object association exceeding a second distance of the first object bounding box to a center position of the second object association.
11. 10. The system of claim 9, wherein the one or more processors that perform adjusting the one or more objects include the one or more processors that perform discarding a first object bounding box in a first video frame in response to the first object bounding box being farther than a threshold distance from a first object association position.
12. 1. A method for performing object association in multi-view video, comprising: receiving bounding box data for each of a plurality of video frames, each of the plurality of video frames corresponding to a particular view of a scene at a same time instance; generating a distance matrix based on the bounding box data, the distance matrix comprising distances between combinations of available bounding boxes between all pairs of the plurality of video frames; determining a plurality of distance submatrices from the distance matrix based on two-way minimum distance corresponding bounding box pairs in the distance matrix; determining a corresponding submatrix separately for each of the plurality of distance submatrices based on optimizing an objective function having each corresponding submatrix and each distance submatrix; providing an association list for the bounding box data of the plurality of video frames based on the corresponding sub-matrix; A method for providing the above.
13. 13. The method of claim 12, wherein a first one of the bi-directional minimum distance correspondence bounding box pairs has a correspondence between a first bounding box in a first video frame and a second bounding box in a second video frame, the first bi-directional minimum distance correspondence bounding box pair being responsive to the second bounding box having a minimum distance from the first bounding box out of all available distances from the first bounding box to all candidate bounding boxes in the second video frame, and the first bounding box having a minimum distance from the second bounding box out of all available distances from the second bounding box to all candidate bounding boxes in the first video frame.
14. 14. The method of claim 12 or 13, wherein determining the plurality of distance submatrices comprises transforming the distance matrices based on the two-way minimum distance corresponding bounding box pairs via row and column elementary transformations to merge the two-way minimum distance corresponding bounding box pairs within the plurality of distance submatrices.
15. generating a first set of tracklets using said same time instance as a starting time frame; generating one or more additional sets of tracklets using additional time instances subsequent to the same time instance as a starting time frame; adjusting an initial association list for one of the time instances using the first set of tracklets and the additional set of tracklets; 15. The method of claim 12, further comprising:
16. A first tracklet of the first set of tracklets comprises: projecting a position of the bounding box for the same time instance onto a second position of the bounding box for a second time instance that follows the same time instance; adding the object position at a second time instance to the first tracklet in response to applying a combinatorial optimization algorithm and the object position at the second time instance being within a threshold distance of the object position at the same time instance; The method of claim 15 , wherein the compound is produced by
17. amalgamating the first set of tracklets with the one or more additional sets of tracklets to generate a resultant set of tracklets. The method of claim 15 further comprising:
18. The step of fusing the first set of tracklets with the one or more additional sets of tracklets comprises: determining that a second tracklet in a second set of tracklets of the one or more additional sets of tracklets does not intersect with any tracklet in the first set of tracklets; adding the second tracklet to the set of resulting tracklets in response to the second tracklet having a length exceeding a threshold; 20. The method of claim 17, comprising:
19. The step of fusing the first set of tracklets with the one or more additional sets of tracklets comprises: determining that a second tracklet in a second set of tracklets of the one or more additional sets of tracklets does not intersect with any tracklet in the first set of tracklets; merging the second tracklet with a third tracklet in the resulting set of tracklets, in response to a sum of lengths of the second tracklet and the third tracklet being less than a buffer size corresponding to the same time instance and the additional time instance; 20. The method of claim 17, comprising:
20. 20. The method of claim 12, wherein providing the association list further comprises adjusting one or more objects in the association list based on a comparison of positions of objects in the association list after position tracking of the bounding box data.
21. 21. The method of claim 20, wherein adjusting the one or more objects comprises changing an association for a first object bounding box in a first video frame from a first object association to a second object association in response to a first distance of the first object bounding box to a center position of the first object association exceeding a second distance of the first object bounding box to a center position of the second object association.
22. 21. The method of claim 20, wherein adjusting the one or more objects comprises discarding a first object bounding box in the first video frame in response to the first object bounding box being farther than a threshold distance from a first object association position.
23. A computer program causing a processor to carry out the method of any one of claims 12 to 22.
24. 1. An apparatus comprising: Means for carrying out the method according to any one of claims 12 to 22. An apparatus comprising:
Citation Information
Patent Citations
Information processing device, three-dimensional position estimation method, computer program, and storage medium
JP2019121019A
Object tracking apparatus, method and program
JP2019174868A
Multi-to-multi tracking in video analytics
US20180046865A1
Cited By
Method and device for target tracking and storage medium
US12700109B2
Method and device for target tracking and storage medium
US20240070881A1
