Method and apparatus for tracking movement of object

Abstract

The present disclosure provides a method and an apparatus for tracking a movement of an object, the method comprising: extracting, by a processor, a first plurality of features for each of a first plurality of objects being tracked at a first time instance based on sensor data obtained up to the first time instance, each feature of the first plurality of features being one of a plurality of feature types; performing, by the processor, object detection on the sensor data obtained at a second time instance that is after the first time instance, and extracting a second plurality of features for each of a second plurality of objects detected from the sensor data obtained at the second time instance, each feature of the second plurality of features being one of the plurality of feature types.

Description

METHOD AND APPARATUS FOR TRACKING MOVEMENT OF OBJECT

[0001] The present disclosure relates broadly, but not exclusively, to a method and an apparatus for tracking a movement of an object.

[0002] Object tracking relates to detection of an object in, for example, an image or a video to track its movement in space or across different camera angles. It has been widely used in different video analysis applications (e.g., tracking suspicious person(s) for crime investigations, looking for lost items, identify a trajectory of a moving object, etc). Video analysis refers to data analysis based on output results of object tracking models (e.g., based on object identifiers (IDs), locations, trajectories, and other similar factors).

[0003] It is important to achieve a high level of accuracy so that an object is correctly identified and tracked. However, such accuracy typically can only be improved at the cost of performance, e.g., taking a longer time, and / or using more computing and power resources to detect and / or track an object.

[0004] Herein disclosed are embodiments of a method and apparatus for tracking a movement of an object that addresses one or more of the above problems.

[0005] Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

[0006] In the present disclosure, one or more apparatus and / or one or more associated methods are disclosed that aim to at least partially contribute to implementing one or more of the above mentioned possible enhancements.

[0007] In a first aspect, the present disclosure provides a method for tracking a movement of an object, comprising: extracting, by a processor, a first plurality of features for each of a first plurality of objects being tracked at a first time instance based on sensor data obtained up to the first time instance, each feature of the first plurality of features being one of a plurality of feature types; performing, by the processor, object detection on the sensor data obtained at a second time instance that is after the first time instance, and extracting a second plurality of features for each of a second plurality of objects detected from the sensor data obtained at the second time instance, each feature of the second plurality of features being one of the plurality of feature types; and selecting, by the processor, at least one feature type from the plurality of feature types based on at least one of: a distribution of each feature type of the first plurality of features among the first plurality of objects, a distribution of each feature type of the second plurality of features among the second plurality of objects, a pattern of each feature type of the first or the second plurality of features, and a feedback from an application utilizing the first and second plurality of features for tracking the object; and tracking, by the processor, the movement of the object from the first time instance to the second time instance by associating at least one of the first plurality of objects with at least one of the second plurality of objects based on a difference or a similarity between at least one feature of the first plurality of features and at least one feature of the second plurality of features that are of the at least one selected feature type.

[0008] In a second aspect, the present disclosure provides an apparatus for tracking a movement of an object, comprising: at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: extract a first plurality of features for each of a first plurality of objects being tracked at a first time instance based on sensor data obtained up to the first time instance, each feature of the first plurality of features being one of a plurality of feature types; perform object detection on the sensor data obtained at a second time instance that is after the first time instance, and extract a second plurality of features for each of a second plurality of objects detected from the sensor data obtained at the second time instance, each feature of the second plurality of features being one of the plurality of feature types; select at least one feature type from the plurality of feature types based on at least one of: a distribution of each feature type of the first plurality of features among the first plurality of objects, a distribution of each feature type of the second plurality of features among the second plurality of objects, a pattern of each feature type of the first or the second plurality of features, and a feedback from an application utilizing the first and second plurality of features for tracking the object; and track the movement of the object from the first time instance to the second time instance by associating at least one of the first plurality of objects with at least one of the second plurality of objects based on a difference or a similarity between at least one feature of the first plurality of features and at least one feature of the second plurality of features that are of the at least one selected feature type.

[0009] In a third aspect, the present disclosure provides a system for tracking a movement of an object, comprising the apparatus of the second aspect and one or more sensor data capturing apparatuses configured to capture one or more sensor data.

[0010] Additional benefits and advantages of the disclosed embodiments will become apparent from the specification and drawings. The benefits and / or advantages may be individually obtained by the various embodiments and features of the specification and drawings, which need not all be provided in order to obtain one or more of such benefits and / or advantages.

[0011] According to the present disclosure, it is possible to provide a method and an apparatus for tracking a movement of an object.

[0012] The accompanying Figs., where like reference numerals refer to identical or functionally similar elements throughout the separate views and which together with the detailed description below are incorporated in and form part of the specification, serve to illustrate various embodiments and to explain various principles and advantages in accordance with a present embodiment, by way of non-limiting example only.

[0013] Embodiments of the disclosure will be better understood and readily apparent to one of ordinary skill in the art from the following written description, by way of example only, and in conjunction with the drawings, in which:

[0014] Fig. 1 shows an illustration of an object tracking and video analysis process on a video according to an example;Fig. 2 shows an illustration of a relationship between accuracy and processing time for object tracking and video analysis according to an example;Fig. 3 shows another illustration of a relationship between accuracy and processing time for object tracking and video analysis according to an example;Fig. 4A shows an exemplary illustration for assigning a correct identifier for each object in an image;Fig. 4B shows an exemplary illustration for assigning an incorrect identifier for one or more objects in an image;Fig. 5 shows an exemplary illustration for how a feature may be extracted according to an embodiment of the present disclosure;Fig. 6 shows an exemplary process for tracking hand movements based on an orientation of each hand according to an embodiment of the present disclosure;Fig. 7 shows exemplary illustrations for when orientation of an object in an image may be utilized according to an embodiment of the present disclosure;Fig. 8 shows a flow chart illustrating a method for tracking a movement of an object according to various embodiments of the present disclosure;Fig. 9 shows an exemplary process for how one or more features may be extracted for hand movement tracking according to an embodiment of the present disclosure;Fig. 10 shows an exemplary illustration of a hand depicted in an image with and without orientation according to an embodiment of the present disclosure;Fig. 11 shows an exemplary illustration of how orientation can be determined for an object in an image according to an embodiment of the present disclosure;Fig. 12 shows an exemplary illustration of how an orientation threshold can be determined for tracking a movement of an object according to an embodiment of the present disclosure;Fig. 13 shows an exemplary illustration of how movement of a pair of hands may be tracked based on orientation according to an embodiment of the present disclosure;Fig. 14 shows an exemplary illustration of how a radius of movement of an object may be determined according to an embodiment of the present disclosure;Fig. 15A shows an exemplary illustration of how a curvature for tracking an object may be determined according to an embodiment of the present disclosure;Fig. 15B shows another exemplary illustration of how a curvature for tracking an object may be determined according to an embodiment of the present disclosure;Fig. 16 shows an exemplary illustration of how curvature may be used for tracking movement of a pair of hands according to an embodiment of the present disclosure;Fig. 17 shows an exemplary illustration of how movement of a pair of hands may be tracked based on orientation according to an embodiment of the present disclosure;Fig. 18 shows an exemplary illustration depicting detection of a pair of hands including orientation according to an embodiment of the present disclosure;Fig. 19 shows a flow chart for feature extraction including orientation for tracking movement of a pair of hands according to an embodiment of the present disclosure;Fig. 20 shows a flow chart for utilizing an orientation threshold for tracking movement of an object according to an embodiment of the present disclosure;Fig. 21 shows a flow chart for feature extraction including curvature for tracking movement of a pair of hands according to an embodiment of the present disclosure;Fig. 22 shows a flow chart for determining, based on a threshold, whether to utilize curvature for tracking movement of an object according to an embodiment of the present disclosure;Fig. 23A shows an exemplary result of tracking movement of a pair of hands without orientation;Fig. 23B shows an exemplary result of tracking movement of a pair of hands with orientation according to an embodiment of the present disclosure;Fig. 24A shows another exemplary illustration of how movement of a pair of hands may be tracked without orientation according to an embodiment of the present disclosure;Fig. 24B shows another exemplary illustration of how movement of a pair of hands may be tracked with orientation according to an embodiment of the present disclosure;Fig. 25 shows an exemplary illustration of how movement of a car may be tracked with orientation according to an embodiment of the present disclosure;Fig. 26 shows a schematic diagram of an exemplary computing device suitable for use to execute the method in Figs. 8, 19, 20, 21 and 22; andFig. 27 shows an exemplary illustration of how orientation change rate may be used for tracking movement of a pair of hands according to an embodiment of the present disclosure.

[0015] Embodiments of the present disclosure will be described, by way of example only, with reference to the drawings. Like reference numbers and characters in the drawings refer to like elements or equivalents.

[0016] Some portions of the description which follows are explicitly or implicitly presented in terms of algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means used by those skilled in the data processing arts to convey most effectively the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities, such as electrical, magnetic or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

[0017] Unless specifically stated otherwise, and as apparent from the following, it will be appreciated that throughout the present specification, discussions utilizing terms such as "detecting", "estimating", "comparing", "receiving", "calculating", "determining", "updating", "generating", "initializing", "outputting", "receiving", "retrieving", "identifying", "dispersing", "authenticating" or the like, refer to the action and processes of a computer system, or similar electronic device, that manipulates and transforms data represented as physical quantities within the computer system into other data similarly represented as physical quantities within the computer system or other information storage, transmission or display devices.

[0018] The present specification also discloses apparatus for performing the operations of the methods. Such apparatus may be specially constructed for the required purposes, or may comprise a computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various machines may be used with programs in accordance with the teachings herein. Alternatively, the construction of more specialized apparatus to perform the required method steps may be appropriate. The structure of a computer will appear from the description below.

[0019] In addition, the present specification also implicitly discloses a computer program, in that it would be apparent to the person skilled in the art that the individual steps of the method described herein may be put into effect by computer code. The computer program is not intended to be limited to any particular programming language and implementation thereof. It will be appreciated that a variety of programming languages and coding thereof may be used to implement the teachings of the disclosure contained herein. Moreover, the computer program is not intended to be limited to any particular control flow. There are many other variants of the computer program, which can use different control flows without departing from the spirit or scope of the disclosure.

[0020] Furthermore, one or more of the steps of the computer program may be performed in parallel rather than sequentially. Such a computer program may be stored on any computer readable medium. The computer readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a computer. The computer readable medium may also include a hard-wired medium such as exemplified in the Internet system, or wireless medium such as exemplified in the GSM mobile telephone system. The computer program when loaded and executed on such a computer effectively results in an apparatus that implements the steps of the preferred method.

[0021] < Exemplary embodiments > 　　Various embodiments of the present disclosure relate to a method and an apparatus for tracking a movement of an object.

[0022] Sensor data refers to any data that is obtained by way of a sensor, for example data (e.g., images, videos, thermal imaging, location coordinates, and other similar data) obtained from a light sensor, thermal sensor, global positioning system (GPS) or other similar sensors. An image refers to a visual representation such as a photograph, a video frame of a video, or other similar media that is captured by an image or video capturing device such as a camera or other similar device. The image may have an axis that is parallel to a height of the image and an axis that is parallel to a length of the image. The image may comprise a plurality of pixels.

[0023] An object refers to something that is depicted in sensor data such as an image. In a particular example, an image may be one of a location (e.g., a work space, desk, other similar location) showing one or more hands, and each hand may be considered as an object. In another broader example, the image may be that of a location, map, a road, a water body, or other similar visual representation in which one or more things (e.g., a living thing (such as a person, an animal, a bird, etc) or a part of a living thing (e.g., hand, feet, appendage, etc), a wheeled vehicle, a ship or boat, an aircraft, or other similar things) may be present, and the one or more things may be considered as one or more objects. It will be appreciated that the object is not limited to the above examples, and other examples of objects are possible. An object may be detected and one or more features of the object may be extracted from an image using a feature extraction unit (e.g., using YOLO object detection model with oriented bounding box prediction unit, or other similar algorithm for feature extraction). The one or more features (e.g., bounding box that surrounds the object, a location or position of the object relative to the image, a confidence level that the object is detected or identified correctly, an orientation, curvature, and other similar features) may be extracted for the object. Each feature can be categorised under one of a plurality of feature types (e.g., bounding box that surrounds the object, a location or position of the object relative to the image, a confidence level that the object is detected or identified correctly, an orientation, curvature, and other similar feature types) which may be selected for tracking movement of an object. Selecting a feature type may be based on a spatial distribution that is determined by comparing a position of each of a plurality of objects (e.g., detected in the sensor data) with one another. The spatial distribution of an object may be determined by the bounding box that surrounds the object, a location or position of the object relative to the image, and size of the bounding box. In object detection domain, a bounding box for an object is described using the x and y coordinates (e.g., based on pixels as well as a X-axis and a Y-axis of an image in which the object is captured) of a centre point, a width (e.g., a length of the bounding box that is parallel to the X-axis of the image, or other similar length) and a height (e.g., a length of the bounding box that is parallel to the Y-axis of the image, or other similar length) of the bounding box. It will be appreciated that a plurality of objects may be detected and tracked in each image or time instance.

[0024] A movement of the object (e.g., in which an object changes its position from a first position (e.g., in a first image) at a first time instance to a second position (e.g., in a second image) at a second time instance) may be tracked by determining a position of the object in each of one or more images or video frames in a video. Tracking a movement of an object typically involves determining whether an object in a first image (e.g., a photograph, a video frame, or other similar media of a location at a first time instance in which the object is present) is same as another object in a second image (e.g., a photograph, a video frame, or other similar media of the location at a second time instance that occur after the first time instance). An identifier may be assigned to each of these objects to indicate a sequence of movement and / or to indicate whether they are the same object (e.g., if the object in the first image is determined to be same as the object in the second image, movement of the object from the first time instance to the second time instance can thus be tracked). It will be appreciated that the second time instance may occur at a time duration (e.g., in nanoseconds, milliseconds, or other similar time duration) after the first time instance.

[0025] Further, it is possible to improve the tracking by determining and utilizing an orientation of the object in each of the one or more images or video frames. For example, the orientation of the object may be defined by an angle between a first axis associated with the sensor data (e.g., an axis that is parallel or perpendicular to a length of the first image, or other similar axis) and a second axis defined by a shape of the object (e.g., an axis that is parallel or perpendicular to a length of an object in a first image, or other similar axis), which may be calculated for the first and second objects and compared with each other to determine whether these two objects are the same object. The determination may be made by comparing a difference between the two angles with a threshold value. It will be appreciated that an angle may be determined for each of one or more objects in each image, and each of the determined angles may be compared with one another to determine and track each object starting from a first image to another second or more images. It will be appreciated that orientation determination is not limited to the examples shown above, and other alternatives or variations for determining orientation of an object are also possible.

[0026] In addition, a rate of change in orientation (also referred to herein as orientation change rate or rate of change in the angle) can also be used for improving the tracking of an object in each of the one or more images or video frames. For example, a first orientation of a first object in a first image and a second orientation of a second object in a second image may be determined, and a first rate of change in orientation of the object from the first orientation to the second orientation may be calculated based on a time duration between the first image and the second image. In the same manner, a third orientation of an object in a third image may be determined, and a second rate of change in orientation of the object from the second orientation to the third orientation may be calculated. An object is associated and tracked by comparing a difference between the first rate of change in orientation and the second rate of change with a certain threshold.

[0027] Furthermore, a rate of change in position (referred to herein as a curvature, or rate / degree of change in trajectory of an object) may be determined and utilized as a feature to further improve tracking of an object in at least three or more images or video frames (e.g., object positions at three or more time instances). For example, a first curvature may be calculated for the first object based on positions of the first object at two time instances occurring before the first time instance and a position of the first object at the first time instance, and a second curvature for the second object may be calculated based on positions of the first object at the first time instance and at a time instance occurring before the first time instance, and a position of the second object at the second time instance. The determination may be made by comparing a difference between the two curvatures with a threshold value. It will be appreciated that a curvature may be determined for each of one or more objects in each image, and each of the determined angles may be compared with one another to determine and track each object starting from a first image to another second or more images.

[0028] Referring to illustration 100 of Fig. 1, based on video footage 102 of a location captured by a video camera 104, one or more objects (e.g., a plurality of persons in the video footage 102) mat be detected and their movements may be tracked in an object tracking process 106. A trajectory (e.g., a line of movement that is determined based on a change in a position of an object in one or more images over a period of time) and object identifier for each of the plurality of persons detected in the video footage 102 may be determined as output 108, and the output 108 may then be used for video analysis applications 110.

[0029] It is an objective in the present disclosure to improve object tracking accuracy within a processing time requirement on capacity-limited hardware. Fig. 2 shows an illustration 200 of a relationship between accuracy (represented by axis 202) and processing time (represented by axis 204) for object tracking and video analysis according to an example. There is a processing time requirement 206 in which all processing required for object tracking should be completed. Thus, shaded portion 208 represents a desired balance to be achieved between accuracy and processing time (e.g., high accuracy of object tracking within the processing time requirement 206). Further referring to illustration 300 of Fig. 3, for a video footage with a video length of 10 minutes (see reference 302), a first case of light-weight processing (see reference 304) may require 9.5 minutes of processing time for detecting an object in the video and tracking movement of the object in the video, but achieves a low accuracy (e.g., 60% accuracy). In a second case of heavy-weight processing (see reference 306), 12 minutes of processing time is used. Although accuracy is high (e.g., 90% accuracy), the processing time exceeded the video length of 10 minutes and thus the object tracking results will be too slow for in particular real-time video analysis.

[0030] Video analysis application accuracy is dependent on object tracking accuracy. Due to limitations in training data-set variety and deep learning model(s), the problems of missing and false object detections may happen in different practical application scenarios, which result in the object tracking accuracy degradation. In time-critical object tracking applications, there is strict latency requirement of running the object tracking models (e.g., less than 30 milli-seconds per frame for processing a 30 frames per second video stream). Further, hardware constraints (e.g., capacity-limited platforms) limits the feasibility of deploying large artificial intelligence (AI) models to achieve better accuracy.

[0031] In a case where object detection and tracking involves detection and tracking of movement of one or more hands (e.g., based on images or a video of one or more hands, for example for monitoring hand movement at a factory, a desk, or other similar environment), assignment of an identifier (ID) to each hand is an important part of hand tracking accuracy. The ID assignment accuracy in a single image or video frame may be defined as a percentage of a number of hands with correct IDs over a total number of hands being monitored in the single image or video frame. Referring to Figs. 4A and 4B, illustration 400 of Fig. 4A depicts ground truth data (e.g., in which each hand 402, 404, 406 and 408 in a video frame are assigned with a correct identifier from left to right of 0, 1, 2 and 3 respectively). On the other hand, illustration 410 of Fig. 4B shows one failure case in ID assignment in which hands 406 and 408 are assigned with wrong IDs (e.g., 3 and 2 respectively).

[0032] The present disclosure provides a solution to improve the object tracking accuracy within the processing time requirement on capacity-limited hardware. For this reason, tracking accuracy is improved by dynamically combining a plurality of features that can be extracted with a low computational load, without using heavy-weight processing.

[0033] With reference to an exemplary functional block diagram 500 of Fig. 5, raw data (e.g., an image, a video, or other similar media) may be input into a feature extraction unit 502 in which one or more features (e.g., attributes of an object (in the image, video or other similar media) to be detected and / or tracked) that may be useful for a target application (e.g., video analysis application) may be extracted from the raw data. The extracted feature(s) may be input into a dynamic feature selection unit 504 in which one or more feature types may be selected based on, for example, application feedback and data distribution or pattern. Application feedback is, for example, object tracking accuracy, and if the feedback is positive, then the feature type is selected. On the other hand, data distribution is a distribution of features among objects being tracked (e.g., at a first or earlier time instance) or among objects that are newly detected (e.g., at a second or later time instance). If values of a feature are distributed so that objects can easily be differentiated, the feature is selected. Regarding pattern, if a pattern is similar to a reference pattern (for example, the distribution may follow a Gaussian / Poisson distribution), the feature is selected. These selected feature types are then input into application unit 506 in which application results based on dynamically selected features of the selected feature types are output or displayed. Further, a feedback loop 508 may enable the application unit 506 to send feedback (e.g., relating to the one or more selected features, or other similar feedback) to the dynamic feature selection unit 504.

[0034] Fig. 6 shows an exemplary process 600 for tracking hand movements based on an orientation of each hand according to an embodiment of the present disclosure. As shown in process 600, a video depicting movement of a plurality of hands may be input into a hand detection module 602 in which the plurality of hands are detected from the video. The detected plurality of hands are used as input for a hand tracking module 604 for tracking movement of each of the plurality of hands in the video. The hand tracking accuracy problem is solved by dynamically introducing an object orientation feature 606 into the hand detection module 602 and hand tracking module 604. In hand tracking algorithms, hand features (e.g., position, size, etc of each hand in an image, a video frame, a video, etc) are used to determine if a hand in a current video frame is same as a hand in a previous video frame. After introducing the new orientation feature dynamically, the accuracy on determining the same hand is expected to increase.

[0035] Orientation of an object detected from sensor data (e.g., detected in an image) refers to an angle between a first axis associated with the sensor data (e.g., an axis that is parallel or perpendicular to a length of the first image, or other similar axis) and a second axis defined by a shape of the object (e.g., an axis that is parallel or perpendicular to a length of an object in a first image, or other similar axis). In the present example, orientation of a hand in an image (e.g., video frame, video or other similar media) refers to an angle between a second axis that is parallel to a length of the hand in the image and a first axis that is parallel to a length of the image. In many application scenarios, orientation can be used to identify left and right hands in addition to tracking hands. For example, referring to illustrations 702 and 704 of Fig. 7 in which orientation of a left hand 710 and a right hand 712 are different from each other, it is possible to differentiate between the left and right hand and thus track each hand accordingly with, for example, a left and / or right hand label (e.g., a label, identifier, or other similar way of indicating right hand or left hand). In some scenarios, such as shown in illustration 706 in which orientation of the left hand 710 and right hand 712 are the same, or in illustration 708 in which orientation of the left hand 710 and right hand 712 cannot be determined, other features (e.g., position, size, curvature, etc.) may be utilized to detect and track hand movements. Thus, one or more features utilized for tracking may be dynamically selected based on a distribution of feature values to improve tracking accuracy. Since orientation calculation is a light-weight operation compared with directly identifying left / right hand, the usage of orientation for hand tracking advantageously improves accuracy without increasing computational resource consumption.

[0036] Fig. 8 shows a flow chart 800 illustrating a method for tracking a movement of an object according to various embodiments of the present disclosure. At step 802, a first plurality of features is extracted for each of a first plurality of objects being tracked at a first time instance based on sensor data obtained up to the first time instance, each feature of the first plurality of features being one of a plurality of feature types. At step 804, object detection is performed on the sensor data obtained at a second time instance that is after the first time instance, and extracting a second plurality of features for each of a second plurality of objects detected from the sensor data obtained at the second time instance, each feature of the second plurality of features being one of the plurality of feature types. At step 806, at least one feature type is selected from the plurality of feature types based on at least one of: a distribution of each feature type of the first plurality of features among the first plurality of objects, a distribution of each feature type of the second plurality of features among the second plurality of objects, a pattern of each feature type of the first or the second plurality of features, and a feedback from an application utilizing the first and second plurality of features for tracking the object. At step 808, the movement of the object is tracked from the first time instance to the second time instance by associating at least one of the first plurality of objects with at least one of the second plurality of objects based on a difference or a similarity between at least one feature of the first plurality of features and at least one feature of the second plurality of features that are of the at least one selected feature type.

[0037] In an implementation, tracking the movement of the object may further comprise determining whether one of the second plurality of objects is same as one of the first plurality of objects based on the difference or the similarity, and assigning an identifier to each of the one of the second plurality of objects and the one of the first plurality of objects based on the determination.

[0038] In an implementation, wherein the plurality of feature types of the first and the second plurality of features may include at least one of an orientation, an orientation change rate, and a curvature of the first plurality of objects and the second plurality of objects respectively, and selecting the at least one feature type may comprise evaluating whether the distribution of the at least one feature type of the first or the second plurality of features is within a predetermined range, whether a pattern of values of the first or second plurality of features is similar to a reference pattern, or whether a feedback from the application is positive.

[0039] In an implementation, wherein the orientation may be defined by an angle between a first axis associated with the sensor data and a second axis defined by a shape of the object, and extracting the first plurality of features may comprise calculating the angle for each of the first plurality of objects, and extracting the second plurality of features may comprise calculating the angle for each of the second plurality of objects.

[0040] In an implementation, wherein the orientation change rate may be defined by a rate of change in the angle, and extracting the first plurality of features may comprise calculating a difference between the angles for each of the first plurality of objects at the first time instance and a time instance occurring before the first time instance, and extracting the second plurality of features may comprise calculating a difference between the angle for each of the first plurality of objects and the angle for each of the second plurality of objects.

[0041] In an implementation, wherein the curvature may be defined by a rate of change of position of the object, and extracting the first plurality of features may comprise calculating a rate of change of position for each of the first plurality of objects based on positions of each of the first plurality of objects at two time instances occurring before the first time instance and a position of each of the first plurality of objects at the first time instance, and extracting the second plurality of features may comprise calculating a rate of change of position for each of the second plurality of objects based on positions of each of the first plurality of objects at the first time instance and at a time instance occurring before the first time instance, and a position of each of the second plurality of objects at the second time instance.

[0042] In an implementation, the method may further comprise selecting the at least one feature type of the first and the second plurality of features if the difference exceeds a threshold for the at least one feature type.

[0043] In an implementation, the method may further comprise selecting the at least one feature type based on a spatial distribution determined by comparing a position of each object of the first and second plurality of objects with one another.

[0044] In an implementation, the method may further comprise determining the threshold based on a monotonically non-increasing function of threshold value over a spatial distance between positions of each pair of objects of the first plurality of objects or a spatial distance between positions of each pair of objects of the second plurality of objects.

[0045] Fig. 9 shows an exemplary process 900 for how one or more features may be extracted for hand movement tracking according to an embodiment of the present disclosure. Raw data comprising, for example, a video depicting movement of one or more hands may be input into a hand feature extraction unit 902 (e.g., using YOLO object detection model with oriented bounding box prediction unit, or other similar algorithm for feature extraction) in which one or more features (e.g., bounding box, location or position, confidence, orientation, curvature, orientation change rate, and other similar features) may be extracted for the one or more hands in the video. These features may be used in a decision making unit 904 to determine which of the one or more features should be utilized based on feature distribution or application feedback for detecting and tracking the one or more hands in the video, and also used in a state estimate unit 906 (e.g., a Kalman filter or other similar algorithm) to estimate, for example, a future position of each of the one or more hands in the video (e.g., predicting a future position of a hand in a next video frame based on a position of the hand in a current video frame). The selected features by the decision making unit 904 and the estimates from the state estimation unit 906 may be utilized as input in a matching unit 908 (e.g., Deep SORT or other similar algorithm) to, for example, assign an identifier to each of the one or more hands in each video frame of the video, tracking a movement of each of the one or more hands in the video, and other similar functions. These tracking results may then be input into an application unit 910 which will utilize the information depending on application purposes.

[0046] Referring to Fig. 10, illustration 1000 shows a hand 1002 (e.g., a hand shown in an image) in which a bounding box 1004 without orientation is generated. For example, the bounding box 1004 may have a width of 135 pixels, a height of 95 pixels, a box centre location having coordinates (55 pixels, 50 pixels), and a confidence level (e.g., a measure of confidence that the hand is detected correctly and accurately) of 0.73. Further, illustration 1006 shows the same hand 1002 in which a bounding box 1008 with orientation is generated. For example, the bounding box 1008 may have a width of 125 pixels, a height of 80 pixels, a box centre location having coordinates (50 pixels, 50 pixels), an orientation angle of 45°, and an improved confidence level of 0.82. The orientation angle (e.g., angle 1106 in illustration 1100 of Fig. 11) is the angle between an axis that is parallel to a length of the hand 1002 (e.g., axis 1012) and an axis that is parallel to a length of the image (e.g., axis 1010). Referring to Fig. 11, an image 1100 shows bounding box 1104 which is a variation of the bounding box 1004. Bounding box 1104 has a centre point 1102 which may be referred to as a position of an object associated with the bounding box 1104. For example, a centre point 1014 of the bounding box 1008 may correspond to a position of the hand 1002. The position 1102 may be represented in terms of pixel coordinates relative to pixel count of an X-axis (e.g., an axis parallel to a length 1108 of the image 1100) and a Y-axis (e.g., an axis parallel to a height 1110 of the image 1100) of the image 1100, for example in a form of (X, Y). It will be appreciated that other angles besides angle 1106 and / or other axis besides axis 1012 and 1010 may be utilized to determine orientation of an object. For example, an axis parallel to a width or other dimension of an object, an axis perpendicular to axis 1012, an axis perpendicular to axis 1010, and / or other variations may be utilized to determine an orientation of an object. It will also be appreciated that image 1100 can be larger than the bounding box 1104, and can comprise more than one bounding boxes if more than one objects are detected.

[0047] The bounding box, orientation angle and position coordinates may be applied to, for example illustration 400 of Fig. 4A. For example, the hands 402, 404, 406 and 408 may be objects in a same image. The hands 402 and 406 are left hands and hands 404 and 408 are right hands. The hand 402 may have an orientation angle of 60°, the hand 404 may have an orientation angle of 110°, the hand 406 may have an orientation angle of 50°, and the hand 408 may have an orientation angle of 105°. The orientation angles of hands 402 and 404 have a difference of 50°, and the hands 402 and 404 are spatially close to each other (e.g., based on a comparison of the position of the hand 402 and the position of the hand 404 in the same image). On the other hand, the orientation angles of hands 402 and 406 have a difference of 10° and are spatially far from each other (e.g., based on a comparison of the position of the hand 402 and the position of the hand 406 in the same image). In this example, if a distribution of only orientation among the hands 402, 404, 406 and 408 is taken into account, orientation may not be selected because some values are too close and not sufficiently separated to be utilized for differentiating the hands from one another. However, by taking position distribution into account as well, orientation can be selected and utilized as a feature for tracking. This decision may be done by using threshold defined by a monotonically non-increasing function of threshold value over a spatial distance between positions of each pair of objects (e.g., spatial distance between positions of each pair of hands 402 and 404, 402 and 406, 402 and 408, 404 and 406, and other similar pairs) as shown in graph 1200 of Fig. 12. For example, if all the pairs of hand orientations exceed the curve, orientation is selected as a feature to be used for tracking.

[0048] Fig. 13 shows an exemplary illustration 1300 of how movement of a pair of hands may be tracked based on orientation according to an embodiment of the present disclosure. A series of positions of a hand 1302 (expressed as points 1304, 1306, 1308 and 1310) may be utilized by a matching unit to generate a first tracklet 1312, and another series of positions of a hand 1314 (expressed as points 1316, 1318 and 1320) may be utilized by the matching unit to generate a second tracklet 1322. Further, orientation may be selected as a feature for tracking if distribution of orientation is within a certain range suitable for differentiating hands. Thus, an orientation angle is first associated with each point. For example, points 1304, 1306, 1308 and 1310 in the first tracklet 1312 have an orientation angle of 40°, 43°, 52° and 53° respectively, and points 1316, 1318 and 1320 in the second tracklet 1322 have an orientation angle of -35°, -25° and 35° respectively. The distribution is then analysed to determine whether orientation is to be used for tracking. In this example, orientation is selected because distribution of orientation at end point of tracklets, i.e., 53° and 35°, is sufficiently separated from each other (e.g., the difference of angles exceeds a certain threshold).

[0049] This decision can also be done by using distribution of orientation among newly detected hand positions, i.e., points 1324 and 1326. For example, points 1324 and 1326 have an orientation angle of 56° and 15°. In this case, orientation is selected because those angles are separately distributed.

[0050] In order to determine whether the new position 1324 and / or the new position 1326 (e.g., detected from an object in a second image, a second video frame in a video, or other similar media) belong to the first tracklet 1312 or the second tracklet 1322, the matching unit may firstly consider the differences between an orientation angle associated with each of the new positions and the orientation angles associated with the previous point in the first tracklet 1312 (e.g., point 1310 detected based on the hand 1302 in a first image, a first video frame in the video, or other similar media) and the second tracklet 1322 (e.g., point 1320 detected based on the hand 1314 in the first image, the first video frame in the video, or other similar media).

[0051] A new position may be added to a tracklet with a lower orientation angle difference e.g., based on a comparison between a difference between the orientation angles of the new position and the point 1310, and a difference between the orientation angles of the new position and the point 1320. For example, new position point 1324 may be added to the first tracklet 1312 since orientation angle difference between that of the new position point 1324 and the point 1310 (e.g., 3° difference) is smaller than orientation angle difference between that of the new position point 1324 and the point 1320 (e.g., 38° difference). Further, new position point 1326 may be added to the second tracklet 1322 since orientation angle difference between that of the new position point 1326 and the point 1320 (e.g., 20° difference) is smaller than orientation angle difference between that of the new position point 1326 and the point 1310 (e.g., 21° difference). If the differences are the same, the matching unit will take into account the bounding box size and position for making a decision of which tracklet to add a new position to. Adding the position 1324 to the first tracklet 1310 indicates that the position 1324 is a new position of the hand 1302 e.g., the object associated with position 1324 is determined to be the hand 1302 by the matching unit. Further, adding the position 1326 to the second tracklet 1322 indicates that the position 1326 is a new position of the hand 1314 e.g., the object associated with position 1324 is determined to be the hand 1314 by the matching unit.

[0052] It will be appreciated that the first image may be an image and / or video frame of a location at a first time instance (e.g., at which points 1310 and 1320 are detected), and the second image may be an image and / or video frame of the location at a second time instance (e.g., at which new position points 1324 and 1326 are detected), wherein the second time instance occurs after the first time instance. The second time instance may occur at a time duration after the first time instance. In determining the orientation angle of the object in the first image and the second image, it will be appreciated that the length of the first image and the length of the second image are the same length, and thus the axis parallel to the length of the first image and the axis parallel to the length of the second image are also the same axis. The matching unit thus determines whether an object in a first image is same as an object in a second image based on a comparison of orientation angles for each object, for example orientation angles for each object in positions 1310 and 1320 (e.g., two objects in a first image) as well as positions 1324 and 1326 (e.g., two objects in a second image). Although Fig. 13 shows two tracklets, it is possible to have any number of tracklets depending on the number of objects to be tracked.

[0053] Fig. 14 shows an exemplary illustration 1400 of how a radius R 1402 of movement of an object (e.g., trajectory) may be determined according to an embodiment of the present disclosure. For example, the curved line 1404 may be a series of positions of an object in a video (examples being the first tracklet 1312 and second tracklet 1322). It is possible to derive a curvature associated with the curved line 1404 for use as a feature to track movement of the object. Curvature may be defined as a rate of change of position or rate / degree of change in trajectory of an object. It may be a rate of a directional and / or angular change in a tracking trajectory of an object (e.g., a tracklet or curved line 1404) and may be expressed as a reciprocal of a radius for moving trajectory (e.g., radius R 1402). The equations for curvature calculation may be as follows: where (x, y) represent coordinate of points on the trajectory, and (X,Y) and R indicate a coordinate of centre position and a radius of a circle approximating the trajectory. To determine the three unknown parameters X, Y, R, the following three conditions may be used: From these conditions, the radius R is derived as: Thus curvature is calculated by:

[0054] Fig. 15A shows an exemplary illustration 1500 of how a curvature for tracking an object may be determined according to an embodiment of the present disclosure. A tracklet 1502 may comprise at least position points 1504 (having position coordinates (x(tn-2), y(tn-2))), 1506 (having position coordinates (x(tn-1), y(tn-1))) and 1508 (having position coordinates (x(tn), y(tn))). Further referring to illustration 1512 of Fig. 15B, based on a slope 1514 calculated for a change in position between points 1504 and 1506, and a slope 1516 calculated for a change in position between points 1506 and 1508, a curvature k 1510 of the tracklet 1502 may be derived as follows: 1stderivative: or 2ndderivative: If x(tn), x(tn-1), or x(tn-2) are the same, denominator of the above equations becomes zero and the value cannot be calculated. Even in such a case, if the coordinates of the three points are different, a rotation of the coordinate system can be applied with an appropriate angle so that x coordinate of the three points are different after the rotation. This makes it possible to calculate curvature.

[0055] It will be appreciated that curvature calculation is not limited to the examples shown above, and other alternatives or variations for determining a rate of change in position of an object are also possible. For example, other different formulas for calculating a first derivative and a second derivative may be utilized. In an implementation, a rate of change in orientation angle may also be utilized to track movement of an object.

[0056] Fig. 16 shows an exemplary illustration 1600 of how curvature may be used for tracking movement of a pair of hands 1602 and 1604 according to an embodiment of the present disclosure. Curvature of a tracklet 1606 for movement of the hand 1602 and curvature of a tracklet 1608 for movement of the hand 1604 may be determined based on rate of change of the positions of the hand 1602 and 1604 respectively (e.g., as shown in Figs. 14, 15A and 15B). Further, four possible curvatures may be derived (e.g., curvature 1614 for an assumed movement from a last position point in hand tracklet 1606 to a new position point 1610, curvature 1616 for an assumed movement from the last position point in hand tracklet 1606 to a new position point 1612, curvature 1618 for an assumed movement from a last position point in hand tracklet 1608 to the new position point 1610, and curvature 1620 for an assumed movement from the last position point in hand tracklet 1610 to the new position point 1612) to determine which tracklet should the new position points 1610 and 1612 belong to. In an implementation, this may be determined by comparing each curvature and selecting the curvature with less curvature change. For example, new position point 1610 may be added to tracklet 1606 if it is determined that curvature 1614 has less curvature change then curvature 1616, and new position point 1612 may be added to tracklet 1608 if it is determined that curvature 1618 has less curvature change than curvature 1620.

[0057] Referring to illustration 2700 of Fig. 27, orientation change rate can be computed by calculating a first angle difference (e.g., represented as an angle difference calculated within reference 2706 or 2708) between the angles for the first object at the first time instance and a time instance occurring before the first time instance, then calculating a second angle difference (e.g., represented as an angle difference calculated within references 2714, 2716, 2718 or 2720) between the angle for the first object and the angle for the second object (e.g., at a second time instance represented by object positions 2710 or 2712), and determining a difference between the first and second angle differences. An exemplary formula for this calculation is as shown below: where ori(t) indicates a function representing orientation at time t.

[0058] Fig. 17 shows an exemplary illustration 1700 of how movement of a pair of hands may be tracked based on orientation according to an embodiment of the present disclosure. Based on determination of position coordinates for hands 1710 and 1712 in a first image 1702, a second image 1704 and third image 1706, it is possible for a matching unit 1708 to generate a tracklet to track movement of each of the pair of hands 1710 and 1712. For example, one or more features for each hand (e.g., bounding box, position coordinates, orientation, confidence, curvature, and other similar features) may be extracted from each image, and the matching unit 1708 may be configured to assign an identifier for each hand 1702 and 1704 in each image, as well as generate a tracklet to track movement of each hand respectively based on the one or more features.

[0059] Advantageously, by using oriented detection, a higher accuracy in tracking a same hand can be achieved. In hand tracking algorithms, features like bounding box size, location, etc are used to determine if detected hands in the current and previous images or video frames are the same hand. By introducing the new orientation-related features (orientation, curvature, etc.), the accuracy on determining the same hand is expected to increase. It is also possible to differentiate left and right hands. The features used in existing hand tracking techniques (e.g., bounding box, location, detection confidence) are not able to clearly differentiate between left and right hands. However, this is important for obtaining correct hand movement tracking results. In scenarios where the images or video frames are of pairs of hands moving and performing work at a (workplace) location, left and right hands can be identified by the orientation degrees because of the manual work / operation requirements. For example, referring to illustration 1800 of Fig. 18, it is possible to identify hand 1802 as a left hand and hand 1804 as a right hand based on an orientation angle associated with each hand.

[0060] Fig. 19 shows a flow chart 1900 for feature extraction including orientation for tracking movement of a pair of hands according to an embodiment of the present disclosure. At step 1902, one or more features are extracted for hand tracking. At step 1904, it is determined whether an object detection confidence (e.g., a confidence that an object detected in an image is a hand) is above a threshold value. If it is determined that the detection confidence is not above the threshold value, the process ends. Otherwise, the process proceeds to step 1906 in which an output flag is set to True if orientation angle is to be used as a feature for hand tracking. The procedure of step 1906 will be described later with flow chart 2000 of Fig. 20. At step 1908, it is determined whether the flag is set to true. If it is not set to True, the process proceeds to step 1912 in which the one or more features are input into a matching unit for hand tracking, and the process ends. If it is set to True, the process proceeds to step 1910 in which orientation angle is included as one of the features to be used for hand tracking, and then to step 1912 in which the one or more features (including orientation angle) are input into the matching unit for hand tracking, and the process ends.

[0061] Fig. 20 shows a flow chart 2000 for explaining how to set the output flag for the orientation feature at step 1906 in Fig. 19 according to an embodiment of the present disclosure. At step 2002, a bounding box and orientation information (e.g., orientation angle) of each detected object in an image or video frame are determined. At step 2004, it is determined whether a difference between orientation angles (e.g., difference in orientation angles between the two detected objects) is more than a threshold. The threshold value of the orientation angle difference may be estimated by considering spatial distance between the two objects, for example according to illustration 1200 of Fig. 12 as described above. If it is determined that the difference does not exceed the threshold, the process proceeds to step 2010 in which a flag is set to False, and the process ends. Otherwise, if the difference exceeds the threshold, the process proceeds to step 2006 in which it is determined whether the bounding boxes of the two objects are overlapping. If it is determined that the bounding boxes are overlapping, the process proceeds to step 2010 in which a flag is set to False, and the process ends. Otherwise, if it is determined that the bounding boxes are not overlapping, the process proceeds to step 2008 in which the flag is set to True, and the process ends.

[0062] Fig. 21 shows a flow chart 2100 for feature extraction including curvature for tracking movement of a pair of hands according to an embodiment of the present disclosure. At step 2102, one or more features (e.g., for hand tracking) are extracted. At step 2104, an output flag is set as True if orientation angle can be used (e.g., as shown in flowchart 1900). At step 1906, it is determined whether the flag is set as True. If it is not set to True, the process proceeds to step 2112 in which the one or more features (e.g., without orientation) is input into the matching unit, and the process ends. Otherwise, if the flag is set as True, the process proceeds to step 2108 in which the output flag is set as True if curvature can be used. At step 2110, it is determined whether the flag is set as True. If it is determined that the flag is not set to True, the process proceeds to step 2112 in which the one or more hand features (e.g., excluding curvature) are input into the matching unit, and the process ends. Otherwise, the process proceeds to step 2111 in which curvature is included as one of the features to be used for hand tracking, and then to step 2112 in which the one or more hand features (e.g., including curvature) are input into the matching unit, and the process ends.

[0063] Fig. 22 shows a flow chart 2200 for explaining how to set the output flag for the curvature feature at step 2108 in Fig. 21 according to an embodiment of the present disclosure. At step 2202, a curvature (rate of change in position of an object e.g., 1 / R, rate of change in degrees, etc.) of each of two objects in an image are calculated. At step 2204, it is determined whether the two calculated curvatures have a difference that exceed a threshold (e.g., 5% difference). If it is determined that the difference exceeds the threshold, the process proceeds to step 2208 in which the flag is set as True, and the process ends. Otherwise, the process proceeds to step 2206 in which the flag is set as False, and the process ends.

[0064] In a case where orientation change rate is extracted instead of curvature, process flows similar to the curvature case shown in Fig. 21 and Fig. 22 are applied, i.e., flow charts in which the word curvature is replaced with orientation change rate are applied.

[0065] Fig. 23A shows an exemplary result of tracking movement of a pair of hands without orientation. For example, tracklet 2300 (e.g., for tracking movement of a left hand) is only able to track 9 out of 21 position points, and tracklet 2302 (e.g., for tracking movement of a right hand) is only able to track 7 out of 21 position points. On the other hand, Fig. 23B shows an exemplary result of tracking movement of a pair of hands with orientation. For example, tracklet 2304 (e.g., for tracking movement of the left hand) is able to track 19 out of 21 position points, and tracklet 2306 (e.g., for tracking movement of the right hand) is able to track 20 out of 21 position points. Thus, the hand tracking results comparison before and after using the orientation feature is presented in these Figs. 23A and 23B respectively. The accuracy is estimated based on the algorithm-calculated tracking points over the ground truth total point number (e.g., 21 position points in total). It can be observed from Fig. 23A that the tracking accuracy with the existing detection model (e.g., tracklets 2300 and 2302 for left and right hands respectively) is 42.8% and 33.3%, respectively. This is because the trajectory breaks due to missing hand detection or feature identification in the tracking algorithm. Utilizing orientation to improve the detection and tracking modules, the accuracy values advantageously increase to 90.4% (for tracklet 2304) and 95.2% (for tracklet 2306), respectively. The left and right hands are thus easier to be detected and the usage of the orientation angle identifies and differentiates between the two hands with a higher confidence.

[0066] Fig. 24A shows another exemplary illustration of how movement of a pair of hands may be tracked without orientation. For example, tracklet 2400 is only able to track 12 out of 20 position points, and tracklet 2402 is only able to track 6 out of 20 position points. On the other hand, in Fig. 24B, movement of the pair of hands are tracked with orientation. For example, tracklet 2404 is able to track 20 out of 20 position points, and tracklet 2406 is able to track 19 out of 20 position points. Thus, tracking accuracy with the existing hand detection model depicted in Fig. 24A is 60% for tracklet 2400 and 30% for tracklet 2402. By including the orientation feature in the hand tracking modules, the accuracy advantageously increases to 100% (tracklet 2404) and 95% (tracklet 2406). Notably, by utilizing orientation and / or curvature for tracking the pair of hands, accuracy is high even if the left and right hands are moving and intersecting each other's trajectory as shown in the tracklets 2404 and 2406.

[0067] Thus, it is possible to achieve high accuracy by incorporating orientation and / or curvature into hand tracking. Further, based on experimental results such those shown in Figs. 23A, 23B, 24A and 24B, there is only minimal change in execution time, in which latency fluctuates between about 20 - 10 milliseconds (ms) when using orientation, compared to about 18 - 12 ms for hand tracking without orientation.

[0068] It is also possible to expand the use of orientation and curvature for other visual tracking applications. For example, Fig. 25 shows an exemplary illustration 2500 of how movement of a vehicle may be tracked e.g., in a digital map. By utilizing orientation and / curvature for tracking, a moving trajectory 2502 of a vehicle can be more accurately detected and tracked, especially for scenarios where there are many vehicles on the road. The techniques can further be applied to, for example, ship tracking and trajectory estimation in a digital map. Based on an average moving speed and direction of a ship, a future moving trajectory of the ship can be estimated for safety and navigation purposes. It will therefore be appreciated that the above examples are not exhaustive, and other tracking applications are also possible. For example, a living thing (such as a person, an animal, a bird, etc) or a part of a living thing (e.g., hand, feet, appendage, etc), a wheeled vehicle, a ship or boat, an aircraft, or other similar things may also be tracked with the same techniques as described above.

[0069] As an extension to other applications using multiple available data features, multiple features can be extracted from raw data (e.g., acoustics, temperature, light intensity, etc.), and it may be dynamically determined which features can be used to achieve better application accuracy and performance (e.g., based on the feature values, distributions, etc.). In an example, it is possible to classify objects based on frequency and / or volume for different types of objects that make sounds in different frequencies and volumes. However, this classification method cannot be used if the multiple sounds have similar frequencies. In another example, temperature values can be used to track target objects. However, in a case where a target object temperature is close to the environment temperature, or there are many other objects with similar temperature, this method is not applicable. It may then be necessary to extract other features or use other methods. In another example, lidar (light detection and ranging), which is commonly used to estimate distance, can be used to identify object types if the signal values among the object types are different from one another.

[0070] Fig. 26 depicts an exemplary computing device 2600, hereinafter interchangeably referred to as a computer system 2600, where one or more such computing devices 2600 may be used to execute the method of Figs. 8, 19, 20, 21 and 22. The exemplary computing device 2600 can be used to implement an apparatus for tracking a movement of an object. In an implementation, a system for tracking a movement of an object may comprise the apparatus and one or more image and video capturing apparatuses configured to capture one or more images or video frames, wherein the one or more images or video frames comprises an image of an object. The following description of the computing device 2600 is provided by way of example only and is not intended to be limiting.

[0071] As shown in Fig. 26, the example computing device 2600 includes a processor 2604 for executing software routines. Although a single processor is shown for the sake of clarity, the computing device 2600 may also include a multi-processor system. The processor 2604 is connected to a communication infrastructure 2606 for communication with other components of the computing device 2600. The communication infrastructure 2606 may include, for example, a communications bus, cross-bar, or network.

[0072] The computing device 2600 further includes a main memory 2608, such as a random access memory (RAM), and a secondary memory 2610. The secondary memory 2610 may include, for example, a storage drive 2612, which may be a hard disk drive, a solid state drive or a hybrid drive and / or a removable storage drive 2614, which may include a magnetic tape drive, an optical disk drive, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), or the like. The removable storage drive 2614 reads from and / or writes to a removable storage medium 2618 in a well-known manner. The removable storage medium 2618 may include magnetic tape, optical disk, non-volatile memory storage medium, or the like, which is read by and written to by removable storage drive 2614. As will be appreciated by persons skilled in the relevant art(s), the removable storage medium 2618 includes a computer readable storage medium having stored therein computer executable program code instructions and / or data.

[0073] In an alternative implementation, the secondary memory 2610 may additionally or alternatively include other similar means for allowing computer programs or other instructions to be loaded into the computing device 2600. Such means can include, for example, a removable storage unit 2622 and an interface 2620. Examples of a removable storage unit 2622 and interface 2620 include a program cartridge and cartridge interface (such as that found in video game console devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a removable solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), and other removable storage units 2622 and interfaces 2620 which allow software and data to be transferred from the removable storage unit 2622 to the computer system 2600.

[0074] The computing device 2600 also includes at least one communication interface 2624. The communication interface 2624 allows software and data to be transferred between computing device 2600 and external devices via a communication path 2626. In various embodiments of the disclosures, the communication interface 2624 permits data to be transferred between the computing device 2600 and a data communication network, such as a public data or private data communication network. The communication interface 2624 may be used to exchange data between different computing devices 2600 which such computing devices 2600 form part an interconnected computer network. Examples of a communication interface 2624 can include a modem, a network interface (such as an Ethernet card), a communication port (such as a serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB), an antenna with associated circuitry and the like. The communication interface 2624 may be wired or may be wireless. Software and data transferred via the communication interface 2624 are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communication interface 2624. These signals are provided to the communication interface via the communication path 2626.

[0075] As shown in Fig. 26, the computing device 2600 further includes a display interface 2602 which performs operations for rendering images or videos to an associated display 2630 and an audio interface 2632 for performing operations for playing audio content via associated speaker(s) 2634.

[0076] As used herein, the term "computer program product" may refer, in part, to removable storage medium 2618, removable storage unit 2622, a hard disk installed in storage drive 2612, or a carrier wave carrying software over communication path 2626 (wireless link or cable) to communication interface 2624. Computer readable storage media refers to any non-transitory, non-volatile tangible storage medium that provides recorded instructions and / or data to the computing device 2600 for execution and / or processing. Examples of such storage media include magnetic tape, CD-ROM, DVD, Blu-ray Disc, a hard disk drive, a ROM or integrated circuit, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), a hybrid drive, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computing device 2600. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and / or data to the computing device 2600 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

[0077] The computer programs (also called computer program code) are stored in main memory 2608 and / or secondary memory 2610. Computer programs can also be received via the communication interface 2624. Such computer programs, when executed, enable the computing device 2600 to perform one or more features of embodiments discussed herein. In various embodiments, the computer programs, when executed, enable the processor 2604 to perform features of the above-described embodiments. Accordingly, such computer programs represent controllers of the computer system 2600.

[0078] Software may be stored in a computer program product and loaded into the computing device 2600 using the removable storage drive 2614, the storage drive 2612, or the interface 2620. The computer program product may be a non-transitory computer readable medium. Alternatively, the computer program product may be downloaded to the computer system 2600 over the communications path 2626. The software, when executed by the processor 2604, causes the computing device 2600 to perform the necessary operations to execute the method as shown in Figs. 8, 19, 20, 21 and 22.

[0079] It is to be understood that the embodiment of Fig. 26 is presented merely by way of example to explain the operation and structure of an apparatus for tracking a movement of an object. Therefore, in some embodiments one or more features of the computing device 2600 may be omitted. Also, in some embodiments, one or more features of the computing device 2600 may be combined together. Additionally, in some embodiments, one or more features of the computing device 2600 may be split into one or more component parts.

[0080] It will be appreciated by a person skilled in the art that numerous variations and / or modifications may be made to the present disclosure as shown in the specific embodiments without departing from the spirit or scope of the disclosure as broadly described. The present embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive.

[0081] Although the present disclosure has been described with reference to the example embodiments, the present disclosure is not limited to the above. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the disclosure.

[0082] This application is based upon and claims the benefit of priority from Singapore patent application No. 10202401645Q, filed on June 7, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0083] The program can be stored and provided to the computer device using any type of non-transitory computer readable media. Non-transitory computer readable media include any type of tangible storage media. Examples of non-transitory computer readable media include magnetic storage media (such as floppy disks, magnetic tapes, hard disk drives, etc.), optical magnetic storage media (e.g. magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memories (such as mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory), etc.). The program may be provided to the computer device using any type of transitory computer readable media. Examples of transitory computer readable media include electric signals, optical signals, and electromagnetic waves. Transitory computer readable media can provide the program to the computer device via a wired communication line, such as electric wires and optical fibers, or a wireless communication line.

[0084] For example, the whole or part of the example embodiments disclosed above can be described as, but not limited to, the following supplementary notes. (Supplementary note 1) 　　A method for tracking a movement of an object, comprising: 　　extracting, by a processor, a first plurality of features for each of a first plurality of objects being tracked at a first time instance based on sensor data obtained up to the first time instance, each feature of the first plurality of features being one of a plurality of feature types; 　　performing, by the processor, object detection on the sensor data obtained at a second time instance that is after the first time instance, and extracting a second plurality of features for each of a second plurality of objects detected from the sensor data obtained at the second time instance, each feature of the second plurality of features being one of the plurality of feature types; 　　selecting, by the processor, at least one feature type from the plurality of feature types based on at least one of: a distribution of each feature type of the first plurality of features among the first plurality of objects, a distribution of each feature type of the second plurality of features among the second plurality of objects, a pattern of each feature type of the first or the second plurality of features, and a feedback from an application utilizing the first and second plurality of features for tracking the object; and 　　tracking, by the processor, the movement of the object from the first time instance to the second time instance by associating at least one of the first plurality of objects with at least one of the second plurality of objects based on a difference or a similarity between at least one feature of the first plurality of features and at least one feature of the second plurality of features that are of the at least one selected feature type. (Supplementary note 2) 　　The method of supplementary note 1, wherein tracking the movement of the object further comprises determining whether one of the second plurality of objects is same as one of the first plurality of objects based on the difference or the similarity, and assigning an identifier to each of the one of the second plurality of objects and the one of the first plurality of objects based on the determination. (Supplementary note 3) 　　The method of supplementary note 1, wherein 　　the plurality of feature types of the first and the second plurality of features include at least one of an orientation, an orientation change rate, and a curvature of the first plurality of objects and the second plurality of objects respectively, and 　　selecting the at least one feature type comprises evaluating whether the distribution of the at least one feature type of the first or the second plurality of features is within a predetermined range, whether a pattern of values of the first or second plurality of features is similar to a reference pattern, or whether a feedback from the application is positive. (Supplementary note 4) 　　The method of supplementary note 3, wherein 　　the orientation is defined by an angle between a first axis associated with the sensor data and a second axis defined by a shape of the object, and 　　extracting the first plurality of features comprises calculating the angle for each of the first plurality of objects, and 　　extracting the second plurality of features comprises calculating the angle for each of the second plurality of objects. (Supplementary note 5) 　　The method of supplementary note 4, wherein 　　the orientation change rate is defined by a rate of change in the angle, and 　　extracting the first plurality of features comprises calculating a difference between the angles for each of the first plurality of objects at the first time instance and a time instance occurring before the first time instance, and 　　extracting the second plurality of features comprises calculating a difference between the angle for each of the first plurality of objects and the angle for each of the second plurality of objects. (Supplementary note 6) 　　The method of supplementary note 3, wherein 　　the curvature is defined by a rate of change of position of the object, and 　　extracting the first plurality of features comprises calculating a rate of change of position for each of the first plurality of objects based on positions of each of the first plurality of objects at two time instances occurring before the first time instance and a position of each of the first plurality of objects at the first time instance, and 　　extracting the second plurality of features comprises calculating a rate of change of position for each of the second plurality of objects based on positions of each of the first plurality of objects at the first time instance and at a time instance occurring before the first time instance, and a position of each of the second plurality of objects at the second time instance. (Supplementary note 7) 　　The method of supplementary note 1, further comprising: 　　selecting the at least one feature type of the first and the second plurality of features if the difference exceeds a threshold for the at least one feature type. (Supplementary note 8) 　　The method of supplementary note 1, further comprising: 　　selecting the at least one feature type based on a spatial distribution determined by comparing a position of each object of the first and second plurality of objects with one another. (Supplementary note 9) 　　The method of supplementary note 7, further comprising: 　　determining the threshold based on a monotonically non-increasing function of threshold value over a spatial distance between positions of each pair of objects of the first plurality of objects or a spatial distance between positions of each pair of objects of the second plurality of objects. (Supplementary note 10) 　　An apparatus for tracking a movement of an object, comprising: 　　at least one processor; and 　　at least one memory including computer program code; the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　extract a first plurality of features for each of a first plurality of objects being tracked at a first time instance based on sensor data obtained up to the first time instance, each feature of the first plurality of features being one of a plurality of feature types; 　　perform object detection on the sensor data obtained at a second time instance that is after the first time instance, and extract a second plurality of features for each of a second plurality of objects detected from the sensor data obtained at the second time instance, each feature of the second plurality of features being one of the plurality of feature types; 　　select at least one feature type from the plurality of feature types based on at least one of: a distribution of each feature type of the first plurality of features among the first plurality of objects, a distribution of each feature type of the second plurality of features among the second plurality of objects, a pattern of each feature type of the first or the second plurality of features, and a feedback from an application utilizing the first and second plurality of features for tracking the object; and 　　track the movement of the object from the first time instance to the second time instance by associating at least one of the first plurality of objects with at least one of the second plurality of objects based on a difference or a similarity between at least one feature of the first plurality of features and at least one feature of the second plurality of features that are of the at least one selected feature type. (Supplementary note 11) 　　The apparatus of supplementary note 10, wherein tracking the movement of the object further comprises determining whether one of the second plurality of objects is same as one of the first plurality of objects based on the difference or the similarity, and assigning an identifier to each of the one of the first plurality of objects and the one of the second plurality of objects based on the determination. (Supplementary note 12) 　　The apparatus of supplementary note 10, wherein: 　　the plurality of feature types of the first and the second plurality of features include at least one of an orientation, an orientation change rate, or a curvature of the first object and the second object respectively, and 　　selecting the at least one feature type comprises evaluating whether the distribution of the first or second plurality of features is within a predetermined range, whether a pattern of values of the first or second plurality of features is similar to a reference pattern, or whether a feedback from the application is positive. (Supplementary note 13) 　　The apparatus of supplementary note 12, wherein 　　the orientation is defined by an angle between a first axis associated with the sensor data and a second axis defined by a shape of the object, and 　　extracting the first plurality of features comprises calculating the angle for each of the first plurality of objects, and 　　extracting the second plurality of features comprises calculating the angle for each of the second plurality of objects. (Supplementary note 14) 　　The apparatus of supplementary note 13, wherein 　　the orientation change rate is defined by a rate of change in the angle, and 　　extracting the first plurality of features comprises calculating a difference between the angles for each of the first plurality of objects at the first time instance and a time instance occurring before the first time instance, and 　　extracting the second plurality of features comprises calculating a difference between the angle for each of the first plurality of objects and the angle for each of the second plurality of objects. (Supplementary note 15) 　　The apparatus of supplementary note 12, wherein 　　the curvature is defined by a rate of change of position of the object, and 　　extracting the first plurality of features comprises calculating a rate of change of position for each of the first plurality of objects based on positions of each of the first plurality of objects at two time instances occurring before the first time instance and a position of each of the first plurality of objects at the first time instance, and 　　extracting the second plurality of features comprises calculating a rate of change of position for each of the second plurality of objects based on positions of each of the first plurality of objects at the first time instance and at a time instance occurring before the first time instance, and a position of each of the second plurality of objects at the second time instance. (Supplementary note 16) 　　The apparatus of supplementary note 10, further configured to select the at least one feature type of the first and the second plurality of features if the difference between the first and second plurality of features exceeds a threshold for the at least one feature type. (Supplementary note 17) 　　The apparatus of supplementary note 10, further configured to select the at least one feature type of the first and the second plurality of features for tracking the movement of the object based on a spatial distribution determined by comparing a position of each object of the first and second plurality of objects with one another. (Supplementary note 18) 　　The apparatus of supplementary note 16, further configured to determine the threshold based on a monotonically non-increasing function of threshold value over a spatial distance between positions of each pair of objects of the first plurality of objects or a spatial distance between positions of each pair of objects of the second plurality of objects. (Supplementary note 19) 　　A system for tracking a movement of an object, comprising the apparatus of any one of supplementary notes 10-18 and one or more sensor data capturing apparatuses configured to capture one or more sensor data.

[0085] 102　　video footage 104 　　video camera 106　　object tracking process 108　　output 110　　video analysis applications 502　　feature extraction unit 504　　dynamic feature selection unit 506　　application unit 508　　feedback loop 902　　hand feature extraction unit 904　　decision making unit 906　　state estimation unit 908　　matching unit 910　　application unit 2600　　computing device 2602　　display interface 2604　　processor 2606　　communication infrastructure 2608　　main memory 2610　　secondary memory

Claims

1. A method for tracking a movement of an object, comprising: 　　extracting, by a processor, a first plurality of features for each of a first plurality of objects being tracked at a first time instance based on sensor data obtained up to the first time instance, each feature of the first plurality of features being one of a plurality of feature types; 　　performing, by the processor, object detection on the sensor data obtained at a second time instance that is after the first time instance, and extracting a second plurality of features for each of a second plurality of objects detected from the sensor data obtained at the second time instance, each feature of the second plurality of features being one of the plurality of feature types; 　　selecting, by the processor, at least one feature type from the plurality of feature types based on at least one of: a distribution of each feature type of the first plurality of features among the first plurality of objects, a distribution of each feature type of the second plurality of features among the second plurality of objects, a pattern of each feature type of the first or the second plurality of features, and a feedback from an application utilizing the first and second plurality of features for tracking the object; and 　　tracking, by the processor, the movement of the object from the first time instance to the second time instance by associating at least one of the first plurality of objects with at least one of the second plurality of objects based on a difference or a similarity between at least one feature of the first plurality of features and at least one feature of the second plurality of features that are of the at least one selected feature type.

2. The method of claim 1, wherein tracking the movement of the object further comprises determining whether one of the second plurality of objects is same as one of the first plurality of objects based on the difference or the similarity, and assigning an identifier to each of the one of the second plurality of objects and the one of the first plurality of objects based on the determination.

3. The method of claim 1, wherein 　　the plurality of feature types of the first and the second plurality of features include at least one of an orientation, an orientation change rate, and a curvature of the first plurality of objects and the second plurality of objects respectively, and 　　selecting the at least one feature type comprises evaluating whether the distribution of the at least one feature type of the first or the second plurality of features is within a predetermined range, whether a pattern of values of the first or second plurality of features is similar to a reference pattern, or whether a feedback from the application is positive.

4. The method of claim 3, wherein 　　the orientation is defined by an angle between a first axis associated with the sensor data and a second axis defined by a shape of the object, and 　　extracting the first plurality of features comprises calculating the angle for each of the first plurality of objects, and 　　extracting the second plurality of features comprises calculating the angle for each of the second plurality of objects.

5. The method of claim 4, wherein 　　the orientation change rate is defined by a rate of change in the angle, and 　　extracting the first plurality of features comprises calculating a difference between the angles for each of the first plurality of objects at the first time instance and a time instance occurring before the first time instance, and 　　extracting the second plurality of features comprises calculating a difference between the angle for each of the first plurality of objects and the angle for each of the second plurality of objects.

6. The method of claim 3, wherein 　　the curvature is defined by a rate of change of position of the object, and 　　extracting the first plurality of features comprises calculating a rate of change of position for each of the first plurality of objects based on positions of each of the first plurality of objects at two time instances occurring before the first time instance and a position of each of the first plurality of objects at the first time instance, and 　　extracting the second plurality of features comprises calculating a rate of change of position for each of the second plurality of objects based on positions of each of the first plurality of objects at the first time instance and at a time instance occurring before the first time instance, and a position of each of the second plurality of objects at the second time instance.

7. The method of claim 1, further comprising: 　　selecting the at least one feature type of the first and the second plurality of features if the difference exceeds a threshold for the at least one feature type.

8. The method of claim 1, further comprising: 　　selecting the at least one feature type based on a spatial distribution determined by comparing a position of each object of the first and second plurality of objects with one another.

9. The method of claim 7, further comprising: 　　determining the threshold based on a monotonically non-increasing function of threshold value over a spatial distance between positions of each pair of objects of the first plurality of objects or a spatial distance between positions of each pair of objects of the second plurality of objects.

10. An apparatus for tracking a movement of an object, comprising: 　　at least one processor; and 　　at least one memory including computer program code; the at least one memory and the computer program code configured to, with at least one processor, cause the apparatus at least to: 　　extract a first plurality of features for each of a first plurality of objects being tracked at a first time instance based on sensor data obtained up to the first time instance, each feature of the first plurality of features being one of a plurality of feature types; 　　perform object detection on the sensor data obtained at a second time instance that is after the first time instance, and extract a second plurality of features for each of a second plurality of objects detected from the sensor data obtained at the second time instance, each feature of the second plurality of features being one of the plurality of feature types; 　　select at least one feature type from the plurality of feature types based on at least one of: a distribution of each feature type of the first plurality of features among the first plurality of objects, a distribution of each feature type of the second plurality of features among the second plurality of objects, a pattern of each feature type of the first or the second plurality of features, and a feedback from an application utilizing the first and second plurality of features for tracking the object; and 　　track the movement of the object from the first time instance to the second time instance by associating at least one of the first plurality of objects with at least one of the second plurality of objects based on a difference or a similarity between at least one feature of the first plurality of features and at least one feature of the second plurality of features that are of the at least one selected feature type.

11. The apparatus of claim 10, wherein tracking the movement of the object further comprises determining whether one of the second plurality of objects is same as one of the first plurality of objects based on the difference or the similarity, and assigning an identifier to each of the one of the first plurality of objects and the one of the second plurality of objects based on the determination.

12. The apparatus of claim 10, wherein: 　　the plurality of feature types of the first and the second plurality of features include at least one of an orientation, an orientation change rate, or a curvature of the first object and the second object respectively, and 　　selecting the at least one feature type comprises evaluating whether the distribution of the first or second plurality of features is within a predetermined range, whether a pattern of values of the first or second plurality of features is similar to a reference pattern, or whether a feedback from the application is positive.

13. The apparatus of claim 12, wherein 　　the orientation is defined by an angle between a first axis associated with the sensor data and a second axis defined by a shape of the object, and 　　extracting the first plurality of features comprises calculating the angle for each of the first plurality of objects, and 　　extracting the second plurality of features comprises calculating the angle for each of the second plurality of objects.

14. The apparatus of claim 13, wherein 　　the orientation change rate is defined by a rate of change in the angle, and 　　extracting the first plurality of features comprises calculating a difference between the angles for each of the first plurality of objects at the first time instance and a time instance occurring before the first time instance, and 　　extracting the second plurality of features comprises calculating a difference between the angle for each of the first plurality of objects and the angle for each of the second plurality of objects.

15. The apparatus of claim 12, wherein 　　the curvature is defined by a rate of change of position of the object, and 　　extracting the first plurality of features comprises calculating a rate of change of position for each of the first plurality of objects based on positions of each of the first plurality of objects at two time instances occurring before the first time instance and a position of each of the first plurality of objects at the first time instance, and 　　extracting the second plurality of features comprises calculating a rate of change of position for each of the second plurality of objects based on positions of each of the first plurality of objects at the first time instance and at a time instance occurring before the first time instance, and a position of each of the second plurality of objects at the second time instance.

16. The apparatus of claim 10, further configured to select the at least one feature type of the first and the second plurality of features if the difference between the first and second plurality of features exceeds a threshold for the at least one feature type.

17. The apparatus of claim 10, further configured to select the at least one feature type of the first and the second plurality of features for tracking the movement of the object based on a spatial distribution determined by comparing a position of each object of the first and second plurality of objects with one another.

18. The apparatus of claim 16, further configured to determine the threshold based on a monotonically non-increasing function of threshold value over a spatial distance between positions of each pair of objects of the first plurality of objects or a spatial distance between positions of each pair of objects of the second plurality of objects.

19. A system for tracking a movement of an object, comprising the apparatus of any one of claims 10-18 and one or more sensor data capturing apparatuses configured to capture one or more sensor data.

Images