Method and system for object tracking in sequences of video frames

Abstract

The present disclosure proposes novel method and system for object tracking in sequences of video frames. The method includes a dynamic update of a relevant Region of Interest (Rol) by calculation of Global Parameters of the Motion Group (GPMG) with respect to each pair of successive video frames. The Rol, which covers the Object for Tracking, is defined in the first frame as a rectangular area Roll with parameters: the coordinates {X1c, Y1c} of its central point and the lengths of the sides of the rectangle {L1x, L1y}. The changes in imagery data contained in the Rol in sequential frames allow the reconstruction of several elementary 2D components of the motion group (MG). The motion group comprises six linear non-homogeneous 2D transformation components: X-translation, Y-translation, Dilatation, Rotation, Shear-out, Shear-in. Each global parameter of the MG represents the mean value of the corresponding elementary 2D motion vector field. The Object tracking in the video frames uses the first three GPMG transformation parameters and the Rol from a given frame to calculate the new Rol parameters for the next successive frame. The changes in Rol parameters thus provide tracking through the dynamic Rol from frame to frame, with the translation of the Rol in the subsequent frame based on the calculated translation movements' GPMG and the Rol resizing, according to the calculated dilatation movements' GPMG. The system according to the present disclosure includes a module for initiation of video frames and the creation of an initial border (rectangle) 200a, and a subsequent second rectangle (300a) in the Rol for tracking, one or more processors 200b, equipped with interconnected reading module (210), module group G (230), module trained operator (220a), which itself is connected (in a two-way connection) to a module Rol (220b), where all participating devices are equipped with memory for the storage of instructions and received data, as well as with one or more pieces of specialised hardware, including specialised circuits, programmable logic circuits, and other hardware devices for execution of the method.

Description

[0001] METHOD AND SYSTEM FOR OBJECT TRACKING IN SEQUENCES OF VIDEO FRAMES

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to a method and system for object tracking in sequences of video frames, with real-life applications in fields including healthcare, robotics, surveillance, traffic management, automated vehicle tracking and more.

[0004] BACKGROUND OF THE INVENTION

[0005] Video tracking is widely used in a multitude of real-life applications, including medical imaging, robotics, traffic monitoring, automated vehicles, and more. It establishes a link between the positions of the object of interest in each frame, analysing the video frames sequentially, and connects the past and present positions of the object through forecasting, creating a bounding border (or rectangle) around the object of interest. The tracking of an object in video usually includes: A) Locating of the object and defining of a region of interest (Rol) for tracking; B) Tracking of the chosen Rol. Part A) represents 'target initialization' and involves defining and detecting the Object of interest and incorporates the process of drawing an Rol (bounding rectangle) around it in the initial frame of the video. Part (A) can be implemented in many ways, for example, using Al, training ML models, etc., depending on the application area.

[0006] Part B) includes an estimation or prediction of the Object's position and size in the remaining frames while simultaneously drawing the Rol (bounding rectangle) around it. Part A) may be realised in multiple ways, for example, through the use of Al (artificial intelligence), the training of ML (machine learning) models, etc., depending on the area of application. Therefore, after the first step, Object tracking in sequences of video frames is equivalent to sequentially drawing the bounding rectangle around a preliminary defined object, capturing its movement in every video frame.

[0007] There are two known methods of tracking when passive tracking an object in video: A) Training to recognise images - segmentation of templates in images, the use of Al on templated data, and training of ML models; B) Optic flow (OF) - for calculating the motion of pixels and estimating velocity of the movement.

[0008] Classical optical flow techniques are restricted to mono-spectral (usually grayscale) images. Current techniques may utilize a modified optical flow, such as SOFIA, as described in more detail in Kalitzin, S., Geertsema, E. and Petkov, G. 2018, Scale-iterative optical flow reconstruction from multichannel image sequences, Frontiers in Artificial Intelligence and Applications. 310, (2018), 302-314. DOI: https: / / doi.org / 10.3233 / 978-1-61499-929-4-302, Kalitzin et al. 2018a(1). Classical and modified optic flow techniques are computationally expensive because they calculate derivatives at each pixel in an image frame.

[0009] There are also other known techniques, such as GLORIA, for direct evaluation of the Global Parameters of the Motion Group (GPMG), which may be applied to both mono-spectral and multi-spectral cases. Such techniques process the image’s spectral components in parallel, while at the same time avoiding the need to calculate the optic flow in every pixel. Kalitzin, S., Geertsema, E. and Petkov, G. 2018, Optical flow groupparameter reconstruction from multichannel image sequences, Frontiers in Artificial Intelligence and Applications. 310, (2018), 290-301 , DOI: https: / / doi.org / 10.3233 / 978- 1-61499-929-4-290, Kalitzin et al. 2018b(2),

[0010] In different real-life applications, real-time video frames are often used and processed for video tracking. Known devices, such as video cameras, can capture consecutive images of a scene from the perspective of the device component capturing the image (where positioning and angle are important). The captured image frames can be sequentially arranged, reproducing a visually moving image or video of the scene, taken by the device (camera). There are several applications for these moving images (video) in contemporary life, such as: recording of important events, film and TV entertainment, archiving, image analysis, video tracking, etc. A single camera captures a 3D scene as a 2D image. This 2-dimentional representation of the 3D scene / environment complicates the image analysis due to: the loss of (or inherent uncertainty in) the depth of field of view, the relationship between the angle of image acquisition and the resulting image plane, as well as the presence of any nonlinear motion with respect to that image plane. For example, a moving device capturing an image, or moving objects within the confines of the captured scene, may present a significant challenge regarding the extraction of suitable features in the video sequences, for the purpose of various applications. There is a known US2022207657 system and method for image stabilisation (of global movement), including memory and a processor, a module for reading frames, an operator for segmentation. The system performs a i) reading of the first and second frames; ii) calculation of the global parameter of the motion group for each elementary 2D motion component; iii) application of each global parameter of the motion group to the corresponding motion group vector field; iv) summation of the vector fields; v) cumulating the global vector field of deformation with the previous cumulative global vector field of deformation; vi) deformation of the second frame by the current cumulative vector field; vii) calculation of the parameters of the global motion group (as a mean quantity of motion for each elementary 2D motion component); viii) segmentation of the first and second frame; ix) generation of the motion group G vector fields; x) calculation of the optic flow (moving the first frame into the second frame) vector field; xi) decomposition of the optic flow vector field by the motion group for determination of each elementary 2D motion component; xii) calculation of the global motion group parameters from the vector fields of the motion group decomposition.

[0011] There is also a known CN 107274433 device and method for target tracking, where the device includes: a module for image reading; a module for area setting; a unit for calculation of inter-frame translation; a block for evaluation; a memory (carrier) for storage. The described method includes: i) reading the image; ii) setting of a zone / area; iii) setting and cropping of the area of interest from the previous frame (including: marking of an object of interest within a rectangular border, controlling the size of the area of interest with a parameter kl and an evaluation of whether the object of interest is moving smoothly within the image); iv) management of the size of the area of interest through parameter k2; v) calculation of the inter-frame movement of the current frame with respect to the previous frame; vi) determination of whether the current frame is the final frame.

[0012] The so-described tracking system analyses the data at pixel level and includes a decomposition of the image's contents, which can be computationally slow and resource intensive.

[0013] Therefore, among tracking techniques, there is a necessity for quick and dynamic actualisation of the relevant region of interest Rol with respect to any pair of consecutive video frames. TECHNICAL DESCRIPTION OF INVENTION

[0014] The task is accomplished using a method and system for the tracking of objects through consecutive video frames with a chosen area (region) of interest (Rol), based on global transformations with a calculation of the global parameters of the motion group (GPMG), moving the Rol based on calculated in Rol translational movements, and resizing the Rol based on dilatational movements with respect to each pair of consecutive video frames.

[0015] In one embodiment, the present disclosure relates to a method for object tracking in a sequence of video frames, comprising: a) reading of pairs of consecutive frames in an ordered sequence of images; b) creation of time-series of images, where each frame appears chronologically earlier than the following frame in the sequence, (tracking of the object contained in the region of interest Rol thorough time).

[0016] The first frame has an earlier timestamp [7s] than the second frame, and the reading of the frames may include the reading of Ts of the first and second frames, so as to order the frames may be ordered for processing according to the method; c) tracking of an object, set up in the first frame via the coordinates of the centre and associated lengths {Xc, YX, Ux, Dy} of a rectangular region / area of interest Rol1, which encompasses the object for tracking in the first frame; d) calculation of the group transformation G1based on the size {Dx, Dy} of Rol1; e) calculation of the Global Parameters of the Motion Group (GPMG) for each of the elementary 2D components of the Motion Group (MG).

[0017] The global parameters of the motion group GPMG are calculated through G1with respect to the movement of Rol1from the first frame to the same Rol1in the second frame, where the motion group MG defines six linear non-homogeneous 2D transformation components, decomposing the complex motion within a 2D plane.

[0018] The calculation of the global parameters of the motion group GPMG includes the calculation of the mean value of motion for each elementary 2D component of the motion group MG.

[0019] The elementary 2D components of the motion group MG include global parameters as follows: X-transiation, Y-translation, Dilatation, Rotation, Shear-out, Shear-in. k 2D image is a projection of a 3D scene. The 3D translations of a given object are thus represented by 2D translations when within the image plane, while the movements towards or away from the camera are represented by the 2D dilatation. The vector fields of the motion group are members of the group transformation G and are set over the size of the Ro / 1from the first frame.

[0020] The first and second frames are multi-spectral (multichannel) images in any of the above (or in any other) examples. An example may also include mono-spectral or grayscale image frames; f) The method additionally includes calculation of the global parameters of the motion group GPMG with respect to the movement of the area Rol1from the first frame to the same area Rol1in the second frame, and the use of the first three global parameters (X-translation, Y-translation, Dilatation) together with the parameters of Rol1for determining the parameters of area / region Rol2in the second frame.

[0021] In another embodiment, the present disclosure also relates to a system for Tracking of objects through consecutive video frames, including: a) a module for initialisation of video frames and the creation of a first border / rectangle (so-called Region of Interest Rol), connected to b) a processor for performing instructions for the completion of operations, equipped with interconnected (as follows): b.l) a reading module for reading of the first frame F1from a sequence of images, simultaneously connected (one-way) to: b.2) module group G (230) for the calculation of the group transform, and to b.3) module trained operator (220a), which is connected (in a two-way connection) with: c) module region (area) of interest Rol (220b) for the generation of an area of interest to be tracked, as well as d) memory for storage of instructions and received data, as well as with e) one or more pieces of specialized hardware, including specialized circuits, programmable logic circuits, and other hardware devices for the execution of the method.

[0022] The processor, together with its participating components performs the following actions: - reading of the first and second frames from a sequence of images, where the first frame is chronologically earlier than the second in the image sequence;

[0023] - reading of the coordinates of the centre {X1c, Y1^} and the lengths {Ux, Uy} of the region of interest for tracking (rectangular region of interest) Rol1encompassing the object for tracking in the first frame;

[0024] - calculation of the group transform G1based on the size {Ux, Uy} of Rol1;

[0025] - calculation of the calculation of the Global Parameters of the Motion Group (GPMG) for each of the elementary 2D components of the Motion Group (MG), using G1with respect to the movement of region Rol1from the first frame to the same region Rol1in the second frame, where the motion group MG defines six linear non-homogeneous 2D transformation components, decomposing the complex 2D motion;

[0026] - using the first three Global Parameters of the Motion Group (GPMG), such as X translation, Y translation, Dilatation, together with the parameters {tfc, Y2^ L2x, Uy} of the region Rol2in the second frame.

[0027] The first frame may have and earlier Tsthan the second frame, and the reading of the frames may include the reading of Ts in the first and second frames, so as to order the frames appropriately for executing the operations. In this or in any other example, the reading of frames may include identification of file type, format type and / or size of the frames.

[0028] The Global Parameters of the Motion Group (GPMG) contain values for the mean quantity of motion for each elementary 2D component of the motion group MG.

[0029] The elementary 2D components of the motion group include X translation, Y translation, Dilatation, Rotation, Shear-out, Shear-in.

[0030] The vector fields of the motion group are members of group G, together with the size of Rol1corresponding to the first frame.

[0031] The first and second frames are multi-spectral (multichannel) images in each of the above (or in any other example) OR can be mono-spectral / grayscale images.

[0032] - Additionally includes calculation of the Global Parameters of the Motion Group (GPMG) with respect to the movement of area Rol1from the first frame to the same area Rol1in the second frame; use of X translation, Y translation, Dilatation, together with the parameters {X2^ Kc, L2x, L2y} of the region Rol1for the determination of the area Rol2in the second frame. An advantage of the created system and method for object tracking in video frame sequences, according to the present disclosure, is the ensuring of swift, convenient and easy tracking of objects with guaranteed quality and speed, which makes the system and method suitable for real-time applications even when using high-speed cameras, or also suitable for the tracking of multiple objects. The method, according to the present disclosure, uses a new approach towards the decomposition of movement along 2D motion components, and towards the evaluation of the mean motion in each spectral layer, in lieu of the known optic flow calculations.

[0033] The proposed innovative method and system process in parallel all spectral components of an image. The components may be RGB video channels but may also be channels from other sensors. This is the main advantage, when compared to other approaches, where Optic Flow is calculated for each spectral component separately, and the results are summed afterwards (or where components are first summed up into a common (total) intensity component, over which Optic Flow is then calculated). The proposed method and system use "sensor fusion" as an innovative feature of the SOFIA-GLORIA algorithms. The included dynamic actuaiisation of the corresponding region / area of interest Rol through the calculation of the Global Parameters of the Motion Group (GPMG) with respect to each pair of consecutive frames, ensures swift tracking.

[0034] The changes in Rol video data in consecutive frames allow for the reconstruction of several elementary 2D components of the motion group MG.

[0035] The system uses only three parameters from the Global Parameters of the Motion Group (GPMG) for the transformation and the Rol of a given frame, in order to calculate with high accuracy the new parameters of the Rol in the next frame, while ensuring tracking through a dynamic Rol from one frame to the next.

[0036] The generated Rol in the following frame is moved / translated according to the calculated Global Parameters of the Motion Group (GPMG) for translational motion and is resized accurately according to the calculated GPMG for dilatational motion.

[0037] The used algorithm for Global transform Reconstruction (GLORIA), processes in parallel all spectral components of an image, while avoiding the reconstruction of the optic flow at a pixel-level or the segmentation of the image contents, and thus reduces its own computational cost.

[0038] The system and method, according to the present disclosure use in parallel the full potential of multi-spectral data, compared to other known methods (which only use mono-spectral intensity values). The described system and method are applied to a sequence of images in video sequences. The tracking of objects through consecutive video frames, based on the described system and method, is used for mono-spectral grayscale images as well as for multispectral (multichannel) images.

[0039] The described system finds a wide range of applications in real life, specifically in areas such as medicine, robotics, security, traffic monitoring, autonomous vehicle tracking, etc.

[0040] DESCRIPTION OT THE FIGURES:

[0041] The present disclosure is illustrated in the accompanying figures (1 - 4), where:

[0042] FIG. 1; illustrates an example of an existing motion group MG with six vector fields representing six linear non-homogeneous 2D transformation components according to the embodiments described herein.

[0043] FIG. 2 : schematically illustrates an object tracking system according to various embodiments described herein. Especially the figure demonstrates the initial step of reading a first frame of an image sequence and reading the coordinates of the centre {Xc, Uc} and the lengths of the sides {Ux, Uy} of a rectangular area called Region of Interest in the first frame RoU, which covers the Object for Tracking.

[0044] FIG. 3: illustrates a method of Object tracking according to various embodiments described herein. The figure demonstrates the method from reading the second frame of an image sequence until obtaining parameters for a new corresponding Rol2area in the second frame. Since the Object tracking method presented in this disclosure is iterative, the described in Fig. 3 procedure is repeated for each newly read frame after the first one.

[0045] FIG. 4; illustrates a method of Object tracking according to various embodiments described herein. The figure generalizes the method from reading a frame with index {i|i > 1} of an image sequence until obtaining parameters for RoParea in the frame with index i. EXAMPLES

[0046] Video tracking according to the present disclosure establishes a connection between the object of interest and its appearance in each video frame. After this, it analyses the video frames sequentially, connecting the past and present positions of the object through forecasting, creating a bounding border (or rectangle) around the object of interest, called a Region of Interest (Rol). It tracks the Rol, containing the object of interest through consecutive video frames by translation and resizing of the Rol according to the global movements of the Rol between frames.

[0047] The system uses techniques for modified optic flow, such as SOFIA, as well as for direct estimation of the global parameters of the motion group (GPMG) through estimation of the mean global movement, such as GLORIA, and is applied to multi- spectral cases (processing each spectral component of the image in parallel) as well as to mono-spectral cases (such as greyscale).

[0048] The number of MG components or layers in the MG used to describe movement between frames may vary based on the image scene and / or the desired level of precision. In various embodiments, the MG may be defined to include two or more elementary 2D linear motion components selected from X-translation, Y-translation, Dilatation, Rotation, Shear-out, Shear-in.

[0049] Using these six motion components, the MG may be used to describe motion between the same Rol in sequential frames such that linear operations approximate each possible nonlinearity on the finite set of the six elements for each point of the Rol.

[0050] FIG. 1, illustrates an example of a motion group MG, including two or more non-homogeneous 2D elementary motion components may be used to generate a group G (100) of MG vector fields, each MG vector field representing a member motion component of the MG.

[0051] In the illustrated example, the MG includes six motion group components: X- transiation, Y-translation, Dilatation, Rotation, Shear-out, Shear-in, and group G includes an MG vector field for each MG component: X-translation (110), Y-translation (120), Dilatation (130), Rotation (140), Shear-out (150), Shear-in (160). The present disclosure only uses the first three out of the illustrated six components of the linear, non-homogenous 2D transformation, namely: X-translation, Y-translation, Dilatation. As described below, MG parameters may be applied to the group G vector fields to generate a group transformation. The group transformation may be a set of modified MG component vector fields wherein the MG parameters add, remove, or modify magnitudes of one or more vectors in one or more vector fields that, if summed pointwise, would otherwise generate an optical flow vector field that would drive the observed motion between the same Rol in two frames. For example, GPMG, including the mean magnitude of motion calculated for each MG component, may be applied to the MG vector fields to generate a group transformation of global MG vector fields that, when summed pointwise, drives global motion between the same Rol in the two frames. The vector fields of each MG vector field are shown for a frame size of 11x21 points. In this illustration, the points may be considered as analogous to pixels in the frame. However, in some embodiments, points may include groupings of multiple pixels.

[0052] In various embodiments, the generating group comprises determining the size of the Rol in image frames. Group G will be applied to estimate the more complex motions between the same Rol in two frames. Using FIG. 1 as an example, with a frame size of 11x21 pixels, the inhomogeneous set of elementary 2D motion components may include six motion components: X-translation (110), Y-translation (120), Dilatation (Y3Q}, Rotation (140), Shear-out (150), Shear-in (160). Thus, group G includes MG vector fields or representative equations for the vector fields having the size and a predetermined number of motion components, wherein each motion group component is represented by a vector field, e.g., for group G shown in FIG. 1 having a vector field of size 11x21 for each of the six MG components. Global components may be calculated from each of the resulting decomposition vector fields of the MG.

[0053] FIG. 2 schematically illustrates an object tracking system (200) according to various embodiments described herein. The figure demonstrates the initial step of reading, through a module for initiation of video frames and the creation of an initial border (rectangle) 200a, related to subsystem - processor 200b, equipped with a reading module (210) for reading of the first frame F1of a sequence of images, connected simultaneously and one-directionally to a module group G 230 for calculation of the group transformation and to a module trained operator 220a, which itself is connected (in a two-way connection) to a module Rol 220b, generating a region of interest for tracking.

[0054] The reading module 210 reads the parameters YU, Ux, Uy} of the RoP, chosen from a trained operator (220a). Where {XU, Wc} are the coordinates of the centre and {Ux, Uy} are the lengths of the sides of a rectangular area called Region of Interest (220b) in the first frame RoP, which covers the Object for Tracking, and calculating the group transformation (? (230) based on the size {Ux, Uy} of the RoP.

[0055] FIG. 3 illustrates a algorithm / method of object tracking (300) according to various embodiments described herein. The figure demonstrates the method from reading the second frame F2of an image sequence until obtaining parameters of the second frame 300a for the new region of interest Rol2area in F2. Reading a second frame F2(310) from a reading module 210; Reading (recall) the coordinates of the centre and the lengths of the sides {XU, Y1^ Ux, Uy} of a rectangular area called Region of Interests the first frame Rol1, which covers the Object for Tracking (320); Reading (recall) the group transformation G3(330) based on the size {Ux, Uy} of the Rol1. Reading RoDfF1) and Rol1(F2) (340); Calculate GPMG for each elementary 2D motion component of an MG using (? concerning motion from the region of interest Fo / 1(F1yito the region of interest RoDfF2), wherein the MG defines a group of six linear non-homogeneous 2D transformation components that decompose complex motion within a 2D plane and collection of the first three values, namely: the Translations along the X and Y axes with corresponding values T / and T1and the Dilatation with a corresponding value D1(350).

[0056] Then obtain the parameter values {XU, Uc, L2x, Uy} of the Rol2’m F2(360) after tracking the Rol1from the first to the second frame using equations: Calculate G2(370).

[0057] Since the Object tracking method in the present disclosure is iterative, the described in FIG. 3 procedure is repeated for each newly read frame after the first one. FIG. 4 presents the iterative step of the method. FIG. 4 illustrates a method of Object tracking according to various embodiments described herein. Assuming (i - 1) is the index of the currently processed frame (previous frame), one needs to process the next frame with index (i). Figure (400) generalizes the method starting with reading a frame Flwith index {i|i > 1} of an image sequence, (called next frame), until obtaining parameters for a RoF area. Reading a frame Fl(410). Recall the centre coordinates and the lengths {X1c, Y^c, L^x, LHy} of the sides of a rectangular area J?o / i-1in the current frame, which covers the Object for Tracking (420). Reading / recall the group transformation G'1(430) based on the size {IX1x, of the RoF-1. Reading ROF"1(F^1) and ROF"1(Fl)

[0058] (the regions of interest) (440). Calculate GPMG for each elementary 2D motion component of an MG using G'1, with respect to motion from the area jRo / i_1fF‘_0 in the current frame to the Rol1"1(Fl) area in the next frame, wherein the MG defines a group of six linear non-homogeneous 2D transformation components that decompose complex motion within a 2D plane. Use of the first three of the obtained six GPMG values, namely those of the Translations along the Xand faxes with values FXand T^y and the Dilatation with a corresponding value D1. Obtain the parameters {Xc, Yc, L'x, Uy} of the RoF in the current frame (460) after tracking RoF-1from the current to the next frame using equations:

[0059] Calculate G‘ for the next iterative step (470).

[0060] The optic flow concepts described here may be applied with the aid of different calculations, based on equations, either known or derived by those skilled in the art (in order to aid the specialist in a multitude of methodologies), which may be applied for the performance of various operations, described here:

[0061] Optical Flow Problem, Vector-Field Reconstruction and Spectral Cost-Function Approximation

[0062] Optical flow attempts to determine a deformation field given the image evolution. Deformation of an image may be described by the change of position x of its points in time t. Therefore, new image values at a given point are those transported from an old or previous image due to the spatial deformation Eq. (1).

[0063] - v(x, t) is a deformation vector field with a tiny parameter change St .

[0064] - Lc(x, t); c = 1, ... , Nc, x e R2is a multichannel image.

[0065] - c = 1, ...,NCare channels, i.e., a set of spectral components - for example, RGB.

[0066] - DV{LC} is a vector-field-induced diffeomorphism applied to the image function.

[0067] Assuming small changes and continuously differentiable functions and using notations from differential geometry where the vector field is a differential operator 7V, the Eq.

[0068] (1) might be rewritten as a differential equation Eq. (2)

[0069] From Eq. (2) follows that in the monochromatic case Nc= l the deformation field is defined only along the image gradient, and the reconstruction problem is underdetermined. On the contrary, if Nc> 2 the problem may be over-determined as the number of equations will exceed the number of unknown variables. However, if the spectrum degenerates, for example, when all spectral components are linearly dependent, the problem is still under-determined.

[0070] To account for both under and over-determined situations, the following minimization problem is defined by the quadratic local cost function in each point (x,t) of the image sequence may be postulated: v(x, t) = argminv[C[v(x, t)}] The solution for v(x, t) always exists because the cost function in Eq. (3) is positive.

[0071] However, this solution may not be unique because of possible zero modes, i.e., local directions of the deformation field along which the cost-functional is invariant.

[0072] A general approach for solving Eq. (3) is presented in Kalitzin et al., 2018a. (1) A fast reconstruction of the global deformation vector field may be executed following Kalitzin et al. 2018b (2).

[0073] Group Parameter Reconstruction

[0074] For several applications, including the present one, looking only for solutions for the optical flow equation that represent known group transformations might be advantageous. Wherein v(x, t) may be represented in a form given in Eq. (4)

[0075] - vu(x~) are the vector fields corresponding to each group generator.

[0076] - Au(t) are the corresponding transformation parameters.

[0077] Then, the minimization problem may be reformulated by substituting Eq. (4) into the Eq. (3) cost function, considering it as a minimization problem for determining the group coefficients 4“(t).

[0078] A (t) = argminA[C{A(t)}]

[0079] The generators of infinitesimal transformations algebra may be introduced as a set of differential operators using notations from differential geometry. The operators defined in Eq. (6) form the transformation group's Lie algebra.

[0080] Structural Tensor, Spatial Smoothening and Regularization

[0081] Applying the stationarity condition for Eq. (5) and introducing the quantities:

[0082] The equation for the coefficients minimizing the function may be written in the form: In Eq. (8) Suqmay refer as the structural tensor and Huas the driving vector field. In cases of an under-determined system, the removal of local singularities (zero or infinitesimal eigenvalues of the structural tensor) is done by a modified Tikhonov regularisation:

[0083] P(0 — (P^maxCO ®min (O)^(P®max(O ®mtn(^))

[0084] • emax / eminare the maximal / the minimal eigenvalues of the structural tensor;

[0085] • I is the unit matrix;

[0086] • e is the Heaviside step function;

[0087] • p is a regularisation constant

[0088] The above regularisation changes the structural tensor only in points on the deformation trajectory t where emin(t) < pemax(t) by setting the minimum eigenvalue to p and leaving the maximum unchanged. Therefore, the procedure leads to a nonsingular structural tensor in all points. Eq. (9) may be inverted to obtain the unique solution for the optical flow vector field for a given scale and regularisation parameter:

[0089] Au= (S^)-1HU(10)

[0090] Two-Dimensional Linear Transformations Group

[0091] The above reconstruction method may be applied in sequences of two- dimensional images. If restricting the transformations to the six parameters non- homogeneous linear group or motion group, the vector generators may be presented in Eq. (11):

[0092] Motion Tracking and Integration of the Group Transformations

[0093] The application of two successive morphisms v(k)and p(k)is not equivalent to one with the sum of the two vector fields. More precisely, one needs to "morph" the first vector field (shift its spatial arguments) by the second one.

[0094] The resulting vector diffeomorphism may be given as

[0095] Eq. (12) may be applied iteratively to reconstruct the global transformation between the initial image and any subsequent sequence of images.

[0096] Further calculations may be utilized, such as those described in Kalitzin et al. 2018a(l); and Kalitzin et al. 2018b(2). both of which are incorporated herein by reference.

[0097] We used GLORIA technology to solve another Computer Vision problem as described in detail in G. H. Petkov, S. Kalitzin, and P. Fornell, "Global movement image stabilization systems and methods", US patent 2022-11-08, Publication of US11494881B2, Petkov et al. 2022 (3), incorporated herein by reference. The application showed the advantages of GLORIA in solving this type of task. In this sense, the GLORIA part is similar to what has been presented in the current application, but both the algorithm and the application goal differ.

[0098] The present disclosure may include the elements described below in various combinations (for example may comprise a few chosen elements or sets of elements). Therefore, the disclosure includes the use of all these elements in isolation as well as in various combinations.

[0099] The present disclosure may include dedicated hardware implementations (not pictured in the figures) including, but not limited to, application-specific integrated circuits, programmable logic arrays, and other hardware devices that can likewise be used to execute the methods described herein.

[0100] Additionally, for the purpose of functionality, an embodiment of the present disclosure may include two or more specific interconnected hardware modules or devices with related control and data signals communicated between and through the modules or as portions of an application-specific integrated circuit.

[0101] "Transmission" of data may be done by wire, wirelessly, or by another method. In some examples, data transmitted to a receiving component may be transmitted to another component or database wherein the data may be further transmitted to the receiving component or otherwise made available to the receiving component.

[0102] The transmission of data from one processing module to a second processing module may be accomplished directly or indirectly. In one example, data may be transmitted by transmitting or another module to a receiving module by transmitting an address, location, or pointer to the data stored in memory, such as one or more databases.

[0103] The present disclosure may include one or more electronic processors, e.g., microprocessors, operable to execute instructions corresponding to the functionalities described herein. Such instructions may be stored on a computer-readable medium. Such systems, modules, units, devices, components, and the like may include functionally related hardware, instructions, firmware, or software. For example, modules or units thereof, which may include generators or engines, may include a physical or logical grouping of functionally related applications, services, resources, assets, systems, programs, databases, or the like. The systems, modules, and units, which may include data storage devices such as databases and / or pattern libraries, may include hardware storing instructions configured to execute disclosed functionalities, which may be physically located in one or more physical locations. For example, systems, modules, units, or components or functionalities thereof may be distributed across one or more networks, systems, devices, or combinations thereof.

[0104] The various functionalities of these features may be modular, distributed, and / or integrated over one or more physical devices. It will be appreciated that such logical partitions may not correspond to physical partitions of the data. For example, all portions of various systems, modules, units, or devices may reside or be distributed among one or more hardware locations.

[0105] The present disclosure contemplates a machine-readable medium containing instruction so that a device connected to the communications network, another network, or a combination thereof, can send or receive voice, video or data and communicate over the communications network, another network, or a combination thereof, using the instructions. The instructions may further be transmitted or received over the communications network, another network, or a combination thereof, via the network interface device.

[0106] The term "machine-readable medium" should include a single medium or multiple medium (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable medium" shall also be taken to include any medium capable of storing, encoding or carrying a set of instructions for execution by the machine that causes the machine to perform any one or more of the methodologies of the present disclosure. The terms "machine-readable medium," "machine-readable device," or "computer-readable device" shall accordingly be taken to include, but not be limited to, memory devices, solid-state memories such as a memory card or other package that houses one or more read-only (non-volatile) memories, random access memories, or other re-writable (volatile) memories; magneto-optical or optical medium such as a disk or tape; or other self-contained information archive or set of archives is considered a distribution medium equivalent to a tangible storage medium. The "machine-readable medium," "machine-readable device," or "computer-readable device" may be non-transitory and, in certain embodiments, may not include a wave or signal per se. Accordingly, the disclosure includes any one or more of a machine-readable medium or a distribution medium, as listed herein and includes art-recognized equivalents and successor media in which the software implementations herein are stored.

[0107] The foregoing is provided for the purposes of illustrating, explaining, and describing possible embodiments of this invention. Modifications and adaptations to these embodiments may be made without departure from the scope of the disclosure, when such modifications / adaptations are of the competency of those skilled in the art.

[0108] The present disclosure has been developed by a team of researchers from "The Big Data for Smart Society Institute" (GATT) (as a structure of Sofia University "St. Kliment Ohridski"), Sofia Bulgaria, with the cooperation of team members from the "Expertise Center for Epilepsy and Sleep Medicine" (SEIN), Netherlands, and the University of Bristol, UK.

[0109] References^

[0110] 1. Kalitzin S, Geertsema E, Petkov G. Scale-iterative optical flow reconstruction from multichannel image sequences. Front Artif Intell Appl [Internet] . 2018;310:302-14. Available from: http: / / ebooks . iospress . nl / volumearticle / 50890

[0111] 2. Kalitzin S, Geertsema E, Petkov G. Optical flow groupparameter reconstruction from multichannel image sequences. Front Artif Intell Appl [Internet] . 2018;310:290-301. Available from: http : / / ebooks .iospress . nl / volumearticle / 50889

[0112] 3. Petkov GH, Kalitzin S, Fornell P. Global movement image stabilization systems and methods [Internet] . 2022 [cited 2022 Aug 24] . Available from: https : / / patents . google . com / patent / US20220207657A1 / en%0Ahtt ps

Claims

PATENT CLAIMS1. Method for object tracking in sequences of video frames comprising: reading of video frames, setting of area (of interest), setting and marking of an object of interest with controlling and managing the size of the region of interest, and calculation of the inter-frame movement of the region of interest in the current frame with respect to the previous frame and the final frame characterised by the fact that: a) the reading of a first frame and a second frame of an image sequence, wherein the first frame is chronologically earlier than the second frame in the image sequence, while the setting of the region of interest and the marking include: b) the reading the coordinates of the centre and the lengths of the sides {X1c, Y1c, L1x, L1y} of a rectangular area called Region of Interest in the first frame Ro!1, which covers the Object for Tracking, where the calculation over and management of the region of interest include: c) the calculation of the group transformation G1based on the size {L1x, L1y] of theRol1, and d) the calculation of the GPMG (global parameters of the motion group) for each elementary 2D motion component of a MG (motion group) using G1, with respect to the motion from the Rol1area in the first frame to the same Rol1area in the second frame, wherein the MG defines a group of six linear non-homogeneous 2D transformation components (such as X translationY translation (120), Dilatation(130), Rotation (140), Shear-out (150), Shear-in (160)), that decompose the complex motion within a 2D plane; e)the using of three of the obtained six values, namely those of the Translations along the X (110) and Y (120) axes with corresponding values Fxand Fyand the Dilatation (130) with a corresponding value D1,- f) obtaining of the parameter values of the region of interest Rol2{tfc, Y2^ L2x, L2y} in the second frame, after tracking the Rol1from the first to the second frame, can be described by the following equations:

2. The method for object tracking according to claim 1, characterised by the fact that the first and second frames being multi-spectral (multichannel) images.

3. The method for object tracking according to claim 1, characterised by the fact that the parallel processing of all spectral components of the image, and additionally includes determination of the global motion group parameters (mean magnitude of motion) for an optical flow that drives the region of interest Rol1area in the first frame to the same Rol1area in the second frame;4. The method for object tracking according to claim 1, characterised by the fact that the further determination of the parameter values {)Fc, Y2^ L2xfL2y} in the second frame after tracking the region of interest Rol1from the first to the second frame, as well as by the indication of the region of interest RoF in the second frame, using the calculated parameters;5. System for object tracking in sequences of video frames comprises: a module for image reading, a module for area / region setting, a module for calculation of interframe movement, a block for evaluation, memory for data storage, characterised by the fact that is equipped with: a) a module for initialisation of video frames and the creation of a first border / rectangle (200a) and a second border / rectangle (300a) in the region of interest to be tracked; b) one or more processors (200b) for performing instructions for the completion of operations, equipped with interconnected (as follows): b.l) a reading module (210) for reading of the first F1and second F2frames, from a sequence of images, simultaneously connected (one-way) to: b.2) module group G (230) for the calculation of the group transform, and b.3) module trained operator (220a), which is connected (in a two-way connection) with: c) module region (area) of interest Rol (220b) for the generation of an area of interest to be tracked, where all participating devices are equipped with: d) memory for storage of instructions and received data, as well as with e) one or more pieces of specialized hardware, including specialized circuits, programmable logic circuits, and other hardware devices for the execution of the method.

6. The system according claim 5, characterised by the fact that the frames of the region of interest Rol (220b) being multi-spectral (multichannel) images.

Images