Single-target tracking method based on sparse optical flow motion enhancement

Abstract

The application provides a single-target tracking method based on sparse optical flow motion enhancement, belongs to the field of target tracking, and aims to improve the accuracy of motion modeling through a sparse optical flow algorithm, and to reduce the parameter amount of a tracking model by using a linear matrix when the tracking model is constructed, so as to avoid the problem that the tracking efficiency is reduced due to excessive model parameter amount and motion modeling error propagation; the method comprises the following steps: based on a current frame image, a previous frame image adjacent to the current frame image and the current frame image, a sparse optical flow algorithm is used to perform motion modeling on a tracking target, so as to determine a target candidate region image in which the tracking target is located in the current frame image; image feature extraction is performed on the target candidate region image to obtain multi-level resolution features; and the multi-level resolution features are input into a preset tracking model to estimate the position of the tracking target in the current frame image.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and in particular to a single-target tracking method based on sparse optical flow motion enhancement. Background Technology

[0002] In related technologies, correlation filtering has been widely used in single-target tracking because it can handle the correlation between two signals. Current correlation filtering methods model the motion information of the target in the tracking sequence, extract gradient features, color features, or depth features from the modeled candidate regions of the target, and calculate the correlation between the frame to be tracked and the template frame (usually the first frame) to predict target localization. To address changes in target appearance scale within the sequence, traditional correlation filtering methods typically employ online updates to the tracking model to avoid the impact of these changes.

[0003] However, current correlation filtering methods for target tracking suffer from low tracking efficiency and accuracy. Summary of the Invention

[0004] In view of this, the present invention aims to propose a single-target tracking method based on sparse optical flow motion enhancement, so as to solve the problems of low tracking efficiency and tracking accuracy of current target tracking methods.

[0005] In a first aspect, the present invention provides a single-target tracking method based on sparse optical flow and enhanced understanding, the method comprising:

[0006] For the current frame image, based on the previous frame image adjacent to the current frame image and the current frame image, a sparse optical flow algorithm is used to model the motion of the tracked target to determine the target candidate region image where the tracked target is located in the current frame image; wherein, the current frame image is any frame image in the image sequence other than the first frame image;

[0007] Image feature extraction is performed on the target candidate region image to obtain multi-level resolution features;

[0008] The multi-level resolution features are input into a preset tracking model to estimate the position of the tracking target in the current frame image; wherein, the preset tracking model includes a multi-resolution filter and a linear matrix, and the linear matrix is ​​used to reduce the number of parameters of the multi-resolution filter.

[0009] Furthermore, the step of modeling the motion of the tracking target using a sparse optical flow algorithm based on the previous frame image adjacent to the current frame image and the current frame image includes:

[0010] Obtain the previous frame image adjacent to the current frame image;

[0011] The previous frame image and the current frame image are converted into grayscale images, and the corner coordinates of corresponding pixels in the two frames are determined using the Harris corner detection algorithm.

[0012] Based on the corner coordinates, determine the optical flow movement pixel distance between the corresponding feature corner points in the previous frame image and the current frame image;

[0013] The distance the tracking target moves in two frames is determined based on the distance the optical flow moves by pixels.

[0014] Based on the predicted target position of the previous frame image and the moving distance, determine the center position of the target candidate region in the current frame image;

[0015] Based on the center position of the target candidate region, a target candidate region image of a preset size is cropped from the current frame image.

[0016] Further, determining the center position of the target candidate region in the current frame image based on the predicted target position of the previous frame image and the moving distance includes:

[0017] Determine the center coordinates of the predicted position of the tracked target in the previous frame image;

[0018] Based on the center coordinates and the moving distance, determine the center point coordinates of the target candidate region in the current frame image;

[0019] The step of cropping the current frame image into a target candidate region image of a preset size based on the center position of the target candidate region includes:

[0020] Based on the center point coordinates of the target candidate region and the preset cropping box, the target candidate region image of the preset size is cropped from the current frame image.

[0021] Furthermore, the construction steps of the multi-resolution filter include:

[0022] For the first frame of the image sequence, a candidate region of a preset size is cropped with the target ground truth bounding box as the center; wherein, the target ground truth bounding box is the bounding box of the tracked target that has been annotated;

[0023] Image features are extracted from the candidate regions to obtain the multi-level resolution features;

[0024] An interpolation model is introduced to map the multi-level resolution features to a continuous space with the same continuous period, so as to make the multi-level resolution features continuous.

[0025] For the continuous space, a set of continuous convolution filters is defined to obtain continuous convolution operators through parameterization;

[0026] Based on the linear matrix, determine the main set of convolutional filters from the continuous convolutional filters;

[0027] Based on the main set of convolutional filters, a dimension-reduced convolutional filter is constructed by mapping, and the dimension-reduced convolutional filter is used as the multi-resolution filter.

[0028] The first frame image is used as a preset sample to optimize the multi-resolution filter, so as to obtain the multi-resolution filter with optimal parameter values.

[0029] Further, the step of using the first frame image as a preset sample to optimize the multi-resolution filter includes:

[0030] For the multi-resolution filter, a loss function is constructed based on a preset sample pair; wherein the preset sample pair is obtained by transforming the first frame image; the loss function is L2 regularized to limit the value of the multi-resolution filter;

[0031] The optimal parameter values ​​of the multi-resolution filter are obtained by iteratively optimizing the minimized loss function using the Gauss-Newton iteration and conjugate gradient methods.

[0032] Furthermore, after inputting the multi-level resolution features into a preset tracking model to estimate the position of the tracking target in the current frame image, the method further includes:

[0033] The Gaussian mixture model is updated using the multi-level resolution features to add the multi-level resolution features to the training sample set; wherein, the Gaussian mixture model is used to perform feature modeling on the multi-level resolution features to express the feature distribution of the multi-level resolution features using Gaussian components; the multi-level resolution features use the target relevance output of the multi-level resolution features as labels;

[0034] At a preset frame interval, the expected loss value of the multi-resolution filter is estimated using the Gaussian component;

[0035] The parameters of the multi-resolution filter are updated so that the output of the updated multi-resolution filter satisfies the desired loss value.

[0036] Further, estimating the expected loss value of the multi-resolution filter using the Gaussian component includes:

[0037] Based on the labels corresponding to the multi-level resolution features in the training sample set, the loss function of the multi-resolution filter is transformed to obtain the expected error function of the multi-resolution filter;

[0038] Substituting the Gaussian component into the error expectation function yields the expected loss value.

[0039] Further, the preset tracking model includes: a mapping module, an acquisition module, a fusion module, and an estimation module; the step of inputting the multi-level resolution features into the preset tracking model to estimate the position of the tracking target in the current frame image includes:

[0040] The multi-level resolution features are input into the mapping module to obtain continuous features;

[0041] The continuous features are input into the acquisition module to obtain the target response map at each resolution through the multi-resolution filter;

[0042] The target response maps at multiple resolutions are input into the fusion module to obtain the target response function;

[0043] The target response function is input into the estimation module to estimate the position of the tracking target in the current frame image based on grid search and conjugate gradient descent.

[0044] A second aspect of the present invention provides a single-target tracking device based on sparse optical flow motion enhancement, the device comprising:

[0045] The motion modeling module is used to perform motion modeling on the current frame image, based on the previous frame image adjacent to the current frame image and the current frame image, using a sparse optical flow algorithm to determine the target candidate region image where the tracked target is located in the current frame image; wherein, the current frame image is any frame image in the image sequence other than the first frame image;

[0046] The feature extraction module is used to extract image features from the target candidate region image to obtain multi-level resolution features;

[0047] An estimation module is used to input the multi-level resolution features into a preset tracking model to estimate the position of the tracking target in the current frame image; wherein the preset tracking model includes a multi-resolution filter and a linear matrix, and the linear matrix is ​​used to reduce the number of parameters of the multi-resolution filter.

[0048] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps of the single-target tracking method based on sparse optical flow motion enhancement as described in the first aspect above.

[0049] Compared with existing technologies, the single-target tracking method based on sparse optical flow motion enhancement described in this invention has the following advantages:

[0050] The present invention provides a single-target tracking method based on sparse optical flow motion enhancement, comprising: for a current frame image, performing motion modeling of the tracked target using a sparse optical flow algorithm based on the previous frame image adjacent to the current frame image and the current frame image, to determine the target candidate region image in the current frame image where the tracked target is located; wherein, the current frame image is any frame image in the image sequence other than the first frame image; extracting image features from the target candidate region image to obtain multi-level resolution features; inputting the multi-level resolution features into a preset tracking model to estimate the position of the tracked target in the current frame image; wherein, the preset tracking model includes a multi-resolution filter and a linear matrix, the linear matrix being used to reduce the number of parameters of the multi-resolution filter; thus, the present invention avoids the target loss problem caused by tracking the target based solely on the cropped region of the previous frame image by using a sparse optical flow algorithm to calculate between two consecutive frames, and improves the accuracy of modeling by using a sparse optical flow algorithm for motion modeling; at the same time, since the multi-resolution filter is constructed using a linear matrix, the number of parameters of the multi-resolution filter is reduced, and the parameter update efficiency of the multi-resolution filter is improved, which helps to improve the tracking efficiency. Attached Figure Description

[0051] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0052] Figure 1 The flowchart of the single-target tracking method based on sparse optical flow motion enhancement provided in Embodiment 1 of the present invention is shown.

[0053] Figure 2 A flowchart illustrating the steps of a method for constructing a multi-resolution filter according to an embodiment of the present invention is shown;

[0054] Figure 3 A flowchart illustrating the steps of a single-target tracking method based on sparse optical flow motion enhancement provided in another embodiment of the present invention is shown.

[0055] Figure 4The flowchart of the single-target tracking method based on sparse optical flow motion enhancement provided in Embodiment 2 of the present invention is shown.

[0056] Figure 5 A schematic diagram of the structure of the single-target tracking device based on sparse optical flow motion enhancement provided in Embodiment 3 of the present invention is shown. Detailed Implementation

[0057] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0058] In related technologies, correlation filtering is commonly used to track target objects. To address changes in target appearance and scale within a sequence, traditional correlation filtering methods typically employ online updates to the tracking model. This involves updating target features every fixed frame using the latest target template to improve the robustness of the tracking model. However, current tracking models have millions of parameters, so updating model parameters using target templates from just a few frames not only reduces tracking performance but may also lead to overfitting.

[0059] Furthermore, in addition to appearance features, target-related attributes in the tracking sequence also require target motion information. Traditional correlation filtering methods model the motion information of the target in the current frame based on the predicted target location from the previous frame, cropping out the region where the predicted target appears in the current frame. However, when the predicted target in the previous frame is unreliable, the region cropped based on the incorrect target may not even contain the tracked target, causing the error to propagate and significantly reducing model accuracy.

[0060] In view of this, the present invention provides a single target tracking method based on sparse optical flow motion enhancement, which reduces the number of model parameters by using linear matrix dimensionality reduction when constructing multi-resolution filters; at the same time, it uses a sparse optical flow algorithm to compensate for camera motion to avoid error propagation caused by unreliable target prediction in the previous frame.

[0061] The single-target tracking method based on sparse optical flow motion enhancement of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0062] Example 1

[0063] Reference Figure 1 , Figure 1 The flowchart illustrating the steps of the single-target tracking method based on sparse optical flow motion enhancement provided in this embodiment of the invention is shown below. Figure 1 As shown, it includes:

[0064] S101, for the current frame image, based on the previous frame image adjacent to the current frame image and the current frame image, a sparse optical flow algorithm is used to perform motion modeling on the tracking target to determine the target candidate region image where the tracking target is located in the current frame image.

[0065] The current frame image refers to any frame image in the image sequence other than the first frame image. Specifically, in the target tracking process based on correlation filtering, the first frame image is used to initialize the filter, so as to determine the position of the target by correlating the image features of the target in the first frame image with the images in subsequent frames. Therefore, subsequent images after the first frame are used to determine the target candidate region where the target exists through motion modeling.

[0066] The image sequence can refer to a sequence of images obtained by capturing a video stream containing a tracking target, or it can be a series of images captured by a camera containing a tracking target. The image sequences are arranged in chronological order.

[0067] Specifically, the sparse optical flow algorithm refers to a method specifically designed for image registration of sparse points in an image. That is, given several points in a reference image, it finds their corresponding points in the current image. In this embodiment of the invention, since there are two modes of motion: camera motion and target motion, feature points can be selected for the image background when the camera is moving; feature points can be selected for the target when the target is moving. In some embodiments, feature points can be selected for both the image background and the target to facilitate the motion modeling process.

[0068] The sparse optical flow algorithm can calculate the target movement distance between the previous frame and the current frame. Based on the movement distance and the target position predicted in the previous frame, the candidate position of the target in the current frame can be determined. For the candidate position, the image of the preset size is cropped as the target candidate region image to reduce the input interference of irrelevant image features on the target tracking process or reduce the tracking efficiency.

[0069] S102, perform image feature extraction on the target candidate region image to obtain multi-level resolution features.

[0070] Among them, multi-level resolution features can be HOG (Histogram of Oriented Gradient), SIFT (Scale-invariant feature transform), or global feature information features.

[0071] S103, input the multi-level resolution features into the preset tracking model to estimate the position of the tracking target in the current frame image.

[0072] The preset tracking model includes a multi-resolution filter and a linear matrix, wherein the linear matrix is ​​used to reduce the number of parameters of the multi-resolution filter.

[0073] Specifically, the preset tracking model includes a mapping module, an acquisition module, a fusion module, and an estimation module. Multi-level resolution features are input into the mapping module of the preset tracking model, and the mapping module maps the multi-level resolution features to a continuous space to obtain continuous features. The continuous features are then input into the acquisition module, and the target response map at each resolution is obtained through the multi-resolution filter in the acquisition module. The multiple target response maps are fused to obtain the target response function.

[0074] The target response function is input into the estimation module, and the response value of the target response function at each location is estimated using a grid search method. Based on the response value at each location, the location with the largest response value in the grid search is taken as the center location of the target. Standard Newton's iterative optimization is applied to the Fourier series expansion to estimate the final target location.

[0075] The construction steps of the multi-resolution filter are as follows: Figure 2 , Figure 2 A flowchart illustrating the steps of constructing a multi-resolution filter according to an embodiment of the present invention is shown, as follows: Figure 2 As shown, it includes:

[0076] S201, for the first frame of the image sequence, a candidate region of a preset size is cropped with the target ground truth bounding box as the center.

[0077] Wherein, the target ground truth bounding box is the bounding box of the tracked target that has been annotated.

[0078] In this embodiment of the invention, the position of the tracking target in the first frame image is manually marked. Thus, the bounding box marked with the tracking target is the target ground truth bounding box. Based on the center of the target ground truth bounding box, a candidate region of a preset size is determined, thereby obtaining a candidate region containing the tracking target. The size of the cropped candidate region is for the purpose of facilitating feature extraction, and the invention does not impose specific limitations.

[0079] S202, an interpolation model is introduced to map the multi-level resolution features to a continuous space with the same continuous period, so as to make the multi-level resolution features continuous.

[0080] Since the multi-level resolution features are discretely distributed multi-dimensional features, an interpolation model is introduced to map the multi-level resolution features into a continuous space to facilitate subsequent filter construction. Specifically, an interpolation operator J is defined for each feature dimension d. d : L 2(T) is a Hilbert space:

[0081]

[0082] Where, N d For the resolution of each feature dimension, b d This indicates that the entire interpolation operator J utilizes a cubic interpolation kernel function. d By interpolation function b d The various translation forms are superimposed to construct x d [n] represents the weighted value of each translated function.

[0083] S203, for the continuous space, define a set of continuous convolution filters to obtain continuous convolution operators through parameterization.

[0084] S204, Based on the linear matrix, determine the main set of convolutional filters from the continuous convolutional filters.

[0085] S205, based on the main set of convolutional filters, a dimension-reduced convolutional filter is constructed by mapping, and the dimension-reduced convolutional filter is used as the multi-resolution filter.

[0086] Specifically, the continuous convolution operator S f :χ→L 2 (T) maps samples x∈χ to a target confidence function defined in the continuous space t∈[0,T); where the continuous convolution operator consists of a series of convolution filters f=(f 1 , ..., f D )∈L 2 (T) D Parameterization, where f d ∈L 2 (T) is a continuous filter for the characteristic channel d. A linear mapping matrix is ​​also defined. Based on the main set of convolutional filters f 1 , ..., f C C < D, through linear mapping A convolutional filter can be constructed in f:

[0087]

[0088] Among them, S Pf {x} is the target confidence function, P is a linear matrix, f is a convolutional filter, and J... d {x d} is the interpolation operator for feature dimension d.

[0089] S206, using the first frame image as a preset sample, the multi-resolution filter is optimized to obtain the multi-resolution filter with optimal parameter values.

[0090] The steps for optimizing the multi-resolution filter include:

[0091] For the multi-resolution filter, a loss function is constructed based on a preset sample pair; wherein the preset sample pair is obtained by transforming the first frame image; the loss function is L2 regularized to limit the value of the multi-resolution filter.

[0092] The loss function is iteratively optimized using Gauss-Newton iteration and the conjugate gradient method to obtain the optimal parameter values ​​of the multi-resolution filter. The number of iterations is not specifically limited in this invention, as long as the optimal parameter values ​​are obtained. For example, the formula is optimized using 10 Gauss-Newton iterations and 20 conjugate gradient method iterations to obtain the optimal values ​​of the filter and the linear matrix.

[0093] Specifically, the first frame image is subjected to image enhancement strategies such as contrast enhancement, brightness enhancement, addition of Gaussian noise, and Laplacian sharpening, expanding the preset sample size; among which, filter f d Given m training sample pairs The minimized functional E(f, P) is expressed as follows:

[0094]

[0095] Where m represents the number of training samples, α j This represents the weight that controls each training sample, and can also be seen as the influence of the training sample on the filter model. It is the L2 regularization term, w is the weight matrix with respect to spatial location; S Pf {x j} is the sample x j The target confidence function.

[0096] Specifically, the filter is trained by minimizing E(f, P) in the Fourier domain, and the Fourier coefficients of the continuous features of the mapping after feature extraction from the first frame image are used. Substituting the above Fourier coefficients into the minimization functional E(f, P) according to Passevar's theorem (the law of conservation of energy) of Fourier transform, we get the following equation:

[0097]

[0098] in, Let J represent the Fourier coefficients of the continuous eigenmap z = J{x}, where It is x dThe expression for the Discrete Fourier Transform; the last regularization term is the F-norm of the linear matrix P, λ = 2 × 10⁻⁶. -7 The first step is to control the weights. Then, the Gauss-Newton iteration and conjugate gradient method are used to iteratively optimize the minimization functional to obtain the optimal values ​​of the filter and the linear matrix. Thus, the multi-resolution filter with optimal parameters is obtained.

[0099] Reference Figure 3 , Figure 3 The following is a flowchart illustrating the steps of a single-target tracking method based on sparse optical flow motion enhancement according to another embodiment of the present invention, as shown below. Figure 3 As shown, it includes:

[0100] S301, Obtain the previous frame image adjacent to the current frame image.

[0101] S302, convert the previous frame image and the current frame image into grayscale images, and use the Harris corner detection algorithm to determine the corner coordinates of corresponding pixel positions in the two frames.

[0102] S303, Based on the corner coordinates, determine the optical flow movement pixel distance between the corresponding feature corner points in the previous frame image and the current frame image.

[0103] The sparse optical flow algorithm calculates the distance traveled by feature points between two frames. Therefore, it first acquires the previous frame image adjacent to the current frame and performs calculations based on the feature points between the two frames. Since the sparse optical flow algorithm relies on the assumption that the grayscale value of a moving object in an image does not change within a very short time, both the previous and current frames are converted to grayscale images. The grayscale values ​​of feature points in one image are used to determine the position of the feature points in the other image, and then the travel distance is determined based on the coordinates. Specifically, the Harris corner detection algorithm is used to determine the corner coordinates H of corresponding pixels in the preceding and following frames. t-1 (x, y), H t (x+u(x,y), y+v(x,y)); Based on the corner coordinates, the formula for calculating the distance traveled is as follows:

[0104] I(x,y,t-1)=I(x+u(x,y),y+v(x,y),t)

[0105] Where I(x, y, t-1) represents the brightness of a pixel position in frame t-1; I(x+u(x, y), y+v(x, y), t represents the brightness of the corresponding pixel position in frame t; u(x, y) represents the distance the pixel position moves on the horizontal axis; and v(x, y) represents the distance the pixel position moves on the vertical axis.

[0106] Solving the above formula: I(x, y, t-1) ≈ I(x, y, t) + I x u(x, y) + I y Expanding v(x, y) using Taylor series, we get:

[0107]

[0108] Solving the above equations yields the specific values ​​of u(x,y) and v(x,y).

[0109] S304, determine the movement distance of the tracking target in two frames based on the optical flow moving pixel distance.

[0110] By obtaining the movement distance of the feature points (i.e., pixel positions) on the horizontal and vertical axes of the coordinate system through the above steps, the movement distance of the tracked target in the two frames can be obtained. Since the target position will also change during the background movement, if the feature points of the background are selected to determine the movement distance of the pixel positions, the movement distance of the tracked target in the two frames can be determined based on the movement distance of the pixel positions.

[0111] Specifically, since target tracking involves two movement modes: target movement and background movement (i.e., camera movement), for target movement, selecting feature points from the target location to determine the movement distance is convenient; therefore, feature points can be directly extracted from the target location. For background movement, feature points can be selected from the image background to facilitate distance calculation. In some embodiments, to improve the accuracy of distance calculation, feature points can be selected from both the target location and the image background location for calculation, and then the distance can be analyzed comprehensively. Since camera movement and target movement are opposite—that is, if the camera moves to the left, it can be considered that the target moves to the right—a direction can be set for the movement distance: positive for target movement and negative for background movement, facilitating subsequent distance calculation and determination of the target center coordinates.

[0112] S305, based on the predicted target position of the previous frame image and the moving distance, determine the center position of the target candidate region in the current frame image.

[0113] S306, Based on the center position of the target candidate region, crop the target candidate region image of a preset size from the current frame image.

[0114] Specifically, after determining the movement distance, the position coordinates (x, y) of the center position of the target location estimated in the previous frame image can be obtained. t-1 y t-1 Then, based on the distances u(x,y) and v(x,y) that the target has moved along the horizontal and vertical axes respectively, the center coordinates (x,y) of the target position in the current frame image can be obtained. t-1+u(x, y), y t-1 If we use +v(x, y)), we can determine the image region to be cropped based on the coordinates of the center position, and use a cropping box of a preset size to crop out the target candidate region to facilitate the subsequent feature extraction process.

[0115] In this embodiment of the invention, although the center point coordinates of the target in the current frame image are determined here, it is essentially a rough estimate of the target position. Therefore, only the center point coordinates are used to crop the image to obtain the target candidate region, so as to prevent the cropped image position from losing the target. Then, the target position is re-determined based on the cropped region.

[0116] S307, Image feature extraction is performed on the target candidate region image to obtain multi-level resolution features.

[0117] S308, input the multi-level resolution features into the preset tracking model to estimate the position of the tracking target in the current frame image.

[0118] S309, the Gaussian mixture model is updated using the multi-level resolution features to add the multi-level resolution features to the training sample set.

[0119] The Gaussian mixture model is used to model the multi-level resolution features, so as to express the feature distribution of the multi-level resolution features using Gaussian components; the multi-level resolution features use the target relevance output of the multi-level resolution features as labels.

[0120] S310, at a preset frame interval, the expected loss value of the multi-resolution filter is estimated using the Gaussian component.

[0121] S311, Update the parameters of the multi-resolution filter so that the output of the updated multi-resolution filter satisfies the desired loss value.

[0122] In this embodiment of the invention, since each frame is used as a training sample for updating, the small interval between consecutive image frames and the high similarity of the images may lead to overfitting of the tracking model; therefore, the method is to update using N frames of samples after an interval of N frames.

[0123] Specifically, the input multi-level resolution feature x is added to the training sample set as a training sample, and the output response map y is used as the label of the multi-level resolution feature x. Since the multi-resolution filter is constructed and optimized on the multi-level resolution feature x and the corresponding output response map y, i.e., given p(x,y), and the target correlation output y on the multi-level resolution feature x is a Gaussian function, the difference in the target correlation output y corresponding to different training samples is reflected in the different peak values ​​of the Gaussian function. Therefore, it can be assumed that the output corresponding to all training sample features is the same, i.e., y = y0. The shift of different y peak values ​​is completed by the multi-level resolution feature x. Therefore, the distribution of the training samples can be decomposed as follows: thus, The offset of the peak value of y is a fixed value; p(x) represents the distribution of the features of the training samples; thus, the optimization of the multi-resolution filter can be performed by estimating only p(x).

[0124] Therefore, a Gaussian mixture model is used to model the features of the training samples: Let

[0125]

[0126] Where L represents the Gaussian component N(x; μ) in the Gaussian mixture model. l The number of I), π l μ represents the prior weight of the l-th Gaussian component. l ∈χ is the mean of the Gaussian components, and the identity matrix I is the covariance of the Gaussian components. Therefore, the Gaussian components in the Gaussian mixture model are used to represent the features of the training samples. Then, the Gaussian components are substituted into the training loss function to estimate the expected loss of the multi-resolution filter output using the corresponding Gaussian mixture model of the training samples. Finally, the multi-resolution filter is updated online based on this expected loss.

[0127]

[0128] Where, π l μ represents the prior weight of the l-th Gaussian component. l ∈χ is the mean of the Gaussian components, and y0 is the fixed response output value of the current hypothesis; S f {μ l} is to use μ l The target relevance of the features is output. The training loss function of the multi-resolution filter can be expressed as minimizing the target relevance expectation: Therefore, substituting the Gaussian component and the fixed response output value into the expectation expression, we obtain the above formula for calculating the expected loss; where, the expectation... It is estimated on the distribution p(x,y).

[0129] In this embodiment of the invention, adding a training sample to the training sample set implemented by the Gaussian mixture model requires updating the Gaussian mixture model using the features of the current frame; the specific update method is as follows:

[0130] Given a new training sample x j First, initialize a π. m =γ,μ m =x j The Gaussian component is m, where the learning rate γ = 0.012. If adding the Gaussian component m results in a total Gaussian component exceeding L, then the two peaks μ are merged. k and μ l Use similar Gaussian components k and l to update the Gaussian model:

[0131]

[0132] Where, π l μ represents the prior weight of the l-th Gaussian component. l ∈χ is the mean of the Gaussian components, π k This represents the prior weight of the k-th Gaussian component.

[0133] This invention initializes a filter based on the first frame of an image sequence. For subsequent frames in the image sequence, a sparse optical flow algorithm is used to calculate the target's movement distance between the two frames using the previous and current frames. The target's center point position in the current frame is predicted using the movement distance and the target's position in the previous frame. Features are then extracted from the center point position image, and the target's position in the current frame is estimated using a preset tracking model. Therefore, by using a linear matrix for feature dimensionality reduction during the initialization of the multi-resolution filter, the number of parameters in the multi-resolution filter is reduced, thus avoiding the problem of slow updates due to a large number of parameters affecting tracking efficiency during optimization. The target movement distance between two frames is calculated using a sparse optical flow algorithm to model the motion of the tracked target. Since the images used are the current frame and the previous frame, rather than just the target candidate region from the previous frame, the problem of target loss that might occur when using candidate regions for motion modeling is avoided. Furthermore, it allows for more accurate cropping of candidate regions containing the tracked target from the current frame image.

[0134] Example 2

[0135] Reference Figure 4 , Figure 4 The flowchart illustrating the steps of the single-target tracking method based on sparse optical flow motion enhancement provided in this embodiment of the invention is shown below. Figure 4 As shown, the method includes:

[0136] S401, based on the first frame image, constructs a multi-resolution filter for a preset tracking model.

[0137] Specifically, based on the ground truth value of the target bounding box given in the first frame image, the image is cropped to extract image features from the cropped part. The convolutional filter of the correlation filter tracking model is initialized by the extracted multi-level resolution features, and the linear matrix is ​​reduced in dimensionality by factorization.

[0138] In some embodiments, since different filters in a set of convolutional filters focus on different target aspects, the weight matrix of the main convolutional filter is set to model the importance of several composite indicators, i.e., the weights of the composite indicators, using a linear matrix.

[0139] Specifically, after constructing the multi-resolution filter, a weight mapping matrix is ​​defined. For the main set of convolutional filters f 1 , ..., f C Weighting:

[0140]

[0141] Among them, W C Describes each primary convolutional filter f c The weight.

[0142] Therefore, the influence of weights is also considered in the process of optimizing the filter: the filter is optimized by minimizing the functional E(f,P). The filter f d Given m training sample pairs The minimized functional E(f, P, W) is expressed as follows:

[0143]

[0144] Where m represents the number of training samples, α j This represents the weight that controls each training sample, and can also be seen as the influence of the training samples on the filter model. β is the L2 regularization term, β is the weight matrix with respect to spatial location, and W is the weight matrix with respect to the convolution filter.

[0145] The filter is trained by minimizing E(f, P, W) in the Fourier domain. The Fourier coefficients of the continuous features obtained after feature extraction and mapping are then used. in, It is x d The expression of the discrete Fourier transform.

[0146] Substituting the above equation into the minimized functional E(f) in step four using Passevar's theorem (the law of conservation of energy) based on Fourier transform, we get the following equation:

[0147]

[0148] in, The Fourier coefficients of the continuous eigenmap z = J{x} are represented by the last regularization term, which is the F-norm (also called the F-norm) of the linear matrix P, λ = 2 × 10⁻⁶. -7 It controls the weights.

[0149] The formula was optimized using 10 Gauss-Newton iterations and 20 conjugate gradient iterations to obtain the optimal values ​​of the filter and the linear matrix for subsequent tracking.

[0150] S402, for the current frame image, based on the previous frame image adjacent to the current frame image and the current frame image, a sparse optical flow algorithm is used to perform motion modeling on the tracked target to determine the target candidate region image where the tracked target is located in the current frame image.

[0151] The current frame image is any frame image in the image sequence other than the first frame image.

[0152] Specifically, the two images are converted into grayscale images, and the Harris corner detection algorithm is used to determine the corner coordinates of corresponding pixels in the two images. The movement distance of the corresponding pixels in the two images, i.e., the optical flow movement distance, is calculated based on the corner coordinates. Then, based on the optical flow movement distance and the target position of the previous image, the center position of the current image is determined, and a target candidate region of a preset size is cropped based on the center position.

[0153] S403, perform image feature extraction on the target candidate region image to obtain multi-level resolution features.

[0154] S404, input the multi-level resolution features into the preset tracking model to estimate the position of the tracking target in the current frame image.

[0155] The preset tracking model includes a multi-resolution filter and a linear matrix, wherein the linear matrix is ​​used to reduce the number of parameters of the multi-resolution filter.

[0156] In this embodiment of the invention, based on the filter f, linear matrix P, and weight matrix W obtained in step S401, a convolutional filter based on factorization dimensionality reduction is used to map samples x∈χ in the continuous feature space into a target confidence function s(t)=S in the continuous interval t∈[0,T). Pf {x}(t):

[0157]

[0158] Among them, s dRepresents the target response map at the d-th dimension resolution, through Fusion of target response functions at different resolutions.

[0159] For the fused target response function, a grid search method is used to obtain the response value at each location. The location with the largest response value is taken as the target center location, and the final target location is estimated based on the target center location.

[0160] S405, the Gaussian mixture model is updated using the multi-level resolution features to add the multi-level resolution features to the training sample set.

[0161] The Gaussian mixture model is used to model the multi-level resolution features, so as to express the feature distribution of the multi-level resolution features using Gaussian components; the multi-level resolution features use the target relevance output of the multi-level resolution features as labels.

[0162] S406, at a preset frame interval, the expected loss value of the multi-resolution filter is estimated using the Gaussian component.

[0163] S407, update the parameters of the multi-resolution filter so that the output of the updated multi-resolution filter satisfies the desired loss value.

[0164] In this embodiment of the invention, since a weight matrix is ​​added when constructing the multi-resolution filter, the influence of the weights must also be incorporated into the expected loss estimation method used when optimizing the parameters of the multi-resolution filter.

[0165]

[0166] Where, π l μ represents the prior weight of the l-th Gaussian component. l ∈χ is the mean of the Gaussian components, and y0 is the fixed response output value of the current hypothesis; S f {μ l} is to use μ l The target relevance output of the feature; β is the weight matrix with respect to spatial location, and W is the weight matrix with respect to the convolution filter.

[0167] This invention initializes a filter based on the first frame of an image sequence. For subsequent frames in the image sequence, a sparse optical flow algorithm is used to calculate the target's movement distance between the two frames using the previous and current frames. The target's center point position in the current frame is predicted using the movement distance and the target's position in the previous frame. Features are then extracted from the center point position image, and the target's position in the current frame is estimated using a preset tracking model. Therefore, by using a linear matrix for feature dimensionality reduction during the initialization of the multi-resolution filter, the number of parameters in the multi-resolution filter is reduced, thus avoiding the problem of slow updates due to a large number of parameters affecting tracking efficiency during optimization. The target movement distance between two frames is calculated using a sparse optical flow algorithm to model the motion of the tracked target. Since the images used are the current frame and the previous frame, rather than just the target candidate region from the previous frame, the problem of target loss that might occur when using candidate regions for motion modeling is avoided. Furthermore, it allows for more accurate cropping of candidate regions containing the tracked target from the current frame image.

[0168] Example 3

[0169] Reference Figure 5 , Figure 5 A schematic diagram of the structure of a single-target tracking device based on sparse optical flow motion enhancement provided in an embodiment of the present invention is shown, as follows: Figure 5 As shown, the device includes:

[0170] The motion modeling module 501 is used to perform motion modeling on the current frame image based on the previous frame image adjacent to the current frame image and the current frame image, using a sparse optical flow algorithm to determine the target candidate region image where the tracking target is located in the current frame image; wherein, the current frame image is any frame image in the image sequence other than the first frame image;

[0171] Feature extraction module 502 is used to extract image features from the target candidate region image to obtain multi-level resolution features;

[0172] The estimation module 503 is used to input the multi-level resolution features into a preset tracking model to estimate the position of the tracking target in the current frame image; wherein, the preset tracking model includes a multi-resolution filter and a linear matrix, and the linear matrix is ​​used to reduce the number of parameters of the multi-resolution filter.

[0173] In some embodiments, the motion modeling module 501 includes:

[0174] The acquisition submodule is used to acquire the previous frame image adjacent to the current frame image;

[0175] The first determining submodule is used to convert the previous frame image and the current frame image into grayscale images, and use the Harris corner detection algorithm to determine the corner coordinates of corresponding pixel positions in the two frames;

[0176] The second determining submodule is used to determine the optical flow movement pixel distance between the corresponding feature corner points in the previous frame image and the current frame image based on the corner point coordinates;

[0177] The third determining submodule is used to obtain the movement distance of the tracking target in two frames based on the optical flow moving pixel distance;

[0178] The fourth determining submodule is used to determine the center position of the target candidate region in the current frame image based on the predicted target position of the previous frame image and the moving distance;

[0179] The cropping module is used to crop the target candidate region image of a preset size from the current frame image based on the center position of the target candidate region.

[0180] In some embodiments, the apparatus further includes:

[0181] The first update module is used to update the Gaussian mixture model using the multi-level resolution features, so as to add the multi-level resolution features to the training sample set; wherein, the Gaussian mixture model is used to perform feature modeling on the multi-level resolution features, so as to express the feature distribution of the multi-level resolution features using Gaussian components; the multi-level resolution features use the target relevance output of the multi-level resolution features as labels;

[0182] The second estimation module is used to estimate the expected loss value of the multi-resolution filter by using the Gaussian component at preset frame intervals.

[0183] The second update module is used to update the parameters of the multi-resolution filter so that the output of the updated multi-resolution filter meets the expected loss value.

[0184] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the single-target tracking method based on sparse optical flow motion enhancement as described in any of the above embodiments.

[0185] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0186] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0188] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0190] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0191] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0192] The single-target tracking method based on sparse optical flow motion enhancement provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A single-target tracking method based on sparse optical flow motion enhancement, characterized in that, The method includes: For the current frame image, based on the previous frame image adjacent to the current frame image and the current frame image, a sparse optical flow algorithm is used to model the motion of the tracked target to determine the target candidate region image where the tracked target is located in the current frame image; wherein, the current frame image is any frame image in the image sequence other than the first frame image; Image feature extraction is performed on the target candidate region image to obtain multi-level resolution features; The multi-level resolution features are input into a preset tracking model to estimate the position of the tracking target in the current frame image; wherein, the preset tracking model includes a multi-resolution filter and a linear matrix, and the linear matrix is ​​used to reduce the number of parameters of the multi-resolution filter; The preset tracking model includes: a mapping module, an acquisition module, a fusion module, and an estimation module; the step of inputting the multi-level resolution features into the preset tracking model to estimate the position of the tracking target in the current frame image includes: The multi-level resolution features are input into the mapping module to obtain continuous features; The continuous features are input into the acquisition module to obtain the target response map at each resolution through the multi-resolution filter; The target response maps at multiple resolutions are input into the fusion module to obtain the target response function; The target response function is input into the estimation module to estimate the position of the tracking target in the current frame image based on grid search and conjugate gradient descent.

2. The single-target tracking method based on sparse optical flow motion enhancement according to claim 1, characterized in that, The step of modeling the motion of the tracked target using a sparse optical flow algorithm based on the previous frame image adjacent to the current frame image and the current frame image includes: Obtain the previous frame image adjacent to the current frame image; The previous frame image and the current frame image are converted into grayscale images, and the corner coordinates of corresponding pixels in the two frames are determined using the Harris corner detection algorithm. Based on the corner coordinates, determine the optical flow movement pixel distance between the corresponding feature corner points in the previous frame image and the current frame image; The distance the tracking target moves in two frames is determined based on the distance the optical flow moves by pixels. Based on the predicted target position of the previous frame image and the moving distance, determine the center position of the target candidate region in the current frame image; Based on the center position of the target candidate region, a target candidate region image of a preset size is cropped from the current frame image.

3. The single-target tracking method based on sparse optical flow motion enhancement according to claim 2, characterized in that, Determining the center position of the target candidate region in the current frame image based on the predicted target position of the previous frame image and the moving distance includes: Determine the center coordinates of the predicted position of the tracked target in the previous frame image; Based on the center coordinates and the moving distance, determine the center point coordinates of the target candidate region in the current frame image; The step of cropping the current frame image into a target candidate region image of a preset size based on the center position of the target candidate region includes: Based on the center point coordinates of the target candidate region and the preset cropping box, the target candidate region image of the preset size is cropped from the current frame image.

4. The single-target tracking method based on sparse optical flow motion enhancement according to claim 1, characterized in that, The construction steps of the multi-resolution filter include: For the first frame of the image sequence, a candidate region of a preset size is cropped with the target ground truth bounding box as the center; wherein, the target ground truth bounding box is the bounding box of the tracked target that has been annotated; Image features are extracted from the candidate regions to obtain the multi-level resolution features; An interpolation model is introduced to map the multi-level resolution features to a continuous space with the same continuous period, so as to make the multi-level resolution features continuous. For the continuous space, a set of continuous convolution filters is defined to obtain continuous convolution operators through parameterization; Based on the linear matrix, determine the main set of convolutional filters from the continuous convolutional filters; Based on the main set of convolutional filters, a dimension-reduced convolutional filter is constructed by mapping, and the dimension-reduced convolutional filter is used as the multi-resolution filter. Using the first frame image as a preset sample, the multi-resolution filter is optimized to obtain the multi-resolution filter with optimal parameter values.

5. The single-target tracking method based on sparse optical flow motion enhancement according to claim 4, characterized in that, The step of using the first frame image as a preset sample to optimize the multi-resolution filter includes: For the multi-resolution filter, a loss function is constructed based on a preset sample pair; wherein the preset sample pair is obtained by transforming the first frame image; the loss function is L2 regularized to limit the value of the multi-resolution filter; The optimal parameter values ​​of the multi-resolution filter are obtained by iteratively optimizing the minimized loss function using the Gauss-Newton iteration and conjugate gradient methods.

6. The single-target tracking method based on sparse optical flow motion enhancement according to claim 1, characterized in that, After inputting the multi-level resolution features into a preset tracking model to estimate the position of the tracking target in the current frame image, the method further includes: The Gaussian mixture model is updated using the multi-level resolution features to add the multi-level resolution features to the training sample set; wherein, the Gaussian mixture model is used to perform feature modeling on the multi-level resolution features to express the feature distribution of the multi-level resolution features using Gaussian components; the multi-level resolution features use the target relevance output of the multi-level resolution features as labels; At a preset frame interval, the expected loss value of the multi-resolution filter is estimated using the Gaussian component; The parameters of the multi-resolution filter are updated so that the output of the updated multi-resolution filter satisfies the desired loss value.

7. The single-target tracking method based on sparse optical flow motion enhancement according to claim 6, characterized in that, The step of estimating the expected loss value of the multi-resolution filter using the Gaussian component includes: Based on the labels corresponding to the multi-level resolution features in the training sample set, the loss function of the multi-resolution filter is transformed to obtain the expected error function of the multi-resolution filter; Substituting the Gaussian component into the error expectation function yields the expected loss value.

Images