A target tracking method and device based on multi-template adaptive updating

Abstract

The application provides a target tracking method and device based on multi-template adaptive updating, and belongs to the technical field of computer vision. The method comprises the following steps: selecting a target to be tracked from the first frame of a video, extracting the features of the target to be tracked in the first frame, and obtaining a target appearance filter template of the first frame; from the second frame of the video, each frame is sequentially taken as a current frame; using the target appearance filter template of the previous frame considering a long-time memory template, obtaining the target appearance filter template of the current frame, and obtaining the tracking result of the target to be tracked in the current frame; the tracking result of the target to be tracked in each frame constitutes the tracking result of the target in the video. By introducing reliable historical frame information, the target template of the current frame is ensured not to be interfered and polluted by obstacles, background clutter and the like, and then the target can be positioned in time and accurately when the occlusion ends or the target reappears, so that better tracking effect can be achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a target tracking method and device based on multi-template adaptive updating. BACKGROUND

[0002] Video target tracking is a research hotspot in the fields of digital image processing and computer vision, and has important research value in many civil and military fields such as video monitoring, intelligent transportation, human-computer interaction, aerial reconnaissance, and guided attack. Target tracking covers a wide range: according to the number of targets, it can be divided into single-target tracking and multi-target tracking; according to the distribution of imaging cameras, it can be divided into single-view tracking and multi-view tracking; according to the imaging data source, it can be divided into tracking based on visible light, infrared, and radar; and according to whether the target information is known in advance, it can be divided into tracking with target priori and tracking without target priori.

[0003] Among them, the single-target tracking problem based on visible light data source without target priori under single view is the focus of current academic research due to the characteristics of not needing camera networking and strong universality to targets. The working process is as follows: for the target position in the starting frame of a given video sequence, a series of target features are extracted using the spatiotemporal correlation, the target position, size, and other parameters are obtained, and finally the complete target motion trajectory is obtained. Since the target information to be tracked is only provided in the first frame, reliable target priori information is extremely scarce. In addition, the algorithm also needs to cope with interference from targets (rotation, scale), background (occlusion, illumination change), platform (shaking, blur, moving out of the field of view), and other aspects.

[0004] In view of the above difficult problems, in recent years, a series of tracking algorithms represented by the BACF method (background-aware correlation filter algorithm) enhance the resistance of the algorithm to external interference by jointly modeling the target and the background, and ensure the accurate positioning of the target. However, the above method fails to consider the rich historical frame target information in the tracking process. When the target is subjected to complex interference such as long-time occlusion and moving out of the field of view, it is often difficult to recapture the target by relying only on single-frame target background modeling. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a target tracking method and device based on multi-template adaptive updating. By introducing reliable historical frame information, the present application ensures that the current frame target template is not polluted by interference such as occlusion and background clutter, and then accurately locates the target in a timely manner when the occlusion ends or the target reappears, thereby achieving better tracking effect.

[0006] The first aspect embodiment of the present application provides a target tracking method based on multi-template adaptive updating, comprising:

[0007] selecting a target to be tracked from a first frame of a video, extracting features of the target to be tracked in the first frame to obtain a target appearance filter template of the first frame, the target appearance filter template representing a mapping relationship between features and corresponding labels of the target to be tracked and a background in a selected search region in a corresponding frame;

[0008] from a second frame of the video, each frame is sequentially taken as a current frame; using the target appearance filter template of the previous frame, the target appearance filter template of the current frame is obtained and a tracking result of the target to be tracked in the current frame is obtained; wherein the target appearance filter template of the previous frame considers a long-time memory template, and the long-time memory template is a best template selected from all retained target appearance filter templates;

[0009] the tracking result of the target to be tracked in each frame is combined to obtain a tracking result of the target to be tracked in the video.

[0010] In one specific embodiment of the present application, the target appearance filter template of the first frame is obtained by extracting features of the target to be tracked in the first frame, including:

[0011] an image slice is obtained in the first frame with the center point of the target to be tracked as the center according to the size of the set search region, the area of the image slice is greater than the area of the target to be tracked and contains the complete target to be tracked;

[0012] feature extraction is performed on the image slice to construct an image slice feature matrix of the first frame

[0013] a Gaussian label matrix is constructed for the image slice

[0014] element-level point division operation is performed on the Gaussian label matrix and the feature matrix to obtain the target appearance filter template w1 of the target to be tracked in the first frame.

[0015] In one specific embodiment of the present application, the target appearance filter template of the current frame is obtained using the target appearance filter template of the previous frame, including:

[0016] 1) using the target appearance filter template of the previous frame, the target appearance filter template of the current frame is obtained by iterative calculation, including:

[0017] denote the current frame as the t-th frame, t≥2, extract the feature matrix of the current frame image slice use the target appearance filter template wt-1 constructing a target appearance filter template w t of a current frame;

[0018] wherein w t is a cost function as follows:

[0019]

[0020] wherein, is an auxiliary variable of a current frame, I K is a K*K unit matrix, K represents a dimension of a feature matrix, F is a discrete Fourier coefficient matrix, M is a size of a search region, w t-1 is a target appearance filter template of a t-1 frame, and μ respectively represent a Lagrange multiplier and a penalty factor;

[0021] 2) performing element-level point multiplication on w t obtained in step 1) and to obtain a tracking response map of the current frame;

[0022] 3) updating the target appearance filter template of the current frame by determining a peak value of the tracking response map of the current frame, comprising:

[0023] if the peak value and the peak state value of the tracking response map of the current frame are greater than the respective corresponding threshold values, the target appearance filter template of the current frame calculated by formula (1) is retained; otherwise, the target appearance filter template calculated by formula (1) is deleted, and the target appearance filter template of the previous frame is updated as the target appearance filter template of the current frame.

[0024] In one specific embodiment of the present application, the solving method of the cost function of w t is as follows:

[0025] After separating variables for the cost function as shown in formula (1), the i-time cross direction multiplier method is used for iterative solving to obtain the target appearance filter template w of the current frame after i+1 iterations. t

[0026] wherein the iterative calculation expression is as follows:

[0027]

[0028] wherein the superscript i represents the i-th iteration, η is a growth rate of each iteration, μ max is an upper limit of the penalty factor; w t ​The iterative initial value of the target appearance filter template is a zero matrix.

[0029] In one embodiment of the present application, the obtaining of the tracking result of the target in the current frame comprises:

[0030] The position of the peak of the tracking response map of the current frame is taken as the target center point position of the current frame, and the target center point position of the current frame is the target tracking result of the current frame.

[0031] In one embodiment of the present application, after the target appearance filter template of the current frame is obtained, the method further comprises:

[0032] If t-1 is equal to an integer multiple of the set frame interval A and the target appearance filter template of the current frame calculated by using formula (1) is retained at least once in the current interval from the t-A+1th frame to the tth frame, a long-time memory template of the current interval is obtained; otherwise, the current interval does not generate a long-time memory template.

[0033] The method for obtaining the long-time memory template comprises:

[0034] The target appearance filter template of the first frame of the video is taken as the first long-time memory template.

[0035] Starting from the second frame, if there is a long-time memory template in the current interval, the corresponding frame of the retained target appearance filter template is obtained according to the retained target appearance filter template in the current interval, and the target appearance filter template of the frame with the highest peak value in the corresponding frame is the long-time memory template of the current interval.

[0036] When the number of long-time memory templates reaches a set upper limit value, the newest long-time memory template replaces the earliest long-time memory template.

[0037] In one embodiment of the present application, the method further comprises:

[0038] After the long-time memory template of the current interval is obtained, all long-time memory templates are used to update the target appearance filter template of the current frame, and the calculation expression is as follows:

[0039]

[0040] wherein w j represents the jth long-time memory template, the function f() is used to measure the matching degree of the selected long-time memory and the target appearance of the current frame, β j represents the adaptive weight of the jth long-time memory template, tp is the number of the current long-time memory template; σ is a regularization parameter, pr jis the prior weight of the jth long-term memory template, and the expression is as follows:

[0041]

[0042] wherein, a is a decay parameter;

[0043] When a new current frame arrives, the result obtained by formula (3) is taken as the target appearance filter template of the updated previous frame.

[0044] The second aspect embodiment of the present application provides a target tracking device based on multi-template adaptive updating, comprising:

[0045] The tracking target acquisition module is configured to select a target to be tracked from the first frame of the video, extract features of the target to be tracked in the first frame, and obtain a target appearance filter template of the first frame, wherein the target appearance filter template represents a mapping relationship between features and corresponding labels of the target to be tracked and a background in a selected search region in the corresponding frame.

[0046] The target appearance filter template update module is configured to take each frame as a current frame from the second frame of the video, obtain the target appearance filter template of the current frame by using the target appearance filter template of a previous frame, and obtain a tracking result of the target to be tracked in the current frame, wherein the target appearance filter template of the previous frame considers a long-term memory template, and the long-term memory template is a best template selected from all retained target appearance filter templates.

[0047] The tracking result generation module is configured to combine the tracking results of the target to be tracked in each frame to obtain a tracking result of the target to be tracked in the video.

[0048] The third aspect embodiment of the present application provides an electronic device, comprising:

[0049] at least one processor; and a memory connected to the at least one processor in communication;

[0050] The memory stores instructions executable by the at least one processor, and the instructions are configured to perform the target tracking method based on multi-template adaptive updating.

[0051] The fourth aspect embodiment of the present application provides a computer readable storage medium, which stores computer instructions for causing the computer to perform the target tracking method based on multi-template adaptive updating.

[0052] The present application has the following characteristics and advantages:

[0053] The present application mainly faces the tracking problem without target prior information, and calculates and counts the data characteristics of the response graph obtained by tracking each frame. On the one hand, the time of the low response value can be identified in time, and the drift correction by introducing the corresponding measures can be carried out; on the other hand, the ideal tracking state can be screened, and the tracking result can be saved as a high-reliability template to provide time sequence guidance for subsequent target positioning.

[0054] The present application retains reliable historical templates in the tracking process, and continuously traces back according to the target appearance and the tracking state, selects the historical template closest to the current state, and realizes the recapture and positioning of the drift problem.

[0055] The present application can be applied to the technical fields of intelligent video monitoring, traffic flow analysis and aviation reconnaissance, improves the accuracy of target tracking, and has high application value. In the aspect of intelligent monitoring, the present application can be used to analyze and process video data in time without manual intervention, and complete the motion analysis and behavior understanding of the target of interest. In the field of intelligent traffic, the present application can be used to automatically identify the model and license plate of a car, and count key information such as vehicle speed and traffic flow, which is helpful to improve the problem of traffic congestion. The present application can also be applied to road sign recognition and pedestrian motion trajectory analysis, and remind the driver at the necessary moment to ensure safe driving and safe traffic. In addition, in the image guidance application, target detection and tracking by the present application can realize accurate positioning and precise attack on air or ground targets. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 It is a whole flow chart of a target tracking method based on multi-template adaptive updating in an embodiment of the present application.

[0057] Figure 2 It is a schematic diagram of target template mutation caused by interference factors in a specific embodiment of the present application.

[0058] Figure 3 It is a typical tracking response schematic diagram in the tracking state discrimination in a specific embodiment of the present application.

[0059] Figure 4 It is a flow chart of multi-template adaptive selection and weight distribution in a specific embodiment of the present application. DETAILED DESCRIPTION

[0060] The embodiment of the present application proposes a target tracking method and device based on multi-template adaptive updating, which is further described in detail as follows in combination with the drawings and specific embodiments.

[0061] The first aspect embodiment of the present application proposes a target tracking method based on multi-template adaptive updating, comprising:

[0062] selecting a to-be-tracked target from a first frame of a video, extracting features of the to-be-tracked target in the first frame to obtain a target appearance filter template of the first frame, the target appearance filter template representing a mapping relationship between features and corresponding labels of the to-be-tracked target and a background in a selected search region in a corresponding frame;

[0063] from a second frame of the video, taking each frame as a current frame in turn; obtaining the target appearance filter template of the current frame and obtaining a tracking result of the to-be-tracked target in the current frame by using the target appearance filter template of a previous frame, wherein the target appearance filter template of the previous frame considers a long-time memory template, and the long-time memory template is a best template selected from all retained target appearance filter templates;

[0064] combining the tracking result of the to-be-tracked target in each frame to obtain a tracking result of the to-be-tracked target in the video.

[0065] The memory system is one of the most powerful but difficult to understand functional systems in the human brain. With the help of a developed memory system, humans can adapt to complex environments and have the ability to handle time series problems. Among them, the memory model jointly proposed by Atkinson and Shiffrin divides the human memory process into three parts: sensory memory, short-term memory, and long-term memory. When the external environment is relatively stable, the short-term memory can quickly give a response, and when the external environment changes dramatically, the brain retrieves the information stored in the long-term memory to obtain a response to the current event. The target tracking problem solved by the present application also has similarities with the human memory process: in the initial frame of tracking, a preliminary impression of the target is first formed; in the subsequent frames, if the external environment is stable and the target itself also does not have a dramatic appearance change, the method described in the present application can quickly locate the target while updating the target impression. However, when the external environment is unstable (for example, there is occlusion or clutter interference) or the target appearance changes rapidly, it is difficult to match the current target well by relying only on a single frame template (i.e., the short-term memory template in the embodiment of the present application). Considering that the angle of observation of the target is relatively limited, in most cases, the observed target appearance has appeared before, so introducing some historical frame template information (i.e., the long-time memory template in the embodiment of the present application) for target positioning can achieve better results, thereby avoiding tracking errors caused by similar interference or occlusion.

[0066] Specifically, the embodiment of the present application proposes a target tracking method based on multi-template adaptive updating, and the overall process is as follows Figure 1The method is shown in the figure. First, the target template features of the previous frame are introduced in each frame tracking time, the target background is jointly modeled, and a short-time memory template is formed; then the tracking state is discriminated to filter out high-reliable filter templates to form long-time memory templates; when the target appearance changes sharply or long-time interference occurs, the long-time memory templates are activated, and the weight of the multiple high-reliable long-time memory templates filtered out is autonomously determined to realize the joint estimation of the current target position. The method comprises the following steps:

[0067] 1) Obtain a single-channel video to be tracked.

[0068] In the embodiment, if the input video is a multi-channel color video, it can be converted into a single-channel video and subjected to frame splitting processing. The embodiment has no other requirements for image resolution and video length.

[0069] 2) Select a target to be tracked in the first frame of the video of step 1), and extract a target appearance filter template of the target to be tracked in the first frame.

[0070] In the embodiment, a target to be tracked is selected in the first frame of the video obtained in step 1) according to autonomous target detection or manual framing, and the center point of the target is used as the center to demarcate an image slice with a size of 3m*3n as a search area for target positioning.

[0071] It should be noted that the embodiment sets the side length of the search area around the target to be 3 times the side length of the target. If the search area cannot meet the size requirement because the target is too large in any frame, a false image meeting the size requirement can be constructed by completing the boundary pixels outward.

[0072] The image slice is subjected to feature extraction to construct an image slice feature matrix of the first frame In a specific embodiment of the application, 31-dimensional gradient histogram features, 10-dimensional color space features, and target gray values are extracted to constitute the feature matrix of the image slice, so the feature matrix in the embodiment has a size of 3m*3n*42. After that, a Gaussian label matrix is constructed for the image slice In a specific embodiment of the application, the Gaussian label matrix has a size of 3m*3n, and the decay factor is 1 / 8.

[0073] The Gaussian label matrix is combined with the feature matrix Element-level point division operation is performed, so that the target appearance filter template w1 of the target to be tracked in the first frame is obtained, and the relationship between the target to be tracked and the background region in the image slice of the first frame is established.

[0074] 3) From the t=2 frame, the t frame is taken as the current frame;

[0075] 4) The target appearance filter template of the previous frame is used to calculate the target appearance filter template of the current frame through iteration.

[0076] In the current frame, the position of the target center point in the previous frame of the current frame in the video is taken as the center to obtain the image slice of the current frame with a size of 3m*3n;

[0077] The feature matrix of the current frame image slice is extracted The target appearance filter template w t-1 of the previous frame (i.e. the t-1 frame) of the current frame is used to construct the target appearance filter template w t of the current frame.

[0078] In one specific embodiment of the present application, the cost function of w t is as follows:

[0079]

[0080] Wherein, w t is the target appearance filter template of the current frame to be solved, is the feature matrix of the current frame image slice, represents the Gaussian label matrix, is an auxiliary variable of the current frame, I K is a K*K unit matrix (K represents the dimension of the feature matrix, which is 42 in the embodiment), F is a discrete Fourier coefficient matrix, M is the size of the search region (the search region size is 3m*3n in the embodiment), w t-1 is the target appearance filter template of the t-1 frame (the template is the target appearance filter template saved before the current frame; when t=2, w1 is the target appearance filter template saved before the current frame). And μ respectively represent the Lagrange multiplier and the penalty factor.

[0081] In the embodiment, the previously saved target filter template w t-1 is introduced in the process of constructing the cost function, so that each independent frame is effectively associated and constrained, Figure 2This paper demonstrates the changes in target template features over time in two video sequences: one featuring a cyclist and the other a toy car. It shows that when the viewpoint of the target in the image changes significantly, the target template features change drastically. By introducing the aforementioned temporal constraints, the changes in these features can be effectively eliminated. Figure 2 The figure illustrates the problem of template abrupt changes caused by drastic changes in the target itself and its external environment. As shown in the figure, due to significant changes in the shooting position relationship between the UAV platform and the moving target, the target's appearance undergoes a considerable change in perspective. Under these circumstances, the feature template exhibits several spikes. This invention, by considering the target's historical appearance information, can effectively reduce the template abrupt change problem caused by these drastic appearance changes.

[0082] In equation (1),

[0083]

[0084] The cost function of the original BACF algorithm continuously updates and learns the appearance template of the target by learning the mapping relationship between target features and label values. From the perspective of template learning and updating, it can be regarded as a relatively "aggressive" updating method.

[0085]

[0086] The temporal constraints introduced in this embodiment of the invention enhance the inter-frame correlation of templates by comparing them with historical target appearance templates. This can be seen as a slightly "conservative" template learning mechanism. Therefore, this embodiment of the invention comprehensively ensures that the template makes a trade-off between two different update strategies, avoiding template fluctuations and abrupt changes caused by drastic changes in target appearance, and even preventing tracking drift.

[0087] The last two terms in equation (1) are the auxiliary variables and Lagrange terms introduced to solve the cost function. After separating the variables, the cost function can be solved iteratively using the i-th cross-direction multiplier method (ADMM) to obtain the target appearance filter template after i+1 iterations in the current frame (i.e., the t-th frame). As shown in the following formula:

[0088]

[0089] Wherein, the superscript i represents the i-th iteration. In this embodiment, the maximum number of iterations is set to 5, the initial penalty factor μ = 3, the growth rate of each iteration is set to η = 3, and the upper limit of the penalty factor μ is set. max =25. Other parameters ( w t The initial values ​​of all matrices are set to zero.

[0090] According to equation (2), when the iteration ends... That is, the target appearance filter template w of the current frame. t .

[0091] 5) Using the target appearance filter template of the current frame obtained in step 4), obtain the tracking response map of the current frame and the target center point position of the current frame.

[0092] In this embodiment, after completing the target appearance filter template w for the current frame... t After solving for w t Feature matrix of the current frame image slice Perform element-wise dot products to obtain the tracking response map for the current frame.

[0093] Because of w t Essentially, the response map reflects the mapping relationship between the target and the surrounding background features and the positive and negative labels. From a physical perspective, the response map represents the degree of matching between the candidate samples at each location and the existing template. Figure 3 This is a schematic diagram of a typical tracking response during tracking state determination in a specific embodiment of the present invention. Since the present invention is geared towards single-target tracking applications, such as... Figure 3 As shown in the left-hand appendix, under ideal conditions, the response map has only a single peak, and the surrounding area is relatively smooth; when the target is disturbed, the contaminated sample cannot perfectly match the template, resulting in multiple peaks in the response map, such as... Figure 3 As shown in the middle attached figure, or an abnormal shape may appear, such as Figure 3 As shown in the attached diagram on the right. Similar to the baseline BACF algorithm, in this embodiment of the invention, the location of the peak value in the tracking response map is selected as the target center point location of the current frame.

[0094] 6) Update the target appearance filter template of the current frame by determining the peak value of the tracking response map of the current frame.

[0095] In this embodiment, since the target is often affected by factors such as background occlusion and similar interference during the tracking process, it is also necessary to determine and monitor the tracking status of the current frame in order to decide whether to include the tracking result of the current frame in the reliable historical tracking result, so as to guide the template update process in subsequent tracking frames.

[0096] This invention employs methods to monitor tracking status and screen high-confidence samples by measuring the peak value and shape of the response graph. The peak value of the response graph can be directly calculated, while the shape is determined by calculating its kurtosis value. According to the definition shown below, the kurtosis value represents the quotient of the squares of the fourth-order cumulant and the second-order cumulant of a variable, and can measure the kurtosis and tailing of the sampled data.

[0097]

[0098] where x is a set of random variables, n represents the number of variables, denotes the average value of variable x. Data with high kurtosis value presents a sharp peak and less tailing; while data with low kurtosis value usually has more peaks or a flat top. Therefore, the kurtosis value and the tracking state can be monitored in time. In practical engineering applications, the kurtosis function in MATLAB can be directly called to calculate the kurtosis value of the response graph (which is essentially a set of two-dimensional variables).

[0099] The embodiment of the present application uses the average peak value and the average kurtosis value of the response graph to determine whether the tracking state of the current frame meets the update condition: if the tracking state is good (in this embodiment, 0.65 times the historical average peak value is used as the peak threshold value and 0.45 times the historical average kurtosis value is used as the kurtosis threshold value for determination, i.e., the peak value and the kurtosis value of the response graph of the current frame both need to be greater than the corresponding threshold value), the target appearance filter template of the current frame is retained for subsequent screening, i.e., w t is retained as the target appearance filter template of the current frame; otherwise, the target appearance filter template obtained in step 4) is deleted, and w t is updated to avoid template pollution, the target appearance filter template obtained in step 4) is deleted, and w t-1 is updated to avoid template pollution, the target appearance filter template obtained in step 4) is deleted, and w t-1 is updated to avoid template pollution, the target appearance filter template obtained in step 4) is deleted, and w

[0100] It should be noted that the historical average peak value and the historical average kurtosis value are the average of the peak values and the kurtosis values of the previous history, for example, if the current frame is the 21st frame, the determination is to compare whether the peak value of the response graph of the current frame is greater than 0.65 times the average of the peak values of the previous 20 frames, and whether the kurtosis value of the response graph of the current frame is greater than 0.45 times the average of the kurtosis values of the previous 20 frames.

[0101] 7) Determine t:

[0102] If t-1 is an integer multiple of the set frame interval A (30 frames are selected in this embodiment) and the target appearance filter template of the current frame calculated by formula (1) has been retained at least once from the t-A+1th frame to the tth frame, the long-term memory template of the current interval composed of the t-A+1th frame to the tth frame is obtained, and then step 8) is entered; otherwise, step 9) is entered.

[0103] wherein the long-term memory template acquisition method is as follows:

[0104] The target appearance filter template of the first frame of the video is used as the first long-term memory template. Starting from the second frame, whenever t-1 is an integer multiple of the set frame interval A (30 frames in this embodiment), all target appearance filter templates retained in the current interval from the t-A+1 frame to the t frame are centrally screened. The frame with the best tracking state (i.e. the frame with the highest peak value of the response map) is selected from the frames corresponding to all target appearance filter templates retained in the A frame (i.e. the current frame when the retained target appearance filter template is calculated according to formula (1)). The target appearance filter template of the frame is used as the long-term memory template of the current interval.

[0105] It should be noted that if the target appearance filter template of any current frame calculated in step 4) is not retained in the current interval, then the current interval will not generate a long-term memory template.

[0106] In one specific embodiment of the present invention, long-term memory templates are generated at 30-frame intervals. Considering the real-time requirements of tracking, the maximum number of long-term memory templates is set to 4, and the obtained long-term memory is continuously updated using a first-in-first-out (FIFO) method to adapt to changes in the target's appearance.

[0107] 8) Update the target appearance filter template for the current frame using all currently retained long-term memory templates.

[0108] After obtaining multiple reliable long-term memory templates, the contribution weight of each template needs to be adaptively determined to achieve adaptive fusion of information from multiple historical frames. Figure 4 This is a flowchart of multi-template adaptive selection and weight allocation in a specific embodiment of the present invention.

[0109] like Figure 4 As shown, the circle represents the long-term memory template obtained after filtering in step 7), and the line represents the process of weight determination and fusion of long-term memory templates at different times. In this embodiment, during the initial tracking stage (frames 2-31), only the target appearance filter template of the initial frame is introduced as a reliable long-term memory template (memory 1) to guide the target localization process. Along with the tracking process, frames 26 (long-term memory 2) and 59 (long-term memory 3) are respectively filtered as reliable long-term memory templates for joint learning. The target person is obscured by trees after frame 73. When the target person reappears in the field of vision at frame 110, although the target has not appeared in the images of frames 73-109, during the target localization process at frame 110, the method described in this embodiment can autonomously find the historical template closest to the current target state and assign it a higher weight based on the target feature template (memory 1) saved at the beginning frame of the tracking task and the highly reliable feature templates (memory 2, memory 3, and memory 4) filtered during the tracking process. Finally, the target is relocated through feature matching.

[0110] The cost function of the multi-template joint modeling is as follows:

[0111]

[0112] The function updates w by adaptively assigning and weighting the reliable target appearance feature filters screened out previously t to be the new w when the next current frame arrives t-1 The ability to continuously improve the resistance to target appearance mutation by fusing multi-time sequence target appearance information, and ensure accurate positioning of the target in the case of temporary target drift, etc. Wherein, w j represents the reliable jth long-time memory template screened out through feature constraint and state discrimination, the function f() is used to measure the matching degree of the long-time memory selected and the current frame target appearance, β j represents the adaptive weight of the jth long-time memory template to be determined, tp is the number of current reliable long-time memory templates, and the upper limit value of tp in the present example is 4. σ=0.05 is a regularization parameter, pr j is the prior weight of the jth long-time memory template, and its expression is shown in formula (4):

[0113]

[0114] Wherein, α is a decay parameter, and its value is 0.5 in the present example. Since the objective function needs to be minimized, for the ideal or matched historical memory, it should be as small as possible. Therefore, the present application uses the reciprocal of the square of the peak value BK of the corresponding frame response map as the matching function f(). In addition, since the target features have time sequence consistency during tracking, the current frame target appearance tends to be closer to the target features of the recent several frames from the probability point of view. Therefore, in the present application, higher weight is given to the most recent target appearance to cope with the appearance changes caused by target deformation, rotation and other factors; lower weight is given to the target appearance with longer time interval. In addition, the sum of each prior weight pr j is equal to 1, that is, j . j

[0115] In engineering practice, the optimal weight combination (β1,...,β tp ) can be directly obtained by calling the QuadProg library function in MATLAB. After obtaining the weights of different historical frame appearance templates, the weights β j are multiplied by the corresponding template appearance filter w j and summed to obtain the updated target appearance filter template w t of the tth frame. ​

[0116] 9) let t = t + 1 ', and then return to step 3) to obtain the position of the peak value in the response map of each frame as the target center point position of the frame.

[0117] When all the frames of the video are traversed, the target center point positions of each frame in the video constitute the target tracking result in the video.

[0118] To achieve the above embodiment, the second aspect embodiment of the present application proposes a target tracking device based on multi-template adaptive update, comprising:

[0119] The tracking target acquisition module is configured to select a target to be tracked from the first frame of the video, extract the features of the target to be tracked in the first frame, and obtain a target appearance filter template of the first frame. The target appearance filter template represents the mapping relationship between the features and corresponding labels of the target to be tracked and the background in the selected search region in the corresponding frame.

[0120] The target appearance filter template update module is configured to take each frame as a current frame from the second frame of the video. The target appearance filter template of the current frame is obtained by using the target appearance filter template of the previous frame, and the tracking result of the target to be tracked in the current frame is obtained. The target appearance filter template of the previous frame considers a long-time memory template, and the long-time memory template is the best template selected from all retained target appearance filter templates.

[0121] The tracking result generation module is configured to combine the tracking results of the target to be tracked in each frame to obtain the tracking result of the target to be tracked in the video.

[0122] It should be noted that the foregoing embodiment of the method for tracking a target based on adaptive update of multiple templates is also applicable to the method for tracking a target based on adaptive update of multiple templates of the present embodiment, which will not be described herein again. According to the method for tracking a target based on adaptive update of multiple templates provided in the embodiments of the present application, a target to be tracked is selected from the first frame of a video, features of the target to be tracked in the first frame are extracted to obtain a target appearance filter template of the first frame, the target appearance filter template representing a mapping relationship between features and corresponding labels of the target to be tracked in the corresponding frame and a background in a selected search region; from the second frame of the video, each frame is sequentially taken as a current frame; the target appearance filter template of the previous frame is used to obtain the target appearance filter template of the current frame and obtain a tracking result of the target to be tracked in the current frame; wherein the target appearance filter template of the previous frame considers a long-time memory template, and the long-time memory template is a best template selected from all retained target appearance filter templates; the tracking result of the target to be tracked in each frame is combined to obtain a tracking result of the target to be tracked in the video. Thus, reliable historical frame information can be introduced, the current frame target template can be ensured not to be polluted by interference such as an occlusion and background clutter, and the target can be accurately positioned in time when the occlusion ends or the target reappears, so that better tracking effect can be achieved.

[0123] To achieve the above-mentioned embodiments, the third aspect of the present application provides an electronic device, comprising:

[0124] at least one processor; and a memory connected with the at least one processor in communication;

[0125] wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to perform the above-mentioned method for tracking a target based on adaptive update of multiple templates.

[0126] To achieve the above-mentioned embodiments, the fourth aspect of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make the computer perform the above-mentioned method for tracking a target based on adaptive update of multiple templates.

[0127] It should be noted that the computer-readable medium in the above disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0128] The computer-readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device. The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the target tracking method based on multi-template adaptive update described above.

[0129] Computer program code for carrying out operations of the present disclosure can be written in any one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0130] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0131] In addition, the terms "first", "second", etc. are used only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0132] Any process or method descriptions or descriptions of the flow diagrams in the specification or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and the various embodiments of the application include additional implementations in which the order of execution or the specific logic functions (or steps) can be changed, including according to the functionality involved, without departing from the scope of the embodiments of the application. It should be understood that the embodiments of the application can include hardware, software, firmware, or some combination thereof.

[0133] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer- readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (a magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0134] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0135] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a computer-readable storage medium. When the programs are executed, they include one or a combination of the steps of the method embodiments.

[0136] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0137] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A target tracking method based on multi-template adaptive update, characterized in that, include: A target to be tracked is selected from the first frame of the video. By extracting the features of the target to be tracked in the first frame, a target appearance filter template for the first frame is obtained. The target appearance filter template represents the mapping relationship between the features and corresponding labels of the target to be tracked and the background in the selected search area in the corresponding frame. Starting from the second frame of the video, each frame is taken as the current frame in sequence; using the target appearance filter template of the previous frame, the target appearance filter template of the current frame is obtained and the tracking result of the target to be tracked in the current frame is obtained; wherein, the target appearance filter template of the previous frame takes into account the long-term memory template, and the long-term memory template is the best template selected from all the retained target appearance filter templates; The tracking results of the target in each frame are combined to form the tracking result of the target in the video; The step of extracting features of the target to be tracked in the first frame to obtain the target appearance filter template of the first frame includes: In the first frame, an image slice is obtained with the center point of the target to be tracked as the center and according to the size of the set search area. The area of ​​the image slice is larger than the area of ​​the target to be tracked and contains the complete target to be tracked. Feature extraction is performed on the image slice to construct the image slice feature matrix of frame 1. Construct a Gaussian label matrix for the image slice Gaussian label matrix With characteristic matrix Perform element-level dot division to obtain the target appearance filter template w1 of the target to be tracked in the first frame; The step of obtaining the target appearance filter template for the current frame using the target appearance filter template from the previous frame includes: 1) Using the target appearance filter template from the previous frame, the target appearance filter template for the current frame is obtained through iterative calculation, including: Let the current frame be frame t, where t ≥ 2. Extract the feature matrix of the image slice in the current frame. Using the target appearance filter template wt from the previous frame -1 Construct the target appearance filter template wt for the current frame; The cost function for wt is shown below: in, For the current frame, IK is an auxiliary variable, where K is the dimension of the feature matrix, F is the discrete Fourier coefficient matrix, M is the size of the search region, and wt is the variable. -1 For the target appearance filter template of frame t-1, μ and μ represent the Lagrange multiplier and the penalty factor, respectively; 2) Combine the wt obtained in step 1) with Perform element-wise dot product to obtain the tracking response map of the current frame; 3) By determining the peak value of the tracking response map of the current frame, update the target appearance filter template of the current frame, including: If the peak value and peak value of the tracking response map of the current frame are greater than their respective thresholds, the target appearance filter template of the current frame calculated by Equation (1) is retained; otherwise, the target appearance filter template calculated by Equation (1) is deleted, and the target appearance filter template of the previous frame is updated to the target appearance filter template of the current frame.

2. The method according to claim 1, characterized in that, The method for solving the cost function of wt is as follows: After separating the variables of the cost function as shown in equation (1), iteratively solving it using the i-th cross-direction multiplier method, we obtain the target appearance filter template after i+1 iterations in the current frame. That is, the target appearance filter template wt for the current frame; The iterative calculation expression is as follows: Where the superscript i represents the i-th iteration, η is the growth rate of each iteration, and μ max This is the upper limit of the penalty factor; The initial values ​​for the iteration of wt are all zero matrices.

3. The method according to claim 1, characterized in that, The step of obtaining the tracking result of the target to be tracked in the current frame includes: The position of the peak value of the tracking response map of the current frame is taken as the target center point position of the current frame, and the target center point position of the current frame is the target tracking result of the current frame.

4. The method according to claim 1, characterized in that, After obtaining the target appearance filter template for the current frame, the method further includes: Determine t: If t-1 is equal to an integer multiple of the set frame interval A and the target appearance filter template of the current frame calculated by formula (1) has been retained at least once in the current interval from frame t-A+1 to frame t, then obtain the long-term memory template of the current interval; otherwise, the current interval does not generate a long-term memory template. The method for obtaining the long-term memory template is as follows: Use the target appearance filter template of the first frame of the video as the first long-term memory template; Starting from the second frame, if a long-term memory template exists in the current interval, the corresponding frame of the retained target appearance filter template obtained according to formula (1) is obtained based on the target appearance filter template retained in the current interval. The target appearance filter template of the frame with the highest peak value of the response map in the corresponding frame is the long-term memory template of the current interval. When the number of long-term memory templates reaches the set upper limit, the oldest long-term memory template will be replaced by a new one.

5. The method according to claim 4, characterized in that, The method further includes: After obtaining the long-term memory template for the current interval, the target appearance filter template for the current frame is updated using all long-term memory templates. The calculation expression is as follows: stβj≥0,j=1,2,...,tp Where wj represents the j-th long-term memory template, the function f() is used to measure the degree of matching between the selected long-term memory and the appearance of the target in the current frame, βj represents the adaptive weight of the j-th long-term memory template, tp is the number of current long-term memory templates, σ is the regularization parameter, and prj is the prior weight of the j-th long-term memory template, expressed as follows: Where α is the attenuation parameter; When a new current frame arrives, the result obtained from equation (3) is used as the target appearance filter template for the updated previous frame.

6. A target tracking device based on multi-template adaptive update that performs the method as described in claim 1, characterized in that, include: The target acquisition module is used to select a target to be tracked from the first frame of the video, and extract the features of the target to be tracked in the first frame to obtain the target appearance filter template of the first frame. The target appearance filter template represents the mapping relationship between the features and corresponding labels between the target to be tracked and the background in the selected search area in the corresponding frame. The target appearance filter template update module is used to sequentially take each frame as the current frame starting from the second frame of the video; using the target appearance filter template of the previous frame, to obtain the target appearance filter template of the current frame and obtain the tracking result of the target to be tracked in the current frame; wherein, the target appearance filter template of the previous frame takes into account the long-term memory template, and the long-term memory template is the best template selected from all the retained target appearance filter templates; The tracking result generation module is used to combine the tracking results of the target to be tracked in each frame to form the tracking result of the target to be tracked in the video.

7. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-5.

Images