Fuzzy detection assisted real-time visual tracking method in complex environment

Abstract

The invention relates to the field of target tracking based on computer vision, and discloses a target tracking method based on computer vision, comprising the following steps: S1, acquiring a source video, and preprocessing the source video to acquire a video frame sequence; s2, constructing a correlation filter, and inputting each input frame in the video frame sequence into the correlation filter in sequence; s3, constructing and training a fuzzy detection system; and S4, respectively adopting a corresponding tracking strategy according to the tracking result of each input frame, determining the real center position of the target in the input frame so as to realize the accurate tracking of the target in each input frame, and setting a memory template by adopting a human vision mechanism system so as to realize the accurate tracking of the target in each input frame. The memory template is updated through the features of the target in the input frame; and S5, circulating the steps S2-S4 until all the video frame sequences are detected. The method provided by the invention has good adaptability and robustness.

Description

Technical Field

[0001] The present invention relates to the field of target tracking based on computer vision, and in particular to a fuzzy detection assisted real-time visual tracking method in a complex environment. Background Art

[0002] The new generation of artificial intelligence has brought about several new research areas, such as computer vision. Therefore, target tracking, which is the basis of computer vision, has always been a hot research area. Algorithms based on correlation filters have become the basis of real-time tracking algorithms due to their high tracking efficiency. The essence of the target tracking method based on correlation filters is to reduce non-target response values ​​and increase target response values, thereby accurately locating the target. However, if the selected best matching position is incorrect, it will cause tracking failure in subsequent frames, thus affecting the performance of the tracking algorithm. During the tracking process, the filter needs to be continuously updated to adapt to the changes in the target's appearance. Each frame updates the filter with the image patch of the current frame. Although the traditional tracking algorithm based on correlation filters has been widely used in current target tracking algorithms, it cannot achieve accurate tracking when the surrounding environment faces challenges such as out-of-view, low resolution or motion blur. Summary of the invention

[0003] In order to provide a fuzzy detection assisted real-time visual tracking method in a complex environment, the present invention constructs a fuzzy detection system and combines it with an algorithm based on a correlation filter to solve the technical problems existing in existing visual tracking algorithms in complex environments.

[0004] The present invention is achieved through the following technical solutions:

[0005] A fuzzy detection assisted real-time visual tracking method in a complex environment, comprising:

[0006] S1, obtaining a source video, and preprocessing the source video to obtain a video frame sequence;

[0007] S2, constructing a correlation filter, and inputting each input frame in the video frame sequence into the correlation filter of the current input frame in sequence, and calculating the response matrix under the target template corresponding to the candidate area of ​​the current input frame and the correlation filter of the current input frame each time the input frame is input;

[0008] S3, constructing and training a fuzzy detection system, and sequentially inputting the response matrix of each input frame into the fuzzy detection system to obtain the reliability of the maximum response value of the response matrix in each input frame, and judging the tracking result of each input frame according to the reliability;

[0009] S4, adopting corresponding tracking strategies according to the tracking results of each input frame, determining the true center position of the tracking target in the input frame, so as to achieve accurate tracking of the tracking target in each input frame, setting a memory template by using the human visual mechanism system, and updating the memory template and the original target template by using the features of the true center position of the tracking target in the input frame;

[0010] S5. Loop S2-S4 until all the video frame sequences are detected.

[0011] As an optimization, in S1, the specific process of preprocessing the source video to obtain a video frame sequence is:

[0012] The source video is divided into a number of video segments at the same time interval, a frame image is randomly sampled in each of the video segments, and the frame images sampled from the several video segments constitute a video frame sequence.

[0013] As an optimization, the specific process of S2 is:

[0014] S2.1, construct and initialize the correlation filter;

[0015] S2.2, in the first input frame, manually mark the tracking target, determine the position and scale of the tracking target, and cyclically sample the tracking target to obtain the image features of the tracking target;

[0016] S2.3, adding a cosine window to the image features for smoothing, and constructing an original target template of the first input frame to train the correlation filter;

[0017] S2.4. In each subsequent input frame, a plurality of candidate samples are generated by circular shifting around the position and proportion of the tracking target of the previous input frame to determine the candidate region of the tracking target of the current input frame, and candidate features are extracted from the candidate region. The candidate features are consistent with the image feature type, and the response values ​​of the final response matrix between the plurality of candidate blocks constituting the candidate region and the original target template determined in the previous input frame are respectively calculated. The maximum position of the response value is the center position of the tracking target of the current input frame, and then the original target template of the previous input frame is updated according to the center position of the tracking target of the current input frame, so as to update the original target template of the current input frame to obtain the correlation filter of the current input frame.

[0018] As an optimization, in S2.4, the specific steps of extracting candidate features from the candidate region and respectively calculating the response values ​​of the final response matrix between several candidate blocks constituting the candidate region and the original target template determined in the previous input frame are:

[0019] S2.4.1. Select a candidate region of the intermediate frame data, and divide the candidate region into a plurality of candidate blocks;

[0020] S2.4.2. Perform similarity calculation on each candidate block in the candidate area corresponding to each input frame and the original target template tracked in the previous input frame, and calculate all response values ​​r between each candidate block and the original target template of the previous input frame. ij , thereby obtaining the initial response matrix R; wherein i represents the candidate block in the i-th row of the candidate area corresponding to the input frame, and j represents the candidate block in the j-th column of the candidate area corresponding to the input frame;

[0021]

[0022] Where m is the number of rows of candidate blocks, and n is the number of columns of candidate blocks;

[0023] S2.4.3, generating an intermediate response matrix R' by zero-filling the initial response matrix to disperse the boundary effect of the initial response matrix R;

[0024] S2.4.4. Reshape the intermediate response matrix through the Boolean matrix to obtain the first Boolean matrix R":

[0025]

[0026] Wherein, R′(i,j) represents the response value of the candidate block in the i-th row and j-th column in the intermediate response matrix R′, and RES′ represents the response value of the adjacent candidate block of the candidate block in the i-th row and j-th column in the intermediate response matrix R′;

[0027] S2.4.5. The second Boolean matrix R1 is obtained by dot multiplication of the first Boolean matrix R" and the intermediate response matrix R':

[0028] R1=R′·R";

[0029] S2.4.6. Assign the maximum response value of the i+1th Boolean matrix to 0 to obtain the matrix R i , and through the matrix R i Get the i-th maximum response value: resmax i =max(R i ); i is a positive integer not less than 1;

[0030] The second Boolean matrix R1 and the matrix R i This is the final response matrix.

[0031] As an optimization, in S2.4, the specific formula for training the correlation filter is:

[0032]

[0033] Among them, H represents the frequency domain representation of the correlation filter, F represents the frequency domain representation of the characteristics of the target area of ​​the input frame; G represents the frequency domain representation of the ideal Gaussian response, is the conjugate complex number of F, λ is the regularization parameter used to prevent overfitting;

[0034] In S2.4, the correlation filter of the original target template updated and determined in the previous input frame is made the correlation filter of the current input frame. In S4, the original target template of the correlation filter of the current input frame is updated by the feature of the true center position of the tracked target in the current input frame, so as to update the original target template of the next input frame to train the correlation filter. The specific formula is:

[0035] H t+1 =(1-η)·H t +η·H new ;

[0036] Among them, η (the value range is 0~1) is the learning rate, which is used to control the update speed, H new is the frequency domain representation of the correlation filter based on the current input frame, H t is the frequency domain representation of the correlation filter of the t-th frame (t-th input frame), H t+1 is the frequency domain representation of the correlation filter of the t+1th frame (t+1th input frame).

[0037] As an optimization, the specific process of building and training the fuzzy detection system is as follows:

[0038] Define positive samples and negative samples;

[0039] determining feature information that can be extracted from the final response matrix;

[0040] Constructing a fuzzy detection system: determining basic properties of the fuzzy detection system, the basic properties including fuzzy set composition operation, implication calculation, output composition operation and defuzzification calculation;

[0041] Determining the membership function of the fuzzy detection system: the membership function of the input variable and the output variable is a triangular membership function;

[0042] Establish fuzzy rules;

[0043] The strategy for optimizing the recognition rate of the fuzzy detection system is as follows: feature information of the final response matrix is ​​correspondingly extracted from positive samples and negative samples, and the feature information is input into the proposed fuzzy detection system for continuous training and testing.

[0044] As an optimization, the definition rules of the positive samples and negative samples are:

[0045] If the target tracked in the previous input frame and the current input frame are both correct, the target tracked in the current input frame is determined as a positive sample; conversely, if the target tracked in the previous input frame is correct but the target tracked in the current input frame is incorrect, the target tracked in the current input frame is determined as a negative sample.

[0046] As an optimization, the feature information includes the maximum number of response values ​​N of the final response matrix mr , large response value ratio R L , the gradient values ​​G in the x and y directions x and G y , the distance between the center point of the tracked target in the previous input frame and the center point of the tracked target in the current input frame D r , variance γ.

[0047] As an optimization, the maximum number of response values ​​N mr The specific formula is:

[0048] N mr =∑ i flag(resmax i )=∑ i flag(max(R i ));

[0049]

[0050] max(R i ) represents the i-th maximum response value in the final response matrix;

[0051] The larger response value is R L The specific formula is:

[0052]

[0053] Among them, N L is the number of maximum response values ​​greater than the threshold τ;

[0054] The gradient values ​​G in the x and y directions x and G y The specific formula is:

[0055]

[0056] x, y represent the row and column positions in the response matrix, r x,y represents the element in the xth row and yth column of the response matrix, r x-1,y 、r x+1,y 、r x,y-1 、r x,y+1They represent the elements of the (x-1)th row and yth column, the elements of the (x+1)th row and yth column, the elements of the xth row and (y-1)th column, and the elements of the xth row and (y+1)th column in the response matrix, respectively. Δ represents the increment in the direction; Δx represents the increment in the row direction, and Δy represents the increment in the column direction; the distance between the center point of the tracked target in the previous input frame and the center point of the tracked target in the current input frame is D r The specific formula is:

[0057]

[0058] ω is the width of the current input frame, h is the height of the current input frame; C is the center point of the tracked target in the current input frame, with coordinates (C x ,C y ) represents; C′ is the center point of the tracked target in the previous input frame, represented by the coordinates (C′ x ,C′ y )express;

[0059]

[0060] The specific formula of variance γ is:

[0061]

[0062] Where number2 represents the maximum number of responses, that is, the number of maximum response values ​​greater than the critical value, R'max represents the maximum response value in the intermediate response matrix R', and R(i) represents the i-th maximum response value in the initial response matrix R.

[0063] As an optimization, the tracking strategy for input frames other than the first input frame includes:

[0064] S4.1. If the tracking result is reliable, jump to S4.4; otherwise, jump to S4.2;

[0065] S4.2, select the memory template corresponding to the previous input frame as the backup template, and calculate the maximum response value of the response matrix of the current input frame and the backup template. If the maximum response value is greater than 0, jump to S4.3, otherwise, jump to S4.4;

[0066] S4.3, using the backup template and selecting the position corresponding to the maximum response value of the final response matrix obtained by calculating the input frame and the backup template as the true center position of the target in the input frame, and then jumping to S4.5;

[0067] S4.4, using the original target template and selecting the position corresponding to the maximum response value of the current input frame as the true center position of the target;

[0068] S4.5 tracks the target according to its true center position, and uses the human visual mechanism to continuously learn and memorize the apparent features of the target at the true center position for a long time, and updates the memory template of the previous input frame in real time through the apparent features of the target at the true center position as the memory template of the current input frame, and then jumps to S4.6;

[0069] S4.6, determine whether the input frame is the last frame in the video frame sequence, if so, end, otherwise, jump back to S2.

[0070] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0071] The present invention proposes a fuzzy detection-assisted real-time visual tracking method in a complex environment. By constructing a fuzzy detection system and combining it with a correlation filter, the proposed fuzzy detection system detects whether the target tracking result is accurate from the final response matrix in advance, and determines whether the current input frame uses a visual memory mechanism. When the fuzzy system detects that the current frame tracking is inaccurate, it will relocate the target according to the target information stored in the memory template to guide subsequent tracking. The results of the implementation show that the proposed method has good adaptability and robustness. When dealing with complex visual challenges (such as illumination changes, background clutter, etc.), the tracking performance greatly verifies the effectiveness of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0073] Figure 1 This is a flow chart of fuzzy detection-assisted real-time visual tracking in a complex environment of the present invention.

[0074] Figure 2 It is a flow chart of the correlation filter tracking algorithm of the present invention.

[0075] Figure 3 This is a tracking result diagram of the correlation filtering-based algorithm of the present invention in three consecutive frames of video. DETAILED DESCRIPTION

[0076] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0077] This embodiment 1 provides a fuzzy detection assisted real-time visual tracking method in a complex environment. Figure 1-Figure 3 As shown, including:

[0078] 1. Video preprocessing to extract frame sequence

[0079] S1, obtaining a source video, and preprocessing the source video to obtain a video frame sequence;

[0080] The specific process is:

[0081] The source video is divided into a number of video segments at the same time interval, a frame image is randomly sampled in each of the video segments, and the frame images sampled from the several video segments constitute a video frame sequence.

[0082] The video clips are sampled only once each, that is, a source video has only one video frame sequence. 2. The correlation filter calculates the final response matrix of the target of each input frame under the current target template (the original target template after the previous frame update), that is,

[0083] S2, constructing a correlation filter, and inputting each input frame in the video frame sequence into the correlation filter of the current input frame in sequence, and calculating the response matrix under the target template corresponding to the candidate area of ​​the current input frame and the correlation filter of the current input frame each time the input frame is input;

[0084] Here, the target template corresponding to the correlation filter of the currently inputted input frame is the updated original target template of the previous frame. The specific updating process is shown in the subsequent content.

[0085] The correlation filter tracking algorithm model was first used in the field of signal processing. Correlation is a measure of the similarity between two signals h and f through correlation operations. Through Fourier transform, the operation of judging the similarity of two signals can be expressed as a dot product operation in the frequency domain. Based on the application of correlation filters in the field of signal processing, they are now applied to the field of object tracking. First, in the initial input frame, the target area is selected, and the correlation filter is initialized by associating the initial target area with a Gaussian distribution with the target center as the peak. Then, in each subsequent frame, features are extracted from the area around the target, including grayscale value, color histogram, HOG, edge features, etc. Feature extraction is to improve the robustness and anti-interference ability of the filter. Next, a cosine window is added to the feature for smoothing. Finally, a reliable and effective correlation filter is constructed to detect the candidate area in the frequency domain. The position with the highest response value will be tracked as the current position of the target. When building a reliable and effective correlation filter, an initial target template is established, and the target position is determined by calculating the similarity between each candidate block in the input frame and the initial target template. The original target template is updated at the position with the highest similarity in the subsequent input frames, thereby achieving continuous tracking. The process is as follows: Figure 2 shown.

[0086] The process of correlation filter is divided into the following steps:

[0087] a. Initialization

[0088] b. Feature extraction

[0089] c. Training filters

[0090] d. Object detection

[0091] e. Filter update

[0092] Combined with step S2 of the present invention, the specific process is:

[0093] S2.1, construct and initialize the correlation filter;

[0094] The above steps are to initialize the correlation filter.

[0095] S2.2, in the first input frame, manually mark the tracking target, determine the position and scale of the tracking target, and cyclically sample the tracking target to obtain the image features of the tracking target;

[0096] This step is to perform feature extraction in the first input frame.

[0097] S2.3, adding a cosine window to the image features for smoothing, and constructing an original target template of the first input frame to train the correlation filter;

[0098] The above steps are feature extraction, which extracts image features from the target area and its surroundings. These features include grayscale images, color histograms, and HOG features. A cosine window is added to the features for smoothing.

[0099] Train the correlation filter:

[0100] The target area (tracking target) and its features in the initial input frame are used to train the correlation filter. The formula is as follows:

[0101]

[0102] H is the frequency domain representation of the correlation filter;

[0103] F is the frequency domain representation of the image features of the tracked target;

[0104] G is the frequency domain representation of the ideal Gaussian response;

[0105] is the complex conjugate of F;

[0106] λ is a regularization parameter used to prevent overfitting;

[0107] Since the number of image features may be small, it is necessary to train a more accurate correlation filter when the number is small. Based on this, the present invention designs a method for optimizing the correlation filter based on the dung beetle algorithm.

[0108] The specific process is:

[0109] A1. Initialize the parameters of the dung beetle algorithm and use the initial regularization parameters of the correlation filter as the initial positions of each dung beetle in the dung beetle population;

[0110] A2, calculating the fitness based on the position of each dung beetle in the dung beetle population in combination with image features, and sorting them according to the size of the fitness, so as to obtain the current optimal position and the current worst position;

[0111] A3, each dung beetle in the dung beetle population performs rolling behavior, dancing behavior, breeding behavior, foraging behavior and stealing behavior according to the fitness ranking to update the position of the dung beetle;

[0112] A4, recalculating the fitness of the dung beetle position after the update based on the image features, and updating the dung beetle population according to the new fitness ranking;

[0113] A5. Determine whether the maximum number of iterations has been reached. If so, output the currently updated dung beetle population to obtain the optimal regularization parameter of the correlation filter. Otherwise, repeat A2 to A5.

[0114] Through the above process, an optimal regularization parameter can be determined to make the correlation filter more accurate.

[0115] The objective function is The fitness function can be specifically P is the sum of the numbers of the candidate blocks corresponding to the first K maximum values ​​of the response matrix after similarity calculation with the candidate area of ​​the first input frame through the correlation filter, and P′ is the sum of the actual numbers of the tracked targets in the candidate blocks in the first input frame, and the tracked targets occupy k candidate blocks.

[0116] In fact, in the first input frame, after manually marking the tracking target, several candidate samples are generated by circular shift around the position and proportion of the tracking target of the input frame to determine the candidate area of ​​the tracking target of the current input frame, and the candidate area is divided into multiple candidate blocks, and the candidate blocks are numbered from left to right and from top to bottom. The sum P' of the candidate blocks where the tracking target is located is first obtained by manual marking, and then the sum of the serial numbers of the candidate blocks corresponding to the first K maximum values ​​of the response matrix obtained by calculating the similarity between the correlation filter and the candidate area of ​​the first input frame is obtained. S2.4. In each subsequent input frame, a number of candidate samples are generated by circular shifting around the position and proportion of the tracking target of the previous input frame to determine the candidate area of ​​the tracking target of the current input frame, and candidate features are extracted from the candidate area. The candidate features are consistent with the image feature type, and the response values ​​of the final response matrix between the several candidate blocks constituting the candidate area and the original target template determined in the previous input frame are calculated respectively. The maximum position of the response value is the center position of the tracking target of the current input frame, and then the original target template of the previous input frame is updated according to the center position of the tracking target of the current input frame, so as to update the original target template of the current input frame to obtain the correlation filter of the current input frame. This step is target detection. In each frame, the current input frame is detected using the trained correlation filter, and the position of the target is determined by calculating the response. The peak position of the response graph corresponds to the center position of the target.

[0117] In S2.4, candidate features are extracted from the candidate region, and the response values ​​of the final response matrix between the candidate blocks constituting the candidate region and the original target template determined in the previous input frame are respectively calculated. The specific steps are:

[0118] S2.4.1, dividing the candidate area into a plurality of candidate blocks;

[0119] Specifically, the candidate region of a certain input frame is a target frame obtained by scaling up the target region of a previous frame of the input frame.

[0120] S2.4.2. Perform similarity calculation on each candidate block in the candidate area corresponding to each input frame and the original target template tracked in the previous input frame, and calculate all response values ​​r between each candidate block and the original target template of the previous input frame. ij , thus obtaining the initial response matrix R.

[0121] That is, in the target tracking process of the current input frame, each candidate block in the given area needs to be compared with the original target template tracked in the previous frame, and all response values ​​r between each sample and the original target template are calculated. ij, thereby combining the initial response matrix R in Formula 3. Wherein, i represents the candidate block in the i-th row in the candidate area corresponding to the input frame, and j represents the candidate block in the j-th column in the candidate area corresponding to the input frame.

[0122]

[0123] Where m is the number of rows of candidate blocks, and n is the number of columns of candidate blocks;

[0124] S2.5.3, generating an intermediate response matrix R' by zero-filling the initial response matrix to disperse the boundary effect of the initial response matrix R;

[0125] By dispersing the boundary effect of the response matrix, the intermediate response matrix R' is finally generated. Generally speaking, the response matrix may have multiple maximum response values. Because the presence of multiple extreme points is likely to interfere with the determination of the true position of the target due to the regional maximum, the regional maximum is used as the dispersion matrix.

[0126] For a response matrix with multiple maximum peaks, a Boolean matrix is ​​used to reshape the response matrix. The maximum value in the matrix R' is defined as 1, and the other elements are defined as 0, thereby obtaining the first Boolean matrix R" in formula 4, that is, S2.5.4, reshaping the intermediate response matrix by a Boolean matrix to obtain the first Boolean matrix R":

[0127]

[0128] Wherein, R′(i,j) represents the response value of the candidate block in the i-th row and j-th column in the intermediate response matrix R′, and RES′ represents the response value of the adjacent candidate block of the candidate block in the i-th row and j-th column in the intermediate response matrix R′;

[0129] S2.5.5. The second Boolean matrix R1 is obtained by dot multiplication of the first Boolean matrix R" and the intermediate response matrix R':

[0130] R1=R′·R" (5);

[0131] S2.5.6. Assign the maximum response value of the i+1th Boolean matrix to 0 in turn to obtain the matrix R i , and through the matrix R i Get the i-th maximum response value: resmax i =max(R i ); i is a positive integer not less than 1;

[0132] The second Boolean matrix R1 and the matrix R i This is the final response matrix.

[0133] Then, the second Boolean matrix R1 is calculated according to Formula 5 to obtain the maximum response value resmax1=max(R1). max(R1) represents the first maximum response value in the second Boolean matrix R1.

[0134] Thus, the i-th maximum response value resmax i =max(R i ) by the matrix R i It turns out that the matrix R i is through R i-1 Generated by assigning the maximum response value to 0.

[0135] In order to adapt to the changes in the object appearance, the correlation filter needs to be updated in each frame.

[0136] In actual target tracking scenarios, the target may change greatly due to various reasons. In order to prevent the target from drifting and eventually causing target tracking failure, the correlation filter is updated in real time during the tracking process. Therefore, the entire target template update step is to determine the real position of the target in the current input frame and then determine the new target position, which can make the update of the correlation filter more accurate. At the same time, the extracted features are used to update the target appearance model and classifier coefficients.

[0137] S3, constructing and training a fuzzy detection system, and sequentially inputting the response matrix of each input frame into the fuzzy detection system to obtain the reliability of the maximum response value of the response matrix in each input frame, and judging the tracking result of each input frame according to the reliability;

[0138] Since the traditional tracking algorithm based on correlation filter has the problem of target template contamination, a fuzzy detection system is proposed in target tracking based on correlation filter to improve the shortcomings of the traditional target tracking algorithm.

[0139] A fuzzy detection system is constructed to detect the reliability of the response matrix. The reliability of the maximum response value is determined using the trained fuzzy detection system, and the current tracking result is judged based on the obtained reliable information.

[0140] The output result of the fuzzy detection system is directly 1 or 0, for example, 1 means reliable and 0 means unreliable.

[0141] The specific process of building and training a fuzzy detection system is as follows:

[0142] Define positive samples and negative samples;

[0143] The rules for positive and negative samples are: if the target tracked in the previous input frame and the current input frame are both correct, the target tracked in the current input frame is determined as a positive sample; conversely, if the target tracked in the previous input frame is correct, but the target tracked in the current input frame is incorrect, the target tracked in the current input frame is determined as a negative sample. (If the previous one is incorrect and the latter one is correct, then the latter one is a positive sample and is discarded in this case).

[0144] Whether the target is correct can be determined through manual marking.

[0145] Figure 3 The tracking results of the algorithm based on correlation filtering in three consecutive frames of video are shown. th The position of the manually marked frame is basically consistent with the target position actually tracked by the algorithm based on correlation filtering, indicating that the tracking effect is good. th The same is true for frames. However, 3 th The position of the manual annotation in the frame deviates greatly from the position actually tracked by the correlation filter-based algorithm, indicating that the tracking effect is not good. th Frame and 2 th The tracking of the frames is correct, so 2 th The frame is defined as a positive sample. Then, due to 3 th The frames are not tracked correctly, so they are defined as negative samples.

[0146] determining feature information that can be extracted from the final response matrix;

[0147] Method for selecting eigenvalues: The fuzzy system identifies positive and negative samples through the response matrix. According to the difference between positive and negative samples, each feature is defined and selected, and the parameters in the proposed fuzzy system are determined, including the maximum response value, the ratio of the larger response value, the gradient value in the X and Y directions, the ratio of the maximum response value to the center of the target frame, and the variance.

[0148] The feature information extracted from positive samples and negative samples and input into the fuzzy detection system is the same. By distinguishing positive and negative samples and extracting the input of positive samples, from "correct tracking" to "correct tracking", the proposed fuzzy system can learn the features of good tracking; through the input of negative samples, from "good tracking" to "bad tracking", the fuzzy system can accurately learn the failed tracking results, which makes the learning of the fuzzy system more complete and accurate.

[0149] Maximum number of response values ​​N mr :

[0150] According to the correlation filter-based algorithm, the determination of the target position is closely related to the peak value, so the number of maximum response values ​​N is calculated in Formula 6 mr, as the feature of the target in the fuzzy detection system.

[0151] The maximum number of response values ​​N mr The specific formula is:

[0152] N mr =∑ i flag(resmax i )=∑ i flag(max(R i )) (6);

[0153]

[0154] max(R i ) represents the i-th maximum response value in the final response matrix;

[0155] Maximum response value ratio R L :

[0156] The number of maximum values ​​in the response matrix is ​​not unique. If most of the maximum values ​​are small, the response matrix is ​​likely to cause "poor tracking". Therefore, it is necessary to set a threshold τ to measure the proportion of large response values. Since the maximum value greater than τ is counted as a large response value N L , so the maximum response value ratio is calculated by Formula 8.

[0157] The larger response value is R L The specific formula is:

[0158]

[0159] Among them, N L is the number of maximum response values ​​greater than the threshold τ;

[0160] The reason why the ratio of larger response values ​​(larger response value than R L ) is used as another input feature of the fuzzy detection system because the relatively large response value is very helpful in distinguishing positive and negative samples. The gradient value G in the x and y directions x ,G y The gradient values ​​in the x and y directions can usually be used to reflect the information difference between the maximum response value position and the surrounding positions. The gradient value represents the steepness of the maximum response value. The larger the gradient value, the steeper the maximum response position. The gradient values ​​in the x and y directions are defined by formulas 9-10.

[0161]

[0162] x, y represent the row and column positions in the response matrix, r x,y represents the element in the xth row and yth column of the response matrix, r x-1,y 、rx+1,y 、r x,y-1 、r x,y+1 They represent the elements of the (x-1)th row and yth column, the elements of the (x+1)th row and yth column, the elements of the xth row and (y-1)th column, and the elements of the xth row and (y+1)th column in the response matrix, respectively. Δ represents the increment in the direction; Δx represents the increment in the row direction, and Δy represents the increment in the column direction; the distance between the center point of the tracked target in the previous input frame and the center point of the tracked target in the current input frame is D r The specific formula is:

[0163]

[0164] ω is the width of the current input frame, h is the height of the current input frame; C is the center point of the tracked target in the current input frame, with coordinates (C x ,C y ) represents; C′ is the center point of the tracked target in the previous input frame, represented by the coordinates (C′ x ,C′ y )express;

[0165]

[0166] After cyclically moving around the actual position of the target in the previous frame, a large number of samples are generated. These samples constitute a target search area, and the distance from each sample to the center of the frame is different. It is not logical if the highest point in the candidate sample has the same distance as the highest point in the previous frame. The similarity is simply used as the best position, ignoring the distance between targets in consecutive frames. Therefore, in Equation 11, (C x ,C y ) is the center point of the tracked target in the current frame, (C′ x ,C′ y ) is the center point of the tracked object in the previous frame, ω is the width of the frame, and h is the height of the frame.

[0167] The specific formula of variance γ is:

[0168]

[0169] Where number2 represents the maximum number of responses, that is, the number of maximum response values ​​greater than the critical value, R'max represents the maximum response value in the intermediate response matrix R', and R(i) represents the i-th maximum response value in the initial response matrix R.

[0170] The variance represents the mean square error value, i.e., the difference between the maximum values ​​greater than the critical value. The difference between each maximum value is represented by the sum of the squares of the differences between the response values ​​greater than the given threshold val. According to Equation 12, the variance γ is also an important input feature in the proposed fuzzy system.

[0171] Finally, the normalized features extracted from positive and negative samples are input into the proposed blur detection system for continuous training and testing.

[0172] Constructing a fuzzy detection system: determining basic properties of the fuzzy detection system, the basic properties including fuzzy set composition operation, implication calculation, output composition operation and defuzzification calculation;

[0173] In the set synthesis operation of the Mamdani type fuzzy system constructed by this method, the basic attribute "and" link uses the "minimum" method, and the "or" uses the "maximum" method. The application calculation uses the "minimum" attribute, and the composite operation (aggregation) of the output uses the "and" method. Fuzzification uses the centroid method.

[0174] The definition of fuzzy sets is as follows:

[0175] Given a domain U, any mapping μA from U to a closed interval [0,1], μA:U→[0,1],u→μA(u) is called a fuzzy subset A of U, and μA is called the membership function of the fuzzy subset. In addition, μA(u) is called the membership degree of u in A, which reflects the membership degree of u′ to the fuzzy subset A. Fuzzy subsets are also called fuzzy sets.

[0176] Determining the membership function of the fuzzy detection system: the membership function of the input variable and the output variable is a triangular membership function;

[0177] Create fuzzy rules:

[0178] Fuzzy control rules are actually a set of fuzzy conditions. Fuzzy control rules are summarized based on the operator's control experience. The principle of determining fuzzy control rules is to ensure that the output of the controller can guarantee the best dynamic and static characteristics of the system output. Fuzzy rules to constrain output variables include state evaluation fuzzy control rules and target evaluation fuzzy control rules. The forms of fuzzy control rules are divided into the following two types:

[0179] The main fuzzy control rules for state assessment are as follows:

[0180] If x1 is A i1 and x2 is A i2 ... and x n Yes A in Then y is C i ;

[0181] Among them, x1, x2, ... x n and y are linguistic variables or fuzzy variables, representing the state variables and control variables of the system. i1 ,A i2 ,……A in and Ci is a linguistic value representing a fuzzy set in the domain.

[0182] The target evaluation fuzzy control rules are as follows:

[0183] If (U is C i →(x is A1, y is B1)), then U is C i .

[0184] The strategy for optimizing the recognition rate of the fuzzy detection system is as follows: feature information of a continuous normalized response matrix (final response matrix) is correspondingly extracted from positive samples and negative samples, and the feature information is input into the proposed fuzzy detection system for continuous training and testing.

[0185] S4, adopting corresponding tracking strategies according to the tracking results of each input frame, determining the true center position of the tracking target in the input frame, so as to achieve accurate tracking of the tracking target in each input frame, setting a memory template by using the human visual mechanism system, and updating the memory template and the original target template by using the features of the true center position of the tracking target in the current input frame;

[0186] In S4, the correlation filter of the original target template updated and determined in the previous input frame is made the correlation filter of the current input frame. In S4, the original target template of the correlation filter of the current input frame is updated by the feature of the true center position of the tracked target in the current input frame, so as to update the original target template of the next input frame to train the correlation filter. The specific formula is:

[0187] H t+1 =(1-η)·H t +η·H new (2)

[0188] Among them, η is the learning rate, η∈(0,1), which is used to control the update speed, H new is the correlation filter calculated based on the updated original target template of the current input frame, H t is the frequency domain representation of the correlation filter of the tth frame, H t+1 is the frequency domain representation of the correlation filter of the t+1th frame;

[0189]

[0190] F t The frequency domain representation of the features representing the center position of the target region in the t-th input frame, Yes F t The complex conjugate of

[0191]

[0192] F new The frequency domain representation of the features of the center position of the target area in the current input frame, Yes F new For example, for the second frame, let t be 2, and obtain the updated correlation filter H2 of the first frame after the first frame is updated, and then perform similarity calculation with the candidate block of the second frame through the correlation filter H2, and calculate the correlation filter H based on the true center position of the tracking target determined after the set tracking strategy. new , then, the correlation filter to be used in the third frame for similarity calculation with the candidate block of the second frame is: H3 = (1-η)·H2 + η·H new .

[0193] The tracking strategy for input frames other than the first input frame includes:

[0194] S4.1. If the tracking result is reliable, jump to S4.4; otherwise, jump to S4.2;

[0195] S4.2, select the memory template corresponding to the previous input frame as the backup template, and calculate the maximum response value of the response matrix of the current input frame and the backup template. If the maximum response value is greater than 0, jump to S4.3, otherwise, jump to S4.4;

[0196] S4.3, using the backup template and selecting the position corresponding to the maximum response value of the final response matrix obtained by calculating the input frame and the backup template as the true center position of the target in the input frame, and then jumping to S4.5;

[0197] S4.4, using the original target template and selecting the position corresponding to the maximum response value of the current input frame as the true center position of the target;

[0198] S4.5 tracks the target according to its true center position, and uses the human visual mechanism to continuously learn and memorize the apparent features of the target at the true center position for a long time, and updates the template used by the target at the true center position as a new target template in real time, and then jumps to S4.6;

[0199] S4.6, determine whether the input frame is the last frame in the video frame sequence, if so, end, otherwise, jump back to S2.

[0200] This step is tracking processing, summarizing the above rules, specifically:

[0201] (1) If the reliability of the maximum response value is determined to be reliable, the original template is used and the position corresponding to the maximum response value of the current input frame is selected as the true center position of the target.

[0202] (2) If the reliability of the maximum response value is determined to be unreliable, the memory template of the previous frame is selected as the backup template and the maximum response value of the response matrix of the current input frame and the backup template is calculated.

[0203] (3) If the maximum response value obtained in the previous step is greater than 0, use the backup template and select the position corresponding to the maximum response value of the backup template as the true center position of the target; otherwise, use the original template and select the position corresponding to the maximum response value of the current frame as the true center position of the target.

[0204] (4) After determining the center position of the target, filter and memorize the appearance features of the current target such as color, HOG, edge features, etc., and store the relevant information (appearance features such as color, HOG, edge features, etc.) in the feature information database (it can be understood that the feature information database is the feature data storage part of the memory template), continuously update the features to improve the database, and finally achieve long-term memory of the target features. When target tracking fails, query the target features stored in the information database, analyze and determine the target position, and relocate the target.

[0205] For the first input frame, the memory template is the same as the original target template, that is, the present invention has two templates, one is a memory template specially trained for long-term memory, and the other is a target template corresponding to the filter.

[0206] S5, looping S2-S4 until all the video frame sequences are detected, thereby realizing the blur detection-assisted real-time visual tracking function in a complex environment.

[0207] It should be noted that one input frame is performed in steps of S2-S4, and multiple input frames are performed in time sequence of S2-S4, that is, when the previous input frame is performing S3, the next input frame can start to perform S2.

[0208] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A fuzzy detection assisted real-time visual tracking method in a complex environment, characterized in that: include: S1, obtaining a source video, and preprocessing the source video to obtain a video frame sequence; S2, constructing a correlation filter, and sequentially inputting each input frame in the video frame sequence into the correlation filter of the current input frame, and calculating a response matrix under the original target template corresponding to the candidate area of ​​the current input frame and the correlation filter of the current input frame each time the input frame is input; S3, constructing and training a fuzzy detection system, and sequentially inputting the response matrix of each input frame into the fuzzy detection system to obtain the reliability of the maximum response value of the response matrix in each input frame, and judging the tracking result of each input frame according to the reliability; S4, adopting corresponding tracking strategies according to the tracking results of each input frame, determining the true center position of the tracking target in the input frame, so as to achieve accurate tracking of the tracking target in each input frame, setting a memory template by using the human visual mechanism system, and updating the memory template and the original target template by using the features of the true center position of the tracking target in the input frame; S5. Loop S2-S4 until all the video frame sequences are detected.

2. The method for real-time visual tracking assisted by blur detection in complex environments according to claim 1, characterized in that: In S1, the specific process of preprocessing the source video to obtain a video frame sequence is as follows: The source video is divided into a number of video segments at the same time interval, a frame image is randomly sampled in each of the video segments, and the frame images sampled from the several video segments constitute a video frame sequence.

3. The method for real-time visual tracking assisted by blur detection in complex environments according to claim 1, characterized in that: The specific process of S2 is: S2.1, construct and initialize the correlation filter; S2.2, in the first input frame, manually mark the tracking target, determine the position and scale of the tracking target, and cyclically sample the tracking target to obtain the image features of the tracking target; S2.3, adding a cosine window to the image features for smoothing, and constructing an original target template of the first input frame to train the correlation filter; S2.

4. In each subsequent input frame, a number of candidate samples are generated using circular shift around the position and proportion of the tracking target of the previous input frame to determine the candidate area of ​​the tracking target of the current input frame, and candidate features are extracted from the candidate area. The candidate features are consistent with the image feature type, and the response values ​​of the final response matrix between the several candidate blocks constituting the candidate area and the original target template determined after updating in the previous input frame are calculated respectively. The position with the maximum response value is the center position of the tracking target of the current input frame.

4. The method for real-time visual tracking assisted by blur detection in complex environments according to claim 3 is characterized in that: In S2.4, candidate features are extracted from the candidate region. The specific steps of respectively calculating the response values ​​of the final response matrix between several candidate blocks constituting the candidate area and the original target template determined in the previous input frame are: S2.4.1, dividing the candidate area into a plurality of candidate blocks; S2.4.

2. Perform similarity calculation on each candidate block in the candidate area corresponding to each input frame and the original target template tracked in the previous input frame, and calculate all response values ​​r between each candidate block and the original target template of the previous input frame. ij , thereby obtaining the initial response matrix R; wherein i represents the candidate block in the i-th row of the candidate area corresponding to the input frame, and j represents the candidate block in the j-th column of the candidate area corresponding to the input frame; Where m is the number of rows of candidate blocks, and n is the number of columns of candidate blocks; S2.4.3, generating an intermediate response matrix R' by zero-filling the initial response matrix to disperse the boundary effect of the initial response matrix R; S2.4.

4. Reshape the intermediate response matrix through the Boolean matrix to obtain the first Boolean matrix R": Wherein, R′(i,j) represents the response value of the candidate block in the i-th row and j-th column in the intermediate response matrix R′, and RES′ represents the response value of the adjacent candidate block of the candidate block in the i-th row and j-th column in the intermediate response matrix R′; S2.4.

5. The second Boolean matrix R1 is obtained by dot multiplication of the first Boolean matrix R" and the intermediate response matrix R': R1=R′·R"; S2.4.

6. Assign the maximum response value of the i+1th Boolean matrix to 0 to obtain the matrix R i , and through the matrix R i Get the i-th maximum response value: resmax i =max(R i ); i is a positive integer not less than 1; the second Boolean matrix R1 and the matrix R i This is the final response matrix.

5. The method for real-time visual tracking assisted by blur detection in complex environments according to claim 4, characterized in that: The specific formula for training the correlation filter is: Among them, H represents the frequency domain representation of the correlation filter, F represents the frequency domain representation of the center position of the tracked target in the input frame; G represents the frequency domain representation of the ideal Gaussian response, is the conjugate complex number of F, λ is the regularization parameter used to prevent overfitting; The correlation filter of the original target template updated and determined in the previous input frame is made the correlation filter of the current input frame. In S4, the original target template of the correlation filter of the current input frame is updated by the feature of the true center position of the tracked target in the current input frame, so as to update the original target template of the next input frame to train the correlation filter. The specific formula is: H t+1 =(1-h)·H t +η·H new ; Among them, η is the learning rate, which is used to control the update speed, H new is the correlation filter calculated based on the original target template updated from the current input frame, H t is the frequency domain representation of the correlation filter of the tth frame, H t+1 is the frequency domain representation of the correlation filter of the t+1th frame.

6. The method for real-time visual tracking assisted by blur detection in complex environments according to claim 1, characterized in that: The specific process of building and training a fuzzy detection system is as follows: Define positive samples and negative samples; determining feature information that can be extracted from the final response matrix; Constructing a fuzzy detection system: determining basic properties of the fuzzy detection system, the basic properties including fuzzy set composition operation, implication calculation, output composition operation and defuzzification calculation; Determining the membership function of the fuzzy detection system: the membership function of the input variable and the output variable is a triangular membership function; Establish fuzzy rules; The strategy for optimizing the recognition rate of the fuzzy detection system is as follows: feature information of the final response matrix is ​​correspondingly extracted from positive samples and negative samples, and the feature information is input into the proposed fuzzy detection system for continuous training and testing.

7. The method for real-time visual tracking assisted by blur detection in complex environments according to claim 6, characterized in that: The definition rules of the positive samples and negative samples are: If the targets tracked in the previous input frame and the current input frame are both correct, the target tracked in the current input frame is determined as a positive sample; On the contrary, if the target tracked in the previous input frame is correct, but the target tracked in the current input frame is incorrect, the target tracked in the current input frame is determined as a negative sample.

8. The method for real-time visual tracking assisted by blur detection in complex environments according to claim 6, characterized in that: The characteristic information includes the maximum number N of response values ​​of the final response matrix mr , large response value ratio R L , the gradient values ​​G in the x and y directions x and G y , the distance between the center point of the tracked target in the previous input frame and the center point of the tracked target in the current input frame D r , variance γ.

9. The method for real-time visual tracking assisted by blur detection in complex environments according to claim 8, characterized in that: The maximum number of response values ​​N mr The specific formula is: N mr =∑ i flag(resmax i )=∑ i flag(max(R i )); max(R1) represents the i-th maximum response value in the final response matrix, where i is a positive integer not less than 1; The larger response value is R L The specific formula is: Among them, N L is the number of maximum response values ​​greater than the threshold τ; The gradient values ​​G in the x and y directions x and G y The specific formula is: x, y represent the row and column positions in the response matrix, r x,y represents the element in the xth row and yth column of the response matrix, r x-1,y 、r x+1,y 、r x,y-1 、r x,y+1 Respectively represent the element of the (x-1)th row and yth column, the element of the (x+1)th row and yth column, the element of the xth row and (y-1)th column, and the element of the xth row and (y+1)th column in the response matrix, Δ represents the increment in the direction; Δx represents the increment in the row direction, and Δy represents the increment in the column direction; The distance between the center point of the tracked target in the previous input frame and the center point of the tracked target in the current input frame is D r The specific formula is: ω is the width of the current input frame, h is the height of the current input frame; C is the center point of the tracked target in the current input frame, with coordinates (C x ,C y ) represents; C′ is the center point of the tracked target in the previous input frame, represented by the coordinates (C′ x ,C′ y )express; The specific formula of variance γ is: Where number2 represents the maximum number of responses, that is, the number of maximum response values ​​greater than the critical value, R'max represents the maximum response value in the intermediate response matrix R', and R(i) represents the i-th maximum response value in the initial response matrix R.

10. The method for real-time visual tracking assisted by blur detection in complex environments according to claim 1, characterized in that: The tracking strategy for input frames other than the first input frame includes: S4.

1. If the tracking result is reliable, jump to S4.4; otherwise, jump to S4.2; S4.2, select the memory template corresponding to the previous input frame as the backup template, and calculate the maximum response value of the response matrix of the current input frame and the backup template. If the maximum response value is greater than 0, jump to S4.3, otherwise, jump to S4.4; S4.3, using the backup template and selecting the position corresponding to the maximum response value of the final response matrix obtained by calculating the input frame and the backup template as the true center position of the target in the input frame, and then jumping to S4.5; S4.4, using the original target template and selecting the position corresponding to the maximum response value of the current input frame as the true center position of the target; S4.5 tracks the target according to its true center position, and uses the human visual mechanism to continuously learn and memorize the apparent features of the target at the true center position for a long time, and updates the memory template of the previous input frame in real time through the apparent features of the target at the true center position as the memory template of the current input frame, and then jumps to S4.6; S4.6, determine whether the input frame is the last frame in the video frame sequence, if so, end, otherwise, jump back to S2.

Images