A multi-scale space semantic perception based occlusion suppression tracking method

Abstract

The present application relates to a kind of based on multi-scale space semantic perception's occlusion suppression tracking method.The algorithm includes: in initial frame, select multi-scale image template feed to convolutional neural network CNN generation filter, and around the tracking target intercept 3 groups of scale image block set for generating transformation matrix;In subsequent tracking process, 3 groups of different scale search area are fed into convolutional neural network CNN to obtain feature map, combined with transformation matrix to generate mask matrix, further calculate to obtain the feature map of occlusion suppression, 3 groups of scale feature map of occlusion suppression are fused to respond the position of maximum as tracking target position, and with peak sidelobe fluctuation as update transformation matrix and filter index.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer image processing and target tracking, and particularly relates to a method for suppressing occlusion tracking based on multi-scale spatial semantic perception. BACKGROUND

[0002] Target tracking is an important research direction in the field of computer vision. Target tracking technology is one of the important basic problems in the field of computer vision. Its main task is to continuously locate the target in subsequent frames after the location of the target in the first frame is given.

[0003] At present, target tracking technology has very wide application in civil and military aspects. In civil aspect, target tracking technology includes intelligent video monitoring, intelligent transportation system, unmanned driving, human-computer interaction, etc. These applications can improve public safety and life quality, and also need to ensure the reliability and safety of technology. In military aspect, target tracking technology includes unmanned aerial vehicle, precision guidance, air early warning, battlefield surveillance, etc. These applications have important significance for improving combat effectiveness and protecting military safety.

[0004] Although great progress has been made in target tracking technology, there are still many challenges and problems in practical application. For example, target tracking technology faces challenges such as complex scene, illumination change, posture change, occlusion, etc.

[0005] In recent years, target tracking methods based on twin networks have attracted widespread attention due to their excellent tracking effect. Twin network is a network structure composed of two convolutional neural networks with the same structure and parameters in parallel. Due to the ability of convolution kernel in convolutional neural network to capture the local invariance feature of two-dimensional data, the target tracking algorithm based on twin network can well overcome the challenges of translation, rotation and scale transformation. However, due to the fact that the template cannot be updated online in the twin network, the target information and the occlusion cannot be effectively distinguished, and the tracking drift often occurs due to the sensitivity of occlusion interference in the tracking process, resulting in tracking failure. SUMMARY

[0006] The technical problem solved by the application is that the target tracking process may encounter the problem of tracking failure caused by target occlusion. The method based on multi-scale spatial semantic perception is adopted to enhance the tracking robustness of the tracker under the condition of target occlusion, improve the tracking accuracy, and effectively solve the problem of partial target occlusion.

[0007] The technical solution for achieving the purpose of the application is a method for suppressing occlusion tracking based on multi-scale spatial semantic perception, comprising the following steps:

[0008] Step 1, selecting multi-scale image templates. In the T-th frame, the target image is cropped as the original image template with a KxK square frame, denoted as z0, and the large-scale and small-scale image templates are cropped with 2Kx2K and K / 2xK / 2 square frames centered at z0, denoted as z + and z - respectively. z0, z + and z - are input into the feature extraction network to obtain filters and where 1≤T≤N, N is the total number of frames of the video sequence.

[0009] Step 2, 3 sets of scale image block sets P0, P + and P - are cropped around the original image template. Specifically, n image blocks containing objects are randomly collected around z0, z + and z - with KxK, 2Kx2K and K / 2xK / 2 square frames respectively, to obtain 3 sets of scale image block sets P0, P + and P - .

[0010] P0, P + and P - are input into the CNN network to generate feature maps, and principal component analysis (PCA) is performed to obtain transformation matrices ε0, ε + and ε - for the 3 sets of scale image blocks. The CNN network is a feature extraction network trained using the data sets Youtube-BB and ImageNet

[0011] Step 3, input the 3 scale search regions into the CNN network to obtain feature maps and Specifically, according to the position of the T-th frame, the search region of the T+1-th frame is determined and 3 scale search region images s0, s + and s - are collected. In order to use the convolution method for target positioning in the subsequent steps, the 3 scale search regions collected in the T-th frame should be larger than and contain the target template in the T-1-th frame.

[0012] Further, the 3 scale search region images s0, s + and s - are input into the feature extraction network to obtain feature maps and

[0013] Step 4, the feature maps and and transform matrix ε0, ε + and ε - , to obtain spatial semantic graph U0, U + and U - , and calculate binary mask C0, C + and C - .

[0014] Step 5, convolve feature map and and filter and to obtain response map h(s0, z0), h(s + , z + ) and h(s - , z - ).

[0015] Step 6, point product of response map h(s0, z0), h(s + , z + ) and h(s - , z - ) and binary mask C0, C + and C - respectively to obtain 3 occlusion suppression peak value maps H0, H + and H - .

[0016] Step 7, fuse occlusion suppression peak value maps H0, H + and H - by point product operation to obtain fused peak value map H.

[0017] Step 8, update the position of the maximum response value of fused peak value map H as the position coordinate of the Tth frame, and calculate the peak sidelobe ratio variance σ 2 T-1 and the peak sidelobe ratio variance σ 2 T of the T-1th frame and before. If σ 2 T is lower than σ 2 T-1 , or the peak sidelobe ratio of the Tth frame is higher than the average value of the T-1th frame and before, then update transform matrix ε0, ε + and ε - , filter and by repeating steps 1, 2, otherwise do not update. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of a multi-scale spatial semantic perception-based occlusion suppression tracking method of the present application.

[0019] Figure 2 Figure is a schematic diagram of the relationship among image blocks, search regions and image templates in the occlusion suppression tracking method based on multi-scale spatial semantic perception according to the present application.

[0020] Figure 3 Figure is a precision diagram of the occlusion suppression tracking method based on multi-scale spatial semantic perception according to the present application and other 13 methods on OTB2015 tracking dataset.

[0021] Figure 4 Figure is a success diagram of the occlusion suppression tracking method based on multi-scale spatial semantic perception according to the present application and other 13 methods on OTB2015 tracking dataset.

[0022] Figure 5 Figure is a visualization comparison of the occlusion suppression tracking method based on multi-scale spatial semantic perception according to the present application and other 3 methods in Jogging video sequence. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described below in combination with the drawings.

[0024] As Figure 1 described, the present application proposes an occlusion suppression tracking method based on multi-scale spatial semantic perception, which comprises the following steps:

[0025] Step 1, selecting multi-scale image templates. In the Tth frame, a 64x64 square frame is used to cut the target image as an original image template, denoted as z0, and a 128x128 and 32x32 square frame is used to cut large-scale and small-scale image templates with z0 as the center, denoted as z + and z - . z0, z + and z - are input into a feature extraction network to obtain filters and wherein 1

[0026] Specifically, a CNN network trained by using dataset Youtube-BB and ImageNet is used as the feature extraction network

[0027] Table 1 is a parameter table of the CNN network

[0028]

[0029]

[0030] The CNN network comprises 4 convolutional layers and 3 pooling layers connected in sequence, and a maximum pooling layer is connected after the first 3 convolutional layers, and there are Dropout random inactivation layers and RELU nonlinear activation functions between them.

[0031] Step 2, 3 sets of scale image block sets P0, P + and P - are cut around the original image template as the center. Specifically, 64x64, 128x128 and 32x32 square frames are used to randomly collect n scale image blocks containing objects around z0, z + and z - , to obtain 3 sets of scale image block sets P0, P + and P - .

[0032] The 3 sets of image block sets P0, P + and P - are input into the CNN network to generate feature maps, and principal component analysis PCA is performed to obtain 3 sets of scale image block transformation matrices ε0, ε + and ε - . Specifically:

[0033] The 3 sets of image block sets P0, P + and P - are input into the feature extraction network to obtain feature map sets and The size of each feature map is 30x30x128.

[0034] Further, the feature maps in each set are connected into a two-dimensional matrix M0, M + and M - , and the size of each two-dimensional matrix is 900x128.

[0035] Further, principal component analysis PCA is applied to the two-dimensional matrices M0, M + and M - to form transformation matrices ε0, ε + and ε - from the main components.

[0036] The principal component analysis PCA is specifically operated as follows: the two-dimensional matrix is centered, i.e. each element of the matrix is subtracted by the average value of all elements of the matrix; the covariance matrix is calculated by the formula , where is the centered two-dimensional matrix, (·) T is the matrix transpose operation, and n is the matrix The number of all elements; the eigenvalue decomposition method is used to solve the covariance matrix The eigenvalues and eigenvectors of the covariance matrix; sort the eigenvalues from large to small, select the largest k, and the corresponding k eigenvectors are respectively used as row vectors to form the eigenvector matrix P; the transformation matrix is obtained by the formula ε=PM.

[0037] Step 3: input the three scale search regions into the CNN network to obtain feature maps And Specifically, according to the position of the Tth frame, the search region of the T+1th frame is determined, and three scale search region images s0, s + And s - The size of the three scale search regions collected in the Tth frame is 256x256, and the three scale search regions contain the target template in the T-1th frame, Figure 2 The relationship between the image block, the search region and the image template in the method is shown.

[0038] Further, the three scale search region images s0, s + And s - are respectively input into the feature extraction network To obtain feature maps And

[0039] Step 4: obtain spatial semantic maps U0, U + And U - by the feature maps And and the transformation matrix ε0, ε + And ε - The formula is as follows:

[0040]

[0041]

[0042]

[0043] Wherein, And are the i-th row and j-th column elements of And , respectively, And are the mean values of And , respectively, And are the i-th row and j-th column elements of U0, U + And U - ;

[0044] Furthermore, calculate the binary mask C0, C + and C - The element in its i-th row and j-th column and All satisfy the following formula,

[0045]

[0046] Step 5, transfer the feature map and and filter and Convolution yields response maps h(s0,z0) and h(s... + ,z + ) and h(s - ,z - The formula is:

[0047]

[0048]

[0049]

[0050] in, This represents the convolution operation.

[0051] Step 6, plot the response graphs h(s0,z0) and h(s... + ,z + ) and h(s - ,z - ) respectively with binary masks C0, C + and C - The dot product yields three occlusion suppression peak maps H0, H1, and H2. + and H - The formula is as follows:

[0052] H0=h(s0,z0)⊙C0 (11)

[0053] H0=h(s0,z0)⊙C0 (12)

[0054] H0=h(s0,z0)⊙C0 (13)

[0055] Here, ⊙ represents the dot product operation.

[0056] Step 7, extract the occlusion suppression peak maps H0 and H... + and H - The fused peak image H is obtained by fusing the components using dot product operations, as shown in the following formula:

[0057] H=H0⊙H + ⊙H -(14)

[0058] Step 8, update the position of the maximum response value of fusion peak map H to the position coordinate of the Tth frame, and judge whether to update the transformation matrix and filter according to the fluctuation of the peak sidelobe. Specifically,

[0059] Calculate the peak sidelobe ratio, the formula is as follows,

[0060]

[0061] Where, h max is the maximum value of the response map, μ and σ represent the standard deviation and average value of the sidelobe around the peak.

[0062] Further, calculate the peak sidelobe ratio variance σ 2 T-1 and the peak sidelobe ratio variance σ 2 T , which can be expressed by the following formula,

[0063]

[0064]

[0065] σ 2 T = (PSR T - μ T-1 ) 2 (18)

[0066] Where, T represents the number of the current frame, represents the parameter sum from the 2nd frame to the Tth frame.

[0067] If is lower than or the peak sidelobe ratio of the Tth frame is higher than the average value before the T-1th frame, update the transformation matrix ε0, ε + and ε - , filter and otherwise, do not update:

[0068] or PSR T > PSR T-1 , update;

[0069] and PSR T ≤ PSR T-1 , do not update;

[0070] The OTB2015 tracking dataset sequence includes the target challenge of occlusion scene, high video resolution, and the video dataset has a total of 100 video sequences. Figure 3 The precision plot of the present method and other 13 methods on the OTB2015 tracking dataset is shown. It reflects the percentage of video frames in which the distance between the center point of the target position estimated by the tracking method and the center point of the target of the true value is less than a given threshold, relative to the total number of frames.

[0071] The intersection over union (IOU) reflects the degree of overlap between the predicted tracking frame and the manually labeled tracking frame, which is defined as the area of the intersection of the predicted frame and the true labeled frame relative to the area of the union of the predicted frame and the true labeled frame, and its formula is:

[0072]

[0073] Where A is the rectangular frame area obtained by the tracking method, B is the rectangular frame area given by the true value, and |·| represents the number of pixels in the area. When the IOU of a certain frame is greater than a set threshold, the frame is considered to be successfully tracked, and the percentage of the total number of successfully tracked frames relative to the total number of frames is the success rate. Figure 4 The success plot of the present method and other 13 methods on the OTB2015 tracking dataset is shown. From Figure 3 and Figure 4 It can be seen that the present method is superior to the other 13 methods in terms of precision and success rate in the occlusion scene.

[0074] The tracking results of the present method and other 3 methods in the jogging video sequence are shown in Figure 5 After passing through the occlusion, the present method can still stably track the target, but the other 3 methods have tracking drift due to insufficient anti-occlusion, and the tracking frame falls on other targets.

[0075] As described above, it should be understood that the present application is not limited to the form disclosed herein, should not be considered as excluding other embodiments, and can be used in various other combinations, modifications and environments, and can be modified by the above teachings or related art or knowledge within the scope of the inventive concept described herein. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application shall be within the protection scope of the claims of the present application.

Claims

1. An occlusion suppression tracking method based on multi-scale spatial semantic perception, characterized in that, Includes the following steps: Step 1, in the Frames are selected from multi-scale image templates and input into a convolutional neural network to obtain filters, where... , The total number of frames in the target video sequence; Step 2: Extract three sets of scale image patches around the target and input them into the convolutional neural network to obtain the transformation matrix of the three sets of scale image patches; Step 3, in the Three scale search regions are extracted from the frame and input into a convolutional neural network to obtain feature maps; Step 4: Obtain the spatial semantic map by searching the feature map of the scale-based search region and the transformation matrix, and further calculate the binary occlusion suppression matrix; Step 5: Convolve the feature maps of the three scale search regions and the filters respectively to obtain the multi-scale response map, and dot product it with the binary occlusion suppression matrix to obtain the peak map of multi-scale occlusion suppression; Step 6: Dot product the multi-scale response map and the binary occlusion suppression matrix to obtain the multi-scale occlusion suppression peak map; Step 7: The multi-scale occlusion suppression peak maps are fused using dot product operations to obtain the fused peak map; Step 8, update the position of the maximum response value in the fused peak map to the position of the peak value. The frame's position coordinates are used, and the fluctuation of the peak sidelobes is used as the basis for whether to update the transform matrix and filter; In step 4, the spatial semantic map is calculated by combining the feature maps and transformation matrices of the three scale search regions extracted by the convolutional neural network, using the following formula: (1) in, The first two are the spatial semantic map and the first two are the feature map. Okay, number Column elements, The transformation matrix is... The mean of the feature map. Based on this, a threshold judgment formula is used. (2) Obtain the binary occlusion suppression matrix .

2. The occlusion suppression tracking method based on multi-scale spatial semantic perception according to claim 1, characterized in that, The set of three scale image patches is fed into a convolutional neural network to obtain feature maps. Then, these stacked feature maps are reshaped into two-dimensional features, one dimension of which is the channel. The transformation matrix of the three scale image patches is obtained using the two-dimensional features by the principal component analysis method.

3. The occlusion suppression tracking method based on multi-scale spatial semantic perception according to claim 1, characterized in that... The peak image of occlusion suppression is obtained by dot productting the response image with the occlusion suppression matrix, as shown in the following formula: (3) in, These are the peak image, the occlusion suppression matrix, and the response image, respectively. Okay, number The elements of the column.

4. The occlusion suppression tracking method based on multi-scale spatial semantic perception according to claim 1, characterized in that, If the The variance of the peak sidelobe ratio of the frame is lower than that of the first frame. The peak sidelobe ratio variance of the previous frame or the first frame The peak sidelobe ratio of the first frame is higher than that of the second frame. The transform matrix and filter are updated based on the average value of the previous frame; otherwise, they are not updated.

Images