A single-target long-time tracking method based on deep neural network

Abstract

The application belongs to the technical field of target tracking, and discloses a single-target long-time tracking method based on a deep neural network, which comprises the following steps: acquiring a tracking target image and a tracking search area image, building a feature extraction network based on a twin network, and performing feature extraction; in the tracking process, a target image is selected by a PNR score to be added to a template library and is sent to a model learning network to perform model learning and updating, a model target feature map and a search feature map are convolved to obtain a response map, the response map is sent to a scale regression network to obtain a scale position score map, and a target scale is acquired; when the target is occluded or lost, target recapture is performed; in the tracking process, information of the target and a suspected target is recorded, and when an interference object appears, a tracking position that is most matched with a historical track is selected. The application reduces model drift phenomenon, makes scale estimation more stable and accurate, and makes the algorithm suitable for long-time target tracking by using a re-search mechanism and an anti-interference matching technology.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of visual target tracking, and particularly relates to a single-target long-time tracking method based on a deep neural network. BACKGROUND

[0002] Single-target tracking refers to tracking a specified target in a video, the target being specified by a human at the beginning and the subsequent position and scale of the specified target being estimated by a tracking algorithm using context information of the video or image sequence. The single-target tracking algorithm has a wide range of application fields, including intelligent video monitoring, unmanned driving, the field of robots, the field of unmanned aerial vehicles, etc. The application scenarios of the target tracking algorithm are diversified, such as in a shopping mall with a large number of people, on a road, on a highway, in the sky, etc., and therefore it faces some difficult challenges, such as illumination change, target deformation, occlusion, complex background, etc.

[0003] At present, in the tracking algorithm based on a deep neural network, the tracking algorithm based on a twin network has been widely studied due to its good performance in speed and accuracy. Methods based on an attention mechanism, methods based on scale estimation of an RPN network, methods based on spatiotemporal modeling, etc. have appeared. However, in the long-time tracking process in reality, the appearance of the target usually changes dynamically, or even rapidly, and the target is occluded, etc. The current methods have the problem of unsatisfactory tracking under the conditions of target deformation and occlusion in the long-time tracking process, such as the phenomenon of tracking drift. SUMMARY

[0004] In view of the above problems, the application provides a long-time tracking algorithm based on RMAD, i.e. a template library updating and scale regression network based on a PNR value, and a scale estimation based on a Hungarian matching algorithm to improve the accuracy and robustness of the tracking algorithm under the conditions of target deformation, target occlusion, etc.

[0005] The single-target long-time tracking method based on a deep neural network disclosed by the application comprises the following steps:

[0006] A tracking target image and a tracking search region image are acquired, a feature extraction network based on a twin network is built, and feature extraction is performed;

[0007] An online learning model: in the tracking process, a target image is selected by a PNR score to be added to a template library, and then is sent to a model learning network for model learning and updating;

[0008] Target scale estimation: a model target feature map and a search feature map are convolved to obtain a response map, the response map is sent to a scale regression network to obtain a scale position score map, and the target scale is acquired;

[0009] Target recapture: in the tracking process, the state of the target is judged to determine whether the target is blocked or lost, and when the target is lost, the target search range is expanded to recapture the target;

[0010] Anti-interference matching tracking: in the tracking process, the position, size, confidence and texture feature information of the target and the suspected target are recorded, when the interference object appears, the Hungarian matching algorithm is used to track match the current estimated tracking position and the historical tracking trajectory, the tracking position most matched with the historical trajectory is selected, so as to exclude the influence of the interference object and guarantee the long-time tracking of the target.

[0011] Further, the twin network adopts ResNet50 as the backbone network.

[0012] Further, the loss function of the model learning network is:

[0013]

[0014]

[0015]

[0016] wherein v j is a feature map of a training image, c j represents the center coordinates of the target, and the purpose is to find a model f, and the target image is represented by A c When the pixel in the image is a background pixel, A c ≈0, and when it is a target, A c ≈1, s=v*f, which represents the target confidence score, y c represents the score of each coordinate position, y c is initialized by using Gaussian distribution, y c ,A c ,η c parameters are learned through online training.

[0017] Further, the calculation formula of the PNR is as follows:

[0018]

[0019]

[0020] wherein S′ max represents the maximum value in the classification score response map S′, S′ min represents the minimum value in the classification score response map S′, M S represents the mean value of S′ after removing the maximum response value, T PNR is the mean value of all historical PNR values, and T S′ is all historical S′max the mean value of the scale regression network.

[0021] Further, the scale regression network is trained according to the following algorithm:

[0022] The loss function is:

[0023]

[0024] L CIOU = -ln(CIOU)

[0025]

[0026] wherein L CIOU is the CIOU loss, (x l , y l ) represents the upper left corner point of the real target frame, (x r , y r ) represents the lower right corner point of the real target frame, (x c , y c ) represents the center point coordinate, and then

[0027]

[0028]

[0029]

[0030]

[0031] According to the classification score map, the highest score target position is obtained, and then the mean value of the top-k target frames around the target position is taken as the frame of the final target.

[0032] Further, according to the classification score of the target, the state of the target is judged: when the score is less than 0.25, it is judged that the target is blocked or the target is not found at present, at this time, a counter is enabled, and when the counter is greater than 5, it is indicated that the target is lost, and then according to the size of the target, the search area is expanded by 1 time every 2 frames, and the target is searched by using the classification network, and when the score of the target is greater than 0.8, it is considered that the target is found; when the search time exceeds a certain number of frames, it is considered that the target is lost.

[0033] Further, when multiple suspected targets appear in the tracking process, the information of the suspected targets is saved, and the information includes the center point position, the scale, the classification confidence and the SILTP feature.

[0034] Further, for a given position (x c , y c ), the encoding formula of the SILTP feature is:

[0035]

[0036] wherein g c is the gray value of the center pixel, g k is the other k pixels in the center pixel field with a radius of R, s is a segmentation function, and epsilon = 0.3:

[0037]

[0038] Further, the cost in the Hungarian matching algorithm is calculated by the cosine similarity between vectors,

[0039]

[0040] wherein obj i is the center point position, scale, classification confidence, and SILTP feature value of the suspected target, tr T is the center point position, scale, classification confidence, and SILTP feature value of the nearest point of the tracking trajectory.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] The template library updating of the present application adds reliable target templates to the template library according to the PNR value of the classification score response map, and reduces the model drift phenomenon;

[0043] The scale estimation uses the scale regression network, so that the scale is more stable and accurate;

[0044] A long-time tracking strategy of the target is designed, and after the target is lost, a dynamic expansion search range and re-identification are enabled;

[0045] An anti-interference matching algorithm is used to reduce similar object interference, so that the tracking algorithm can track the target for a long time. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flowchart of the present application;

[0047] Figure 2 Classification score map when similar objects appear;

[0048] Figure 3 Structure diagram of the deep neural network of the present application;

[0049] Figure 4 Tracking result map of the present application. DETAILED DESCRIPTION

[0050] The present application will be further described below in conjunction with the drawings, but the present application is not limited in any way by the drawings, and any transformation or replacement based on the teaching of the present application belongs to the protection scope of the present application.

[0051] To achieve the above object, the application adopts the following technical scheme:

[0052] S1 obtains a tracking target image and a tracking search area image, builds a feature extraction network based on a twin network, and performs feature extraction.

[0053] S2 online learning model: in the tracking process, the target image is selected to join the template library through the PNR score, and then is sent to the model learning network for model learning and updating.

[0054] S3 target scale estimation: the model target feature map and the search feature map are convolved to obtain a response map, the response map is sent to a scale regression network to obtain a scale position score map, and the target scale is obtained.

[0055] S4 target recapture: in the tracking process, the state of the target is judged to determine whether the target is occluded or lost, and when the target is lost, the target search range is expanded for target recapture.

[0056] S5 anti-interference matching tracking: in the tracking process, the position, scale, confidence and texture feature information of the target and the suspected target are recorded, when the interference object appears, the Hungarian matching algorithm is used to track match the current estimated tracking position and the historical tracking trajectory, the tracking position most matched with the historical trajectory is selected, so as to exclude the influence of the interference object and ensure long-time tracking of the target.

[0057] The application will be further described below in combination with the drawings.

[0058] Example 1

[0059] As shown in Figure 1 , the application constructs a single target long-time tracking algorithm based on a RMAD deep neural network, the basic idea of which is to extract the deep features of the target and the search area by using a twin network, to establish a template library for the target features, to obtain the target template features by using an online learning model, to obtain a classification score map by convolving the target template features and the search area features, and to obtain the position information of the target. Then the scale regression network is used to determine the target size; through the tracking state judgment mechanism, the search range is expanded when the target is lost to search again. When the interference object appears, the Hungarian matching algorithm is used for tracking matching to exclude the influence of the interference object.

[0060] Specifically, the following steps are included:

[0061] S1 build a feature extraction network based on a twin network: resnet50 is used as the backbone network, the input of the network is composed of a tracking target image and a tracking search area image, and the input is obtained after the network to obtain the feature map of the tracking target and the feature map of the search area.

[0062] S2 sends the feature map into the classification network to get the classification score map: the classification score network is composed of an online learning model network and a related convolution network. The input of the online learning model network is a template set of 50 target feature templates, which contains 15 initial target feature templates and target feature templates in the subsequent tracking process, and online learning is performed every 20 frames. The online learning model network is trained according to the following algorithm:

[0063] 1) the loss function adopted is

[0064]

[0065]

[0066] where v j is the feature map of the training image, c j represents the target center coordinates. The purpose is to find the model f, and the target image is represented by A c , when the pixel in the image is a background pixel, A c ≈0, and when it is a target, A c ≈1. s=v*f represents the target confidence score, y c represents the score of each coordinate position, and y c is initialized using a Gaussian distribution. Where y c , A c , η c parameters are learned through online training.

[0067] Then, according to the PNR peak signal-to-noise ratio score of the classification score response map S', it is judged whether the target feature image is added to the template library. When the PNR value is greater than the threshold T PNR and the classification score is also greater than the threshold T S′ , the current target feature map is added to the template library. When the template library is full, the oldest template except the initial template is replaced, and the PNR calculation formula is as follows:

[0068]

[0069]

[0070] where S' max represents the maximum value in the classification score response map S', S' min represents the minimum value in the classification score response map S', M S represents the mean value of S' after removing the maximum response value, T PNR is the mean value of all historical PNR values, and TS′ For all history S' max The mean value.

[0071] S3 calculates the rectangular frame of the tracking target through the scale regression network: obtain the model target feature map of step (2), convolve the model target feature map and the search feature map to obtain a response map, and send the response map into the scale regression network to obtain a four-dimensional position score map A w×h×4 Each point in it is composed of a four-dimensional vector, and t(i,j) = (l,t,r,b) represents the distance from the target center point (i,j) to the four edges of the target rectangular frame. The scale regression network is specifically trained according to the following algorithm:

[0072] The loss function is

[0073] L CIOU = -ln(CIOU)

[0074]

[0075] Where L CIOU is the CIOU loss, (x l , y l ) represents the upper left corner point of the real target frame, (x r , y r ) represents the lower right corner point of the real target frame, and (x c , y c ) represents the center point coordinate, then

[0076]

[0077] According to the classification score map, the highest score target position is obtained, and then the mean value of the top-k target frames around it is taken as the frame of the final target.

[0078] S4 judges the target state according to the classification score of the target: when the score is less than 0.25, it is judged that the target is blocked or the target is not found at present. At this time, the counter is enabled, and when the counter is greater than 5, it indicates that the target is lost, and then the search area is expanded by 1 times every 2 frames according to the target size, and the target is searched again using the classification network. When the target score is greater than 0.8, it is considered that the target is found. When the search time exceeds a certain number of frames, it is considered that the target is lost.

[0079] S5 anti-interference matching: when multiple peak points higher than a certain threshold appear on the classification score map, as shown in Figure 2 , it indicates that multiple suspected targets appear in the tracking process. At this time, the information of the suspected target obj i is saved, including the center point position, the scale, the classification confidence, and the SILTP feature. The Hungarian matching algorithm is used to match the target historical tracking trajectory trT (p1,p2,…p T The final tracking result is obtained by matching the results. The SILTP (Scale Invariant Local Ternary Pattern) operator is an improved version of the LBP operator, which is more robust to local image noise and illumination changes, and has stronger texture recognition capabilities. For a given location (x... c ,y c The SILTP encoding formula is:

[0080]

[0081] Where g c g is the grayscale value of the center pixel. k Let be the N other pixels within the neighborhood of the center pixel (a circle with radius R), s be a piecewise function, and ε = 0.3.

[0082]

[0083] The method for extracting target texture features is as follows: Take the target image, perform grayscale transformation, calculate the SILTP value of each pixel in the target image, using a 4-neighborhood and calculating counter-clockwise. The cost in Hungarian matching is calculated using cosine similarity, and its formula is:

[0084]

[0085] Among them obj i The vector contains information about the suspected target (center point location, scale, classification confidence, SILTP feature value), tr T The vector represents information about the nearest point in the tracking trajectory (center point location, scale, classification confidence, SILTP feature value).

[0086] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0087] The template library update of this invention adds reliable target templates to the template library based on the PNR value of the classification score response map, thereby reducing model drift.

[0088] The use of a scale regression network in scale estimation makes the scale more stable and accurate;

[0089] A long-term target tracking strategy was designed, and dynamic expansion of the search range and re-identification were enabled after the target was lost;

[0090] An anti-interference matching algorithm is adopted to reduce interference from similar objects, enabling the tracking algorithm to track the target for a long time.

[0091] As used herein, the word "preferably" is used interchangeably with the word "preferably," to mean that an example, implementation or embodiment is preferred, but not necessarily advantageous over other examples, implementations or embodiments. As used herein, the word "preferably" is used to mean that an aspect, design, or implementation so described is preferred, but not necessarily advantageous over other aspects, designs, or implementations. The use of the terms "preferably," "more preferably," "most preferably," and the like are intended to further clarify the order of preference of various aspects, designs, or implementations. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise, or clear from context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied under any of the foregoing instances. In addition, the articles "a," "an," and "the" as used in the context of this document are to be construed to be open-ended terms (i.e., to mean "one or more"). Follow-on Related Application

[0092] Moreover, although the present disclosure has been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based on the foregoing description and accompanying drawings. The present disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular, with reference to the various functions performed by the above described components (e.g., elements, etc.) to achieve the results, the terms used to describe certain claim components should not be construed to be limited to the specific aspects as set forth above but rather should be interpreted as broadly as is reasonable. For example, the term "comprises" is used in the specification and claims to mean that the disclosed implementations include, but are not limited to, the features, elements, and / or components described in the specification and / or claims. In addition, the term "comprises" is used in the specification and claims to mean that the disclosed implementations include, but are not limited to, the features, elements, and / or components described in the specification and / or claims. Further, the term "comprises" is used in the specification and claims to mean that the disclosed implementations include, but are not limited to, the features, elements, and / or components described in the specification and / or claims.

[0093] The various functional units in the embodiments of the present application can be integrated in one processing module, or each unit can exist physically, or a plurality of or more units can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of a software functional module. If the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. The above-mentioned devices or systems can execute the storage method in the corresponding method embodiments.

[0094] In summary, the above-mentioned embodiments are one embodiment of the present application, but the embodiments of the present application are not limited by the above-mentioned embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principle of the present application are equivalent replacement modes and are included in the protection scope of the present application.

Claims

1. A single-target long-time tracking method based on a deep neural network, characterized in that, The method comprises the following steps: Obtaining a tracking target image and a tracking search area image, building a feature extraction network based on a twin network, and performing feature extraction; In the tracking process, a target image is selected by a PNR score to join a template library and is sent to a model learning network for model learning and updating to obtain a model target feature map and a search feature map; The model target feature map and the search feature map are convolved to obtain a response map, the response map is sent to a scale regression network to obtain a scale position score map, and a target scale is obtained; In the tracking process, the state of the target is judged to determine whether the target is occluded or lost, and the target search range is expanded when the target is occluded or lost to recapture the target; In the tracking process, the position, scale, confidence and texture feature information of the target and the suspected target are recorded, when an interference object appears, a Hungarian matching algorithm is used to track and match the current estimated tracking position and the historical tracking trajectory, the tracking position most matched with the historical trajectory is selected, and thus the influence of the interference object is excluded to ensure long-time tracking of the target. The calculation formula of the PNR is as follows: ， ， wherein represents the maximum value in the classification score response plot , represents the minimum value in the classification score response plot , represents the mean value of after removing the maximum response value is the mean value of all historical PNR values is the mean value of all historical .

2. The deep neural network based single object long-term tracking method according to claim 1, wherein, The twin network adopts ResNet50 as a backbone network.

3. The deep neural network based single object long-term tracking method according to claim 1, wherein, The loss function of the model learning network is as follows: ， ， , wherein is a feature map of the training image, denotes the target center coordinates, the goal is to ask the model , the target image is denoted by , when the pixel in the image is a background pixel , for the target , , denotes the target confidence score, denotes the score for each coordinate position, Gaussian distribution is used for initialization, parameters are learned through online training.

4. The deep neural network based single object long-term tracking method according to claim 1, wherein, The scale regression network is trained according to the following algorithm: The loss function is as follows: wherein is the CI0U loss, with denotes the top-left point of the real target frame, denotes the bottom-right point of the real target frame, denotes the center point coordinate, then According to the classification score map, the target position with the highest score is obtained, and then the mean value of top-k target boxes around the target position is taken as the final target box.

5. The deep neural network based single object long-term tracking method according to claim 1, wherein, The state of the target is judged according to the classification score of the target: when the score is less than 0.25, it is judged that the target is occluded or the target is not found, at this time, a counter is enabled, and when the counter is greater than 5, it is considered that the target is lost, and then the search area is expanded by 1 time every 2 frames according to the size of the target, and the target is searched by using the classification network, and when the target score is greater than 0.8, it is considered that the target is found; when the search time exceeds a certain number of frames, it is considered that the target is lost.

6. The deep neural network based single object long-term tracking method according to claim 1, wherein, When multiple suspected targets appear in the tracking process, the information of the suspected targets is saved, and the information includes the center point position, the scale, the classification confidence and the SILTP feature.

7. The deep neural network based single object long-term tracking method according to claim 6, wherein, For a given position The encoding formula for the SILTP feature is: wherein is the gray value of the center pixel, is the gray value of the center pixel, : 。 8. The deep neural network based single object long-term tracking method according to claim 7, wherein, In the Hungarian matching algorithm, the cost is calculated by the cosine similarity to calculate the similarity between vectors. wherein The vector is composed of the center point position, scale, classification confidence, SILTP feature value of the suspected target, The vector is composed of the center point position, scale, classification confidence, SILTP feature value of the suspected target,

Images