Target tracking method and tracking system based on deep learning

Abstract

The present invention provides a target tracking method and a tracking system based on deep learning, and relates to the field of computer vision technology. The present invention uses a pre-trained convolutional neural network to extract features, and combines optical flow information to determine the target search area, so as to more accurately capture the position change of the target between consecutive frames. Compared with the traditional method of manually designing features, this method can better adapt to different targets and environments. The consistency standard deviation of the optical flow vector is used to evaluate the reliability of each search area, which can help the algorithm distinguish interference factors such as background noise or occlusion, thereby improving the robustness of tracking. The method of the present invention also adopts a series of efficient processing steps, such as non-maximum suppression, smoothing filtering, and peak detection, which help to reduce the computational burden, so that the entire process can run at a lower latency, and is suitable for real-time applications.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target tracking method and tracking system based on deep learning. Background Art

[0002] Object tracking is an important task in the field of computer vision, which involves continuously locating and tracking one or more moving objects of interest in a video sequence. Traditional object tracking methods mainly include feature-based methods and model-based methods, which usually rely on manually designed features and simple appearance models. However, these methods have the following limitations when dealing with complex scenes: limited feature expression capabilities, manually designed features often have difficulty capturing complex changes in the target, and simple appearance models are prone to failure when the target appearance changes significantly. Many traditional methods require a lot of computing resources to process high-resolution video data. In recent years, the successful application of deep learning technology in the field of computer vision has promoted the development of object tracking technology. Object tracking methods based on deep learning can automatically extract high-level semantic features through an end-to-end learning mechanism, and show better robustness and accuracy when dealing with complex scenes.

[0003] Therefore, it is necessary to provide a target tracking method and tracking system based on deep learning to solve the above technical problems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a target tracking method and tracking system based on deep learning, which achieve the beneficial effect of showing better robustness and accuracy when processing complex scenes.

[0005] The present invention provides a target tracking method based on deep learning, and the target tracking method comprises the following steps:

[0006] S1: Obtain the initial frame image and use the pre-trained convolutional neural network to extract the original feature map and initial state information of the target area to be tracked;

[0007] S2: obtaining a current frame image, calculating the optical flow information between the initial frame image and the current frame image through a pre-trained optical flow estimation network, and determining multiple target search areas in the current frame image according to the initial state information and optical flow information of the target to be tracked, wherein the current frame image is the next frame image of the initial frame image;

[0008] S3: Calculate the reliability score of each target search area, select at least one target search area to form a target candidate area according to the reliability score of each target search area, and obtain a target candidate area feature map of the target candidate area;

[0009] S4: performing cross-correlation encoding processing on the original feature map of the target region and the feature map of the target candidate region to obtain a similarity response map, wherein the similarity response map indicates the similarity between the original feature map of the target region and the feature map of the target candidate region at corresponding positions;

[0010] S5: performing cross-correlation decoding processing on the similarity response map, and determining the final position of the target to be tracked in the current frame image according to the decoding result.

[0011] Preferably, step S1 includes the following specific steps:

[0012] S101: extracting features of the initial frame image using a pre-trained convolutional neural network to obtain a feature map of the initial frame image;

[0013] S102: Applying a region proposal unit of a convolutional neural network to extract an initial target candidate region from the initial frame image feature map;

[0014] S103: Execute a non-maximum suppression algorithm on the initial target candidate area to obtain initial state information of the initial target locking area and the target to be tracked;

[0015] S104: Use the pooling layer of the convolutional neural network to process the initial target locking area to obtain an original feature map of the target area of ​​the target to be tracked.

[0016] Preferably, in step S2, the optical flow information is a pixel displacement vector between an initial frame image and a current frame image, and the pixel displacement vector includes displacement components in a horizontal direction and a vertical direction.

[0017] Preferably, the calculation steps of the reliability score of each target search area in step S3 are:

[0018] Calculate the direction consistency standard deviation and size consistency standard deviation of the optical flow vector in the target search area for each pixel;

[0019] Calculate the aggregated reliability score of each pixel in the target search area and normalize the aggregated reliability score, where the aggregated reliability score is the weighted average of the direction consistency standard deviation and the size consistency standard deviation;

[0020] The reliability score of the target search region is calculated based on the aggregated reliability score of each pixel in the target search region, where the reliability score of the target search region is the standard deviation of the aggregated reliability scores of all pixels within the target search region.

[0021] Preferably, the direction consistency standard deviation is obtained by calculating the standard deviation of the direction cosine similarity of the optical flow vectors in the neighborhood, and the size consistency standard deviation is obtained by calculating the standard deviation of the Euclidean norm of the optical flow vectors in the neighborhood.

[0022] Preferably, after step S4 and before step S5, the method further includes applying a smoothing filter to perform denoising processing on the obtained similarity response graph.

[0023] Preferably, step S5 includes the following specific steps:

[0024] S501: using a peak detection algorithm to determine the local maximum point in the similarity response graph;

[0025] S502: applying a non-maximum suppression technique to remove repeated peaks adjacent to the local maximum point, and determining the corresponding position coordinates of the local maximum point of the peak value with the highest response value in the target candidate area as the target center coordinates;

[0026] S503: Based on the target center coordinates and the initial state information of the target to be tracked, determine the target frame position and size of the target to be tracked in the current frame image, thereby determining the final position of the target to be tracked in the current frame image, and updating the initial state information of the target to be tracked.

[0027] Preferably, in step S501, a peak threshold is preset, and the peak threshold is used to remove false peaks caused by noise changes and background changes in the similarity response graph.

[0028] The present invention also provides a target tracking system based on deep learning, which is applied to a target tracking method based on deep learning. The target tracking system includes:

[0029] The initial target feature extraction module is used to obtain the initial frame image and use the pre-trained convolutional neural network to extract the original feature map and initial state information of the target area of ​​the target to be tracked;

[0030] A target search area generation module is used to obtain a current frame image, calculate the optical flow information between the initial frame image and the current frame image through a pre-trained optical flow estimation network, and determine multiple target search areas in the current frame image according to the initial state information and optical flow information of the target to be tracked, wherein the current frame image is the next frame image of the initial frame image;

[0031] A target candidate region selection module is used to calculate the reliability score of each target search region, select at least one target search region to form a target candidate region according to the reliability score of each target search region, and obtain a target candidate region feature map of the target candidate region;

[0032] A similarity response map generation module is used to perform cross-correlation coding processing on the original feature map of the target region and the feature map of the target candidate region to obtain a similarity response map, wherein the similarity response map indicates the similarity between the original feature map of the target region and the feature map of the target candidate region at corresponding positions;

[0033] The target position decoding and updating module is used to perform cross-correlation decoding processing on the similarity response map and determine the final position of the target to be tracked in the current frame image according to the decoding result.

[0034] Compared with the related art, the target tracking method and tracking system based on deep learning provided by the present invention have the following beneficial effects:

[0035] The present invention uses a pre-trained convolutional neural network to extract features and combines optical flow information to determine the target search area, which can more accurately capture the position changes of the target between consecutive frames. This method can better adapt to different targets and environments than the traditional method of manually designing features. Using the consistency standard deviation of the optical flow vector to evaluate the reliability of each search area can help the algorithm distinguish interference factors such as background noise or occlusion, thereby improving the robustness of tracking. The method of the present invention also adopts a series of efficient processing steps, such as non-maximum suppression, smoothing filtering, and peak detection, which helps to reduce the computational burden, so that the entire process can run at a lower latency and is suitable for real-time applications. Finally, the entire tracking process is decomposed into multiple functional modules, which is not only easy to implement and maintain, but also facilitates possible future functional expansion or optimization work. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a flow chart of a target tracking method based on deep learning of the present invention;

[0037] Figure 2 This is a schematic diagram of the module structure of a target tracking system based on deep learning in the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0039] It should also be noted that, for ease of description, only the parts related to the present invention, but not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the various operations (or steps) as being processed sequentially, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the various operations can be rearranged. The processing can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0040] Embodiment 1

[0041] The present invention provides a target tracking method based on deep learning. In the specific implementation process, Figure 1 As shown, it shows a flowchart of a target tracking method based on deep learning, including:

[0042] Step S1, obtaining an initial frame image, and using a pre-trained convolutional neural network to extract the original feature map and initial state information of the target area of ​​the target to be tracked.

[0043] Specifically, step S1 includes the following specific steps:

[0044] Step S101, using a pre-trained convolutional neural network to extract features from an initial frame image to obtain a feature map of the initial frame image.

[0045] In the specific implementation process, first, a convolutional neural network that has been pre-trained using a large-scale dataset is selected, the obtained initial frame image is normalized and then input into the MobileNet model to perform a forward propagation operation, and the initial frame image feature map is output in the middle layer.

[0046] Step S102, applying the region proposal unit of the convolutional neural network to extract the initial target candidate region from the initial frame image feature map.

[0047] During the specific implementation process, after obtaining the feature map of the initial frame image, it is necessary to extract the feature map of the target area from the feature map, which is used as the feature matching basis of the target to be tracked in the subsequent frame image. The feature map of the initial frame image is passed as input to the region proposal unit of the convolutional neural network to generate a series of rectangular candidate boxes containing the target, namely the initial target candidate area.

[0048] Step S103, performing a non-maximum suppression algorithm on the initial target candidate area to obtain initial state information of the initial target locking area and the target to be tracked.

[0049] In the specific implementation process, in order to further lock the target in which initial target candidate area, further screening operations are required. In the specific implementation, the region proposal unit generates a series of rectangular candidate boxes containing the target, and assigns a confidence score to each rectangular candidate box. The non-maximum suppression algorithm is executed to remove the boxes with overlap higher than the set threshold and confidence scores lower than the set threshold. The candidate box with the highest confidence score among the remaining rectangular candidate boxes is selected as the final initial target locking area, and the bounding box coordinates, center point position, and feature vector of the initial target locking area are extracted and combined into the initial state information of the target to be tracked.

[0050] Step S104, using the pooling layer of the convolutional neural network to process the initial target locking area to obtain an original feature map of the target area of ​​the target to be tracked.

[0051] In the specific implementation process, after determining the initial target locking area, find the corresponding area on the original feature map and crop this area, and apply the pooling layer to downsample the cropped feature map in the spatial dimension to obtain the original feature map of the target area.

[0052] Step S2, obtaining the current frame image, calculating the optical flow information between the initial frame image and the current frame image through a pre-trained optical flow estimation network, and determining multiple target search areas in the current frame image according to the initial state information and optical flow information of the target to be tracked, wherein the current frame image is the next frame image of the initial frame image.

[0053] Specifically, in step S2, the optical flow information is a pixel displacement vector between an initial frame image and a current frame image, and the pixel displacement vector includes displacement components in a horizontal direction and a vertical direction.

[0054] In the specific implementation process, a current frame image is obtained, and the current frame image is the next frame image of the initial frame image. The optical flow information between the initial frame image and the current frame image is calculated through a pre-trained optical flow estimation network. During implementation, a PWC-Net optical flow estimation network based on deep learning is selected, and the initial frame image and the current frame image are provided as input to the optical flow estimation network PWC-Net. The two input frames are processed using the loaded model to obtain an optical flow field between them. The optical flow field is a two-dimensional vector field. The two-dimensional vector field contains a pixel displacement vector for each pixel, wherein each pixel displacement vector represents the direction and distance of the pixel point at the corresponding position moving from the initial frame image to the current frame image. The pixel displacement vector includes displacement components in the horizontal and vertical directions. The optical flow information related to the target to be tracked is extracted from the obtained optical flow field. Based on the initial state information and optical flow information of the target to be tracked, multiple new positions containing the target are determined in the current frame image as the target search area.

[0055] Step S3, calculating the reliability score of each target search area, selecting at least one target search area to form a target candidate area according to the reliability score of each target search area, and obtaining a target candidate area feature map of the target candidate area.

[0056] During the specific implementation process, it is necessary to screen out the target search area that is most likely to contain the target to be tracked from multiple target search areas. To do this, it is necessary to evaluate each target search area, obtain a reliability score for each target search area through the evaluation, and decide which target search area to select as the area containing the target to be tracked based on the reliability score. The higher the reliability score, the higher the possibility that the target search area contains the target to be tracked. Based on the reliability score, at least one target search area is selected as a target candidate area, and a pre-trained convolutional neural network is used to extract features from the target candidate area to obtain a target candidate area feature map.

[0057] Specifically, in step S3, the direction consistency standard deviation is obtained by calculating the standard deviation of the direction cosine similarity of the optical flow vectors in the neighborhood, and the size consistency standard deviation is obtained by calculating the standard deviation of the Euclidean norm of the optical flow vectors in the neighborhood.

[0058] The calculation steps of the reliability score of each target search area in step S3 are as follows:

[0059] Calculate the direction consistency standard deviation and size consistency standard deviation of the in-domain optical flow vector for each pixel in the target search area.

[0060] During the specific implementation process, when an object moves between video frames, the movement of its internal pixels should be relatively consistent. If the optical flow vectors in a certain area are relatively uniform, it usually means that the area is a coherent whole, rather than a background or a mixture of multiple different moving objects. Therefore, it can be used to track moving targets. Using the optical flow vector to calculate the reliability score helps to screen out target candidate areas with a greater possibility of containing targets in multiple target searches. First, the direction consistency standard deviation and size consistency standard deviation of the optical flow vector in the field of each pixel in the target search area are calculated. Among them, this consistency is measured by calculating the standard deviation of the direction cosine similarity of the optical flow vector in the neighborhood. The smaller the standard deviation, the more consistent the direction, and the higher the possibility of this area as a candidate target. The size consistency standard deviation is measured by calculating the standard deviation of the Euclidean norm of the optical flow vector in the neighborhood. The smaller the standard deviation, the more consistent the size, and the higher the possibility of this area as a candidate target.

[0061] The aggregated reliability score of each pixel in the target search area is calculated and normalized, where the aggregated reliability score is the weighted average of the direction consistency standard deviation and the size consistency standard deviation.

[0062] During the specific implementation process, the weighted average of the directional consistency standard deviation and the size consistency standard deviation is used as the aggregated reliability score of each pixel in the target search area to form a comprehensive evaluation index. Relying on one of them alone is not enough to accurately judge the reliability. The aggregated reliability score takes into account the directional difference and size difference of the optical flow vector at the same time to make the scoring more accurate. The two factors of direction and size are equally important, and the same weight value is assigned to the directional consistency standard deviation and the size consistency standard deviation to calculate the aggregated reliability score. The aggregated reliability score is used as a comprehensive evaluation indicator for each pixel in the target search area, providing a basis for the subsequent calculation of the reliability score of the target search area.

[0063] The reliability score of the target search region is calculated based on the aggregated reliability score of each pixel in the target search region, where the reliability score of the target search region is the standard deviation of the aggregated reliability scores of all pixels within the target search region.

[0064] In the specific implementation process, by performing statistical analysis on the aggregated reliability scores of each pixel in the target search area, the standard deviation is used as the value of the reliability score to obtain the reliability score of each target search area. This score reflects the consistency level of all pixels in the area. If more than 90% (can be set as the case may be) of the pixels in a region have high reliability scores and the changes in these scores are small, then the reliability score of this region will be high, indicating that it is likely to be the target to be tracked. One or several regions with the highest reliability scores are selected as the final target candidate regions. This step excludes unreliable regions and regions containing more interference factors, improves tracking accuracy, and uses a pre-trained convolutional neural network to extract features from the target candidate regions to obtain feature maps of the target candidate regions.

[0065] Step S4, performing cross-correlation coding processing on the original feature map of the target region and the feature map of the target candidate region to obtain a similarity response map, wherein the similarity response map indicates the similarity between the original feature map of the target region and the feature map of the target candidate region at corresponding positions.

[0066] In the specific implementation process, the cross-correlation coding technology is used to calculate the similarity between the original feature map of the target area and the feature map of the target candidate area, and these similarity results are combined to form a complete similarity response map. Each element in the similarity response map represents the similarity between the corresponding position in the feature map of the target candidate area and the original feature map of the target area. The position of the maximum value is found in the similarity response map. This position is the most likely position of the target to be tracked. Therefore, the similarity response map provides an intuitive and quantitative way to evaluate the basis of a more precise position of the target. Among them, the cross-correlation coding adopts a sliding window method, and applies the original feature map of the target area as a template pixel by pixel on the feature map of the target candidate area to calculate the similarity between the two, and combines the similarity results at all positions to form a complete similarity response map.

[0067] Step S5, performing cross-correlation decoding processing on the similarity response graph, and determining the final position of the target to be tracked in the current frame image according to the decoding result.

[0068] Specifically, step S5 includes the following specific steps:

[0069] Step S501: Use a peak detection algorithm to determine the local maximum point in the similarity response graph.

[0070] In the specific implementation process, the peak detection algorithm is used to detect the significant peaks in the similarity response graph. These significant peaks are local maximum points, and these peaks correspond to the possible positions of the target to be tracked.

[0071] Step S502: applying non-maximum suppression technology to remove repeated peaks adjacent to the local maximum point, and determining the corresponding position coordinates of the local maximum point of the peak value of the highest response value in the target candidate area as the target center coordinates.

[0072] In the specific implementation process, the non-maximum suppression method is applied to remove redundant detection boxes or peaks in the detection task. By comparing each local maximum point with other points in its neighborhood, it is determined whether to retain the point. If a local maximum point is not the largest in its neighborhood, it will be removed. In the present invention, the non-maximum suppression method is used to remove adjacent repeated peaks in the similarity response graph to ensure that each target has only one most significant peak, thereby improving the accuracy of target positioning and simplifying subsequent processing steps. Next, the coordinates of the position in the target candidate area corresponding to the peak with the highest response value among the remaining local maximum points after non-maximum suppression processing are selected as the target center coordinates. The target center coordinates are the final center position coordinates of the target to be tracked in the current frame image.

[0073] Step S503, based on the target center coordinates and according to the initial state information of the target to be tracked, determine the target frame position and size of the target to be tracked in the current frame image, thereby determining the final position of the target to be tracked in the current frame image, and updating the initial state information of the target to be tracked.

[0074] During the specific implementation process, according to the target center coordinates and the initial state information, during implementation, the center position and bounding box coordinates of the target frame in the initial state information are extracted as the basis for the size of the new target frame. The target center coordinates are used as the center coordinates, and the target frame position and size of the target to be tracked in the current frame image are determined through calculation, thereby determining the final position of the target to be tracked in the current frame image, realizing the tracking of the target, and updating the initial state information of the target to be tracked so that tracking can be continued in the next frame.

[0075] Specifically, after step S4 and before step S5, the process further includes applying a smoothing filter to perform denoising processing on the obtained similarity response graph.

[0076] In the specific implementation process, preprocessing is required to reduce the influence of noise before cross-correlation decoding of the similarity response map. During preprocessing, a Gaussian filter is used to smooth the similarity response map. The Gaussian filter can effectively reduce noise while retaining the main structural information of the image and controlling the degree of smoothing.

[0077] Specifically, in step S501, a peak threshold is preset, and the peak threshold is used to remove false peaks caused by noise changes and background changes in the similarity response graph.

[0078] During the specific implementation process, the similarity response map will be affected by the noise, lighting changes, and texture factors in the image, and some insignificant local maximum points will appear. These points are not the real target positions, but are caused by interference factors. In order to ensure that those significant and reliable peaks are retained, a peak threshold is set to exclude those local maximum points whose response values ​​are lower than the peak threshold, thereby reducing the interference of subsequent screening.

[0079] The working principle of a target tracking method based on deep learning provided by the present invention is as follows:

[0080] First, the initial frame image is obtained, and the pre-trained convolutional neural network is used to extract features from the initial frame image to obtain the original feature map of the target area to be tracked and the initial state information. The original feature map of the target area is used as the matching basis for the target to be tracked in the subsequent frame image, and the original state information is used to determine the state information of the target to be tracked in the new frame image. Secondly, the current frame image is obtained, and the pre-trained optical flow estimation network is used to calculate the optical flow information between the initial frame image and the current frame image. According to the optical flow information, multiple target search areas of the target to be tracked in the current frame image are predicted. Thirdly, the consistency of the optical flow vector in the optical flow information is used to calculate the possibility evaluation of the existence of the target to be tracked in each target search area, and the reliability score of each target search area is obtained. The higher the reliability score, the higher the possibility that the target search area contains the target to be tracked. According to the reliability score, at least one target search area is selected to form a target candidate area, and the target candidate area feature map of the target candidate area is obtained. Then, the original feature map of the target area and the feature map of the target candidate area are cross-correlated encoded, and a sliding window is used. The method is to apply the original feature map of the target area as a template pixel by pixel on the feature map of the target candidate area, calculate the similarity between the two, and combine the similarity results at all positions to form a complete similarity response map. Each element in the response map corresponds to a position in the feature map of the target candidate area, and its value represents the similarity between the position and the original feature map of the target area. Finally, the similarity response map is cross-correlation decoded, and the peak detection algorithm is used to detect all local maximum points in the similarity response map. According to the preset peak threshold, those peaks below a certain intensity are filtered out, and the non-maximum suppression algorithm is used to remove the repeated peaks adjacent to the local maximum points. The peak with the highest response value is selected to correspond to the position in the current frame image as the target center coordinates. According to the target center coordinates and the initial state information of the target to be tracked, the position and size of the target selection box of the target to be tracked in the current frame image are determined. The position of this target selection box is the final position of the target to be tracked in the current frame image. According to the newly determined position and size, the state information of the target to be tracked is updated for tracking in the next frame.

[0081] Embodiment 2

[0082] The present invention provides a target tracking system based on deep learning, which is used to perform a target tracking method based on deep learning, such as Figure 2 As shown, it shows a schematic diagram of the module structure of a target tracking system based on deep learning, and the tracking system includes:

[0083] The initial target feature extraction module 100 is used to obtain an initial frame image and use a pre-trained convolutional neural network to extract the original feature map and initial state information of the target area of ​​the target to be tracked;

[0084] The target search area generation module 200 is used to obtain the current frame image, calculate the optical flow information between the initial frame image and the current frame image through the pre-trained optical flow estimation network, and determine multiple target search areas in the current frame image according to the initial state information and optical flow information of the target to be tracked, wherein the current frame image is the next frame image of the initial frame image;

[0085] The target candidate region selection module 300 is used to calculate the reliability score of each target search region, select at least one target search region to form a target candidate region according to the reliability score of each target search region, and obtain a target candidate region feature map of the target candidate region;

[0086] A similarity response map generation module 400 is used to perform cross-correlation coding processing on the original feature map of the target region and the feature map of the target candidate region to obtain a similarity response map, wherein the similarity response map indicates the similarity between the original feature map of the target region and the feature map of the target candidate region at corresponding positions;

[0087] The target position decoding and updating module 500 is used to perform cross-correlation decoding processing on the similarity response graph, and determine the final position of the target to be tracked in the current frame image according to the decoding result.

[0088] The working principle of a target tracking system based on deep learning provided by the present invention is as follows:

[0089] First, the initial target feature extraction module 100 uses a pre-trained convolutional neural network to obtain the original feature map of the target area of ​​the target to be tracked in the initial frame image and the initial state of the target, which are used as the matching basis for the target to be tracked in the subsequent frame image. Secondly, the target search area generation module 200 obtains the current frame image, and calculates the optical flow information between the initial frame image and the current frame image through the pre-trained optical flow estimation network. Based on the optical flow information, the position of the target to be tracked in the current frame image is predicted to obtain multiple target search areas. Thirdly, the target candidate area selection module 300 uses the consistency of the optical flow vector in the optical flow information to calculate the reliability score of each target search area. The search area with a high reliability score is more likely to contain the target to be tracked. At least one target search area with a high reliability score is selected to form a target candidate area, and a target candidate area feature map of the target candidate area is obtained. Then, the similarity response map generation module 400 performs cross-correlation coding processing on the original feature map of the target area and the feature map of the target candidate area, and adopts a sliding window method to image by image on the feature map of the target candidate area. The original feature map of the target area is used as a template to calculate the similarity between the two, and the similarity results at all positions are combined to form a complete similarity response map. Each element in the response map corresponds to a position in the feature map of the target candidate area, and its value represents the similarity between the position and the original feature map of the target area. Finally, the target position decoding and updating module 500 performs cross-correlation decoding processing on the similarity response map, uses a peak detection algorithm to detect all local maximum points in the similarity response map, and filters out those peaks below a certain intensity according to a preset peak threshold. Then, a non-maximum suppression algorithm is used to remove repeated peaks adjacent to the local maximum points, and the position coordinates of the peak corresponding to the highest response value in the target candidate area are selected as the target center coordinates. The position and size of the target selection box of the target to be tracked in the current frame image are determined according to the target center coordinates and the initial state information of the target to be tracked. The position of this target selection box is the final position of the target to be tracked in the current frame image. According to the newly determined position and size, the state information of the target to be tracked is updated for tracking in the next frame.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0091] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A target tracking method based on deep learning, characterized in that: The target tracking method comprises: S1: Obtain the initial frame image and use the pre-trained convolutional neural network to extract the original feature map and initial state information of the target area to be tracked; S2: obtaining a current frame image, calculating the optical flow information between the initial frame image and the current frame image through a pre-trained optical flow estimation network, and determining multiple target search areas in the current frame image according to the initial state information and optical flow information of the target to be tracked, wherein the current frame image is the next frame image of the initial frame image; S3: Calculate the reliability score of each target search area, select at least one target search area to form a target candidate area according to the reliability score of each target search area, and obtain a target candidate area feature map of the target candidate area; The calculation steps of the reliability score of each target search area are as follows: Calculate the direction consistency standard deviation and size consistency standard deviation of the optical flow vector in the target search area for each pixel; Calculate the aggregated reliability score of each pixel in the target search area and normalize the aggregated reliability score, where the aggregated reliability score is the weighted average of the direction consistency standard deviation and the size consistency standard deviation; Calculate the reliability score of the target search area based on the aggregated reliability score of each pixel in the target search area, where the reliability score of the target search area is the standard deviation of the aggregated reliability scores of all pixels in the target search area; S4: performing cross-correlation encoding processing on the original feature map of the target region and the feature map of the target candidate region to obtain a similarity response map, wherein the similarity response map indicates the similarity between the original feature map of the target region and the feature map of the target candidate region at corresponding positions; S5: performing cross-correlation decoding processing on the similarity response map, and determining the final position of the target to be tracked in the current frame image according to the decoding result; Step S5 includes the following specific steps: S501: using a peak detection algorithm to determine the local maximum point in the similarity response graph; S502: applying a non-maximum suppression technique to remove repeated peaks adjacent to the local maximum point, and determining the corresponding position coordinates of the local maximum point of the peak value with the highest response value in the target candidate area as the target center coordinates; S503: Based on the target center coordinates and the initial state information of the target to be tracked, determine the target frame position and size of the target to be tracked in the current frame image, thereby determining the final position of the target to be tracked in the current frame image, and updating the initial state information of the target to be tracked.

2. The target tracking method based on deep learning according to claim 1, characterized in that: Step S1 includes the following specific steps: S101: extracting features of the initial frame image using a pre-trained convolutional neural network to obtain a feature map of the initial frame image; S102: Applying a region proposal unit of a convolutional neural network to extract an initial target candidate region from the initial frame image feature map; S103: Execute a non-maximum suppression algorithm on the initial target candidate area to obtain initial state information of the initial target locking area and the target to be tracked; S104: Use the pooling layer of the convolutional neural network to process the initial target locking area to obtain an original feature map of the target area of ​​the target to be tracked.

3. The target tracking method based on deep learning according to claim 2, characterized in that: In step S2, the optical flow information is a pixel displacement vector between an initial frame image and a current frame image, and the pixel displacement vector includes displacement components in a horizontal direction and a vertical direction.

4. The target tracking method based on deep learning according to claim 3, characterized in that: The direction consistency standard deviation is obtained by calculating the standard deviation of the direction cosine similarity of the optical flow vectors in the neighborhood, and the size consistency standard deviation is obtained by calculating the standard deviation of the Euclidean norm of the optical flow vectors in the neighborhood.

5. The target tracking method based on deep learning according to claim 4, characterized in that: After step S4 and before step S5, the process further includes applying a smoothing filter to perform denoising on the obtained similarity response graph.

6. The target tracking method based on deep learning according to claim 5, characterized in that: In step S501, a peak threshold is preset, and the peak threshold is used to remove false peaks caused by noise changes and background changes in the similarity response graph.

7. A target tracking system based on deep learning, characterized in that: The target tracking method based on deep learning as described in any one of claims 1 to 6, wherein the target tracking system comprises: The initial target feature extraction module is used to obtain the initial frame image and use the pre-trained convolutional neural network to extract the original feature map and initial state information of the target area of ​​the target to be tracked; A target search area generation module is used to obtain a current frame image, calculate the optical flow information between the initial frame image and the current frame image through a pre-trained optical flow estimation network, and determine multiple target search areas in the current frame image according to the initial state information and optical flow information of the target to be tracked, wherein the current frame image is the next frame image of the initial frame image; A target candidate region selection module is used to calculate the reliability score of each target search region, select at least one target search region to form a target candidate region according to the reliability score of each target search region, and obtain a target candidate region feature map of the target candidate region; A similarity response map generation module is used to perform cross-correlation coding processing on the original feature map of the target region and the feature map of the target candidate region to obtain a similarity response map, wherein the similarity response map indicates the similarity between the original feature map of the target region and the feature map of the target candidate region at corresponding positions; The target position decoding and updating module is used to perform cross-correlation decoding processing on the similarity response map and determine the final position of the target to be tracked in the current frame image according to the decoding result.

Images