Data processing method for updating target image based on contextual information feature fusion

Abstract

The invention relates to a data processing method for updating a target image based on contextual information feature fusion, belongs to the technical field of computer vision, and solves the problem that a tracking target is lost when the appearance of the tracking target is greatly changed in the prior art. Comprising the following steps: acquiring a video image sequence including a to-be-tracked target and a to-be-tracked target image; inputting the to-be-tracked target image and the video image sequence into a trained target tracking model to obtain a target tracking result; wherein the target tracking model is used for tracking each frame of image in a video image sequence by using target template features to obtain a to-be-tracked target; the target template feature is dynamically updated according to the confidence score of the target area identified by each frame of image in the video image sequence. The target template is dynamically updated by fusing the context information, and the performance and robustness of a target tracking algorithm in a complex environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a data processing method for updating a target image based on context information feature fusion. Background Art

[0002] Single object tracking is a widely studied topic in computer vision. Its core concept is to accurately estimate the position of an object at any given moment in a subsequent video sequence using only the initial position information of the object. Object tracking is widely used in fields such as autonomous driving, robotics, and video surveillance. With the introduction of deep learning technology, its performance has been significantly improved, promoting its widespread application in real-world scenarios.

[0003] In recent years, deep learning-based object tracking has made significant progress, among which the Siamese network is one of the most successful tracking frameworks. It defines the visual object tracking task as a target matching problem, aiming to learn a general similarity graph between the target template and the search area. The Siamese network tracker consists of a template branch and a detection branch. The template branch inputs the initial target image, and the detection branch inputs the current frame image patch. The two branches share convolutional neural network (CNN) parameters and the same transformation fitting to find the target in the search area.

[0004] However, most current twin networks are learned offline, selecting the target image as the tracking template in the first frame, and the template is not updated during the tracking process. When the appearance of the tracked target changes significantly, such as in a changing and complex environment such as motion, deformation, occlusion, etc., it is easy to cause the tracked target to be lost. That is, changes in the target's appearance or features will seriously affect the tracking effect of the algorithm. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a data processing method for updating a target image based on context information feature fusion, so as to solve the existing problem of losing the tracking target when the appearance of the tracking target changes significantly.

[0006] The purpose of the present invention is mainly achieved through the following technical solutions:

[0007] The present invention provides a data processing method for updating a target image based on context information feature fusion, which is characterized by comprising the following steps:

[0008] Acquire a video image sequence including a target to be tracked and an image of the target to be tracked;

[0009] The target image to be tracked and the video image sequence are input into a trained target tracking model to obtain a target tracking result; wherein the target tracking model is used to track each frame image in the video image sequence using a target template feature to obtain the target to be tracked; the target template feature is dynamically updated according to the confidence score of the target area identified in each frame image in the video image sequence.

[0010] Further, the target template feature is generated based on the target image to be tracked and the target template feature set; wherein the initial value of the target template feature set is a plurality of feature copies of the target image to be tracked;

[0011] Dynamically updating the target template feature according to the confidence score of the target area identified in each frame image in the video image sequence, including:

[0012] Obtaining a target region of a current frame image and a confidence score of the target region of the current frame image based on the current target template feature recognition;

[0013] When the confidence score is greater than the average confidence score of the target region of the historical frame, one of the feature copies is replaced with the target image feature in the target region to update the target template feature set.

[0014] Further, generating the target template feature based on the target image to be tracked and the target template feature set includes:

[0015] Extracting features of the target image to be tracked;

[0016] Performing feature splicing on the features of the target image to be tracked and the target template features in the target template feature set to obtain spliced ​​features;

[0017] The concatenated features are sequentially passed through two two-dimensional convolutional layers to perform dimensionality reduction operations on the concatenated features to obtain dimensionality reduced features;

[0018] Based on the reduced-dimensional features, a dynamic convolution method is used to obtain the network weights of each convolution kernel of the dynamic convolution layer, and a weighted sum of the convolution kernels in the dynamic convolution layer is calculated;

[0019] Performing a convolution operation on the weighted sum of the convolution kernels in the dynamic convolution layer and the dimensionally reduced features to obtain convolutional features;

[0020] The convolutional features are batch normalized and activated to obtain target template features.

[0021] Furthermore, based on the dimensionally reduced features, the network weights of each convolution kernel of the dynamic convolution layer are obtained by a dynamic convolution method, including:

[0022] Performing an average pooling operation on the reduced-dimensional features to further reduce the spatial dimension of the features;

[0023] The features after the average pooling operation pass through the fully connected layer to obtain the weight coefficients of each convolution kernel in the dynamic convolution layer;

[0024] The weight coefficients of each convolution kernel of the dynamic convolution layer are normalized by the Sigmoid activation function to obtain the network weights of each convolution kernel of the dynamic convolution layer.

[0025] Furthermore, the target template feature is used to track the target to be tracked in each frame of the video image sequence, including:

[0026] The search area of ​​the current frame image is obtained by cutting the preset size with the target area of ​​the previous frame image as the center;

[0027] After extracting the image features of the current frame image search area, interoperating with the target template features to obtain a feature response map of the current frame image search area;

[0028] Based on the feature response map of the search area of ​​the current frame image and the size of the target area, classification regression is used to predict the target bounding box of the current frame image, and the predicted target area is identified; wherein the information of the target area includes the target area position, the target area response score and the average peak correlation energy.

[0029] Furthermore, the average peak correlation energy of the target area is calculated using the following formula:

[0030]

[0031] Among them, F max Indicates the highest response value in the feature response map of the search area of ​​the current frame image; F min Indicates the lowest response value in the feature response map of the search area of ​​the current frame image; N anchors Indicates the number of candidate anchor boxes in the search area of ​​the current frame image; F i Represents the response value of the i-th candidate anchor box;

[0032] The candidate anchor point frame is a window of a preset size centered on each pixel in the search area of ​​the current frame image;

[0033] The response value of the candidate anchor point frame is an intersection-over-union ratio between the candidate anchor point frame and the target area.

[0034] Furthermore, the method of obtaining the confidence score of the target area of ​​the current frame image includes: passing the position of the target area of ​​the current frame image, the target area response score, the average peak correlation energy and the position of the target area of ​​the previous frame image, and the intersection and union ratio of the target area position of the previous frame image and the target area position of the current frame image through three fully connected layers containing activation functions in sequence to obtain the confidence score of the target area of ​​the current frame image.

[0035] Furthermore, the image of the target to be tracked is obtained by capturing the first frame of the video image sequence including the target to be tracked.

[0036] Furthermore, the target tracking model includes a target template feature generation module, a target tracking module and a confidence assessment module, which are trained by the following method:

[0037] Constructing training data sets for a target template feature generation module, a target tracking module, and a confidence assessment module respectively; wherein the training data set for the target template feature generation module includes sample video images and actual target images corresponding to each frame image; the training data set for the target tracking module includes the same sample video images and annotated target regions of each frame image as the training data set for the target template feature generation module; the training data set for the confidence assessment module includes the target region of each frame image and annotated confidence scores;

[0038] Loading the training data set of the target template feature generation module, training the target template feature generation module using an improved structural similarity index measurement function, and updating the parameters of the target template feature generation module by back propagation using a stochastic gradient descent method to obtain a trained target template feature generation module;

[0039] Loading the training data set of the target tracking module, training the target tracking module using the cross entropy loss function, and updating the parameters of the target tracking module using the stochastic gradient descent method for back propagation to obtain a trained target tracking module;

[0040] Loading the training data set of the confidence assessment module, training the confidence assessment module using the cross entropy loss function, and updating the parameters of the confidence assessment module by back propagation using the stochastic gradient descent method to obtain a trained confidence assessment module;

[0041] Based on the trained target template feature generation module, the trained target tracking module and the trained confidence assessment module, a trained target tracking model is obtained.

[0042] Furthermore, the improved structural similarity index measurement function has the following formula:

[0043]

[0044]

[0045] Among them, T x represents the target template feature map generated by the target template feature generation module; T y Represents the feature map of the actual target image; C represents the number of channels of the feature map; m represents the mth channel of the feature map; N represents the number of eigenvalues; and n represents the nth eigenvalue.

[0046] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0047] 1. The present invention generates predicted target template features by using target feature values ​​and the latest and reliable feature template set, integrating the most recent target change information, and can obtain a more accurate updated template. The fusion of the initial target and multiple features containing the target image helps to obtain a more accurate target description.

[0048] 2. The present invention continuously updates the target template features. When the appearance of the tracked target changes significantly, such as in a changing and complex environment such as motion, deformation, occlusion, etc., it is not easy to cause the tracked target to be lost. That is, the target can still be tracked when the appearance or features change.

[0049] 3. The present invention determines whether to update the feature template set by tracking the confidence score of the target, thereby ensuring the use of high-quality target features for updating and reducing the noise caused by adding unreliable targets, thereby enhancing the model's anti-drift capability.

[0050] 4. The present invention uses a target template feature generation module and dynamic convolution to learn specific convolution kernel parameters for each set of spatiotemporal information, which can increase the size and capacity of the model while ensuring the efficiency and real-time performance of the tracking process.

[0051] 5. The present invention enhances the sensitivity of the model to the similarity between the target features and the search frame image features by minimizing the improved structural similarity index metric (SSIM) as the loss function, which helps to improve the stability of tracking.

[0052] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0054] Figure 1 It is a flowchart of a data processing method for updating a target image based on context information feature fusion in an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a process for generating target template features in an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of the structure of the target template generation module in the embodiment of the present invention. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0058] A specific embodiment of the present invention discloses a data processing method for updating a target image based on context information feature fusion, such as Figure 1 As shown, the steps S1 to S2 are included:

[0059] Step S1: Acquire a video image sequence including a target to be tracked and an image of the target to be tracked.

[0060] Specifically, a video image including a target to be tracked is obtained and each frame of the video image is captured using OpenCV. Exemplarily, the target to be tracked may be a pedestrian, an animal or other object in the video image.

[0061] Furthermore, an image of the target to be tracked is captured from the first frame of the video image sequence.

[0062] In this embodiment, an image of 127×127 in size is cropped from the first frame of the video image sequence with the target to be tracked as the center, as the target image to be tracked.

[0063] Step S2, inputting the target image to be tracked and the video image sequence into a trained target tracking model to obtain a target tracking result; wherein the target tracking model is used to track each frame image in the video image sequence using a target template feature to obtain the target to be tracked; the target template feature is dynamically updated according to the confidence score of the target area identified in each frame image in the video image sequence.

[0064] Furthermore, the target template feature is generated based on the target image to be tracked and the target template feature set; wherein the initial value of the target template feature set is a number of feature copies of the target image to be tracked, and is dynamically updated according to the confidence score of the target area identified in each frame image.

[0065] Specifically, the target template feature is composed of the feature T0 of the target image to be tracked and the target template feature set T k (k=1, 2, 3 . . . K) is constructed, wherein K represents the number of target template features in the target template feature set. Exemplarily, K is set to 10 in this embodiment.

[0066] The feature T0 of the target image to be tracked is obtained by capturing and extracting the features of the first frame of the video image sequence. It is the feature information initially set as the tracking target and will not be updated or replaced. In the target template feature set initialization stage, the feature T0 of the target image to be tracked is copied into K feature copies and stored in the target template feature set T k In the tracking process, the latest and most effective target features will be selected to update the target template feature set T k The update strategy used in the feature is the first-in-first-out principle.

[0067] Furthermore, the target template feature is generated based on the target image to be tracked and the target template feature set, such as Figure 2 As shown, it includes steps S201 to S206:

[0068] Step S201: extracting features of the target image to be tracked.

[0069] Specifically, the feature map of an image is a set of data extracted from the original image through a specific algorithm or model in image processing and computer vision. These data represent certain features or attributes of the image. The feature map is usually used to represent important information in the image. The method for extracting image features can use edge detection methods, deep learning methods, etc., and the present invention does not limit this.

[0070] Step S202: performing feature splicing on the features of the target image to be tracked and the target template features in the target template feature set to obtain spliced ​​features.

[0071] Specifically, the features of the target image T0 to be tracked and the target template feature set T extracted in step S201 are k The features in are concatenated in sequence to obtain the concatenated features.

[0072] Step S203: The concatenated features are sequentially passed through two two-dimensional convolutional layers to perform dimensionality reduction operations on the concatenated features to obtain dimensionality reduced features.

[0073] Specifically, the convolution layer can extract features from the input data. The first two-dimensional convolution layer is used to extract simple features, such as edges and corners. The second two-dimensional convolution layer is used to extract more complex features, such as higher-level features composed of basic features. At the same time, after the convolution layer, the pooling layer is used to reduce the spatial dimension of the feature map, thereby reducing the number of parameters and computational complexity, while making feature detection more robust.

[0074] Step S204: Based on the reduced-dimensional features, a dynamic convolution method is used to obtain the network weights of the convolution kernels of the dynamic convolution layer, and the weighted sum of the convolution kernels in the dynamic convolution layer is calculated.

[0075] Specifically, the dynamic convolution method is used to perform convolution operations in combination with the features after dimensionality reduction, which can achieve adaptive feature extraction and enhancement to improve the accuracy and robustness of the target tracking algorithm. The network can dynamically adjust the number of convolution kernels and the network weights of the convolution kernels according to the input data to capture more complex and subtle features, while optimizing the feature fusion process, better processing the spatiotemporal information in the video sequence, and generating more accurate target templates, thereby effectively responding to the challenges of target appearance changes and complex environments.

[0076] Furthermore, the network weights of each convolution kernel of the dynamic convolution layer are obtained by a dynamic convolution method, including steps S2041 to S2043:

[0077] Step S2041: perform an average pooling operation on the reduced-dimensional features to further reduce the spatial dimension of the features.

[0078] Specifically, average pooling can reduce the spatial size of the feature map and the dimension of the feature, thereby reducing the amount of computation for subsequent processing. At the same time, through the pooling operation, features can be further abstracted, important information can be retained, and some unnecessary details or noise can be removed.

[0079] Step S2042: The features after the average pooling operation are passed through a fully connected layer to obtain the weight coefficients of each convolution kernel of the dynamic convolution layer.

[0080] Specifically, the fully connected layer can learn to effectively linearly combine the features extracted by the convolution layer through a large number of weight parameters. These weight coefficients represent the contribution of different features to the final task, so that the network can identify and strengthen the features that are useful for the prediction results. Through model training, the model can use different numbers of convolution kernels according to different features in the image, thereby obtaining the network weight parameters of each convolution kernel, so as to most effectively utilize the features extracted by the convolution kernel and obtain the features that are closest to the actual prediction target template.

[0081] Step S2043: Normalize the weight coefficients of each convolution kernel of the dynamic convolution layer through the Sigmoid activation function to obtain the network weights of each convolution kernel of the dynamic convolution layer.

[0082] Specifically, the formula for obtaining the network weights of each convolution kernel of the dynamic convolution layer based on the dimensionally reduced features by the dynamic convolution method is as follows:

[0083] a j =Sigmoid(FC(GAP(X)))

[0084] Among them, a j represents the network weight of the j-th convolution kernel; X represents the feature after dimensionality reduction; GAP() represents the average pooling operation; FC() represents the operation through the fully connected layer; Sigmoid() represents the normalization of the weight coefficient to the [0,1] interval through the Sigmoid function.

[0085] It should be noted that based on the network weights of each convolution kernel in the dynamic convolution layer, the weighted sum of the convolution kernels in the dynamic convolution layer is calculated. The features extracted by multiple convolution kernels can be fused to form a more comprehensive feature representation, which helps to capture different aspects of the input data.

[0086] Step S205: performing a convolution operation on the weighted sum of the convolution kernels in the dynamic convolution layer and the dimensionality reduced features to obtain convolution features.

[0087] Specifically, the weighted sum of the convolution kernels in the dynamic convolution layer is convolved with the reduced-dimensional features. By adaptively adjusting the importance of different convolution kernels, the optimal extraction and integration of input features is achieved, which enhances the model's ability to capture key information. At the same time, the computational complexity is reduced, making the model more flexible and efficient.

[0088] Step S206: The convolutional features are subjected to batch normalization and activation function to obtain target template features.

[0089] Specifically, using batch normalization and ReLU activation function to process the convolutional features can accelerate model training convergence, enhance model stability and generalization ability, and enrich feature representation by introducing nonlinearity, so that the model can more effectively learn and adapt to target search and matching tasks in dynamic environments.

[0090] Furthermore, dynamically updating the target template features according to the confidence scores of the target regions identified in each frame image in the video image sequence includes:

[0091] Obtaining a target region of a current frame image and a confidence score of the target region of the current frame image based on the current target template feature recognition;

[0092] When the confidence score is greater than the average confidence score of the target region of the historical frame, one of the feature copies is replaced with the target image feature in the target region to update the target template feature set.

[0093] Specifically, in the target tracking process, when the target is blocked or the target position is not accurate enough, the predicted target will contain noise. In order to avoid adding the features of the wrong predicted target into the sample set and affecting the subsequent tracking effect, features with high reliability must be selected and added into the template set. In the present invention, the target area identified by the current target template feature is used to calculate the confidence score of the target area of ​​the current frame based on the information contained in the target area. The confidence score can be used as a filtering mechanism to exclude low-confidence predictions and reduce the impact of these errors on tracking performance. The target image features of the target area with a high confidence score are updated in the target template feature set to improve the accuracy of the target during the tracking process. Further, the target template feature is used to track the target to be tracked in each frame of the video image sequence, including steps S211-step S213:

[0094] Step S211 , cropping a target area of ​​a previous frame image to a preset size as the center to obtain a search area of ​​the current frame image.

[0095] Specifically, in a video sequence, based on the premise that the movement of the target object between consecutive frames is usually continuous and gentle, and no violent jumps occur, by limiting the search area, the number of pixels that need to be processed in each frame can be greatly reduced, thereby reducing the computational cost. In this embodiment, a 303×303 image is cropped with the target area of ​​the previous frame as the center as the search area.

[0096] Step S212: extracting the image features of the current frame image search area and performing interoperation with the target template features to obtain a feature response map of the current frame image search area.

[0097] Specifically, the image features of the current frame image search area and the target template features are subjected to feature interaction using methods including but not limited to dot product operations, Euclidean distance calculations, cosine similarity calculations, and the like. A response map is generated by calculating the similarity or difference between the image features of the current frame image search area and the target template features, wherein the response value of each pixel represents the degree of matching, and locations with high response values ​​may be the true locations of the targets.

[0098] Step S213, based on the feature response map of the search area of ​​the current frame image and the size of the target area, use classification regression to predict the target bounding box of the current frame image and identify the predicted target area; wherein the information of the target area includes the target area position, the target area response score and the average peak correlation energy.

[0099] Specifically, the response peak area in the feature response map of the search area of ​​the current frame image is centered, and a candidate target bounding box is generated according to the size of the target area. For multiple overlapping candidate bounding boxes, non-maximum suppression (NMS) is used to select the best bounding box and remove redundant candidate areas. A classifier is applied to determine whether the candidate area contains the target, and a regressor is used to adjust the position and size of the bounding box to match the target more accurately.

[0100] It should be noted that, in the present invention, the template branch of the twin network is used to obtain the target area and target area information in the current frame image. The twin network can use SiameseFC or SiamRPN++, and the present invention does not limit it.

[0101] On the other hand, the average peak correlation energy (APCE) in the information of the target area is used to indicate the degree of fluctuation of the feature response graph obtained by the interaction between the target features and the search frame special frame. A significant decrease in the APCE value indicates that the fluctuation of the response graph has increased, indicating that the target may be occluded or lost.

[0102] In this embodiment, the average peak correlation energy (apce) of the target area is calculated to evaluate the fluctuation degree of the target response and determine whether the tracked target is blocked.

[0103] Furthermore, the average peak correlation energy of the target area is calculated using the following formula:

[0104]

[0105] Among them, F max Indicates the highest response value in the feature response map of the search area of ​​the current frame image; F min Indicates the lowest response value in the feature response map of the search area of ​​the current frame image; N anchors Indicates the number of candidate anchor boxes in the search area of ​​the current frame image; F i Represents the response value of the i-th candidate anchor box.

[0106] The candidate anchor point frame is a window of a preset size centered on each pixel in the search area of ​​the current frame image; the preset size is the same as the target size, or is a multiple of the target size.

[0107] The response value of the candidate anchor point frame is an intersection-over-union ratio between the candidate anchor point frame and the target area.

[0108] Specifically, the intersection-over-union ratio of the candidate anchor point frame and the target area can help determine the degree of overlap between the candidate anchor point frame and the real target area, thereby being used to evaluate the accuracy of positioning.

[0109] Furthermore, obtaining the confidence score of the target area of ​​the current frame image includes: passing the position of the target area of ​​the current frame image, the target area response score, the average peak correlation energy and the position of the target area of ​​the previous frame image, and the intersection and union ratio of the target area position of the previous frame image and the target area position of the current frame image through three fully connected layers containing activation functions in sequence to obtain the confidence score of the target area of ​​the current frame image.

[0110] Specifically, the intersection-over-union (IOU) between the position of the target region in the previous frame image and the position of the target region in the current frame image is calculated, which is an indicator for evaluating the degree of overlap between the two bounding boxes.

[0111] The position of the target area of ​​the current frame image, the target area response score, the average peak correlation energy and the position of the target area of ​​the previous frame image, and the intersection and union ratio of the target area position of the previous frame image and the target area position of the current frame image are integrated into a feature vector and input into the first fully connected layer of the neural network. Matrix multiplication is performed in the first fully connected layer to transform the input features into a new high-dimensional space, and an activation function is applied to the output of the first fully connected layer to introduce nonlinearity and help the network learn complex feature representations; the output of the activation function connected to the first fully connected layer is used as the input of the second fully connected layer, and the matrix multiplication operation is also performed to transform the input features into another dimensional space, and the activation function is applied to the output of the second fully connected layer; the output of the activation function connected to the second fully connected layer is used as the input of the third fully connected layer, and matrix multiplication and activation function operations are performed again; the output of the activation function connected to the third fully connected layer is directly mapped to a value, which is the confidence score of the target area of ​​the current frame image.

[0112] Furthermore, when the confidence score of the target region in the current frame is greater than the average confidence score of the target region in the historical frame, it means that the recognition of this target region has a higher confidence level, and its features are extracted and the target template feature set T is updated using the first-in-first-out method. k , that is, replace the target template feature with the longest time in the target template feature set with the target area feature of the current array.

[0113] Furthermore, the target tracking model includes a target template feature generation module, a target tracking module and a confidence assessment module.

[0114] The target template feature generation module is used to dynamically update the target template feature according to the target image tracked in each frame;

[0115] The target tracking module is used to track each frame of the video image sequence using the target template feature to obtain the target to be tracked;

[0116] The confidence evaluation module is used to evaluate the confidence of the target image tracked in each frame to determine whether to update the target template feature.

[0117] Furthermore, the target tracking model is obtained by training through the following method:

[0118] Constructing training data sets for a target template feature generation module, a target tracking module, and a confidence assessment module respectively; wherein the training data set for the target template feature generation module includes sample video images and actual target images corresponding to each frame image; the training data set for the target tracking module includes the same sample video images and annotated target regions of each frame image as the training data set for the target template feature generation module; the training data set for the confidence assessment module includes the target region of each frame image and annotated confidence scores;

[0119] Specifically, this embodiment uses the training set of LaSOT for training. The training set of LaSOT is a long-term tracking data set. The test set contains nearly 1,400 video sequences and 280 videos. Each video has an average of about 2,500 frames. The data set has 70 categories. 30 videos are randomly selected and the first 1,000 frames of the video are obtained.

[0120] The training data set of the target template feature generation module is loaded, the target template feature generation module is trained using an improved structural similarity index measurement function, and the parameters of the target template feature generation module are updated by back propagation using a stochastic gradient descent method to obtain a trained target template feature generation module.

[0121] The target template feature generation module includes an initial target feature extraction module, a feature template set management module and a target template generation module.

[0122] The initial target feature extraction module uses a residual neural network to extract features of the target image to be tracked;

[0123] The feature template management module is used to determine whether to update the target template feature set based on the confidence score of the target area tracked by the current frame image, and to extract the features of the target area to be updated and update the target template feature set;

[0124] The target template generation module is as follows Figure 3As shown, it includes a feature splicing layer, two two-dimensional convolution layers, a dynamic convolution module, a convolution layer and a batch normalization and activation function layer in sequence; wherein the feature splicing layer is used to perform feature splicing on the features of the target image to be tracked and the target template features in the target template feature set; the two two-dimensional convolution layers are used to perform dimensionality reduction operation on the spliced ​​features to obtain the reduced-dimensional features X; the dynamic convolution module includes an average pooling layer, a fully connected layer and an activation function layer, which is used to obtain the network weights (a of each convolution kernel of the dynamic convolution layer) j ) and the corresponding convolution kernel W j The weighted sum is performed; the convolution layer is used to perform a convolution operation on the weighted sum of the feature X after dimensionality reduction and the convolution kernel in the dynamic convolution layer; the batch normalization and activation function layer is used to obtain the target template feature.

[0125] Furthermore, the improved structural similarity index measurement function has the following formula:

[0126]

[0127] Among them, T x represents the target template feature map generated by the target template feature generation module; T y Represents the feature map of the actual target image; C represents the number of channels of the feature map; m represents the mth channel of the feature map; N represents the number of eigenvalues; and n represents the nth eigenvalue.

[0128] Specifically, in order to describe the similarity between the predicted target features and the actual target features, an improved structural similarity index metric ssim is used as the loss function of the present invention in model training. Ssim_loss is used to measure the structural similarity between two images, which is more representative of the similarity between two features than the Euclidean distance. The value range of ssim_loss is between [0,1]. The larger the value of ssim_loss, the more similar the two features are. The training of the target template feature generation module is completed by this method.

[0129] Furthermore, a training data set of the target tracking module is loaded, the target tracking module is trained using a cross entropy loss function, and the parameters of the target tracking module are updated by back propagation using a stochastic gradient descent method to obtain a trained target tracking module.

[0130] The target tracking module uses the template branch of the twin network to obtain the target area of ​​the current frame image based on the current target template feature recognition. The twin network can use SiameseFC or SiamRPN++, which is not limited by the present invention.

[0131] Furthermore, the training data set of the confidence assessment module is loaded, the confidence assessment module is trained using the cross entropy loss function, and the parameters of the confidence assessment module are updated by back propagation using the stochastic gradient descent method to obtain a trained confidence assessment module.

[0132] The confidence assessment module includes three fully connected layers including activation functions connected in sequence, which are used to obtain the confidence score of the target area of ​​the current frame image based on the position of the target area of ​​the current frame image, the target area response score, the average peak correlation energy and the position of the target area of ​​the previous frame image, and the intersection and union ratio of the target area position of the previous frame image and the target area position of the current frame image.

[0133] Specifically, the cross entropy loss function formula of the training confidence assessment module is as follows:

[0134]

[0135] Among them, NS represents the number of samples in the training set; p l Represents the confidence score of the output of the lth sample; y l Represents the sample label of the lth sample.

[0136] Furthermore, based on the trained target template feature generation module, the trained target tracking module and the trained confidence assessment module, a trained target tracking model is obtained.

[0137] In summary, the data processing method for updating a target image based on context information feature fusion according to an embodiment of the present invention has the following beneficial effects:

[0138] 1. The present invention generates predicted target template features by using target feature values ​​and the latest and reliable feature template set, integrating the most recent target change information, and can obtain a more accurate updated template. The fusion of the initial target and multiple features containing the target image helps to obtain a more accurate target description.

[0139] 2. The present invention continuously updates the target template features. When the appearance of the tracked target changes significantly, such as in a changing and complex environment such as motion, deformation, occlusion, etc., it is not easy to cause the tracked target to be lost. That is, the target can still be tracked when the appearance or features change.

[0140] 3. The present invention determines whether to update the feature template set by tracking the confidence score of the target, thereby ensuring the use of high-quality target features for updating and reducing the noise caused by adding unreliable targets, thereby enhancing the model's anti-drift capability.

[0141] 4. The present invention uses a target template feature generation module and dynamic convolution to learn specific convolution kernel parameters for each set of spatiotemporal information, which can increase the size and capacity of the model while ensuring the efficiency and real-time performance of the tracking process.

[0142] 5. The present invention enhances the sensitivity of the model to the similarity between the target features and the search frame image features by minimizing the improved structural similarity index metric (SSIM) as the loss function, which helps to improve the stability of tracking.

[0143] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A data processing method for updating a target image based on context information feature fusion, characterized in that: The steps include: Acquire a video image sequence including a target to be tracked and an image of the target to be tracked; The target image to be tracked and the video image sequence are input into a trained target tracking model to obtain a target tracking result; wherein the target tracking model is used to track each frame image in the video image sequence using a target template feature to obtain the target to be tracked; the target template feature is dynamically updated according to the confidence score of the target area identified in each frame image in the video image sequence.

2. The method according to claim 1, characterized in that: Generate the target template feature based on the target image to be tracked and the target template feature set; wherein the initial value of the target template feature set is a number of feature copies of the target image to be tracked; Dynamically updating the target template feature according to the confidence score of the target area identified in each frame image in the video image sequence, including: Obtaining a target region of a current frame image and a confidence score of the target region of the current frame image based on the current target template feature recognition; When the confidence score is greater than the average confidence score of the target region of the historical frame, one of the feature copies is replaced with the target image feature in the target region to update the target template feature set.

3. The method according to claim 2, characterized in that: Generating the target template feature based on the target image to be tracked and the target template feature set includes: Extracting features of the target image to be tracked; Performing feature splicing on the features of the target image to be tracked and the target template features in the target template feature set to obtain spliced ​​features; The concatenated features are sequentially passed through two two-dimensional convolutional layers to perform dimensionality reduction operations on the concatenated features to obtain dimensionality reduced features; Based on the reduced-dimensional features, a dynamic convolution method is used to obtain the network weights of each convolution kernel of the dynamic convolution layer, and a weighted sum of the convolution kernels in the dynamic convolution layer is calculated; Performing a convolution operation on the weighted sum of the convolution kernels in the dynamic convolution layer and the dimensionally reduced features to obtain convolutional features; The convolutional features are batch normalized and activated to obtain target template features.

4. The method according to claim 3, characterized in that: Based on the reduced-dimensional features, the network weights of each convolution kernel of the dynamic convolution layer are obtained by a dynamic convolution method, including: Performing an average pooling operation on the reduced-dimensional features to further reduce the spatial dimension of the features; The features after the average pooling operation pass through the fully connected layer to obtain the weight coefficients of each convolution kernel in the dynamic convolution layer; The weight coefficients of each convolution kernel of the dynamic convolution layer are normalized by the Sigmoid activation function to obtain the network weights of each convolution kernel of the dynamic convolution layer.

5. The method according to claim 2, characterized in that: The target template feature is used to track the target to be tracked in each frame of the video image sequence, including: The search area of ​​the current frame image is obtained by cutting the preset size with the target area of ​​the previous frame image as the center; After extracting the image features of the current frame image search area, interoperating with the target template features to obtain a feature response map of the current frame image search area; Based on the feature response map of the search area of ​​the current frame image and the size of the target area, classification regression is used to predict the target bounding box of the current frame image, and the predicted target area is identified; wherein the information of the target area includes the target area position, the target area response score and the average peak correlation energy.

6. The method according to claim 5, characterized in that: The average peak correlation energy of the target region was calculated using the following formula: Among them, F max Indicates the highest response value in the feature response map of the search area of ​​the current frame image; F min Indicates the lowest response value in the feature response map of the search area of ​​the current frame image; N anchors Indicates the number of candidate anchor boxes in the search area of ​​the current frame image; F i Represents the response value of the i-th candidate anchor box; The candidate anchor point frame is a window of a preset size centered on each pixel in the search area of ​​the current frame image; The response value of the candidate anchor point frame is an intersection-over-union ratio between the candidate anchor point frame and the target area.

7. The method according to claim 6, characterized in that: The method of obtaining the confidence score of the target area of ​​the current frame image includes: passing the position of the target area of ​​the current frame image, the target area response score, the average peak correlation energy and the position of the target area of ​​the previous frame image, and the intersection and union ratio of the target area position of the previous frame image and the target area position of the current frame image through three fully connected layers containing activation functions in sequence to obtain the confidence score of the target area of ​​the current frame image.

8. The method according to any one of claims 1 to 7, characterized in that: The image of the target to be tracked is obtained by capturing the first frame of the video image sequence including the target to be tracked.

9. The method according to claim 1, characterized in that: The target tracking model includes a target template feature generation module, a target tracking module and a confidence assessment module, which are trained by the following method: Constructing training data sets for a target template feature generation module, a target tracking module, and a confidence assessment module respectively; wherein the training data set for the target template feature generation module includes sample video images and actual target images corresponding to each frame image; the training data set for the target tracking module includes the same sample video images and annotated target regions of each frame image as the training data set for the target template feature generation module; the training data set for the confidence assessment module includes the target region of each frame image and annotated confidence scores; Loading the training data set of the target template feature generation module, training the target template feature generation module using an improved structural similarity index measurement function, and updating the parameters of the target template feature generation module by back propagation using a stochastic gradient descent method to obtain a trained target template feature generation module; Loading the training data set of the target tracking module, training the target tracking module using the cross entropy loss function, and updating the parameters of the target tracking module using the stochastic gradient descent method for back propagation to obtain a trained target tracking module; Loading the training data set of the confidence assessment module, training the confidence assessment module using the cross entropy loss function, and updating the parameters of the confidence assessment module by back propagation using the stochastic gradient descent method to obtain a trained confidence assessment module; Based on the trained target template feature generation module, the trained target tracking module and the trained confidence assessment module, a trained target tracking model is obtained.

10. The method according to claim 9, characterized in that: The improved structural similarity index measurement function has the following formula: Among them, T x represents the target template feature map generated by the target template feature generation module; T y Represents the feature map of the actual target image; C represents the number of channels of the feature map; m represents the mth channel of the feature map; N represents the number of eigenvalues; and n represents the nth eigenvalue.

Images