Video target tracking method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a video target tracking method and device, equipment, a storage medium and a program product, and relates to the field of big data. The method comprises the steps that after appearance feature matrixes of target images in video frames are determined, correlation filtering operation is carried out on the appearance feature matrixes of the target images in the two adjacent video frames to obtain response matrixes, and according to the maximum response value of the response matrixes and the weighting result of the deviation distance, the deviation distance of the target images in the two adjacent video frames is calculated. According to the method, the similarity of the appearance characteristics of the target images in the two adjacent video frames is determined, and compared with calculation of a cosine distance for measurement, the measurement mode does not excessively depend on the recognition performance of a detector in the target detection stage, and the accuracy of the detector is better inclusive. Moreover, the motion feature similarity of the target images in the two adjacent video frames is determined, and the motion trail of the target object is determined by combining the appearance feature similarity and the motion feature similarity, so that the accuracy of the motion trail is further improved, and target tracking is effectively realized.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a video target tracking method, apparatus, device, storage medium and program product. Background Art

[0002] In places like banks and shopping malls, intelligent video analysis technology is often used to provide early warnings of abnormal behavior and manage risks. For example, intelligent video analysis systems monitor bank lobbies, ATM areas, and other key locations in real time to identify unusual behaviors such as prolonged stays or the placement of suspicious items. Object tracking is a key task in intelligent video analysis, aiming to continuously track the movement of a target within a video sequence.

[0003] In existing target tracking methods, cosine distance is mainly used to calculate the similarity of the appearance features of each target in adjacent video frames, and then the same target in different video frames is connected in series based on the similarity of appearance features, and finally the motion trajectory of each target in the entire video stream is constructed.

[0004] However, when using cosine distance to measure the appearance features of the target, the extracted target appearance features are required to be one-dimensional feature vectors, which will overly rely on the recognition performance of the detector. When the detector is not accurate enough in target recognition and there is a certain offset in the identified target position, the similarity measurement will be inaccurate, making it impossible to construct an accurate motion trajectory. Summary of the Invention

[0005] The present application provides a video target tracking method, apparatus, device, storage medium and program product for constructing an accurate motion trajectory.

[0006] In a first aspect, the present application provides a video target tracking method, the method comprising:

[0007] Detecting multiple target objects in each video frame of the video stream to obtain multiple target images;

[0008] Determining an appearance feature matrix of the target image, performing a correlation filtering operation on the appearance feature matrix of each target image in two adjacent video frames to obtain a response matrix, and determining the similarity of appearance features of each target image in the two adjacent video frames based on a weighted result of a maximum response value and a deviation distance of the response matrix, where the deviation distance includes the distance of the maximum response value relative to the center position of the matrix;

[0009] Predicting the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame, and determining the similarity of motion features of each target image in two adjacent video frames based on the detected position and the predicted position of each target image in the current video frame;

[0010] The final similarity of each target image in two adjacent video frames is determined according to the appearance feature similarity and the motion feature similarity, and the motion trajectory of the target object is determined according to the final similarity to achieve target tracking.

[0011] In a possible implementation, determining the appearance feature matrix of the target image includes:

[0012] The target image is input into a convolutional neural network with an asymmetric convolution kernel to obtain an appearance feature matrix of the target image.

[0013] In a possible implementation, determining the final similarity of each target image in two adjacent video frames based on the appearance feature similarity and the motion feature similarity includes:

[0014] Obtaining a cost matrix according to a weighted result of the appearance feature similarity and the motion feature similarity;

[0015] The final similarity of each target image in two adjacent video frames is determined according to the cost matrix.

[0016] In a possible implementation, determining the final similarity of each target image in two adjacent video frames according to the cost matrix includes:

[0017] The optimal solution of the cost matrix is ​​solved according to the Hungarian algorithm to obtain the final similarity of each target image in two adjacent video frames.

[0018] In a possible implementation, determining the similarity of motion features of each target image in two adjacent video frames based on the detected position of each target image in the current video frame and the predicted position includes:

[0019] The Mahalanobis distance between the detected position of each target image in the current video frame and the predicted position is determined to obtain the similarity of motion features of each target image in two adjacent video frames.

[0020] In a possible implementation, predicting the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame includes:

[0021] Kalman filtering is used to predict the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame;

[0022] And / or, detecting a plurality of target objects in each video frame of the video stream includes:

[0023] The target detector detects multiple target objects in each video frame of the video stream.

[0024] In a second aspect, the present application provides a video target tracking device, the device comprising:

[0025] A detection module, configured to detect multiple target objects in each video frame of the video stream to obtain a target image;

[0026] a first processing module, configured to determine an appearance feature matrix of the target image, perform a correlation filtering operation on the appearance feature matrix of each target image in two adjacent video frames to obtain a response matrix, and determine the similarity of appearance features of each target image in the two adjacent video frames based on a weighted result of a maximum response value and a deviation distance of the response matrix, where the deviation distance includes the distance of the maximum response value relative to the center position of the matrix;

[0027] a second processing module, predicting a predicted position of each target image in a current video frame based on the detected position and motion parameters of each target image in a previous video frame, and determining similarity of motion features of each target image in two adjacent video frames based on the detected position and the predicted position of each target image in the current video frame;

[0028] The third processing module is used to determine the final similarity of each target image in two adjacent video frames according to the appearance feature similarity and the motion feature similarity, and determine the motion trajectory of the target object according to the final similarity to achieve target tracking.

[0029] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor;

[0030] The memory stores computer-executable instructions;

[0031] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0033] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0034] The video target tracking method, apparatus, device, storage medium and program product provided by the present application, after determining the appearance feature matrix of the target image in the video frame, performs a correlation filtering operation on the appearance feature matrix of each target image in two adjacent video frames to obtain a response matrix, and determines the appearance feature similarity of each target image in the two adjacent video frames according to the weighted result of the maximum response value and the deviation distance of the response matrix; predicts the predicted position of each target image in the current video frame according to the detection position and motion parameter of each target image in the previous video frame, and determines the motion feature similarity of each target image in the two adjacent video frames according to the detection position and the predicted position of each target image in the current video frame; then, determines the final similarity of each target image in the two adjacent video frames according to the appearance feature similarity and the motion feature similarity, and determines the motion trajectory of the target object according to the final similarity to achieve target tracking. The scheme of the present application, by introducing the correlation filtering operation into the measurement of the target appearance feature similarity, uses the maximum response value of the response matrix obtained by the operation and its offset for weighting, and jointly measures the similarity of the target appearance feature. Compared to calculating cosine distance, this metric is less reliant on the detector's recognition performance during the target detection phase and is more tolerant of detector accuracy. Furthermore, by combining appearance feature similarity with motion feature similarity, the target's trajectory is determined, further improving trajectory accuracy and effectively enabling target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] Figure 1 Schematic diagram of the process of video target tracking method provided in this application Figure 1 ;

[0037] Figure 2 Schematic diagram of the process of video target tracking method provided in this application Figure 2 ;

[0038] Figure 3 A schematic diagram of the structure of the video target tracking device provided in this application;

[0039] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application.

[0040] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0043] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0044] It should be noted that the video target tracking method, device, equipment, storage medium and product provided in this application can be used in the field of big data, and can also be used in any field other than the field of big data. The application field of the video target tracking method, device, equipment, storage medium and product in this application is not limited.

[0045] In banking facilities, intelligent video analytics systems monitor bank halls, ATM areas, or other key locations in real time to identify abnormal behavior. Object tracking is a key task in intelligent video analytics.

[0046] Currently, targets can be tracked using the Deep SORT (Simple Online and Realtime Tracking with a Deep Association Metric) algorithm. This algorithm is used for multi-target tracking. It uses an RCNN (Region-based Convolutional Neural Network) as a detector to identify each target in each video frame. A lightweight convolutional neural network is then used to extract appearance features from each target. The cosine distance is used to calculate the similarity of the appearance features of each target in adjacent video frames. Based on this similarity, the same target in different video frames is then concatenated, ultimately constructing the trajectory of each target throughout the entire video stream.

[0047] However, using lightweight convolutional neural networks to extract target appearance features is not robust to large pose changes and rotational deformations, making accurate target matching difficult. Furthermore, using cosine distance to measure target appearance requires the extracted features to be one-dimensional feature vectors, which overly relies on the detector's recognition performance. If the detector's target recognition is inaccurate and the identified target position is offset, the similarity measurement will be inaccurate, making it impossible to construct an accurate motion trajectory.

[0048] To this end, this application proposes a video target tracking method that incorporates correlation filtering operations into the measurement of target appearance feature similarity. The maximum response value of the response matrix obtained by the operation and its offset are weighted to jointly measure the similarity of the target appearance features. Compared to calculating the cosine distance for measurement, this measurement method does not overly rely on the recognition performance of the detector in the target detection stage and is more inclusive of the detector's accuracy. Furthermore, by combining the similarity of appearance features with the similarity of motion features, the motion trajectory of the target object is determined, further improving the accuracy of the motion trajectory and effectively achieving target tracking.

[0049] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0050] Figure 1 Schematic diagram of the process of video target tracking method provided in this application Figure 1 ,like Figure 1 As shown, the video target tracking method provided by this application may include:

[0051] S101 : Detect multiple target objects in each video frame of a video stream to obtain multiple target images.

[0052] For example, a video stream includes multiple video frames, and multiple target objects may be detected in each video frame. The target objects may include people, vehicles, and the like.

[0053] For example, a convolutional neural network that achieves target detection functionality while balancing speed and accuracy and supporting edge device adaptation can be used to quickly identify and label one or more target objects in a video frame.

[0054] For example, after detecting the target object, the attributes of the target image are obtained. The attributes of the target image may include the category of the target object (human, vehicle, animal, other object, etc.), the position of the target object, the size of the target image, etc.

[0055] In a possible implementation, a target detector is used to detect multiple target objects in each video frame of a video stream. The target detector can detect and identify target objects in the video stream in real time with high accuracy and precision.

[0056] For example, multiple target objects may be detected in each video stream using an existing target detector, for example, a target detection algorithm related to YOLO (You Only Look Once).

[0057] S102. Determine the appearance feature matrix of the target image, perform a correlation filtering operation on the appearance feature matrix of each target image in two adjacent video frames to obtain a response matrix, and determine the similarity of the appearance features of each target image in the two adjacent video frames based on the maximum response value of the response matrix and the weighted result of the deviation distance.

[0058] The deviation distance includes the distance of the maximum response value relative to the center position of the response matrix.

[0059] It should be noted that correlation filtering is a technique used to compare two signals or images. Correlation filtering can slide one feature matrix on another feature matrix, calculate the similarity score at each position, and generate a response matrix. Therefore, the response matrix is ​​the result of the correlation filtering operation, and each element represents the similarity score of the two feature matrices at the corresponding position.

[0060] The maximum response value indicates the highest similarity between the two feature matrices at a certain position. Generally, a higher maximum response value means that the two feature matrices have a higher similarity at this position, and a lower maximum response value means that the two feature matrices have a lower similarity at this position.

[0061] The offset distance refers to the distance between the position of the maximum response value and the center of the response matrix. It can be used to determine the degree of deviation between the best matching position of the two feature matrices and the expected center. If the position of the maximum response value is close to the center of the matrix, it means that the two feature matrices are highly aligned in space.

[0062] In the embodiments of the present application, by combining the maximum response value and the offset distance, a more comprehensive measure of the similarity between two target images can be achieved. The maximum response value provides the strength of the similarity, and the offset distance provides the spatial consistency of the similarity. At the same time, considering that when the position of the detected target object is not accurate enough, the target detection results may have a certain range of offset, the combined measurement of the maximum response value and the offset distance can make the results more accurate. Based on this, the similarity of the appearance features of two targets can be more accurately assessed.

[0063] For example, the weighting process can adjust the weights according to specific application scenarios so as to attach different importance to the strength and spatial consistency of similarity.

[0064] For ease of description, two adjacent video frames are referred to as the previous video frame and the current video frame, and a correlation filtering operation is performed on the appearance features of each target image in the previous video frame and the appearance features of each target image in the current video frame. Since the previous video frame includes multiple target images and the current video frame also includes multiple target images, a correlation filtering operation is performed on each target image in the previous video frame and each target image in the current video frame to obtain multiple response matrices, wherein one target image in the previous video frame and one target image in the current video frame correspond to one response matrix. Accordingly, the similarity of the appearance features of each target image refers to the similarity of the appearance features of each target image in the previous video frame and each target image in the current video frame in two adjacent video frames, that is, the similarity of the appearance features between any target image in the previous video frame and any target image in the current video frame.

[0065] To facilitate understanding of the solution of the present application, the following describes in detail the related filtering operations and the calculation process of the appearance feature similarity, assuming that the previous video frame includes target image 1, target image 2, and target image 3, and the current video frame includes target image 4, target image 5, and target image 6.

[0066] The appearance feature matrix of target image 1 and the appearance feature matrix of target image 4 are subjected to correlation filtering operation to obtain response matrix 1, the appearance feature matrix of target image 1 and the appearance feature matrix of target image 5 are subjected to correlation filtering operation to obtain response matrix 2, and the appearance feature matrix of target image 1 and the appearance feature matrix of target image 6 are subjected to correlation filtering operation to obtain response matrix 3.

[0067] The appearance feature matrix of target image 2 and the appearance feature matrix of target image 4 are subjected to correlation filtering operation to obtain response matrix 4, the appearance feature matrix of target image 2 and the appearance feature matrix of target image 5 are subjected to correlation filtering operation to obtain response matrix 5, and the appearance feature matrix of target image 2 and the appearance feature matrix of target image 6 are subjected to correlation filtering operation to obtain response matrix 6.

[0068] The appearance feature matrix of target image 3 and the appearance feature matrix of target image 4 are subjected to correlation filtering operation to obtain response matrix 7, the appearance feature matrix of target image 3 and the appearance feature matrix of target image 5 are subjected to correlation filtering operation to obtain response matrix 8, and the appearance feature matrix of target image 3 and the appearance feature matrix of target image 6 are subjected to correlation filtering operation to obtain response matrix 9.

[0069] Based on the maximum response value and deviation distance of response matrix 1, the appearance feature similarity between target image 1 and target image 4 can be measured; based on the maximum response value and deviation distance of response matrix 2, the appearance feature similarity between target image 1 and target image 5 can be measured; based on the maximum response value and deviation distance of response matrix 3, the appearance feature similarity between target image 1 and target image 6 can be measured. Based on the maximum response value and deviation distance of response matrix 4, the appearance feature similarity between target image 2 and target image 4 can be measured; based on the maximum response value and deviation distance of response matrix 5, the appearance feature similarity between target image 2 and target image 5 can be measured; based on the maximum response value and deviation distance of response matrix 6, the appearance feature similarity between target image 2 and target image 6 can be measured. Based on the maximum response value and deviation distance of response matrix 7, the appearance feature similarity between target image 3 and target image 7 can be measured; based on the maximum response value and deviation distance of response matrix 8, the appearance feature similarity between target image 3 and target image 8 can be measured; based on the maximum response value and deviation distance of response matrix 9, the appearance feature similarity between target image 3 and target image 9 can be measured.

[0070] The applicant also found that when using a lightweight convolutional neural network to extract the appearance features of a target, when the target undergoes large posture changes and rotational deformations, the extracted target appearance features are not robust, making it difficult to achieve accurate target matching.

[0071] Therefore, in one possible implementation, the target image is fed into a convolutional neural network with an asymmetric convolution kernel to obtain an appearance feature matrix for the target image. Embedding the asymmetric convolution kernel within a convolutional neural network creates an asymmetric convolutional network that can overcome rotational deformation when used to extract the target's appearance features, resulting in more robust appearance features.

[0072] Asymmetric convolution is an orthogonal approach that exploits the relationship between the kernel weights and their spatial positions to decompose a symmetric convolution kernel into multiple asymmetric branches. This decomposition reduces the computational complexity of each asymmetric branch, thus reducing the amount of computation required. During network training, the asymmetric branches perform convolution operations on the input separately. The convolution results of these branches are combined in subsequent layers to form a complete feature representation. This decomposition captures features of different orientations and scales, allowing the network to learn richer feature representations during training. During inference (i.e., model deployment and operational use), the weights of the asymmetric branches learned during training can be fused to form an enhanced symmetric convolution kernel. The fused kernel skeleton has a larger weight and is embedded in a convolutional neural network to form an asymmetric convolutional network for object appearance feature extraction. This makes the extracted appearance features rotationally invariant and robust to edge deformation, which helps achieve more accurate object matching.

[0073] For example, the original d×d convolution kernel is decomposed into three parallel convolution kernel branches of sizes d×d, 1×d, and d×1. During the network training phase, these three convolution kernel branches are separately convolved with the image to enrich the feature space. During the inference phase, the weights of the three convolution kernel branches are fused to form an enhanced d×d convolution kernel, which is used for the convolution operation when extracting the appearance features of the target object.

[0074] Accordingly, the detected target image is input into the convolutional neural network with the asymmetric convolution kernel described above. After processing by the convolutional neural network with the asymmetric convolution kernel, an appearance feature matrix of the target image can be obtained. The appearance feature matrix of each target image represents the appearance features of each target image. For example, the appearance feature matrix can be a W×H×C feature matrix, where W refers to the width, H refers to the height, and C refers to the number of channels.

[0075] In a possible implementation, features of the target image may be extracted using a symmetric convolutional neural network to obtain an appearance feature matrix.

[0076] S103. Predict the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame, and determine the similarity of motion characteristics of each target image in two adjacent video frames based on the detected position and predicted position of each target image in the current video frame.

[0077] In this embodiment of the present application, the predicted position of each target image in the current video frame is predicted based on the detected position and motion parameters of each target object in the previous video frame, that is, multiple predicted positions are obtained. At the same time, multiple target images are detected in the current video frame through step 1, resulting in multiple detected positions. Subsequently, similarity calculations are performed between the multiple detected positions in the current video frame and the multiple predicted positions to obtain motion feature similarity.

[0078] For example, the previous video frame includes target image 1, target image 2, and target image 3, and the current video frame includes target image 4, target image 5, and target image 6.

[0079] Based on the detection position 1 and motion parameter 1 of target image 1, target image 1 is predicted to be at prediction position 1 in the current video frame. Based on the detection position 2 and motion parameter 2 of target image 2, target image 2 is predicted to be at prediction position 2 in the current video frame. Based on the detection position 3 and motion parameter 3 of target image 3, target image 3 is predicted to be at prediction position 3 in the current video frame. The detection position of target image 4 in the current frame is recorded as detection position 4, the detection position of target image 5 in the current video frame is recorded as detection position 5, and the detection position of target image 6 in the current video frame is recorded as detection position 6.

[0080] A similarity calculation is performed between detection position 4 and prediction position 1 to obtain the motion feature similarity between target image 4 and target image 1. A similarity calculation is performed between detection position 4 and prediction position 2 to obtain the motion feature similarity between target image 4 and target image 2. A similarity calculation is performed between detection position 4 and prediction position 3 to obtain the motion feature similarity between target image 4 and target image 3. A similarity calculation is performed between detection position 5 and prediction position 1 to obtain the motion feature similarity between target image 5 and target image 1. A similarity calculation is performed between detection position 5 and prediction position 2 to obtain the motion feature similarity between target image 5 and target image 2. A similarity calculation is performed between detection position 5 and prediction position 3 to obtain the motion feature similarity between target image 5 and target image 3. A similarity calculation is performed between detection position 6 and prediction position 1 to obtain the motion feature similarity between target image 6 and target image 1. A similarity calculation is performed between detection position 6 and prediction position 2 to obtain the motion feature similarity between target image 6 and target image 2. A similarity calculation is performed between detection position 6 and prediction position 6 to obtain the motion feature similarity between target image 4 and target image 3.

[0081] In one possible implementation, the Mahalanobis distance between the detected and predicted positions of each target image in the current video frame is determined to obtain the similarity of motion features of each target image in two adjacent video frames. Because the Mahalanobis distance considers the covariance structure, it is insensitive to scale changes in the data. Therefore, even if the data has different units or scales in different dimensions, the Mahalanobis distance can still provide a consistent metric.

[0082] For example, the Mahalanobis distance between detection position 4 and prediction position 1 is calculated to obtain the motion feature similarity between target image 4 and target image 1; the Mahalanobis distance between detection position 4 and prediction position 2 is calculated to obtain the motion feature similarity between target image 4 and target image 2; and the Mahalanobis distance between detection position 4 and prediction position 3 is calculated to obtain the motion feature similarity between target image 4 and target image 3. The Mahalanobis distance between detection position 5 and prediction position 1 is calculated to obtain the motion feature similarity between target image 5 and target image 1; the Mahalanobis distance between detection position 5 and prediction position 2 is calculated to obtain the motion feature similarity between target image 5 and target image 2; and the Mahalanobis distance between detection position 5 and prediction position 3 is calculated to obtain the motion feature similarity between target image 5 and target image 3. The Mahalanobis distance between detection position 6 and prediction position 1 is calculated to obtain the motion feature similarity between target image 6 and target image 1; the Mahalanobis distance between detection position 6 and prediction position 2 is calculated to obtain the motion feature similarity between target image 6 and target image 2; and the Mahalanobis distance between detection position 6 and prediction position 6 is calculated to obtain the motion feature similarity between target image 4 and target image 3.

[0083] For example, the similarity of motion features of target images in two adjacent video frames may also be determined by using Euclidean distance, cosine distance, and the like.

[0084] In one possible implementation, a Kalman filter is used to predict the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame. The Kalman filter constructs the motion characteristics of the target image by analyzing its motion state (e.g., position, velocity, acceleration, etc.), predicting its possible position in the current video frame. Combined with the size of the target image, the predicted position of each target image in the current video frame can be obtained. The Kalman filter is computationally efficient and suitable for real-time applications, such as target tracking in video streams. It can also adapt to dynamic changes in target motion, allowing the Kalman filter to quickly adjust its predictions even if the target's motion pattern changes.

[0085] For example, the predicted position of the target image in the current video frame may also be predicted through motion model prediction, deep learning methods, and the like.

[0086] S104 , determining the final similarity of each target image in two adjacent video frames based on the appearance feature similarity and the motion feature similarity, and determining the motion trajectory of the target object based on the final similarity to achieve target tracking.

[0087] In the embodiments of the present application, the final similarity of each target image in two adjacent video frames is determined based on the similarity of the appearance features and motion features of each target image in two adjacent video frames. Accordingly, the final similarity of each target image in each pair of adjacent video frames in the video stream can be calculated, thereby determining the motion trajectory of each target image and achieving tracking of the target motion trajectory throughout the entire video stream.

[0088] In one possible implementation, a cost matrix is ​​obtained based on the weighted results of the similarity of appearance features and the similarity of motion features. Each element in the cost matrix represents the final similarity between a target image in the previous video frame and a target image in the current video frame. For example, the cost matrix represents the possibility of matching n known targets with m detection results identified in the current video frame. Therefore, based on the cost matrix, the final similarity of each target image in two adjacent video frames can be determined. The cost matrix provides a structured way to represent the cost of different options or solutions, so that the complexity of the problem can be simplified to matrix operations, and the cost matrix can be used in combination with a variety of optimization algorithms, such as the Hungarian algorithm for solving bipartite graph matching problems, linear programming for solving transportation problems, etc. These algorithms can efficiently process the cost matrix and find the optimal solution.

[0089] In one possible implementation, the Hungarian algorithm is used to find the optimal solution to the cost matrix, yielding the final similarity of each target image in two adjacent video frames. Using the Hungarian algorithm to find the optimal solution to the cost matrix allows for finding a matching solution with the lowest total cost in polynomial time. Solving the cost matrix yields a matching solution that indicates which detection result (the target image in the current frame) is most likely to match each known target (e.g., the target image in the previous frame).

[0090] The video target tracking method provided in this application incorporates correlation filtering operations into the measurement of target appearance feature similarity. The maximum response value and its offset of the response matrix obtained by the operation are weighted to jointly measure the similarity of the target appearance features. Compared with calculating the cosine distance for measurement, this measurement method does not overly rely on the recognition performance of the detector during the target detection phase and is more inclusive of the detector's accuracy. Furthermore, by combining the similarity of appearance features with the similarity of motion features, the motion trajectory of the target object is determined, further improving the accuracy of the motion trajectory and effectively achieving target tracking.

[0091] Figure 2 This is a flow chart of the video target tracking method of this application Figure 2 ,like Figure 2 As shown, the video target tracking method provided by this application includes:

[0092] S201 : Detect multiple target objects in each video frame of a video stream using a target detector to obtain multiple target images.

[0093] S202. Input the target image into a convolutional neural network with an asymmetric convolution kernel to obtain an appearance feature matrix of the target image, perform a correlation filtering operation on the appearance feature matrix of each target image in two adjacent video frames to obtain a response matrix, and determine the similarity of the appearance features of each target image in two adjacent video frames based on a weighted result of a maximum response value and a deviation distance of the response matrix.

[0094] S203. Use Kalman filtering to predict the predicted position of each target image in the current video frame based on the detection position and motion parameters of each target image in the previous video frame, determine the Mahalanobis distance between the detection position of each target image in the current video frame and the predicted position, and obtain the similarity of the motion features of each target image in two adjacent video frames.

[0095] S204. Obtain a cost matrix based on the weighted results of the appearance feature similarity and the motion feature similarity; solve the optimal solution of the cost matrix according to the Hungarian algorithm to obtain the final similarity of each target image in two adjacent video frames; and determine the motion trajectory of the target object based on the final similarity to achieve target tracking.

[0096] The video target tracking method provided by this application embeds an asymmetric convolution kernel into a convolutional neural network. The asymmetric convolutional network formed can overcome the rotational deformation of the target when used for target appearance feature extraction, and extract more robust features of the target appearance for target similarity measurement. In addition, the correlation filtering operation is introduced into the measurement of target appearance feature similarity, and the maximum response value and offset of the response matrix obtained by the operation are used for weighting to jointly measure the similarity of the target appearance features. Compared with calculating the cosine distance for measurement, this measurement method does not overly rely on the recognition performance of the detector in the target detection stage, and has better tolerance for the accuracy of the detector.

[0097] Figure 3 The schematic diagram of the structure of the video target tracking device provided in this application is as follows: Figure 3 As shown, the video target tracking device 10 provided in this embodiment includes:

[0098] A detection module 11 is configured to detect a plurality of target objects in each video frame of the video stream to obtain a plurality of target images;

[0099] A first processing module 12 is configured to determine an appearance feature matrix of the target image, perform a correlation filtering operation on the appearance feature matrix of each target image in two adjacent video frames to obtain a response matrix, and determine the similarity of appearance features of each target image in the two adjacent video frames based on a weighted result of a maximum response value and a deviation distance of the response matrix, where the deviation distance includes the distance of the maximum response value relative to the center position of the matrix;

[0100] A second processing module 13 is configured to predict the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame, and determine the similarity of motion features of each target image in two adjacent video frames based on the detected position and the predicted position of each target image in the current video frame;

[0101] The fourth processing module 14 is configured to determine the final similarity of each target image in two adjacent video frames based on the appearance feature similarity and the motion feature similarity, and determine the motion trajectory of the target object based on the final similarity to achieve target tracking.

[0102] In a possible implementation, the first processing module 12 is specifically configured to input the target image into a convolutional neural network with an asymmetric convolution kernel to obtain an appearance feature matrix of the target image.

[0103] In a possible implementation, the fourth processing module 14 is specifically configured to obtain a cost matrix based on a weighted result of the appearance feature similarity and the motion feature similarity; and determine a final similarity of each target image in two adjacent video frames based on the cost matrix.

[0104] In a possible implementation, the fourth processing module 14 is specifically configured to solve the optimal solution of the cost matrix according to the Hungarian algorithm to obtain the final similarity of each target image in two adjacent video frames.

[0105] In a possible implementation, the second processing module 13 is specifically configured to determine the Mahalanobis distance between the detected position of each target image in the current video frame and the predicted position, and obtain the similarity of motion features of each target image in two adjacent video frames.

[0106] In a possible implementation, the second processing module 13 is specifically configured to use Kalman filtering to predict the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame.

[0107] In a possible implementation, the detection module 11 is specifically configured to detect a plurality of target objects in each video frame of the video stream using a target detector.

[0108] The video target tracking device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0109] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 4 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus.

[0110] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that the at least one processor 501 performs the above method.

[0111] The specific implementation process of the processor 501 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0112] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0113] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0114] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0115] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0116] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0117] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0118] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0119] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0120] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0121] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0123] Those skilled in the art will understand that all or part of the steps for implementing the above-mentioned method embodiments can be accomplished by hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes various media that can store program codes, such as ROM, RAM, magnetic disks or optical disks. After considering the specification and practicing the invention disclosed herein, those skilled in the art will easily think of other implementation plans of the present application. This application is intended to cover any variations, uses or adaptive changes of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the technical field that are not disclosed in this application. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0124] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video target tracking method, characterized in that: The method comprises: Detecting multiple target objects in each video frame of the video stream to obtain multiple target images; Determining an appearance feature matrix of the target image, performing a correlation filtering operation on the appearance feature matrix of each target image in two adjacent video frames to obtain a response matrix, and determining the similarity of appearance features of each target image in the two adjacent video frames based on a weighted result of a maximum response value and a deviation distance of the response matrix, where the deviation distance includes the distance of the maximum response value relative to the center position of the matrix; Predicting the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame, and determining the similarity of motion features of each target image in two adjacent video frames based on the detected position and the predicted position of each target image in the current video frame; The final similarity of each target image in two adjacent video frames is determined according to the appearance feature similarity and the motion feature similarity, and the motion trajectory of the target object is determined according to the final similarity to achieve target tracking.

2. The method according to claim 1, characterized in that Determining the appearance feature matrix of the target image includes: The target image is input into a convolutional neural network with an asymmetric convolution kernel to obtain an appearance feature matrix of the target image.

3. The method according to claim 1, characterized in that Determining the final similarity of each target image in two adjacent video frames according to the appearance feature similarity and the motion feature similarity includes: Obtaining a cost matrix according to a weighted result of the appearance feature similarity and the motion feature similarity; The final similarity of each target image in two adjacent video frames is determined according to the cost matrix.

4. The method according to claim 3, characterized in that Determining the final similarity of each target image in two adjacent video frames according to the cost matrix includes: The optimal solution of the cost matrix is ​​solved according to the Hungarian algorithm to obtain the final similarity of each target image in two adjacent video frames.

5. The method according to claim 1, wherein The determining, based on the detected position and the predicted position of each target image in the current video frame, the similarity of motion features of each target image in two adjacent video frames includes: The Mahalanobis distance between the detected position of each target image in the current video frame and the predicted position is determined to obtain the similarity of motion features of each target image in two adjacent video frames.

6. The method according to claim 1, characterized in that The step of predicting the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame includes: Kalman filtering is used to predict the predicted position of each target image in the current video frame based on the detected position and motion parameters of each target image in the previous video frame; And / or, detecting a plurality of target objects in each video frame of the video stream includes: The target detector detects multiple target objects in each video frame of the video stream.

7. A video target tracking device, characterized in that: The device comprises: A detection module, configured to detect multiple target objects in each video frame of the video stream to obtain a target image; a first processing module, configured to determine an appearance feature matrix of the target image, perform a correlation filtering operation on the appearance feature matrix of each target image in two adjacent video frames to obtain a response matrix, and determine the similarity of appearance features of each target image in the two adjacent video frames based on a weighted result of a maximum response value and a deviation distance of the response matrix, where the deviation distance includes the distance of the maximum response value relative to the center position of the matrix; a second processing module, predicting a predicted position of each target image in a current video frame based on the detected position and motion parameters of each target image in a previous video frame, and determining similarity of motion features of each target image in two adjacent video frames based on the detected position and the predicted position of each target image in the current video frame; The third processing module is used to determine the final similarity of each target image in two adjacent video frames according to the appearance feature similarity and the motion feature similarity, and determine the motion trajectory of the target object according to the final similarity to achieve target tracking.

8. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images