Video coding motion estimation method and apparatus based on multi-target tracking
Through multi-objective tracking technology, using deep learning and Kalman filtering and other methods, the problem of limited search efficiency and accuracy in vigorous motion images in traditional video encoding methods is solved, and more efficient motion estimation and coding are achieved.
Patent Information
- Application Number
- PCT/CN2023/140277
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2023-12-20
- Publication Date
- 2024-12-05
AI Technical Summary
Traditional video encoding motion estimation methods cannot adaptively change according to the trajectory of objects in the image, resulting in limited search efficiency and accuracy in images with severe motion.
Using video encoding motion estimation method based on multi-object tracking, multi-object recognition and tracking of video images, using deep learning neural networks and Kalman filtering and other technologies, the feature vectors and distance costs between targets are calculated, and the targets are determined at different levels. The position information in the frame, thereby accurately matching the motion vectors of the encoded macroblocks.
It improves the search efficiency and accuracy of video encoding motion estimation, and can quickly find the best matching points in images with intense motion, improving coding efficiency.
Smart Images

Figure CN2023140277_05122024_PF_FP_ABST
Abstract
Description
A video coding motion estimation method and device based on multi-target tracking Technical Field
[0001] The present application belongs to the field of video coding technology, and in particular relates to a video coding motion estimation method and device based on multi-target tracking. Background Art
[0002] Among the current mainstream video coding methods, inter-frame predictive coding is a very important link, which can significantly reduce the redundancy of video data in the time domain and achieve better video coding compression effects.
[0003] Inter-frame prediction is accomplished through motion estimation of the coded macroblocks, which mainly searches for the most similar matching macroblock in the reference frame. However, traditional motion estimation methods do not utilize relevant information of the image itself. They usually use the current macroblock as the search starting point and then search in a fixed pattern. These traditional methods cannot adapt to the trajectory of the object in the image. For images with intense motion, multiple searches are required to find the matching point, which limits the search efficiency and accuracy.
[0004] Summary of the Invention
[0005] This application aims to solve the technical problems mentioned above, such as the inability of traditional methods to adaptively change according to the trajectory of the object in the image, the need for multiple searches to find matching points for images with intense motion, and the limited search efficiency and accuracy. A video coding motion estimation method and device based on multi-target tracking is proposed. The technical solution is as follows:
[0006] In a first aspect, an embodiment of the present application provides a video coding motion estimation method based on multi-target tracking, comprising:
[0007] Decoding the video to be processed to obtain at least two frames of video images;
[0008] Performing multi-target recognition processing on each frame of video image to obtain each frame of video image marked with at least two targets; wherein each target is marked based on a detection frame;
[0009] The position information and detection frame size information corresponding to each target in the first frame of the video image are used as the tracking set, and the position information and detection frame size information corresponding to each target in the m-th frame of the video image are used as the detection set. The detection frame position information corresponding to each target in the first frame of the video image in the m-th frame of the video image is determined based on the tracking set and the detection set; where m is a positive integer greater than or equal to 2;
[0010] Determining optimal matching position information of the coded macroblock in the j-th frame of the video image based on position information of the coded macroblock in the i-th frame of the video image, and calculating an inter-frame motion vector of the coded macroblock based on the position information of the coded macroblock and the optimal matching position information; wherein i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is not equal to j;
[0011] All video images are encoded according to the inter-frame motion vectors of all coded macroblocks.
[0012] In an optional solution of the first aspect, performing multi-target recognition processing on each frame of video image to obtain each frame of video image marked with at least two targets includes:
[0013] Each frame of video image is input into a trained deep learning neural network to obtain each frame of video image marked with at least two targets; wherein the deep learning neural network is trained based on multiple sample images marked with sample targets.
[0014] In another optional solution of the first aspect, determining, based on the tracking set and the detection set, detection frame position information corresponding to each target in the first frame of the video image in the m-th frame of the video image includes:
[0015] Calculate the cosine distance between the feature vector corresponding to each target in the tracking set and the feature vector corresponding to each target in the detection set;
[0016] Perform Kalman filtering on the position information and detection box size information corresponding to each target in the tracking set to obtain the predicted position information and covariance information of the detection box corresponding to each target;
[0017] Based on the detection box size information corresponding to each target in the detection set, and the predicted position information and covariance information of the detection box corresponding to each target in the tracking set, the Mahalanobis distance between each target in the tracking set and each target in the detection set is calculated;
[0018] Perform weighted summation of the cosine distance and the corresponding Mahalanobis distance between each target in the tracking set and each target in the detection set to obtain the total distance cost;
[0019] According to the total distance cost between each target in the tracking set and each target in the detection set, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image is determined.
[0020] In another optional solution of the first aspect, determining, based on the total distance cost between each target in the tracking set and each target in the detection set, detection frame position information corresponding to each target in the first frame of video image in the m-th frame of video image includes:
[0021] Convert the total distance cost between each target in the tracking set and each target in the detection set into a matrix expression; where the value corresponding to the sth row and tth column in the matrix expression is the total distance cost between the sth target in the tracking set and the tth target in the detection set, and s and t are both positive integers greater than or equal to 1;
[0022] Determine the minimum value corresponding to each row in the matrix expression, calculate the difference between each value in each row and the minimum value, and update the matrix expression based on all the differences;
[0023] Determine the minimum value corresponding to each column in the updated matrix expression, calculate the difference between each value in each column and the minimum value, and update the matrix expression again based on all the differences;
[0024] Get the minimum number of rows and columns that cover all zero values in the updated matrix expression;
[0025] When it is detected that the minimum number of rows and columns exceeds a preset threshold, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image is determined according to the updated matrix expression.
[0026] In another optional solution of the first aspect, after obtaining the minimum number of rows and columns covering all zero values in the updated matrix expression, the method further includes:
[0027] When it is detected that the minimum number of rows and columns does not exceed the preset threshold, a minimum value is determined in the updated matrix expression;
[0028] Calculate the difference between each value in each row and the minimum value, and calculate the sum of each value in each column and the minimum value;
[0029] Again obtain the minimum number of rows and columns that covers all zero values in the updated matrix expression.
[0030] In another optional solution of the first aspect, determining the optimal matching position information of the coded macroblock in the jth frame of the video image based on the position information of the coded macroblock in the i-th frame of the video image includes:
[0031] Determine whether the coded macroblock is within the detection frame corresponding to the target in the i-th frame video image;
[0032] When it is determined that the coded macroblock is within the detection frame corresponding to the target in the i-th frame of the video image, at least two reference position information of the coded macroblock in the j-th frame of the video image are determined based on the position information of the coded macroblock in the i-th frame of the video image, and image residuals between the position information of the coded macroblock and each reference position information are calculated respectively;
[0033] The reference position information corresponding to the minimum image residual is used as the optimal matching position information of the coded macroblock in the j-th frame video image.
[0034] In another optional solution of the first aspect, calculating the inter-frame motion vector of the coded macroblock according to the position information of the coded macroblock and the best matching position information includes:
[0035] Determining a first coordinate based on position information of the coded macroblock, and determining a second coordinate based on the best matching position information;
[0036] Calculating a first motion vector based on the first coordinate and the second coordinate;
[0037] Determining a second motion vector based on motion vectors corresponding to adjacent coded macroblocks of the coded macroblock;
[0038] A difference calculation is performed on the first motion vector and the second motion vector to obtain an inter-frame motion vector of the coded macroblock.
[0039] In a second aspect, an embodiment of the present application provides a video coding motion estimation device based on multi-target tracking, comprising:
[0040] A video decoding module is used to decode the video to be processed to obtain at least two frames of video images;
[0041] The target recognition module is used to perform multi-target recognition processing on each frame of video image to obtain each frame of video image marked with at least two targets; wherein each target is marked based on the detection frame;
[0042] an object tracking module, configured to use the position information and detection frame size information corresponding to each object in the first frame of video image as a tracking set, use the position information and detection frame size information corresponding to each object in the m-th frame of video image as a detection set, and determine the detection frame position information corresponding to each object in the first frame of video image in the m-th frame of video image based on the tracking set and the detection set; where m is a positive integer greater than or equal to 2;
[0043] a motion estimation module, configured to determine optimal matching position information of the coded macroblock in the jth frame of the video image based on the position information of the coded macroblock in the i-th frame of the video image, and calculate an inter-frame motion vector of the coded macroblock based on the position information of the coded macroblock and the optimal matching position information; wherein i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is not equal to j;
[0044] The video encoding module is used to encode all video images according to the inter-frame motion vectors of all encoded macroblocks.
[0045] In a third aspect, an embodiment of the present application further provides a video coding motion estimation device based on multi-target tracking, comprising a processor and a memory;
[0046] The processor is connected to the memory;
[0047] a memory for storing executable program code;
[0048] The processor runs the program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the video coding motion estimation method based on multi-target tracking provided by the first aspect of the embodiment of the present application or any implementation method of the first aspect.
[0049] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the video coding motion estimation method based on multi-target tracking provided by the first aspect of the embodiment of the present application or any one of the implementation methods of the first aspect can be implemented.
[0050] In an embodiment of the present application, during the process of encoding the video, the video to be processed can be decoded to obtain at least two frames of video images; multi-target recognition processing can be performed on each frame of the video image to obtain each frame of the video image marked with at least two targets; the position information and detection frame size information corresponding to each target in the first frame of the video image are used as a tracking set, and the position information and detection frame size information corresponding to each target in the m-th frame of the video image are used as a detection set, and the detection frame position information corresponding to each target in the first frame of the video image in the m-th frame of the video image is determined based on the tracking set and the detection set; based on the position information of the encoded macroblock in the i-th frame of the video image, the optimal matching position information of the encoded macroblock in the j-th frame of the video image is determined, and the inter-frame motion vector of the encoded macroblock is calculated based on the position information of the encoded macroblock and the optimal matching position information; all video images are encoded according to the inter-frame motion vectors of all encoded macroblocks. By tracking and detecting multiple targets in video images and then providing the detection information to the video encoder, the method of using the target tracking information to perform motion estimation can make full use of the relevant information of object movement in the video image in the motion estimation of video encoding, accurately search for the optimal matching point, and improve the search efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0052] FIG1 is an overall flow chart of a video coding motion estimation method based on multi-target tracking provided by an embodiment of the present application;
[0053] FIG2 is a schematic diagram of a target recognition effect of a video image provided by an embodiment of the present application;
[0054] FIG3 is a schematic diagram of an effect of a target detection frame of a video image provided by an embodiment of the present application;
[0055] FIG4 is a schematic diagram of an optimal matching position effect of a video image provided by an embodiment of the present application;
[0056] FIG5 is a schematic diagram of an inter-frame motion vector effect of a video image provided by an embodiment of the present application;
[0057] FIG6 is a schematic structural diagram of a video coding motion estimation device based on multi-target tracking provided by an embodiment of the present application;
[0058] FIG7 is a structural diagram of another video coding motion estimation device based on multi-target tracking provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.
[0060] In the following introduction, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following introduction provides multiple embodiments of the present application. Different embodiments can be replaced or combined, so the present application can also be considered to include all possible combinations of the same and / or different embodiments described. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present application should also be considered to include embodiments containing one or more of all other possible combinations of A, B, C, and D, even though the embodiment may not be clearly described in the following text.
[0061] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the elements described without departing from the scope of the present application. Various examples may appropriately omit, replace, or add various processes or components. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted, or combined. In addition, features described in some examples may be combined in other examples.
[0062] Please refer to FIG1 , which shows an overall flow chart of a video coding motion estimation method based on multi-target tracking provided by an embodiment of the present application.
[0063] As shown in FIG1 , the video coding motion estimation method based on multi-target tracking may include at least the following steps:
[0064] Step 102: Decode the video to be processed to obtain at least two frames of video images.
[0065] In an embodiment of the present application, a video coding motion estimation method based on multi-target tracking can be applied to, but is not limited to, a control terminal. The control terminal can be used to decode the acquired video, then perform multi-target tracking detection on the decoded video image, and then provide the detection information to a video encoder so that the target tracking information can be used to perform motion estimation, and finally, the video image is encoded in combination with the estimated motion vector. The acquired video can be, but is not limited to, a road traffic video containing vehicles or a street crowd video containing human bodies. The control terminal can acquire the video directly through a connected camera device, but is not limited to this.
[0066] Specifically, during the video encoding process, the acquired video to be processed may be first decoded to obtain the original data corresponding to each frame of the video to be processed. It is understood that in the embodiment of the present application, the video to be processed may be decoded by a predetermined video decoder, but is not limited to, to obtain at least two frames of video images, wherein the time interval between any two adjacent frames of video images remains consistent. For example, a video to be processed with a duration of 1 minute may be decoded to obtain 6 frames of video images with a time interval of 10 seconds.
[0067] Step 104: Perform multi-target recognition processing on each frame of video image to obtain each frame of video image marked with at least two targets.
[0068] Specifically, after decoding the video to be processed to obtain at least two frames of video images, multi-target recognition processing can be performed on each frame of video images to identify multiple targets from each frame of video images, and obtain each frame of video images marked with at least two targets. Each frame of video images contains at least two targets, and during the target recognition process, the position information of all targets contained in each frame of video images and the detection frame size information corresponding to each target can be identified. The detection frame size information here can be, but is not limited to, the width and height information of the detection frame marking the target.
[0069] It can be understood that when the video is a road traffic video, the multiple targets corresponding to each frame of the video image can be but are not limited to all moving vehicles appearing in the road traffic video; when the video is a street crowd video, the multiple targets corresponding to each frame of the video image can be but are not limited to all human bodies appearing in the street crowd video. The type of video is not limited here.
[0070] As an option of the embodiment of the present application, performing multi-target recognition processing on each frame of video image to obtain each frame of video image marked with at least two targets includes:
[0071] Each frame of video image is input into a trained deep learning neural network to obtain each frame of video image marked with at least two targets; wherein the deep learning neural network is trained based on multiple sample images marked with sample targets.
[0072] Specifically, in the process of performing multi-target recognition processing on each frame of video image, each frame of video image can be input into a trained deep learning neural network. The type of the deep learning neural network can be but is not limited to a neural network such as CenterNet, YOLO, SSD or Faster RCNN, and the deep learning neural network outputs each frame of video image marked with at least two targets.
[0073] Here, the type of deep learning neural network is taken as an example, the corresponding data set can be collected according to the target type corresponding to the video to be processed, so as to train the CenterNet target detection network. For example, when the type of video to be processed is a road traffic video, the data set of the CenterNet target detection network can be, but is not limited to, images of vehicles of various categories and the range of detection frames used to mark the vehicle target positions in the image. Then, the CenterNet target detection network can be trained based on this data set until the verification loss of the CenterNet target detection network is stable and no longer decreases. In order to improve the accuracy of the model, its verification loss should be as small as possible. Then, after the training of the CenterNet target detection network is completed, each frame of the video image can be input into the CenterNet target detection network to obtain the detection frame size information for each target in each frame of the video image.
[0074] Please also refer to Figure 2, which is a schematic diagram of the target recognition effect of a video image of an embodiment of the present application. As shown in Figure 2, the video image contains four targets (the shapes are inconsistent). After the video image is input into the deep learning neural network, each target in the output video image is marked with a detection frame, and the width and height corresponding to each detection frame are different.
[0075] Step 106: Use the position information and detection frame size information corresponding to each target in the first frame of video image as a tracking set, use the position information and detection frame size information corresponding to each target in the m-th frame of video image as a detection set, and determine the detection frame position information corresponding to each target in the first frame of video image in the m-th frame of video image based on the tracking set and the detection set.
[0076] Specifically, after obtaining each frame of video images marked with at least two targets, all targets in the first frame of video images, the position information corresponding to each target, and the detection frame size information can be selected from all video images as a tracking set, and all targets in all video images other than the first frame of video images, the position information corresponding to each target, and the detection frame size information can be selected as a detection set. The tracking set can be understood as initializing each target into a tracking object, each tracking object including corresponding position information and detection frame size information; the detection set can also be understood as initializing each target into a tracking object, each tracking object including corresponding position information and detection frame size information.
[0077] Furthermore, after obtaining the tracking set and the detection set respectively, the cosine distance between the feature vector corresponding to each target in the tracking set and the feature vector corresponding to each target in the detection set can be calculated, wherein the feature vector corresponding to each target can be but is not limited to the contour feature vector of each target (also called ReID (Re-identification) feature vector), which can be obtained through the deep learning neural network mentioned above, that is, the deep learning neural network can also output the contour feature vector corresponding to each target.
[0078] The expression of the cosine distance here can be but is not limited to the following:
[0079] In the above formula, d (1) (i, j) can correspond to the cosine distance between the i-th tracking object (i.e., target) in the tracking set and the j-th detection object (i.e., target) in the detection set, f i It can be corresponded to the feature vector of the i-th tracking object in the tracking set, g j It can correspond to the feature vector of the j-th detection object in the detection set.
[0080] Furthermore, after obtaining the tracking set and detection set, Kalman filtering can be performed on the position information and detection box size information corresponding to each target in the tracking set to obtain the predicted position information and covariance information of the detection box corresponding to each target. The Kalman filtering method is generally understood to be a conventional technique in the art and will not be further described here.
[0081] Furthermore, after obtaining the predicted position information and covariance information of the detection box corresponding to each target in the tracking set through the Kalman filter processing technology, the Mahalanobis distance between each target in the tracking set and each target in the detection set can be calculated based on the predicted position information and covariance information of the detection box corresponding to each target in the tracking set and the detection box size information corresponding to each target in the detection set. The calculation expression of the Mahalanobis distance can be, but is not limited to, as follows:
[0082] In the above formula, d (2) (i, j) can correspond to the Mahalanobis distance between the i-th tracking object (i.e., target) in the tracking set and the j-th detection object (i.e., target) in the detection set, d j It can correspond to the detection box size information of the jth detection object in the detection set, y i It can correspond to the predicted position information of the detection box of the i-th tracking object in the tracking set, S i It can correspond to the covariance information of the i-th tracking object in the tracking set.
[0083] Furthermore, after obtaining the cosine distance and the corresponding Mahalanobis distance between each target in the tracking set and each target in the detection set, the cosine distance and the corresponding Mahalanobis distance between each target in the tracking set and each target in the detection set can be weighted and summed by, but not limited to, a preset weight value to obtain a total distance cost between each target in the tracking set and each target in the detection set, wherein the calculation expression of the total distance cost can be, but not limited to, as follows: c(i, j) = λ·d (1) (i, j)+(1-λ)d (2) (i, j)
[0084] In the above formula, c(i, j) can be corresponded to the total distance cost between the i-th tracking object in the tracking set and the j-th detection object in the detection set, d (1) (i, j) can correspond to the cosine distance between the i-th tracking object (i.e., target) in the tracking set mentioned above and the j-th detection object (i.e., target) in the detection set, d (2)(i, j) may correspond to the Mahalanobis distance between the i-th tracked object (ie, target) in the aforementioned tracking set and the j-th detected object (ie, target) in the detection set, and λ may correspond to a preset weight value.
[0085] Furthermore, after obtaining the total distance cost between each target in the tracking set and each target in the detection set, the total distance cost between each target in the tracking set and each target in the detection set can be calculated using, but is not limited to, the Hungarian algorithm to obtain the optimal match for each target in the tracking set, and then determine the detection box position information corresponding to each target in the first frame of the video image in the mth frame of the video image.
[0086] As another optional embodiment of the present application, determining, based on the total distance cost between each target in the tracking set and each target in the detection set, detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image includes:
[0087] Convert the total distance cost between each target in the tracking set and each target in the detection set into a matrix expression; where the value corresponding to the sth row and tth column in the matrix expression is the total distance cost between the sth target in the tracking set and the tth target in the detection set, and s and t are both positive integers greater than or equal to 1;
[0088] Determine the minimum value corresponding to each row in the matrix expression, calculate the difference between each value in each row and the minimum value, and update the matrix expression based on all the differences;
[0089] Determine the minimum value corresponding to each column in the updated matrix expression, calculate the difference between each value in each column and the minimum value, and update the matrix expression again based on all the differences;
[0090] Get the minimum number of rows and columns that cover all zero values in the updated matrix expression;
[0091] When it is detected that the minimum number of rows and columns exceeds a preset threshold, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image is determined according to the updated matrix expression.
[0092] Specifically, in the process of determining the detection frame position information corresponding to each target in the first frame of video image in the mth frame of video image, the total distance cost between each target in the tracking set and each target in the detection set can be first converted into a matrix expression. For example, the value corresponding to the sth row and tth column in the matrix expression is the total distance cost between the sth target in the tracking set and the tth target in the detection set, and s and t are both positive integers greater than or equal to 1.
[0093] Next, the minimum value corresponding to each row in the matrix expression is determined, the difference between each value in each row and the minimum value is calculated, and each value in each row is replaced by the difference between the minimum value and the minimum value. It is understood that at least one value in each row is 0.
[0094] Next, after replacing each row, the minimum value corresponding to each column can be determined in the matrix expression, the difference between each value in each column and the minimum value can be calculated, and each value in each column can be replaced by the difference between the minimum value and the minimum value. It can be understood that at least one value in each column is 0.
[0095] Next, all zero values in the matrix can be covered with as few column or row markers as possible to obtain the minimum number of rows and columns covering all zero values in the updated matrix expression. It will be understood that to ensure the minimum number of rows and columns, it is possible but not limited to making the number of zero values corresponding to the marked rows or columns as large as possible.
[0096] When it is detected that the minimum number of rows and columns exceeds a preset threshold, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image can be determined based on the final replacement matrix. For example, but not limited to, when all values in the sth row of the final replacement matrix except the tth column are zero, it can be determined that the tth target in the detection set is the optimal matching target for the sth target in the tracking set. It can be understood that the detection frame position information can be, but is not limited to, the position coordinates corresponding to any at least one vertex of the detection frame in the video image. It can be calculated by the center coordinates of each target and the corresponding detection frame size information. The position coordinates and center coordinates can both be determined based on the plane rectangular coordinate system established by the video image (the plane rectangular coordinate system corresponding to each frame of the video image can remain consistent), and this is not limited to this.
[0097] As another optional embodiment of the present application, when it is detected that the minimum number of rows and columns does not exceed the preset threshold, the minimum value (non-zero value) can be determined in the updated matrix expression, the difference between each value (non-zero value) in each row and the minimum value can be calculated, and each value in each column can be replaced by the difference between the minimum value and the minimum value again.
[0098] Then, after replacing each row, the sum between each value in each column and the minimum value can be calculated in the matrix expression, and each value in each column can be replaced by the sum between the minimum value and the minimum value.
[0099] Next, all zero values in the matrix can be covered with as few column or row markers as possible to obtain the minimum number of rows and columns covering all zero values in the updated matrix expression. The above steps are repeated until the minimum number of rows and columns is detected to exceed the preset threshold.
[0100] Please also refer to Figure 3, which is a schematic diagram of the effect of a target detection frame of a video image provided by an embodiment of the present application. As shown in Figure 3, targets 1 and 2 on the left side of the video image can correspond to the target position and detection frame in the first frame of the video image, and targets 1 and 2 on the right side can correspond to the optimal matching target position and detection frame in the mth frame of the video image.
[0101] Step 108: Determine the optimal matching position information of the coded macroblock in the jth frame of video image based on the position information of the coded macroblock in the i-th frame of video image, and calculate the inter-frame motion vector of the coded macroblock based on the position information of the coded macroblock and the optimal matching position information.
[0102] Specifically, after obtaining the detection frame position information corresponding to each target in the first frame of video image in each subsequent frame of video image, in order to achieve more accurate coding macroblock search and matching, thereby reducing residual data with the reference frame and achieving the effect of reducing the encoding bit rate, it is possible to first determine whether the coded macroblock is within the detection frame corresponding to the target in the i-th frame of video image (which can also be understood as the coded frame video image). It can be understood that it is assumed here that the detection frame corresponding to the target can be divided into 8x16 macroblocks, and the judgment method can be to determine whether the coded macroblock is any one of all the macroblocks divided by the detection frame.
[0103] Furthermore, when it is determined that the coded macroblock is within the detection frame corresponding to the target in the i-th frame of the video image, at least two reference position information of the coded macroblock in the j-th frame of the video image (which can also be understood as a reference frame of the video image, which can be but not limited to an adjacent frame of the i-th frame of the video image) can be determined based on the position coordinates of the coded macroblock in the i-th frame of the video image, and the reference position information corresponding to the minimum image residual can be used as the optimal matching position information of the coded macroblock in the j-th frame of the video image by calculating the image residual, but not limited to. Here, taking the macroblock of the i-th row and j-th column of a detection frame in the i-th frame of the video image as an example, the corresponding macroblock of the i-th row and j-th column can be determined in the j-th frame of the video image based on the i-th row and j-th column, and the macroblock of the i-th row and j-th column in the j-th frame of the video image is used as the center macroblock, and the position information of the center macroblock and all the surrounding macroblocks is used as at least two reference position information, and the corresponding image residuals are calculated respectively. The calculation expression of the image residual can be, but is not limited to, expressed as follows:
[0104] In the above formula, SAD corresponds to calculating the sum of absolute errors of a macroblock (which can be understood as a method for calculating image residuals), W and H are the number of pixels in the width and height of the macroblock, S(x, y) corresponds to the value of the pixel in the xth row and yth column of the encoded macroblock in the jth frame of video, and T(x, y) corresponds to the value of the pixel in the xth row and yth column of the encoded macroblock in the ith frame of video. In the embodiment of the present application, the method for calculating image residuals can also be, but is not limited to, any one of SAD (sum of absolute errors), SSD (sum of squared differences), MAD (mean absolute difference), or SATD (sum of absolute values after Hadamard transformation).
[0105] Furthermore, the position information of the macroblock with the smallest image residual value can be used as the optimal matching position information of the encoded macroblock in the j-th frame video image. It can also be understood that after determining the center macroblock, the residual of the center macroblock and the residuals of the eight macroblocks with a distance of 4 macroblocks around and diagonally opposite to each other can be calculated, but is not limited to it; then, the macroblock with the smallest residual is used as the center point with a distance of 2 macroblocks to calculate the residuals of the center point, the surrounding macroblocks, and the diagonally opposite to each other; then, the macroblock with the smallest residual is used as the center point with a distance of 1 macroblock to calculate the residuals of the center point, the surrounding macroblocks, and the diagonally opposite to each other, wherein the macroblock with the smallest residual is the optimal matching macroblock, and this is not limited to this.
[0106] Please also refer to Figure 4, which is a schematic diagram of the optimal matching position effect of a video image provided by an embodiment of the present application. As shown in Figure 4, when the coded macroblock in the coded frame video image on the right is within the detection frame range of target 1, the corresponding center macroblock in the reference frame video image on the left can be determined based on the position information of the coded macroblock in the detection frame of target 1, and the nine adjacent macroblocks around the center macroblock can be used as the search range to calculate the image residual to obtain the optimal matching position information.
[0107] Furthermore, after obtaining the optimal matching position information of the coded macroblock in the j-th frame video image, a first coordinate may be determined based on the position information of the coded macroblock, the first coordinate may be, but not limited to, the coordinate of the upper left corner of the coded macroblock in a plane rectangular coordinate system, and a second coordinate may be determined based on the optimal matching position information, the second coordinate may be, but not limited to, the coordinate of the upper left corner of the optimal matching macroblock in a plane rectangular coordinate system, and a first motion vector may be calculated in combination with the first coordinate and the second coordinate. The first motion vector may be, but not limited to, expressed as MV: (x t -x,y t -y), the upper left corner coordinates of the coded macroblock correspond to (x, y), and the upper left corner coordinates of the best matching macroblock correspond to (x t ,y t ).
[0108] Furthermore, a second motion vector can be determined based on the motion vectors corresponding to the adjacent coded macroblocks of the coded macroblock, and the difference between the first motion vector and the second motion vector is calculated to obtain the inter-frame motion vector of the coded macroblock. Since the detection frame of the tracking target usually contains multiple coded macroblocks, the motion vectors of spatially adjacent coded macroblocks have a strong correlation. Therefore, the motion vector of each macroblock can be predicted based on the adjacent previously coded blocks. For example, the predicted second motion vector is represented as MV p , taking the first motion vector as MV as an example, the inter-frame motion vector of the coded macroblock can be, but is not limited to, represented as: MV d =MV-MV p
[0109] Here, the calculation method of the second motion vector can be based on the prediction method of the spatiotemporal domain, such as using the vector median prediction in the spatial domain, that is, using the motion vectors of the left block A, the upper block B and the upper right block C adjacent to the current block E, and taking the median as the predicted motion vector of the current block; of course, the motion vector prediction of the corresponding block of the previous frame in the time domain can also be used, that is, using the block motion vector of the same coordinate position in the previous frame and the current frame as the predicted motion vector of the current block, which is not limited to this here.
[0110] Please also refer to Figure 5, which is a schematic diagram of the inter-frame motion vector effect of a video image provided by an embodiment of the present application. As shown in Figure 5, the vector between the upper left corner coordinates of the encoded macroblock in the encoded frame video image on the right and the upper left corner coordinates of the encoded macroblock in the reference frame video image on the left is the inter-frame motion vector of the encoded macroblock.
[0111] Step 110: Encode all video images according to the inter-frame motion vectors of all encoded macroblocks.
[0112] Specifically, after obtaining the inter-frame motion vectors of the coded macroblocks, the motion optimization processing of all video images can be performed in combination with the inter-frame motion vectors of all coded macroblocks during the encoding process of all video images. Compared with the traditional video coding motion estimation method that uses a fixed search mode, it can fully utilize the information related to the video image content to make the motion estimation matching more accurate. Especially in high-speed motion video images, its motion estimation advantage will be more obvious and will not fall into the trap of local optimality. In addition, the method of the present application can also perform targeted target recognition and tracking network training based on the main body of the video image content, which can adapt to the needs of video encoding in different scenes and further improve the targeting.
[0113] Please refer to FIG. 6 , which shows a schematic structural diagram of a video coding motion estimation device based on multi-target tracking provided by an embodiment of the present application.
[0114] As shown in FIG6 , the video coding motion estimation apparatus based on multi-target tracking may include at least a video decoding module 601, a target recognition module 602, a target tracking module 603, a motion estimation module 604, and a video coding module 605, wherein:
[0115] The video decoding module 601 is used to decode the video to be processed to obtain at least two frames of video images;
[0116] The target recognition module 602 is configured to perform multi-target recognition processing on each frame of the video image to obtain each frame of the video image marked with at least two targets; wherein each target is marked based on a detection frame;
[0117] The target tracking module 603 is configured to use the position information and detection frame size information corresponding to each target in the first frame of the video image as a tracking set, use the position information and detection frame size information corresponding to each target in the m-th frame of the video image as a detection set, and determine the detection frame position information corresponding to each target in the first frame of the video image in the m-th frame of the video image based on the tracking set and the detection set, where m is a positive integer greater than or equal to 2;
[0118] a motion estimation module 604 configured to determine, based on the position information of the coded macroblock in the i-th frame of the video image, the optimal matching position information of the coded macroblock in the j-th frame of the video image, and calculate an inter-frame motion vector of the coded macroblock based on the position information of the coded macroblock and the optimal matching position information; wherein i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is not equal to j;
[0119] The video encoding module 605 is configured to perform encoding processing on all video images according to the inter-frame motion vectors of all encoded macroblocks.
[0120] In some possible embodiments, the target recognition module is configured to:
[0121] Each frame of video image is input into a trained deep learning neural network to obtain each frame of video image marked with at least two targets; wherein the deep learning neural network is trained based on multiple sample images marked with sample targets.
[0122] In some possible embodiments, the target tracking module is used to:
[0123] Calculate the cosine distance between the feature vector corresponding to each target in the tracking set and the feature vector corresponding to each target in the detection set;
[0124] Perform Kalman filtering on the position information and detection box size information corresponding to each target in the tracking set to obtain the predicted position information and covariance information of the detection box corresponding to each target;
[0125] Based on the detection box size information corresponding to each target in the detection set, and the predicted position information and covariance information of the detection box corresponding to each target in the tracking set, the Mahalanobis distance between each target in the tracking set and each target in the detection set is calculated;
[0126] Perform weighted summation of the cosine distance and the corresponding Mahalanobis distance between each target in the tracking set and each target in the detection set to obtain the total distance cost;
[0127] According to the total distance cost between each target in the tracking set and each target in the detection set, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image is determined.
[0128] In some possible embodiments, the target tracking module is used to:
[0129] Convert the total distance cost between each target in the tracking set and each target in the detection set into a matrix expression; where the value corresponding to the sth row and tth column in the matrix expression is the total distance cost between the sth target in the tracking set and the tth target in the detection set, and s and t are both positive integers greater than or equal to 1;
[0130] Determine the minimum value corresponding to each row in the matrix expression, calculate the difference between each value in each row and the minimum value, and update the matrix expression based on all the differences;
[0131] Determine the minimum value corresponding to each column in the updated matrix expression, calculate the difference between each value in each column and the minimum value, and update the matrix expression again based on all the differences;
[0132] Get the minimum number of rows and columns that cover all zero values in the updated matrix expression;
[0133] When it is detected that the minimum number of rows and columns exceeds a preset threshold, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image is determined according to the updated matrix expression.
[0134] In some possible embodiments, the target tracking module is used to:
[0135] After obtaining the minimum number of rows and columns covering all zero values in the updated matrix expression, when it is detected that the minimum number of rows and columns does not exceed a preset threshold, determining the minimum value in the updated matrix expression;
[0136] Calculate the difference between each value in each row and the minimum value, and calculate the sum of each value in each column and the minimum value;
[0137] Again obtain the minimum number of rows and columns that covers all zero values in the updated matrix expression.
[0138] In some possible embodiments, the motion estimation module is configured to:
[0139] Determine whether the coded macroblock is within the detection frame corresponding to the target in the i-th frame video image;
[0140] When it is determined that the coded macroblock is within the detection frame corresponding to the target in the i-th frame of the video image, at least two reference position information of the coded macroblock in the j-th frame of the video image are determined based on the position information of the coded macroblock in the i-th frame of the video image, and image residuals between the position information of the coded macroblock and each reference position information are calculated respectively;
[0141] The reference position information corresponding to the minimum image residual is used as the optimal matching position information of the coded macroblock in the j-th frame video image.
[0142] In some possible embodiments, the motion estimation module is configured to:
[0143] Determining a first coordinate based on position information of the coded macroblock, and determining a second coordinate based on the best matching position information;
[0144] Calculating a first motion vector based on the first coordinate and the second coordinate;
[0145] Determining a second motion vector based on motion vectors corresponding to adjacent coded macroblocks of the coded macroblock;
[0146] A difference calculation is performed on the first motion vector and the second motion vector to obtain an inter-frame motion vector of the coded macroblock.
[0147] Those skilled in the art will clearly understand that the technical solutions of the embodiments of the present application can be implemented with the help of software and / or hardware. "Unit" and "module" in this specification refer to software and / or hardware that can independently perform or cooperate with other components to perform specific functions, where the hardware can be, for example, a field programmable gate array (FPGA), an integrated circuit (IC), etc.
[0148] Please refer to FIG. 7 , which shows a schematic structural diagram of another apparatus for video coding motion estimation based on multi-target tracking provided by an embodiment of the present application.
[0149] As shown in FIG. 7 , a video coding motion estimation apparatus 700 based on multi-target tracking may include at least one processor 701 , at least one network interface 704 , a user interface 703 , a memory 705 , and at least one communication bus 702 .
[0150] The communication bus 702 may be used to implement connection and communication among the above components.
[0151] The user interface 703 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.
[0152] The network interface 704 may include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, and the like.
[0153] The processor 701 may include one or more processing cores. The processor 701 utilizes various interfaces and circuits to connect the various components within the multi-target tracking-based video coding motion estimation device 700. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 705, and invoking data stored in the memory 705, the processor 701 executes various functions and processes data within the multi-target tracking-based video coding motion estimation device 700. Optionally, the processor 701 may be implemented in at least one hardware form selected from the group consisting of a DSP, an FPGA, and a PLA. The processor 701 may integrate one or a combination of a CPU, a GPU, and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is responsible for handling wireless communications. It is understood that the modem may not be integrated into the processor 701 and may be implemented as a separate chip.
[0154] The memory 705 may include RAM or ROM. Optionally, the memory 705 includes a non-transitory computer-readable medium. The memory 705 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 705 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 705 may also optionally be at least one storage device located away from the aforementioned processor 701. As shown in Figure 7, the memory 705 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a video coding motion estimation application based on multi-target tracking.
[0155] Specifically, the processor 701 may be configured to call a video coding motion estimation application based on multi-target tracking stored in the memory 705, and specifically perform the following operations:
[0156] Decoding the video to be processed to obtain at least two frames of video images;
[0157] Performing multi-target recognition processing on each frame of video image to obtain each frame of video image marked with at least two targets; wherein each target is marked based on a detection frame;
[0158] The position information and detection frame size information corresponding to each target in the first frame of the video image are used as the tracking set, and the position information and detection frame size information corresponding to each target in the m-th frame of the video image are used as the detection set. The detection frame position information corresponding to each target in the first frame of the video image in the m-th frame of the video image is determined based on the tracking set and the detection set; where m is a positive integer greater than or equal to 2;
[0159] Determining optimal matching position information of the coded macroblock in the j-th frame of the video image based on position information of the coded macroblock in the i-th frame of the video image, and calculating an inter-frame motion vector of the coded macroblock based on the position information of the coded macroblock and the optimal matching position information; wherein i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is not equal to j;
[0160] All video images are encoded according to the inter-frame motion vectors of all coded macroblocks.
[0161] In some possible embodiments, performing multi-target recognition processing on each frame of video image to obtain each frame of video image marked with at least two targets includes:
[0162] Each frame of video image is input into a trained deep learning neural network to obtain each frame of video image marked with at least two targets; wherein the deep learning neural network is trained based on multiple sample images marked with sample targets.
[0163] In some possible embodiments, determining the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image according to the tracking set and the detection set includes:
[0164] Calculate the cosine distance between the feature vector corresponding to each target in the tracking set and the feature vector corresponding to each target in the detection set;
[0165] Perform Kalman filtering on the position information and detection box size information corresponding to each target in the tracking set to obtain the predicted position information and covariance information of the detection box corresponding to each target;
[0166] Based on the detection box size information corresponding to each target in the detection set, and the predicted position information and covariance information of the detection box corresponding to each target in the tracking set, the Mahalanobis distance between each target in the tracking set and each target in the detection set is calculated;
[0167] Perform weighted summation of the cosine distance and the corresponding Mahalanobis distance between each target in the tracking set and each target in the detection set to obtain the total distance cost;
[0168] According to the total distance cost between each target in the tracking set and each target in the detection set, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image is determined.
[0169] In some possible embodiments, determining, based on the total distance cost between each target in the tracking set and each target in the detection set, the detection box position information corresponding to each target in the first frame of the video image in the mth frame of the video image includes:
[0170] Convert the total distance cost between each target in the tracking set and each target in the detection set into a matrix expression; where the value corresponding to the sth row and tth column in the matrix expression is the total distance cost between the sth target in the tracking set and the tth target in the detection set, and s and t are both positive integers greater than or equal to 1;
[0171] Determine the minimum value corresponding to each row in the matrix expression, calculate the difference between each value in each row and the minimum value, and update the matrix expression based on all the differences;
[0172] Determine the minimum value corresponding to each column in the updated matrix expression, calculate the difference between each value in each column and the minimum value, and update the matrix expression again based on all the differences;
[0173] Get the minimum number of rows and columns that cover all zero values in the updated matrix expression;
[0174] When it is detected that the minimum number of rows and columns exceeds a preset threshold, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image is determined according to the updated matrix expression.
[0175] In some possible embodiments, after obtaining the minimum number of rows and columns covering all zero values in the updated matrix expression, the method further includes:
[0176] When it is detected that the minimum number of rows and columns does not exceed the preset threshold, a minimum value is determined in the updated matrix expression;
[0177] Calculate the difference between each value in each row and the minimum value, and calculate the sum of each value in each column and the minimum value;
[0178] Again obtain the minimum number of rows and columns that covers all zero values in the updated matrix expression.
[0179] In some possible embodiments, determining the optimal matching position information of the coded macroblock in the jth frame of the video image based on the position information of the coded macroblock in the i-th frame of the video image includes:
[0180] Determine whether the coded macroblock is within the detection frame corresponding to the target in the i-th frame video image;
[0181] When it is determined that the coded macroblock is within the detection frame corresponding to the target in the i-th frame of the video image, at least two reference position information of the coded macroblock in the j-th frame of the video image are determined based on the position information of the coded macroblock in the i-th frame of the video image, and image residuals between the position information of the coded macroblock and each reference position information are calculated respectively;
[0182] The reference position information corresponding to the minimum image residual is used as the optimal matching position information of the coded macroblock in the j-th frame video image.
[0183] In some possible embodiments, calculating the inter-frame motion vector of the coded macroblock according to the position information of the coded macroblock and the best matching position information includes:
[0184] Determining a first coordinate based on position information of the coded macroblock, and determining a second coordinate based on the best matching position information;
[0185] Calculating a first motion vector based on the first coordinate and the second coordinate;
[0186] Determining a second motion vector based on motion vectors corresponding to adjacent coded macroblocks of the coded macroblock;
[0187] A difference calculation is performed on the first motion vector and the second motion vector to obtain an inter-frame motion vector of the coded macroblock.
[0188] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a microdrive, a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.
[0189] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.
[0190] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0191] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.
[0192] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0193] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0194] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0195] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0196] The above are merely exemplary embodiments of the present disclosure and are not intended to limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure herein, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not recorded in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.
Claims
1. A video coding motion estimation method based on multi-target tracking, characterized in that: include: Decoding the video to be processed to obtain at least two frames of video images; Performing multi-target recognition processing on each frame of the video image to obtain each frame of the video image marked with at least two targets; wherein each target is marked based on a detection frame; Taking the position information and the detection frame size information corresponding to each target in the first frame of the video image as a tracking set, taking the position information and the detection frame size information corresponding to each target in the m-th frame of the video image as a detection set, and determining the detection frame position information corresponding to each target in the first frame of the video image in the m-th frame of the video image according to the tracking set and the detection set; wherein m is a positive integer greater than or equal to 2; Determine the optimal matching position information of the coded macroblock in the jth frame of the video image based on the position information of the coded macroblock in the i-th frame of the video image, and calculate the inter-frame motion vector of the coded macroblock according to the position information of the coded macroblock and the optimal matching position information; wherein i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is not equal to j; All the video images are coded according to the inter-frame motion vectors of all the coded macroblocks.
2. The method according to claim 1, characterized in that The performing multi-target recognition processing on each frame of the video image to obtain each frame of the video image marked with at least two targets includes: Each frame of the video image is input into a trained deep learning neural network to obtain each frame of the video image marked with at least two targets; wherein the deep learning neural network is trained based on multiple sample images marked with sample targets.
3. The method according to claim 1, characterized in that The determining, according to the tracking set and the detection set, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image includes: Calculating the cosine distance between the feature vector corresponding to each of the targets in the tracking set and the feature vector corresponding to each of the targets in the detection set; Performing Kalman filtering on the position information and detection frame size information corresponding to each target in the tracking set to obtain predicted position information and covariance information of the detection frame corresponding to each target; Based on the detection box size information corresponding to each target in the detection set, and the detection box prediction position information and covariance information corresponding to each target in the tracking set, a Mahalanobis distance between each target in the tracking set and each target in the detection set is calculated; Performing weighted sum processing on the cosine distance and the corresponding Mahalanobis distance between each target in the tracking set and each target in the detection set to obtain a total distance cost; According to the total distance cost between each target in the tracking set and each target in the detection set, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image is determined.
4. The method according to claim 3, characterized in that The determining, according to the total distance cost between each target in the tracking set and each target in the detection set, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image includes: The total distance cost between each target in the tracking set and each target in the detection set is converted into a matrix expression; wherein the value corresponding to the sth row and tth column in the matrix expression is the sth row and tth column in the tracking set. The total distance cost between the target and the t-th target in the detection set, s and t are both positive integers greater than or equal to 1; Determine the minimum value corresponding to each row in the matrix expression, calculate the difference between each value in each row and the minimum value, and update the matrix expression according to all the differences; Determine the minimum value corresponding to each column in the updated matrix expression, calculate the difference between each value in each column and the minimum value, and update the matrix expression again according to all the differences; Obtain the minimum number of rows and columns covering all zero values in the updated matrix expression; When it is detected that the minimum number of rows and columns exceeds a preset threshold, the detection frame position information corresponding to each target in the first frame of the video image in the mth frame of the video image is determined according to the updated matrix expression.
5. The method according to claim 4, characterized in that After obtaining the minimum number of rows and columns covering all zero values in the updated matrix expression, the method further includes: When it is detected that the minimum number of rows and columns does not exceed the preset threshold, a minimum value is determined in the updated matrix expression; Calculate the difference between each value in each row and the minimum value, and calculate the sum of each value in each column and the minimum value; The minimum number of rows and columns covering all zero values in the updated matrix expression is obtained again.
6. The method according to claim 1, characterized in that The determining the optimal matching position information of the coded macroblock in the jth frame of the video image based on the position information of the coded macroblock in the i-th frame of the video image comprises: Determine whether the coded macroblock is within the detection frame corresponding to the target in the i-th frame of the video image; When it is determined that the coded macroblock is within the detection frame corresponding to the target in the i-th frame of the video image, at least two reference position information of the coded macroblock in the j-th frame of the video image are determined according to the position information of the coded macroblock in the i-th frame of the video image, and the image residual between the position information of the coded macroblock and each of the reference position information is calculated respectively; The reference position information corresponding to the minimum image residual is used as the optimal matching position information of the coded macroblock in the j-th frame video image.
7. The method according to claim 6, characterized in that The step of calculating the inter-frame motion vector of the coded macroblock according to the position information of the coded macroblock and the optimal matching position information comprises: Determine the first coordinate according to the position information of the coded macroblock, and determine the second coordinate according to the best matching position information; Calculate a first motion vector based on the first coordinate and the second coordinate; Determine a second motion vector according to motion vectors corresponding to adjacent coded macroblocks of the coded macroblock; A difference calculation is performed on the first motion vector and the second motion vector to obtain an inter-frame motion vector of the coded macroblock.
8. A video coding motion estimation device based on multi-target tracking, characterized in that: include: A video decoding module, used for decoding the video to be processed to obtain at least two frames of video images; A target recognition module, used for performing multi-target recognition processing on each frame of the video image to obtain each frame of the video image marked with at least two targets; wherein each target is marked based on a detection frame; The target tracking module is used to use the position information corresponding to each target in the first frame of the video image and the detection frame size information as a tracking set, use the position information corresponding to each target in the m-th frame of the video image and the detection frame size information as a detection set, and determine the first frame according to the tracking set and the detection set. The detection frame position information corresponding to each target in the video image in the mth frame of the video image; wherein m is a positive integer greater than or equal to 2; A motion estimation module, configured to determine the optimal matching position information of the coded macroblock in the jth frame of the video image based on the position information of the coded macroblock in the i-th frame of the video image, and calculate the inter-frame motion vector of the coded macroblock according to the position information of the coded macroblock and the optimal matching position information; wherein i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is not equal to j; The video encoding module is used to encode all the video images according to the inter-frame motion vectors of all the encoded macroblocks.
9. A video coding motion estimation device based on multi-target tracking, characterized in that: including a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the steps of the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Target tracking technology based on video compression domain
CN108200432A
Inter-frame prediction method and device
CN110933423A
Video motion estimation method and device, electronic equipment and storage medium
CN111526369A
Detection tracking method and system based on DeepSort and DeepEMD
CN115511920A
Video coding motion estimation method and device based on multi-target tracking
CN116366866A
Cited By
Target tracking and behavior detection method based on camera equipment
CN120219772A
