Video target tracking method, device, medium and system

Abstract

The invention provides a video target tracking method, device, medium and system. According to the method, an instance segmentation algorithm is used for foreground extraction, which is a necessary step for crowd estimation, tracking and behavior analysis in a crowded scene; in order to still ensure the accuracy of the model and improve the precision of the model under the condition of less parameter quantity, the gravitation model and the human motion analysis are fused by adopting a human motion analysis and gravitation model weighted average method, and non-human and non-human portrait elements in the scene are removed; by using a Hungary algorithm, association and position matching are performed in each frame of a video, so that the real-time performance is ensured, and the problems that a video target tracking method based on attention feature fusion in an existing scheme has a large number of parameters and is long in detection time, video target detection needs to detect a plurality of images in a short time, and the detection efficiency is high are solved. And therefore, the real-time performance of target tracking is relatively poor.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video target tracking method, device, medium and system. Background Art

[0002] Crowd target tracking is a challenging research field in computer vision, and the existing technical solutions are as follows. The first existing solution is a drone video multi-target tracking method based on attention feature fusion. This method uses a deep learning network based on the attention mechanism for target detection, and then uses the Kalman filter for target tracking. The second existing solution is a video target tracking and prediction method and system based on a graph neural network. The Faster RCNN model is used to generate a spatial graph of target categories and target relationship categories, and then the target is tracked, and the long short-term memory network is used to predict the target trajectory. The third existing solution is a video target tracking method based on Kalman filtering. Based on YOLOv5, this method uses Kalman filtering for smooth target state estimation, and uses the Hungarian algorithm to achieve efficient data association, thereby improving the stability and accuracy of target tracking.

[0003] In summary, deep learning-based methods have been widely used in the field of video target detection. However, the existing video target tracking methods based on attention feature fusion have a large number of parameters and a long detection time. Video target detection requires detecting multiple images in a short time, so it is difficult to ensure the real-time performance of target tracking.

[0004] That is, the existing video target tracking method based on attention feature fusion has a large number of parameters and a long detection time, while video target detection requires detecting multiple images in a short time, which makes the real-time performance of target tracking poor. Summary of the invention

[0005] The main purpose of the present application is to provide a video target tracking method, device, medium and system, so as to at least solve the problem that the existing video target tracking method based on attention feature fusion has a large number of parameters and a long detection time, while video target detection requires detecting multiple images in a short time, which makes the real-time performance of target tracking poor.

[0006] In order to achieve the above object, according to one aspect of the present application, a video target tracking method is provided, the method comprising:

[0007] Acquire an initial acquisition video, and preprocess the initial acquisition video to convert each frame image in the initial acquisition video into a color frame image to obtain a first acquisition video;

[0008] Extracting a target outline from the first collected video using an instance segmentation algorithm, and verifying whether the target outline is a human outline using human motion analysis and a gravity model;

[0009] When it is determined that the target outline is the human body outline, a fuzzy mean algorithm is used to perform pedestrian counting processing on the first collected video to obtain the total number of pedestrians;

[0010] The Hungarian algorithm is used to match the pedestrians in the first collected video based on the total number of pedestrians, and some frame images in the first collected video are deleted to obtain a second collected video, where the second collected video represents the tracking trajectory video of each pedestrian corresponding to the total number of pedestrians.

[0011] Optionally, in the process of preprocessing the initial captured video to convert each frame image in the initial captured video into a color frame image to obtain the first captured video, the method further includes:

[0012] use Adjusting the contrast of each frame of the image in the initially acquired video;

[0013] Among them, L-1 represents white, 0 represents black, and r j is the jth input image intensity to be processed, p r is the probability density function of r, s k is the contrast of the k-frame image after processing.

[0014] Optionally, in the process of performing pedestrian counting processing on the first collected video by using a fuzzy mean algorithm to obtain the total number of pedestrians, the method further includes:

[0015] The objective function is constructed as:

[0016]

[0017] Among them, J m is the objective function value of the cluster family of the mth cluster, N is the number of samples, C is the number of cluster centers of the jth cluster, c j represents the jth cluster center, x i represents the i-th sample, Represents sample x i For cluster center c j The degree of membership.

[0018] Optionally, in the process of performing pedestrian counting processing on the first collected video by using a fuzzy mean algorithm to obtain the total number of pedestrians, the method further includes:

[0019] Using L c =I m p k, marking the clusters of each frame of video, each of the clusters includes all particles constituting a contour of the target;

[0020] Among them, p k is the total number of particles in the target contour in cluster k, L c The post-processed data for a single target profile, I m is the label of the mth contour.

[0021] Optionally, a fuzzy mean algorithm is used to perform pedestrian counting processing on the first collected video to obtain the total number of pedestrians, including:

[0022] according to determining the total number of pedestrians;

[0023] Among them, H k is the total number of pedestrians in cluster k, p n∈k is the average number of particles required for a single pedestrian, p k is the total number of particles in the target contour in cluster k.

[0024] Optionally, using the Hungarian algorithm to match pedestrians in the first collected video based on the total number of pedestrians, and deleting some frame images in the first collected video, includes:

[0025] Using a preset detection list and a preset tracking list, calculating the center distance and IoU score and storing them in a target matrix, and sorting each of the IoU scores using a corresponding cost function;

[0026] The Hungarian algorithm is used to search for the minimum tracking value of each detection in the target matrix to obtain a detection tracking matching matrix, wherein the detection tracking matching matrix represents a matrix matching between the preset detection list and the preset tracking list;

[0027] When the pedestrian is not matched in a preset number of consecutive frames, the corresponding image frames in the first captured video are deleted.

[0028] Optionally, the method further comprises:

[0029] When it is determined that the target contour is not a human contour, an instance segmentation algorithm is used again to extract the target contour from the first captured video until it is determined that the target contour is the human contour.

[0030] According to another aspect of the present application, a video target tracking device is provided, the device comprising:

[0031] An acquisition unit, used for acquiring an initial acquisition video, and preprocessing the initial acquisition video to convert each frame image in the initial acquisition video into a color frame image to obtain a first acquisition video;

[0032] A first processing unit is used to extract a target contour from the first collected video by using an instance segmentation algorithm, and verify whether the target contour is a human contour by using human motion analysis and a gravity model;

[0033] A second processing unit is used to perform pedestrian counting processing on the first collected video by using a fuzzy mean algorithm to obtain a total number of pedestrians when it is determined that the target outline is the human body outline;

[0034] The third processing unit is used to match the pedestrians in the first collected video based on the total number of pedestrians by using the Hungarian algorithm, and delete some frame images in the first collected video to obtain a second collected video, wherein the second collected video represents the tracking trajectory video of each pedestrian corresponding to the total number of pedestrians.

[0035] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0036] According to another aspect of the present application, a video target tracking system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0037] The technical solution of the present application is applied to foreground extraction through an instance segmentation algorithm, which is a necessary step for crowd estimation, tracking and behavior analysis in crowded scenes; in order to ensure the accuracy of the model and improve the model precision with a smaller number of parameters, the gravity model and the human motion analysis are fused by weighted averaging of the human motion analysis and the gravity model to remove non-human and non-pedestrian pixels in the scene; the Hungarian algorithm is used to ensure real-time performance by performing association and position matching in each frame of the video. At the same time, since the present application does not incorporate manual experience, it is not necessary to readjust the model parameters when changing the scene, thereby solving the problem that the existing video target tracking method based on attention feature fusion has a large number of parameters and a long detection time, while video target detection requires detecting multiple images in a short time, resulting in poor real-time performance of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 A schematic diagram of a process of a video target tracking method provided according to an embodiment of the present application is shown;

[0040] Figure 2 A schematic diagram showing the principle of the processing process of the initial acquisition video provided in the embodiment of the present application is shown;

[0041] Figure 3 A schematic diagram of a process flow of another video target tracking method provided according to an embodiment of the present application is shown;

[0042] Figure 4 A structural block diagram of a video target tracking device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0047] Instance segmentation is a computer vision algorithm that aims to label each pixel in an image as belonging to which object instance. Unlike semantic segmentation, instance segmentation can not only distinguish the boundaries between different objects, but also distinguish different instances within the same category. Instance segmentation algorithms usually use deep learning techniques, such as convolutional neural networks (CNN) and region proposal networks (RPN), to achieve accurate segmentation of different object instances in an image. This algorithm is widely used in many computer vision tasks, such as object detection, image segmentation, and autonomous driving.

[0048] The Hungarian algorithm is a classic algorithm for solving bipartite graph matching problems. Its basic idea is to find the maximum matching by continuously augmenting the path until no more augmenting paths can be found. The time complexity of the Hungarian algorithm is O(V*E), where V is the number of vertices and E is the number of edges. This algorithm is widely used in practical applications, such as solving marriage stability problems and task allocation problems.

[0049] As introduced in the background technology, crowd target tracking is a challenging research field in computer vision, and the existing technical solutions are as follows. The first existing solution is a drone video multi-target tracking method based on attention feature fusion, which uses a deep learning network based on the attention mechanism for target detection, and then uses Kalman filtering for target tracking. The second existing solution is a video target tracking and prediction method and system based on a graph neural network, which uses the Faster RCNN model to generate a spatial graph of target categories and target relationship categories, then tracks the target, and uses a long short-term memory network to predict the target trajectory. The third existing solution is a video target tracking method based on Kalman filtering. Based on YOLOv5, this method uses Kalman filtering for smooth target state estimation, and uses the Hungarian algorithm to achieve efficient data association, thereby improving the stability and accuracy of target tracking. In order to solve the problem that the video target tracking method based on attention feature fusion in the existing solution has a large number of parameters and a long detection time, and the video target detection needs to detect multiple images in a short time, so that the real-time performance of target tracking is poor, the embodiments of the present application provide a video target tracking method, device, medium and system.

[0050] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0051] In this embodiment, a video target tracking method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] Figure 1 FIG. 1 is a flow chart of a video target tracking method provided according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0053] Step S101, obtaining an initial acquisition video, and preprocessing the initial acquisition video to convert each frame image in the initial acquisition video into a color frame image to obtain a first acquisition video;

[0054] During preprocessing, the video from a static camera is first converted into color frames Where N is the total number of frames, and each color frame is filtered out through an adaptive median filter (AMF) to remove noise and make the image have smooth contour edges. The principle of the adaptive median filter is to use the average value of adjacent pixels to replace the noise pixels, so as to achieve the purpose of filtering and smoothing, and finally perform histogram equalization on the filtered image.

[0055] In the process of preprocessing the initial captured video in step S101 to convert each frame image in the initial captured video into a color frame image to obtain the first captured video, the method further includes:

[0056] use Adjusting the contrast of each frame of the image in the initial captured video;

[0057] Among them, L-1 represents white, 0 represents black, and r j is the jth input image intensity to be processed, p r is the probability density function of r, s k is the contrast of the k-frame image after processing.

[0058] After the image preprocessing is completed, the proposed instance segmentation method is used for multi-target detection. The image is composed of pixels, and instance segmentation is to classify each pixel into a specific label class to obtain an image segmented by instance, and each person is an instance. The encoder-decoder structure is used here. The encoder is used to obtain complex and detailed semantic information in the image, and the decoder is used to parse and restore the complex semantic information in the image to generate a segmentation result with refined boundaries. The encoder uses the Alessi spatial pyramid pooling module, and uses Alessi convolution modules of different sizes to extract multi-scale features. Multi-scale contextual information can be extracted through the encoder. In the decoder part, the features are first bilinearly upsampled by a multiple of 4, and then the low-level features are connected. In addition, in order to smooth the training and increase the importance of the encoder features, this application reduces the channels by adding 1×1 convolutions to the underlying features. Finally, after the features are improved by 3×3 convolutions, bilinear upsampling is performed again by a multiple of 4 to obtain the segmented image. Separable convolution and dilated convolution can significantly reduce the number of model parameters and computation while maintaining similar or better performance. Separable convolution can increase the running speed of the model while reducing the complexity of the model.

[0059] After instance segmentation, only pixels belonging to humans are extracted, and all pixels except those of humans are set to zero using the following formula:

[0060]

[0061] Among them, I(x, y) is an image containing only human pixels, b(x, y) is a synthetic image containing only human contours, and u is between [0, (L-1)].

[0062] That is, the processing process of the initial acquisition video is as follows Figure 2 shown.

[0063] Step S102, extracting a target outline from the first collected video using an instance segmentation algorithm, and verifying whether the target outline is a human outline using human motion analysis and a gravity model;

[0064] Specifically, the robust instance segmentation algorithm is used for foreground extraction, which is a necessary step for crowd estimation, tracking and behavior analysis in crowded scenes. In order to ensure the accuracy of the model and improve the precision of the model with fewer parameters, the gravity model and the human motion analysis are fused by weighted average of the human motion analysis and gravity model to remove non-human and non-pedestrian pixels in the scene. The dynamic template matching method is used to establish a trajectory of all pedestrians in a scene by associating and matching positions in each frame of the video.

[0065] In order to ensure the accuracy of target detection, human body contour verification is performed before the next target tracking step. By combining human motion analysis and gravity model with weighted average, all objects other than the human body are eliminated to achieve accurate target tracking. The principle is to use the star skeleton strategy to determine the law of motion contour change over time. In order to generate a star skeleton on the extracted contour, the center of mass is first found, and then three extreme points are obtained by traversing the boundary. Then, the center of mass of each contour is connected to the three extreme points with lines. The human body movement mode is analyzed by calculating the angle change between the lines, so as to achieve accurate tracking prediction.

[0066] Step S103, when it is determined that the target outline is the human body outline, a fuzzy mean algorithm is used to perform pedestrian counting processing on the first collected video to obtain the total number of pedestrians;

[0067] In the crowd counting step, the fuzzy mean algorithm is used for cluster estimation to count the detected human silhouettes.

[0068] The basic principles are as follows:

[0069] Objective function:

[0070] Fuzzy mean clustering obtains cluster centers by minimizing the objective function. The objective function is essentially the sum of the Euclidean distances from each point to each cluster. The clustering process is the process of minimizing the objective function. Through repeated iterative operations, the error value of the objective function is gradually reduced. When the objective function converges, the final clustering result can be obtained. The objective function is as follows:

[0071]

[0072] Among them, J m is the objective function value of the cluster family of the mth cluster, N is the number of samples, C is the number of cluster centers of the jth cluster, c j represents the jth cluster center, x i represents the i-th sample, Represents sample x i For cluster center c j The degree of membership.

[0073] ||*|| is any measure of data similarity, the most common of which is the Euclidean norm:

[0074]

[0075] d represents the Euclidean distance of each sample. The distance is used to represent the measure of the similarity of vector x. It is calculated using the Euclidean norm L2 norm and represents the similarity of vector x in the Euclidean space.

[0076] Membership matrix uij and cluster center c j :

[0077] The membership matrix is ​​an N*C matrix, which indicates the degree to which each sample point belongs to each class. i , the sum of its membership for each cluster is 1, and each sample point is assigned to the class to which it has the largest membership. The closer it is to 1, the higher the membership, and vice versa. Find the cluster center c of each group j , which minimizes the objective function. This ensures the clustering principle of the highest similarity within the group and the lowest similarity between groups. k is the kth cluster center.

[0078]

[0079] Termination conditions:

[0080]

[0081] ε is a small constant representing the error threshold, and t is the number of iterations.

[0082] Since each contour is a cluster composed of a set of particles, we first label these clusters in each video frame using a labeling formula;

[0083] Using the marking formula: L c =I m p k , mark the clusters of each frame of video, each of the above clusters includes all particles that constitute the above target contour;

[0084] Among them, p k is the total number of particles in the target contour in cluster k, L c The post-processed data for the single target profile is I m is the label of the mth contour.

[0085] In addition, the present application also draws a rectangular frame for each outline to make it more visually obvious. By labeling and clustering these data, all verified pedestrians can be counted. Since the number of outlines in each video frame is different, the number of particles in each outline is also different. However, the counting method based on the clustering process is only applicable to scenes with clear pedestrian outlines. In real scenes, due to factors such as pedestrians being close to each other or being blocked by obstacles, the captured pedestrian outlines may be incomplete or missing, which will affect the pedestrian count. Therefore, in order to solve this problem, the present application proposes a particle-based measurement scheme. Since a single pedestrian is usually composed of a specific number of particles, the minimum number of particles required for a single pedestrian is first measured, and then the total number of pedestrians is calculated using the total number of pedestrians calculation formula.

[0086] Calculation formula based on the total number of pedestrians: Determine the total number of pedestrians mentioned above;

[0087] Among them, H k is the total number of pedestrians in cluster k, p n∈k is the average number of particles required for a single pedestrian, p k is the total number of particles in the target contour in cluster k.

[0088] Step S104, using the Hungarian algorithm to match the pedestrians in the first captured video based on the total number of pedestrians, and deleting some frame images in the first captured video to obtain a second captured video, wherein the second captured video represents the tracking trajectory video of each pedestrian corresponding to the total number of pedestrians.

[0089] In the above steps, the instance segmentation algorithm is used for foreground extraction, which is a necessary step for crowd estimation, tracking and behavior analysis in crowded scenes; in order to ensure the accuracy of the model and improve the model precision with fewer parameters, the gravity model and the human motion analysis are fused by weighted averaging of the human motion analysis and the gravity model to remove non-human and non-pedestrian pixels in the scene; the Hungarian algorithm is used to ensure real-time performance by performing association and position matching in each frame of the video. At the same time, because this application does not add manual experience, it does not need to readjust the model parameters when changing the scene, thereby solving the problem that the existing video target tracking method based on attention feature fusion has a large number of parameters and a long detection time, and video target detection needs to detect multiple images in a short time, which makes the real-time performance of target tracking poor.

[0090] Step S104, i.e., using the Hungarian algorithm to match pedestrians in the first collected video based on the total number of pedestrians, and deleting some frame images in the first collected video, includes:

[0091] Using the preset detection list and the preset tracking list, calculate the center distance and IoU score and store them in the target matrix, and sort each of the above IoU scores using the corresponding cost function;

[0092] The Hungarian algorithm is used to search for the minimum tracking value of each detection in the target matrix to obtain a detection tracking matching matrix, wherein the detection tracking matching matrix represents a matrix matching between the preset detection list and the preset tracking list;

[0093] When the pedestrian is not matched in a preset number of consecutive frames, the corresponding image frame in the first captured video is deleted.

[0094] In the target tracking algorithm, the Hungarian algorithm is first used to associate a person from one image to another based on the center distance and IoU value of the bounding boxes in the two images. The algorithm includes:

[0095] Define two empty lists, one for tracking and one for detection;

[0096] Calculate the center distance and IoU score through the detection and tracking list and store it in a matrix, sort each score using the cost function;

[0097] Run the Hungarian algorithm, which uses bipartite graph logic to search the matrix for the minimum track value for each detection, resulting in a matrix describing the matches between detections and tracks.

[0098] In complex cases where bounding boxes overlap, a single candidate box has two or more matches. By setting the maximum IoU to 1, all matches are set to 0. To facilitate the search for the minimum value, 1 is replaced by -1.

[0099] If the target is not matched in 5 consecutive frames, it will be deleted. For unmatched detection, a new target will be initialized. If the new target exists in 3 frames, it will be retained, otherwise it will be deleted.

[0100] The result of the Hungarian algorithm association is used as the input of the target tracking algorithm. Target tracking uses the dynamic template matching method to establish the trajectory of all pedestrians in the scene by associating and matching positions in each frame of the video. The key to the problem is to search for where the pedestrian appears in the target image, or at least where a sufficiently similar template appears. To this end, this application uses a correlation coefficient to measure the similarity between each pedestrian position in the target image and the reference target. For each pedestrian position and the sub-image reference target position, the value of the correlation coefficient is the largest.

[0101] The tracking module of the present application obtains the position information from the detection module and associates it with the pedestrian, and then uses this association information to extract the reference template from the previous frame image. If the same rectangle is found when searching in the frame captured from that point, the module flashes the same rectangle above the pedestrian, otherwise, it indicates that a new template has been generated. Since the template used for searching is dynamically generated, a dynamic template matching algorithm is demonstrated. Therefore, data association is used, normalized cross-correlation is used as the cost function and dynamic template matching is used to associate and track the predicted and detected pedestrians. In the tracking method of the present application, each pedestrian is represented as a rectangle.

[0102] By associating and matching positions in each frame of the video, a trajectory of all pedestrians in the scene is established.

[0103] In one embodiment of the present application, the above method further includes:

[0104] When it is determined that the target contour is not a human contour, the instance segmentation algorithm is used again to extract the target contour from the first captured video until the target contour is determined to be a human contour, thereby ensuring successful tracking.

[0105] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the video object tracking method of the present application will be described in detail below in conjunction with specific embodiments.

[0106] This embodiment relates to a specific video target tracking method, such as Figure 3 As shown, the following steps are included:

[0107] Acquire an initial acquisition video, and preprocess the initial acquisition video to convert each frame image in the initial acquisition video into a color frame image to obtain a first acquisition video;

[0108] An instance segmentation algorithm is used to extract a target contour from the first acquired video, and human motion analysis and a gravity model are used to verify whether the target contour is a human contour;

[0109] When it is determined that the target outline is a human body outline, a fuzzy mean algorithm is used to perform pedestrian counting processing on the first collected video to obtain the total number of pedestrians;

[0110] Define two empty lists, one for tracking and one for detection;

[0111] Calculate the center distance and IoU score through the detection and tracking list and store it in a matrix, sort each score using the cost function;

[0112] Run the Hungarian algorithm, which uses bipartite graph logic to search the matrix for the minimum track value for each detection, resulting in a matrix describing the matches between detections and tracks.

[0113] In complex cases where bounding boxes overlap, a single candidate box has two or more matches. By setting the maximum IoU to 1, all matches are set to 0. To facilitate the search for the minimum value, 1 is replaced by -1.

[0114] If the target is not matched in 5 consecutive frames, it will be deleted. For the unmatched detection, a new target will be initialized. If the new target exists in 3 frames, it will be retained, otherwise it will be deleted to obtain the second captured video.

[0115] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0116] The embodiment of the present application also provides a video target tracking device. It should be noted that the video target tracking device of the embodiment of the present application can be used to execute the method for video target tracking provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0117] The following is an introduction to the video target tracking device provided in an embodiment of the present application.

[0118] Figure 4 FIG. 1 is a structural block diagram of a video target tracking device provided according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0119] The acquisition unit 41 is used to acquire an initial acquisition video and preprocess the initial acquisition video to convert each frame image in the initial acquisition video into a color frame image to obtain a first acquisition video;

[0120] A first processing unit 42 is used to extract a target contour from the first collected video using an instance segmentation algorithm, and verify whether the target contour is a human contour using human motion analysis and a gravity model;

[0121] The second processing unit 43 is used to perform pedestrian counting processing on the first collected video by using a fuzzy mean algorithm to obtain the total number of pedestrians when the target outline is determined to be the human body outline;

[0122] The third processing unit 44 is used to match the pedestrians in the first captured video based on the total number of pedestrians using the Hungarian algorithm, and delete some frame images in the first captured video to obtain a second captured video, wherein the second captured video represents the tracking trajectory video of each pedestrian corresponding to the total number of pedestrians.

[0123] In the above device, an instance segmentation algorithm is used for foreground extraction, which is a necessary step for crowd estimation, tracking and behavior analysis in crowded scenes; in order to ensure the accuracy of the model and improve the model precision with fewer parameters, the gravity model and the human motion analysis are fused by weighted averaging of the human motion analysis and the gravity model to remove non-human and non-pedestrian pixels in the scene; the Hungarian algorithm is used to ensure real-time performance by performing association and position matching in each frame of the video. At the same time, because the present application does not add manual experience, it is not necessary to readjust the model parameters when changing the scene, thereby solving the problem that the existing video target tracking method based on attention feature fusion has a large number of parameters and a long detection time, and the video target detection needs to detect multiple images in a short time, which makes the real-time performance of target tracking poor.

[0124] In one embodiment of the present application, the acquisition unit includes a first processing module,

[0125] The first processing module is used for preprocessing the initial acquisition video to convert each frame image in the initial acquisition video into a color frame image to obtain the first acquisition video.

[0126] use Adjusting the contrast of each frame of the image in the initial captured video;

[0127] Among them, L-1 represents white, 0 represents black, and r j is the jth input image intensity to be processed, p r is the probability density function of r, s k is the contrast of the k-frame image after processing.

[0128] In one embodiment of the present application, the second processing unit includes a second processing module, and the second processing module is used to construct an objective function in the process of using a fuzzy mean algorithm to perform pedestrian counting processing on the first collected video to obtain the total number of pedestrians:

[0129]

[0130] Among them, J m is the objective function value of the cluster family of the mth cluster, N is the number of samples, C is the number of cluster centers of the jth cluster, c j represents the jth cluster center, x i represents the i-th sample, Represents sample x i For cluster center c j The degree of membership.

[0131] In one embodiment of the present application, the second processing unit includes a third processing module, which is used to use L to calculate the total number of pedestrians in the process of using the fuzzy mean algorithm to perform pedestrian counting processing on the first collected video. c =I m p k , mark the clusters of each frame of video, each of the above clusters includes all particles that constitute the above target contour;

[0132] Among them, p k is the total number of particles in the target contour in cluster k, L c The post-processed data for the single target profile is I m is the label of the mth contour.

[0133] In one embodiment of the present application, the second processing unit includes a fourth processing module,

[0134] The fourth processing module is used to Determine the total number of pedestrians mentioned above;

[0135] Among them, H k is the total number of pedestrians in cluster k, p n∈k is the average number of particles required for a single pedestrian, p k is the total number of particles in the target contour in cluster k.

[0136] In one embodiment of the present application, the third processing unit includes a fifth processing module, a sixth processing module and a seventh processing module.

[0137] The fifth processing module is used to use the preset detection list and the preset tracking list, calculate the center distance and the IoU score and store them in the target matrix, and sort each of the above IoU scores using the corresponding cost function;

[0138] The sixth processing module is used to use the Hungarian algorithm to search for the minimum tracking value of each detection in the target matrix to obtain a detection tracking matching matrix, where the detection tracking matching matrix represents a matrix matching between the preset detection list and the preset tracking list;

[0139] The seventh processing module is used to delete the corresponding image frame in the first captured video when the pedestrian is not matched in a continuous preset number of frames.

[0140] In one embodiment of the present application, the above-mentioned device further includes a fourth processing unit,

[0141] The fourth processing unit is used for, when it is determined that the target contour is not a human contour, again using an instance segmentation algorithm to extract the target contour from the first collected video until it is determined that the target contour is a human contour.

[0142] The video target tracking device includes a processor and a memory. The acquisition unit, the first processing unit, the second processing unit and the third processing unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0143] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to solve the problem that the existing solution based on attention feature fusion video target tracking method has a large number of parameters and a long detection time, while video target detection needs to detect multiple images in a short time, which makes the real-time performance of target tracking poor.

[0144] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0145] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the video target tracking method.

[0146] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the video target tracking method is executed when the program is run.

[0147] The embodiment of the present invention provides a device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements at least the following steps when executing the program: obtaining an initial acquisition video, and preprocessing the initial acquisition video to convert each frame image in the initial acquisition video into a color frame image to obtain a first acquisition video; extracting a target contour from the first acquisition video using an instance segmentation algorithm, and verifying whether the target contour is a human contour using human motion analysis and a gravity model; when determining that the target contour is a human contour, performing pedestrian counting processing on the first acquisition video using a fuzzy mean algorithm to obtain the total number of pedestrians; matching pedestrians in the first acquisition video based on the total number of pedestrians using a Hungarian algorithm, and deleting some frame images in the first acquisition video to obtain a second acquisition video, wherein the second acquisition video represents the tracking trajectory video of each pedestrian corresponding to the total number of pedestrians. The device in this article may be a server, a PC, a PAD, a mobile phone, etc.

[0148] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with at least the following method steps: obtaining an initial acquired video, and preprocessing the initial acquired video to convert each frame image in the initial acquired video into a color frame image to obtain a first acquired video; using an instance segmentation algorithm to extract a target contour from the first acquired video, and using human motion analysis and a gravity model to verify whether the target contour is a human contour; when it is determined that the target contour is the human contour, using a fuzzy mean algorithm to perform pedestrian counting processing on the first acquired video to obtain the total number of pedestrians; using a Hungarian algorithm to match pedestrians in the first acquired video based on the total number of pedestrians, and deleting some frame images in the first acquired video to obtain a second acquired video, wherein the second acquired video represents the tracking trajectory video of each pedestrian corresponding to the total number of pedestrians.

[0149] The present application also provides a video target tracking system, including: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the above methods. The example segmentation algorithm is used for foreground extraction, which is a necessary step for crowd estimation, tracking and behavior analysis in crowded scenes; in order to ensure the accuracy of the model and improve the accuracy of the model with fewer parameters, the gravity model and the human motion analysis are fused by the weighted average method of the human motion analysis and the gravity model to remove non-human and non-pedestrian pixels in the scene; the Hungarian algorithm is used to associate and position match in each frame of the video. At the same time, because the present application does not add artificial experience, it does not need to readjust the model parameters when changing the scene, thereby solving the problem that the existing video target tracking method based on attention feature fusion has a large number of parameters and a long detection time, and the video target detection needs to detect multiple images in a short time, which makes the real-time performance of the target tracking poor.

[0150] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0152] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0156] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0157] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0158] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0159] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0160] 1) The video target tracking method of the present application uses an instance segmentation algorithm for foreground extraction, which is a necessary step for crowd estimation, tracking and behavior analysis in crowded scenes; in order to ensure the accuracy of the model and improve the model precision with fewer parameters, the gravity model and the human motion analysis are fused by weighted averaging of the human motion analysis and the gravity model to remove non-human and non-pedestrian pixels in the scene; the Hungarian algorithm is used to ensure real-time performance by performing association and position matching in each frame of the video. At the same time, because the present application does not add manual experience, it does not need to readjust the model parameters when changing the scene, thereby solving the problem that the existing video target tracking method based on attention feature fusion has a large number of parameters and a long detection time, and video target detection needs to detect multiple images in a short time, which makes the real-time performance of target tracking poor.

[0161] 2) The video target tracking device of the present application uses an instance segmentation algorithm for foreground extraction, which is a necessary step for crowd estimation, tracking and behavior analysis in crowded scenes; in order to ensure the accuracy of the model and improve the model precision with fewer parameters, the gravity model and the human motion analysis are fused by weighted averaging of the human motion analysis and the gravity model to remove non-human and non-pedestrian pixels in the scene; the Hungarian algorithm is used to ensure real-time performance by performing association and position matching in each frame of the video. At the same time, because the present application does not add manual experience, it does not need to readjust the model parameters when changing the scene, thereby solving the problem that the existing video target tracking method based on attention feature fusion has a large number of parameters and a long detection time, while video target detection needs to detect multiple images in a short time, which makes the real-time performance of target tracking poor.

[0162] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A video target tracking method, characterized in that: include: Acquire an initial acquisition video, and preprocess the initial acquisition video to convert each frame image in the initial acquisition video into a color frame image to obtain a first acquisition video; Extracting a target outline from the first collected video using an instance segmentation algorithm, and verifying whether the target outline is a human outline using human motion analysis and a gravity model; When it is determined that the target outline is the human body outline, a fuzzy mean algorithm is used to perform pedestrian counting processing on the first collected video to obtain the total number of pedestrians; The Hungarian algorithm is used to match the pedestrians in the first collected video based on the total number of pedestrians, and some frame images in the first collected video are deleted to obtain a second collected video, where the second collected video represents the tracking trajectory video of each pedestrian corresponding to the total number of pedestrians.

2. The method according to claim 1, characterized in that In the process of preprocessing the initial captured video to convert each frame image in the initial captured video into a color frame image to obtain the first captured video, the method further includes: use Adjusting the contrast of each frame of the image in the initially acquired video; Among them, L-1 represents white, 0 represents black, and r j is the jth input image intensity to be processed, p r is the probability density function of r, s k is the contrast of the k-frame image after processing.

3. The method according to claim 1, characterized in that In the process of using the fuzzy mean algorithm to perform pedestrian counting processing on the first collected video to obtain the total number of pedestrians, the method further includes: The objective function is constructed as: Among them, J m is the objective function value of the cluster family of the mth cluster, N is the number of samples, C is the number of the jth cluster center, cj represents the jth cluster center, xi represents the ith sample, Represents the sample xi to the cluster center c j The degree of membership.

4. The method according to claim 1, characterized in that: In the process of using the fuzzy mean algorithm to perform pedestrian counting processing on the first collected video to obtain the total number of pedestrians, the method further includes: Using L c =I m p k , marking the clusters of each frame of video, each of the clusters includes all particles constituting a contour of the target; Among them, p k is the total number of particles in the target contour in cluster k, L c The post-processed data for a single target profile, I m is the label of the mth contour.

5. The method according to claim 1, characterized in that The fuzzy mean algorithm is used to perform pedestrian counting processing on the first collected video to obtain the total number of pedestrians, including: according to determining the total number of pedestrians; Among them, H k is the total number of pedestrians in cluster k, p n∈k is the average number of particles required for a single pedestrian, p k is the total number of particles in the target contour in cluster k.

6. The method according to claim 1, characterized in that The Hungarian algorithm is used to match pedestrians in the first collected video based on the total number of pedestrians, and some frame images in the first collected video are deleted, including: Using a preset detection list and a preset tracking list, calculating the center distance and IoU score and storing them in a target matrix, and sorting each of the IoU scores using a corresponding cost function; The Hungarian algorithm is used to search for the minimum tracking value of each detection in the target matrix to obtain a detection tracking matching matrix, wherein the detection tracking matching matrix represents a matrix matching between the preset detection list and the preset tracking list; When the pedestrian is not matched in a preset number of consecutive frames, the corresponding image frames in the first captured video are deleted.

7. The method according to claim 1, characterized in that The method further comprises: When it is determined that the target contour is not a human contour, an instance segmentation algorithm is used again to extract the target contour from the first captured video until it is determined that the target contour is the human contour.

8. A video target tracking device, characterized in that: include: An acquisition unit, used for acquiring an initial acquisition video, and preprocessing the initial acquisition video to convert each frame image in the initial acquisition video into a color frame image to obtain a first acquisition video; A first processing unit is used to extract a target contour from the first collected video by using an instance segmentation algorithm, and verify whether the target contour is a human contour by using human motion analysis and a gravity model; A second processing unit is used to perform pedestrian counting processing on the first collected video by using a fuzzy mean algorithm to obtain a total number of pedestrians when it is determined that the target outline is the human body outline; The third processing unit is used to match the pedestrians in the first collected video based on the total number of pedestrians by using the Hungarian algorithm, and delete some frame images in the first collected video to obtain a second collected video, wherein the second collected video represents the tracking trajectory video of each pedestrian corresponding to the total number of pedestrians.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A video target tracking system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images