Target capturing method and system in video sequence scene and medium

Abstract

The invention discloses a target capturing method and system in a video sequence scene and a medium, and belongs to the field of image single target tracking, and the method comprises the steps: carrying out the target detection of a video sequence image, obtaining the position and size of a target, and deleting the target of which the size is not in a specified range; initializing a multi-target tracker by using the position of the undeleted target so as to perform multi-target tracking and accumulated confidence calculation; for each target tracker, when the tracking hit times of the target tracker reach a first set value and the accumulated confidence reaches a set confidence threshold, outputting the tracked target and the accumulated confidence, and when the tracking continuous miss times of the target tracker reach a second set value, deleting the target tracker; and initializing a single-target tracker by using the target with the highest cumulative confidence coefficient to perform single-target tracking and tracking confidence coefficient calculation and output, and performing cross validation on tracking output results of the two types of targets to determine whether to output or not and output results. The method realizes stable capture and identification of the target.

Description

Technical Field

[0001] The present invention belongs to the field of image single target tracking, and more specifically, relates to a target capturing method, system and medium in a video sequence scene. Background Art

[0002] As an information-intensive carrier, video plays an increasingly important role in modern society. Whether it is security monitoring in daily life or complex military surveillance tasks, video data provides a rich source of information. Among them, accurate detection and stable tracking of targets in videos is one of the key technologies for realizing applications such as behavior recognition analysis, intelligent monitoring analysis, abnormal event detection and even military monitoring. This technology not only has broad application prospects in the civilian field, such as intelligent transportation, smart cities, etc., but also has strategic significance in the military field. For example, target recognition, tracking, and battlefield situation awareness are hot topics for development and competitive research in countries around the world.

[0003] However, there are still some problems with the existing technical means of target detection and tracking in video sequence scenarios. The traditional tracking method adopts the method of first detecting and then directly tracking. This method is extremely dependent on the results of the first detection and the performance of the subsequent tracking method. Once a false alarm occurs in the initial detection of the target, it will cause tracking initialization errors, thereby affecting the accuracy of subsequent tracking. If the target area is re-detected at regular intervals during the tracking process to verify the accuracy of the tracking results, although this method avoids the impact of the initial capture, if a false alarm occurs during the verification, it may cause tracking failure, and ultimately still affect the performance of tracking capture. In addition, factors such as the appearance, movement pattern, and shooting environment of the target will change with the changes in time and space. Simple tracking algorithms are difficult to adapt to these changes and will also cause tracking failures.

[0004] Therefore, the existing target detection and tracking technology for video scenes still has bottleneck problems and urgently needs further research and development. How to improve the anti-interference ability of algorithm detection, the stability and accuracy of tracking, reduce false alarms and missed detections, and ensure effective detection and tracking of various targets in various complex environments is of great research significance. Summary of the invention

[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a target capture method, system and medium in a video sequence scenario, which aims to improve the anti-interference ability, tracking stability and accuracy of the target capture method in the video sequence scenario, reduce false alarms and missed detections, and ensure effective detection and tracking of various targets in various complex environments.

[0006] To achieve the above-mentioned purpose, according to one aspect of the present invention, a method for capturing targets in a video sequence scene is provided, comprising: S1, performing target detection on an input video sequence image to obtain the position and size of all targets in each frame of the image; S2, designing an imaging size range of the desired target according to its theoretical size in the image, deleting targets whose sizes in S1 are not within the imaging size range, and outputting the targets that are not deleted; S3, initializing a multi-target tracker using the positions of all targets output in S2 to perform multi-target tracking and cumulative confidence calculation on the video sequence image; during the multi-target tracking process, for each target tracker, when its tracking hit count reaches a first set value and the cumulative set value is reached, the target tracker is initialized. When the confidence reaches the set confidence threshold, the tracked target and the cumulative confidence are output. When the number of consecutive misses in tracking reaches a second set value, the target tracker is deleted; S4, the single target tracker is initialized using the target with the highest cumulative confidence to perform single target tracking and tracking confidence calculation on the video sequence image and output it; S5, the target and cumulative confidence output in S3 and the target and tracking confidence output in S4 are cross-validated using the intersection-union-ratio matching method to determine whether to output the target tracked in S4, or to output the target with the largest cumulative confidence tracked in S3, or to output the fusion result of the targets matched in S4 and S3, or not to output it.

[0007] Furthermore, the method also includes: judging whether each frame image is captured according to the result of cross-validation in S5; when the number of frames in which the image is not captured in a consecutive number of frames reaches a third set value, exiting the tracking, if there is a target output in S3, re-executing S4 and S5, and if there is no output target in S3, no output.

[0008] Furthermore, S1 specifically includes: preprocessing the input video sequence images; performing target detection on the preprocessed images to obtain a horizontal detection frame; using border shrinkage to find the minimum circumscribed rectangle of the target in the horizontal detection frame to obtain a rotated detection frame, and the position and size of the rotated detection frame are the position and size of the target therein.

[0009] Furthermore, the method of using border shrinkage to search for the minimum bounding rectangle of the target in the horizontal detection frame specifically includes: performing saliency detection, image segmentation, filtering and edge detection on the image in the horizontal detection frame in sequence to achieve border shrinkage, and obtaining the minimum bounding rectangle of the frame after border shrinkage as the rotation detection frame.

[0010] Furthermore, the imaging size range is:

[0011] Th wmax =target w ×rate w

[0012] Th hmax =target h ×rate w

[0013] Th wmin =target w ÷rate h

[0014] Th hmin =target h ÷rate h

[0015]

[0016] Among them, Th wmax ,Th hmax ,Th wmin ,Th hmin are the upper limit of imaging length, upper limit of imaging width, lower limit of imaging length, and lower limit of imaging width of the desired target, respectively. w 、target h are the theoretical length and width of the target in the image, respectively. The theoretical size includes target w and target h , rate w is the length scaling factor, rate h is the width scaling factor, base is the first adjustable parameter, α is the second adjustable parameter, β is the third adjustable parameter, and the imaging size range includes Th wmax ,Th hmax ,Th wmin and Th hmin .

[0017] Furthermore, the multi-target tracking of the video sequence images specifically includes: for each target tracker, using the target tracker to track the video sequence images, if the tracked target matches the corresponding target obtained by target detection, the tracking is successful, and the target position obtained by target detection is used to correct the position of the matched corresponding tracked target; if the tracked target does not match all targets obtained by target detection, the tracking is a miss; for each target output in S2, if the target does not match all targets obtained by the multi-target tracker, a new target tracker is initialized for multi-target tracking.

[0018] Furthermore, when the tracking hit count is greater than a fourth set value, the cumulative confidence is:

[0019] cumConf←cumConf×(1-cum_factor)+conf×cum_factor

[0020] When the tracking hit count is not greater than the fourth set value, the cumulative confidence is:

[0021] cumConf←1-(1-cumConf)×(1-conf×cum_factor)

[0022] Among them, cumConf is the cumulative confidence, the left and right sides are the current cumulative confidence and the previous cumulative confidence respectively, cum_factor is the adjustable weight, and conf is the target detection confidence of the current frame.

[0023] Furthermore, the tracking confidence is: a correlation score of the position of the target tracked by a single target.

[0024] According to another aspect of the present invention, a target capture system in a video sequence scenario is provided, comprising: a processor; a memory storing a computer executable program, wherein when the program is executed by the processor, the processor executes the target capture method in the video sequence scenario as described above.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the target capturing method in the video sequence scenario as described above is implemented.

[0026] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0027] (1) A target capture method in a video sequence scenario is provided. After preliminary target detection is performed on an image, some detected targets are initially filtered out according to the imaging size of the target in the image to achieve spatial filtering. The undeleted targets are passed to a multi-target tracking module. At this stage, temporal filtering is implemented to eliminate occasional false detections in the detection and to achieve multi-frame accumulation of confidence. The output is controlled by setting a cumulative number threshold (a first set value) and a confidence threshold. The output result is sent to a single target tracking module, and the target with the highest confidence is initially selected to initialize single target tracking to achieve capture. Subsequently, the single target tracking module completes the tracking prediction of the captured target by itself each time, and completes cross-validation with the target output by the multi-target tracking to determine whether the target is captured and tracked correctly, and implements capture change in a timely manner to ensure that the target is captured and tracked correctly.

[0028] (2) An optimal target detection method is provided. After obtaining a horizontal detection frame, the minimum circumscribed rectangle of the target is found in the horizontal detection frame, and a rotation detection frame is obtained to determine the position and size of the target, which has higher target detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flow chart of a method for capturing a target in a video sequence scenario provided by an embodiment of the present invention;

[0030] Figure 2 An example diagram of controllable generation of cloud and fog interference provided by an embodiment of the present invention;

[0031] Figure 3 A flowchart of a border shrinking algorithm implementation provided by an embodiment of the present invention;

[0032] Figure 4 An example diagram of imaging size calculation provided by an embodiment of the present invention;

[0033] Figure 5 SORT multi-target tracking time series information filtering flow chart provided for the embodiment of the present invention

[0034] Figure 6 DSST single target tracking prediction flow chart provided for the embodiment of the present invention

[0035] Figure 7 A DSST and SORT cross-validation flow chart provided in one embodiment of the present invention;

[0036] Figure 8 A DSST and SORT cross-validation flow chart is provided for another embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0039] Embodiment 1

[0040] A method for capturing objects in video sequence scenarios. Figure 1 , combined with Figure 2-Figure 8, the target capturing method in the video sequence scenario in this embodiment is described in detail, and the method includes operations S1 to S5.

[0041] Operation S1, performing object detection on the input video sequence images to obtain the positions and sizes of all objects in each frame of the image.

[0042] Preferably, operation S1 specifically includes the following sub-operations S11 to S13.

[0043] In sub-operation S11, the input video sequence images are preprocessed. Specifically, for example, the video sequence is input frame by frame, and image preprocessing such as quantization and denoising is performed on the video sequence before input; or the video sequence can be directly input without preprocessing.

[0044] In sub-operation S12, object detection is performed on the preprocessed image to obtain a horizontal detection frame.

[0045] Specifically, you can directly use traditional target detection methods for target detection; you can also use intelligent methods for target detection, but you need to perform network training before use. When using intelligent methods for target detection, you can choose to directly use rotating box target detection to better represent the position and size of the target; you can also use horizontal box target detection to improve detection speed and accuracy. After detecting the horizontal box, find the target's rotating box in the local area represented by the horizontal detection box.

[0046] Taking the use of yolov5 target detection algorithm for target detection as an example, the specific implementation process of sub-operation S12 is described. Other target detection methods can also be used to implement sub-operation S12.

[0047] Before using yolov5 for target detection, network training must be performed first. In order to improve the anti-interference ability, target images with interference such as occlusion and fog are added during training. Therefore, a controllable generation method of cloud occlusion data based on target maps and a controllable simulation method based on foggy scenes are used to generate target images with cloud occlusion. Figure 2 As shown in the figure, cloud and fog simulation is controlled and generated on the ship target in DOTAv1.0. The cloud layer image is extracted by manual cutout or intelligent cloud layer segmentation method for subsequent image mapping, while the original image is directly selected from the DOTA dataset; the fog images of various degrees use a fog image simulator based on the atmospheric scattering model to simulate a more realistic haze effect at any altitude.

[0048] The core architecture of yolov5 includes four parts: input, Backbone, Neck and Head. The input uses Mosaic data enhancement and adaptive anchor box calculation to enrich the detection data set, especially random scaling adds many small targets, making the network more robust. The two important structures in Backbone are Focus and CSP. Focus is a special downsampling method that can speed up the speed. CSP can solve the problem of large amount of calculation in reasoning and can also play a role in downsampling. Neck adopts the structure of FPN+PAN. FPN conveys strong semantic features from top to bottom, while PAN conveys strong positioning features from bottom to top. It can aggregate parameters of different detection layers from different backbone layers. Head uses the neighborhood positive and negative sample allocation strategy, which can significantly accelerate convergence. As an intelligent target detection method, yolov5 can learn the characteristics of the target through training. At the same time, due to the addition of interference such as cloud and fog occlusion during training, it has anti-interference ability during reasoning test and can achieve accurate detection of targets in images. In addition, during the target detection phase, more attention should be paid to the recall rate, which can ensure that the target is detected as much as possible without worrying about false alarms, because subsequent operations will filter the detection area.

[0049] In sub-operation S13, the minimum circumscribed rectangle of the target is found in the horizontal detection frame by using the frame shrinkage to obtain a rotated detection frame, and the position and size of the rotated detection frame are the position and size of the target in the rotated detection frame.

[0050] Preferably, the minimum bounding rectangle of the target is found in the horizontal detection frame by using the frame shrinkage, which specifically includes: performing saliency detection, image segmentation, filtering and edge detection on the image in the horizontal detection frame in sequence to realize frame shrinkage, and obtaining the minimum bounding rectangle of the frame after the frame shrinkage as the rotation detection frame, such as Figure 3 It should be noted that Figure 3 The method shown is only a preferred method provided by this embodiment, and other frame shrinking methods can also be used to obtain the rotation detection frame.

[0051] by Figure 3 Taking the frame shrinking method shown as an example, the specific implementation process of sub-operation S13 is explained, including the following steps (1) to (5).

[0052] Step (1): firstly extract salient features of the input image using saliency detection to obtain a saliency map. The saliency detection is, for example, FT saliency detection.

[0053] Specifically, the image is first Gaussian smoothed, and then its color space is converted, converting the RGB color space to the CIELAB color space. The average values ​​of L, a, and b of the entire image are calculated, and the Euclidean distance between the L, a, and b values ​​of each pixel after Gaussian smoothing and the average values ​​of L, a, and b of the image are calculated according to the following formula:

[0054]

[0055] Among them, I μ =[L μ a μ b μ ] T represents the mean matrix of image L, a, b, A matrix representing the L, a, and b values ​​of each pixel after Gaussian smoothing. After that, normalization is performed to obtain the final saliency map.

[0056] Step (2), different regions in the image are obtained by image segmentation, and the image segmentation is, for example, OTSU image segmentation. OTSU is an algorithm for determining the threshold of image binarization segmentation, and its essential idea is to maximize the inter-class variance.

[0057] Taking a grayscale image as an example, an image can be regarded as an M×N matrix, that is, the pixels in the image, each value is a pixel value, where the pixel value is between (0, 255). The segmentation threshold of the foreground (i.e., the target) and the background is denoted as optimal threshold , the proportion of foreground pixels in the entire image is recorded as w0, and the average grayscale of the foreground is recorded as μ0; the proportion of background pixels in the entire image is recorded as w1, and its average grayscale is recorded as μ1. The total average grayscale of the image is recorded as μ, and the inter-class variance is recorded as maximum. Assume that the background of the image is dark, and the size of the image is M×N, and the grayscale value of the pixel in the image is less than the threshold optimal threshold The number of pixels is recorded as N0, and the pixel grayscale is greater than or equal to the threshold optimal threshold The number of pixels is recorded as N1, then:

[0058]

[0059] N0+N1=M×N

[0060] w0+w1=1

[0061] μ=w0×μ0+w1×μ1

[0062] maximum=w0×(μ0-μ) 2 +w1×(μ1-μ) 2

[0063] Substituting the expression of μ into the expression of maximum, we get:

[0064] maximum=w0×w1×(μ0-μ1) 2

[0065] The traversal method is used to obtain the maximum threshold that maximizes the variance between classes. After solving the optimal threshold using the OSTU algorithm, the binary method is used for segmentation.

[0066] In step (3), the remaining image noise is suppressed by filtering operation, and the filtering method is, for example, morphological filtering. Morphological filtering includes erosion operation and dilation operation. Erosion operation can eliminate noise points, filter sea clutter, suppress clutter interference, and eliminate some boundary values, which may cause the overall reduction of the target image. Dilation operation can increase the target feature value, resulting in the overall enlargement of the target image. The combination of erosion operation and dilation operation can achieve the purpose of better segmentation of independent graphic elements.

[0067] Step (4), extracting target edge information through edge detection algorithm, the edge detection algorithm is, for example, LSD edge detection method. LSD edge detection method is used to detect local straight contours in an image. First, the horizontal line angle of each pixel is calculated to form a horizontal line field, that is, a unit vector field. Here, the horizontal line angle of the pixel is the vertical angle of the gradient direction of the point, and the horizontal line field is a matrix corresponding to each point in the image. The value of the element in the matrix is ​​the horizontal line angle corresponding to the midpoint of the image. For all horizontal lines in the horizontal field, if the direction is approximately within the tolerance range, these areas are divided into the same support domain, and then the pixel range of the support domain is obtained to obtain the minimum circumscribed rectangle. Compared with traditional edge detection operators such as Canny, LSD edge detection algorithm has stronger robustness when the image is interfered with or blocked.

[0068] Step (5), obtain the directed frame after the frame shrinkage through the minimum circumscribed rectangular frame, and obtain the rotated target detection frame.

[0069] Operation S2, designing an imaging size range according to a theoretical size of the desired target in the image, deleting targets whose sizes are not within the imaging size range in operation S1, and outputting the targets that are not deleted.

[0070] The theoretical size of the desired target in the image can be calculated based on the camera's posture position and the actual size of the desired target. The specific calculation method is as follows.

[0071] In real scenes, the camera’s posture and position are known. For example, taking aerial photography, Figure 4As shown, with the ground as the reference system, the position of the camera is (x, y, z), and the angle attitude is (α, β, γ), where α is the pitch angle, the field of view in the length and width directions are 40° and 30° respectively, the target is on the ground, and the real length and width of the target are real w 、real h ,but:

[0072]

[0073] Therefore, the actual range of imaging is:

[0074]

[0075] The image size is 1920×1280, and the long side (length) and short side (width) of the target in the image can be obtained by scaling proportionally:

[0076]

[0077] Thus, the theoretical size of the desired target in the image is obtained. w and target h ).

[0078] A certain range of thresholds is set according to the calculated imaging size and aspect ratio to determine whether the detection frame exceeds the range. The target frame that exceeds the range is filtered out to eliminate false alarms.

[0079] Preferably, the imaging size range set for the desired target is:

[0080] Th wmax =target w ×rate w

[0081] Th hmax =target h ×rate w

[0082] Th wmin =target w ÷rate h

[0083] Th hmin =target h ÷rate h

[0084]

[0085] Among them, Th wmax ,Th hmax ,Th wmin ,Thhmin are the upper limit of imaging length, upper limit of imaging width, lower limit of imaging length, and lower limit of imaging width of the desired target, respectively. w 、target h are the theoretical length and width of the expected target in the image, respectively. The theoretical size includes target w and target h , rate w is the length scaling factor, rate h is the width scaling factor, base is the first adjustable parameter, α is the second adjustable parameter, β is the third adjustable parameter, and the imaging size range includes Th wmax ,Th hmax ,Th wmin and Th hmin .

[0086] The three parameters base, α and β are introduced in the imaging size range. This is because the width and height of the target in the field of view vary greatly when the distance is different, so there should be different zoom ranges for targets of different sizes. The base value is adjusted to determine the tolerable zoom range of large targets, and the α value is adjusted to change the zoom range of small targets. Preferably, base = 1.2, α = 2.5, β = 0.9.

[0087] By judging whether Th wmin <det w <Th wmax ,Th hmin <det h <Th hmax , delete the rotation detection frame that is not within this range, and complete the prior knowledge spatial information filtering. w 、det h The length and width of the detected object are calculated for the rotation detection frame obtained in operation S1 , respectively.

[0088] Operation S3, using the positions of all targets output in operation S2 to initialize a multi-target tracker to perform multi-target tracking and cumulative confidence calculation on the video sequence images; during the multi-target tracking process, for each target tracker, when its tracking hit count reaches a first set value and the cumulative confidence reaches a set confidence threshold, it outputs the tracked targets and cumulative confidences, and when its tracking consecutive miss counts reaches a second set value, the target tracker is deleted.

[0089] The position of the target output in operation S2 is the result of filtering by prior knowledge and meets the requirements in terms of spatial scale, so it is used to initialize the multi-target tracker. The multi-target tracker adopts, for example, the SORT multi-target tracking algorithm, or other multi-target tracking algorithms.

[0090] The SORT multi-target tracking algorithm consists of three parts: target detection, Kalman filtering, and Hungarian matching. In this embodiment, the target detection result comes from the output of operation S2, so the focus is on Kalman filtering and Hungarian matching. The specific process is as follows: Figure 5 shown.

[0091] After obtaining the output result of the target detection module, the exact position of the target cannot be directly obtained. This is because the detection result may contain errors, resulting in the output detection frame being noisy. Kalman filtering can fuse the target detection result with the prediction result of the tracking algorithm to further improve the accuracy of the detection frame obtained by the detection algorithm.

[0092] Kalman filtering uses the signal and noise state space model, and uses the estimated value of the previous moment and the observed value of the current moment to estimate the state variable and find the estimated value at the moment of occurrence. Kalman filtering assumes that the observation value credibility model is a normal distribution and the control model is a normal distribution. All observable signals change linearly.

[0093] In the SORT multi-target tracking algorithm, the observed variables Where u represents the horizontal pixel position of the target center, v represents the vertical pixel position of the target center, s and r represent the scale (area) and aspect ratio of the target bounding box, respectively. Represent the horizontal speed, vertical speed and area change speed of the target respectively. After the target is detected, the target state is updated with the detected bounding box, where the velocity component is optimized and solved through the Kalman filter framework. If there is no detection associated with the target, the linear velocity model is used to simply predict its state without correction.

[0094] Hungarian matching is mainly used to solve the assignment problem. There are n different tasks, and n people are required to complete one of them respectively. The time it takes for each person to complete the task is different. So there is a question, how to assign tasks so that the time spent is the least. The optimal solution problem mainly relies on the properties of the optimal solution of the matrix: if the minimum element of a row (column) of the matrix is ​​subtracted from each element of the row (column), the reduced matrix is ​​obtained, and its optimal solution is the same as the optimal solution of the original matrix. In the SORT multi-target tracking algorithm, the distance between the detected target and the tracked target is determined by calculating the IOU distance between the two targets, and then generating the corresponding cost matrix. The Hungarian matching algorithm is then used to match the current frame detection result with the tracker prediction result based on the cost matrix.

[0095] Since target detection can only output the confidence of the current frame each time, but the confidence of the target box of the tracking result after accumulation of multiple frames should also be cumulative, the cumulative confidence of multiple frames is used as the confidence of the tracking result of the current frame.

[0096] In one embodiment of the present invention, the cumulative confidence is calculated as follows:

[0097] cumConf←cumConf×(1-cum_factor)+conf×cum_factor

[0098] In another embodiment of the present invention, the cumulative confidence is calculated as follows:

[0099] cumConf←1-(1-cumConf)×(1-conf×cum_factor)

[0100] In another embodiment of the present invention, the cumulative confidence is calculated as follows:

[0101]

[0102] Among them, cumConf is the cumulative confidence, the left and right sides of ← are the current cumulative confidence and the previous cumulative confidence respectively, cum_factor is an adjustable weight, conf is the target detection confidence of the current frame, N is the number of tracking hits, and N4 is the fourth set value. The initial value of cumConf is 0; cum_factor is used to prevent cumConf from growing too fast; N4 is, for example, 20. It should be noted that other cumulative confidence calculation methods are also applicable to this method.

[0103] According to an embodiment of the present invention, during multi-target tracking: for each target tracker, the target tracker is used to track the video sequence images; if the tracked target matches the corresponding target obtained by target detection, the tracking is successful; the target position obtained by target detection is used to correct the position of the matched corresponding tracked target; if the tracked target does not match all targets obtained by target detection, the tracking is a failure; for each target output in operation S2, if the target does not match all targets obtained by the multi-target tracker, a new target tracker is initialized for multi-target tracking.

[0104] When the number of consecutive misses of a tracker reaches the second set value (such as 10), it is considered to have reached the survival period and the tracker will be deleted to remove occasional false alarms. When the cumulative hit count of a tracker reaches the first set value (such as 5) and the cumulative confidence reaches the set confidence threshold (such as 0.5), the target in the current tracker is considered to be a real and valid target and will be output to the subsequent single target tracking module for initialization or cross-validation.

[0105] Operation S4: Initialize a single target tracker using the target with the highest cumulative confidence, so as to perform single target tracking on the video sequence images and calculate and output the tracking confidence.

[0106] Specifically, first, the target with the highest confidence in the results after spatiotemporal information filtering is selected to initialize the single target tracker to complete the initial capture; then, the initialized single target tracker is used to predict the target position and tracking confidence.

[0107] Taking a single target tracker using the DSST single target tracking algorithm as an example, the specific process of operation S4 is described. Figure 6 ,After obtaining the region center in the initialization phase, feature extraction f is performed to train the filter, and after determining the candidate region, feature z is extracted, and the correlation score y is calculated, so as to determine the target center, and iterate continuously to achieve tracking.

[0108] DSST introduces a multi-feature fusion mechanism, using the features of HOG (Histogram of Oriented Gradients) + CN (color features) + grayscale features, and adds a scale filter to cope with the scale changes of the target in the image. Therefore, DSST mainly has two important modules: position filter and scale filter. The two filters are independent of each other and complement each other.

[0109] The position filter can complete the position estimation when tracking a single target. Assuming that the size of the image block P where the target is located is M×N, extract the features of P (such as Fhog features) from it to obtain the feature f of size M×N×d, where the dimension of the feature is d-dimensional. Then construct the response g through the Gaussian function, which is M×N in size, with the largest response value in the middle and decreasing in order around. Perform a two-dimensional discrete Fourier transform (DFT) on the features of each dimension of f to obtain Fl, perform a two-dimensional DFT on g to obtain G, and obtain the filter template H.

[0110] The image feature of a new frame is z. The two-dimensional DFT of each dimension feature is obtained to obtain Zl. Each dimension of Z is passed through the filter template to obtain the final response matrix y. The position of the maximum value in y is the new center position of the target, and the target positioning is completed.

[0111] The scale filter can complete the scale estimation when tracking a single target. Assuming that the size of the image block P where the target is located is M×N, with the center of the image block as the center, pictures of different scales are intercepted according to different scale factors to obtain a series of image blocks of different scales. For each image block, its feature descriptor is calculated (the dimension is d, and the d here has nothing to do with the dimension d in the position estimation). g is the output response constructed by the Gaussian function, with a size of 1×S, the middle value is the largest, and it decreases towards both ends. If 33 scales are selected, then S=33, and the feature fl of each dimension is a 1×S vector. A one-dimensional DFT is performed on the feature of each dimension of f to obtain Fl, and a one-dimensional DFT is performed on g to obtain G. The calculation process is the same as that of the position estimation, except that the dimension becomes one-dimensional, and finally the filter template H is obtained.

[0112] The new frame of the image is centered on the position obtained by the position filter, and S image blocks of different scales are intercepted. Their feature descriptors are calculated respectively to form a new feature z. The one-dimensional DFT of each dimensional feature is calculated to obtain Zl. Through the regularization loss, the matrix y containing the responses of each scale is obtained. y is a 1×S-dimensional vector, and the scale corresponding to the maximum value in y is the result of the final scale estimation.

[0113] The position filter and scale filter are used to obtain the position and size of the target for tracking, and the y value of the position filter is used as the confidence of the single target tracking itself. The position size and confidence are also output for the single target tracking result. Preferably, the tracking confidence of the single target tracker in this embodiment is: the correlation score of the position of the target tracked by the single target.

[0114] Operation S5, uses the intersection-over-union matching method to cross-validate the target and cumulative confidence output in operation S3 and the target and tracking confidence output in operation S4 to determine whether to output the target tracked in operation S4, or to output the target with the largest cumulative confidence tracked in operation S3, or to output the fusion result of the targets matched in operation S4 and operation S3, or not to output it.

[0115] In this embodiment, the results of the multi-target tracker with the highest cumulative confidence (such as 3) are cross-validated with the prediction results of the single target tracking by alternating ratio matching, and the following is provided: Figure 7 and Figure 8 Two specific cross-validation methods are shown.

[0116] See also Figure 7 The cross-validation method shown in the figure has the following specific process.

[0117] When the tracking confidence of single target tracking is greater than the set value A1 (such as 0.4): if there is no output in operation S3 at this time, only the result of single target tracking in operation S4 is output, and the number of uncaptured frames remains unchanged; if there is output in operation S3 at this time, select the target detection frames with the top three highest cumulative confidences and perform alternating ratio matching with the single target tracking frame output in operation S4. If the alternating ratio is greater than the set value B1 (such as 0.2), the number of uncaptured frames remains unchanged, and the result of single target tracking in operation S4 is output; if the alternating ratio is not greater than the set value B1, the number of uncaptured frames + 1 is output, and the result of single target tracking in operation S4 is output. At this time, the tracking frame does not need to be updated with the detection result.

[0118] When the tracking confidence of single target tracking is not greater than the set value A1: if there is no output in operation S3 at this time, no output is performed, and the number of uncaptured frames is +1; if there is output in operation S3 at this time, the target detection frames with the top three highest cumulative confidences are selected to perform alternation ratio matching with the single target tracking frame output in operation S4. If the alternation ratio is greater than the set value B2 (such as 0.5), the number of uncaptured frames remains unchanged, and the result of single target tracking in operation S4 is output. The tracking confidence is output according to the weighted fusion of the confidence, and the tracking frame is updated with the detection result. If the alternation ratio is not greater than the set value B2, the number of uncaptured frames is +1, no output is performed, and the tracking frame is updated without the detection result.

[0119] See also Figure 8 The cross-validation method shown in the figure has the following specific process.

[0120] When the tracking confidence of single target tracking is greater than the set value A2 (such as 0.6): if there is no output in operation S3 at this time, only the result of single target tracking in operation S4 is output, and the number of uncaptured frames remains unchanged; if there is output in operation S3 at this time, select the target detection frames with the top three highest cumulative confidences and perform alternating ratio matching with the single target tracking frame output in operation S4. If the alternating ratio is greater than the set value B3 (such as 0.2), the number of uncaptured frames remains unchanged, and the result of single target tracking in operation S4 is output, and the tracking frame does not need to be updated with the detection result. If the alternating ratio is not greater than the set value B3, the number of uncaptured frames is +1, and the result of single target tracking in operation S4 is output, and the tracking frame does not need to be updated with the detection result.

[0121] When the tracking confidence of the single target tracking is between the set value A2 and the set value A3 (such as 0.1), where A3<A2: if there is no output in operation S3 at this time, only the result of the single target tracking in operation S4 is output, and the number of frames not captured is +1; if there is an output in operation S3 at this time, the target detection frames with the top three highest cumulative confidences are selected to perform alternation ratio matching with the single target tracking frame output in operation S4, if the alternation ratio is greater than the set value B4 (such as 0.5), the number of frames not captured remains unchanged, the result of the single target tracking in operation S4 and the weighted fusion confidence are output, and the tracking frame is updated with the detection result, if the alternation ratio is between the set value B4 and the set value B3, the number of frames not captured remains unchanged, the weighted fusion result of the confidence of operation S4 and operation S3 is output, and the tracking frame is updated with the detection result, if the alternation ratio is less than the set value B3, the number of frames not captured is +1, no output, and the tracking frame is not updated with the detection result.

[0122] When the tracking confidence of a single target tracking is less than the set value A3: if there is no output in operation S3 at this time, no output is performed; if there is an output in operation S3 at this time, the target detection frames with the top three highest cumulative confidences are selected for alternation ratio matching with the single target tracking frame output in operation S4. If the alternation ratio is greater than the set value B5 (such as 0.8), the tracking frame is updated with the detection result, the number of frames not captured is +1, and a weighted fusion result according to a given value (such as 0.5) is output. If the alternation ratio is not greater than the set value B5, the number of frames not captured is +1, no output is performed, and the tracking frame is not updated with the detection result.

[0123] According to an embodiment of the present invention, the method also includes: judging whether each frame image is captured according to the result of cross-validation in operation S5; when the number of frames in which the image is not captured in a consecutive number of frames reaches a third set value, exiting tracking, if there is a target output in operation S3, re-executing operation S4 and operation S5, and if there is no output target in operation S3, no output is performed.

[0124] The target capture method in the video sequence scene provided by the embodiment of the present invention first performs preliminary detection on the target in the image; then performs preliminary filtering according to the prior knowledge such as the imaging size and aspect ratio of the target in the image; then the result after the target detection is transmitted to the multi-target tracking module, and the time series filtering is implemented at this stage to eliminate the occasional false detection in the detection and realize the multi-frame accumulation of confidence, and the output is controlled by setting the cumulative number threshold and the confidence threshold; the output result is sent to the single target tracking module, and the target with the highest confidence is initially selected to initialize the single target tracking to realize the capture, and the single target tracking module completes the tracking prediction of the captured target each time, and completes the cross-validation with the target output by the multi-target tracking, judges whether the target is captured and tracked correctly, and realizes the capture change in time to ensure the correct capture and tracking of the target. The method can solve the following technical problems existing in the prior art: the detection and recognition ability of the target detection is weak in the case of interference; when the target is weak, more false alarms will be generated, resulting in inaccurate capture; the single target tracking ability is limited, and when the time series is too long and the target feature becomes weak, it will cause the tracking and capture to fail.

[0125] Embodiment 2

[0126] A target capture system in a video sequence scene includes: a processor; a memory storing a computer executable program, and when the program is executed by the processor, the processor executes the target capture method in the video sequence scene. The related technical solution is the same as the first embodiment, and will not be repeated here.

[0127] Embodiment 3

[0128] A computer-readable storage medium stores a computer program, and when the program is executed by a processor, the target capture method in the above video sequence scene is implemented. The related technical solution is the same as that in the first embodiment, and will not be described in detail here.

[0129] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for capturing a target in a video sequence scene, characterized in that: include: S1, performs target detection on the input video sequence images to obtain the position and size of all targets in each frame image; S2, designing an imaging size range of the desired target according to its theoretical size in the image, deleting the targets whose sizes in S1 are not within the imaging size range, and outputting the targets that are not deleted; S3, initializing a multi-target tracker using the positions of all targets output in S2 to perform multi-target tracking and cumulative confidence calculation on the video sequence images; During multi-target tracking, for each target tracker, when the number of tracking hits reaches a first set value and the cumulative confidence reaches a set confidence threshold, the tracked target and the cumulative confidence are output; when the number of consecutive tracking misses reaches a second set value, the target tracker is deleted; S4, initializing a single target tracker using the target with the highest cumulative confidence, so as to perform single target tracking and tracking confidence calculation and output for the video sequence images; S5, cross-validates the target and cumulative confidence output in S3 and the target and tracking confidence output in S4 using the intersection-over-union matching method to determine whether to output the target tracked in S4, or to output the target with the largest cumulative confidence tracked in S3, or to output the fusion result of the targets matched in S4 and S3, or not to output it.

2. The target capture method in a video sequence scene according to claim 1, characterized in that: The method further comprises: Determine whether each frame image is captured according to the result of the cross-validation in S5; When the number of frames in which the image is not captured in a continuous number of frames reaches a third set value, the tracking is exited. If a target is output in S3, S4 and S5 are re-executed. If no target is output in S3, no output is performed.

3. The target capture method in a video sequence scene according to claim 1, characterized in that: The S1 specifically includes: Preprocess the input video sequence images; Perform object detection on the preprocessed image to obtain a horizontal detection frame; The minimum circumscribed rectangle of the target is found in the horizontal detection frame by using the frame shrinkage to obtain a rotation detection frame, and the position and size of the rotation detection frame are the position and size of the target in the rotation detection frame.

4. The target capture method in a video sequence scene as claimed in claim 3, characterized in that: The step of searching for the minimum bounding rectangle of the target in the horizontal detection frame by shrinking the frame specifically includes: The image in the horizontal detection frame is sequentially subjected to saliency detection, image segmentation, filtering and edge detection to achieve frame shrinkage, and the minimum circumscribed rectangular frame of the frame after frame shrinkage is obtained as the rotation detection frame.

5. The target capture method in a video sequence scene according to claim 1, characterized in that: The imaging size range is: Th wmax =target w ×rate w Th hmax =target h ×rate w Th wmin =target w ÷rate h Th hmin =target h ÷rate h Among them, Th wmax ,Th hmax ,Th wmin ,Th hmin are the upper limit of imaging length, upper limit of imaging width, lower limit of imaging length, and lower limit of imaging width of the desired target, respectively. w 、target h are the theoretical length and width of the target in the image, respectively. The theoretical size includes target w and target h , rate w is the length scaling factor, rate h is the width scaling factor, base is the first adjustable parameter, α is the second adjustable parameter, β is the third adjustable parameter, and the imaging size range includes Th wmax ,Th hmax ,Th wmin and Th hmin .

6. The target capture method in a video sequence scene according to any one of claims 1 to 5, characterized in that: The performing multi-target tracking on the video sequence images specifically includes: For each target tracker, the target tracker is used to track the target of the video sequence image. If the tracked target matches the corresponding target obtained by target detection, the tracking is successful. The position of the corresponding tracked target is corrected by using the target position obtained by target detection. If the tracked target does not match any target obtained by target detection, the tracking is unsuccessful. For each target output in S2, if the target does not match any target obtained by the multi-target tracker, a new target tracker is initialized for multi-target tracking.

7. The target capture method in a video sequence scene according to claim 6, characterized in that: When the tracking hit count is greater than the fourth set value, the cumulative confidence is: cumConf←cumConf×(1-cum_factor)+conf×cum_factor When the tracking hit count is not greater than the fourth set value, the cumulative confidence is: cumConf←1-(1-cumConf)×(1-conf×cum_factor) Among them, cumConf is the cumulative confidence, the left and right sides are the current cumulative confidence and the previous cumulative confidence respectively, cum_factor is the adjustable weight, and conf is the target detection confidence of the current frame.

8. The target capture method in a video sequence scene according to any one of claims 1 to 5, characterized in that: The tracking confidence is: a correlation score of the position of the target tracked by a single target.

9. A target capture system in a video sequence scene, characterized in that: include: processor; A memory storing a computer executable program, wherein when the program is executed by the processor, the processor executes the target capturing method in the video sequence scene according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a target capturing method in a video sequence scenario as described in any one of claims 1 to 8 is implemented.

Images