Method, device and storage medium for detecting stranded targets

Abstract

The present application provides a method for detecting a retained target and related equipment, which can effectively judge the retention of targets that disappear for a short time and targets that disappear for a long time. The method includes: determining the pedestrian detection result corresponding to the current image frame; determining the appearance features and motion features corresponding to each target in the current image frame based on the pedestrian detection result; determining the first final cost matrix corresponding to the current image frame based on the appearance features and motion features; determining the matching result corresponding to the current image frame based on the first final cost matrix and the second final cost matrix corresponding to N image frames, where N image frames are image frames before the current image frame and N is an integer greater than or equal to 2; updating the status information of the tracking target corresponding to the current image frame based on the matching result of the current image frame; and determining whether there is a retained target in the current image frame based on the status information of the tracking target corresponding to the current image frame.

Description

Technical field

[0001] The present application relates to the field of image processing, and in particular to a method, device and storage medium for detecting a retained target. [Background Technology]

[0002] In recent years, video surveillance systems have been applied to many industries. For example, video surveillance equipment has been installed in public places such as train stations, supermarkets, banks, residential communities, and schools, using cameras to monitor the scene. However, having human personnel monitor the scene through cameras is not only labor-intensive, financially demanding, and resource-intensive, but also prone to false positives and missed detections. Consequently, intelligent video surveillance systems that use AI to replace human monitoring have emerged and are gaining increasing attention. This is particularly true in densely populated public places such as train stations, airports, and shopping malls, where significant safety risks exist and real-time monitoring is essential. This demand has led to the development of intelligent detection systems for delayed personnel. In densely populated areas, if individuals such as the elderly, children, or people with disabilities are stranded in less crowded areas for various reasons and are not detected in time, they can pose a serious threat to their lives. Traditional manual detection suffers from these issues, making intelligent delayed personnel detection systems of great practical research and application value.

[0003] In order to develop intelligent people detention detection systems, multi-target tracking methods are widely deployed on these platforms. Current multi-target tracking methods are mainly divided into detection-based methods and query-based methods.

[0004] The detection-based method mainly performs target detection on the input video frame to obtain the pedestrian detection frame, obtains the appearance features of each detection frame through the feature extraction network, and then predicts the position of the previously tracked target at the current moment. The tracked target is matched with the detection frame through the association algorithm, and the attributes of the successfully matched tracked target are updated.

[0005] Currently available methods can only handle the impact of short-term occlusion (<5s) on tracking. However, when applied to practical projects such as pedestrian detention detection, it is necessary to consider the impact of longer disappearances on tracking, which makes it difficult to meet the needs in practical applications. [Summary of the invention]

[0006] The present application provides a method, device and storage medium for detecting a retained target, which can effectively judge the retention of a target that disappears for a short time or a target that disappears for a long time.

[0007] The first aspect of the present application provides a method for detecting a retained target, comprising:

[0008] Determine a pedestrian detection result corresponding to a current image frame, where the current image frame is any image frame in the target video;

[0009] Determining the appearance features and motion features corresponding to each target in the current image frame according to the pedestrian detection result;

[0010] Determining a first final cost matrix corresponding to the current image frame according to the appearance features and the motion features;

[0011] Determining a matching result corresponding to the current image frame according to the first final cost matrix and second final cost matrices corresponding to N image frames, where the N image frames are image frames before the current image frame, and N is an integer greater than or equal to 2;

[0012] Updating the state information of the tracking target corresponding to the current image frame according to the matching result of the current image frame;

[0013] It is determined whether there is a retained target in the current image frame according to the state information of the tracking target corresponding to the current image frame.

[0014] In a second aspect of the present application, a device for detecting a retained target includes:

[0015] a determination unit, configured to determine a pedestrian detection result corresponding to a current image frame, wherein the current image frame is any image frame in a target video;

[0016] a feature extraction unit, configured to determine appearance features and motion features corresponding to each target in the current image frame according to the pedestrian detection result;

[0017] a cost matrix determining unit, configured to determine a first final cost matrix corresponding to the current image frame according to the appearance feature and the motion feature;

[0018] a matching unit, configured to determine a matching result corresponding to the current image frame based on the first final cost matrix and second final cost matrices corresponding to N image frames, where the N image frames are image frames before the current image frame, and N is an integer greater than or equal to 2;

[0019] an updating unit, configured to update the state information of the tracking target corresponding to the current image frame according to the matching result of the current image frame;

[0020] The judgment unit is configured to determine whether there is a retained target in the current image frame according to the state information of the tracking target corresponding to the current image frame.

[0021] A third aspect of an embodiment of the present application provides a computer device comprising at least one connected processor, memory, and transceiver, wherein the memory is used to store program code, and the processor is used to call the program code in the memory to execute the steps of the method for detecting a retained target described in the first aspect above.

[0022] A fourth aspect of an embodiment of the present application provides a computer storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the method for detecting a retained target as described in any one of the above aspects.

[0023] Compared to related technologies, the embodiments provided in this application track multiple targets in a given video, detect pedestrians using a detector, and then calculate a cost matrix by calculating the Mahalanobis distance of motion features and the cosine distance of appearance features. Cascade matching and Hungarian matching are used based on the cost matrix to obtain matching results, thereby determining whether the tracked target remains. This allows for effective judgment of the retention of targets that disappear for a short time or for a long time.

Brief Description of the Drawings

[0024] Figure 1 A schematic flow chart of a method for detecting a retained target provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a state transition matrix provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a virtual structure of a detection device for a retained target provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the hardware structure of the server provided in an embodiment of the present application. [Specific implementation method]

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0029] The following describes a method for detecting a stranded target from the perspective of a stranded target detection device. The stranded target detection device may be a server or a service unit in the server, and is not specifically limited thereto.

[0030] Please refer to Figure 1 , Figure 1 A schematic flow chart of a method for detecting a retained target provided in an embodiment of the present application includes:

[0031] 101. Determine a pedestrian detection result corresponding to the current image frame.

[0032] In this embodiment, the detection device for the retained target may first obtain an untrimmed target video V = {v1, v2, ..., v n}, where v i (i=1,2,...,n) is the i-th image frame in the target video. After that, the pedestrian detection result corresponding to the current image frame can be obtained. For a given untrimmed target video, the pedestrian detection box result corresponding to the current image frame is obtained using the yolov5 detector. The pedestrian detection result of one frame is defined as D={d1,d2,d3,......,d i}d i is one of the test results, and d i =(x1,y1,x2,y2), (x,y) is the coordinate of the upper left corner and the lower right corner of the detection box; in order to obtain the motion information conveniently, it is expressed as d i =(x,y,a,h), where (x,y) represents the center coordinates of the detection box, a represents the aspect ratio of the coordinate box, and h represents the height of the coordinate box.

[0033] 102. Determine the appearance features and motion features corresponding to each target in the current image frame according to the pedestrian detection result.

[0034] In this embodiment, after determining the pedestrian detection result corresponding to the current frame, the detection device for the retained target can determine the appearance features and motion features corresponding to each target in the current image frame based on the pedestrian detection result. The following is a detailed description:

[0035] Extracting the appearance features corresponding to each target in the current image frame based on a person re-identification network. The person re-identification network is pre-trained on a training dataset and is used to identify the appearance features of an image.

[0036] If there is a newly added first tracking target in the current image frame, initializing the first Kalman filter parameters of the first tracking target;

[0037] predicting a first motion feature of a first tracking target in a current image frame according to a first Kalman filter parameter;

[0038] If the first tracking target does not exist in the current image frame, obtaining a second Kalman filter parameter corresponding to the first detection target in the first image frame, the first detection target is included in the current image frame, and the first image frame is an image frame previous to the current image frame;

[0039] predicting a second motion feature of the first detection target in the current image frame according to the second Kalman filter parameter;

[0040] Determining a Kalman gain based on a third motion feature and a second motion feature of the first detection target detected in the current image frame;

[0041] The target motion feature of the first detected target in the current image frame is updated according to the Kalman gain.

[0042] That is, the detection device of the retained target can pre-train a pedestrian re-identification network FastReID to extract appearance features. Through the pedestrian re-identification network FastReID, the appearance features corresponding to the current image frame can be determined, and the appearance features are 128-dimensional feature vectors.

[0043] The current image frame may include some newly added tracking targets or may not include them. For example, the previous image frame of the current image frame includes 5 detection targets, each detection target has a corresponding detection frame. In addition to the 5 detection targets in the previous frame, the current image frame also includes a new target or one of the targets disappears and a new target appears. The newly added target is defined as the first tracking target. When the detection device for tracking target retention determines that the first tracking target exists in the current image frame, it can initialize the Kalman filter parameters of the first tracking target. The Kalman filter parameters include the position velocity mean m and the covariance matrix p. According to the detection frame information of the newly generated tracking target (the current image frame does not have a matching detection frame), a position velocity vector m = (x, y, a, h, 0, 0, 0, 0) is generated. The first 4 values in the position velocity vector m are the values of the detection frame, and the last 4 0s represent the change value of the first 4 values, which are initialized to 0. The covariance matrix p of the first tracking target is calculated by the following formula:

[0044] p=diag(std 2 );

[0045] std=(2*0.05*h,2*0.05*h,0.01,2*0.05*h,

[0046] 10*0.00625*h, 10*0.00625*h, 0.00001, 10*0.00652*h)

[0047] Among them, diag means diagonalizing the matrix, and h is the height of the detection box.

[0048] If the detection device for tracking target retention determines that the first tracking target does not exist in the current image frame, the second motion feature of the first detection target in the current image frame can be predicted using the following two formulas based on the second Kalman filter parameters (position velocity mean m and covariance matrix p) corresponding to the first detection target obtained after updating the image frame previous to the current image frame:

[0049] M'=F*M;

[0050] M' is the predicted position and speed average of the first detection target in the current image frame, M is the position and speed average of the first detection target in the first image frame, and F is an 8x8 state transition matrix, assuming that the first detection target is moving in a uniform linear motion. Figure 2 As shown, dt represents the time interval, the default value is 1;

[0051] P'=F*P*F Τ +Q;

[0052] Where P' is the predicted covariance matrix of the first detected target in the current image frame, P is the covariance matrix of the first detected target in the first image frame, F is the state transition matrix, and Q is the Kalman filter motion estimation error, where Q is calculated as follows:

[0053]

[0054] Then, according to the third motion feature and the second motion feature of the first detection target in the current image frame, the Kalman gain is determined by the following formula:

[0055] K=P′H T S -1 ;

[0056] Where K is the Kalman gain, P′ is the covariance matrix in the second motion feature, H is the state transfer matrix, S is the deviation of the covariance matrix, S = HP′H T +R, R is random noise, R=diag((0.05*h,0.05*h,0.1,005*h) 2 ), h is the height of the detection box corresponding to the first detection target, and diag represents the diagonalization of the matrix;

[0057] Finally, the target motion feature of the first detected target in the current image frame is updated according to the Kalman gain using the following formula:

[0058]

[0059] Wherein, m is the mean position velocity in the target motion feature, P is the covariance matrix in the target motion feature, m′ is the mean position velocity in the second motion feature, y is the deviation of the mean position velocity, y=z-Hm′, z is the detection frame information of the actual position of the first detection target in the current image frame, K is the Kalman gain, and P′ is the covariance matrix in the second motion feature.

[0060] It should be noted that H is a 4x8 state transition matrix, and the matrix is a matrix with equal row and column numbers, the values of positions are 1, and the other positions are 0.

[0061] 103. Determine a first final cost matrix corresponding to the current image frame according to the appearance features and the motion features.

[0062] In this embodiment, the detection device for the retained target can determine the cosine distance between the second tracked target and the second detected target in the current image frame based on the appearance features, and construct an initial cost matrix based on the cosine distance. Then, the Mahalanobis distance between the second tracked target and the second detected target is determined based on the motion features. The initial cost matrix is constrained based on the Mahalanobis distance to obtain a first final cost matrix. The following is a detailed description:

[0063] Assume that the current image frame includes a tracking target set and a detection target set, and the tracking target set is T = {t1, t2, t3, ..., t i}, where t i is the i-th tracking target, each tracking target includes an appearance feature set F i ={f i1 ,f i2 ,f i3 ,......,f ij} and a current motion feature S i ={x,y,a,h,v x ,v y ,v a ,v h}, where f ij represents the appearance features of the detection results of the jth matching of the i-th tracking target, S i It represents the motion characteristics of the i-th tracked target. The first four values are the horizontal and vertical coordinates of the center point, the aspect ratio and height of the tracking box, and the last four values represent the change vector of the first four values. The detection target set is D = {d1, d2, d3, ..., d i}, each detection target has appearance features and motion characteristics S d ={x,y,a,h,0,0,0,0}, the cosine distance between the second tracking target and the second detection target is determined by the following formula:

[0064]

[0065] Among them, cos ij is the cosine distance between the second tracking target i and the second detection target j in the current image frame, n is the number of appearance features of the second tracking target i, f ik is the kth appearance feature of the second tracking target, is the appearance feature of the second detection target j, the current image frame includes a tracking target set and a detection target set, and the tracking target set is T = {t1, t2, t3, ..., t i}, the detection target set is D={d1,d2,d3,......,d i}, each detection target in the detection target set includes appearance features and motion characteristics S d ={x,y,a,h,0,0,0,0}, where x and y are the center coordinates of the detection box of each detection target, a is the aspect ratio of the detection box, and h is the height of the detection box. The initial cost matrix is then constructed based on the cosine distance.

[0066] At the same time, the detection device of the retained target can determine the Mahalanobis distance between the second tracking target and the second detection target according to the motion characteristics, as follows:

[0067] Obtain the first position velocity mean and first covariance matrix of the second tracking target, the second position velocity mean and second covariance matrix of the second detection target, the first position velocity mean and first covariance matrix are the updated Kalman filter parameters of the second tracking target in the current image frame, the second position velocity mean and second covariance matrix are the updated Kalman filter parameters of the second detection target in the current image frame; and decompose the covariance matrix to obtain a lower triangular matrix; determine the target distance based on the second position velocity mean and the first position velocity mean; determine the eigenvalue vector based on the target distance; calculate the Mahalanobis distance between the second tracking target and the second detection target based on the eigenvalue vector. That is, use the Kalman filter update formula to obtain the motion feature vector m and the covariance matrix P (note that m and P here are only position features, that is, m only has xyah, and P is a 4x4 matrix); perform Cholesky decomposition on the covariance matrix P to obtain a lower triangular matrix L; use the motion features (only position features) of the detection result minus the motion features (only position features) of the tracking target to obtain the distance d ij Solve Lx = d ij Τ The eigenvalue vector x ij ; Mahalanobis distance

[0068] Finally, the cost matrix is constrained according to the obtained Mahalanobis distance. Specifically, if the Mahalanobis distance dm between the i-th tracking target and the j-th detection result is ijIf the cosine distance between the two targets is greater than a preset threshold of 9.4877 (the critical value of a chi-square distribution with 4 degrees of freedom and a probability of 0.05), the two targets are considered unlikely to be the same target, and the corresponding value in the cost matrix is set to infinity (10e5). Unlike deepsort, different thresholds are set based on the target's state (how long it has disappeared). The longer the disappearance time, the less accurate the prediction results of the target's motion features, or even unusable. At the same time, the appearance features themselves have constraints. That is, if the cosine distance of the appearance features is greater than 0.2, the two targets are considered unlikely to be the same target, and the cosine distance is also set to infinity (10e5), thus obtaining the final cost matrix COV.

[0069] 104. Determine a matching result corresponding to the current image frame according to the first final cost matrix and second final cost matrices corresponding to the N image frames.

[0070] In this embodiment, after determining the first final cost matrix, the detection device for the retained target can determine the matching result corresponding to the current image frame based on the first final cost matrix and the second final cost matrix corresponding to N image frames, wherein the N images are image frames before the current image frame, and N is an integer greater than or equal to 2. That is, for the tracking target t i There is an attribute time_since_update (tsu), which indicates the number of frames since the last successful match. First, the original cascade matching algorithm sets a priority for the tracking target based on the tsu attribute, that is, it is considered that the target with a shorter interval from the last match is more likely to be matched. Therefore, according to tsu from small to large, all tracking targets with tsu equal to this value are selected for matching each time, and the unmatched detection targets are used for the next round of matching until the tracking targets are matched or the detection targets are matched. However, when two people cross each other (that is, one person briefly blocks the other person), if the ID of the blocked person changes before and after being blocked (because the appearance features will fluctuate greatly when occlusion occurs, which may cause a match failure), two tracking targets will be generated for the same person, and the newly generated tracking target will match this person first, resulting in tracking interruption. This application modifies the conditions for selecting tracking targets for matching each time. Instead of selecting a tracking target with a tsu equal to a certain value, the target selected each time has a tsu within a range. For example, the first time a tracking target with a tsu of [0, 30) is selected, that is, the tracking target in the 30 image frames before the current image frame is matched with the detection target in the current image frame. The next time, the tracking target is [30, 60), and so on. By expanding the search range, the algorithm can simultaneously match two tracking targets for the same person, greatly increasing the possibility that the previously generated tracking target will continue to track the pedestrian when the pedestrian is not obscured.

[0071] Then, the classic Hungarian matching method is used, taking the cost matrix of the tracked target selected by the time segment cascade matching and the unmatched detection target as input and outputting the matching result. This can obtain the matching results of each target in the current frame.

[0072] In one embodiment, the detection device for the retained target further performs the following operations:

[0073] If there are unmatched detection target sets and tracking target sets in the matching result corresponding to the current image frame, then calculating the target cosine distance between the target appearance feature of the third tracking target in the tracking target set and the appearance feature of the third detection target, where the third tracking target is a target that has not appeared in the tracking target set for a preset time period;

[0074] If the target cosine distance meets the preset conditions, then calculating the third final cost matrix of the third tracking target;

[0075] Matching unmatched objects in the current image frame is performed according to the third final cost matrix.

[0076] In this embodiment, almost all current target tracking algorithms have not considered the situation where the target disappears for a long time, because in this situation it is almost impossible to use motion features to predict the position of the target when it appears. Through research, it is found that when the target disappears for a long time, the appearance features will not change much when it reappears, so this problem can be simplified to a pedestrian re-identification problem. Pedestrian re-identification is to extract the appearance features of the image by training an appearance feature extraction network, and by giving a pedestrian query image, find other images of the target from the database. Therefore, the retrieval algorithm handles the target that has disappeared for a long time by executing it on the specified tracking target (which has disappeared for a long time and has not been matched).

[0077] Specifically: If there are unmatched detection targets and tracking targets after the matching algorithm is completed, the target marked as having disappeared for a long time among the tracking targets is selected; and the cosine distance of the appearance features of each tracking target and the detection result is calculated using the following formula:

[0078] cos ijk represents the cosine distance between the kth appearance feature of the i-th tracked target and the j-th detection target. If cos ijk Satisfy the preset condition (the preset condition can be, for example, cos ijk The total number of values exceeding the specified percentage (30%) is less than the threshold (0.25). Of course, other conditions are also possible, such as cos ijkIf the total number of values exceeding a specified percentage (35%) is less than a threshold (0.26, with no specific limit), the two targets are considered to be the same pedestrian. After calculating the cost matrix using the above method for calculating the final cost matrix (in addition, when calculating the corresponding cost matrix using the above method for calculating the final cost matrix, the Mahalanobis distance constraint of the motion feature and the appearance threshold constraint are not used), the Hungarian matching algorithm is used to match the above cost matrix to obtain a matching result. This can greatly improve the tracking success rate of targets that have disappeared for a long time, and because appearance features must exceed a certain proportion to participate in the matching, matching errors are effectively avoided.

[0079] 105. Update the state information of the tracking target corresponding to the current image frame according to the matching result of the current frame.

[0080] In this embodiment, after obtaining the matching result of the current frame, the detection device for the retained target can update the state information of the tracking target corresponding to the current image frame according to the matching result of the current frame. The state update of the tracking target mainly includes the update of attributes and the update of state values. The attribute values include age (the total number of frames generated by the tracking target), time_since_update (the number of frames between the last successful tracking), and Kalman filter attributes (position velocity vector m and covariance matrix P); the state value includes three states, continue, short_miss, and long_miss, which represent the three states of continuous tracking success, short-term disappearance, and long-term disappearance. The state value is set according to the size of the attribute time_since_update. The reason for setting the state value is that if a target disappears for a long time, the accuracy of the tracking target prediction result will decrease due to the increase in the disappearance time, so different thresholds should be set according to different states.

[0081] 106. Determine whether there is a retained target in the current image frame according to the state information of the tracking target corresponding to the current image frame.

[0082] In this embodiment, after updating the status information of the tracking target corresponding to the current image frame, the detection device of the retained target can determine whether the attribute age of each tracking target in the current image frame is greater than the retention threshold. If age is greater than the threshold, it is considered that the tracking target is retained in the scene.

[0083] In summary, the embodiments provided herein track multiple targets in a given video, detect pedestrians using a detector, and then calculate a cost matrix derived from the Mahalanobis distance of motion features and the cosine distance of appearance features. Cascade matching and Hungarian matching are then used based on the cost matrix to obtain matching results, thereby determining whether the tracked target remains. This effectively determines the retention of targets that disappear for both short periods and long periods.

[0084] The above describes the present application from the perspective of a method for detecting a retained target, and the following describes the present application from the perspective of a device for detecting a retained target.

[0085] See also Figure 3 , Figure 3 This is a virtual structural diagram of a detection device for a retained target provided in an embodiment of the present application. The detection device 300 for a retained target includes:

[0086] A determination unit 301 is configured to determine a pedestrian detection result corresponding to a current image frame, where the current image frame is any image frame in a target video;

[0087] A feature extraction unit 302 is configured to determine appearance features and motion features corresponding to each target in the current image frame based on the pedestrian detection result;

[0088] A cost matrix determining unit 303 is configured to determine a first final cost matrix corresponding to the current image frame according to the appearance features and the motion features;

[0089] a matching unit 304 configured to determine a matching result corresponding to the current image frame based on the first final cost matrix and second final cost matrices corresponding to N image frames, where the N image frames are image frames before the current image frame, and N is an integer greater than or equal to 2;

[0090] An updating unit 305 is configured to update the state information of the tracking target corresponding to the current image frame according to the matching result of the current image frame;

[0091] The judgment unit 306 is configured to determine whether there is a retained target in the current image frame according to the state information of the tracking target corresponding to the current image frame.

[0092] In one possible design, the feature extraction unit 302 is specifically configured to:

[0093] extracting appearance features corresponding to each target in the current image frame according to a person re-identification network, wherein the person re-identification network is pre-trained on a training data set and is used to identify appearance features of an image;

[0094] If there is a newly added first tracking target in the current image frame, initializing a first Kalman filter parameter of the first tracking target;

[0095] determining a first motion feature of the first tracking target in the current image frame according to the first Kalman filter parameter;

[0096] If the first tracking target does not exist in the current image frame, obtaining a second Kalman filter parameter corresponding to a first detection target in a first image frame, where the first detection target is any target in the current image frame, and the first image frame is an image frame previous to the current image frame;

[0097] predicting a second motion feature of the first detection target in the current image frame according to the second Kalman filter parameter;

[0098] determining a Kalman gain corresponding to the first detection target based on a third motion feature and the second motion feature of the first detection target detected in the current image frame;

[0099] The target motion feature of the first detection target in the current image frame is updated according to the Kalman gain corresponding to the first detection target.

[0100] In one possible design, the motion features include a position velocity mean and a covariance matrix, and the feature extraction unit 302 determines the Kalman gain corresponding to the first detection target based on the third motion feature and the second motion feature of the first detection target detected in the current image frame, including:

[0101] The Kalman gain corresponding to the first detection target is determined by the following formula:

[0102] K=P′H T S -1 ;

[0103] Wherein, K is the Kalman gain corresponding to the first detection target, P′ is the covariance matrix in the second motion feature, H is the state transfer matrix, S is the deviation of the covariance matrix, S=HP′H T +R, R is random noise, R=diag((0.05*h,0.05*h,0.1,005*h) 2 ), h is the height of the detection box corresponding to the first detection target, and diag represents diagonalizing the matrix;

[0104] The updating of the target motion feature of the first detection target in the current image frame according to the Kalman gain corresponding to the first detection target includes:

[0105] The target motion feature of the first detection target in the current image frame is determined by the following formula:

[0106]

[0107] Wherein, m is the mean position velocity in the target motion feature, P is the covariance matrix in the target motion feature, m′ is the mean position velocity in the second motion feature, y is the deviation of the mean position velocity, y=z-Hm′, z is the detection frame information of the actual position of the first detection target in the current image frame, K is the Kalman gain, and P′ is the covariance matrix in the second motion feature.

[0108] In one possible design, the cost matrix determining unit 303 is specifically configured to:

[0109] determining a cosine distance between a second tracking target and a second detection target in the current image frame by using the appearance feature;

[0110] Constructing an initial cost matrix based on the cosine distance;

[0111] determining a Mahalanobis distance between the second tracking target and the second detection target according to the motion feature;

[0112] The initial cost matrix is constrained according to the Mahalanobis distance to obtain the first final cost matrix.

[0113] In a possible design, the cost matrix determining unit 303 determines the cosine distance between the second tracking target and the second detection target in the current image frame by using the appearance feature, including:

[0114] The cosine distance is determined by the following formula:

[0115]

[0116] Among them, cos ij is the cosine distance between the second tracking target i and the second detection target j in the current image frame, n is the number of appearance features of the second tracking target i, f ik is the kth appearance feature of the second tracking target, is the appearance feature of the second detection target j, the current image frame includes a tracking target set and a detection target set, and the tracking target set is T={t1, t2, t3, ..., t i}, the detection target set is D = {d1, d2, d3, ..., d i}, each detection target in the detection target set includes appearance features and motion characteristics S d ={x,y,a,h,0,0,0,0}, where x and y are the center coordinates of the detection frame of each detection target, a is the aspect ratio of the detection frame, and h is the height of the detection frame.

[0117] In a possible design, the cost matrix determining unit 303 determines the Mahalanobis distance between the second tracking target and the second detection target according to the motion feature, including:

[0118] Acquire a first position velocity mean and a first covariance matrix of the second tracking target, and a second position velocity mean and a second covariance matrix of the second detection target;

[0119] Decomposing the covariance matrix to obtain a lower triangular matrix;

[0120] determining a target distance based on the second position speed average and the first position speed average;

[0121] determining a eigenvalue vector according to the target distance;

[0122] A Mahalanobis distance between the second tracking target and the second detection target is calculated based on the eigenvalue vector.

[0123] In one possible design, the matching unit 304 is further configured to:

[0124] If there is an unmatched detection target set and tracking target set in the matching result corresponding to the current image frame, calculating a target cosine distance between a target appearance feature in a third tracking target in the tracking target set and an appearance feature in the third detection target, where the third tracking target is a target that has not appeared in the tracking target set for a preset time period;

[0125] If the target cosine distance meets the preset condition, calculating the third final cost matrix of the third tracking target;

[0126] Matching unmatched objects in the current image frame is performed according to the third final cost matrix.

[0127] In one possible design, the status information includes:

[0128] Attribute value information and status value information;

[0129] The attribute values include the total number of frames generated by the tracking target, the number of frames between the last successful tracking of the tracking target, the position and velocity vectors, and the covariance matrix;

[0130] The status value information includes: tracking success status, short-term disappearance status, and long-term disappearance status.

[0131] Figure 4 This is a schematic diagram of the structure of the application server, such as Figure 4As shown, the server 400 of this embodiment includes at least one processor 401, at least one network interface 404 or other user interface 403, a memory 405, and at least one communication bus 402. The server 400 optionally includes a user interface 403, including a display, a keyboard or a clicking device. The memory 405 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 405 stores execution instructions. When the server 400 is running, the processor 401 communicates with the memory 405, and the processor 401 calls the instructions stored in the memory 405 to execute the above-mentioned detection method of the retained target. The operating system 406 includes various programs for implementing various basic services and processing tasks based on hardware.

[0132] The server provided in the embodiment of the present application can execute the technical solution of the embodiment of the above-mentioned detection method of the retained target. Its implementation principle and technical effects are similar and will not be repeated here.

[0133] The present application also provides a computer-readable storage medium having a computer program stored thereon. When executed by a computer, the computer program implements the method flow associated with the apparatus for detecting a retained target in any of the above-mentioned method embodiments. Correspondingly, the computer may be the apparatus for detecting a retained target.

[0134] Embodiments of the present application also provide a computer program or a computer program product including a computer program. When executed on a computer, the computer program causes the computer to implement the method flow associated with the apparatus for detecting a retained target in any of the aforementioned method embodiments. Accordingly, the computer may be the aforementioned apparatus for detecting a retained target.

[0135] In the above Figure 1 In the corresponding embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0136] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting a stranded target, characterized in that: include: Determine a pedestrian detection result corresponding to a current image frame, where the current image frame is any image frame in the target video; Determining the appearance features and motion features corresponding to each target in the current image frame according to the pedestrian detection result; Determining a first final cost matrix corresponding to the current image frame according to the appearance features and the motion features; Determining a matching result corresponding to the current image frame according to the first final cost matrix and second final cost matrices corresponding to N image frames, where the N image frames are image frames before the current image frame, and N is an integer greater than or equal to 2; Updating the state information of the tracking target corresponding to the current image frame according to the matching result of the current image frame; Determine whether there is a retained target in the current image frame according to the state information of the tracking target corresponding to the current image frame, Determining the appearance features and motion features corresponding to each target in the current image frame according to the pedestrian detection result includes: extracting appearance features corresponding to each target in the current image frame according to a person re-identification network, wherein the person re-identification network is pre-trained on a training data set and is used to identify appearance features of an image; If there is a newly added first tracking target in the current image frame, initializing a first Kalman filter parameter of the first tracking target; determining a first motion feature of the first tracking target in the current image frame according to the first Kalman filter parameter; If the first tracking target does not exist in the current image frame, obtaining a second Kalman filter parameter corresponding to a first detection target in a first image frame, where the first detection target is any target in the current image frame, and the first image frame is an image frame previous to the current image frame; predicting a second motion feature of the first detection target in the current image frame according to the second Kalman filter parameter; determining a Kalman gain corresponding to the first detection target based on a third motion feature and the second motion feature of the first detection target detected in the current image frame; The target motion feature of the first detection target in the current image frame is updated according to the Kalman gain corresponding to the first detection target.

2. The method according to claim 1, characterized in that The motion feature includes a position velocity mean and a covariance matrix, and the determining of the Kalman gain corresponding to the first detection target based on the third motion feature and the second motion feature of the first detection target detected in the current image frame includes: The Kalman gain corresponding to the first detection target is determined by the following formula: ; in, is the Kalman gain corresponding to the first detection target, is the covariance matrix of the second motion feature, is the state transition matrix, is the deviation of the covariance matrix, , is random noise, , is the height of the detection frame corresponding to the first detection target, Indicates diagonalizing the matrix; The updating of the target motion feature of the first detection target in the current image frame according to the Kalman gain corresponding to the first detection target includes: The target motion feature of the first detection target in the current image frame is determined by the following formula: ； in, is the mean value of the position and velocity in the target motion feature, is the covariance matrix of the target motion features, is the mean value of the position velocity in the second motion feature, is the deviation of the mean position and velocity, , is the detection frame information of the actual position of the first detection target in the current image frame, is the Kalman gain, is the covariance matrix of the second motion feature.

3. The method according to claim 1, characterized in that The determining of the first final cost matrix corresponding to the current image frame according to the appearance feature and the motion feature includes: determining a cosine distance between a second tracking target and a second detection target in the current image frame by using the appearance feature; Constructing an initial cost matrix based on the cosine distance; determining a Mahalanobis distance between the second tracking target and the second detection target according to the motion feature; The initial cost matrix is constrained according to the Mahalanobis distance to obtain the first final cost matrix.

4. The method according to claim 3, characterized in that Determining the cosine distance between the second tracking target and the second detection target in the current image frame by using the appearance feature includes: The cosine distance is determined by the following formula: ； in, is the cosine distance between the second tracking target i and the second detection target j in the current image frame, is the number of appearance features of the second tracking target i, is the kth appearance feature of the second tracking target, is the appearance feature of the second detection target j, the current image frame includes a tracking target set and a detection target set, and the tracking target set is , the detection target set is Each detection target in the detection target set includes appearance features and motion characteristics , is the center coordinate of the detection frame of each detection target, is the value of the aspect ratio of the detection frame, and h is the height of the detection frame.

5. The method according to claim 3, characterized in that Determining the Mahalanobis distance between the second tracking target and the second detection target according to the motion feature includes: Acquire a first position velocity mean and a first covariance matrix of the second tracking target, and a second position velocity mean and a second covariance matrix of the second detection target; Decomposing the covariance matrix of the second tracking target to obtain a lower triangular matrix; determining a target distance based on the second position speed average and the first position speed average; determining a eigenvalue vector according to the target distance; A Mahalanobis distance between the second tracking target and the second detection target is calculated based on the eigenvalue vector.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: If there is an unmatched detection target set and tracking target set in the matching result corresponding to the current image frame, calculating a target cosine distance between a target appearance feature in a third tracking target in the tracking target set and an appearance feature in the third detection target, where the third tracking target is a target that has not appeared in the tracking target set for a preset time period; If the target cosine distance meets the preset condition, calculating the third final cost matrix of the third tracking target; Matching unmatched objects in the current image frame is performed according to the third final cost matrix.

7. The method according to any one of claims 1 to 5, characterized in that The status information includes: Attribute value information and status value information; The attribute values include the total number of frames generated by the tracking target, the number of frames between the last successful tracking of the tracking target, the position and velocity vectors, and the covariance matrix; The status value information includes: tracking success status, short-term disappearance status, and long-term disappearance status.

8. A detection device for a retained target, characterized in that: include: a determination unit, configured to determine a pedestrian detection result corresponding to a current image frame, wherein the current image frame is any image frame in a target video; a feature extraction unit, configured to determine appearance features and motion features corresponding to each target in the current image frame according to the pedestrian detection result; a cost matrix determining unit, configured to determine a first final cost matrix corresponding to the current image frame according to the appearance feature and the motion feature; a matching unit, configured to determine a matching result corresponding to the current image frame based on the first final cost matrix and second final cost matrices corresponding to N image frames, where the N image frames are image frames before the current image frame, and N is an integer greater than or equal to 2; an updating unit, configured to update the state information of the tracking target corresponding to the current image frame according to the matching result of the current image frame; a judgment unit, configured to determine whether there is a retained target in the current image frame according to the state information of the tracking target corresponding to the current image frame, The feature extraction unit is specifically used for: extracting appearance features corresponding to each target in the current image frame according to a person re-identification network, wherein the person re-identification network is pre-trained on a training data set and is used to identify appearance features of an image; If there is a newly added first tracking target in the current image frame, initializing a first Kalman filter parameter of the first tracking target; determining a first motion feature of the first tracking target in the current image frame according to the first Kalman filter parameter; If the first tracking target does not exist in the current image frame, obtaining a second Kalman filter parameter corresponding to a first detection target in a first image frame, where the first detection target is any target in the current image frame, and the first image frame is an image frame previous to the current image frame; predicting a second motion feature of the first detection target in the current image frame according to the second Kalman filter parameter; determining a Kalman gain corresponding to the first detection target based on a third motion feature and the second motion feature of the first detection target detected in the current image frame; The target motion feature of the first detection target in the current image frame is updated according to the Kalman gain corresponding to the first detection target.

9. A computer storage medium, characterized in that include: The instructions, when executed on a computer, cause the computer to execute the steps of the method for detecting a retained target according to any one of claims 1 to 7.

Images