A method, device and equipment for target detection and tracking

Abstract

The present invention discloses a method, device and equipment for target detection and tracking, which belongs to the field of image processing technology. The method comprises: detecting the confidence and tracking area of ​​all targets in the current frame image; extracting features of the tracking area of ​​each target in the current frame image to obtain a feature point set of the target; if the current frame image is not the first frame image or the second frame image, obtaining an associated frame image of the current frame image, wherein the associated frame image is the first N frame images adjacent to the current frame image, and N is a positive integer greater than or equal to 2; for each target in the current frame image, determining whether the target is a tracked target based on the confidence and feature point set of the associated frame image and the target in the current frame image. The present invention can improve the accuracy of target detection and tracking by associating the target information of the current frame image with the associated frame image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target detection and tracking method, device and equipment. Background Art

[0002] The accuracy and real-time performance of target detection is an important capability of the entire system. Since various objects have different appearances, shapes, and postures, and are interfered by factors such as illumination and occlusion during imaging, target detection has always been the most challenging problem in the field of computer vision.

[0003] In existing target detection and tracking schemes, a single video frame in a video is usually detected separately. If the video frame is disturbed by lighting or occlusion, the detection result of the video frame may be falsely detected. There is a difference between the target detection and tracking result and the actual target motion, which affects the accuracy of target detection and tracking, and thus makes target tracking difficult. Summary of the invention

[0004] The embodiments of the present invention provide a target detection and tracking method, device and equipment to improve the accuracy of target detection and tracking.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] On the one hand, an embodiment of the present invention provides a target detection and tracking method, including:

[0007] Detect the confidence and tracking area of ​​all targets in the current frame image;

[0008] Extracting features from the tracking area of ​​each target in the current frame image to obtain a set of feature points of the target;

[0009] If the current frame image is not the first frame image or the second frame image, acquiring an associated frame image of the current frame image, wherein the associated frame image is the first N frame images adjacent to the current frame image, where N is a positive integer greater than or equal to 2;

[0010] For each of the targets in the current frame image, it is determined whether the target is a tracked target based on the confidence and feature point set of the target in the associated frame image and the current frame image.

[0011] On the other hand, an embodiment of the present invention provides a target detection and tracking device, including:

[0012] Image detection module, used to detect the confidence and tracking area of ​​all targets in the current frame image;

[0013] A feature point extraction module, used to extract features from the tracking area of ​​each target in the current frame image to obtain a feature point set of the target;

[0014] an associated image acquisition module, configured to acquire an associated frame image of the current frame image if the current frame image is not the first frame image or the second frame image, wherein the associated frame image is the first N frame images adjacent to the current frame image, and N is a positive integer greater than or equal to 2;

[0015] The feature point matching module is used to determine, for each target in the current frame image, whether the target is a tracked target based on the confidence and feature point set of the target in the associated frame image and the current frame image.

[0016] On the other hand, an embodiment of the present invention provides a target detection and tracking device, which includes a memory and a processor, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the above-mentioned target detection and tracking method.

[0017] The beneficial effects brought by the technical solution of the present invention are:

[0018] When detecting and tracking each target in the current frame image, the confidence and feature point set of the target in the previous N frames are associated with the confidence and feature point set of the target in the current frame image. This can improve the accuracy of target detection and tracking, and further improve the situation where low confidence in single-frame detection causes target loss and leads to tracking failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a schematic diagram of an application environment of the target detection and tracking method provided by an embodiment of the present invention.

[0021] Figure 2 It is a flowchart of a target detection and tracking method provided by an embodiment of the present invention.

[0022] Figure 3 It is a schematic diagram of a process of detecting a current frame image provided by an embodiment of the present invention.

[0023] Figure 4It is a schematic diagram of a flow chart of feature matching of a target provided by an embodiment of the present invention.

[0024] Figure 5 This is another flowchart of feature matching for a target provided by an embodiment of the present invention.

[0025] Figure 6 It is a flowchart of another target detection and tracking method provided by an embodiment of the present invention.

[0026] Figure 7 It is an example schematic diagram of a target detection and tracking method provided by an embodiment of the present invention.

[0027] Figure 8 4 is a schematic diagram of an example of pedestrian detection and tracking results in a low-light scenario provided by an embodiment of the present invention.

[0028] Fig. 9 It is a schematic diagram of the structure of a target detection and tracking device provided by an embodiment of the present invention.

[0029] Fig.10 It is a schematic diagram of the hardware structure of the target detection and tracking device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0032] See also Figure 1, which shows a schematic diagram of an application scenario of the target detection and tracking method provided by an embodiment of the present invention, the application scenario includes a monitoring camera 101 and a server 102, wherein the monitoring camera 101 is used to send real-time monitored frame images to the server 102, and the server 102 is used to execute the target detection and tracking method provided by an embodiment of the present invention, so as to perform real-time detection and tracking of the target in each frame image sent by the monitoring camera 101.

[0033] When the user needs to track the target in the frame image monitored in real time by the surveillance camera 101, the server 102 performs the following processing on the received current frame image: first, use the image detection model to detect within the entire range of the current frame image to determine the confidence and tracking area of ​​all targets in the current frame image; then use the feature point matching module to extract features of each target in the current frame image and the tracking area corresponding to the target to obtain a feature point set of the target; then obtain the associated frame image of the current frame image, which is the adjacent first N (N is a positive integer greater than or equal to 2) frame images of the current frame image; and then determine whether the target is the tracked target based on the confidence and feature point set of the target in the associated frame image and the current frame image.

[0034] In the video streaming scenario, the server 102 can match the feature points of the current frame image with the corresponding associated frame image by associating the information between each frame, which can improve the situation where low confidence in single-frame detection affects target loss and leads to tracking failure.

[0035] It should be noted that the above Figure 1 The scenario shown is only an example. In practical applications, the target detection and tracking method provided by the embodiment of the present invention can also be applied to other scenarios. No specific limitation is made to the application scenario of the target detection and tracking method herein.

[0036] See also Figure 2 , which shows a flow chart of a target detection and tracking method provided by an embodiment of the present invention. For the sake of convenience, this embodiment is described with a server as the execution subject. It should be understood that the execution subject of the method is not limited to the server, but can also be applied to devices with image processing functions such as terminal devices. Figure 2 As shown, the method may include:

[0037] S201, detecting the confidence and tracking area of ​​all targets in the current frame image.

[0038] Among them, the tracking area is the image area of ​​the target in the current frame image, and the confidence can also be called reliability, confidence level or confidence coefficient. The confidence can characterize the degree of credibility of the detection of the target. In statistics, a confidence interval can refer to the estimated interval of a population parameter constructed by a sample statistic. The confidence interval of a probability sample is an interval estimate of a population parameter of this sample. The confidence interval shows the degree to which the true value of this parameter has a certain probability of falling around the measurement result. The confidence interval gives the degree of credibility of the measured value of the measured parameter, that is, the "certain probability" required above. This probability can be called confidence.

[0039] For details, please refer to Figure 3 , which shows a schematic diagram of the process of detecting the current frame image provided by an embodiment of the present invention. Figure 3 As shown, the confidence and tracking area of ​​detecting all targets in the current frame image may include:

[0040] S2011, preprocessing the current frame image to obtain a preprocessed image.

[0041] In the embodiment of the present invention, the preprocessing may include graying, median filtering, layer-by-layer image scaling and differential processing. Preprocessing the current frame image may highlight the edge features of each target in the current frame image, facilitating clearer detection.

[0042] S2012, extract edge gradient features from the preprocessed image to obtain pre-detection areas of all targets in the current frame.

[0043] Specifically, edge features and directional features are extracted for each layer of the preprocessed image, and the extracted features are input into the pre-detection model for detection to obtain the pre-detection area of ​​all targets in the current frame. The pre-detection model may include a pre-trained classifier, which may use a support vector machine method to identify features. The pre-detection area of ​​the target is the image area of ​​the target in the preprocessed image. By extracting the edge features and directional features of the target and preliminarily confirming the area where the target may appear in the image, the accuracy of subsequent detection can be improved with less computational effort.

[0044] S2013, input the pre-detection area of ​​each target into the target detection model, and the target detection model outputs the confidence and tracking area of ​​the target.

[0045] In an embodiment of the present invention, the target detection model is constructed using a deep convolutional neural network, which includes 5 convolutional layers, 5 downsampling layers and 1 fully connected layer, a total of 11 layers. Among them, the convolutional layer is the key to feature extraction, which is implemented by convolution operation. The convolutional layer, downsampling layer and fully connected layer are composed of related kernel functions, and the kernel function is used to extract target features. The convolutional layer uses a kernel function with a size of 5×5, and the convolutional layer step size is 1. The downsampling layer uses a kernel function with a size of 2×2, and the downsampling layer step size is 1. The downsampling layer takes the average value of the output of the convolutional layer to reduce the number of output weights of the network to improve computational efficiency, and at the same time, it can also increase the image translation invariance ability of feature expression. The fully connected layer integrates the vector extracted by the front-end features, which is used for binary classification and highlights the feature expression.

[0046] In practical applications, the image detection model in the server can be divided into three parts: preprocessing module, pre-detection model and target detection model. When the server detects the confidence and tracking area of ​​all targets in the current frame image, it first pre-processes the current frame image from the surveillance camera through the processing module to enhance the target edge features; then the pre-processed image is sent to the pre-detection model constructed by the classifier for training and classification to obtain the pre-detection area of ​​each target; and then the target detection model detects the confidence and tracking area of ​​each target. By reclassifying the target detection model based on the pre-detection area, the target detection effect is further improved.

[0047] S202, extracting features from the tracking area of ​​each target in the current frame image to obtain a feature point set of the target.

[0048] Specifically, for each pixel in the tracking area of ​​the target, the following operations are performed: calculating the corner point response function value corresponding to the pixel; if the corner point response function value corresponding to the pixel is greater than a preset threshold, adding the pixel to the feature point set.

[0049] In the embodiment of the present invention, for each target tracking area in the current frame image, the feature points of the tracking area can be extracted by the Harris corner detection algorithm to obtain a feature point set of the target. In specific implementation, for each pixel point in the tracking area of ​​the target, if (x, y) represents the coordinates of the pixel point, a small window is defined in the tracking area, and the window moves around the pixel point (x, y). For a small local pixel change (u, v), the corresponding grayscale change can be expressed as:

[0050]

[0051] Among them, I(x,y) is the grayscale value of the pixel (x,y), and I(x+u,y+v) is the grayscale value of the pixel (x+u,y+v).

[0052] By diagonalizing F(u,v), we can get the simplified expression of F(u,v):

[0053]

[0054] Among them, I x is the partial derivative of I(x,y) in the X direction, I y is the partial derivative of I(x,y) in the Y direction.

[0055] The matrix Defined as M, the corner point response function d can be obtained by using the rank and trace of the matrix M:

[0056] d = det(M) - k (trace(M)) 2

[0057] Where det(M) represents the determinant of the matrix M, trace(M) represents the trace of the matrix M, and k represents the proportional coefficient. If the d value is greater than the preset threshold, the pixel point (x, y) is the feature point to be extracted, and the pixel point is added to the feature point set.

[0058] In practical applications, the proportional coefficient k can usually be set to 0.05, and the preset threshold can be set to the maximum value of d in the above small window. For example, in pedestrian detection and tracking, the size of the small window can be set to 4*8.

[0059] S203, if the current frame image is not the first frame image or the second frame image, obtain an associated frame image of the current frame image, wherein the associated frame image is the first N frame images adjacent to the current frame image, and N is a positive integer greater than or equal to 2.

[0060] Usually, various types of targets in an image have different appearances, shapes, and postures (such as motion or stillness). Single-frame detection only detects single-frame images, which is simple to implement, but does not utilize the multi-frame correlation information of moving targets. During the entire life cycle of the target, the detection rate is not high. In addition, due to the interference of factors such as illumination and occlusion during imaging, the target features are partially blurred, and the confidence of single-frame target detection will be greatly reduced, resulting in a difference between the target detection and tracking results and the actual target motion situation, affecting the accuracy of target detection and tracking, and thus making target tracking difficult.

[0061] In view of this, the embodiment of the present invention considers the correlation between adjacent frames, and detects and tracks all targets in the current frame image by acquiring the first N frame images adjacent to the current frame image, thereby improving the detection rate.

[0062] S204: for each target in the current frame image, determine whether the target is a tracked target according to the confidence and feature point set of the target in the associated frame image and the current frame image.

[0063] Usually, a target with a confidence level greater than or equal to a preset confidence threshold can be regarded as a trusted target or a high-quality target. However, due to interference factors such as target motion or occlusion, if the confidence level of the target is less than the preset confidence threshold, the target is not necessarily an untrustworthy target or a low-quality target. Therefore, under certain conditions, even if the confidence level of the target is less than the preset confidence threshold, the target can be regarded as a trusted target or a high-quality target. When tracking a target, it is usually a trusted target or a high-quality target. If the judgment is made only based on the confidence level, the target may be lost.

[0064] Therefore, the embodiment of the present invention divides all the targets in the current frame image into three categories, namely, the first target, the second target and the third target. The first target represents that the confidence of the target is greater than or equal to the preset confidence threshold, the second target represents that the confidence of the target is less than the preset confidence threshold, and the second target meets the preset conditions, and the third target is a target other than the first target and the second target. Therefore, the first target and the second target can also be called high-quality targets.

[0065] In a preferred embodiment, it is considered that there are two frame images adjacent to the current frame image in the associated frame image, that is, the tracking target is determined based on the first two frame images of the current frame image.

[0066] For details, please refer to Figure 4 , which shows a schematic diagram of a process of performing feature matching on a target provided by an embodiment of the present invention. Figure 4 As shown, step S204 may include:

[0067] S2041, determining whether the target is the first target or the second target in the first associated frame image.

[0068] If the target is the first target or the second target in the first associated frame image, it is considered that the target is a high-quality target in the first associated frame image, and step S2042 is executed; if the target is not the first target or the second target in the first associated frame image, it is considered that the target is a low-quality target in the first associated frame image, and step S2046 is executed. The first associated frame image belongs to the associated frame image, and the first associated frame image is the previous frame image adjacent to the current frame image.

[0069] S2042: Determine whether the target is the first target or the second target in the second associated frame image.

[0070] If the target is the first target or the second target in the second associated frame image, it is considered that the target is a high-quality target in the second associated frame image, and step S2043 is executed; if the target is not the first target or the second target in the second associated frame image, it is considered that the target is a low-quality target in the second associated frame image, and step S2046 is executed. The second associated frame image belongs to the associated frame image, and the second associated frame image is the previous frame image adjacent to the first associated frame image.

[0071] S2043: Determine whether the target is the first target in the current frame image according to the confidence level of the target in the current frame image.

[0072] Specifically, it is determined whether the confidence of the target is greater than or equal to a preset confidence threshold. If the confidence of the target is greater than or equal to the preset confidence threshold, the target is the first target in the current frame image. If the target is the first target in the current frame image, step S2045 is executed; if the target is not the first target in the current frame image, step S2044 is executed.

[0073] Preferably, the preset confidence threshold may be set to 0.9.

[0074] S2044: If the target is not the first target in the current frame image, determine whether the target is a tracked target based on a set of feature points of the target in the first associated frame image and the current frame image.

[0075] If the target is not the first target in the current frame image, that is, the confidence of the target in the current frame image is less than the preset confidence threshold, considering that the target may be affected by complex environments such as interference, it cannot be concluded that the target is not a credible target, that is, the tracked target. Therefore, it is necessary to further determine based on the information of the target in the associated frame image.

[0076] For details, please refer to Figure 5 , which shows another flow chart of feature matching of a target provided by an embodiment of the present invention. Figure 5 As shown, determining whether the target is a tracked target according to the feature point set of the target in the first associated frame image and the current frame image may include:

[0077] S20441, obtaining a first difference coefficient according to a difference between a feature point set of the target in the current frame image and a feature point set of the target in the first associated frame image.

[0078] In the embodiment of the present invention, the first difference coefficient can be obtained by calculating the Euclidean norm. In specific implementation, if D represents the feature point set of the target in the current frame image, and D1 represents the feature point set of the target in the first associated frame image, the first difference coefficient can be expressed as:

[0079] U=‖D-D1‖ 2

[0080] S20442, obtaining a preset matching threshold and a preset matching threshold coefficient corresponding to the target.

[0081] In the embodiment of the present invention, the preset matching threshold is related to the current frame image and the frame image before the current frame image, and the preset matching threshold can be dynamically adjusted during the entire life cycle of the target. The value range of the preset matching threshold coefficient is (0,1], and the preset matching threshold coefficient remains unchanged during the life cycle of the target, and the life cycle refers to the target during the entire detection and tracking process.

[0082] S20443, determine whether the first difference coefficient is greater than or equal to the product of the preset matching threshold and the preset matching threshold.

[0083] If the first difference coefficient is greater than or equal to the product of the preset matching threshold and the preset matching threshold, even if the confidence of the target is less than the preset confidence threshold, the target can be considered to be a high-quality target and a credible target, and step S20444 is executed; if the first difference coefficient is less than the product of the preset matching threshold and the preset matching threshold, the target is an untrustworthy target, that is, the target is not a tracked target, and step S20445 is executed.

[0084] S20444: The target is a tracked target, and the target is determined as a second target.

[0085] When the target is not the first target, if the first difference coefficient is greater than or equal to the product of the preset matching threshold and the preset matching threshold, the target is considered to be a credible target, and is therefore determined as the second target, which facilitates detection and tracking of the target in subsequent frame images.

[0086] S20445, the target is not a tracking target.

[0087] S2045: If the target is the first target in the current frame image, then the target is the tracked target.

[0088] It should be noted that, for each target, the premise for it to be a tracked target (i.e., a credible target) in the current frame image is that the target is a high-quality target (i.e., the first target or the second target) in all frame images in the associated frame image. In other words, the confidence of the target in all frame images in the associated frame image is greater than or equal to the preset confidence threshold, or the confidence of the target is less than the preset confidence threshold and certain conditions are met at the same time.

[0089] S2046, the target is not a tracking target.

[0090] It should be noted that the confidence corresponding to the first target is greater than or equal to a preset confidence threshold, the confidence corresponding to the second target is less than the preset confidence threshold, and the second target meets the preset conditions.

[0091] In a possible embodiment, the tracking target may also be determined based on the N frames before the current frame, where N is greater than 2, that is, there are more than two frame images adjacent to the current frame in the associated frame. Specifically, for each target in the current frame, when the target is the first target or the second target in the N frames, the process starts at step S2043, and the execution process is the same as that in the above embodiment, which will not be repeated here.

[0092] When the server performs feature matching on the current frame image, it can determine whether the target is the target to be tracked by comparing the preset matching threshold with the first difference coefficient. When the target is in motion, if the target is affected by interference such as occlusion, the difference between the feature point set of the target in the current frame image and the associated frame image may become larger or smaller. If the same preset matching threshold is always used, there is a possibility of misjudgment and loss of tracking.

[0093] Based on the above description, in a possible embodiment, the target detection and tracking method provided by the embodiment of the present invention may also include: if the target is the first target or the second target in the first associated frame image, and the target is the first target or the second target in the second associated frame image, if the target is the first target in the current frame image, when the preset matching threshold is greater than the first difference coefficient, the preset matching threshold is updated to the first difference coefficient.

[0094] In a possible embodiment, when the server initializes the preset matching threshold, the preset matching threshold may be calculated based on the first frame image and the second frame image. Figure 6 As shown, the target detection and tracking method of the embodiment of the present invention may also include:

[0095] S601, if the current frame image is the second frame image, for each target in the current frame image, a second difference coefficient is obtained according to the difference between a feature point set of the target in the current frame image and a feature point set of the target in the first frame image.

[0096] In the embodiment of the present invention, the second difference coefficient can be obtained by calculating the Euclidean norm. In specific implementation, if D represents the feature point set of the target in the current frame image, and D0 represents the feature point set of the target in the first frame image, the second difference coefficient can be expressed as:

[0097] U 2 =‖D-D0‖ 2

[0098] S602: Determine the second difference coefficient as a preset matching threshold corresponding to the target.

[0099] The target detection and tracking method of the embodiment of the present invention is more comprehensively described below in conjunction with a specific embodiment.

[0100] When the server receives each frame, it detects the confidence and tracking area of ​​all targets in the frame, and calculates the feature point set of each target based on the tracking area. In the tracking process of the target contained in the video stream, the detection is usually based on the previous frame. Therefore, when the server receives the first frame and the second frame, it only detects the target in the image and does not match the features.

[0101] When the current frame image received by the server is not the first frame image or the second frame image, the associated frame image of the current frame image is obtained, and the associated frame image is the first N frame images (N≥2) adjacent to the current frame image. For each target in the current frame image, it is determined whether the target is a tracked target based on the confidence and feature point set of the target in the associated frame image and the current frame image.

[0102] For details, please refer to Figure 7 , which shows an example schematic diagram of a target detection and tracking method provided by an embodiment of the present invention. Figure 7 Middle,U i represents the difference coefficient of the feature point set of the target between the i-th frame image and the i-1-th frame image. For the convenience of description, the preset matching threshold is represented by T, where T is the difference coefficient U between the second frame image and the first frame image. 2 , set the preset matching threshold coefficient to 0.8 and the preset confidence threshold to 0.9.

[0103] Taking the third frame image as the current frame image, the difference coefficient between the third frame image and the second frame image is U 3 Under the premise that the target is a high-quality target in both the first and second frames, if the confidence of the target in the third frame is greater than or equal to 0.9, then the target is a credible target in the third frame, i.e., the target to be tracked. At the same time, if T is greater than U 3 , then update T to U 3 That is, T = U 3 , otherwise T is not updated; if the confidence of the target in the third frame image is less than 0.9, then compare U 3 Is it greater than or equal to 0.8*T? 3 If U is greater than or equal to 0.8*T, then the target in the third frame image is a credible target, i.e., the tracked target, and the target is the second target; if U 3If it is less than 0.8*T, the target in the third frame image is not a credible target, i.e., a tracked target.

[0104] Taking the fourth frame as an example, assuming that in the third frame, T has been updated to U 3 The difference coefficient between the fourth frame image and the third frame image is U 4 Under the premise that the target is a high-quality target in both the second and third frames, if the confidence of the target in the fourth frame is greater than or equal to 0.9, then the target is a credible target in the fourth frame, i.e., the tracked target. At the same time, if T is greater than U 4 , then update T to U 4 That is, T = U 4 , otherwise T is not updated; if the confidence of the target in the fourth frame image is less than 0.9, then compare U 4 Is it greater than or equal to 0.8*T? 4 If U is greater than or equal to 0.8*T, then the target in the fourth frame image is a credible target, i.e., the tracked target, and the target is also the second target; 4 If it is less than 0.8*T, the target in the third frame image is not a credible target, i.e., a tracked target.

[0105] When the server processes subsequent frame images, it processes them in the same way as the third or fourth frame images to complete the multi-frame associated dynamic detection and tracking of the target. During the life cycle of each target, the preset matching threshold T of the target will be updated to ensure that the value of T is the minimum value of the difference coefficient between all adjacent frames.

[0106] By collecting videos from surveillance cameras to test and analyze pedestrians, if the total number of pedestrians in the video is represented by NP, and the total number of correctly detected pedestrians is represented by NTDP, the detection rate DR can be expressed as:

[0107] DR=NTDP / NP

[0108] After comparison, it is found that in the dynamic scene of pedestrians, the DR of the YOLO algorithm is 0.7, while the DR of the target detection and tracking method of the embodiment of the present invention is 0.85, and the detection and tracking rate is improved by about 20%. It can be seen that the target detection and tracking method of the embodiment of the present invention improves the problem of target tracking difficulty caused by single-frame detection under ambient light difference conditions. For details, please refer to Figure 8 , Figure 8 This is a diagram showing an example of pedestrian detection and tracking results in a low-light scene. Figure 8 It can be seen that pedestrians can be effectively detected and tracked even in low-light scenes.

[0109] It can be seen from the above technical scheme of the embodiment of the present invention that after determining the tracking area of ​​each target, the embodiment of the present invention extracts the feature points of the target, matches the extracted feature points with relevant frames, confirms the state of the target in the current frame image, obtains the detection result of the current frame image, and tracks the target according to the detection result. By considering the correlation between the feature point sets of the target between adjacent frames, the accuracy of target detection and tracking can be improved, thereby improving the situation in which the confidence level is low during single-frame detection, resulting in target loss and tracking failure.

[0110] See also Fig. 9 , which shows a schematic diagram of the structure of a target detection and tracking device provided by an embodiment of the present invention. Fig. 9 As shown, the device 900 may include:

[0111] Image detection module 901, used to detect the confidence and tracking area of ​​all targets in the current frame image;

[0112] A feature point extraction module 902 is used to extract features from the tracking area of ​​each target in the current frame image to obtain a feature point set of the target;

[0113] The associated image acquisition module 903 is used to acquire an associated frame image of the current frame image if the current frame image is not the first frame image or the second frame image, wherein the associated frame image is the first N frame images adjacent to the current frame image, and N is a positive integer greater than or equal to 2;

[0114] The feature point matching module 904 is used to determine, for each target in the current frame image, whether the target is a tracked target based on the confidence and feature point set of the target in the associated frame image and the current frame image.

[0115] In a possible embodiment, the image detection module 901 may include:

[0116] A preprocessing unit, used for preprocessing the current frame image to obtain a preprocessed image;

[0117] An edge feature extraction unit, used for extracting edge gradient features from the preprocessed image to obtain a pre-detection area of ​​all targets in the current frame;

[0118] The target detection unit is used to input the pre-detection area of ​​each target into the target detection model, and the target detection model outputs the confidence and tracking area of ​​the target.

[0119] In practical applications, the target detection model can be constructed using a deep convolutional neural network, which can include 5 convolutional layers, 5 downsampling layers and 1 fully connected layer.

[0120] In a possible embodiment, the feature point extraction module 902 may be used to perform the following operations on each pixel point in the tracking area of ​​the target:

[0121] Calculate the corner point response function value corresponding to the pixel point;

[0122] If the corner point response function value corresponding to the pixel point is greater than a preset threshold, the pixel point is added to the feature point set.

[0123] In a possible embodiment, the feature point matching module 904 may include:

[0124] a first judgment unit, configured to determine whether the target is the first target in the current frame image according to the confidence of the target in the current frame image if the target is the first target or the second target in the first associated frame image, and the target is the first target or the second target in the second associated frame image, wherein the first associated frame image and the second associated frame image both belong to the associated frame images, the first associated frame image is a previous frame image adjacent to the current frame image, and the second associated frame image is a previous frame image adjacent to the first associated frame image;

[0125] A first determining unit, configured to determine that if the target in the current frame image is the first target, the target is the tracked target;

[0126] a second determining unit, configured to determine whether the target is a tracked target according to a set of feature points of the target in the first associated frame image and the current frame image if the target is not the first target in the current frame image;

[0127] Among them, the confidence corresponding to the first target is greater than or equal to a preset confidence threshold, the confidence corresponding to the second target is less than the preset confidence threshold, and the second target meets a preset condition.

[0128] In a possible embodiment, the second determining unit may also be used to:

[0129] Obtaining a first difference coefficient according to a difference between a feature point set of the target in the current frame image and a feature point set of the target in the first associated frame image;

[0130] Obtaining a preset matching threshold and a preset matching threshold coefficient corresponding to the target;

[0131] If the first difference coefficient is greater than or equal to the product of the preset matching threshold and the preset matching threshold coefficient, the target is the tracked target, and the target is determined as the second target;

[0132] If the first difference coefficient is smaller than the product of the preset matching threshold and the preset matching threshold coefficient, the target is not the tracked target.

[0133] In a possible embodiment, the device 900 may also include a threshold updating module, which is used to: if the target is the first target or the second target in the first associated frame image, and the target is the first target or the second target in the second associated frame image, if the target is the first target in the current frame image, when the preset matching threshold is greater than the first difference coefficient, update the preset matching threshold to the first difference coefficient.

[0134] In a possible embodiment, the device 900 may also include a threshold initialization module, which is used to: if the current frame image is the second frame image, then for each target in the current frame image, obtain a second difference coefficient based on the difference between the feature point set of the target in the current frame image and the feature point set of the target in the first frame image; and determine the second difference coefficient as a preset matching threshold corresponding to the target.

[0135] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0136] An embodiment of the present invention also provides a target detection and tracking device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the target detection and tracking method provided in the above method embodiment.

[0137] Further, Fig.10 FIG. 1 is a schematic diagram showing a hardware structure of a target detection and tracking device provided by an embodiment of the present invention. The target detection and tracking device may participate in or include the target detection and tracking device provided by an embodiment of the present invention. Fig.10As shown, the target detection and tracking device 100 may include one or more (1002a, 1002b, ..., 1002n are used to illustrate) processors 1002 (the processor 1002 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.10 The structure shown is only for illustration and does not limit the above device structure. Fig.10 More or fewer components as shown, or with Fig.10 Different configurations are shown.

[0138] It should be noted that the one or more processors 1002 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuit may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the target detection and tracking device 100 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0139] The memory 1004 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method in the embodiment of the present invention. The processor 1002 executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, that is, to implement the above-mentioned digital car key management method. The memory 1004 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1004 may further include a memory remotely arranged relative to the processor 1002, and these remote memories may be connected to the target detection and tracking device 100 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0140] The transmission device 1006 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the target detection and tracking device 100. In one example, the transmission device 1006 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one embodiment, the transmission device 1006 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0141] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the target detection and tracking device 100 (or a mobile device).

[0142] It should be noted that the sequence of the embodiments of the present invention described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0144] The above description has fully disclosed the specific embodiments of the present invention. It should be pointed out that any changes made by those skilled in the art to the specific embodiments of the present invention do not deviate from the scope of the claims of the present invention. Accordingly, the scope of the claims of the present invention is not limited to the above specific embodiments.

Claims

1. A target detection and tracking method, characterized in that: include: Detect the confidence and tracking area of ​​all targets in the current frame image; Extracting features from the tracking area of ​​each target in the current frame image to obtain a set of feature points of the target; If the current frame image is not the first frame image or the second frame image, acquiring an associated frame image of the current frame image, wherein the associated frame image is the first N frame images adjacent to the current frame image, where N is a positive integer greater than or equal to 2; For each of the targets in the current frame image, determining whether the target is a tracked target according to the confidence and feature point set of the target in the associated frame image and the current frame image includes: If the target is the first target or the second target in the first associated frame image, and the target is the first target or the second target in the second associated frame image, determining whether the target is the first target in the current frame image according to the confidence of the target in the current frame image, wherein the first associated frame image and the second associated frame image both belong to the associated frame images, the first associated frame image is a previous frame image adjacent to the current frame image, and the second associated frame image is a previous frame image adjacent to the first associated frame image; If the target is the first target in the current frame image, then the target is the tracked target; If the target is not the first target in the current frame image, determining whether the target is a tracked target according to the first associated frame image and the feature point set of the target in the current frame image; wherein the confidence corresponding to the first target is greater than or equal to a preset confidence threshold, the confidence corresponding to the second target is less than the preset confidence threshold, and the second target meets a preset condition; Obtaining a first difference coefficient according to a difference between a feature point set of the target in the current frame image and a feature point set of the target in the first associated frame image; Obtaining a preset matching threshold and a preset matching threshold coefficient corresponding to the target; If the first difference coefficient is greater than or equal to the product of the preset matching threshold and the preset matching threshold coefficient, the target is the tracked target, and the target is determined as the second target; If the first difference coefficient is smaller than the product of the preset matching threshold and the preset matching threshold coefficient, the target is not the tracked target.

2. The method according to claim 1, characterized in that The confidence and tracking area of ​​all targets detected in the current frame image include: Preprocessing the current frame image to obtain a preprocessed image; Extracting edge gradient features from the preprocessed image to obtain pre-detection areas of all targets in the current frame; The pre-detection area of ​​each target is input into the target detection model, and the target detection model outputs the confidence and tracking area of ​​the target.

3. The method according to claim 2, characterized in that The target detection model is constructed using a deep convolutional neural network, which includes 5 convolutional layers, 5 downsampling layers and 1 fully connected layer.

4. The method according to any one of claims 1 to 3, characterized in that: The step of extracting features from the tracking area of ​​each target in the current frame image to obtain a set of feature points of the target includes: For each pixel in the tracking area of ​​the target, the following is performed: Calculate the corner point response function value corresponding to the pixel point; If the corner point response function value corresponding to the pixel point is greater than a preset threshold, the pixel point is added to the feature point set.

5. The method according to claim 1, characterized in that The method further comprises: If the target is the first target or the second target in the first associated frame image, and the target is the first target or the second target in the second associated frame image, if the target is the first target in the current frame image, when the preset matching threshold is greater than the first difference coefficient, the preset matching threshold is updated to the first difference coefficient.

6. The method according to claim 5, characterized in that The method further comprises: If the current frame image is the second frame image, for each target in the current frame image, a second difference coefficient is obtained according to a difference between a feature point set of the target in the current frame image and a feature point set of the target in the first frame image; The second difference coefficient is determined as a preset matching threshold corresponding to the target.

7. A target detection and tracking device, characterized in that: include: Image detection module, used to detect the confidence and tracking area of ​​all targets in the current frame image; A feature point extraction module, used to extract features from the tracking area of ​​each target in the current frame image to obtain a feature point set of the target; an associated image acquisition module, configured to acquire an associated frame image of the current frame image if the current frame image is not the first frame image or the second frame image, wherein the associated frame image is the first N frame images adjacent to the current frame image, and N is a positive integer greater than or equal to 2; A feature point matching module is used to determine, for each target in the current frame image, whether the target is a tracked target based on the confidence and feature point set of the targets in the associated frame image and the current frame image, including: if the target is the first target or the second target in the first associated frame image, and the target is the first target or the second target in the second associated frame image, then determine whether the target is the first target in the current frame image based on the confidence of the target in the current frame image, wherein the first associated frame image and the second associated frame image both belong to the associated frame images, the first associated frame image is the previous frame image adjacent to the current frame image, and the second associated frame image is the previous frame image adjacent to the first associated frame image; if the target is the first target in the current frame image, then the target is the tracked target; if the target is not the first target in the current frame image, then the target is the tracked target; wherein the first target is a first target, then according to the feature point set of the target in the first associated frame image and the current frame image, it is determined whether the target is a tracked target; wherein the confidence corresponding to the first target is greater than or equal to a preset confidence threshold, the confidence corresponding to the second target is less than the preset confidence threshold, and the second target meets a preset condition; according to the difference between the feature point set of the target in the current frame image and the feature point set of the target in the first associated frame image, a first difference coefficient is obtained; a preset matching threshold and a preset matching threshold coefficient corresponding to the target are obtained; if the first difference coefficient is greater than or equal to the product of the preset matching threshold and the preset matching threshold coefficient, the target is the tracked target, and the target is determined as the second target; if the first difference coefficient is less than the product of the preset matching threshold and the preset matching threshold coefficient, the target is not the tracked target.

8. A target detection and tracking device, characterized in that: The target detection and tracking device includes a memory and a processor, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the method described in any one of claims 1-6.

Images