Target tracking method and device

Abstract

The invention provides a target tracking method and device. The method comprises the following steps: acquiring a first feature map of a current frame image; aligning and registering the first feature map to a feature map of a previous frame of image of the current frame of image to obtain a registration map corresponding to the current frame of image; performing target detection on the registration image corresponding to the current frame image to obtain position information of a plurality of detection frames in the current frame image and categories to which targets in the plurality of detection frames belong; performing target tracking according to the position information of the same type of detection frames in the current frame image and the previous frame image; therefore, the target position deviation caused by the difference between the video frames is reduced, the relevance of the target between the continuous frames is improved, the target tracking is more stable and accurate, and the robustness and accuracy of the target tracking are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a target tracking method and device. Background Art

[0002] Object tracking is one of the hot topics in computer vision research. Object tracking has a wide range of applications in video surveillance, navigation, military, human-computer interaction, virtual reality, autonomous driving, and many other fields. Simply put, object tracking is to analyze and track a given object in a video to determine the exact location of the object in the video. Since the accuracy and real-time performance of object tracking are crucial for many application scenarios, it is very important to accurately track the object in the video. Summary of the invention

[0003] The present disclosure provides a target tracking method and device to solve one of the technical problems in the related art at least to a certain extent. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a target tracking method is provided, comprising: obtaining a first feature map of a current frame image; aligning and registering the first feature map to a feature map of a previous frame image of the current frame image to obtain a registration map corresponding to the current frame image; performing target detection on the registration map corresponding to the current frame image to obtain position information of multiple detection frames in the current frame image and categories to which targets in the multiple detection frames belong; and performing target tracking based on the position information of detection frames of the same category in the current frame image and the previous frame image.

[0005] According to a second aspect of an embodiment of the present disclosure, a target tracking device is provided, including: an acquisition module, used to acquire a first feature map of a current frame image; a registration module, used to align and register the first feature map to the feature map of a previous frame image of the current frame image, so as to obtain a registration map corresponding to the current frame image; a detection module, used to perform target detection on the registration map corresponding to the current frame image, so as to obtain position information of multiple detection frames in the current frame image and categories to which targets in the multiple detection frames belong; and a tracking module, used to track the target based on the position information of detection frames of the same category in the current frame image and the previous frame image.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a target tracking method as described in any of the above embodiments of the present disclosure.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the target tracking method as described in any of the above embodiments of the present disclosure.

[0008] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising: a computer program, wherein when the computer program is executed by a processor, the target tracking method as described in any of the above embodiments of the present disclosure is implemented.

[0009] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0010] In the present technical scheme, firstly, by acquiring the first feature map of the current frame image and aligning and registering it with the feature map of the previous frame image, the target position deviation caused by the difference between video frames (such as motion, illumination change, etc.) can be significantly reduced, and the correlation between the target in consecutive frames is improved, making the target tracking more stable and accurate. Secondly, target detection is performed on the registered image, and the position information and category of multiple targets in the current frame can be accurately identified, which provides a reliable basis for subsequent tracking and enables the system to process multiple targets at the same time, improving the processing power and flexibility of the system. Finally, target tracking is performed according to the position information of the detection boxes of the same category in the current frame and the previous frame, which can achieve continuous and stable tracking of the target and improve the robustness and accuracy of target tracking. Among them, when performing target detection on the registered image, multi-scale feature fusion is used to identify the current frame. It can detect multiple targets in the image and output the location information and category of each target, which improves the performance of target detection in different scales and complex backgrounds, realizes pixel-level target detection, and improves the accuracy and comprehensiveness of target detection. In addition, after target detection is performed on the current frame image, the location information of the detection box corresponding to the current frame image is obtained. By comparing the location information of the detection boxes of the same category in the current frame and the previous frame, it is possible to more accurately judge whether these detection boxes contain the same target, which helps to reduce false detection and missed detection and improve the reliability of target detection. When the same target exists, the target existence classification model is used to further verify whether there is a target in the detection box, thereby reducing false positives and missed negatives, and thus tracking the target in the current frame image based on the detection result of the same target in the previous frame image, ensuring the continuity and stability of the target, and can handle the movement, occlusion and disappearance of the target.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0013] Figure 1 is a flowchart of a target tracking method shown in the first embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of a target tracking method shown in the second embodiment of the present disclosure;

[0015] Figure 3 is a flowchart of a target tracking method shown in the third embodiment of the present disclosure;

[0016] Figure 4 is a flowchart of a target tracking method shown in the fourth embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of the target tracking principle shown in an embodiment of the present disclosure;

[0018] Figure 6 is a schematic diagram of the structure of the target detection and tracking model shown in the embodiment of the present disclosure;

[0019] Figure 7 is a schematic diagram of a specific structure of a target detection and tracking model shown in an embodiment of the present disclosure;

[0020] Figure 8 is a schematic diagram of a false detection model shown in an embodiment of the present disclosure;

[0021] Fig. 9 is a schematic structural diagram of a target tracking device shown in the fifth embodiment of the present disclosure;

[0022] Fig.10 It is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0025] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, and are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0026] Object detection and tracking is one of the most basic downstream tasks in the field of computer vision, and is also one of the most common deep learning tasks that can be implemented in the industry. The specific task of object detection is to regress the coordinate box of the target to be identified in the input image, and output the specific category of the target. At present, there are two mature technical solutions based on convolutional neural networks and Transformer models. Algorithms based on convolutional neural networks are widely used in server-side and mobile devices because they have relatively mature compression and quantization technologies, which greatly improve the model frame rate while ensuring model accuracy. Object detection based on convolutional neural networks can be divided into two types according to the technical route: Anchor_base (based on anchor boxes) and Anchor_free (without anchor boxes). Anchor_base algorithms are the first algorithms studied and implemented in the industry. They need to set a priori boxes of several different aspect ratios according to the strategy in advance. In order to identify target objects of different scales, it is usually necessary to set a priori boxes containing small targets and large targets. This method has a very high recognition accuracy for normal targets and large targets, but it is unsatisfactory for small target detection. The Anchor_free technical route does not require setting a priori boxes with different aspect ratios in advance, but it needs to obtain the heatmap of the target. The position of the target will have a large response on the heatmap, while the other positions will be weak. The Anchor_free technical route is more friendly to small targets and pixel-level targets, and the recognition effect of targets of different sizes is also more extensive. The disadvantage is that it is difficult to train and the loss function is more complex to construct. The target detection algorithms based on the Transformer model are relatively large in size and are usually used for open domain target detection and recognition of all things. Since the algorithms based on the attention mechanism have a global receptive field and representation ability than convolutional neural networks, Transformer becomes more effective in realizing target detection on large models.

[0027] The main difficulty of the tracking task is to ensure the continuity of the two target IDs (identifications). How to determine whether the target frames of the two frames belong to the same ID and how to continue to follow the target after occlusion are the two problems that the tracking task needs to solve. Currently, tracking is mainly based on the Kalman filter algorithm. The Kalman filter has good performance for linear motion, but is not effective for obvious self-jitter or nonlinear motion.

[0028] In addition, in the related art, a target tracker is also used for target tracking. The core of the tracker is the Kalman filter algorithm and the Hungarian algorithm. It uses the Kalman filter to predict the state of the target in the next frame, and then uses the Hungarian algorithm to match this state with the detection result of the next frame to achieve target tracking. If the target is not detected in the next frame, for example, due to occlusion or other reasons, the state information predicted by the Kalman filter will not match the detection result, resulting in the premature end of the tracking segment.

[0029] Therefore, in order to solve the above problems, the present disclosure proposes a target tracking method and device.

[0030] The target tracking method and device according to the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0031] Figure 1 It is a flowchart of the target tracking method shown in the first embodiment of the present disclosure.

[0032] like Figure 1 As shown, the target tracking method includes the following steps:

[0033] Step 101, obtaining a first feature map of the current frame image.

[0034] In order to improve the accuracy of target detection, as a possible implementation method, key features of the current frame image are extracted, and a first feature map is generated based on the key features.

[0035] As an example, a relevant feature extraction algorithm is used to extract features from the current frame image to obtain a first feature map.

[0036] As another example, the current frame image is input into a convolutional network or an encoding network to perform feature extraction on the current frame image to obtain a first feature map of the current frame image.

[0037] Step 102: align and register the first feature map with the feature map of the previous frame image of the current frame image to obtain a registration map corresponding to the current frame image.

[0038] In order to eliminate jitter and offset during image capture, as a possible implementation method, the first feature map of the current frame is accurately aligned and registered with the feature map of the previous frame of the current frame to ensure that the correspondence between the two frames in the feature space remains consistent, thereby obtaining a registration map that reflects the feature distribution of the current frame.

[0039] In an embodiment of the present disclosure, based on the homography matrix of any two frames of images, the current frame image is aligned to the previous frame image to obtain an aligned image of the current frame image; wherein the homography matrix is ​​used to indicate the perspective transformation relationship of the current frame image to the previous frame image; the optical flow vector field between the aligned image and the previous frame image is calculated; wherein the optical flow vector field is used to indicate the moving direction and moving speed of the pixel points in the aligned image relative to the corresponding pixel points in the previous frame image; based on the optical flow vector field of the aligned image, the aligned images are locally aligned to obtain a registration map corresponding to the current frame image.

[0040] That is to say, based on the homography matrix between the current frame image and the previous frame image, the current frame image is perspective transformed so that the current frame image is visually consistent with the previous frame image, thereby generating an aligned image of the current frame image. Next, the optical flow vector field between the aligned image and the previous frame image is calculated, wherein the optical flow vector field is a two-dimensional vector field, and each vector represents the moving direction and speed of a pixel point in the aligned image relative to a corresponding pixel point in the previous frame image. Based on the optical flow information, the local area in the aligned image is adjusted so that the adjusted aligned image is aligned with the previous frame image, and the alignment map corresponding to the current frame image is obtained.

[0041] Step 103, performing target detection on the registration map corresponding to the current frame image to obtain position information of multiple detection frames in the current frame image and the categories to which the targets in the multiple detection frames belong.

[0042] As a possible implementation method, a target detection model is used to perform target detection on the registration map corresponding to the current frame image. The target detection model can predict the location and category of multiple targets. The target detection model outputs at least one detection frame and its corresponding confidence score and category label. It should be noted that each detection frame is usually defined by four coordinate values, namely the (x, y) coordinates of the upper left corner and the lower right corner. These coordinates specify the specific location of the target in the image, and the category label indicates the category to which the target in each detection frame belongs, such as "person", "car", "cat", etc. The confidence score reflects the model's confidence in the detection result, usually between 0 and 1, and the higher the value, the stronger the confidence.

[0043] In order to achieve pixel-level target detection, in the embodiment of the present disclosure, the registration image is downsampled according to multiple different sampling rates to obtain multiple downsampled images; the feature maps of the multiple downsampled images are fused to obtain a fused feature map; and target detection is performed on the fused feature map to obtain the position information of multiple detection boxes in the current frame image and the categories to which the targets in the multiple detection boxes belong.

[0044] That is to say, firstly, the registration map is downsampled by using multiple different sampling rates to generate a series of downsampled maps with decreasing resolution, wherein these downsampled maps include information of different scales from fine to coarse; then, a feature map is extracted for each downsampled map, and the feature map contains the key information of the target in the image; then, these feature maps of different scales are fused to integrate cross-scale information and enhance the feature expression capability, so that the fused feature map contains both fine detail information and global context information; finally, target detection is performed on the fused feature map to identify multiple targets in the current frame image, and the location information of each target (such as the coordinates of the bounding box) and the category to which it belongs are output. Thus, through multi-scale feature fusion, the performance of target detection at different scales and complex backgrounds is improved, pixel-level target detection is achieved, and the accuracy and comprehensiveness of target detection are improved.

[0045] Step 104 , target tracking is performed based on the position information of the detection frames of the same category in the current frame image and the previous frame image.

[0046] In order to improve the accuracy of target tracking, in an embodiment of the present disclosure, target detection results are obtained from the current frame image and the previous frame image respectively, wherein the detection results generally include the position information of each detection frame (generally the upper left corner and lower right corner coordinates of the bounding box) and the category label of the target in the detection frame; based on the target detection results of the current frame image and the previous frame image, detection frames of the same category are matched between the current frame image and the previous frame image, and target tracking is performed based on the matching results.

[0047] In summary, firstly, by obtaining the first feature map of the current frame image and aligning and registering it with the feature map of the previous frame image, the target position deviation caused by differences between video frames (such as motion, illumination changes, etc.) can be significantly reduced, and the correlation between targets in consecutive frames is improved, making target tracking more stable and accurate. Secondly, target detection is performed on the registered image, which can accurately identify the position information and categories of multiple targets in the current frame, providing a reliable basis for subsequent tracking, and enabling the system to process multiple targets at the same time, improving the processing power and flexibility of the system. Finally, target tracking is performed according to the position information of the detection boxes of the same category in the current frame and the previous frame, which can achieve continuous and stable tracking of the target and improve the robustness and accuracy of target tracking.

[0048] In order to clearly explain how the target is tracked according to the position information of the detection frames of the same category in the current frame image and the previous frame image in the above embodiment, the present disclosure proposes another target tracking method.

[0049] Figure 2 It is a flowchart of a target tracking method shown in the second embodiment of the present disclosure.

[0050] like Figure 2 As shown, the target tracking method includes the following steps:

[0051] Step 201, obtaining a first feature map of the current frame image.

[0052] Step 202: align and register the first feature map with the feature map of the previous frame image of the current frame image to obtain a registration map corresponding to the current frame image.

[0053] Step 203: Perform target detection on the registration map corresponding to the current frame image to obtain position information of multiple detection frames in the current frame image and the categories to which the targets in the multiple detection frames belong.

[0054] Step 204 , judging whether the detection frames of the same category contain the same target based on the position information of the detection frames of the same category in the current frame image and the previous frame image.

[0055] In order to improve the accuracy of target tracking, in the embodiment of the present disclosure, based on the position information of the detection frames of the same category in the current frame image and the previous frame image, the target center points in the detection frames of the same category in the current frame image and the previous frame image are determined, and based on the distance between the target center points in the current frame image and the previous frame image, it is determined whether the detection frames of the same category contain the same target.

[0056] Step 205 , in response to containing the same target, using the target existence classification model to detect the detection frame of the same category in the current frame image to determine whether there is a target in the detection frame of the same category in the current frame image.

[0057] In order to further improve the accuracy of target tracking, as a possible implementation method, when confirming that the same target is contained in the detection frame of the same category in the current frame image and the previous frame image, a target existence classification model (such as a binary classification model) is further used to detect the detection frame of the same category. This can more accurately determine the existence of the target in the current frame, further verifying the accuracy of containing the same target in the detection frame of the same category in the current frame image and the previous frame image, thereby improving the accuracy of target tracking.

[0058] In an embodiment of the present disclosure, a target image block is input into a target existence classification model to obtain a probability value output by the target existence classification model; wherein the probability value is used to indicate the possibility of the existence of a target in the target image block; and based on the comparison result between the probability value and a set probability threshold, it is determined whether the target exists in the target image block.

[0059] For example, the probability value output by the target existence classification model is 0.75, and the probability threshold is set to 0.7. Since 0.75 is greater than or equal to 0.7, it can be determined that the target exists in the target image block.

[0060] Step 206 , in response to the existence of the target, tracking the same target in the current frame image based on the first detection result of the same target in the previous frame image.

[0061] In order to improve the accuracy of target tracking, in the embodiment of the present disclosure, when there is a target in the same detection frame and it is the same target as the previous frame, the detection result of the same target in the current frame is updated based on the first detection result of the same target in the previous frame image, and the best detection result of the same target in the current frame image is obtained (for example, including the position, speed, etc. of the target), so that the same target in the current frame image is tracked based on the best detection result of the same target in the current frame.

[0062] In summary, by comparing the position information of the detection frames of the same category in the current frame and the previous frame, it is possible to more accurately determine whether these detection frames contain the same target, which helps to reduce false detections and missed detections and improve the reliability of target detection; the target existence classification model can further verify whether there is a target in the detection frame, thereby reducing false positives and missed negatives; based on the detection results of the same target in the previous frame image, the target in the current frame image is tracked, ensuring the continuity and stability of the target, and can handle situations such as target movement, occlusion and disappearance.

[0063] Based on any of the above embodiments, multiple detection frames of the current frame image are cropped to obtain image blocks and generate a heat map. Combined with the offset of the target center point, it is determined whether the detection frames of the same category in the current frame and the previous frame contain the same target. Figure 3 The embodiments are described in detail.

[0064] Figure 3 It is a flowchart of a target tracking method shown in the third embodiment of the present disclosure.

[0065] like Figure 3 As shown, whether the detection frames of the same category contain the same target is determined by the following steps 301 to 304:

[0066] Step 301 : cropping the current frame image according to multiple detection frames in the current frame image to obtain multiple image blocks of the current frame image.

[0067] The image block includes a detection box.

[0068] In order to analyze specific areas in an image more efficiently or more accurately, in the embodiment of the present disclosure, the image is cropped according to multiple detection frames that have been detected in the image. For example, with each detection frame as the center, a local area containing these targets is intercepted from the current frame image according to a predetermined size or ratio, thereby generating multiple image blocks. Each image block corresponds to a detected target in the original image and its surrounding context information.

[0069] Step 302: Obtain heat maps corresponding to multiple image blocks.

[0070] Among them, the heat map is used to indicate the center point of the image block.

[0071] In order to improve the accuracy and robustness of target tracking, in the embodiment of the present disclosure, a heat map corresponding to each image block is obtained. The heat map usually represents the significance or activity of the target area in a pixel-level manner. The heat map is used to locate the center point of the target in the corresponding detection frame, that is, the heat map is used to indicate the center point of the target in the image block.

[0072] Step 303: Determine the offset of the target center point in the target image block according to the coordinates of the target center point in the target image block in the current frame image and the previous frame image.

[0073] The target image block is an image block containing a detection box of the same category.

[0074] It is important to understand that during target tracking, target detection may produce errors due to interference from factors such as lighting changes, occlusion, and noise. By calculating the offset, the tracking instability caused by detection errors can be reduced.

[0075] Therefore, in order to improve the target tracking accuracy and stability, in the embodiment of the present disclosure, the coordinates of the target center point in the current frame are compared with the coordinates of the corresponding target center point in the previous frame to obtain the moving distance and direction of the target between two consecutive frames, that is, the offset of the target center point. The offset is a vector, which contains two components: one offset in the horizontal direction (usually the x-axis) and the other offset in the vertical direction (usually the y-axis).

[0076] For example, the center point coordinates of the previous frame are (x prev ,y prev ), the center point coordinates of the current frame are (x curr ,y curr), then the offset of the center point can be calculated by the following formula:

[0077] Δx=x curr -x prev ; Δy=y curr -y prev .

[0078] Step 304 , determining whether the detection boxes of the same category in the current frame image and the previous frame image contain the same target based on the offset and the coordinates of the target center point in the target image block in the current frame image.

[0079] In an embodiment of the present disclosure, the predicted coordinate value of the target image block in the previous frame image is determined based on the offset and the coordinates of the center point of the target image block in the current frame image; based on the difference between the predicted coordinate value and the actual coordinate value of the target center point in the target image block in the previous frame image, it is determined that the detection frames of the same category contain the same target.

[0080] For example, with an offset of The target center point in the target image block in the current frame image is For example, the predicted coordinate value of the target center point in the target image block in the previous frame image

[0081]

[0082] by Taking the actual coordinate value of the target center point in the target image block in the previous frame image as an example, the difference between the predicted coordinate value and the actual coordinate value of the target center point in the target image block in the previous frame image is

[0083] In an embodiment of the present disclosure, in response to the absolute value of the difference being less than a set difference threshold, it is determined that the detection frames of the same category contain the same target; in response to the absolute value of the difference being not less than the set difference threshold, it is determined that the detection frames of the same category contain non-identical targets.

[0084] That is, the difference between the predicted coordinate value and the actual coordinate value of the target center point in the target image block in the previous frame image For example, when dist is less than the set difference threshold, it is determined that the target maintains a relatively stable position between the two frames, so that the detection frames of the same category contain the same target. When dist is greater than or equal to the set difference threshold, it is determined that the detection frames of the same category contain non-identical targets.

[0085] In summary, by cropping multiple detection frames in the current frame image, multiple image blocks containing targets can be obtained, which can significantly reduce the interference of background information and make target tracking more focused on the target itself, thereby improving tracking accuracy; then, the heat map corresponding to the image block is obtained, and the center point of the target in the detection frame is located based on the heat map, and the offset of the target center point is calculated according to the coordinates of the center point of the target image block in the current frame and the previous frame. Combined with the offset and the coordinates of the center point of the target image block in the current frame, it can be accurately determined whether the detection frames of the same category in the current frame and the previous frame contain the same target.

[0086] On the basis of any of the above embodiments, when there is a target in the same detection frame and it is the same target as the previous frame, the detection result of the same target in the current frame is updated based on the first detection result of the same target in the previous frame image, and the best detection result of the same target in the current frame image is obtained, so that the same target in the current frame image is tracked based on the best detection result of the same target in the current frame. Figure 4 The embodiments are described in detail.

[0087] Figure 4 It is a flowchart of a target tracking method shown in the fourth embodiment of the present disclosure.

[0088] like Figure 4 As shown, the same target in the current frame image is tracked using the following steps 401 to 404:

[0089] Step 401 , in response to the existence of a target in the target image block, a Kalman filter is used to predict a second detection result of the same target in the current frame image based on a first detection result of the same target in the previous frame image.

[0090] In the embodiments of the present disclosure, a second detection result of the target in the current frame image is estimated based on a first detection result of the same target in the previous frame image (for example, the first detection result may include status information such as the position and speed of the target). This allows the position and status of the target in the current frame to be predicted more accurately even when the target is partially obscured or the image quality is poor, thereby achieving continuous and stable target tracking.

[0091] Step 402, obtaining a third detection result of the same target in the current frame image.

[0092] Among them, the third detection result is obtained by performing target detection on the current frame image using a target detection model.

[0093] In order to further improve the accuracy of target tracking, in the embodiment of the present disclosure, the best detection result of the same target in the current frame image is determined based on the third detection result of the same target in the current frame image output by the target detection model and the second detection result of the same target in the current frame image predicted by the Kalman filter. Therefore, the third detection result of the same target obtained by the target detection model performing target detection on the current frame image can be obtained first.

[0094] Step 403, determining the best detection result of the same target in the current frame image according to the second detection result and the third detection result of the same target in the current frame image.

[0095] In the embodiment of the present disclosure, based on the second detection result and the third detection result obtained for the same target in the current frame image, the optimal detection position or state of the same target in the current frame, that is, the best detection result of the same target in the current frame, is determined.

[0096] Step 404: Track the same target in the current frame image based on the best detection result.

[0097] In the disclosed embodiment, the position and state information of the same target is continuously recorded in consecutive video frames to achieve accurate and stable target tracking.

[0098] In summary, in response to the existence of a target in a target image block, a Kalman filter is used to predict a second detection result of the same target in a current frame image based on a first detection result of the same target in a previous frame image; a third detection result of the same target in the current frame image is obtained; based on the second detection result and the third detection result of the same target in the current frame image, the best detection result of the same target in the current frame image is determined; based on the best detection result, the same target in the current frame image is tracked, thereby fully utilizing the prediction ability of the Kalman filter to predict possible detection results of the target in the current frame based on the detection result of the previous frame; further, by combining the prediction result (second detection result) and the actual detection result (third detection result) in the current frame, the best detection result of the same target is determined, thereby improving the accuracy and robustness of target tracking, while improving the real-time performance of target tracking, and effectively dealing with complex scenes such as target occlusion and rapid movement.

[0099] Based on any of the above embodiments, Figure 5 As shown, the target tracking method of the embodiment of the present disclosure is implemented based on the following steps.

[0100] 1. Get the feature map of the current frame image and the previous frame image;

[0101] Using an encoder to extract features from a current frame image and a previous frame image to obtain a first feature map of the current frame image and a feature map of the previous frame image;

[0102] 2. Align and register the feature map of the current frame image with the feature map of the previous frame image;

[0103] In the embodiment of the present disclosure, a frame alignment model is used to align and register the feature map of the current frame image with the feature map of the previous frame image to obtain a registration map corresponding to the current frame image;

[0104] 3. Target detection and tracking

[0105] In the embodiment of the present disclosure, a target detection and tracking model is used to perform target detection on the registration image, where: Figure 6 As shown in Figure 1, the target detection and tracking model includes two cascaded high-resolution networks Hrnet (High-Resolution Network), where the first-level Hrnet outputs candidate target areas, and the second-level Hrnet processes these areas for five specific tasks; the specific structure of the target detection and tracking model can be shown as follows Figure 7 As shown;

[0106] The first level of Hrnet processing includes the following steps:

[0107] (1) Input image: Input the configuration image into the first-level Hrnet network;

[0108] (2) Feature extraction: Utilize the high-resolution characteristics of HrNet to extract multi-scale features in the image; for example, the registration map is downsampled at different sampling rates to obtain downsampled images at different sampling rates. Feature maps of different downsamples are grouped, that is, feature maps of the same size are added together to enhance feature representation, while feature maps of different sizes are subjected to feature pyramid operations (e.g., a small-size feature map is enlarged to the same size as a large-size feature map, or a large-size feature map is reduced to the same size as a small-size feature map) to fuse feature maps of different sizes and capture multi-scale information in the image.

[0109] (3) Object detection: Based on the extracted features, object detection is performed and the top-K (K=4) candidate object regions (detection boxes) with the highest confidence are output;

[0110] (4) Region cropping: Cropping the original image according to the output candidate target regions to obtain image blocks containing these regions (croped regions);

[0111] The second-level Hrnet includes 5 parallel branches, which can be used to predict the target's heat map, size, center offset, tracking offset and distance, as follows:

[0112] (1) Heatmap: Predict a heatmap, which is used to locate the center point of the target;

[0113] (2) Size: Calculate the width (w) and height (h) of the bounding box;

[0114] (3) Center Offset: Calculate the offset of the coordinates of the target center point relative to the specified grid on the feature map;

[0115] (4) Track Offset: Calculate the offset of the target center point between the current frame and the previous frame (or the next frame). This offset is used in the subsequent inter-frame target matching process. Specifically, the tracking target matching of the previous and next frames is based on the predicted distance value between the target center points, as follows:

[0116] First, the predicted coordinates of the target center point in the previous frame are calculated by subtracting the offset of the target center point in the previous frame from the predicted coordinates of the target center point in the current frame. The specific formula is as follows:

[0117]

[0118] in, is the offset of the target center point between the current frame and the previous frame, is the coordinate value of the target center point in the target image block in the current frame image, is the predicted coordinate value of the target center point in the target image block in the previous frame image;

[0119] Secondly, calculate the distance between the predicted coordinates and the actual target center coordinates obtained from the previous frame, which is specifically expressed as the following formula:

[0120]

[0121] Where dist is the difference between the predicted coordinate value and the actual coordinate value of the target center point in the image block in the previous frame image;

[0122] Finally, if the distance is within a threshold, the current frame is considered to predict the same target ID, otherwise it is another new target ID.

[0123] (5) Distance: Calculate the distance between the target and the camera;

[0124] 4. False Detection Model and State Estimation

[0125] In the embodiments of the present disclosure, Figure 8As shown in the figure, by combining deep learning (such as ResNet) and Kalman filter to deal with false positives (FP) results in target detection tasks and perform state estimation; the details are as follows:

[0126] (1) ResNet (residual network) is used as a binary classification model to determine whether the predicted bounding box actually contains the object;

[0127] (2) Kalman filter is used for state estimation;

[0128] In the embodiment of the present disclosure, a Kalman filter is used to estimate the optimal state of a target, for example, to estimate the position and speed of the target, which is specifically expressed as the following formula:

[0129]

[0130] in, represents the optimal state estimate (best detection result) of the target at time t, represents the observed value at time t (the second detection result), K t represents the Kalman gain, represents the observation error, where

[0131]

[0132] in, represents the state of the target at time t-1 (the first detection result), F represents the state transfer matrix, and Bu represents the control input;

[0133]

[0134] Among them, z t represents the observation value at time t, H represents the observation matrix, and Vt represents the observation noise;

[0135]

[0136] P t - =FP t-1 F T +Q t ;

[0137] P t =P t - -K t HP t - =(IK t H)P t - ;

[0138] Among them, P t - represents the covariance matrix of the prior error, P t-1 represents the covariance matrix at time t-1, Q t represents the covariance matrix of the prediction process noise; P t Represents the covariance matrix of the posterior errors.

[0139] In the disclosed embodiment, ResNet is used to reduce false detection, and a Kalman filter is used to perform state estimation (e.g., position, velocity, acceleration, direction) to improve the accuracy and robustness of target detection.

[0140] With the above Figures 1 to 8 Corresponding to the target tracking method provided in the embodiment, the present disclosure also provides a target tracking device. Since the target tracking device provided in the embodiment of the present disclosure corresponds to the target tracking method provided in the above embodiment, the implementation method of the target tracking method is also applicable to the target tracking device provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0141] Fig. 9 It is a schematic diagram of the structure of a target tracking device shown in the fifth embodiment of the present disclosure.

[0142] like Fig. 9 As shown, the target tracking device 900 includes: an acquisition module 910, a registration module 920, a detection module 930 and a tracking module 940.

[0143] Among them, the acquisition module 910 is used to acquire the first feature map of the current frame image; the registration module 920 is used to align and register the first feature map to the feature map of the previous frame image of the current frame image to obtain the registration map corresponding to the current frame image; the detection module 930 is used to perform target detection on the registration map corresponding to the current frame image to obtain the position information of multiple detection frames in the current frame image and the categories to which the targets in the multiple detection frames belong; the tracking module 940 is used to track the target according to the position information of the detection frames of the same category in the current frame image and the previous frame image.

[0144] As a possible implementation method of the embodiment of the present disclosure, the tracking module 940 is used to determine whether the detection frames of the same category contain the same target based on the position information of the detection frames of the same category in the current frame image and the previous frame image; in response to containing the same target, use the target existence classification model to detect the detection frames of the same category in the current frame image to determine whether there is a target in the detection frames of the same category in the current frame image; in response to the existence of the target, track the same target in the current frame image based on the first detection result of the same target in the previous frame image.

[0145] As a possible implementation method of an embodiment of the present disclosure, a tracking module 940 is used to crop the current frame image according to multiple detection frames in the current frame image to obtain multiple image blocks of the current frame image; wherein the image block includes a detection frame; obtain a heat map corresponding to the multiple image blocks; wherein the heat map is used to indicate the target center point in the image block; determine the offset of the target center point in the target image block according to the coordinates of the target center point in the target image block in the current frame image and the previous frame image; wherein the target image block is an image block containing detection frames of the same category; determine whether the detection frames of the same category in the current frame image and the previous frame image contain the same target according to the offset and the coordinates of the target center point in the target image block in the current frame image.

[0146] As a possible implementation method of an embodiment of the present disclosure, the tracking module 940 is used to determine the predicted coordinate value of the target center point in the target image block in the previous frame image based on the offset and the coordinates of the target center point in the target image block in the current frame image; and determine whether the detection boxes of the same category contain the same target based on the difference between the predicted coordinate value and the actual coordinate value of the target center point in the target image block in the previous frame image.

[0147] As a possible implementation method of the embodiment of the present disclosure, the tracking module 940 is used to determine that the detection boxes of the same category contain the same target in response to the absolute value of the difference being less than the set difference threshold; and to determine that the detection boxes of the same category contain non-identical targets in response to the absolute value of the difference being not less than the set difference threshold.

[0148] As a possible implementation method of an embodiment of the present disclosure, a tracking module 940 is used to respond to the presence of a target in the target image block, and to use Kalman filtering to predict a second detection result of the same target in the current frame image based on a first detection result of the same target in the previous frame image; to obtain a third detection result of the same target in the current frame image; wherein the third detection result is obtained by performing target detection on the current frame image using a target detection model; to determine the best detection result of the same target in the current frame image based on the second detection result and the third detection result of the same target in the current frame image; and to track the same target in the current frame image based on the best detection result.

[0149] As a possible implementation method of the embodiment of the present disclosure, the tracking module 940 is used to input the target image block into the target existence classification model in response to containing the same target, so as to obtain the probability value output by the target existence classification model; wherein the probability value is used to indicate the possibility of the existence of the target in the target image block; and determine whether the target exists in the target image block based on the comparison result of the probability value and the set probability threshold.

[0150] As a possible implementation method of the embodiment of the present disclosure, the detection module 930 is used to downsample the registration image according to multiple different sampling rates to obtain multiple downsampled images; fuse the feature maps of the multiple downsampled images to obtain a fused feature map; perform target detection on the fused feature map to obtain the position information of multiple detection boxes in the current frame image and the categories to which the targets in the multiple detection boxes belong.

[0151] As a possible implementation method of the embodiment of the present disclosure, the registration module 920 is used to align the current frame image with the previous frame image based on the homography matrix of any two frame images to obtain an aligned image of the current frame image; wherein the homography matrix is ​​used to indicate the perspective transformation relationship of the current frame image to the previous frame image; calculate the optical flow vector field between the aligned image and the previous frame image; wherein the optical flow vector field is used to indicate the moving direction and moving speed of the pixel points in the aligned image relative to the corresponding pixel points in the previous frame image; based on the optical flow vector field of the aligned image, the aligned image is locally registered to obtain a registration map corresponding to the current frame image.

[0152] The target tracking device of the disclosed embodiment can significantly reduce the target position deviation caused by the difference between video frames by acquiring the first feature map of the current frame image and aligning and registering it with the feature map of the previous frame image, thereby improving the correlation between the target between consecutive frames, making the target tracking more stable and accurate. Furthermore, target detection is performed on the registered image, so that the position information and categories of multiple targets in the current frame can be accurately identified, providing a reliable basis for subsequent tracking, and enabling the system to process multiple targets at the same time, thereby improving the processing power and flexibility of the system. Finally, target tracking is performed according to the position information of the detection boxes of the same category in the current frame and the previous frame, so that continuous and stable tracking of the target can be achieved, thereby improving the robustness and accuracy of target tracking.

[0153] In an exemplary embodiment, an electronic device is also provided.

[0154] Among them, electronic equipment includes:

[0155] processor;

[0156] a memory for storing processor-executable instructions;

[0157] The processor is configured to execute instructions to implement the target tracking method proposed in any of the aforementioned embodiments.

[0158] As an example, Fig.10 is a schematic diagram of the structure of an electronic device 1000 shown in an exemplary embodiment of the present disclosure, such as Fig.10 As shown, the electronic device 1000 may further include:

[0159] The memory 1010 and the processor 1020, a bus 1030 connecting different components (including the memory 1010 and the processor 1020), the memory 1010 stores a computer program, and when the processor 1020 executes the program, the target tracking method described in the embodiment of the present disclosure is implemented.

[0160] Bus 1030 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0161] The electronic device 1000 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 1000, including volatile and non-volatile media, removable and non-removable media.

[0162] The memory 1010 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1040 and / or cache memory 1050. The server 1000 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 1060 may be used to read and write non-removable, non-volatile magnetic media ( Fig.10 not shown, usually called a "hard drive"). Although Fig.10 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 1030 via one or more data medium interfaces. The memory 1010 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0163] A program / utility 1080 having a set (at least one) of program modules 1070 may be stored, for example, in the memory 1010, such program modules 1070 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 1070 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0164] The electronic device 1000 may also communicate with one or more external devices 1090 (e.g., keyboard, pointing device, display 1091, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 1092. In addition, the electronic device 1000 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 1093. As shown, the network adapter 1093 communicates with other modules of the electronic device 1000 through a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0165] The processor 1020 executes various functional applications and data processing by running the programs stored in the memory 1010 .

[0166] It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the target tracking method of the embodiment of the present disclosure, and will not be repeated here.

[0167] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to complete the risk assessment method proposed in any of the above embodiments. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0168] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the target tracking method proposed in any of the above embodiments is implemented.

[0169] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0170] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A target tracking method, characterized in that: include: Obtaining a first feature map of the current frame image; Aligning and registering the first feature map with a feature map of a previous frame image of the current frame image to obtain a registration map corresponding to the current frame image; Performing target detection on the registration map corresponding to the current frame image to obtain position information of multiple detection frames in the current frame image and categories to which targets in the multiple detection frames belong; Target tracking is performed according to position information of detection frames of the same category in the current frame image and the previous frame image.

2. The method according to claim 1, characterized in that The target tracking according to the position information of the detection frames of the same category in the current frame image and the previous frame image includes: Determining whether the detection frames of the same category contain the same target according to the position information of the detection frames of the same category in the current frame image and the previous frame image; In response to containing the same target, using a target existence classification model to detect the detection frame of the same category in the current frame image to determine whether there is a target in the detection frame of the same category in the current frame image; In response to the existence of the target, the same target in the current frame image is tracked based on the first detection result of the same target in the previous frame image.

3. The method according to claim 2, characterized in that The determining, based on the position information of the detection frames of the same category in the current frame image and the previous frame image, whether the detection frames of the same category contain the same target includes: According to the multiple detection frames in the current frame image, the current frame image is cropped to obtain multiple image blocks of the current frame image; wherein the image block includes a detection frame; Obtaining a heat map corresponding to the plurality of image blocks; wherein the heat map is used to indicate a target center point in the image block; Determine the offset of the target center point in the target image block according to the coordinates of the target center point in the target image block in the current frame image and the previous frame image; wherein the target image block is an image block containing a detection frame of the same category; According to the offset and the coordinates of the center point of the target in the target image block in the current frame image, it is determined whether the detection frames of the same category in the current frame image and the previous frame image contain the same target.

4. The method according to claim 3, characterized in that The determining, based on the offset and the coordinates of the center point of the target in the target image block in the current frame image, whether the detection frames of the same category in the current frame image and the previous frame image contain the same target includes: Determine the predicted coordinate value of the target center point in the target image block in the previous frame image according to the offset and the coordinate of the target center point in the target image block in the current frame image; Whether the detection frames of the same category contain the same target is determined based on the difference between the predicted coordinate value and the actual coordinate value of the target center point in the target image block in the previous frame image.

5. The method according to claim 4, characterized in that The determining, based on the difference between the predicted coordinate value and the actual coordinate value of the center point of the target image block in the previous frame image, whether the detection frames of the same category contain the same target includes: In response to an absolute value of the difference being less than a set difference threshold, determining that the detection frames of the same category contain the same object; In response to the absolute value of the difference being not less than the set difference threshold, it is determined that the detection frames of the same category contain non-identical objects.

6. The method according to claim 2, characterized in that In response to the existence of the target in the target image block, tracking the same target in the current frame image based on the first detection result of the same target in the previous frame image includes: In response to the existence of the target in the target image block, using Kalman filtering to predict a second detection result of the same target in the current frame image based on a first detection result of the same target in the previous frame image; Obtaining a third detection result of the same target in the current frame image; wherein the third detection result is obtained by performing target detection on the current frame image using a target detection model; Determining the best detection result of the same target in the current frame image according to the second detection result and the third detection result of the same target in the current frame image; Based on the best detection result, the same target in the current frame image is tracked.

7. The method according to claim 3, characterized in that In response to containing the same target, detecting the target image block using a target existence classification model to determine whether the target image block contains the target, including: In response to containing the same target, inputting the target image block into the target existence classification model to obtain a probability value output by the target existence classification model; wherein the probability value is used to indicate the possibility of the target existing in the target image block; According to the comparison result of the probability value and the set probability threshold, it is determined whether the target image block exists.

8. The method according to claim 1, characterized in that The performing target detection on the registration map corresponding to the current frame image to obtain position information of multiple detection frames in the current frame image and categories to which the targets in the multiple detection frames belong includes: Downsampling the registration image according to a plurality of different sampling rates to obtain a plurality of downsampled images; Fusing the feature maps of the multiple downsampled images to obtain a fused feature map; Target detection is performed on the fused feature map to obtain position information of multiple detection frames in the current frame image and categories to which targets in the multiple detection frames belong.

9. The method according to claim 1, characterized in that: The step of aligning and registering the first feature map with a feature map of a previous frame image of the current frame image to obtain a registration map corresponding to the current frame image includes: Based on the homography matrix of any two frames of images, align the current frame image with the previous frame image to obtain an aligned image of the current frame image; wherein the homography matrix is ​​used to indicate the perspective transformation relationship of the current frame image being warped to the previous frame image; Calculating an optical flow vector field between the aligned image and the previous frame image; wherein the optical flow vector field is used to indicate a moving direction and a moving speed of a pixel point in the aligned image relative to a corresponding pixel point in the previous frame image; Based on the optical flow vector field of the aligned image, the aligned image is locally registered to obtain a registration map corresponding to the current frame image.

10. A target tracking device, characterized in that: include: An acquisition module, used to acquire a first feature map of a current frame image; A registration module, used for aligning and registering the first feature map with the feature map of the previous frame image of the current frame image, so as to obtain a registration map corresponding to the current frame image; A detection module, configured to perform target detection on the registration map corresponding to the current frame image, so as to obtain position information of multiple detection frames in the current frame image and categories to which the targets in the multiple detection frames belong; A tracking module is used to track the target according to the position information of the detection frame of the same category in the current frame image and the previous frame image.

Images