A deep learning-based automatic matching and target tracking method and system

Abstract

The application discloses a kind of automatic matching and target tracking method and system based on deep learning.The method first obtains target image and video to be processed, and video is extracted video frame set according to preset frame interval interval;Through the feature extraction network in image feature extraction and matching model, the target image and video frame set are carried out feature extraction, and the feature information corresponding to each image is obtained;Then the target image feature information is compared with each video frame feature information by the matching layer in image feature extraction and matching model, and the most similar feature information is determined, and the video frame corresponding to the most similar feature information is regarded as the specific position of target image;Finally, the motion trajectory of the target to be tracked is formed based on the specific position of the video frame corresponding to the most similar feature information in video.The present application realizes the accurate positioning and continuous tracking of target in video.

Description

Technical Field

[0001] This invention relates to the field of computer image processing, and in particular to an automatic matching and target tracking method and system based on deep learning. Background Technology

[0002] Image matching and tracking is an important research direction in computer vision, with many practical applications. In the field of surveillance and security, image matching and tracking technology can track targets within a monitored area in real time, thereby improving the intelligence and accuracy of the monitoring system, strengthening the monitoring of potential threats, and helping to promptly detect and respond to various security issues. Therefore, accurate image matching and target tracking technology is crucial, as it can reduce the possibility of false alarms and missed detections, making security systems more efficient and reliable.

[0003] Traditional image matching methods typically rely on extracting feature points or feature descriptors from images, such as SIFT (Scale Invariant Feature Transform) and SURF (Speed-Up Robust Feature Transform). Then, nearest neighbor algorithms (e.g., k-nearest neighbors) and RANSAC (Random Sample Consensus) are used to process the matching relationships between feature points in the reference and query images. These methods perform well in certain scenarios, but their generalization ability and accuracy are often limited due to the subjectivity of feature selection and parameter tuning.

[0004] In summary, there is an urgent need for a highly accurate and stable automatic image matching and target tracking technology solution. Summary of the Invention

[0005] Based on this, embodiments of this application provide an automatic matching and target tracking method and system based on deep learning. An end-to-end image matching network is designed using convolutional neural network technology. The input is a video and an image of the target to be tracked, and the output is the real-time position of the target in the video, thereby realizing real-time and accurate tracking and monitoring of multiple targets in intelligent security scenarios.

[0006] Firstly, a deep learning-based automatic matching and target tracking method is provided, which includes:

[0007] Acquire a target image and a video to be processed, and extract a set of video frames from the video at preset frame intervals; wherein, the target image contains at least one target to be tracked;

[0008] The feature extraction network in the image feature extraction and matching model is used to extract features from the target image and the video frame set to obtain the feature information corresponding to each image.

[0009] The matching layer in the image feature extraction and matching model compares the feature information of the target image with the feature information of each video frame to determine the most similar feature information, and the video frame corresponding to the most similar feature information is taken as the specific location of the target image.

[0010] The motion trajectory of the target to be tracked is formed based on the specific location of the video frame corresponding to the most similar feature information in the video.

[0011] Optionally, the feature extraction network performs feature extraction on the target image and the video frame set to obtain feature information corresponding to each image. Specifically, the feature extraction network includes a three-block ResNet and a Feature Pyramid Network (FPN), wherein:

[0012] After the original image is input, it is passed through three blocks of ResNet in sequence. These three blocks extract the features of the image in a multi-level and multi-scale manner, reducing the size of the image to 1 / 2, 1 / 4 and 1 / 8 of the original size, and obtaining feature maps C1, C2 and C3 respectively.

[0013] Perform 1x1 convolution on feature maps C1, C2, and C3 respectively to obtain outputs P1, P2, and P3;

[0014] The image features are obtained by upsampling P3 by two times and fusing it with P2, and then upsampling it again by two times and fusing it with P1.

[0015] The fused image features and the original P3 output are used together as the image feature information and input to the matching layer.

[0016] Optionally, the matching layer compares the target image feature information with the feature information of each video frame to determine the most similar feature information, including:

[0017] Similarity features are determined by matching points and corresponding relationships between two images being compared.

[0018] Optionally, the motion trajectory of the target to be tracked is formed based on the specific location of the video frame corresponding to the most similar feature information in the video, including:

[0019] Perform frame-to-frame matching in the video; when performing frame-to-frame matching in the video, continue to track the target's position if it is below the inter-frame constraint threshold; if it is above the threshold, then re-perform the matching between the video frame and the target image.

[0020] Optionally, the method further includes:

[0021] The image feature extraction and matching model is trained using a publicly available matching dataset. Training parameters are set and the model is trained iteratively until the loss function converges, resulting in a fully trained image feature extraction and matching model. The training parameters include at least the learning rate, training batch size, maximum number of iterations, and optimizer.

[0022] The trained image feature extraction and matching model is then lightweighted according to preset requirements, and the output head is deployed; wherein, the lightweighting process includes at least distillation and pruning.

[0023] Secondly, a deep learning-based automatic matching and target tracking system is provided, which includes:

[0024] The input module is used to acquire a target image and a video to be processed, and to extract a set of video frames from the video at preset frame intervals; wherein the target image contains at least one target to be tracked;

[0025] The feature extraction module is used to extract features from the target image and the video frame set through the feature extraction network in the image feature extraction and matching model to obtain feature information corresponding to each image;

[0026] The matching module is used to compare the feature information of the target image with the feature information of each video frame through the matching layer in the image feature extraction and matching model to determine the most similar feature information, and to take the video frame corresponding to the most similar feature information as the specific location of the target image.

[0027] The output module is used to form the motion trajectory of the target to be tracked based on the specific location of the video frame corresponding to the most similar feature information in the video.

[0028] Optionally, the feature extraction network performs feature extraction on the target image and the video frame set to obtain feature information corresponding to each image. Specifically, the feature extraction network includes a three-block ResNet and a Feature Pyramid Network (FPN), wherein:

[0029] After the original image is input, it is passed through three blocks of ResNet in sequence. These three blocks extract the features of the image in a multi-level and multi-scale manner, reducing the size of the image to 1 / 2, 1 / 4 and 1 / 8 of the original size, and obtaining feature maps C1, C2 and C3 respectively.

[0030] Perform 1x1 convolution on feature maps C1, C2, and C3 respectively to obtain outputs P1, P2, and P3;

[0031] The image features are obtained by upsampling P3 by two times and fusing it with P2, and then upsampling it again by two times and fusing it with P1.

[0032] The fused image features and the original P3 output are used together as the image feature information and input to the matching layer.

[0033] Optionally, the matching layer compares the target image feature information with the feature information of each video frame to determine the most similar feature information, including:

[0034] Similarity features are determined by matching points and corresponding relationships between two images being compared.

[0035] Optionally, the motion trajectory of the target to be tracked is formed based on the specific location of the video frame corresponding to the most similar feature information in the video, including:

[0036] Perform frame-to-frame matching in the video; when performing frame-to-frame matching in the video, continue to track the target's position if it is below the inter-frame constraint threshold; if it is above the threshold, then re-perform the matching between the video frame and the target image.

[0037] Optionally, the system further includes:

[0038] The training module is used to train the image feature extraction and matching model using a publicly available matching dataset, and to set training parameters to iteratively train the model until the loss function converges, thus obtaining the trained image feature extraction and matching model; the training parameters include at least the learning rate, training batch size, maximum number of iterations, and optimizer.

[0039] The deployment module is used to perform model lightweighting processing on the trained image feature extraction and matching model according to preset requirements, and to deploy the output head; wherein, the model lightweighting processing includes at least distillation processing and pruning processing.

[0040] The technical solution provided in this application first acquires the target image and the video to be processed, and extracts a set of video frames from the video according to a preset frame interval; then, the feature extraction network in the image feature extraction and matching model extracts features from the target image and the set of video frames to obtain feature information corresponding to each image; next, the matching layer in the image feature extraction and matching model compares the feature information of the target image with the feature information of each video frame to determine the most similar feature information, and uses the video frame corresponding to the most similar feature information as the specific location of the target image; finally, the motion trajectory of the target to be tracked is formed based on the specific location of the video frame corresponding to the most similar feature information in the video. It can be seen that the beneficial effect of this invention is that the position information of the target to be tracked can be tracked by the system in real time, and accurate positioning and continuous tracking of the target in the video are achieved. Attached Figure Description

[0041] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0042] Figure 1 A flowchart illustrating the steps of an automatic matching and target tracking method based on deep learning, provided for an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the feature extraction and matching network provided in the embodiments of this application;

[0044] Figure 3 A flowchart illustrating the design of the target matching and tracking model provided in this application embodiment;

[0045] Figure 4 This is a schematic diagram of a deep learning-based image matching and target tracking system provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] In the description of this invention, the terms "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations of the inventive concept.

[0048] Traditional image matching algorithms typically rely on complex feature engineering and optimization strategies, such as feature extraction, distance metrics, and weighting functions, to model the similarity or matching relationship between two or more objects. This approach suffers from low robustness, making it difficult to adapt to complex application scenarios, and poor generalization ability, with the algorithm results heavily influenced by parameter adjustments. Therefore, this patent proposes a deep learning-based automatic image matching and target tracking method and system. It designs an end-to-end image matching network using convolutional neural network technology, taking a video and the target image to be tracked as input, and outputting the real-time position of the target in the video. This enables real-time and accurate tracking and monitoring of multiple targets in intelligent security scenarios.

[0049] For details, please refer to Figure 1 The diagram illustrates a flowchart of a deep learning-based automatic matching and target tracking method provided in an embodiment of this application. The method may include the following steps:

[0050] S1: Acquire the target image and the video to be processed, and extract the video frame set according to the preset frame interval.

[0051] The target image contains the target to be tracked. This step primarily implements input processing: first, it accepts two inputs: the target image to be tracked and the video to be processed. In the video, the system extracts one frame every n frames to prepare for subsequent processing.

[0052] S2 extracts features from the target image and video frame set through the feature extraction network in the image feature extraction and matching model to obtain the feature information corresponding to each image.

[0053] In this step, video frames and target images are processed by the feature extraction module, which is specifically responsible for capturing key features in the image.

[0054] S3 uses the matching layer in the image feature extraction and matching model to compare the feature information of the target image with the feature information of each video frame to determine the most similar feature information, and uses the video frame corresponding to the most similar feature information as the specific location of the target image.

[0055] In this step, feature matching is specifically implemented. The feature information of the two images is passed to the matching layer. The task of this layer is to compare the features in the target image with the features extracted from the video frame, so as to find the most similar part.

[0056] In the embodiments of this application, such as Figure 2 A schematic diagram of the feature extraction and matching network is given.

[0057] The feature extraction network in this application consists of a three-block ResNet and a feature pyramid network FPN. The specific process is described as follows:

[0058] (1) After the original image is input, it passes through three blocks of ResNet in sequence. These three blocks extract the features of the image in multiple levels and at multiple scales, reduce the size of the image to 1 / 2, 1 / 4 and 1 / 8 of the original size, and obtain feature maps C1, C2 and C3 respectively.

[0059] (2) Perform 1x1 convolution on feature maps C1, C2, and C3 respectively to obtain outputs P1, P2, and P3;

[0060] (3) P3 is upsampled twice and fused with P2, and then upsampled twice again and fused with P1 to obtain image features. This series of operations effectively integrates the image features, forming a richer and more representative feature representation.

[0061] (4) The fused image features and the original P3 output are used as the image feature information and input to the matching layer;

[0062] (5) After the target image and the extracted video frame are subjected to feature extraction in the above steps, the matching layer can obtain the matching points and matching correspondence between the two images.

[0063] Through this process, the system can efficiently extract image features and automatically match the target image with video frames at the matching layer.

[0064] S4. The motion trajectory of the target to be tracked is formed based on the specific position of the video frame corresponding to the most similar feature information in the video.

[0065] In the embodiments of this application, such as Figure 3 The diagram shown illustrates the design of the target matching and tracking model. Specifically:

[0066] (1) After the video extraction frames and the target image to be tracked are input into the feature extraction and matching network, it is necessary to determine whether the matching is successful;

[0067] (2) Assuming that the video frame and the target image are successfully matched, the target image is located in the video according to the correspondence between the two images obtained by the matching network; if the match is not successful, the frame is extracted from the video again for matching until the match is successful.

[0068] (3) After the target is successfully located, the matching between video frames begins. The reason for doing this is that the matching between video frames can obtain more similar information, thereby improving the accuracy of target tracking.

[0069] (4) When performing inter-frame matching in the video, there are certain inter-frame constraints. If the constraint is less than the inter-frame constraint threshold, the target position tracking continues. If the constraint is greater than the threshold (such as when the video field of view changes), the matching between the video frame and the target image is performed again.

[0070] (5) Model training strategy: The model is trained using a public matching dataset. Training parameters (learning rate, training batch size, maximum number of iterations, optimizer, etc.) are set and the network model is trained iteratively until the loss function converges, thus obtaining the image feature extraction and matching model.

[0071] In optional embodiments of this application, the method further includes automatic matching and target tracking model deployment:

[0072] Model deployment is completed according to system requirements. In the automatic matching and target tracking system designed in this patent, only the output head of the model needs to be deployed into the system. Based on the system's hardware limitations and real-time requirements, appropriate model lightweighting processing (distillation, pruning, etc.) is performed to complete the model deployment and realize automatic video target matching and tracking based on deep learning.

[0073] like Figure 4 This application provides a schematic diagram of an image matching and target tracking system based on deep learning. This application also provides an automatic matching and target tracking system based on deep learning. The system includes:

[0074] The input module is used to acquire the target image and the video to be processed, and extract the video frame set according to the preset frame interval; wherein, the target image contains the target to be tracked;

[0075] The feature extraction module is used to extract features from the target image and video frame set through the feature extraction network in the image feature extraction and matching model, so as to obtain the feature information corresponding to each image.

[0076] The matching module is used to compare the feature information of the target image with the feature information of each video frame through the matching layer in the image feature extraction and matching model to determine the most similar feature information, and to take the video frame corresponding to the most similar feature information as the specific location of the target image.

[0077] The output module is used to form the motion trajectory of the target to be tracked based on the specific position of the video frame corresponding to the most similar feature information in the video.

[0078] In an optional embodiment of this application, the feature extraction network performs feature extraction on the target image and video frame set to obtain feature information corresponding to each image. Specifically, the feature extraction network includes a three-block ResNet and a feature pyramid network FPN, wherein:

[0079] After the original image is input, it is passed through three blocks of ResNet in sequence. These three blocks extract the features of the image in a multi-level and multi-scale manner, reducing the size of the image to 1 / 2, 1 / 4 and 1 / 8 of the original size, and obtaining feature maps C1, C2 and C3 respectively.

[0080] Perform 1x1 convolution on feature maps C1, C2, and C3 respectively to obtain outputs P1, P2, and P3;

[0081] The image features are obtained by upsampling P3 by two times and fusing it with P2, and then upsampling it again by two times and fusing it with P1.

[0082] The fused image features and the original P3 output are used together as the image feature information and input to the matching layer.

[0083] In optional embodiments of this application, the matching layer compares the target image feature information with the feature information of each video frame to determine the most similar feature information, including:

[0084] Similarity features are determined by matching points and corresponding relationships between two images being compared.

[0085] In optional embodiments of this application, forming the motion trajectory of the target to be tracked based on the specific location of the video frame corresponding to the most similar feature information in the video includes:

[0086] Perform frame-to-frame matching in the video; when performing frame-to-frame matching in the video, continue to track the target's position if it is below the inter-frame constraint threshold; if it is above the threshold, then re-perform the matching between the video frame and the target image.

[0087] In optional embodiments of this application, the system further includes:

[0088] The training module is used to train the image feature extraction and matching model using a publicly available matching dataset. It sets training parameters and iteratively trains the model until the loss function converges, resulting in a trained image feature extraction and matching model. The training parameters include at least the learning rate, training batch size, maximum number of iterations, and optimizer.

[0089] The deployment module is used to perform lightweight processing on the trained image feature extraction and matching model according to preset requirements, and deploy the output head; wherein, the lightweight processing includes at least distillation processing and pruning processing.

[0090] The deep learning-based automatic matching and target tracking system provided in this application embodiment is used to implement the aforementioned deep learning-based automatic matching and target tracking method. Specific limitations regarding the deep learning-based automatic matching and target tracking system can be found in the above-described limitations of the deep learning-based automatic matching and target tracking method, and will not be repeated here. Each component of the aforementioned deep learning-based automatic matching and target tracking system can be implemented wholly or partially through software, hardware, or a combination thereof. The aforementioned modules can be embedded in or independent of the processor in the device in hardware form, or stored in the memory of the device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A deep learning-based automatic matching and target tracking method, characterized in that, The method includes: Acquire a target image and a video to be processed, and extract a set of video frames from the video at preset frame intervals; wherein, the target image contains at least one target to be tracked; The feature extraction network in the image feature extraction and matching model is used to extract features from the target image and the video frame set to obtain the feature information corresponding to each image. The matching layer in the image feature extraction and matching model compares the feature information of the target image with the feature information of each video frame to determine the most similar feature information, and the video frame corresponding to the most similar feature information is taken as the specific location of the target image. The motion trajectory of the target to be tracked is formed based on the specific location of the video frame corresponding to the most similar feature information in the video.

2. The method according to claim 1, characterized in that, The feature extraction network extracts features from the target image and the video frame set to obtain feature information corresponding to each image. Specifically, the feature extraction network includes a three-block ResNet and a Feature Pyramid Network (FPN), wherein: After the original image is input, it is passed through three blocks of ResNet in sequence. These three blocks extract the features of the image in a multi-level and multi-scale manner, reducing the size of the image to 1 / 2, 1 / 4 and 1 / 8 of the original size, and obtaining feature maps C1, C2 and C3 respectively. Perform 1x1 convolution on feature maps C1, C2, and C3 respectively to obtain outputs P1, P2, and P3; The image features are obtained by upsampling P3 by two times and fusing it with P2, and then upsampling it again by two times and fusing it with P1. The fused image features and the original P3 output are used together as the image feature information and input to the matching layer.

3. The method according to claim 1, characterized in that, The matching layer compares the feature information of the target image with the feature information of each video frame to determine the most similar feature information, including: Similarity features are determined by matching points and corresponding relationships between two images being compared.

4. The method according to claim 1, characterized in that, The motion trajectory of the target to be tracked is formed based on the specific location of the video frame corresponding to the most similar feature information in the video, including: Perform frame-to-frame matching in the video; when performing frame-to-frame matching in the video, continue to track the target's position if it is below the inter-frame constraint threshold; if it is above the threshold, then re-perform the matching between the video frame and the target image.

5. The method according to claim 1, characterized in that, The method further includes: The image feature extraction and matching model is trained using a publicly available matching dataset. Training parameters are set and the model is trained iteratively until the loss function converges, resulting in a fully trained image feature extraction and matching model. The training parameters include at least the learning rate, training batch size, maximum number of iterations, and optimizer. The trained image feature extraction and matching model is then lightweighted according to preset requirements, and the output head is deployed; wherein, the lightweighting process includes at least distillation and pruning.

6. A deep learning-based automatic matching and target tracking system, characterized in that, The system includes: The input module is used to acquire a target image and a video to be processed, and to extract a set of video frames from the video at preset frame intervals; wherein the target image contains at least one target to be tracked; The feature extraction module is used to extract features from the target image and the video frame set through the feature extraction network in the image feature extraction and matching model to obtain feature information corresponding to each image; The matching module is used to compare the feature information of the target image with the feature information of each video frame through the matching layer in the image feature extraction and matching model to determine the most similar feature information, and to take the video frame corresponding to the most similar feature information as the specific location of the target image. The output module is used to form the motion trajectory of the target to be tracked based on the specific location of the video frame corresponding to the most similar feature information in the video.

7. The system according to claim 6, characterized in that, The feature extraction network extracts features from the target image and the video frame set to obtain feature information corresponding to each image. Specifically, the feature extraction network includes a three-block ResNet and a Feature Pyramid Network (FPN), wherein: After the original image is input, it is passed through three blocks of ResNet in sequence. These three blocks extract the features of the image in a multi-level and multi-scale manner, reducing the size of the image to 1 / 2, 1 / 4 and 1 / 8 of the original size, and obtaining feature maps C1, C2 and C3 respectively. Perform 1x1 convolution on feature maps C1, C2, and C3 respectively to obtain outputs P1, P2, and P3; The image features are obtained by upsampling P3 by two times and fusing it with P2, and then upsampling it again by two times and fusing it with P1. The fused image features and the original P3 output are used together as the image feature information and input to the matching layer.

8. The system according to claim 6, characterized in that, The matching layer compares the feature information of the target image with the feature information of each video frame to determine the most similar feature information, including: Similarity features are determined by matching points and corresponding relationships between two images being compared.

9. The system according to claim 6, characterized in that, The motion trajectory of the target to be tracked is formed based on the specific location of the video frame corresponding to the most similar feature information in the video, including: Perform frame-to-frame matching in the video; when performing frame-to-frame matching in the video, continue to track the target's position if it is below the inter-frame constraint threshold; if it is above the threshold, then re-perform the matching between the video frame and the target image.

10. The system according to claim 6, characterized in that, The system also includes: The training module is used to train the image feature extraction and matching model using a publicly available matching dataset, and to set training parameters to iteratively train the model until the loss function converges, thus obtaining the trained image feature extraction and matching model; the training parameters include at least the learning rate, training batch size, maximum number of iterations, and optimizer. The deployment module is used to perform model lightweighting processing on the trained image feature extraction and matching model according to preset requirements, and to deploy the output head; wherein, the model lightweighting processing includes at least distillation processing and pruning processing.

Images