A convolutional network-based unsupervised behavior tracking and recognition method and system

Abstract

The present invention discloses an unsupervised behavior tracking and recognition method and system based on a convolutional network, which relates to the field of visual tracking technology, and includes the following steps: collecting a video sequence, obtaining a template image and a current frame search image from the video sequence, and constructing a training data set; constructing a behavior tracking model based on the SiamFC algorithm; using the training data set to train the behavior tracking model until the preset requirements are met, and a trained behavior tracking model is obtained; using the trained behavior tracking model to track the target object in the monitoring video segment, and obtaining the user behavior tracking and recognition result. The present invention introduces Transformer into the visual tracking technology, effectively determines the importance of different parts in the input sequence through the attention mechanism, solves the global spatiotemporal correlation problem in the sequence data, and realizes efficient tracking tasks.

Description

Technical Field

[0001] The present invention relates to the field of visual tracking technology, and more specifically to an unsupervised behavior tracking and recognition method and system based on a convolutional network. Background Art

[0002] Visual tracking is to detect, extract, identify and track moving targets in image sequences, obtain the motion parameters of moving targets, and then carry out the next step of processing and analysis to understand the behavior of moving targets and complete higher-level detection tasks. With the widespread application of artificial intelligence and computer vision, visual tracking technology has become a hot research frontier. However, due to factors such as the high-speed movement of the target, the interference of the target's environment, and the posture changes of the tracking device itself, complex scenarios such as occlusion, blur and size changes of the target often occur, which in turn affects the tracking effect of the tracker.

[0003] In recent years, trackers based on twin networks have been widely used. These trackers use the excellent feature extraction capabilities of convolutional neural networks and the efficient matching method of cross-correlation operations to balance the accuracy and speed of trackers and achieve certain results. However, existing twin trackers usually only use the initial frame as the target template, fail to fully consider the spatiotemporal information of historical frames, and cannot achieve efficient tracking tasks. This has become a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0004] In view of this, the present invention provides an unsupervised behavior tracking and recognition method and system based on a convolutional network, which solves the problems existing in the background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] An unsupervised behavior tracking and recognition method based on a convolutional network comprises the following steps:

[0007] Collect video sequences and obtain template images and current frame search images from the video sequences to construct a training data set;

[0008] Based on the SiamFC algorithm, a behavior tracking model is constructed;

[0009] The behavior tracking model is trained using the training data set until the preset requirements are met to obtain a trained behavior tracking model;

[0010] The trained behavior tracking model is used to track the target object in the monitored video segment to obtain the user behavior tracking and recognition results.

[0011] Optionally, collecting a video sequence specifically includes the following steps:

[0012] Shoot multiple target objects separately to obtain several video sequences;

[0013] In each video sequence, foreground frames with target objects and background frames without target objects are determined, and the quality of each video sequence is analyzed;

[0014] All video sequences are sorted from high to low based on the quality analysis results, and the top N video sequences with high quality are retained.

[0015] Optionally, analyzing the quality of each video sequence includes the following steps:

[0016] Perform smoothing filtering on each video frame in the video sequence to determine the motion vector field;

[0017] The target area is confirmed based on the motion vector field, the square mean of the pixel difference between the target area in each video frame and the corresponding area in the previous frame is obtained, and the quality of each video frame and the quality of the video sequence are determined.

[0018] Optionally, obtaining a template image and a current frame search image specifically includes the following steps:

[0019] The target object in the first frame of the video sequence is cropped according to the marked position and size to obtain a template image;

[0020] The video sequence is identified frame by frame to determine each frame image contained in the video sequence, and each frame image except the first frame image is defined as a search image.

[0021] Optionally, the behavior tracking model includes: a twin backbone network, an adaptive spatiotemporal information extraction module, a spatiotemporal context mapping module and a classification regression module; the twin backbone network, the adaptive spatiotemporal information extraction module, and the classification regression module are all connected to the spatiotemporal context mapping module;

[0022] The Siamese backbone network is used to extract features from the template image and the search image to generate the reference template and the real-time search feature map;

[0023] An adaptive spatiotemporal information extraction module is used to generate a template image suitable for the current frame based on the tracking information of the previous frame;

[0024] The spatiotemporal context mapping module is used to capture the spatiotemporal context mapping relationship between consecutive frames;

[0025] The classification and regression module is used to generate tracking results through the classification function and the regression function of the bounding box.

[0026] Optionally, the Siamese backbone network adopts a fully convolutional Siamese network SiamFC, which consists of two branches with the same structure and shared weights;

[0027] After the template image and the search image are input into the twin backbone network, the two branches are used to extract the convolution features of the template image and the search image, and the similarity scores of the two convolution features are calculated. The position with the highest similarity score is the target position, and a response map is generated based on the search image centered on the target position of the previous frame.

[0028] Optionally, the adaptive spatiotemporal information extraction module is built based on a multi-head attention mechanism, which cascades the extracted template image features and search image features, performs multi-head attention feature mapping on the cascaded features, obtains features enhanced by weighted mapping of the attention layer, and generates a template image suitable for the current frame.

[0029] Optionally, the spatiotemporal context mapping module is an encoder-decoder structure, in which the encoder encodes the extracted template image features and search image features, and the decoder decodes the encoded features and target query, captures context information, and performs nonlinear transformation and mapping on the features.

[0030] Optionally, the classification regression module is constructed using multi-layer convolution, the classification function uses a binary cross entropy loss function, and the bounding box regression function uses L1 loss and IOU loss function.

[0031] An unsupervised behavior tracking and recognition system based on a convolutional network for implementing the above method comprises:

[0032] A data acquisition module is used to collect video sequences and obtain template images and current frame search images from the video sequences to construct a training data set;

[0033] The model building module is used to build a behavior tracking model through the SiamFC algorithm;

[0034] A model training module is used to train the behavior tracking model by using a training data set until the preset requirements are met to obtain a trained behavior tracking model;

[0035] The tracking and identification module is used to track the target object in the monitoring video segment by using the trained behavior tracking model to obtain the user behavior tracking and identification results.

[0036] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses an unsupervised behavior tracking and recognition method and system based on a convolutional network, obtains a template image and a current frame search image from a video sequence, and constructs a training data set; constructs a behavior tracking model based on the SiamFC algorithm, preprocesses the template image and the search image, and extracts features through a twin network, generates a template more suitable for the current frame according to the tracking information of the previous frame, and determines the search area in the current frame with the highest similarity to the tracking target of the previous frame; fuses and enhances the feature map of the template image and the search area using an encoder-decoder structure to obtain an enhanced feature vector, and captures the spatiotemporal context mapping relationship between consecutive frames; constructs a classification regression module using multi-layer convolution, and generates tracking results through a classification function and a regression function of a bounding box. The behavior tracking model is trained using the training data set until the upper limit of the number of iterations is met or the model converges, and the trained model is used to track the target in the video sequence to achieve efficient tracking tasks, which can avoid the influence of complex scenarios such as target occlusion, blur and size change on the tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of the unsupervised behavior tracking and recognition method based on convolutional network provided by the present invention;

[0039] Figure 2 This is a schematic diagram of the structure of the behavior tracking model provided by the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The embodiment of the present invention discloses an unsupervised behavior tracking and recognition method based on a convolutional network, such as Figure 1 As shown, the following steps are included:

[0042] Collect video sequences and obtain template images and current frame search images from the video sequences to construct a training data set;

[0043] Based on the SiamFC algorithm, a behavior tracking model is constructed;

[0044] The behavior tracking model is trained using the training data set until the preset requirements are met to obtain a trained behavior tracking model;

[0045] The trained behavior tracking model is used to track the target object in the monitored video segment to obtain the user behavior tracking and recognition results.

[0046] A video sequence is a sequence of multiple video images that are connected in time and space and arranged in the order of shooting time. Not every video sequence is suitable for user behavior analysis, and manual quality analysis of video sequences requires a lot of manpower and time. Therefore, in order to improve the efficiency of analyzing video sequences, in this embodiment, collecting video sequences specifically includes the following steps:

[0047] Shoot multiple target objects separately to obtain several video sequences;

[0048] In each video sequence, foreground frames with target objects and background frames without target objects are determined, and the quality of each video sequence is analyzed;

[0049] All video sequences are sorted from high to low based on the quality analysis results, and the top N video sequences with high quality are retained.

[0050] Specifically, for each video frame in the video sequence, the grayscale value corresponding to each pixel of the video frame and the number of pixels contained in the video frame are obtained, the grayscale value corresponding to each pixel of the video frame is added to obtain the corresponding total grayscale value, the quotient of the total grayscale value and the number of pixels contained is taken as the grayscale mean, the video frame corresponding to the largest grayscale mean is taken as the background frame, and the video frame corresponding to the smallest grayscale mean is taken as the foreground frame.

[0051] Furthermore, the quality of each video sequence is analyzed, specifically including the following steps:

[0052] Perform smooth filtering on each video frame in the video sequence to determine the motion vector field; in this embodiment, a two-dimensional Gaussian filtering method can be used for smooth filtering; after the video frame is smooth filtered, the previous frame image is used as a reference frame, and a block matching algorithm is used to predict the motion vector for each pixel in the current frame image, thereby obtaining the motion vector field of the entire frame image;

[0053] The target area is confirmed based on the motion vector field, the square mean of the pixel difference between the target area in each video frame and the corresponding area in the previous frame is obtained, and the quality of each video frame and the quality of the video sequence are determined.

[0054] Further, obtaining the template image and the current frame search image specifically includes the following steps:

[0055] The target object in the first frame of the video sequence is cropped according to the marked position and size to obtain a template image;

[0056] The video sequence is identified frame by frame to determine each frame image contained in the video sequence, and each frame image except the first frame image is defined as a search image.

[0057] In this embodiment, the center point of the target object can be assumed to be at the center of the first frame image, and a region of 127*127 is cropped, and all pixel values ​​in the cropped region are copied to a newly created three-channel RGB image to obtain a template image. Starting from the second frame, the images of subsequent frames are cropped to a size of 255*255 with the target object as the center. In addition, before cropping, the image can be perturbed by randomly adjusting the contrast, flipping, rotating, center point offset, etc. to enhance the robustness of the training.

[0058] Furthermore, if Figure 2 As shown, the behavior tracking model includes: a twin backbone network, an adaptive spatiotemporal information extraction module, a spatiotemporal context mapping module and a classification regression module; the twin backbone network, the adaptive spatiotemporal information extraction module, and the classification regression module are all connected to the spatiotemporal context mapping module;

[0059] The Siamese backbone network is used to extract features from the template image and the search image to generate the reference template and the real-time search feature map;

[0060] An adaptive spatiotemporal information extraction module is used to generate a template image suitable for the current frame based on the tracking information of the previous frame;

[0061] The spatiotemporal context mapping module is used to capture the spatiotemporal context mapping relationship between consecutive frames;

[0062] The classification and regression module is used to generate tracking results through the classification function and the regression function of the bounding box.

[0063] Furthermore, the twin backbone network adopts a fully convolutional twin network SiamFC, which consists of two branches with the same structure and shared weights; after the template image and the search image are input into the twin backbone network, the two branches are used to extract the convolution features of the template image and the search image, and the similarity scores of the two convolution features are calculated. The position with the highest similarity score is the target position, and a response map is generated based on the search image centered on the target position of the previous frame.

[0064] The similarity score is calculated as follows: In the formula, z represents the template image, x represents the search image, It represents the feature extraction corresponding to the embedded deep network, * represents the convolution operation, through which the part of x that is closest to z is extracted; b represents the value of each position in the similarity score map.

[0065] Furthermore, the adaptive spatiotemporal information extraction module is built based on the multi-head attention mechanism, which cascades the extracted template image features and search image features, performs multi-head attention feature mapping on the cascade features, obtains the features enhanced by the weighted mapping of the attention layer, and generates a template image suitable for the current frame.

[0066] Furthermore, the spatiotemporal context mapping module is an encoder-decoder structure, which encodes the extracted template image features and search image features through the encoder, and decodes the encoded features and target query through the decoder, captures context information, performs nonlinear transformation and mapping on the features, and enhances the expressiveness of feature representation. Among them, the target query can focus on each position of the template image and search image features and learn a robust expression.

[0067] Furthermore, the classification and regression module is constructed using multi-layer convolution, the classification function uses the binary cross entropy loss function, and the bounding box regression function uses the L1 loss and IOU loss function.

[0068] and Figure 1 Corresponding to the method described above, the embodiment of the present invention also provides an unsupervised behavior tracking and recognition system based on a convolutional network, for Figure 1 The specific implementation of the method in the embodiment of the present invention is an unsupervised behavior tracking and recognition system based on a convolutional network, which can be applied to a computer terminal or various mobile devices, and specifically includes:

[0069] A data acquisition module is used to collect video sequences and obtain template images and current frame search images from the video sequences to construct a training data set;

[0070] The model building module is used to build a behavior tracking model through the SiamFC algorithm;

[0071] A model training module is used to train the behavior tracking model by using a training data set until the preset requirements are met to obtain a trained behavior tracking model;

[0072] The tracking and identification module is used to track the target object in the monitoring video segment by using the trained behavior tracking model to obtain the user behavior tracking and identification results.

[0073] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0074] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An unsupervised behavior tracking and recognition method based on convolutional network, characterized in that: The following steps are involved: Collect video sequences and obtain template images and current frame search images from the video sequences to construct a training data set; Based on the SiamFC algorithm, a behavior tracking model is constructed; The behavior tracking model is trained using the training data set until the preset requirements are met to obtain a trained behavior tracking model; Use the trained behavior tracking model to track the target object in the monitored video segment to obtain the user behavior tracking and recognition results; The video sequence is collected, including the following steps: Shoot multiple target objects separately to obtain several video sequences; In each video sequence, foreground frames with target objects and background frames without target objects are determined, and the quality of each video sequence is analyzed; All video sequences are sorted from high to low based on the quality analysis results, and the top N video sequences with high quality are retained.

2. The unsupervised behavior tracking and recognition method based on convolutional network according to claim 1 is characterized in that: Analyze the quality of each video sequence, including the following steps: Perform smoothing filtering on each video frame in the video sequence to determine the motion vector field; The target area is confirmed based on the motion vector field, the square mean of the pixel difference between the target area in each video frame and the corresponding area in the previous frame is obtained, and the quality of each video frame and the quality of the video sequence are determined.

3. The unsupervised behavior tracking and recognition method based on convolutional network according to claim 1 is characterized in that: Obtaining the template image and the current frame search image includes the following steps: The target object in the first frame of the video sequence is cropped according to the marked position and size to obtain a template image; The video sequence is identified frame by frame to determine each frame image contained in the video sequence, and each frame image except the first frame image is defined as a search image.

4. The unsupervised behavior tracking and recognition method based on convolutional network according to claim 1 is characterized in that: The behavior tracking model includes: a twin backbone network, an adaptive spatiotemporal information extraction module, a spatiotemporal context mapping module, and a classification and regression module; the twin backbone network, the adaptive spatiotemporal information extraction module, and the classification and regression module are all connected to the spatiotemporal context mapping module; The Siamese backbone network is used to extract features from the template image and the search image to generate the reference template and the real-time search feature map; An adaptive spatiotemporal information extraction module is used to generate a template image suitable for the current frame based on the tracking information of the previous frame; The spatiotemporal context mapping module is used to capture the spatiotemporal context mapping relationship between consecutive frames; The classification and regression module is used to generate tracking results through the classification function and the regression function of the bounding box.

5. The unsupervised behavior tracking and recognition method based on convolutional network according to claim 4 is characterized in that: The twin backbone network uses the fully convolutional twin network SiamFC, which consists of two branches with the same structure and shared weights; After the template image and the search image are input into the twin backbone network, the two branches are used to extract the convolution features of the template image and the search image, and the similarity scores of the two convolution features are calculated. The position with the highest similarity score is the target position, and a response map is generated based on the search image centered on the target position of the previous frame.

6. The unsupervised behavior tracking and recognition method based on convolutional network according to claim 4 is characterized in that: The adaptive spatiotemporal information extraction module is built based on the multi-head attention mechanism. It cascades the extracted template image features and search image features, performs multi-head attention feature mapping on the cascade features, obtains the features enhanced by the attention layer weighted mapping, and generates a template image suitable for the current frame.

7. The unsupervised behavior tracking and recognition method based on convolutional network according to claim 4 is characterized in that: The spatiotemporal context mapping module is an encoder-decoder structure. The encoder encodes the extracted template image features and search image features, and the decoder decodes the encoded features and target query to capture context information and perform nonlinear transformation and mapping on the features.

8. The unsupervised behavior tracking and recognition method based on convolutional network according to claim 4 is characterized in that: The classification and regression module is constructed using multi-layer convolution. The classification function uses the binary cross entropy loss function, and the bounding box regression function uses the L1 loss and IOU loss functions.

9. An unsupervised behavior tracking and recognition system based on a convolutional network that implements the method according to any one of claims 1 to 8, characterized in that: include: A data acquisition module is used to collect video sequences and obtain template images and current frame search images from the video sequences to construct a training data set; The model building module is used to build a behavior tracking model through the SiamFC algorithm; A model training module is used to train the behavior tracking model by using a training data set until the preset requirements are met to obtain a trained behavior tracking model; The tracking and identification module is used to track the target object in the monitoring video segment by using the trained behavior tracking model to obtain the user behavior tracking and identification results.

Images