A single sample learning video target tracking method

Abstract

The application discloses a kind of single sample learning video target tracking methods, video image target object semantic segmentation dataset construction and image semantic segmentation model definition, construct image frame sequence perception feature extraction module, the construction of single sample learning information extraction module, the construction of segmentation tracking module, the segmentation inference of subsequent frame image, the optimal segmentation encoding of this module will be output by segmentation tracking module as input, and the final output of segmentation decoder is multi-channel target semantic segmentation result.The single sample learning video target tracking method of the application is combined with the pre-trained vectorization representation, a model parameter learning specific to target object is designed in the inference process, better generalizes target object appearance information, realizes the optimal dynamic target tracking of existing, so as to realize the optimal autonomous action planning of intelligent robot.

Description

Technical Field

[0001] This invention relates to the fields of intelligent manufacturing and machine vision, and in particular to a video target tracking method based on single-sample learning. Background Technology

[0002] Single-sample learning-based video target tracking is a method for dynamic target tracking. In the intelligent manufacturing production scenario of digital factories, intelligent robots need to dynamically track the moving target objects to be manipulated during the execution of given production tasks and the realization of autonomous motion planning, so as to achieve real-time and precise control downstream. Therefore, target object tracking based on video streams is one of the key technologies for effectively realizing this application. The implementation of video target tracking can be achieved by performing semantic segmentation of the target objects in the image frame by frame in the video sequence, thereby achieving the separation of the foreground and background of the target objects and realizing the autonomous motion planning of intelligent robots. Existing methods include online learning trackers and offline learning trackers. Specific algorithms include target trackers based on support vector machines, offline model training trackers or online model training trackers based on convolutional neural networks, target trackers based on long short-term memory and convolutional neural networks, etc. With the continuous development of technology, people's requirements for single-sample learning-based video target tracking methods are also getting higher and higher.

[0003] Existing single-sample learning methods for video object tracking have certain drawbacks. Current work has adapted semantic segmentation networks to video object tracking tasks through online fine-tuning. However, this method is prone to overfitting to the appearance of the target object set in the first frame and high latency due to excessive computational complexity. Subsequent methods have integrated the appearance model specific to the target object into the segmentation model, improving the runtime and achieving end-to-end learning. However, feature matching techniques based on feature embedding are often difficult to effectively implement through deep learning in order to generalize to new target objects. Therefore, we propose a single-sample learning method for video object tracking. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides a single-sample learning video target tracking method. This method not only combines pre-trained vectorized representations but also designs target object-specific model parameter learning during the inference process, thereby better generalizing the target object's appearance information and achieving the best dynamic target tracking available. This enables optimal autonomous action planning for intelligent robots and effectively solves the problems in the background technology.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a video target tracking method based on single-sample learning, comprising the following steps:

[0008] S1: Model Construction: Construction of video image target object semantic segmentation dataset and definition of image semantic segmentation model;

[0009] S2: Module Construction: Construct an image frame sequence perceptual feature extraction module;

[0010] S3: Sample Module Construction: Single Sample Learning Information Extraction Module The structure;

[0011] S4: Tracking Module Construction: Segmented Tracking Module The structure;

[0012] S5: Image segmentation inference: Information extraction module based on single-sample learning of the first frame image. and segmentation tracking module Segmentation inference for subsequent frame images;

[0013] S6: Output: Multi-channel target segmentation unit The construction of this module will segment the tracking module. The optimal segmentation code is used as input to the segmentation decoder. The final output is the multi-channel target semantic segmentation result.

[0014] As a preferred technical solution of this application, step S1 specifically includes the following operational steps:

[0015] A1: Construct a semi-supervised image segmentation dataset;

[0016] A2: In the image segmentation dataset of video target tracking time series, the target object is defined only by the labels of the reference target foreground and background segmentation annotations given in the first frame. For a specific video sequence, only the semantic segmentation annotation labels of its first frame image are given, and then the segmentation target inference work needs to be performed in each subsequent frame.

[0017] A3: Based on the characteristics of the above dataset, we define the video object segmentation framework as follows: in These represent learnable parameters, acquired through learning during model training.

[0018] A4: Video target tracking is implemented as an end-to-end network using video image segmentation methods.

[0019] As a preferred technical solution of this application, step S2 specifically includes the following operational steps:

[0020] B1: The image frame sequence sensing feature extraction module includes a pixel association information aggregation unit. Pixel classifier With attention mechanism unit

[0021] B2: Construct a pixel classifier Pixel classifier The input is the single-sample ground truth label Yseg1. By encoding the input target ground truth label Yseg1, the other input image segmentation ground truth labels of the single-sample learning information extraction module are predicted.

[0022] B3: By employing inference methods within the video sequence, the goal is to automatically label the new image in the next frame, thereby achieving feature segmentation label pair (x) that includes additional frames. t Yseg t This expands the small sample learning dataset.

[0023] As a preferred technical solution of this application, step S3 specifically includes the following operational steps:

[0024] C1: Solve by applying the steepest descent iteration to achieve a trade-off between accuracy and efficiency;

[0025] C2: All computations are implemented using standard neural network operations, starting from a given initialization. Start executing N iterations of steepest descent. Since the convergence of steepest descent is fast and efficient, setting the number of iterations N=5 during training and inference will achieve the expected convergence effect.

[0026] C3: In this new optimization, setting the number of iterations to N=2 can achieve a very ideal update effect, while minimizing the amount of computation and enabling real-time processing;

[0027] C4: Constructs a single-sample learning information extraction module. This can then be applied to the subsequent test frame sequences input in time sequence, combined with the real-time data from the previous step. The updated values ​​are applied to downstream processing stages as part of the predictive segmentation and tracking module. The segmentation code used to obtain the target for video tracking is finally provided as input to the segmentation decoder.

[0028] As a preferred technical solution of this application, step S4 specifically includes the following operational steps:

[0029] D1: This segmentation tracking module can perform real-time lightweight computation on the depth feature map, and can predict the rich encoded information of the target segmentation;

[0030] D2: The model parameters in the segmentation and tracking module are learned using the single-sample information extraction module. By segmentation tracking module Output and generated true value labeling The result is obtained by minimizing the squared error between the two values, calculated by the attention mechanism unit. Provided per-pixel importance weights Weighted;

[0031] D3: Segmentation Tracking Module Implemented as a convolutional filter with a kernel size of K=5, the first hidden layer in the middle uses dilated convolution with a dilation rate D. r =2, which allows for the extraction of a wider range of pixel mutual information with minimal computational effort.

[0032] As a preferred technical solution of this application, step S5 specifically includes the following operational steps:

[0033] E1: Segmentation and tracking module parameters based on the first frame of the video sequence. It is obtained by calculation based on the initial input image Ima1 and the given ground truth label Yseg1;

[0034] E2: The image annotations "(x1,Yseg1) pairs" of the video sequence constitute training samples for learning to segment a given target;

[0035] E3: Based on the previous step of predicting the segmentation tracking module parameters by directly minimizing the segmentation error in the first frame, robust segmentation prediction for upcoming frames can be ensured.

[0036] E4: Use the true value of the first frame to label Yseg1 (i.e., the corresponding...). In in ) as labels in our single-sample learning information extraction module;

[0037] E5: Generated by encoding the true value label Yseg1. Multidimensional labels, thus allowing segmentation tracking modules Predict richer target segmentation representations in test frames;

[0038] E6: To achieve optimal learning and fastest convergence of the single-sample learning information extraction module, and to realize the real-time optimal segmentation encoding output of the segmentation tracking module.

[0039] As a preferred technical solution of this application, step S6 specifically includes the following operational steps:

[0040] F1: Multi-channel target segmentation unit The input consists of three information streams: the first is the real-time optimal segmentation encoding output from the segmentation and tracking module; the second is the single-sample learning information extraction module. Output Thirdly, it is the output of the image frame sequence sensing feature extraction module;

[0041] F2: First, construct the memory bank. The input information for the memory bank comes from two branches: one is the single-sample learning information extraction module. Output Secondly, it is the output of the image frame sequence perception feature extraction module;

[0042] F3: The semantic memory stores semantic information by setting dynamic parameters for region semantic comparison and pixel semantic aggregation. Depend on Composed of target features;

[0043] F4: The input information stream of the target segmentation and decoding module includes two branches: one is the real-time optimal segmentation encoding output of the segmentation tracking module; the other is the embedded representation output of the memory bank.

[0044] F5: The target segmentation and decoding module retrieves data from memory. All categorical features derived from the model are represented as real tensors.

[0045] F6: Further in Capturing the global context of consecutive frame sequences between images can enrich the representability of semantic understanding;

[0046] F7: The memory bank retains compressed global feature representations during the inference phase, and outputs video target tracking results after feature information fusion.

[0047] As a preferred technical solution of this application, steps S1-S6 combine pre-trained vectorized representations and design target object-specific model parameter learning during inference to generalize the appearance information of the target object and achieve the best dynamic target tracking available.

[0048] (III) Beneficial Effects

[0049] Compared with existing technologies, this invention provides a single-sample learning method for video target tracking, which has the following beneficial effects: This single-sample learning method not only combines pre-trained vectorized representations but also designs target-object-specific model parameter learning during inference, better generalizing the appearance information of the target object and achieving the best dynamic target tracking currently available. This enables optimal autonomous motion planning for intelligent robots, and this invention solves the problem of single-sample learning in video target tracking. Video target tracking refers to the process of locating moving target objects in a video time series, including single-target and multi-target tracking. Problems involved include occlusion and motion blur, viewpoint and scale changes, background and lighting changes, and the target object moving out of the field of view during tracking, etc. Few-shot learning is a machine learning method that uses a small amount of labeled data combined with a large amount of unlabeled data for classification or regression. Compared to mainstream deep learning methods, which require a large amount of labeled datasets to effectively train the model and achieve acceptable generalization performance, few-shot learning effectively solves the problem of scarce labeled datasets in real-world production scenarios. It significantly reduces the human and time costs of data labeling and significantly improves the robustness and generalization performance of the algorithm system. This invention raises the requirement for labeled data in few-shot learning to a single-sample scale. That is, in the process of target tracking in a series of time-series video frames, only the target object in the first frame of the entire video needs to be labeled to effectively train and infer the algorithm model. This invention uses single-sample learning, which is one of the most advanced methods in few-shot learning, and uses a video image segmentation algorithm to achieve video target tracking. Compared to other existing single-sample learning methods, our method extracts richer target information during the training phase, effectively memorizing key correlation information between consecutive frames of video sequences and contextual semantic information between near and far pixels, thus improving the accuracy of multi-target construction. Furthermore, our single-sample learning method has significantly lower computation time during the model inference phase than other existing models, improving the real-time performance of the algorithm's inference.

[0050] 1. This method successfully achieves video target tracking using single-sample learning, outperforming other existing similar methods. (Segmentation and tracking module) The system learns to predict initial ground truth labels for target object segmentation from the first frame. These ground truth labels are then generated by the network. Through refinement and detail, the network constructed in this invention possesses a powerful ability to learn segmentation priors. Furthermore... It is not limited to operating on approximate target labels to perform conditional segmentation of target objects. Yseg1, as a label in the single-sample learning information extraction module, introduces a trainable pixel classifier. By employing automated machine learning through its internal single-sample learning information extraction module—rather than simply using the ground truth values ​​from the first frame for annotation—our segmentation and tracking module can predict multi-channel segmentation, thus providing powerful associated target perception information. The segmentation prediction is more accurate.

[0051] 2. STM implements video object segmentation using a spatiotemporal memory network, offering a novel solution for semi-supervised video object segmentation. Depending on the nature of the problem, video frames with object masks can be transformed into richer intermediate predictions. FEELVOS proposes a fast end-to-end embedding learning method for video object segmentation. PreMVOS first generates a set of accurate object segmentation mask proposals for each video frame, then selects and merges these proposals in a designed manner into accurate and temporally consistent pixel object tracking over the video sequence, specifically addressing the problem of segmenting multiple objects in a video sequence. FRTM proposes a novel VOS architecture consisting of two network components. The target appearance model is composed of a lightweight module that learns during the inference phase, using fast optimization techniques to predict coarse but robust target segmentation. SiamRCNN, combined with a dynamic programming-based algorithm, models the complete history of the object to be tracked and potential interfering objects by re-detecting the template of the first frame and the prediction of the previous frame. AGSSVOS segments multiple objects in a single feedforward path using both instance-independent and instance-specific modules. Information from both modules is guided by attention to a fusion decoder to segment all object instances simultaneously along a single path. Our method achieves state-of-the-art performance compared to other similar methods, outperforming the average performance of existing state-of-the-art methods. The score improved by 2.9 points, at the boundary. The score improved by 2.5 points. The performance of our new method is 3.4% higher than that of the optimal method STM, and the experimental results confirm that our new method achieves the best performance in video target tracking algorithms.

[0052] 3. During the algorithm inference process, given a test sequence, along with the annotation of the first frame, an initial training set is first created for the single-sample learning information extraction module, including single-sample pairs. Feature maps are extracted from the first frame. Then, the single-sample learning information extraction module predicts the parameters of the segmentation and tracking module through iterative learning. The initial estimate of the segmentation tracking module is set to all zeros to simplify computational complexity and improve system real-time performance. Then the learned model... Apply to subsequent test frames Ima2 to obtain annotation label encoding To adapt to changing scenes, we further update our segmentation and tracking module using frame information from the processed data, based on a memory-based approach. We ensure memory updates by removing outdated samples. Retaining the first frame, if a video sequence contains multiple targets, we process each target independently and in parallel, utilizing the re-detection template from the first frame and predictive modeling from the previous frame to model the complete tracking history of both objects and potential interfering objects. Our method achieves optimal tracking decisions and is optimized for re-detection and tracking of target objects after prolonged occlusion. Experiments demonstrate that the algorithm achieves optimal performance.

[0053] The single-sample learning video target tracking method is simple, easy to operate, and performs better than traditional methods. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the overall algorithm flow of a single-sample learning video target tracking method according to the present invention. Detailed Implementation

[0055] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. However, those skilled in the art will understand that the embodiments described below are some embodiments of the present invention, but not all embodiments, and are only used to illustrate the present invention, and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Where specific conditions are not specified in the embodiments, conventional conditions or conditions recommended by the manufacturer shall be followed. Where the manufacturer of the instruments used is not specified, they are all conventional products that can be purchased commercially.

[0056] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0057] like Figure 1 As shown, a single-sample learning video target tracking method includes the following steps:

[0058] S1: Model Construction: Construction of video image target object semantic segmentation dataset and definition of image semantic segmentation model;

[0059] S2: Module Construction: Construct an image frame sequence perceptual feature extraction module;

[0060] S3: Sample Module Construction: Single Sample Learning Information Extraction Module The structure;

[0061] S4: Tracking Module Construction: Segmented Tracking Module The structure;

[0062] S5: Image segmentation inference: Information extraction module based on single-sample learning of the first frame image. and segmentation tracking module Segmentation inference for subsequent frame images;

[0063] S6: Output: Multi-channel target segmentation unit The construction of this module will segment the tracking module. The optimal segmentation code is used as input to the segmentation decoder. The final output is the multi-channel target semantic segmentation result.

[0064] Furthermore, step S1 specifically includes the following operational steps:

[0065] A1: Construct a semi-supervised image segmentation dataset;

[0066] A2: In the image segmentation dataset of video target tracking time series, the target object is defined only by the labels of the reference target foreground and background segmentation annotations given in the first frame. For a specific video sequence, only the semantic segmentation annotation labels of its first frame image are given, and then the segmentation target inference work needs to be performed in each subsequent frame.

[0067] A3: Based on the characteristics of the above dataset, we define the video object segmentation framework as follows: in These represent learnable parameters, acquired through learning during model training.

[0068] A4: Video target tracking is implemented as an end-to-end network using video image segmentation methods.

[0069] Furthermore, step S2 specifically includes the following operational steps:

[0070] B1: The image frame sequence sensing feature extraction module includes a pixel association information aggregation unit. Pixel classifier With attention mechanism unit

[0071] B2: Construct a pixel classifier Pixel classifier The input is the single-sample ground truth label Yseg1. By encoding the input target ground truth label Yseg1, the other input image segmentation ground truth labels of the single-sample learning information extraction module are predicted.

[0072] B3: By employing inference methods within the video sequence, the goal is to automatically label the new image in the next frame, thereby achieving feature segmentation label pair (x) that includes additional frames. t Yseg t This expands the small sample learning dataset.

[0073] Furthermore, step S3 specifically includes the following operational steps:

[0074] C1: Solve by applying the steepest descent iteration to achieve a trade-off between accuracy and efficiency;

[0075] C2: All computations are implemented using standard neural network operations, starting from a given initialization. Start executing N iterations of steepest descent. Since the convergence of steepest descent is fast and efficient, setting the number of iterations N=5 during training and inference will achieve the expected convergence effect.

[0076] C3: In this new optimization, setting the number of iterations to N=2 can achieve a very ideal update effect, while minimizing the amount of computation and enabling real-time processing;

[0077] C4: Constructs a single-sample learning information extraction module. This can then be applied to the subsequent test frame sequences input in time sequence, combined with the real-time data from the previous step. The updated values ​​are applied to downstream processing stages as part of the predictive segmentation and tracking module. The segmentation code used to obtain the target for video tracking is finally provided as input to the segmentation decoder.

[0078] Furthermore, step S4 specifically includes the following operational steps:

[0079] D1: This segmentation tracking module can perform real-time lightweight computation on the depth feature map, and can predict the rich encoded information of the target segmentation;

[0080] D2: The model parameters in the segmentation and tracking module are learned using the single-sample information extraction module. By segmentation tracking module Output and generated true value labeling The result is obtained by minimizing the squared error between the two values, calculated by the attention mechanism unit. Provided per-pixel importance weights Weighted;

[0081] D3: Segmentation Tracking Module Implemented as a convolutional filter with a kernel size of K=5, the first hidden layer in the middle uses dilated convolution with a dilation rate D. r =2, which allows for the extraction of a wider range of pixel mutual information with minimal computational effort.

[0082] Furthermore, step S5 specifically includes the following operational steps:

[0083] E1: Segmentation and tracking module parameters based on the first frame of the video sequence. It is obtained by calculation based on the initial input image Ima1 and the given ground truth label Yseg1;

[0084] E2: The image annotations "(x1,Yseg1) pairs" of the video sequence constitute training samples for learning to segment a given target;

[0085] E3: Based on the previous step of predicting the segmentation tracking module parameters by directly minimizing the segmentation error in the first frame, robust segmentation prediction for upcoming frames can be ensured.

[0086] E4: Use the true value of the first frame to label Yseg1 (i.e., the corresponding...). In in ) as labels in our single-sample learning information extraction module;

[0087] E5: Generated by encoding the true value label Yseg1. Multidimensional labels, thus allowing segmentation tracking modules Predict richer target segmentation representations in test frames;

[0088] E6: To achieve optimal learning and fastest convergence of the single-sample learning information extraction module, and to realize the real-time optimal segmentation encoding output of the segmentation tracking module.

[0089] Furthermore, step S6 specifically includes the following operational steps:

[0090] F1: Multi-channel target segmentation unit The input consists of three information streams: the first is the real-time optimal segmentation encoding output from the segmentation and tracking module; the second is the single-sample learning information extraction module. Output Thirdly, it is the output of the image frame sequence sensing feature extraction module;

[0091] F2: First, construct the memory bank. The input information for the memory bank comes from two branches: one is the single-sample learning information extraction module. Output Secondly, it is the output of the image frame sequence perception feature extraction module;

[0092] F3: The semantic memory stores semantic information by setting dynamic parameters for region semantic comparison and pixel semantic aggregation. Depend on Composed of target features;

[0093] F4: The input information stream of the target segmentation and decoding module includes two branches: one is the real-time optimal segmentation encoding output of the segmentation tracking module; the other is the embedded representation output of the memory bank.

[0094] F5: The target segmentation and decoding module retrieves data from memory. All categorical features derived from the model are represented as real tensors.

[0095] F6: Further in Capturing the global context of consecutive frame sequences between images can enrich the representability of semantic understanding;

[0096] F7: The memory bank retains compressed global feature representations during the inference phase, and outputs video target tracking results after feature information fusion.

[0097] Furthermore, steps S1-S6 incorporate pre-trained vectorized representations and design target object-specific model parameter learning during inference to generalize target object appearance information, thereby achieving the best dynamic target tracking currently available.

[0098] The performance comparison analysis of this invention with other existing best-in-class methods is shown in the table below. The experimental dataset uses our constructed intelligent manufacturing production scenario video target tracking dataset, containing 50 video clips. Evaluation metrics include the average Jakarta. and boundaries Score, and total score The comparison algorithms include STM, FEELVOS, PreMVOS, FRTM, SiamRCNN, and AGSSVOS.

[0099]

[0100] Example:

[0101] Overall method implementation steps:

[0102] 1. Construction of video image target object semantic segmentation dataset and definition of image semantic segmentation model.

[0103] 2. Construct an image frame sequence perception feature extraction module.

[0104] 3. Single-sample learning information extraction module The structure.

[0105] 4. Segmentation and Tracking Module The structure of.

[0106] 5. Single-sample learning information extraction module based on the first frame image and segmentation tracking module Segmentation inference for subsequent frame images.

[0107] 6. Multi-channel target segmentation unit The construction of this module will segment the tracking module. The optimal segmentation code is used as input to the segmentation decoder. The final output is the multi-channel target semantic segmentation result.

[0108] The above general steps include: 1. Constructing the video image target object semantic segmentation dataset and defining the image semantic segmentation model.

[0109] Step 1: Construct a semi-supervised image segmentation dataset. A typical video dataset consists of a series of multiple frames arranged chronologically, represented as follows: Represents a video dataset, where Representative video Frame time series images, where In the collection represent Each labeled segmentation tag corresponds to Image frames To distinguish foreground objects from the image background. Therefore, this video image segmentation dataset contains... Group "Image-Tagged Data" in Also includes Unsupervised image data.

[0110] Step 2: In the image segmentation dataset for video target tracking time series, the target object is defined only by the labels of the foreground and background segmentation annotations given in the first frame. For a specific video sequence, often only the semantic segmentation annotations of its first frame are provided, and then the segmentation target inference needs to be performed in each subsequent frame; this process belongs to few-shot learning. The video sequence dataset is represented in chronological order. in This represents the latest frame of the image at the current moment. Therefore, in this application scenario, only... This single image frame contains labeled data with actual values. The other frames in the data are all unsupervised data, which means that Unsupervised image data is Therefore, the final representation of the dataset in this invention is as follows:

[0111] Step 3: Based on the features of the above dataset, we define the video object segmentation framework as follows: in These represent learnable parameters, acquired through learning during model training. (Network) Obtain the current image (Ima) and segmentation / tracking module The output. Although It is unrelated to the goal itself, but it is based on As a condition, it integrates information about the target object and encodes it into its parameters. Among them It is a convolution kernel. The weights that make up the convolutional layer with a kernel size of K thus integrate information about the target object and encode it into its parameters. The purpose of this.

[0112] Step 4: Video target tracking is implemented using a video image segmentation method as an end-to-end network, such as... Figure 1 As shown, it consists of an image frame sequence perception feature extraction module and a single-sample learning information extraction module. Segmentation tracking module and multi-channel target segmentation unit composition. This represents the network parameters learned during offline training, while These are parameters predicted by the single-sample learning information extraction module during inference.

[0113] Step 2 above: Constructing an image frame sequence perceptual feature extraction module

[0114] Step 1: The image frame sequence sensing feature extraction module includes a pixel association information aggregation unit. Pixel classifier With attention mechanism unit First, construct the pixel association information aggregation unit. The backbone network is constructed using NASNet for pixel association information aggregation units. The NASNet network structure is built using a neural architecture search framework. Time-series image frames (Ima) are input to... Represented as pixel association information convergence feature Then The feature x is output in three directions, one of which is input to the multi-channel target segmentation unit. The memory in the middle is used for two purposes: firstly, it is used for the memory bank, and secondly, it is used for the segmentation and tracking module. Thirdly, it is input into the single-sample learning information extraction module. (See Figure 1 (Details will be provided in subsequent steps). The multi-channel object segmentation unit uses three residual blocks with a spatial stride of s = 16. These features are first fed through an additional convolution. Simultaneously, at the input... Before, The output feature dimension is reduced to C=512.

[0115] Step 2: Construct a pixel classifier Pixel classifier The input is a single-sample ground truth label, Yseg1. Encoding the input target ground truth label Yseg1 predicts the segmentation ground truth labels for other input images in the single-sample learning information extraction module. Pixel classifier. The tensor to the deep feature map is Where H, W, and D are the height, width, and dimension of the segmentation tracking module features, and s is the feature stride. Pixel classifier It is implemented as a dilated convolutional network.

[0116] Step 3: Attention Mechanism Unit The target ground truth label Yseg is used as input to construct the attention mechanism unit, and its mapping tensor is: According to the convergence function Generate the element values ​​of the mapped tensor. or It is a set of feature segmentation label pairs (x) of size Q. t ,Tseg t This includes a single segmentation ground truth labeled frame (x1, Yseg1). By employing inference methods within the video sequence, it aims to automatically label the next frame with new image labels, thereby achieving feature segmentation label pairs (x1, Yseg1) that include additional frames. t Yseg t This expands the few-shot learning dataset. The scalar λ is a regularization parameter learned through training. The attention mechanism unit is implemented as a 6-hidden-layer convolutional network. Similar to... and Shared feature map tensor structure unit.

[0117] Step 3 above, single-sample learning information extraction module Construction

[0118] Step 1: Constructing a single-sample learning information extraction module It is continuously differentiable, it will Minimize to yes The function is continuously differentiable. A compromise between accuracy and efficiency is achieved by applying the steepest descent iterative method. The optimized iterative calculation method is expressed as: in This represents gradient operation, T represents transpose, ⊙ represents convolution operation, t represents the t-th frame, and x represents the gradient operation. t Yseg represents the pixel association information convergence feature of the image in frame t. t The segmentation label represents the segmentation label of frame t. This represents the result obtained from the i-th iteration calculation for the t-th frame of the image. value, This represents the result obtained during the (i+1)th iteration of the t-th frame image. Value. Single-sample learning information extraction module. The input is The output is The weights that make up the convolutional layer with a kernel size of K integrate information about the target object. These are parameters predicted by the single-sample learning information extraction module during inference.

[0119] Step 2: All calculations in the above optimization iterative calculation method are implemented using standard neural network operations. Since it is continuously differentiable, this applies to all network parameters. The segmentation tracking module parameters obtained after i iterations It is also differentiable. The single-sample learning information extraction module is implemented as a network module. As mentioned above: It is a set of feature segmentation label pairs (x) of size Q. t Yseg t It starts from a given initialization. Begin executing N iterations of steepest descent (as described in step one). Because steepest descent converges quickly and efficiently, setting the number of iterations N=5 during training and inference will achieve the desired convergence.

[0120] Step 3: Parameters obtained in the previous step Building upon this foundation, further optimization-based formulas are used. For segmentation tracking module parameters The dataset is updated in a timely manner by incorporating new input frame samples, and new frame images that are subsequently input are added to the dataset. Then, iterative optimization calculations are applied. In this new optimization, setting the number of iterations to N=2 can achieve a very ideal update effect, while minimizing the amount of computation and enabling real-time processing.

[0121] Step 4: The segmentation and tracking module parameters obtained by iterative prediction of the first frame using the steepest descent method. This constructs the single-sample learning information extraction module. This can then be applied to the subsequent test frame sequences input in time sequence, combined with the real-time data from the previous step. The updated values ​​are applied to downstream processing stages as part of the predictive segmentation and tracking module. The segmentation code used to obtain the target for video tracking is finally provided as input to the segmentation decoder.

[0122] Step 4 of the overall process described above: Segmentation and tracking module The structure of.

[0123] Step 1: Constructing the segmentation and tracking module The structure is implemented as follows: Single-sample learning information extraction module Output results As The input is used as parameters in the segmentation and tracking module. The structure of the segmentation and tracking module maps the input C-dimensional depth features x to a D-dimensional encoding of the target segment with the same spatial resolution H×W through training. To ensure... Since it is continuously differentiable, the segmentation tracking module can be constructed as follows: in The convolutional kernel size is K. This segmentation and tracking module can perform real-time, lightweight computations on the depth feature map, predicting rich encoded information for target segmentation.

[0124] Step 2: The model parameters in the segmentation and tracking module are learned using the single-sample information extraction module. By segmentation tracking module Output and generated true value labeling The result is obtained by minimizing the squared error between the two values, calculated by the attention mechanism unit. Provided per-pixel importance weights Weighted.

[0125] Step 3: Segmentation and Tracking Module Implemented as a convolutional filter with a kernel size of K=5, the first hidden layer in the middle uses dilated convolution with a dilation rate D. r =2, which allows for the extraction of a wider range of pixel mutual information with minimal computational effort. The number of output channels D is set to 32.

[0126] Step 5 above: Single-sample learning information extraction module based on the first frame image and segmentation tracking module Segmentation inference for subsequent frame images.

[0127] Step 1: Based on the first frame of the video sequence, segment and track module parameters. It is obtained by calculating the initial input image Ima1 combined with the given ground truth label Yseg1, where the ground truth label Yseg1 defines the semantic segmentation feature information of the target object itself. Represented as... That is, it is obtained through the single-sample learning information extraction module.

[0128] Step 2: The image annotations "(x1, Yseg1) pairs" in the video sequence constitute training samples for learning to segment a given target. However, these training samples are only provided during the inference phase, thus belonging to the few-shot learning problem in video target segmentation. In reality, there is only a single ground truth annotation sample from the first frame, strictly speaking, a single-shot learning problem. Single-shot learning information extraction module. A segmentation tracking module is generated from a single ground truth pair labeled "(x1, Yseg1)". Parameters required The approach is to minimize the supervised learning objective. in Follow these rules

[0129] Step 3: Based on the aforementioned pixel association information aggregation unit For the input image The deep embedding feature representation is used for operations, with the backbone network employing the DenseNet architecture. Segmentation and tracking module. The learning process outputs a segmentation of the target object in the initial frame. During inference, given a new frame Ima, the object is segmented into... i.e., segmentation tracking module The segmentation inference is applied to the new frame Ima to generate the first new segmentation inference. Due to the strong correlation between consecutive video frames, predicting the segmentation tracking module parameters by directly minimizing the segmentation error in the first frame, based on the previous step, ensures robust segmentation prediction for upcoming frames.

[0130] Step 4: Label Yseg1 (i.e., the corresponding value) using the true value of the first frame. In in As labels in our single-sample learning information extraction module, combined with subsequent unsupervised image data A trainable label generator is introduced. It takes the ground truth segmentation label Yseg as input and predicts the label (i.e., image sequence) of the dataset used by the single-sample learning information extraction module for training. The labels used for the single-sample learning information extraction module are based on After being input (Inference and prediction obtained). Therefore, the parameters of the segmentation tracking module are predicted as follows:

[0131] Step 5: Generate by encoding the true value label Yseg1. Multidimensional labels, thus allowing segmentation tracking modules Predict richer target segmentation representations in test frames.

[0132] Step Six: To address the imbalanced training dataset issue, assign higher weights to the target region and lower weights to regions with blurred vision, using an attention mechanism unit. Adjusting loss value Using the true value labeled Yseg as input, predict the loss. The importance weight of each pixel is determined. This guides the single-sample learning information extraction module to achieve optimal learning and fastest convergence, enabling the segmentation and tracking module to output the optimal segmentation encoding in real time.

[0133] Step 6 of the overall plan above: Multi-channel target segmentation unit The construction of this module enables the segmentation tracking module. The optimal segmentation code output is used as input to the target segmentation unit. The final output is the multi-channel target semantic segmentation result.

[0134] Step 1: Multi-channel target segmentation unit The input consists of three information streams: the first is the real-time optimal segmentation encoding output from the segmentation and tracking module; the second is the single-sample learning information extraction module. Output Thirdly, it is the output of the image frame sequence perceptual feature extraction module. Multi-channel target segmentation unit. It consists of two parts: a target segmentation and decoding module and a memory bank. The construction process of the memory bank is described in steps two and three, and the construction of the multi-channel target segmentation unit is described in steps four to seven.

[0135] Step 2: First, construct the memory bank. The input information for the memory bank comes from two branches: one is the single-sample learning information extraction module. Output Secondly, there is the output of the image frame sequence perceptual feature extraction module. These information streams contain pixel-level shallow and deep semantic features of the input image. The role of the memory bank is that the semantic context of target objects between parts of the image and between consecutive frame sequences is crucial for understanding the correlation between pixels. The memory bank selectively memorizes the feature information of the input image sequence to achieve semantic aggregation and construction of pixels. It uses the temporal sequence image frame context information in the memory bank to enhance semantic understanding and perform semantic segmentation. The construction of the memory bank provides a large number of regional semantic segmentation references.

[0136] Step 3: The semantic memory stores semantic information by setting dynamic parameters for region semantic comparison and pixel semantic aggregation. Depend on Composed of target features, namely Each target feature corresponds to a category. Each term represents the first term in the image Ima observed throughout the learning phase. Overall region perception representation of each category express A 3D real-valued vector. In each training phase, the memory is updated with the new feature learning results. Specifically, the current depth information of the image Ima. pass Smooth update to memory representation In the context of , ∈ is a hyperparameter of the memory bank, with a value range of 0 << ∈ << 0.06. When the th... Update when each category appears in the image Ima and the classification confidence is greater than the threshold.

[0137] Step four: Next, we introduce the construction of the target segmentation decoding module (steps four through seven). The input information stream of the target segmentation decoding module includes two branches: one is the real-time optimal segmentation encoding output of the segmentation tracking module; the other is the embedding representation output from the memory bank. The target segmentation decoding module compresses the embedding representation from the memory bank into a compact set of representative example embedding representations. For each category... element All features are processed by a 7-layer convolutional network for multi-class classification to obtain... Each feature represents Matrix representation of a 3D vector Multiple features are used to represent each class (i.e.) This is used to explain the changes in characteristics within a class.

[0138] Step 5: The target segmentation and decoding module retrieves data from the memory. All categorical features derived from the model are represented as real tensors. Then for each with characteristics The subclass image Ima, where W and H represent the width and height of the image respectively, is represented using embedding. Calculate its deep semantics Algorithm representation in yes 3D real matrix, tensor and They are respectively reduced in dimension and flattened into matrix form. To achieve high-speed computing, This indicates the transpose operation. Represents matrix multiplication, where Let be the hyperbolic tangent function, with the expression: Each item in reflects Each row (i.e., feature) in the data is compared with... The canonical generalized distance metric between each column (i.e., feature representative) in the dataset. Based on deep semantics. and feature embedding Computational feature representation in express The rich feature representation adjusts the tensor structure to

[0139] Step Six, Next and original features Connection as Not only in Encode local contextual semantics within the image, and further in Capturing the global context of consecutive frame sequences between images can enrich the representability of semantic understanding.

[0140] Step 7: The target segmentation and decoding module outputs multi-channel target pixel classification, including the backbone network using DenseNet to map the input image Ima into a convolutional representation. Classification is achieved using a convolutional neural network with a hidden layer depth of 9. A classification-aware attention map is generated from the feature embeddings, and a 5×5 convolutional kernel is used. (Memory bank) All region patterns are stored in the training data. The memory retains compressed global feature representations during the inference phase, and the video target tracking results are output after feature fusion.

[0141] It should be noted that, in this document, relational terms such as first and second (number one, number two), etc., are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A video target tracking method based on single-sample learning, characterized in that: The following steps are included: S1: Model Construction: Construction of video image target object semantic segmentation dataset and definition of image semantic segmentation model; S2: Module Construction: Construct an image frame sequence perceptual feature extraction module; S3: Sample Module Construction: Single Sample Learning Information Extraction Module The structure; S4: Tracking Module Construction: Segmented Tracking Module The structure; S5: Image segmentation inference: Information extraction module based on single-sample learning of the first frame image. and segmentation tracking module Segmentation inference for subsequent frame images; S6: Output: Multi-channel target segmentation unit The construction of this module will segment the tracking module. The optimal segmentation code is used as input to the segmentation decoder. The final output is the multi-channel target semantic segmentation result; The S1 step specifically includes the following operations: A1: Construct a semi-supervised image segmentation dataset; A2: In the image segmentation dataset of video target tracking time series, the target object is defined only by the labels of the reference target foreground and background segmentation annotations given in the first frame. For a specific video sequence, only the semantic segmentation annotation labels of its first frame image are given, and then the segmentation target inference work needs to be performed in each subsequent frame. A3: Based on the characteristics of the above dataset, the video object segmentation framework is defined as follows: ,in These represent learnable parameters, acquired through learning during model training. A4: Video target tracking is implemented using a video image segmentation method as an end-to-end network; The S2 step specifically includes the following operations: B1: The image frame sequence sensing feature extraction module includes a pixel association information aggregation unit. Pixel classifier With attention mechanism unit ; B2: Construct a pixel classifier Pixel classifier The input is the label of a single sample's true value. By labeling the actual input target value Encode the image to predict the ground truth labels for other input image segments in the single-sample learning information extraction module; B3: By employing inference methods within the video sequence, the goal is to automatically label the new image in the next frame, thereby achieving feature segmentation labeling pairs that include additional frames. This expands the small sample learning dataset; The S3 step specifically includes the following operations: C1: Solve by applying the steepest descent iteration to achieve a trade-off between accuracy and efficiency; C2: All computations are implemented using standard neural network operations, starting from a given initialization. Start execution The steepest descent iteration is used because the convergence of steepest descent is fast and efficient, so the number of iterations is set during training and inference. This will achieve the expected convergence effect; C3: Setting the number of iterations in this new optimization. This achieves a very good update effect while minimizing the amount of computation and enabling real-time processing; C4: Constructs a single-sample learning information extraction module. This can then be applied to the subsequent test frame sequences input in time sequence, combined with the real-time data from the previous step. The updated values ​​are applied to downstream processing stages as part of the predictive segmentation and tracking module. This is used to obtain the segmentation code of the video tracking target, which is then provided as input to the segmentation decoder. The S4 step specifically includes the following operations: D1: This segmentation tracking module can perform real-time lightweight computation on the depth feature map, and can predict the rich encoded information of the target segmentation; D2: The model parameters in the segmentation and tracking module are learned using the single-sample information extraction module. By segmentation tracking module Output and generated true value labeling The result is obtained by minimizing the squared error between the two values, calculated by the attention mechanism unit. Provided per-pixel importance weights Weighted; D3: Segmentation Tracking Module Implemented with a kernel size of The convolutional filter uses dilated convolution in the first hidden layer, with a dilation rate of [missing information]. It can extract a wider range of pixel mutual information with minimal computational effort; The S5 step specifically includes the following operations: E1: Segmentation and tracking module parameters based on the first frame of the video sequence. It is for the initial input image Labeling based on given true values Obtained through calculation; E2: Image annotation of video sequences The training samples constitute the basis for learning to segment a given target. E3: Based on the previous step of predicting the segmentation tracking module parameters by directly minimizing the segmentation error in the first frame, robust segmentation prediction for upcoming frames can be ensured. E4: Labeled using the true values ​​from the first frame That is, corresponding In ,in , as labels in the single-sample learning information extraction module; E5: Labeling the true values Encode generation Multidimensional labels, thus allowing segmentation tracking modules Predict richer target segmentation representations in test frames; E6: To achieve optimal learning and fastest convergence of the single-sample learning information extraction module, and to realize the real-time optimal segmentation encoding output of the segmentation tracking module; Step S6 specifically includes the following operational steps: F1: Multi-channel target segmentation unit The input includes three information streams: the first is the real-time optimal segmentation code output of the segmentation tracking module; The second is the single-sample learning information extraction module. Output Thirdly, it is the output of the image frame sequence perceptual feature extraction module; F2: First, construct the memory bank. The input information for the memory bank comes from two branches: one is the single-sample learning information extraction module. Output Secondly, it is the output of the image frame sequence perceptual feature extraction module; F3: The semantic memory stores semantic information by setting dynamic parameters for region semantic comparison and pixel semantic aggregation. Depend on Composed of target features; F4: The input information stream of the target segmentation and decoding module includes two branches: one is the real-time optimal segmentation encoding output of the segmentation tracking module; the other is the embedded representation output of the memory bank. F5: The target segmentation and decoding module retrieves data from memory. All categorical features derived from the model are represented as real tensors. ; F6: Further in Capturing the global context of consecutive frame sequences between images can enrich the representability of semantic understanding; F7: The memory bank retains compressed global feature representations during the inference phase, and outputs video target tracking results after feature information fusion.

2. The video target tracking method based on single-sample learning according to claim 1, characterized in that: In steps S1-S6, the pre-trained vectorized representation is combined, and model parameter learning specific to the target object is designed during the inference process to generalize the appearance information of the target object and achieve the best dynamic target tracking available.

Images