Recurrent difference feedback video object counting network modeling method

Abstract

The present invention discloses a cyclic difference feedback video target counting network modeling method, comprising: S1: establishing a target counting video data set; S2: constructing a cyclic difference feedback video target counting network, the cyclic difference feedback video target counting network comprising a basic network, a cyclic difference feedback learning module and a step constraint strategy; S3: using a high-density target video data set to train the cyclic difference feedback video target counting network to obtain a trained cyclic difference feedback video target counting network; S4: using a test video to evaluate the trained cyclic difference feedback video target counting network. The present invention realizes the accurate detection of high-density targets in video sequences and improves the robustness of the detection network.

Description

Technical Field

[0001] The invention relates to computer vision technology, and in particular to a cyclic difference feedback video target counting network modeling method. Background Art

[0002] The object counting task is to detect or regress objects in images or video frames to count objects of interest. It not only achieves accurate counting in sparse scenes, but also can count more challenging dense objects. Based on this, the task has a wide range of applicability, including multiple types of objects such as people, vehicles, and fruits.

[0003] Most existing methods use a single image or video frame for detection, so it is difficult to use spatiotemporal information to improve the performance of the model. Although operations such as LSTM or Transformer can be used to process video sequences, these operations are usually slow in parallel computing efficiency and difficult to optimize under limited samples. To this end, the present invention proposes a cyclic difference feedback video object counting network modeling method. Summary of the invention

[0004] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a cyclic difference feedback video target counting network modeling method to achieve a robust video frame target counting function.

[0005] Technical solution: The cyclic difference feedback video target counting network modeling method of the present invention comprises:

[0006] S1: Establish a target counting video dataset, use a video capture device to obtain the video of the target of interest, mark the target in each video frame, and obtain the label of the current frame;

[0007] S2: constructing a cyclic difference feedback video target counting network, wherein the cyclic difference feedback video target counting network includes a basic network, a cyclic difference feedback learning module and a step constraint strategy, wherein the basic network is used for feature extraction of video frames to obtain a video frame feature vector, the cyclic difference feedback learning module performs difference information feedback learning based on the video frame feature vector to obtain a density output and a difference metric, and the step constraint strategy is used for performing structured and unstructured constraints on the learning of the current network learning parameters;

[0008] S3: training the cyclic difference feedback video target counting network using the target counting video dataset to obtain a trained cyclic difference feedback video target counting network;

[0009] S4: Evaluation of the trained recurrent difference feedback video object counting network using test videos.

[0010] Furthermore, the cyclic difference feedback learning module in S2 specifically includes:

[0011] The first convolutional layer: includes two 3×3 convolutional layers, which are used to transform the current frame features extracted by the basic network to obtain transformation features;

[0012] The second convolutional layer: includes three 3×3 convolutional layers with a dilation rate of 2, which further learn the transformation features to obtain the current frame difference features, and the difference features are passed through a single 1×1 convolutional layer to obtain a difference metric;

[0013] Fusion layer: used to aggregate the transformation features and the ikth frame difference features along the channel dimension, and obtain the density output through two 3×3 convolution layers with a dilation rate of 2 and a single 1×1 convolution layer.

[0014] Furthermore, the step constraint strategy in S2 is specifically to use the features of the network obtained by training the previous batch to perform structural and unstructured constraints on the features of the current batch:

[0015]

[0016] Among them, F j and Represent the feature outputs of the j-th layer network of the current batch and the previous batch, respectively, N j F i j and The number of feature pixels, α and β are hyper parameters.

[0017] Furthermore, the basic network in S2 is specifically the network layer before the fourth downsampling in the ImageNet pre-trained network model, which extracts basic features of the video frames.

[0018] Furthermore, the S3 specifically includes:

[0019] S3-1: Select video data from the target counting video dataset in a ratio of 7:1:2 and divide the dataset into video training set, validation set and test set;

[0020] S3-2: Set the parameters required for network training and initialize the network model parameters;

[0021] S3-3: Send the samples in the video training set to the cyclic difference feedback video object counting network for training, and calculate the total loss L t1 And optimize the network parameters through Adam optimizer;

[0022] S3-4: Use samples from the video validation set to validate the cyclic difference feedback video object counting network and save the network parameters with the smallest error;

[0023] S3-5: Return to step S3-3 until the training is completed.

[0024] Furthermore, the total loss L in S3-3 is t1 The calculation steps include:

[0025] S3-3-1: Calculate the supervised loss L1 of the difference metric:

[0026]

[0027] Among them, M represents the number of each input pixel, y i represents the true difference map, p i Represents a measure of dissimilarity.

[0028] S3-3-2: Calculate the supervised loss L2 of the density output:

[0029]

[0030] Among them, Z and Represent the density output of the network and the true density map respectively.

[0031] S3-3-3: Calculate the weighted sum of each loss as the total loss L t1 :

[0032]

[0033] in, Represents a hyperparameter.

[0034] Furthermore, the specific calculation of the true difference map in S3-3-1 is:

[0035]

[0036] Among them, κ is the scaling factor.

[0037] Furthermore, the ikth frame difference feature in S2 refers to the frame difference feature obtained from the ikth frame before the current ikth frame. If the current frame is the previous k frames, the ikth frame difference feature is an all-zero feature vector.

[0038] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0039] 1) The present invention designs a cyclic difference feedback video object counting network modeling method to realize the interaction of features between video frames, and at the same time utilizes feature constraints to enhance the regression performance of the network.

[0040] 2) The present invention designs a cyclic difference feedback learning module and proposes a new difference representation learning strategy, which uses difference information to help the network mine effective features, and uses a cyclic strategy to utilize historical spatiotemporal features to help better regress the density map in the current stage.

[0041] 3) The present invention designs a step constraint strategy, which imposes structured and unstructured constraints on the parameters of the current stage network, so that the network can achieve continuous parameter updates while avoiding the influence of excessive counting losses on network parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the cyclic difference feedback video target counting network modeling method provided by the present invention;

[0043] Figure 2 The framework diagram of the cyclic difference feedback video target counting network provided in this aspect. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] This embodiment provides a high-density target counting network modeling method driven by dark light enhancement, such as Figure 1 As shown, the following steps are included:

[0046] S1: Establish a target counting video dataset, use a video capture device to obtain the video of the target of interest, mark the target in each video frame, and obtain the label of the current frame;

[0047] S2: constructing a cyclic difference feedback video target counting network, wherein the cyclic difference feedback video target counting network includes a basic network, a cyclic difference feedback learning module and a step constraint strategy, wherein the basic network is used for feature extraction of video frames to obtain a video frame feature vector, the cyclic difference feedback learning module performs difference information feedback learning based on the video frame feature vector to obtain a density output and a difference metric, and the step constraint strategy is used for performing structured and unstructured constraints on the learning of the current network learning parameters;

[0048] The basic network is specifically the network layer before the fourth downsampling in the ImageNet pre-trained network model, which extracts basic features of the video frames. In the embodiment, the first 10 layers of VGG16 are used as the basic network.

[0049] Among them, the cyclic difference feedback learning module specifically includes:

[0050] The first convolutional layer: includes two 3×3 convolutional layers, which are used to transform the current frame features extracted by the basic network to obtain transformation features;

[0051] The second convolutional layer: includes three 3×3 convolutional layers with a dilation rate of 2, which further learn the transformation features to obtain the current frame difference features, and the difference features are passed through a single 1×1 convolutional layer to obtain a difference metric;

[0052] Fusion layer: used to aggregate the transformation features and the ikth frame difference features along the channel dimension, and obtain the density output through two 3×3 convolution layers with a dilation rate of 2 and a single 1×1 convolution layer.

[0053] The ikth frame difference feature refers to the frame difference feature obtained from the ikth frame before the current ikth frame. If the current frame is the previous k frames, the ikth frame difference feature is an all-zero feature vector.

[0054] Among them, the step constraint strategy is to use the characteristics of the network obtained by the previous batch training to perform structured and unstructured constraints on the characteristics of the current batch:

[0055]

[0056] Among them, F j and Represent the feature outputs of the j-th layer network of the current batch and the previous batch, respectively, N j F i j and The number of feature pixels, α and β are hyper parameters.

[0057] S3: training the cyclic difference feedback video target counting network using the target counting video dataset to obtain a trained cyclic difference feedback video target counting network;

[0058] The steps of network training are as follows:

[0059] S3-1: Select video data from the target counting video dataset in a ratio of 7:1:2 and divide the dataset into video training set, validation set and test set;

[0060] S3-2: Set the parameters required for network training and initialize the network model parameters;

[0061] S3-3: Send the samples in the video training set to the cyclic difference feedback video object counting network for training, and calculate the total loss L t1 And optimize the network parameters through Adam optimizer;

[0062] S3-4: Use samples from the video validation set to validate the cyclic difference feedback video object counting network and save the network parameters with the smallest error;

[0063] S3-5: Return to step S3-3 until the training is completed.

[0064] Furthermore, the total loss L in S3-3 is t1 The calculation steps include:

[0065] S3-3-1: Calculate the supervised loss L1 of the difference metric:

[0066]

[0067] Among them, M represents the number of each input pixel, y i represents the true difference map, p i Represents a measure of dissimilarity.

[0068] S3-3-2: Calculate the supervised loss L2 of the density output:

[0069]

[0070] Among them, Z and Represent the density output of the network and the true density map respectively.

[0071] S3-3-3: Calculate the weighted sum of each loss as the total loss L t1 :

[0072]

[0073] in, Represents a hyperparameter.

[0074] Furthermore, the specific calculation of the true difference map in S3-3-1 is:

[0075]

[0076] Among them, κ is the scaling factor.

[0077] S4: Evaluation of the trained recurrent difference feedback video object counting network using test videos.

[0078] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A cyclic difference feedback video object counting network modeling method, characterized in that: The method comprises the following steps: S1: Establish a target counting video dataset, use a video capture device to obtain the video of the target of interest, mark the target in each video frame, and obtain the label of the current frame; S2: constructing a cyclic difference feedback video target counting network, wherein the cyclic difference feedback video target counting network includes a basic network, a cyclic difference feedback learning module and a step constraint strategy, wherein the basic network is used for feature extraction of video frames to obtain a video frame feature vector, the cyclic difference feedback learning module performs difference information feedback learning based on the video frame feature vector to obtain a density output and a difference metric, and the step constraint strategy is used for performing structured and unstructured constraints on the learning of the current network learning parameters; S3: training the cyclic difference feedback video target counting network using the target counting video dataset to obtain a trained cyclic difference feedback video target counting network; S4: Evaluate the trained cyclic difference feedback video object counting network using test videos; The cyclic difference feedback learning module in S2 specifically includes: The first convolutional layer: includes two 3×3 convolutional layers, which are used to transform the current frame features extracted by the basic network to obtain transformation features; The second convolutional layer: includes three 3×3 convolutional layers with a dilation rate of 2, which further learn the transformation features to obtain the current frame difference features, and the difference features are passed through a single 1×1 convolutional layer to obtain a difference metric; Fusion layer: used to aggregate the transformation features and the ikth frame difference features along the channel dimension, and obtain the density output through two 3×3 convolution layers with a dilation rate of 2 and a single 1×1 convolution layer; The step constraint strategy in S2 is specifically to use the features of the network obtained by the previous batch training to perform structural and unstructured constraints on the features of the current batch: Among them, F j and Represent the feature outputs of the j-th layer network of the current batch and the previous batch, respectively, N j F i j and The number of feature pixels, α and β are hyper parameters.

2. The cyclic difference feedback video object counting network modeling method according to claim 1 is characterized in that: The basic network in S2 is specifically the network layer before the fourth downsampling in the ImageNet pre-trained network model, which extracts basic features of the video frames.

3. The cyclic difference feedback video object counting network modeling method according to claim 1, characterized in that: The S3 specifically includes: S3-1: Select video data from the target counting video dataset in a ratio of 7:1:2 and divide the dataset into video training set, validation set and test set; S3-2: Set the parameters required for network training and initialize the network model parameters; S3-3: Send the samples in the video training set to the cyclic difference feedback video object counting network for training, and calculate the total loss L t1 And optimize the network parameters through Adam optimizer; S3-4: Use samples from the video validation set to validate the cyclic difference feedback video object counting network and save the network parameters with the smallest error; S3-5: Return to step S3-3 until the training is completed.

4. The cyclic difference feedback video object counting network modeling method according to claim 3 is characterized in that: The total loss L in S3-3 t1 The calculation steps include: S3-3-1: Calculate the supervised loss L1 of the difference metric: Among them, M represents the number of each input pixel, y i represents the true difference map, p i represents the difference measure, S3-3-2: Calculate the supervised loss L2 of the density output: Among them, Z and Represent the density output and true density map of the network respectively, S3-3-3: Calculate the weighted sum of each loss as the total loss L t1 : in, Represents a hyperparameter.

5. The cyclic difference feedback video object counting network modeling method according to claim 1, characterized in that: The specific calculation of the true difference map in S3-3-1 is: Among them, κ is the scaling factor.

6. The cyclic difference feedback video object counting network modeling method according to claim 2, characterized in that: The ikth frame difference feature in S2 refers to the frame difference feature obtained from the ikth frame before the current ikth frame. If the current frame is the previous k frames, the ikth frame difference feature is an all-zero feature vector.

Images