High-density target counting network modeling method driven by dark light enhancement

Abstract

The present invention discloses a modeling method of a high-density target counting network driven by dark light enhancement, comprising: S1: making a low-light / normal-light sample pair and a high-density target data set; S2: establishing a low-light enhancement learning network, which includes a low-light enhancement module, a backbone network and a texture feature perception module; S3: using the low-light / normal-light sample pair data set to train the low-light enhancement learning network, and obtaining a trained low-light enhancement learning network; S4: establishing a high-density target regression network, which includes a low-light enhancement learning network and a texture feature perception density regression module; S5: using the trained low-light enhancement learning network in S3 to initialize the parameters of the S4 high-density target regression network, and using the high-density target data set to train the high-density target regression network, and obtaining a trained high-density target regression network. Robust high-density target detection of samples under different lighting conditions is achieved, and the applicability of high-density target complex scenes is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a dark-light enhancement driven high-density target counting network modeling method. Background Art

[0002] High-density target counting is an important task in computer vision, and is widely used in statistics of various high-density targets, such as crowd counting, fruit counting, pest counting, vehicle density detection, etc. Taking fruit counting as an example, fruits such as dates and cherry tomatoes usually have high density and severe occlusion. Accurate prediction of them enables planters to grasp the growth of plants in a timely manner, and also helps them to achieve efficient resource allocation.

[0003] Most existing methods count high-density targets in pictures taken under good lighting conditions. When the lighting conditions are poor, the performance of the model will drop significantly. This is because the pictures taken under poor lighting conditions have poor image quality, many noise points, and unclear context information, which reduces the universality of existing methods. Although the above problems can be alleviated by using low-light pictures to annotate high-density targets, it is difficult for annotators to accurately annotate each target, which leads to disadvantages such as long annotation time and high cost. To this end, this aspect proposes a high-density target counting network modeling method driven by dark light enhancement. Summary of the invention

[0004] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a dark-light-enhanced driven high-density target counting network modeling method to achieve the task of counting high-density targets under various lighting conditions.

[0005] Technical solution: The high-density target counting network modeling method driven by dark light enhancement of the present invention comprises the following steps:

[0006] S1: Create low-light / normal-light sample pairs and high-density target datasets. Obtain low-light / normal-light samples by adjusting the camera exposure. Obtain high-density target datasets through monitoring equipment. Manually mark the targets in the images in the dataset.

[0007] S2: Establish a low-light enhancement learning network, the low-light enhancement learning network includes a low-light enhancement module, a backbone network and a texture feature perception module, the low-light enhancement module is used to input picture illumination enhancement to obtain an illumination enhancement feature map, the backbone network performs basic feature learning according to the illumination enhancement feature map to obtain a basic feature vector, and the texture feature perception module performs manual feature learning according to the basic feature vector to obtain manual feature prediction;

[0008] S3: training the low-light enhancement learning network using the low-light / normal-light sample data set to obtain a trained low-light enhancement learning network;

[0009] S4: Establishing a high-density target regression network, the high-density target regression network includes the low-light enhancement learning network and the texture feature perception density regression module, the texture feature perception density regression module uses the manual feature prediction in S2 to perform feature learning, and fuses and learns with the basic feature vector to obtain a density map prediction;

[0010] S5: Initialize the parameters of the high-density target regression network in S4 using the low-light enhancement learning network trained in S3, and train the high-density target regression network using a high-density target data set to obtain a trained high-density target regression network.

[0011] Furthermore, the low light enhancement module in S2 specifically includes:

[0012] The first convolutional layer includes three 3×3 convolutional layers;

[0013] The Fourier transform layer includes a two-dimensional Fourier transform and a component transform. The two-dimensional Fourier transform is used to perform a fast Fourier transform on the output of the first convolutional layer to obtain a real vector R(x) and an imaginary vector I(x). The component transform performs A(x) and P(x) operations on the real vector and the imaginary vector to obtain an amplitude (A) and an amplitude (P), respectively, where x∈R c×h×w represents the input features, and P(x) = arctan[I(x) / R(x)];

[0014] The component reinforcement layer includes an amplitude interaction layer and an amplitude learning layer. The amplitude interaction layer generates a channel dimension parameter m∈R c×1×1 The amplitude (A) is multiplied to obtain a refined amplitude feature; the amplitude learning layer performs feature learning on the amplitude (P) through two 3×3 convolutional layers to obtain an enhanced amplitude feature;

[0015] The inverse Fourier transform layer is used to perform a fast inverse Fourier transform on the refined amplitude features and the enhanced amplitude features to obtain the Fourier enhanced features;

[0016] The second convolutional layer learns the Fourier enhancement features through a 3×3 convolutional layer to obtain a dark-light enhanced image.

[0017] Furthermore, the backbone network in S2 is specifically the first 10 layers of the pre-trained VGG16 network, and the backbone network accepts the output of the low-light enhancement module as input to obtain basic features.

[0018] Furthermore, the texture feature perception module in S2 specifically includes:

[0019] The third convolutional layer includes three 3×3 convolutional layers with a dilation rate of 2;

[0020] The activation layer includes a 1×1 convolution layer, a Sigmoid function and an upsampling layer connected in sequence. The 1×1 convolution layer is used to down-channel the output of the third convolution layer, and then obtain normalized features after passing through the Sigmoid function. The upsampling layer performs bilinear difference upsampling on the normalized features to obtain manual feature predictions.

[0021] Furthermore, the S3 specifically includes:

[0022] S3-1: Divide the low-light / normal-light sample pair dataset into a training dataset and a validation dataset;

[0023] S3-2: Setting training parameters, sending sample pairs to the low-light enhancement learning network for training, and optimizing the network by calculating the total loss;

[0024] S3-3: Test the low-light enhancement learning network using samples in the validation dataset and save the network parameters with the minimum error;

[0025] S3-4: Return to step S3-2 until the training is completed to obtain the final low-light enhancement learning network.

[0026] Furthermore, the texture feature perception density regression module in S4 specifically includes:

[0027] The fourth convolutional layer uses the output dark-light enhanced image in S2 as input and passes through two 3×3 convolutional layers with a dilation rate of 2 to obtain the dark-light enhanced feature vector;

[0028] A fusion layer, including a splicing layer and a 1×1 convolution layer, wherein the splicing splices the dark light enhancement feature vector and the output of the backbone network in S2 along the channel dimension, and the 1×1 convolution layer performs channel dimension reduction on the spliced ​​feature vector;

[0029] The fifth convolution layer includes two 3×3 convolution layers, a 1×1 convolution layer and an upsampling layer connected in sequence. The 3×3 convolution layer performs feature learning on the output of the fusion layer, and then performs channel dimension reduction through the 1×1 convolution layer. Finally, the features after channel dimension reduction are bilinearly upsampled to obtain a density map prediction.

[0030] Furthermore, the S5 specifically includes:

[0031] S5-1: Divide the high-density target data set into a training data set and a validation data set;

[0032] S5-2: Setting training parameters, sending sample pairs into the high-density target regression network for training, and optimizing the network by calculating the total loss;

[0033] S5-3: Testing the high-density target regression network with samples in the validation data set, and saving the network parameters with the minimum error;

[0034] S5-4: Return to step S5-2 until the training is completed to obtain the final high-density target regression network.

[0035] Furthermore, the step of calculating the total loss in S3-2 includes:

[0036] S3-2-1: Calculate the supervision loss L of the intermediate output 1 :

[0037]

[0038] Among them, M i Indicates the amount of data loaded in each batch, X i represents a normal light picture, Indicates intermediate output;

[0039] S3-2-2: Calculate the supervised loss L of texture output 2 :

[0040]

[0041] Among them, X i Represents the data loaded in each batch, a i ∈{0,1} is the true label in the hand-crafted features, 0 represents the background area, 1 represents the texture area, and p i Represents texture output;

[0042] S3-2-3: Calculate the weighted sum of each loss as the total loss L t1 :

[0043] L t1 =L 1 +λ*L 2

[0044] Here, λ represents a hyperparameter.

[0045] Furthermore, the step of calculating the total loss in S5-2 includes:

[0046] S5-2-1: Calculate the supervised loss L of the density output 3 :

[0047]

[0048] Among them, Mi Indicates the amount of data loaded in each batch, Y i represents the true label of the density map, Indicates density output;

[0049] S5-2-2: Calculate the supervised loss L of texture output 4 :

[0050]

[0051] Among them, X i Represents the data loaded in each batch, a i ∈{0,1} is the true label in the hand-crafted features, 0 represents the background area, 1 represents the texture area, and p i Represents texture output;

[0052] S5-2-3: Calculate the weighted sum of each loss as the total loss L t2 :

[0053] L Lt2 =L 3 +β*L 4

[0054] Here, β represents a hyperparameter.

[0055] Furthermore, the manual features in S2 are binary feature representations obtained by a manual feature calculation method.

[0056] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0057] 1) This paper designs a dark-light enhancement-driven high-density target counting network modeling method, which enhances low-light images online through a two-stage learning method and uses manual feature enhancement to perform density map regression. It can be widely used in security monitoring, smart agriculture and forestry, and medical fields.

[0058] 2) The present invention designs a low-light enhancement module, which is placed at the bottom layer of the network and realizes the network's perception of light through Fourier transform reinforcement learning.

[0059] 3) The present invention designs a texture learning strategy. First, in the first stage, the network enhances the low-light pictures while roughly learning the input texture information through manual features, which helps the network to perceive the input context information more robustly.

[0060] 4) The present invention designs a texture output enhancement learning strategy. In the second stage of learning, by copying the parameters learned by the network in the first stage, the context information is refined through the texture output further learning strategy to help the network better learn the density map. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flow chart of the high-density target counting network modeling method driven by dark light enhancement provided by the present invention;

[0062] Figure 2 This is a framework diagram of a high-density target counting network driven by dark light enhancement provided in this aspect. DETAILED DESCRIPTION

[0063] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] Embodiment: This embodiment provides a high-density target counting network modeling method driven by dark light enhancement, such as Figure 1 As shown, the following steps are included:

[0065] S1: Create low-light / normal-light sample pairs and high-density target datasets. Obtain low-light / normal-light samples by adjusting the camera exposure. Obtain high-density target datasets through monitoring equipment. Manually mark the targets in the images in the dataset.

[0066] S2: Establish a low-light enhancement learning network, which includes a low-light enhancement module, a backbone network and a texture feature perception module. The low-light enhancement module is used to input image illumination enhancement to obtain an illumination enhancement feature map. The backbone network performs basic feature learning based on the illumination enhancement feature map to obtain a basic feature vector. The texture feature perception module performs manual feature learning based on the basic feature vector to obtain manual feature prediction.

[0067] Among them, the low-light enhancement module specifically includes:

[0068] The first convolutional layer includes three 3×3 convolutional layers;

[0069] The Fourier transform layer includes a two-dimensional Fourier transform and a component transform. The two-dimensional Fourier transform is used to perform a fast Fourier transform on the output of the first convolutional layer to obtain a real vector R(x) and an imaginary vector I(x). The component transform performs A(x) and P(x) operations on the real vector and the imaginary vector to obtain an amplitude (A) and an amplitude (P), respectively, where x∈R c×h×w represents the input features, and P(x) = arctan[I(x) / R(x)];

[0070] The component reinforcement layer includes an amplitude interaction layer and an amplitude learning layer. The amplitude interaction layer generates a channel dimension parameter m∈R c×1×1The amplitude (A) is multiplied to obtain a refined amplitude feature; the amplitude learning layer performs feature learning on the amplitude (P) through two 3×3 convolutional layers to obtain an enhanced amplitude feature;

[0071] The inverse Fourier transform layer is used to perform a fast inverse Fourier transform on the refined amplitude features and the enhanced amplitude features to obtain the Fourier enhanced features;

[0072] The second convolutional layer learns the Fourier enhancement features through a 3×3 convolutional layer to obtain a dark-light enhanced image.

[0073] The backbone network is specifically the first 10 layers of the pre-trained VGG16 network, and the backbone network accepts the output of the low-light enhancement module as input to obtain basic features.

[0074] Furthermore, the texture feature perception module in S2 specifically includes:

[0075] The third convolutional layer includes three 3×3 convolutional layers with a dilation rate of 2;

[0076] The activation layer includes a 1×1 convolution layer, a Sigmoid function and an upsampling layer connected in sequence. The 1×1 convolution layer is used to down-channel the output of the third convolution layer, and then obtain normalized features after passing through the Sigmoid function. The upsampling layer performs bilinear difference upsampling on the normalized features to obtain manual feature predictions.

[0077] The manual features are binary feature representations obtained by a manual feature calculation method (such as the LBP algorithm).

[0078] S3: Using the low-light / normal-light sample pair data set to train the low-light enhancement learning network to obtain a trained low-light enhancement learning network.

[0079] The specific steps include:

[0080] S3-1: Divide the low-light / normal-light sample pair dataset into a training dataset and a validation dataset;

[0081] S3-2: Setting training parameters, sending sample pairs to the low-light enhancement learning network for training, and optimizing the network by calculating the total loss;

[0082] S3-3: Test the low-light enhancement learning network using samples in the validation dataset and save the network parameters with the minimum error;

[0083] S3-4: Return to step S3-2 until the training is completed to obtain the final low-light enhancement learning network.

[0084] Among them, the calculation steps of total loss in S3-2 include:

[0085] S3-2-1: Calculate the supervision loss L of the intermediate output 1 :

[0086]

[0087] Among them, M i Indicates the amount of data loaded in each batch, X i represents a normal light picture, Indicates intermediate output;

[0088] S3-2-2: Calculate the supervised loss L of texture output 2 :

[0089]

[0090] Among them, X i Represents the data loaded in each batch, a i ∈{0,1} is the true label in the hand-crafted features, 0 represents the background area, 1 represents the texture area, and p i Represents texture output;

[0091] S3-2-3: Calculate the weighted sum of each loss as the total loss L t1 :

[0092] L t1 =L 1 +λ*L 2

[0093] Here, λ represents a hyperparameter.

[0094] S4: Establish a high-density target regression network, which includes the low-light enhancement learning network and the texture feature-aware density regression module. The texture feature-aware density regression module uses the manual feature prediction described in S2 to perform feature learning, and fuses and learns with the basic feature vector to obtain a density map prediction.

[0095] Among them, the texture feature perception density regression module in S4 specifically includes:

[0096] The fourth convolutional layer uses the output dark-light enhanced image in S2 as input and passes through two 3×3 convolutional layers with a dilation rate of 2 to obtain the dark-light enhanced feature vector;

[0097] A fusion layer, including a splicing layer and a 1×1 convolution layer, wherein the splicing splices the dark light enhancement feature vector and the output of the backbone network in S2 along the channel dimension, and the 1×1 convolution layer performs channel dimension reduction on the spliced ​​feature vector;

[0098] The fifth convolution layer includes two 3×3 convolution layers, a 1×1 convolution layer and an upsampling layer connected in sequence. The 3×3 convolution layer performs feature learning on the output of the fusion layer, and then performs channel dimension reduction through the 1×1 convolution layer. Finally, the features after channel dimension reduction are bilinearly upsampled to obtain a density map prediction.

[0099] S5: Initialize the parameters of the high-density target regression network in S4 using the low-light enhancement learning network trained in S3, and train the high-density target regression network using a high-density target data set to obtain a trained high-density target regression network.

[0100] The specific steps include:

[0101] S5-1: Divide the high-density target data set into a training data set and a validation data set;

[0102] S5-2: Setting training parameters, sending sample pairs into the high-density target regression network for training, and optimizing the network by calculating the total loss;

[0103] S5-3: Testing the high-density target regression network with samples in the validation data set, and saving the network parameters with the minimum error;

[0104] S5-4: Return to step S5-2 until the training is completed to obtain the final high-density target regression network.

[0105] Among them, the calculation steps of the total loss in S5-2 include:

[0106] S5-2-1: Calculate the supervised loss L of the density output 3 :

[0107]

[0108] Among them, M i Indicates the amount of data loaded in each batch, Y i represents the true label of the density map, Indicates density output;

[0109] S5-2-2: Calculate the supervised loss L of texture output 4 :

[0110]

[0111] Among them, X i Represents the data loaded in each batch, a i ∈{0,1} is the true label in the hand-crafted features, 0 represents the background area, 1 represents the texture area, and p i Represents texture output;

[0112] S5-2-3: Calculate the weighted sum of each loss as the total loss L t2 :

[0113] L Lt2 =L 3 +β*L 4

[0114] Here, β represents a hyperparameter.

[0115] In order to verify the performance of the network, the test data set can be further used to test the final network model and obtain the average error and mean square error for evaluation.

[0116] The above disclosure is only a preferred embodiment of the present invention, and cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A high-density target counting network modeling method driven by dark light enhancement, characterized in that , the method comprises the following steps: S1. Create low-light / normal-light sample pairs and high-density target datasets. Obtain low-light / normal-light samples by adjusting the camera exposure. Obtain high-density target datasets through monitoring equipment. Manually mark the targets in the images in the dataset. S2. Establish a low-light enhancement learning network, wherein the low-light enhancement learning network includes a low-light enhancement module, a backbone network, and a texture feature perception module. The low-light enhancement module performs illumination enhancement on the input image to obtain an illumination enhancement feature map. The backbone network performs basic feature learning based on the illumination enhancement feature map to obtain a basic feature vector. The texture feature perception module performs manual feature learning based on the basic feature vector to obtain manual feature prediction. S3, training the low-light enhancement learning network using the low-light / normal-light sample data set to obtain a trained low-light enhancement learning network; S4, establishing a high-density target regression network, wherein the high-density target regression network includes the low-light enhancement learning network and the texture feature perception density regression module, wherein the texture feature perception density regression module uses the manual feature prediction described in S2 to perform feature learning, and fuses and learns with the basic feature vector to obtain a density map prediction; S5, using the low-light enhancement learning network trained in S3 to initialize the parameters of the high-density target regression network in S4, and using the high-density target data set to train the high-density target regression network to obtain a trained high-density target regression network; The low light enhancement module in step S2 specifically includes: The first convolutional layer includes three 3×3 convolutional layers; The Fourier transform layer includes a two-dimensional Fourier transform and a component transform. The two-dimensional Fourier transform is used to perform a fast Fourier transform on the output of the first convolutional layer to obtain a real vector R(x) and an imaginary vector I(x). The component transform performs A(x) and P(x) operations on the real vector and the imaginary vector to obtain an amplitude (A) and an amplitude (P), respectively, where x represents an input feature. and P(x) = arctan[I(x) / R(x)]; The component enhancement layer includes an amplitude interaction layer and an amplitude learning layer. The amplitude interaction layer performs a product operation on the amplitude (A) through a randomly generated channel dimension parameter to obtain a refined amplitude feature. The amplitude learning layer performs feature learning on the amplitude (P) through two 3×3 convolutional layers to obtain an enhanced amplitude feature. The inverse Fourier transform layer is used to perform a fast inverse Fourier transform on the refined amplitude features and the enhanced amplitude features to obtain the Fourier enhanced features; The second convolutional layer learns the Fourier enhancement features through a 3×3 convolutional layer to obtain dark-light enhanced images; The backbone network in S2 is specifically the first 10 layers of the pre-trained VGG16 network, and the backbone network receives the output of the low-light enhancement module as input to obtain basic features; The texture feature perception module in S2 specifically includes: The third convolutional layer includes three 3×3 convolutional layers with a dilation rate of 2; The activation layer includes a 1×1 convolution layer, a Sigmoid function and an upsampling layer connected in sequence, wherein the 1×1 convolution layer is used to down-channel the output of the third convolution layer, and then obtain normalized features after passing through the Sigmoid function, and the upsampling layer performs bilinear difference upsampling on the normalized features to obtain manual feature prediction; The texture feature perception density regression module in S4 specifically includes: The fourth convolutional layer uses the output dark-light enhanced image in S2 as input and passes through two 3×3 convolutional layers with a dilation rate of 2 to obtain the dark-light enhanced feature vector; A fusion layer, including a splicing layer and a 1×1 convolution layer, wherein the splicing splices the dark light enhancement feature vector and the output of the backbone network in S2 along the channel dimension, and the 1×1 convolution layer performs channel dimension reduction on the spliced ​​feature vector; The fifth convolution layer includes two 3×3 convolution layers, a 1×1 convolution layer and an upsampling layer connected in sequence. The 3×3 convolution layer performs feature learning on the output of the fusion layer, and then performs channel dimension reduction through the 1×1 convolution layer. Finally, the features after channel dimension reduction are bilinearly upsampled to obtain a density map prediction.

2. According to the dark light enhanced driven high-density target counting network modeling method of claim 1, it is characterized by: The S3 specifically includes: S3-1: Divide the low-light / normal-light sample pair dataset into a training dataset and a validation dataset; S3-2: Setting training parameters, sending sample pairs to the low-light enhancement learning network for training, and optimizing the network by calculating the total loss; S3-3: Test the low-light enhancement learning network using samples in the validation dataset and save the network parameters with the minimum error; S3-4: Return to step S3-2 until the training is completed to obtain the final low-light enhancement learning network.

3. According to the dark light enhanced driven high-density target counting network modeling method of claim 1, it is characterized by: The S5 specifically includes: S5-1: Divide the high-density target data set into a training data set and a validation data set; S5-2: Setting training parameters, sending sample pairs into the high-density target regression network for training, and optimizing the network by calculating the total loss; S5-3: Testing the high-density target regression network with samples in the validation data set, and saving the network parameters with the minimum error; S5-4: Return to step S5-2 until the training is completed to obtain the final high-density target regression network.

4. According to the dark light enhanced driven high-density target counting network modeling method of claim 3, it is characterized by: The steps for calculating the total loss in S3-2 include: S3-2-1: Calculate the supervision loss L1 of the intermediate output: Among them, M i Indicates the amount of data loaded in each batch, X i represents a normal light picture, Indicates intermediate output; S3-2-2: Calculate the supervised loss L2 of texture output: Among them, X i represents the data loaded in each batch, ai∈{0,1} is the true label in the manual features, 0 represents the background area, 1 represents the texture area, and p i Represents texture output; S3-2-3: Calculate the weighted sum of each loss as the total loss L t1 : L t1 =L1+λ*L2 Here, λ represents a hyperparameter.

5. According to the dark light enhanced driven high-density target counting network modeling method of claim 4, it is characterized by: The steps for calculating the total loss in S5-2 include: S5-2-1: Calculate the supervised loss L3 of the density output: Among them, M i Indicates the amount of data loaded in each batch, Y i represents the true label of the density map, Indicates density output; S5-2-2: Calculate the supervised loss L4 of texture output: Among them, X i represents the data loaded in each batch, ai∈{0,1} is the true label in the manual features, 0 represents the background area, 1 represents the texture area, and p i Represents texture output; S5-2-3: Calculate the weighted sum of each loss as the total loss L t2 : L Lt2 =L3+β*L4 Here, β represents a hyperparameter.

6. The high-density target counting network modeling method driven by dark light enhancement according to claim 1 is characterized by: The manual features in S2 are binary feature representations obtained by a manual feature calculation method.

Images