A crowd counting system and method in foggy scenes based on image adaptive restoration

By adopting image adaptive recovery and feature fusion technology in foggy scenes, the problem of low population counting accuracy in foggy images is solved, and accurate counting of foggy populations is achieved, reducing work complexity and labor costs.

CN117935156BActive Publication Date: 2024-11-26YANSHAN UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410099935.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-24
Publication Date
2024-11-26
Estimated Expiration
2044-01-24

AI Technical Summary

Technical Problem

The prior art crowd counting methods in foggy days scenarios cannot fully recover potential information in the original fuzzy object, and rely on a large number of manual annotations or complex domain adaptation strategies, resulting in increased work complexity and reduced counting accuracy.

Method used

The system based on image adaptive recovery is adopted, including image adaptive recovery module, dual-branch feature guidance module, frequency domain attention fusion module and population density map regression module. Through defog and pixel enhancement, dual-branch feature extraction and frequency domain attention fusion , combined with Bayesian loss and fog-day robust loss terms, realize adaptive processing of fog-day images and crowd counting.

Benefits of technology

The accuracy of population counting in foggy days images is improved, the complexity of work is reduced, manual annotation and complex domain adaptation strategies are avoided, and the accuracy and reliability of population counting in foggy days is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117935156B_ABST
    Figure CN117935156B_ABST
Patent Text Reader

Abstract

The invention discloses a crowd counting system and method for foggy scenes based on image adaptive restoration, which belongs to the field of image processing, and comprises the following steps: obtaining a crowd image of a foggy scene to be processed; inputting the crowd image of the foggy scene to be processed into an image adaptive restoration module for clarity restoration; inputting the crowd image of the foggy scene to be processed and the cleared image into a double-branch feature guidance module for initial feature extraction; inputting the initial features of the foggy scene and the cleared initial features into a frequency domain attention fusion module for fusion to obtain high-level semantic features; performing density map regression on the high-level semantic features to obtain a crowd density map; summing up the pixel values ​​of the crowd density map to obtain a crowd counting result. The invention can effectively realize accurate counting of crowd images of foggy scenes, and perform targeted enhancement on foggy scenes without clearing labels to improve the robustness of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a system and method for counting people in foggy scenes based on image adaptive restoration. Background Art

[0002] In recent years, crowd counting technology has demonstrated its importance in multiple application scenarios, especially in the field of intelligent video surveillance. Many studies have focused on crowd counting analysis under clear weather conditions and have achieved remarkable results. However, under complex weather conditions such as foggy days, these methods developed for clear weather conditions often fail to achieve the expected results due to factors such as color degradation and reduced image contrast. At present, although there has been some progress in the study of crowd counting in foggy scenes, these methods mainly focus on enhancing the model's feature extraction ability in foggy scenes through attention mechanisms, thereby achieving end-to-end crowd number prediction. Most of these methods ignore the importance of targeted enhancement of images according to weather conditions, which may result in the inability to fully restore the potential information in the original blurred objects. In addition, for some other computer vision tasks performed in foggy scenes, such as foggy object detection, the object detection task is often decomposed into two independent stages: defogging and object detection. This approach either treats it as a domain adaptation problem. These techniques usually rely on a large amount of manual annotation in the defogging stage or rely on the design of complex domain adaptation strategies. This not only increases the complexity of the work, but also these methods may be limited in application scenarios that require higher-level reasoning capabilities, such as crowd counting tasks in foggy days. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a crowd counting system and method for foggy scenes based on image adaptive restoration. The foggy scene images are adaptively enhanced to reduce the negative impact of foggy weather factors on the foggy crowd counting performance, thereby achieving accurate prediction of the number of people in the foggy scene images.

[0004] To solve the above technical problems, the technical solution adopted by the present invention is: a crowd counting system for foggy scenes based on image adaptive restoration, comprising an image acquisition module for acquiring foggy scene images to be counted, after which an image acquisition module is provided with an image adaptive restoration module, a dual-branch feature guidance module, a frequency domain attention fusion module, a crowd density map regression module and a crowd counting module for summing the pixel values ​​of the crowd density map.

[0005] The further improvement of the technical solution of the present invention is that the specific composition structure of each module of the system is as follows:

[0006] Image adaptive restoration module: including a defogging module and a pixel enhancement module, wherein the defogging module includes four convolutional layers and element multiplication, element subtraction and element addition operations, wherein the size of the convolution kernel of the first convolutional layer is 1×1, the size of the convolution kernel of the second convolutional layer is 3×3, the size of the convolution kernel of the third convolutional layer is 5×5, and the size of the convolution kernel of the fourth convolutional layer is 7×7; the pixel enhancement module includes a parameter learning unit, a global feature adjustment unit and an image adjustment unit, wherein the parameter learning unit includes three convolutional layers and an average pooling layer, the convolution kernel size of the first convolutional layer is 7×7, and the convolution kernel sizes of the second and third convolutional layers are both 3×3; the global feature adjustment unit contains a fully connected layer; the image adjustment unit includes a backbone network for extracting basic features of the image, the backbone network is three convolutional layers, and the convolution kernel size is 1×1;

[0007] Dual-branch feature-guided module: includes a clarity counting submodule, a fog counting submodule and a fog robust loss term; the clarity counting submodule and the fog counting submodule are composed of the first 16 layers of two parallel VGG-19 networks and 5 maximum pooling layers. From input to output, they are two convolutional layers, one pooling layer, two convolutional layers, one pooling layer, four convolutional layers, one pooling layer, four convolutional layers, one pooling layer, four convolutional layers and one pooling layer, respectively. The convolution kernel size of each convolutional layer is 3×3, and the number of channels of each convolutional layer from input to output is 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, and 512, respectively. The step size of the five maximum pooling layers is 2; the robust loss term L HR The calculation formula is:

[0008]

[0009] In the formula, GAP represents the global average pooling operation;

[0010] Frequency domain attention fusion module: includes a 2D discrete cosine transform layer, an average pooling layer, two fully connected layers, a ReLU layer, and a Sigmoid layer;

[0011] Crowd density map regression module: It includes three convolutional layers. The convolution kernel size of the first convolution layer is 3×3, the convolution kernel size of the second convolution layer is 3×3, and the convolution kernel size of the third convolution layer is 1×1. The number of channels of the convolution layer from input to output is 256, 128, and 1 respectively.

[0012] A further improvement of the technical solution of the present invention is that: before the counting system is used, a plurality of foggy scene images and real crowd density maps are used to train the established counting system. During the training process, Bayesian loss is used as the crowd counting loss term of the real crowd density map and the generated crowd density map, and the foggy robust loss term L is used in combination. HR , forming a total loss function of L whole =L c +λL HR Make L whole Minimum loss

[0013] Training is performed for optimization purposes; the specific population count loss term is expressed as:

[0014]

[0015]

[0016]

[0017] Where Γ(·) is the distance function, E[C n ] represents the expected count of the nth person, D est represents the estimated density map, p(y n |x m ) represents the posterior probability of a person appearing in each pixel, N(x m: ;z n ,σ 2 1 2×2 ) means that in x m A two-dimensional Gaussian distribution evaluated at x m Represents the position distribution of each target in the image, y n represents the target label, M represents the number of pixels in each density map, and N represents the total number of people in each image training sample;

[0018] Establish the total loss function L of the crowd counting system in foggy scenes based on image adaptive restoration whole It is expressed as:

[0019] L whole =L c +λL HR

[0020] Specifically, the crowd counting loss term L c Used to measure the error distance between the estimated number of people and the actual number of people, the foggy robust loss term L HR is used to measure the distribution difference between generated features and features, and λ is the weight coefficient of the two loss terms, which is determined by model training.

[0021] The further improvement of the technical solution of the present invention is that the specific training method is to establish a crowd counting system for foggy scenes based on image adaptive restoration, the parameters of the dual-branch feature guidance module in the established system are initialized by the first sixteen convolutional layers of two parallel pre-trained VGG-19, and the parameters of other modules are randomly set to an initial value. After each time the counting system is used to obtain the generated crowd density map of the image sample, according to the total loss function L whole The real crowd density map and the generated crowd density map are compared to determine each iteration error of the counting system, the error of the counting system is back-propagated each time, the parameters of the crowd counting system are adjusted, and the crowd counting system for restored foggy images based on image adaptive restoration is optimized; the Adam optimization algorithm is used to update the parameters in each optimization iteration until the loss function converges to a smaller value, and the parameters and the trained counting system are saved.

[0022] The specific steps of a crowd counting method in foggy scenes based on image adaptive restoration are as follows:

[0023] Step 1: Obtain the crowd image of the foggy scene to be processed;

[0024] Step 2: Input the crowd image of the foggy scene to be processed into the image adaptive restoration module for clear restoration to obtain a clear image;

[0025] Step 3: Input the crowd image of the foggy scene to be processed and the cleared image into the dual-branch feature guidance module to extract initial features, and obtain the initial features of the foggy scene and the cleared initial features;

[0026] Step 4: Input the initial features of the foggy scene and the clear initial features into the frequency domain attention fusion module for fusion to obtain high-level semantic features;

[0027] Step 5: Input the high-level semantic features into the density map regression module to perform density map regression to obtain the crowd density map;

[0028] Step 6: Sum the pixel values ​​of the crowd density map to obtain the crowd counting result.

[0029] The further improvement of the technical solution of the present invention is that the specific steps of step 2 are as follows:

[0030] Step 2.1: Input the foggy scene image into the defogging module and then into the first convolutional layer to obtain the feature map f1;

[0031] Step 2.2: Input f1 into the second convolutional layer for feature extraction to obtain feature map f2;

[0032] Step 2.3: Link f1 and f2 through channels to obtain feature map f3;

[0033] Step 2.4: Input f3 into the third convolutional layer to obtain feature map f4;

[0034] Step 2.5: Channel link f3 and f4 to obtain feature map f5;

[0035] Step 2.6: Input f5 into the fourth convolutional layer to obtain feature map f6;

[0036] Step 2.7: Channel link f1, f2, f4 and f5 to obtain the estimated feature map θ;

[0037] Step 2.8: Take the estimated feature map and the foggy scene image as input, perform element-wise multiplication, element-wise subtraction, and element-wise addition operations to obtain the defogged image, specifically J(x)=θI(x)-θ+b, where J(x) is the defogged image, I(x) is the foggy scene image, and b is a constant with a default value of 1;

[0038] Step 2.9: Input the defogging image obtained in step 2.8 into the pixel enhancement submodule and then into the parameter learning unit to obtain the inherent attribute vector of foggy weather;

[0039] Step 2.10: Use the global feature adjustment unit to scale and move the fog weather inherent attribute vector to obtain the parameter learning result;

[0040] Step 2.11: Input the parameter learning result into the image adjustment unit to obtain a sharpened image.

[0041] The further improvement of the technical solution of the present invention is that the specific steps of step 3 are as follows:

[0042] Step 3.1: Input the cleared image into the dual-branch feature guidance module and then into the clearing counting submodule to obtain the cleared initial feature f c ;

[0043] Step 3.2: Input the crowd image of the foggy scene obtained in step 1 into the dual-branch feature guidance module and then into the foggy counting submodule to obtain the initial feature f of the foggy scene. h ;

[0044] Step 3.3: Clear the initial feature f c With the initial characteristics of the foggy scene f h Input into the foggy robust loss term to measure the two features f c and feature f h The distribution difference between them.

[0045] The further improvement of the technical solution of the present invention is that the specific steps of step 4 are as follows:

[0046] Step 4.1: Clear the initial feature f obtained in step 3.1 c And the initial feature f of the foggy scene obtained in step 3.2 h Input into the frequency domain attention fusion module;

[0047] Step 4.2: Clarify the initial feature f c The n parts are evenly divided in the channel dimension, expressed as Here i∈{0,1,…,n-1},H is f c The height of W is f c The width of each part is In addition, C must be divisible by n;

[0048] Step 4.3: For each Generate frequency domain components using a 2D discrete cosine transform layer

[0049] Step 4.4: and Perform element-wise multiplication to get F i Freq ;

[0050] Step 4.5: For all F i Freq After concatenation, the weight F is obtained through two fully connected layers, ReLU layer and Sigmoid layer. att-weight ;

[0051] Step 4.6: Set the weight F att-weight With the initial characteristics of the foggy scene f h Perform element-wise multiplication to obtain high-level semantic features.

[0052] Due to the adoption of the above technical scheme, the technical progress achieved by the present invention is as follows: by setting an image adaptive restoration module, a given foggy image can be restored in a targeted manner to improve the accuracy of counting under low visibility conditions; by setting a dual-branch feature guidance module, features of the restored image and the foggy image can be extracted; by setting a frequency domain attention fusion module, the features of the clear counting block can be used as weights to guide the features of the foggy counting block; the above modules combined with the counting method can effectively realize the accurate counting of people under foggy conditions, without the need to rely on a large amount of manpower costs for manual annotation of clear images, and without the need to design complex domain adaptation strategies, the complexity of the work is reduced, and the application of counting people under foggy conditions is more convenient to operate and the counting results are more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 It is a schematic flow chart of the counting method of the present invention;

[0055] Figure 2 It is a structural schematic diagram of the counting system of the present invention;

[0056] Figure 3 It is a structural schematic diagram of the image adaptive restoration module of the present invention;

[0057] Figure 4 It is a schematic diagram of the structure of the frequency domain attention fusion module of the present invention;

[0058] Figure 5 It is a schematic diagram of using the present invention to count the crowd in a crowd scene image taken in foggy weather; DETAILED DESCRIPTION

[0059] The present invention is further described in detail below in conjunction with embodiments:

[0060] like Figure 2 As shown, it is a structural schematic diagram of a crowd counting system for foggy scenes based on image adaptive restoration, which includes an image acquisition module for acquiring foggy scene images to be counted, an image adaptive restoration module for appropriately restoring a given foggy image to improve the accuracy of counting under low visibility conditions, a dual-branch feature guidance module for extracting features from the restored image and the foggy image, a frequency domain attention fusion module for guiding the features of the foggy counting block by using the features of the clear counting block as weights, a crowd density map regression module for performing density map regression on the fused features, and a crowd counting module for summing the pixel values ​​of the crowd density map.

[0061] The specific structure of each module is as follows:

[0062] like Figure 3The figure shows a schematic diagram of the structure of the image adaptive restoration module, which specifically includes a defogging submodule for defogging foggy images and a pixel enhancement submodule for enhancing images after defogging, wherein the defogging submodule includes four convolutional layers and element multiplication, element subtraction and element addition operations, wherein the size of the convolution kernel of the first convolutional layer is 1×1, the size of the convolution kernel of the second convolutional layer is 3×3, the size of the convolution kernel of the third convolutional layer is 5×5, and the size of the convolution kernel of the fourth convolutional layer is 7×7; the pixel submodule includes a parameter learning unit, a global feature adjustment unit and an image adjustment unit, wherein the parameter learning unit includes three convolutional layers and an average pooling layer, the convolution kernel size of the first convolutional layer is 7×7, and the convolution kernel sizes of the second and third convolutional layers are both 3×3; the global feature adjustment unit includes a fully connected layer; the image adjustment module includes a backbone network for extracting basic features of the image, the backbone network is three convolutional layers, and the convolution kernel size is 1×1;

[0063] Dual-branch feature-guided module: includes a clarity counting submodule, a fog counting submodule and a fog robust loss term; the clarity counting submodule consists of the first 16 layers of the VGG-19 network and 5 maximum pooling layers, and from input to output, there are two convolutional layers, one pooling layer, two convolutional layers, one pooling layer, four convolutional layers, one pooling layer, four convolutional layers, one pooling layer, four convolutional layers and one pooling layer, and the convolution kernel of each convolutional layer is 3×3. From input to output, each convolutional layer generates feature maps with channel numbers of 64, 64, 128, 128, 256, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512 and 512, and each of the pooling layers is a maximum pooling layer with a step size of 2.

[0064] The fog counting submodule consists of the first 16 layers of the VGG-19 network and 5 maximum pooling layers. From input to output, there are two convolutional layers, one pooling layer, two convolutional layers, one pooling layer, four convolutional layers, one pooling layer, four convolutional layers, one pooling layer, four convolutional layers and one pooling layer. The convolution kernel of each convolutional layer is 3×3. From input to output, each convolutional layer generates feature maps with channel numbers of 64, 64, 128, 128, 256, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512 and 512 respectively. Each of the pooling layers is a maximum pooling layer with a step size of 2.

[0065] Foggy weather robust loss term L HR The calculation formula is:

[0066]

[0067] In the formula, GAP represents the global average pooling operation;

[0068] like Figure 4 The figure shows the structure of the frequency domain attention fusion module, which includes a two-dimensional discrete cosine transform layer, an average pooling layer, two fully connected layers, a ReLU layer, and a Sigmoid layer.

[0069] Crowd density map regression module: It includes three convolutional layers. The convolution kernel size of the first convolution layer is 3×3, the convolution kernel size of the second convolution layer is 3×3, and the convolution kernel size of the third convolution layer is 1×1. The number of channels of the convolution layer from input to output is 256, 128, and 1 respectively.

[0070] The above-mentioned crowd counting system in foggy scenes needs to be trained before being used for crowd counting, so that the total loss function value is small. Several foggy scene images and real crowd density maps are used to train the established counting system. During the training process, Bayesian loss is used as the crowd counting loss term of the real crowd density map and the generated crowd density map, and the foggy robust loss term L is used. HR , the total loss function is L whole =L c +λL HR Make L whole The loss is minimized for the purpose of optimizing training; the specific crowd counting loss term is expressed as:

[0071]

[0072]

[0073]

[0074] Where Γ(·) is the distance function, E[C n ] represents the expected count of the nth person, D est represents the estimated density map, p(y n |x m ) represents the posterior probability of a person appearing in each pixel, N(x m: ;z n ,σ 2 1 2×2 ) means that in x m A two-dimensional Gaussian distribution evaluated at x m Represents the position distribution of each target in the image, y n represents the target label, M represents the number of pixels in each density map, and N represents the total number of people in each image training sample;

[0075] Establish the total loss function L of the crowd counting system in foggy scenes based on image adaptive restoration whole It is expressed as:

[0076] L whole =Lc +λL HR

[0077] Specifically, the crowd counting loss term L c Used to measure the error distance between the estimated number of people and the actual number of people, the foggy robust loss term L HR to measure the distribution difference between generated features and features, λ is the weight coefficient of the two loss terms, which is determined by model training. The specific training method is to establish a crowd counting system for foggy scenes based on image adaptive restoration. The parameters of the dual-branch feature guidance module in the established system are initialized by the first sixteen convolutional layers of two parallel pre-trained VGG-19, and the parameters of other modules are randomly set to an initial value. After each time the counting system is used to obtain the generated crowd density map of the image sample, the error of each iteration of the counting system is determined by comparing the real crowd density map with the generated crowd density map according to the total loss function, and the error of the counting system is back-propagated each time, and the parameters of the crowd counting system are adjusted to optimize the crowd counting system for restored foggy images based on image adaptive restoration; the Adam optimization algorithm is used to update the parameters in each optimization iteration until the loss function converges to a smaller value, and the parameters and the trained counting system are saved. The trained crowd counting system for foggy scene images can be used to count the crowd of any input foggy scene image. The crowd counting result can be obtained by directly inputting any single image into the crowd counting system in conjunction with the crowd counting method.

[0078] like Figure 2 The flowchart of the crowd counting method in foggy scene based on image adaptive restoration using the above counting system is shown, and the specific steps are as follows:

[0079] Step 1: Obtain the crowd image of the foggy scene to be processed;

[0080] Step 2: Input the crowd image of the foggy scene to be processed into the image adaptive restoration module for clear restoration to obtain a clear image;

[0081] Step 2.1: Input the foggy scene image into the defogging module and then into the first convolutional layer to obtain the feature map f1;

[0082] Step 2.2: Input f1 into the second convolutional layer for feature extraction to obtain feature map f2;

[0083] Step 2.3: Link f1 and f2 through channels to obtain feature map f3;

[0084] Step 2.4: Input f3 into the third convolutional layer to obtain feature map f4;

[0085] Step 2.5: Channel link f3 and f4 to obtain feature map f5;

[0086] Step 2.6: Input f5 into the fourth convolutional layer to obtain feature map f6;

[0087] Step 2.7: Channel link f1, f2, f4 and f5 to obtain the estimated feature map θ;

[0088] Step 2.8: Take the estimated feature map and the foggy scene image as input, perform element-wise multiplication, element-wise subtraction, and element-wise addition operations to obtain the defogged image, specifically J(x) = θI(x) - θ + b, where J(x) is the defogged image, I(x) is the foggy scene image, and b is a constant with a default value of 1;

[0089] Step 2.9: Input the defogging image obtained in step 2.8 into the pixel enhancement submodule and then into the parameter learning unit to obtain the inherent attribute vector of foggy weather;

[0090] Step 2.10: In order to adjust the overall appearance of the image while maintaining the image content, the global feature adjustment unit is used to scale and move the foggy weather intrinsic attribute vector to obtain the parameter learning result;

[0091] Step 2.11: Input the parameter learning result into the image adjustment unit to obtain a sharpened image.

[0092] Step 3: Input the crowd image of the foggy scene to be processed and the cleared image into the dual-branch feature guidance module to extract initial features, and obtain the initial features of the foggy scene and the cleared initial features;

[0093] Step 3.1: Input the cleared image into the dual-branch feature guidance module and then into the clearing counting submodule to obtain the cleared initial feature f c ;

[0094] Step 3.2: Input the crowd image of the foggy scene obtained in step 1 into the dual-branch feature guidance module and then into the foggy counting submodule to obtain the initial feature f of the foggy scene. h ;

[0095] Step 3.3: Clear the initial feature f c With the initial characteristics of the foggy scene f h Input into the foggy robust loss term to measure the two features f c and feature f h The distribution difference between them.

[0096] Step 4: Input the initial features of the foggy scene and the clear initial features into the frequency domain attention fusion module for fusion to obtain high-level semantic features;

[0097] Step 4.1: Clear the initial feature f obtained in step 3.1c And the initial feature f of the foggy scene obtained in step 3.2 h Input into the frequency domain attention fusion module;

[0098] Step 4.2: Clarify the initial feature f c The n parts are evenly divided in the channel dimension, expressed as Here i∈{0,1,…,n-1},H is f c The height of W is f c The width of each part is In addition, C must be divisible by n;

[0099] Step 4.3: For each Generate frequency domain components using 2D discrete cosine transform layers

[0100] Step 4.4: and Perform element-wise multiplication to get F i Freq ;

[0101] Step 4.5: For all F i Freq After concatenation, the weight F is obtained through two fully connected layers, ReLU layer and Sigmoid layer. att-weight ;

[0102] Step 4.6: Set the weight F att-weight With the initial characteristics of the foggy scene f h Perform element-wise multiplication to obtain high-level semantic features.

[0103] Step 5: Input the high-level semantic features into the density map regression module to perform density map regression to obtain the crowd density map;

[0104] Step 6: Sum the pixel values ​​of the crowd density map to obtain the crowd counting result.

[0105] When the present invention is used to count the crowd in a crowd picture taken in foggy weather, Figure 5 As shown, any crowd image taken in foggy weather is directly input into the crowd counting system of foggy weather scene based on image adaptive restoration, and the crowd counting result can be obtained by combining with the counting method. The present invention can effectively realize the accurate counting of crowd images in foggy weather, save the manual labeling work, liberate the labor cost, and perform targeted enhancement on foggy weather images without clear labels to improve the robustness of the model.

[0106] The embodiments described above are merely descriptions of preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A crowd counting system for foggy scenes based on image adaptive restoration, comprising an image acquisition module for acquiring foggy scene images to be counted, characterized in that: After the image acquisition module, there are sequentially connected image adaptive restoration module, dual-branch feature guidance module, frequency domain attention fusion module, crowd density map regression module and crowd counting module for summing the pixel values ​​of the crowd density map; the specific composition structure of each module of the system is as follows: Image adaptive restoration module: includes a defogging module and a pixel enhancement module. The defogging module includes four convolutional layers and element multiplication, element subtraction and element addition operations. The size of the convolution kernel of the first convolutional layer is 1×1, the size of the convolution kernel of the second convolutional layer is 3×3, the size of the convolution kernel of the third convolutional layer is 5×5, and the size of the convolution kernel of the fourth convolutional layer is 7×7. The pixel enhancement submodule includes a parameter learning unit, a global feature adjustment unit, and an image adjustment unit. The parameter learning unit includes three convolutional layers and an average pooling layer. The convolution kernel size of the first convolutional layer is 7×7, and the convolution kernel sizes of the second and third convolutional layers are both 3×3. The global feature adjustment unit contains a fully connected layer; the image adjustment unit includes a backbone network for extracting basic features of the image. The backbone network consists of three convolutional layers, and the convolution kernel size is 1×1. Dual-branch feature-guided module: includes a clarity counting submodule, a fog counting submodule and a fog robust loss term; the clarity counting submodule and the fog counting submodule are composed of the first 16 layers of two parallel VGG-19 networks and 5 maximum pooling layers. From input to output, they are two convolutional layers, one pooling layer, two convolutional layers, one pooling layer, four convolutional layers, one pooling layer, four convolutional layers, one pooling layer, four convolutional layers and one pooling layer, respectively. The convolution kernel size of each convolutional layer is 3×3, and the number of channels of each convolutional layer from input to output is 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, and 512, respectively. The step size of the five maximum pooling layers is 2; the robust loss term L HR The calculation formula is: In the formula, GAP represents the global average pooling operation; Frequency domain attention fusion module: includes a 2D discrete cosine transform layer, an average pooling layer, two fully connected layers, a ReLU layer, and a Sigmoid layer; Crowd density map regression module: including three convolution layers, the convolution kernel size of the first convolution layer is 3×3, the convolution kernel size of the second convolution layer is 3×3, and the convolution kernel size of the third convolution layer is 1×1. The number of channels of the convolution layer from input to output is 256, 128 and 1 respectively; before using the counting system, the counting system is trained with several foggy scene images and real crowd density maps. During the training process, Bayesian loss is used as the crowd counting loss item of the real crowd density map and the generated crowd density map, and the foggy robust loss item L is used in combination. HR , forming a total loss function of L whole , so that L whole The loss is minimized for the purpose of optimizing training; the specific crowd counting loss term is expressed as: Where Γ(·) is the distance function, E[cn] represents the expected count of the nth person, and D est represents the estimated density map, p(y n |x m ) represents the posterior probability of a person appearing in each pixel, N(x m ;z n ,σ 2 1 2×2 ) means that in x m A two-dimensional Gaussian distribution evaluated at x m Represents the position distribution of each target in the image, y n represents the target label, M represents the number of pixels in each density map, and N represents the total number of people in each image training sample; Establish the total loss function L of the crowd counting system in foggy scenes based on image adaptive restoration whole It is expressed as: THE whole =L c +λL HR Specifically, the crowd counting loss term L c Used to measure the error distance between the estimated number of people and the actual number of people, the foggy robust loss term L HR to measure the distribution difference between generated features and features, λ is the weight coefficient of the two loss terms, which is determined by model training; the specific training method is to establish a crowd counting system for foggy scenes based on image adaptive restoration. The parameters of the dual-branch feature guidance module in the established system are initialized by the first sixteen convolutional layers of two parallel pre-trained VGG-19, and the parameters of other modules are randomly set to an initial value. After each use of the counting system to obtain the generated crowd density map of the image sample, according to the total loss function L whole The real crowd density map and the generated crowd density map are compared to determine each iteration error of the counting system, the error of the counting system is back-propagated each time, the parameters of the crowd counting system are adjusted, and the crowd counting system for restored foggy images based on image adaptive restoration is optimized; the Adam optimization algorithm is used to update the parameters in each optimization iteration until the loss function converges to a smaller value, and the parameters and the trained counting system are saved.

2. A crowd counting method in foggy scenes based on image adaptive restoration, characterized by: Applied to the counting system described in claim 1, the specific steps are as follows: Step 1: Obtain the crowd image of the foggy scene to be processed; Step 2: Input the crowd image of the foggy scene to be processed into the image adaptive restoration module for clear restoration to obtain a clear image; Step 3: Input the crowd image of the foggy scene to be processed and the cleared image into the dual-branch feature guidance module to extract initial features, and obtain the initial features of the foggy scene and the cleared initial features; Step 4: Input the initial features of the foggy scene and the clear initial features into the frequency domain attention fusion module for fusion to obtain high-level semantic features; Step 5: Input the high-level semantic features into the density map regression module to perform density map regression to obtain the crowd density map; Step 6: Sum the pixel values ​​of the crowd density map to obtain the crowd counting result.

3. The method for counting people in foggy scenes based on image adaptive restoration according to claim 2, characterized in that: Step 2 The specific steps are as follows: Step 2.1: Input the foggy scene image into the defogging module and then into the first convolutional layer to obtain the feature map f1; Step 2.2: Input f1 into the second convolutional layer for feature extraction to obtain feature map f2; Step 2.3: Link f1 and f2 through channels to obtain feature map f3; Step 2.4: Input f3 into the third convolutional layer to obtain feature map f4; Step 2.5: Channel link f3 and f4 to obtain feature map f5; Step 2.6: Input f5 into the fourth convolutional layer to obtain feature map f6; Step 2.7: Channel link f1, f2, f4 and f5 to obtain the estimated feature map θ; Step 2.8: Take the estimated feature map θ and the foggy scene image as input, perform element-wise multiplication, element-wise subtraction, and element-wise addition operations to obtain the defogged image, specifically J(x)=θI(x)-θ+b, where J(x) is the defogged image, I(x) is the foggy scene image, and b is a constant with a default value of 1; Step 2.9: Input the defogging image obtained in step 2.8 into the pixel enhancement submodule and then into the parameter learning unit to obtain the inherent attribute vector of foggy weather; Step 2.10: Use the global feature adjustment unit to scale and move the fog weather inherent attribute vector to obtain the parameter learning result; Step 2.11: Input the parameter learning result into the image adjustment unit to obtain a sharpened image.

4. The method for counting people in foggy scenes based on image adaptive restoration according to claim 2, characterized in that: Step 3 The specific steps are as follows: Step 3.1: Input the cleared image into the dual-branch feature guidance module and then into the clearing counting submodule to obtain the cleared initial feature f c ; Step 3.2: Input the crowd image of the foggy scene obtained in step 1 into the dual-branch feature guidance module and then into the foggy counting submodule to obtain the initial feature f of the foggy scene. h ; Step 3.3: Clear the initial feature f c With the initial characteristics of the foggy scene f h Input into the foggy robust loss term to measure the two features f c and feature f h The distribution difference between them.

5. The method for counting people in foggy scenes based on image adaptive restoration according to claim 2, characterized in that: Step 4 The specific steps are as follows: Step 4.1: Clear the initial feature f obtained in step 3.1 c And the initial feature f of the foggy scene obtained in step 3.2 h Input into the frequency domain attention fusion module; Step 4.2: Clarify the initial feature f c The n parts are evenly divided in the channel dimension, denoted as f c 0 , f c 1 ,…,f c n-1 , here i∈{0,1,…,n-1},H is f c The height of W is f c The width of each part is In addition, C must be divisible by n; Step 4.3: For each f c i , using a 2D discrete cosine transform layer to generate frequency domain components Step 4.4: With f c i Perform element-wise multiplication to get F i Freq ; Step 4.5: For all F i Freq After concatenation, the weight F is obtained through two fully connected layers, ReLU layer and Sigmoid layer. att-weight ; Step 4.6: Set the weight F att-weight With the initial characteristics of the foggy scene f h Perform element-wise multiplication to obtain high-level semantic features.

Citation Information

Patent Citations

  • Deep learning-based image high-density population counting method

    CN107301387A

  • High-resolution dense target counting method based on convolutional neural network

    CN113239904A

  • Cross-modal crowd counting method and system

    CN113538400A

  • Method for counting crowds in foggy day scene

    CN114399728A

  • Video crowd counting method and device, terminal equipment and storage medium

    CN116665117A