A Target Localization and Counting Method Based on Point Domain Feature Learning

Abstract

This invention relates to a target localization and counting method based on point domain feature learning. The method includes: inputting the image to be detected into a backbone network to extract a three-level feature pyramid comprising top-level features, intermediate-level features, and bottom-level features; inputting the three-level feature pyramid into a multilayer perceptron module to capture the global long-distance dependence of its top-level features and obtain global information features; inputting the three-level feature pyramid into a visual centering mechanism module to aggregate local region features of its top-level features, obtain local information features, and connect them with global information features along the channel dimension to obtain mapped top-level features; then, combining these with intermediate-level and bottom-level features to form a mapped three-level feature pyramid; inputting this pyramid into a global centralization adjustment module; outputting an initial feature map to a multi-perceptron module; extracting channel features and spatial features and concatenating them to obtain a multi-perceptron feature map; then, passing through a regression head to output the predicted target count of the image to be detected; and finally, passing through a classification head to output the predicted target localization of the image to be detected.

Description

Technical Field

[0001] This invention relates to the field of target localization technology, and in particular to a target localization and counting method based on point neighborhood feature learning. Background Technology

[0002] In recent years, an increasing number of researchers have focused on the crucial role of automated crowd counting in public surveillance and intelligent transportation systems. The analysis of crowd behavior and traffic density significantly impacts the efficiency of public transportation. In smart city construction, the ability of intelligent transportation systems to accurately and efficiently acquire crowd behavior and vehicle information from public surveillance is a critical element in achieving effective traffic planning. Crowd counting and vehicle counting are fundamental tasks in crowd behavior analysis and traffic flow analysis. For example, crowd counting has been widely applied in public safety, video surveillance, and other fields. There are generally two methods to solve the target counting problem: computational population density map, where the input is an image and the output is the total number of people in the image; and prediction density map, where the input is an image and the output is a predicted population density map, which is then integrated to obtain the total number of people.

[0003] In earlier years, researchers studied the object counting problem using detection-based or regression-based methods. Detection-based methods treat the entire object in an image as a detectable feature and then easily compute the detection result. These methods use image processing techniques to extract handcrafted features of the object; however, this assumption does not always hold true in studies, especially when objects are very dense and severely occluded. Regression-based methods use global features, such as texture and gradients, and then use machine learning methods to learn a regression model, such as support vector machines or ridge regression, to establish a mapping between handcrafted features and the number of people in the image. Detection-based methods suffer from problems such as severe occlusion and perspective, and their performance is significantly negatively affected on high-density images. Regression-based methods use global features for counting, solving problems such as image occlusion and perspective, but they perform poorly on high-density object images.

[0004] With the rapid development of deep learning, more and more computer vision researchers are focusing on convolutional neural networks (CNNs). CNNs possess strong automatic feature extraction capabilities, and recent research in object counting has demonstrated that they outperform all traditional methods. Traditional object counting methods typically output the total number of people in an image, while CNN-based methods usually predict density maps, which contain density information unavailable in traditional methods. However, in the process of generating density maps using Gaussian kernels, the location information of the target is lost, posing a significant challenge to target localization. Most existing methods for target localization rely on bounding box-annotated datasets, making many purely point-annotated counting datasets unusable for target localization tasks, such as Shanghai Tech partA and partB. Furthermore, simultaneously achieving target counting and localization requires designing two separate methods—one for counting and one for localization—greatly increasing the workload for researchers.

[0005] In summary, existing target counting methods learn features through density map learning. The learned knowledge is not visually intuitive knowledge, but rather a mapping between the density map and the original image. When performing localization based on this mapping, the predicted density map is usually parsed, and the local maximum value in the density map is regarded as the target localization point. This is also a non-visually intuitive prediction, which is quite sensitive to the complexity of the image. For images with complex backgrounds, overlapping targets, or occlusions, the counting results may be inaccurate. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the knowledge learned in the prior art is not visual intuitive knowledge, but a mapping between density map and original image, which leads to large prediction error and low efficiency in the presence of occlusion, overlap or background noise.

[0007] To address the aforementioned technical problems, this invention provides a target localization and counting method based on point neighborhood feature learning, comprising:

[0008] The image to be detected is input into the backbone network, and a three-level feature pyramid of the image to be detected is extracted and output; the three-level feature pyramid includes top-level features, middle-level features and bottom-level features;

[0009] The three-level feature pyramid is input into the multilayer perceptron module to capture the global long-distance dependency of its top-level features and obtain global information features.

[0010] The three-level feature pyramid is input into the visual center mechanism module, which aggregates the local region features of its top-level features to obtain local information features.

[0011] The global information features and the local information features are connected together along the channel dimension to obtain the top-level mapping features, which, together with the intermediate-level features and the bottom-level features, form a three-level mapping feature pyramid.

[0012] The mapped three-level feature pyramid is input into the global centralized adjustment module, which maps it to the top-level features, middle-level features and bottom-level features, and splices them along the channel dimension to output the initial feature map;

[0013] The initial feature map is input into the multi-sensor module, and the channel features and spatial features of the initial feature map are extracted and concatenated to obtain the multi-sensor feature map;

[0014] The multi-sensory feature map is input into the classification and regression module. The regression head outputs the predicted number of targets in the image to be detected, and the classification head outputs the predicted target location in the image to be detected.

[0015] In one embodiment of the present invention, a smoothing block is further included, which is set after the backbone network, so that the three-level feature pyramid output by the backbone network passes through the convolution block, batch normalization layer and activation function layer in sequence along the forward propagation direction, and outputs a smoothed three-level feature pyramid, which is then input to the multilayer perceptron module and the visual center mechanism module.

[0016] In one embodiment of the present invention, the step of inputting the image to be detected into the backbone network and extracting and outputting the three-level feature pyramid of the image to be detected includes:

[0017] The image to be detected is input into the backbone network and passes through the Embedding layer, the first stage block, the Merging layer, the second stage block, the Merging layer, the third stage block, the Merging layer, and the fourth stage in sequence along the forward propagation direction. The output of the second stage block is used as the bottom layer feature, the output of the third stage block is used as the middle layer feature, and the output of the fourth stage block is used as the top layer feature, thus forming a three-level feature pyramid of the image to be detected.

[0018] The first stage block comprises three cascaded basic structural units along the forward propagation direction; the second stage block comprises four cascaded basic structural units along the forward propagation direction; the third stage block comprises eighteen cascaded basic structural units along the forward propagation direction; and the fourth stage block comprises three cascaded basic structural units along the forward propagation direction.

[0019] The basic structural unit, along the forward propagation direction, sequentially includes a Pconv network, a convolutional layer with a kernel size of 1, a batch normalization layer, an activation function layer, and a convolutional layer with a kernel size of 3.

[0020] The Embedding layer is a convolutional layer with a kernel size of 4 and a stride of 4;

[0021] The Merging layer is a convolutional layer with a kernel size of 2 and a stride of 2.

[0022] In one embodiment of the present invention, the step of inputting the three-level feature pyramid into a multilayer perceptron module to capture the global long-range dependency of its top-level features and obtain global information features includes:

[0023] After group normalization, the three-level feature pyramid is input into a depthwise separable convolutional unit for depthwise convolution to obtain the convolutional feature map. Represented as:

[0024]

[0025] The convolutional feature map is input into the channel MLP unit, and after group normalization, the channel MLP operation is performed to output the global information feature MLP(X). in ), represented as:

[0026]

[0027] Among them, X in GN() represents a three-level feature pyramid; DConv() represents a group normalization; CMLP() represents a channel multilayer perceptron.

[0028] In one embodiment of the present invention, the step of inputting the three-level feature pyramid into the visual center mechanism module, aggregating the local region features of its top-level features, and obtaining local information features includes:

[0029] The three-level feature pyramid is input into the visual center mechanism module. After being combined and encoded by a set of preset convolutional layers, it passes through convolutional layers, batch normalization layers, and activation function layers to output the encoded features.

[0030] The encoded features are input into the inherent codebook B = b1, b2, ..., b of the visual center mechanism module. k ,…,b K In the process, a set of scaling factors S = s1, s2, ..., s is used. k ,…,s K Encode features Learnable visual encoding mapped to the intrinsic codebook b k Obtain the mapping encoding information e of each pixel in the encoded features. k , is represented as:

[0031]

[0032] Using φ to fuse the mapping encoding information e of all pixels in the encoding feature k The fused coding information e of the coding features is obtained and represented as:

[0033]

[0034] The fused encoded information is then passed through a fully connected layer and a convolutional layer before being output, and then combined with the three-level feature pyramid X. in Perform channel multiplication to obtain the local region feature Z, represented as:

[0035]

[0036] The local region features are compared with the three-level feature pyramid X. in Perform channel addition to obtain local information features LVC(X) in ), represented as: LVC(X in ) = X in ⊕Z;

[0037] in, Representing encoded features The i-th pixel in the array, b k This represents the k-th learnable visual code in the intrinsic codebook. This represents the positional information of each pixel relative to a learnable visual code; the total spatial number of the three-level feature pyramid is K = H × W, where H and W refer to the height and width of the three-level feature pyramid, respectively; φ is the fusion function, which includes BN layers, ReLU layers, and mean layers; X in Represents a three-level feature pyramid; Conv 1×1 This represents a 1×1 convolutional layer, where δ(·) refers to the Sigmoid function. Indicates channel multiplication; This indicates channel addition.

[0038] In one embodiment of the present invention, the mapping three-level feature pyramid is represented as follows:

[0039] X = cat(MLP(X) in ),LVC(X in ));

[0040] Among them, X in Represents a three-level feature pyramid; MLP(X) in ) represents global information features, LVC(X) in ) represents local information features, and cat represents the concatenation of feature maps along the channel dimension.

[0041] In one embodiment of the present invention, obtaining the channel features of the initial feature map includes:

[0042] The initial feature map F0 is input into a multi-scale feature extraction network to obtain multiple feature maps of different scales, which are then concatenated to obtain a multi-scale feature map F.

[0043] The initial channel features C(F) of the multi-scale feature map are extracted using a channel feature extraction network. initial , is represented as:

[0044] C(F) initial =cat(δ(L(r(L(v,W0),W1)))(F));

[0045] The initial channel feature C(F) initial The initial feature map is obtained by element-wise multiplying it with the multi-scale feature map F to obtain the channel features C(F);

[0046] Where F represents the multi-scale feature map, v represents the feature vector after the average pooling layer, L is the linear layer, W0 and W1 are the parameters of the two linear layers respectively, r represents the ReLU activation function; δ(·) represents the Sigmoid function; and cat represents the concatenation of feature maps along the channel dimension.

[0047] In one embodiment of the present invention, the multi-scale feature extraction network includes four parallel convolutional units, wherein:

[0048] The first convolutional unit includes an average pooling layer and a first dilated convolutional layer sequentially along the forward propagation direction;

[0049] The second convolutional unit includes, in sequence along the forward propagation direction, a second dilated convolutional layer, a third dilated convolutional layer, and a fourth dilated convolutional layer;

[0050] The third convolutional unit includes, in sequence, the fifth and sixth dilated convolutional layers along the forward propagation direction;

[0051] The fourth convolutional unit includes the seventh dilated convolutional layer.

[0052] In one embodiment of the present invention, the acquisition of spatial features of the initial feature map includes:

[0053] The initial feature map F0 is input into the spatial feature extraction network to obtain the initial spatial features S(F0) of the initial feature map F0. initial , is represented as:

[0054] S(F0) initial =δ(Conv 7×7 (MAP(F0),θ))⊙F0;

[0055] The initial spatial feature S(F0) initialThe spatial feature S(F0) is obtained by element-wise multiplication with the initial feature map F0;

[0056] Where F0 represents the initial feature map, Conv 7×7 denoted as a 7×7 convolutional layer, MAP represents max pooling and average pooling, and θ represents the parameters of the spatial feature extraction network.

[0057] In one embodiment of the present invention, the step of inputting the multi-sensor feature map into a classification and regression module, outputting the predicted number of targets in the image to be detected through a regression head, and outputting the predicted target location in the image to be detected through a classification head includes:

[0058] The multi-sensor feature map is input into the regression head of the classification and regression module, and after passing through four 3×3 convolutional layers, the predicted target number is output.

[0059] The multi-sensory feature map is input into the classification head of the classification and regression module, and after passing through four 3×3 convolutional layers and the Sigmoid function, the predicted target location is output.

[0060] The technical solution of the present invention has the following advantages compared with the prior art:

[0061] The target localization and counting method based on point-domain feature learning described in this invention abandons the traditional density map-based learning approach. Instead, it learns direct visual knowledge based on the image to be detected, directly acquiring the image features of the image to be detected, and performing target localization and counting based on these image features. This invention uses a backbone network to construct a three-level feature pyramid of the image to be detected, employs a lightweight MLP architecture to capture the global long-distance dependencies of the top-level features of the three-level feature pyramid, and uses a learnable visual center mechanism to aggregate local region features of the three-level feature pyramid, constructing a mapped three-level feature pyramid. Global centralization adjustment is used to concatenate the mapped top-level, intermediate-level, and bottom-level features along the channel dimension, allowing the intermediate-level and bottom-level features of the mapped three-level feature pyramid to simultaneously benefit from the visual centralization information of the mapped top-level features, achieving multi-scale feature fusion, thereby improving the detection performance and classification accuracy of the classification and regression module; furthermore, global centralization adjustment can effectively reduce computational load and the number of parameters. After obtaining the initial feature map of the image to be detected, the multi-sensor module is used to extract channel features and spatial features and then concatenate them to obtain a multi-sensor feature map. After passing through the classification and regression module, the predicted target location and the predicted target number are obtained. The use of the multi-sensor module reduces the calculation error of point neighborhood feature learning in low-density areas and improves the target location accuracy. The classification and regression module is used to simultaneously realize target counting, which improves the generalization ability and robustness of the model. Attached Figure Description

[0062] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein...

[0063] Figure 1 This is a flowchart of the steps of the target localization and counting method based on point domain feature learning provided by the present invention;

[0064] Figure 2 This is a schematic diagram of the basic structural unit provided by the present invention;

[0065] Figure 3 This is a network model structure diagram of the target localization and counting method based on point domain feature learning provided by the present invention. Detailed Implementation

[0066] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0067] Reference Figure 1 The flowchart shown illustrates the steps of the target localization and counting method based on point neighborhood feature learning of the present invention. The specific steps include:

[0068] S101: Input the image to be detected into the backbone network and extract the three-level feature pyramid of the image to be detected; the three-level feature pyramid includes top-level features, middle-level features and bottom-level features;

[0069] S102: Input the three-level feature pyramid into the multilayer perceptron module to capture the global long-distance dependency of its top-level features and obtain global information features;

[0070] S103: Input the three-level feature pyramid into the visual center mechanism module, aggregate the local region features of its top-level features, and obtain local information features;

[0071] S104: Connect the global information features and the local information features along the channel dimension to obtain the top-level mapping features, which, together with the intermediate-level features and the bottom-level features, form a three-level mapping feature pyramid.

[0072] S105: Input the mapped three-level feature pyramid into the global centralized adjustment module, map it to the top-level features, middle-level features and bottom-level features, and stitch them together along the channel dimension to output the initial feature map;

[0073] S106: Input the initial feature map into the multi-sensor module, extract the channel features and spatial features of the initial feature map and concatenate them to obtain the multi-sensor feature map;

[0074] S107: Input the multi-sensor feature map into the classification and regression module, including:

[0075] The multi-sensor feature map is input into the regression head of the classification and regression module, and after passing through four 3×3 convolutional layers, the predicted target number is output.

[0076] The multi-sensory feature map is input into the classification head of the classification and regression module, and after passing through four 3×3 convolutional layers and the Sigmoid function, the predicted target location is output.

[0077] Specifically, in this embodiment, a smoothing block is further provided after the backbone network. This causes the three-level feature pyramid output by the backbone network to sequentially pass through a convolutional block, a batch normalization layer, and an activation function layer along the forward propagation direction, resulting in a smoothed three-level feature pyramid that is then input to the multilayer perceptron module and the visual center mechanism module. The smoothing block consists of a 7x7 convolution with an output channel size of 256, a batch normalization layer (BN), and an activation function layer, used to smooth the features of the three-level feature pyramid.

[0078] Specifically, in step S101, the extraction of the three-level feature pyramid includes:

[0079] The image to be detected is input into the backbone network and passes through the Embedding layer, the first stage block, the Merging layer, the second stage block, the Merging layer, the third stage block, the Merging layer, and the fourth stage in sequence along the forward propagation direction. The output of the second stage block is used as the bottom layer feature, the output of the third stage block is used as the middle layer feature, and the output of the fourth stage block is used as the top layer feature, thus forming a three-level feature pyramid of the image to be detected.

[0080] The first stage block comprises three cascaded basic structural units along the forward propagation direction; the second stage block comprises four cascaded basic structural units along the forward propagation direction; the third stage block comprises eighteen cascaded basic structural units along the forward propagation direction; and the fourth stage block comprises three cascaded basic structural units along the forward propagation direction.

[0081] Reference Figure 2 As shown, the basic structural unit, along the forward propagation direction, includes, in sequence, a Pconv network, a convolutional layer with a kernel size of 1, a batch normalization layer, an activation function layer, and a convolutional layer with a kernel size of 3.

[0082] The Embedding layer is a convolutional layer with a kernel size of 4 and a stride of 4;

[0083] The Merging layer is a convolutional layer with a kernel size of 2 and a stride of 2.

[0084] Specifically, in step S102, the acquisition of global information features includes:

[0085] After group normalization, the three-level feature pyramid is input into a depthwise separable convolutional unit for depthwise convolution to obtain the convolutional feature map. Represented as:

[0086]

[0087] The convolutional feature map is input into the channel MLP unit, and after group normalization, the channel MLP operation is performed to output the global information feature MLP(X). in ), represented as:

[0088]

[0089] Among them, X in GN() represents a three-level feature pyramid; DConv() represents a group normalization; CMLP() represents a channel multilayer perceptron.

[0090] Specifically, in step S103, the acquisition of local information features includes:

[0091] The three-level feature pyramid is input into the visual center mechanism module. After being combined and encoded by a set of preset convolutional layers, it passes through convolutional layers, batch normalization layers, and activation function layers to output the encoded features.

[0092] The encoded features are input into the inherent codebook B = b1, b2, ..., b of the visual center mechanism module. k ,…,b K In the process, a set of scaling factors S = s1, s2, ..., s is used. k ,…,s K Encode features Learnable visual encoding mapped to the intrinsic codebook b k Obtain the mapping encoding information e of each pixel in the encoded features. k , is represented as:

[0093]

[0094] Using φ to fuse the mapping encoding information e of all pixels in the encoding feature k The fused coding information e of the coding features is obtained and represented as:

[0095]

[0096] The fused encoded information is then passed through a fully connected layer and a convolutional layer before being output, and then combined with the three-level feature pyramid X. inPerform channel multiplication to obtain the local region feature Z, represented as:

[0097]

[0098] The local region features are compared with the three-level feature pyramid X. in Perform channel addition to obtain local information features LVC(X) in ), represented as:

[0099] in, Representing encoded features The i-th pixel in the array, b k This represents the k-th learnable visual code in the intrinsic codebook. This represents the positional information of each pixel relative to a learnable visual code; the total spatial number of the three-level feature pyramid is K = H × W, where H and W refer to the height and width of the three-level feature pyramid, respectively; φ is the fusion function, which includes BN layers, ReLU layers, and mean layers; X in Represents a three-level feature pyramid; Conv 1×1 This represents a 1×1 convolutional layer, where δ(·) refers to the Sigmoid function. ⊕ represents channel multiplication; ⊕ represents channel addition.

[0100] Based on the global and local information features obtained above, the top-level mapping feature is obtained; the global and local information features of each level of the three-level feature pyramid are concatenated to obtain the three-level mapping feature pyramid, represented as: X = cat(MLP(X in ),LVC(X in ));

[0101] Among them, X in Represents a three-level feature pyramid; MLP(X) in ) represents global information features, LVC(X) in ) represents local information features, and cat represents the concatenation of feature maps along the channel dimension.

[0102] Specifically, in step S106, the initial feature map is input to the multi-sensor module, and the channel features and spatial features of the initial feature map are extracted and concatenated to obtain the multi-sensor feature map, which specifically includes:

[0103] S106-1: Input the initial feature map F0 into the multi-scale feature extraction network to obtain multiple feature maps of different scales, and then stitch them together to obtain the multi-scale feature map F;

[0104] The multi-scale feature extraction network includes four parallel convolutional units; the first convolutional unit includes an average pooling layer and a first dilated convolutional layer in sequence along the forward propagation direction; the second convolutional unit includes a second dilated convolutional layer, a third dilated convolutional layer and a fourth dilated convolutional layer in sequence along the forward propagation direction; the third convolutional unit includes a fifth dilated convolutional layer and a sixth dilated convolutional layer in sequence along the forward propagation direction; and the fourth convolutional unit includes a seventh dilated convolutional layer.

[0105] S106-2: Extract the initial channel features C(F) of the multi-scale feature map using a channel feature extraction network. initial , is represented as:

[0106] C(F) initial =cat(δ(L(r(L(v,W0),W1)))(F));

[0107] S106-3: The initial channel feature C(F) is... initial The initial feature map is obtained by element-wise multiplying it with the multi-scale feature map F to obtain the channel features C(F);

[0108] Where F represents the multi-scale feature map, v represents the feature vector after the average pooling layer, L is the linear layer, W0 and W1 are the parameters of the two linear layers respectively, r represents the ReLU activation function; δ(·) represents the Sigmoid function; cat represents the concatenation of feature maps along the channel dimension;

[0109] S106-4: Input the initial feature map F0 into the spatial feature extraction network to obtain the initial spatial features S(F0) of the initial feature map F0. initial , is represented as:

[0110] S(F0) initial =δ(Conv 7×7 (MAP(F0),θ))⊙F0;

[0111] S106-5: The initial spatial feature S(F0) is... initial The spatial feature S(F0) is obtained by element-wise multiplication with the initial feature map F0;

[0112] Where F0 represents the initial feature map, Conv 7×7 denoted as a 7×7 convolutional layer, MAP represents max pooling and average pooling, and θ represents the parameters of the spatial feature extraction network.

[0113] S106-6: Concatenate the channel features C(F) and spatial features S(F0) of the initial feature map to obtain a multi-sensory feature map.

[0114] The target localization and counting method based on point-domain feature learning described in this invention abandons the traditional density map-based learning approach. Instead, it learns direct visual knowledge based on the image to be detected, directly acquiring the image features of the image to be detected, and performing target localization and counting based on these image features. This invention uses a backbone network to construct a three-level feature pyramid of the image to be detected, employs a lightweight MLP architecture to capture the global long-distance dependencies of the top-level features of the three-level feature pyramid, and uses a learnable visual center mechanism to aggregate local region features of the three-level feature pyramid, constructing a mapped three-level feature pyramid. Global centralization adjustment is used to concatenate the mapped top-level, intermediate-level, and bottom-level features along the channel dimension, allowing the intermediate-level and bottom-level features of the mapped three-level feature pyramid to simultaneously benefit from the visual centralization information of the mapped top-level features, achieving multi-scale feature fusion, thereby improving the detection performance and classification accuracy of the classification and regression module; furthermore, global centralization adjustment can effectively reduce computational load and the number of parameters. After obtaining the initial feature map of the image to be detected, the multi-sensor module is used to extract channel features and spatial features and then concatenate them to obtain a multi-sensor feature map. After passing through the classification and regression module, the predicted target location and the predicted target number are obtained. The use of the multi-sensor module reduces the calculation error of point neighborhood feature learning in low-density areas and improves the target location accuracy. The classification and regression module is used to simultaneously realize target counting, which improves the generalization ability and robustness of the model.

[0115] Based on the above embodiments, in this embodiment of the invention, the network model of the target localization and counting method based on point neighborhood feature learning is referred to... Figure 3 As shown, it includes three parts:

[0116] The first part is the backbone network, which is used for the initial extraction of image features.

[0117] The second part is the feature pyramid, which performs multi-scale processing on the extracted features. The feature pyramid consists of object corner processing (OCP) and global centralization adjustment. After the input image passes through the backbone network, a three-level feature pyramid X is extracted. i Each layer of features X i The spatial dimensions of (i = 0, 1, 2) are 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively. A lightweight MLP architecture is then used to capture the global long-range dependencies of X2, and a learnable visual center mechanism is used to aggregate the local angular regions of the input image.

[0118] The third part is the multi-sensor head, which includes a multi-sensor module and a classification and regression head. The multi-sensor module processes the feature maps at each scale obtained in the second part using spatial and channel attention, and then the classification and regression head performs crowd counting and localization. The main purpose of this part is to address the computational errors in low-density regions during point neighborhood feature learning.

[0119] S201: The image to be detected first enters the backbone network, which has four stages. Before stage 1, the image passes through the Embedding layer, which is a regular convolutional layer with a kernel size of 4 and a stride of 4. Before other stages, a Merging layer is used, which is a regular convolutional layer with a kernel size of 2 and a stride of 2. The Merging layer's function is spatial downsampling and channel expansion. Each stage uses a partial convolutional neural network (Pconv) as the basic operator. Pconv is a variant of a regular convolutional neural network. This type of convolutional layer only needs to apply regular convolutions to a portion of the input channels for spatial feature extraction, keeping the remaining channels unchanged. For continuous or regular memory accesses, the first or last consecutive channel is treated as a representative of the entire feature map for computation. Without loss of generality, the input and output feature maps are assumed to have the same number of channels.

[0120] The basic structural units within a stage block are as follows: Figure 2 As shown. The first stage block has 3 basic structural units, the second stage block has 4 basic structural units, the third stage block has 18 basic structural units, and the fourth stage block has 3 basic structural units. A tensor of shape (h, w, c) becomes (h / 4, w / 4, c) after passing through the first stage block. After that, each time it passes through a stage block, its h and w become half of their original values. The tensor output by the backbone network has the shape (h / 32, w / 32, c).

[0121] S202: To perform feature smoothing, the features pass through a smoothing block consisting of a 7x7 convolution with an output channel size of 256, a batch normalization (BN) layer, and an activation function layer before entering the target angle visual processing. X in The output of the smooth block is represented as:

[0122] X in =σ(BN(Conv) 7×7 (X2)));

[0123] Where σ(·) represents the ReLU activation function, BN represents the batch normalization layer, and Conv 7×7 This indicates a convolutional layer with a kernel size of 7.

[0124] S203: Target corner visual processing consists of two parallel blocks. One lightweight MLP (Multilayer Perceptron) captures the global long-range dependencies (i.e., global information) of the top-level X2 features. Simultaneously, to preserve local corner regions (i.e., local information), a learnable Visual Central Convergence (LVC) mechanism operates on X2 to aggregate local region features within the layer. The resulting feature maps from these two blocks are concatenated along the channel dimension and passed as the output of target corner visual processing to the downstream recognition model.

[0125] X = cat(MLP(X) in ); LVC(X in ));

[0126] Where X is the output of the target angle visual processing, cat is the concatenation of feature maps along the channel dimension, and MLP(X) in ) and LVC(X in The tensor refers to the output features of the lightweight MLP and learnable visual center mechanism used; the shape of the tensor does not change after the target angle visual processing.

[0127] S203-1: The MLP consists of two residual modules: a depthwise separable convolution-based module and a channel-based MLP module. The input to the MLP module is the output of the depthwise separable convolution module. Both modules undergo channel scaling and DropPath operations to improve feature generalization and robustness.

[0128] The input to the depthwise separable convolutional module is a group-normalized feature map X. in Depthwise separable convolution can improve feature representation while reducing computational cost. A depthwise separable convolution module is represented as:

[0129]

[0130] in, GN(·) represents the output of the depthwise separable convolution module, GN(·) represents group normalization, and Dconv(·) represents a depthwise convolution with a kernel of 1.

[0131] The input to the channel MLP module is the output of the depthwise separable convolution module, which is then normalized before the channel MLP operation is performed.

[0132]

[0133] Where CMLP(·) represents a channel multilayer sensor.

[0134] S203-2: LVC is an encoder with an intrinsic dictionary, consisting of an intrinsic codebook and a set of learnable visual center scaling factors. The intrinsic codebook is represented as: B = b1, b2, ..., bk Where K = H × W is the total number of input feature spaces, and H and W refer to the height and width of the input feature map; the learnable visual center scaling factor is represented as S = s1, s2, ..., s k Specifically, the feature X after the smooth block in First, the features are encoded using a combination of convolutional layers consisting of 1×1, 3×3, and 1×1 convolutions. Then, the encoded features undergo CBR block processing, where each block consists of a 3×3 convolution with a BN layer and a ReLU activation function. After these steps, the encoded features... The input is entered into the codebook, and then a set of scaling factors s is used to sequentially... and b k Map the corresponding positional information. The information of the entire image relative to the k-th code is represented in the following way:

[0135]

[0136] in It is the i-th pixel, b k It is the k-th learnable visual code, s k It is the kth scaling factor. This represents the positional information of each pixel relative to a codeword. K is the total number of visual centers. Then, φ is used to fuse all ek, where φ includes a BN layer, ReLU, and a mean layer. Based on this, the complete information of the entire image relative to K codewords is calculated as follows:

[0137]

[0138] After obtaining the codebook output, e is further input into a fully connected layer and a 1×1 convolutional layer to predict features that highlight key classes. Then, we use features from the smoothing block X... in The channel multiplication between the input features and the scaling factor coefficients. The above process can be expressed as:

[0139]

[0140] Where Conv 1×1 It refers to a 1×1 convolutional layer, and δ(·) refers to the Sigmoid function. This refers to channel multiplication.

[0141] Finally, in the feature X output from the smooth block in Channel addition is performed between the feature Z and the local corner region feature, with the following formula:

[0142]

[0143] in, This indicates channel addition.

[0144] S204: Global centralization adjustment is used to ensure that shallow features of the feature pyramid can simultaneously benefit from the visual centralization information of the deepest features. In this embodiment, spatial OCP is first implemented at the top layer of the feature pyramid (i.e., X2). Then, using the obtained feature X containing spatial target angle visual processing, all frontal shallow features (i.e., X0 to X1) are simultaneously adjusted. On each corresponding bottom-layer feature, the features obtained from the deep layers are upsampled to the same spatial scale as the bottom-layer features, and then connected along the channel dimension. Based on this, the concatenated features are downsampled to a 256-channel size by a 1×1 convolution.

[0145] S205: Finally, these features are aggregated into the multi-head perception part.

[0146] S205-1: Input the feature map F0 into a multi-scale feature extraction network, extract features of four different scales from the feature map F0 through the multi-scale feature extraction network, and concatenate the features of the four different scales to obtain the multi-scale feature F;

[0147] The multi-scale feature extraction network includes four parallel convolutional units.

[0148] The first convolutional unit includes a first dilated convolutional layer, and an average pooling layer is disposed before the first dilated convolutional layer;

[0149] The second convolutional unit includes a second dilated convolutional layer, a third dilated convolutional layer, and a fourth dilated convolutional layer connected in sequence;

[0150] The third convolutional unit includes a fifth and a sixth dilated convolutional layer connected in sequence;

[0151] The fourth convolutional unit includes the seventh dilated convolutional layer.

[0152] S205-2: Extracting the channel features of the multi-scale features, the method includes:

[0153] The initial channel features C(F) of the multi-scale features are extracted using a channel feature extraction network. inital ;

[0154] The initial channel feature C(F) initial The channel feature C(F) is obtained by element-wise multiplication with the multi-scale feature;

[0155] Specifically, the initial channel features C(F) of the multi-scale features are extracted using a channel feature extraction network. initial The formula is:

[0156] C(F) initial=cat(δ(L(r(L(v,W0),W1)))(F));

[0157] Where F represents multi-scale features, v represents the feature vector after average pooling, W0 and W1 are the parameters of two linear layers, L is a linear layer, and r is the ReLU activation function.

[0158] S205-3: Extracting spatial features from the feature map, the method includes:

[0159] The initial spatial features S(F0) of the feature map are extracted using a spatial feature extraction network. initial ;

[0160] The initial channel feature S(F0) initial The spatial feature S(F0) is obtained by element-wise multiplication with the feature map;

[0161] Specifically, the initial spatial features S(F0) of the feature map are extracted using a spatial feature extraction network. initial The formula is:

[0162] S(F0) initial =δ(Conv 7×7 (MAP(F0),θ))⊙F0;

[0163] Where F0 represents the feature map, Conv 7×7 denoted as a 7×7 convolutional layer, MAP represents max pooling and average pooling, and θ represents the parameters of the spatial feature extraction network.

[0164] S205-4: After calculating the corresponding channel weights, the channel sensing block and the spatial sensing block are applied to the feature mapping. The overall multi-sensor module is represented as follows:

[0165] Output MA =C(F)+S(F0);

[0166] Among them, Output MA This indicates the output of the multi-sensor module.

[0167] S206: The classification and regression head consists of two parts: a classification head and a regression head. Both the regression head and the classification head are composed of four 3×3 convolutional layers. The difference is that the output layer of the regression head is a 3×3 convolutional layer, which is used to output the number of predicted targets; the output layer of the classification head is a sigmoid function, which is used to achieve target localization.

[0168] In this embodiment of the invention, Target Angle Visual Criterion (OCC) is a generalized intra-layer feature adjustment method that can extract global long-range dependencies while preserving local corner region information of the input image as much as possible, which is crucial for dense prediction tasks. However, using OCP at each level of the feature pyramid leads to significant computational overhead. To improve the computational efficiency of intra-layer feature adjustment, this embodiment further proposes a top-down Global Centralized Criterion Adjustment (GCR) for the feature pyramid. Specifically, considering that the deepest features typically contain the most abstract feature representations, which are scarce in shallow features, the spatial OCP of this embodiment is first implemented at the top level of the feature pyramid (i.e., X2). Then, using the obtained feature X containing spatial target angle visual processing, all frontal shallow features (i.e., X0 to X1) are simultaneously adjusted. For each corresponding bottom-level feature, the features obtained from the deep layers are upsampled to the same spatial scale as the bottom-level features and then concatenated along the channel dimension. Based on this, the concatenated features are downsampled to a 256-channel size by a 1×1 convolution. In this way, the spatial weights of the global representation can be explicitly increased, thereby effectively achieving comprehensive and differentiated feature representation.

[0169] Based on the above embodiments, in order to verify the performance of the present invention, experiments were conducted on multiple public target counting datasets using the target localization and counting method based on point neighborhood feature learning proposed in this invention. Furthermore, a large-scale comparison was made with existing methods using mean absolute error (MAE) and mean squared error (MSE). The comparison results are shown in Table 1.

[0170] Table 1 Comparison of experimental results

[0171]

[0172] As shown in Table 1, the target localization and counting method based on point neighborhood feature learning provided by this invention has the smallest mean absolute error and mean square error in the datasets SHTech Part A, SHTech Part B, UCF CC 50, and UCF-QNRF compared to existing methods such as MCNN, CSRNet, CAN, SDANet, ASNet, GLoss, S3, MAN, FusionCount, CLTR, and OrdinalEntropy, indicating that the target localization and counting technology of this invention is accurate.

[0173] The target localization and counting method based on point-domain feature learning provided by this invention abandons the traditional density map-based learning approach. Instead, it learns direct visual knowledge based on the image to be detected, directly acquiring the image features of the image to be detected, and performing target localization and counting based on these image features. This invention uses a backbone network to construct a three-level feature pyramid of the image to be detected, employs a lightweight MLP architecture to capture the global long-distance dependencies of the top-level features of the three-level feature pyramid, and uses a learnable visual center mechanism to aggregate local region features of the three-level feature pyramid, constructing a mapped three-level feature pyramid. Global centralization adjustment is used to concatenate the mapped top-level, intermediate-level, and bottom-level features along the channel dimension, allowing the intermediate-level and bottom-level features of the mapped three-level feature pyramid to simultaneously benefit from the visual centralization information of the mapped top-level features, achieving multi-scale feature fusion, thereby improving the detection performance and classification accuracy of the classification and regression module; furthermore, global centralization adjustment can effectively reduce computational load and the number of parameters. After obtaining the initial feature map of the image to be detected, the multi-sensor module is used to extract channel features and spatial features and then concatenate them to obtain a multi-sensor feature map. After passing through the classification and regression module, the predicted target location and the predicted target number are obtained. The use of the multi-sensor module reduces the calculation error of point neighborhood feature learning in low-density areas and improves the target location accuracy. The classification and regression module is used to simultaneously realize target counting, which improves the generalization ability and robustness of the model.

[0174] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0178] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A target locating and counting method based on point field feature learning, characterized in that, include: The image to be detected is input into the backbone network, and a three-level feature pyramid of the image to be detected is extracted and output; the three-level feature pyramid includes top-level features, middle-level features and bottom-level features; The three-level feature pyramid is input into the multilayer perceptron module to capture the global long-distance dependency of its top-level features and obtain global information features. The three-level feature pyramid is input into the visual center mechanism module, which aggregates the local region features of its top-level features to obtain local information features. The global information features and the local information features are connected together along the channel dimension to obtain the top-level mapping features, which, together with the intermediate-level features and the bottom-level features, form a three-level mapping feature pyramid. The mapped three-level feature pyramid is input into the global centralized adjustment module, which maps it to the top-level features, middle-level features and bottom-level features, and splices them along the channel dimension to output the initial feature map; The initial feature map is input into the multi-sensor module, and the channel features and spatial features of the initial feature map are extracted and concatenated to obtain the multi-sensor feature map; The multi-sensory feature map is input into the classification and regression module, and the predicted number of targets in the image to be detected is output through the regression head, and the predicted target location in the image to be detected is output through the classification head. The acquisition of channel features from the initial feature map includes: processing the initial feature map... Input the data into a multi-scale feature extraction network to obtain feature maps of different scales, then concatenate them to obtain a multi-scale feature map. The initial channel features of the multi-scale feature map are extracted using a channel feature extraction network. , represented as: The initial channel features With the multi-scale feature map Element-wise multiplication is performed to obtain the channel features of the initial feature map. ;in, Represents multi-scale feature maps, This represents the feature vector after the average pooling layer. For linear layers, and These are the parameters for two linear layers, Represents the ReLU activation function; Represents the Sigmoid function; This indicates the concatenation of feature maps along the channel dimension.

2. The target localization and counting method based on point neighborhood feature learning according to claim 1, characterized in that, It also includes a smoothing block, which is set after the backbone network, so that the three-level feature pyramid output by the backbone network passes through the convolution block, batch normalization layer and activation function layer in sequence along the forward propagation direction, and outputs a smoothed three-level feature pyramid, which is then input to the multilayer perceptron module and the visual center mechanism module.

3. The target localization and counting method based on point domain feature learning according to claim 1, characterized in that, The process of inputting the image to be detected into the backbone network and extracting and outputting the three-level feature pyramid of the image to be detected includes: The image to be detected is input into the backbone network and passes through the Embedding layer, the first stage block, the Merging layer, the second stage block, the Merging layer, the third stage block, the Merging layer, and the fourth stage block in sequence along the forward propagation direction. The output of the second stage block is used as the bottom layer feature, the output of the third stage block is used as the middle layer feature, and the output of the fourth stage block is used as the top layer feature, thus forming a three-level feature pyramid of the image to be detected. The first stage block comprises three cascaded basic structural units along the forward propagation direction; the second stage block comprises four cascaded basic structural units along the forward propagation direction; the third stage block comprises eighteen cascaded basic structural units along the forward propagation direction; and the fourth stage block comprises three cascaded basic structural units along the forward propagation direction. The basic structural unit, along the forward propagation direction, sequentially includes a Pconv network, a convolutional layer with a kernel size of 1, a batch normalization layer, an activation function layer, and a convolutional layer with a kernel size of 3. The Embedding layer is a convolutional layer with a kernel size of 4 and a stride of 4; The Merging layer is a convolutional layer with a kernel size of 2 and a stride of 2.

4. The target localization and counting method based on point neighborhood feature learning according to claim 1, characterized in that, The step of inputting the three-level feature pyramid into the multilayer perceptron module to capture the global long-range dependency of its top-level features and obtain global information features includes: After group normalization, the three-level feature pyramid is input into a depthwise separable convolutional unit for depthwise convolution to obtain the convolutional feature map. , represented as: ； The convolutional feature map is input into the channel MLP unit, and after group normalization, the channel MLP operation is performed to output global information features. , represented as: ； in, This represents a three-level feature pyramid. Indicates group normalization, This represents a depthwise convolution with a kernel of 1. This refers to a multilayer sensor for the channel.

5. The target localization and counting method based on point neighborhood feature learning according to claim 1, characterized in that, The step of inputting the three-level feature pyramid into the visual center mechanism module, aggregating the local region features of its top-level features, and obtaining local information features includes: The three-level feature pyramid is input into the visual center mechanism module. After being combined and encoded by a set of preset convolutional layers, it passes through convolutional layers, batch normalization layers, and activation function layers to output the encoded features. ; The encoded features are input into the inherent codebook of the visual center mechanism module. In, using a set of scaling factors Encode features Learnable visual encoding mapped to the intrinsic codebook Obtain the mapping encoding information of each pixel in the encoded features. , represented as: ； use The mapping encoding information of all pixels in the fusion encoding feature Obtain fusion coding information of coding features , represented as: ； The fused encoded information is then passed through fully connected layers and convolutional layers before being output, and then combined with the three-level feature pyramid. Perform channel multiplication to obtain local region features. , represented as: ； The local region features are compared with the three-level feature pyramid. Perform channel addition to obtain local information features , represented as: ; in, Representing encoded features The i-th pixel in This represents the k-th learnable visual code in the intrinsic codebook. Represents the positional information of each pixel relative to a learnable visual code; the total spatial number of the three-level feature pyramid. , These refer to the height and width of the three-level feature pyramid, respectively; φ is the fusion function, which includes the BN layer, ReLU, and mean layers. This represents a three-level feature pyramid. This represents a 1×1 convolutional layer. Refers to the Sigmoid function. Indicates channel multiplication; This indicates channel addition.

6. The target localization and counting method based on point neighborhood feature learning according to claim 1, characterized in that, The mapping three-level feature pyramid is represented as follows: ； in, This represents a three-level feature pyramid. Represents global information characteristics. Representing local information features, This indicates the concatenation of feature maps along the channel dimension.

7. The target localization and counting method based on point neighborhood feature learning according to claim 1, characterized in that, The multi-scale feature extraction network comprises four parallel convolutional units, wherein: The first convolutional unit includes an average pooling layer and a first dilated convolutional layer sequentially along the forward propagation direction; The second convolutional unit includes, in sequence along the forward propagation direction, a second dilated convolutional layer, a third dilated convolutional layer, and a fourth dilated convolutional layer; The third convolutional unit includes, in sequence, the fifth and sixth dilated convolutional layers along the forward propagation direction; The fourth convolutional unit includes the seventh dilated convolutional layer.

8. The target localization and counting method based on point neighborhood feature learning according to claim 1, characterized in that, The acquisition of spatial features of the initial feature map includes: The initial feature map The initial feature map is obtained in the input spatial feature extraction network. initial spatial features , represented as: ； The initial spatial features and the initial feature map Element-wise multiplication yields spatial features ; in, Represents the initial feature map. This represents a 7×7 convolutional layer. This represents max pooling and average pooling. This represents the parameters of the spatial feature extraction network.

9. The target localization and counting method based on point neighborhood feature learning according to claim 1, characterized in that, The step of inputting the multi-sensor feature map into the classification and regression module, outputting the predicted number of targets in the image to be detected through the regression head, and outputting the predicted target localization in the image to be detected through the classification head includes: The multi-sensor feature map is input into the regression head of the classification and regression module, and after passing through four 3×3 convolutional layers, the predicted target number is output. The multi-sensory feature map is input into the classification head of the classification and regression module, and after passing through four 3×3 convolutional layers and the Sigmoid function, the predicted target location is output.

Images