A method and apparatus for counting image targets based on image-text pairs

Abstract

This invention relates to the field of image processing technology, and in particular to an image target counting method and apparatus based on image-text pairs. The method includes: constructing a target counting model, comprising a text encoder, an image encoder, a text-image enhancement module, and a decoupling head; inputting the original image and text representing the target category into the target counting model, outputting the coordinates of predicted points of several individuals belonging to the target category in the original image, and thus obtaining the total number of individuals belonging to the target category. This invention preserves local detail information of the image, enhances the model's ability to perceive local information, and further combines global information extracted by a multi-head self-attention mechanism, effectively improving the model's ability to recognize targets and increasing the accuracy of target counting.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image target counting method and apparatus based on image-text pairs. Background Technology

[0002] Recently, open set models have attracted the attention of object counting researchers because these models have good generalization performance by predicting new and unseen categories.

[0003] Zero-point learning has been introduced into density map-based counting methods to process image-text data. For example, Contrastive Language-Image Pre-training (CLIP) is an embedded zero-point learning method that has been applied to zero-point object counting methods such as CrowdCLIP and CLIP-Count. Both methods extract multimodal information from image-text data through CLIP, enabling them to process image-text data with multiple target classes.

[0004] However, since existing methods are all based on density maps for counting, they ignore the key location information of the target in the image and cannot effectively utilize the local features of the image, resulting in inaccurate target counting. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the inability to utilize local image information in the prior art leads to inaccurate target counting.

[0006] To address the aforementioned technical problems, this invention provides an image target counting method based on image-text pairs, comprising:

[0007] Input the original image into the image encoder and output the image encoding; input the text of the target category to be counted in the original image into the text encoder and output the text encoding.

[0008] Text encoding and image encoding are input into the text-image enhancement module, which outputs target fusion features, including:

[0009] The text encoding and image encoding are input into the first feature enhancement module of the text-image enhancement module to obtain the first enhanced feature;

[0010] The first enhanced feature is passed through a convolutional layer and an upsampling layer and then input into the second feature enhancement module of the text image enhancement module to obtain the second enhanced feature;

[0011] The first and second enhancement features are input into the feature fusion module of the text image enhancement module to obtain the target fusion features;

[0012] Each feature enhancement module includes a local enhancement multi-head self-attention mechanism, a cross-attention mechanism, and a multilayer perceptron connected in sequence; the local enhancement multi-head self-attention mechanism has a three-branch structure, including a channel attention branch, a multi-head self-attention branch, and a spatial attention branch;

[0013] The target fusion features are input into the decoupling head. The regression head of the decoupling head outputs the coordinates of the predicted points of the individuals, and the classification head of the decoupling head outputs the class confidence of each predicted point. Based on the class confidence of the predicted points, it is determined whether the predicted points belong to the target class. If so, the coordinates and class confidence of the predicted points are output, and the total number of individuals belonging to the target class is obtained.

[0014] Preferably, the text encoder includes a text preprocessing module and a text encoding module connected in sequence; the text preprocessing module is used to reduce noise in the input text and output preprocessed text; the text encoding module is used to receive the preprocessed text and output text encoding.

[0015] The image encoder includes an image preprocessing module and an image encoding module; the image preprocessing module is used to scale, randomly flip, and normalize the input image, and output a preprocessed image; the image encoding module is used to receive the preprocessed image and output an image encoding.

[0016] Preferably, the text encoding module adopts the BERT model.

[0017] Preferably, the image encoding module adopts the ViT model.

[0018] Preferably, the step of inputting text encoding and image encoding into the text-image enhancement module and outputting target fusion features includes:

[0019] Image encoding is used as the input feature of the first local augmentation multi-head self-attention mechanism to obtain the first image augmentation feature; text encoding and the first image augmentation feature are input into the first cross-attention mechanism, where text encoding is used as the query and the first image augmentation feature is used as the key and value, to output the first text-image fusion feature; the first text-image fusion feature is input into the first multilayer perceptron to obtain the first augmentation feature;

[0020] The first enhanced feature is passed through a convolutional layer and an upsampling layer, and then input into the first local enhancement multi-head self-attention mechanism of the second feature enhancement module to obtain the second image enhanced feature; the text encoding and the second image enhanced feature are input into the second cross-attention mechanism, where the text encoding is used as a query and the second image enhanced feature is used as a key and value, and the second text-image fusion feature is output; the second text-image fusion feature is input into the second multilayer perceptron to obtain the second enhanced feature;

[0021] The first and second enhanced features are input into the feature fusion module. The first and second enhanced features are added together and then passed through two upsampling sub-modules in sequence to obtain the target fused features.

[0022] Preferably, the input features of the local augmented multi-head self-attention mechanism are processed through the channel attention branch, the multi-head self-attention branch, and the spatial attention branch to obtain channel augmented features, self-attention augmented features, and spatial augmented features; the channel augmented features, self-attention augmented features, and spatial augmented features are added together to obtain the output features of the local augmented multi-head self-attention mechanism.

[0023] Preferably, the input features of the local augmentation multi-head self-attention mechanism are processed through a channel attention branch to obtain channel-enhanced features, including:

[0024] Channel attention performs max pooling and average pooling on the height and width dimensions of the input features respectively to obtain two feature weights; the two feature weights are added together and then multiplied by the input features to obtain the channel-enhanced features.

[0025] Preferably, the input features of the local enhancement multi-head self-attention mechanism are processed through a multi-head self-attention branch to obtain self-attention enhancement features, including:

[0026] The input feature sizes of the local augmentation multi-head self-attention mechanism are B, C, H, W, where B represents the batch size, C represents the channel, H represents the height, and W represents the width.

[0027] First, the input features of the local augmentation multi-head self-attention mechanism are converted into a three-dimensional sequence of size B, C, H×W. Then, the two dimensions C and H×W are swapped to convert it into a three-dimensional sequence of size B, H×W, C, which serves as the input features of the multi-head self-attention mechanism.

[0028] The input features of the multi-head self-attention mechanism are processed by the multi-head self-attention mechanism to obtain output features of size B,H×W,C;

[0029] The H×W and C dimensions of the output feature are swapped, and it is converted into a three-dimensional sequence of size B, H×W, C. Finally, the three-dimensional sequence is converted into a feature map of size B, C, H, W, to obtain the self-attention enhancement feature.

[0030] Preferably, the input features of the local augmentation multi-head self-attention mechanism are processed through a spatial attention branch to obtain spatial augmentation features, including:

[0031] Spatial attention takes the maximum and average values ​​of the input features along the channel dimension to obtain two feature weights; the two feature weights are added together and then multiplied by the input features to obtain the spatially enhanced features.

[0032] The present invention also provides an image target counting device based on image-text pairs, comprising:

[0033] The encoding module is used to input the raw image into the image encoder and output the image encoding; and to input the text of the target category to be counted in the raw image into the text encoder and output the text encoding.

[0034] The feature fusion module takes text encoding and image encoding as input to the text-image enhancement module and outputs target fused features, including:

[0035] The text encoding and image encoding are input into the first feature enhancement module of the text-image enhancement module to obtain the first enhanced feature;

[0036] The first enhanced feature is passed through a convolutional layer and an upsampling layer and then input into the second feature enhancement module of the text image enhancement module to obtain the second enhanced feature;

[0037] The first and second enhancement features are input into the feature fusion module of the text image enhancement module to obtain the target fusion features;

[0038] Each feature enhancement module includes a local enhancement multi-head self-attention mechanism, a cross-attention mechanism, and a multilayer perceptron connected in sequence; the local enhancement multi-head self-attention mechanism has a three-branch structure, including a channel attention branch, a multi-head self-attention branch, and a spatial attention branch;

[0039] The prediction module is used to input the target fusion features into the decoupling head. The regression head of the decoupling head outputs the coordinates of the predicted points of the individuals, and the classification head of the decoupling head outputs the class confidence of each predicted point. Based on the class confidence of the predicted point, it is determined whether the predicted point belongs to the target class. If so, the coordinates and class confidence of the predicted point are output, and the total number of individuals belonging to the target class is obtained.

[0040] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0041] This invention discloses an image target counting method based on image-text pairs. It constructs a target counting model and utilizes a text-image enhancement module to perform feature enhancement and feature fusion on the input text and image. Furthermore, it introduces a local enhancement multi-head self-attention mechanism within the text-image enhancement module. Building upon existing multi-head self-attention mechanisms, and leveraging the advantage of convolution in capturing local features, it adds two branches: channel attention and spatial attention. By calculating attention weights at different spatial and channel locations, it preserves local detail information of the image, enhancing the model's ability to perceive local information. Further, by combining this with global information extracted through the multi-head self-attention mechanism, it effectively improves the model's target recognition ability and increases the accuracy of target counting. Attached Figure Description

[0042] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0043] Figure 1 This is a structural diagram of the target counting model proposed in this invention;

[0044] Figure 2 This is a structural diagram of the text image enhancement module;

[0045] Figure 3 This is a structural diagram of a locally enhanced multi-head self-attention mechanism. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0047] Example 1

[0048] Reference Figure 1 As shown, the present invention provides an image target counting method based on image-text pairs, comprising:

[0049] Construct a target counting model, including a text encoder, an image encoder, a text-image enhancement module, and a decoupling head;

[0050] The original image and the text of the target category to be counted in the original image are input into the target counting model. The model outputs the coordinates of the predicted points of several individuals belonging to the target category in the original image, and then obtains the total number of individuals belonging to the target category.

[0051] The target counting model will be introduced in detail below.

[0052] The text encoder includes a text preprocessing module and a text encoding module connected in sequence.

[0053] The text preprocessing module processes the input text for the target category, used for model training or inference, and outputs preprocessed text. First, the module converts all text to lowercase to reduce vocabulary variations. Simultaneously, it segments the text into words or subwords and removes pause words to reduce unwanted noise. This target category text preprocessing eliminates noise in the text, contributing to improved model stability.

[0054] The text encoding module takes preprocessed text as input and outputs text encoding. To enable interaction with image data, this embodiment's text encoding module employs the BERT (Bidirectional Encoder Representations from Transformers) model. The text encoding module first segments the sentence into specially marked word segments and adds a unique position vector E to each word segment. pos1 This process is performed so that the model can understand the order of word segments. The word segments are then converted into embedding representations, which are input into the BERT model for encoding, outputting a context-dependent vector representation for each word segment. Finally, the context-dependent vector representations of all word segments are input into the Transform block for fusion, outputting the text encoding ε. t .

[0055] The image encoder includes an image preprocessing module and an image encoding module.

[0056] The image preprocessing module processes the input raw image for model training or inference, outputting a preprocessed image. First, the module scales the raw image to ensure its size meets the model's input requirements. Then, with a probability of 0.5, the module randomly flips the scaled image, increasing data diversity and improving the model's generalization ability. Finally, the mean and standard deviation of the randomly flipped image are calculated to standardize the pixel values, ensuring they fall within the range expected by the model during training, thus achieving image normalization. Preprocessing the raw image helps eliminate differences in brightness and contrast between images in the dataset, accelerating model convergence and improving training stability.

[0057] The image encoding module takes the preprocessed image as input and outputs an image encoding. For interaction with text data, this invention uses a ViT (Vision Transformer) model for the image encoding module. The image encoding module first divides the preprocessed image N times in both the height and width directions, obtaining N... 2 There are several image blocks. To save the position information of each image block, a position vector E is added to each image block. pos2 Each image patch is flattened into a one-dimensional vector using a flattening operation, and then input into a stacked Transform block. Global information is extracted from each image patch, and the resulting image encoding ε is output. p .

[0058] The text image enhancement module takes as input the text encoding and image encoding output by the text encoder and image encoder, respectively, and outputs the target fusion features.

[0059] Reference Figure 2As shown, the text image enhancement module includes a first feature enhancement module, a second feature enhancement module, and a feature fusion module connected in sequence.

[0060] The feature enhancement module includes a Locally Enhanced Multi-Head Self-Attention (L-MHSA) mechanism, a cross-attention mechanism, and a multilayer perceptron, which are connected in sequence.

[0061] The text encoding and image encoding are input into the first feature enhancement module of the text-image enhancement module; in the first feature enhancement module, the image encoding is used as the input feature of the first local enhancement multi-head self-attention mechanism to obtain the first image enhancement feature; the text encoding and the first image enhancement feature are input into the first cross-attention mechanism, where the text encoding is used as the query and the first image enhancement feature is used as the key and value, and the first text-image fusion feature is output; the first text-image fusion feature is input into the first multilayer perceptron to obtain the first enhancement feature.

[0062] The first enhanced feature is passed through a convolutional layer and an upsampling layer, and then input into the first local enhancement multi-head self-attention mechanism of the second feature enhancement module to obtain the second image enhanced feature; the text encoding and the second image enhanced feature are input into the second cross-attention mechanism, where the text encoding is used as a query and the second image enhanced feature is used as a key and value, and the second text-image fusion feature is output; the second text-image fusion feature is input into the second multilayer perceptron to obtain the second enhanced feature.

[0063] Reference Figure 3 The local enhancement multi-head self-attention mechanism shown is a three-branch structure, including a channel attention branch, a multi-head self-attention branch, and a spatial attention branch, which respectively perform attention calculations on the channel, spatial, and self of the input features.

[0064] The input features of the local augmented multi-head self-attention mechanism are processed through the channel attention branch, the multi-head self-attention branch, and the spatial attention branch to obtain the channel augmented features, the self-attention augmented features, and the spatial augmented features. The output features of the local augmented multi-head self-attention mechanism are obtained by adding the channel augmented features, the self-attention augmented features, and the spatial augmented features.

[0065] Specifically, the dimensions of the input feature X of the local augmented multi-head self-attention mechanism are B, C, H, W, where B represents the batch size, C represents the channels, H represents the height, and W represents the width.

[0066] In the channel attention branch, max pooling and average pooling are performed on the height and width dimensions of the input feature X, respectively, to obtain two feature weights of shape B,C,1,1. The two feature weights are added together and then multiplied by the input feature X to obtain the channel enhancement feature X. c The formula is expressed as:

[0067] X c =X*(MaxPool(X)+AvgPool(X))

[0068] Among them, X c X represents the channel enhancement feature, X represents the input feature of the local enhancement multi-head self-attention mechanism, and MaxPool and AvgPool represent the max pooling operation and average pooling operation in the height and width dimensions, respectively.

[0069] In the multi-head self-attention branch, the shape of the input feature X is transformed into a three-dimensional sequence of size B, C, H × W. Then, the C and H × W dimensions are swapped, resulting in another three-dimensional sequence of size B, H × W, C, which serves as the input feature for the multi-head self-attention mechanism. After passing through the multi-head self-attention mechanism, the input feature yields an output feature of size B, H × W, C. The H × W and C dimensions of this output feature are then swapped, again transforming it into a three-dimensional sequence of size B, H × W, C. Finally, this three-dimensional sequence is converted into a feature map of size B, C, H, W, yielding the self-attention enhancement feature X. m The formula is expressed as:

[0070] X seq =X.flatten(2).view(B,H×W,C)

[0071] X m =MHSA(X seq ).unsqueez(-1).view(B,C,H,W)

[0072] Where X represents the input feature of the local augmentation multi-head self-attention mechanism, B represents the batch size, C represents the channel, H represents the height, W represents the width, flatten represents the flattening operation, and view represents the shape transformation operation. seq The input features are for the multi-head self-attention mechanism; MHSA is a multi-head self-attention mechanism, where unsqueez(-1) represents adding a dimension of size 1 at the last position of the tensor, X m This is a feature that enhances self-attention.

[0073] In the spatial attention branch, the maximum and average values ​​are taken along the channel dimension of the input feature X to obtain two feature weights of shape B, 1, H, W. The two feature weights are added together and then multiplied by the input feature X to obtain the spatial augmentation feature X. s The formula is expressed as:

[0074] X s =X*(Max(X)+Avg(X))

[0075] Among them, X s For spatial enhancement features, Max and Avg are the maximum and average values ​​taken in the channel dimension, respectively.

[0076] The feature fusion module includes a summation operation, a first upsampling submodule, and a second upsampling submodule connected in sequence. Each upsampling submodule includes a convolutional layer and an upsampling layer connected in sequence. The first enhanced feature and the second enhanced feature are input into the feature fusion module. After the first enhanced feature and the second enhanced feature are added together, they are passed through the first upsampling submodule and the second upsampling submodule in sequence to obtain the target fused feature.

[0077] The input to the decoupling head is the target fusion feature, and its structure includes a regression head and a classification head. Both the regression and classification heads consist of three sequentially connected 3×3 convolutional layers, with the first two convolutional layers followed by a ReLU activation function. The last convolutional layer of the regression head outputs the coordinates of the predicted points for each individual, and the last convolutional layer of the classification head outputs the class confidence score for each predicted point.

[0078] In this embodiment, the decoupling head generates a prediction point every two pixels of the target fusion feature. The regression head calculates the offset value of each prediction point and outputs the coordinates of the prediction point for each individual. The classification head outputs the class confidence score for each prediction point. The model determines whether each prediction point is a positive sample based on its class confidence score. If a prediction point is a positive sample, it outputs the coordinates and class confidence score of that prediction point, ultimately obtaining the total number of individuals belonging to the target class.

[0079] Example 2

[0080] To verify the effectiveness of this invention, experiments were conducted on the target counting dataset FSC147, which consists of 6135 images and 343,818 annotations. The training set contains 3659 images, the validation set contains 1286 images, and the test set contains 1190 images.

[0081] In the image preprocessing module, this embodiment randomly crops the input original image, changing its size to 224×224, then randomly flips the image, and finally normalizes the image using the mean values ​​of 0.485, 0.456, and 0.406 and the standard deviations of 0.229, 0.224, and 0.225.

[0082] In the image encoding module, the preprocessed image is divided into 14 segments on average. 2 Image blocks.

[0083] The effectiveness of this invention can be verified by counting results in the test set portion of the FSC147 dataset:

[0084] The target counting model proposed in this invention is compared with network models such as CLIP-Count on the same dataset. The evaluation metric is the mean absolute error (MAE), which is calculated as follows:

[0085]

[0086] Where Num is the total number of samples in the test set, and y i This represents the total number of individuals labeled as the target category in the i-th sample. This represents the total number of individuals belonging to the target category in the i-th sample predicted by the model.

[0087] The experimental results are shown in Table 1.

[0088] Table 1. Comparison of target counting results on the FSC147 dataset

[0089] method Zaro-shotCounting CLIP-Count This invention MAE 22.09 17.78 15.91

[0090] The experimental results in Table 1 demonstrate that the method of this invention significantly improves the counting results on the FSC147 dataset compared to existing methods.

[0091] In summary, the image target counting method based on image-text pairs described in this invention constructs a target counting model, utilizes a text image enhancement module to perform feature enhancement and feature fusion on the input text and input image, and introduces a local enhancement multi-head self-attention mechanism in the text image enhancement module. Based on the existing multi-head self-attention mechanism, and leveraging the advantage of convolution in capturing local features, two branches, channel attention and spatial attention, are added. By calculating attention weights at different spatial and channel positions, local detail information of the image is preserved, enhancing the model's ability to perceive local information. Furthermore, combined with the global information extracted by the multi-head self-attention mechanism, the model's ability to identify targets is effectively improved, thus increasing the accuracy of target counting.

[0092] Example 3

[0093] Based on the image target counting method based on image-text pairs described in Embodiment 1, this embodiment also provides an image target counting device based on image-text pairs, including:

[0094] The encoding module is used to input the raw image into the image encoder and output the image encoding; and to input the text of the target category to be counted in the raw image into the text encoder and output the text encoding.

[0095] The feature fusion module takes text encoding and image encoding as input to the text-image enhancement module and outputs target fused features, including:

[0096] The text encoding and image encoding are input into the first feature enhancement module of the text-image enhancement module to obtain the first enhanced feature;

[0097] The first enhanced feature is passed through a convolutional layer and an upsampling layer and then input into the second feature enhancement module of the text image enhancement module to obtain the second enhanced feature;

[0098] The first and second enhancement features are input into the feature fusion module of the text image enhancement module to obtain the target fusion features;

[0099] Each feature enhancement module includes a local enhancement multi-head self-attention mechanism, a cross-attention mechanism, and a multilayer perceptron connected in sequence; the local enhancement multi-head self-attention mechanism has a three-branch structure, including a channel attention branch, a multi-head self-attention branch, and a spatial attention branch;

[0100] The prediction module is used to input the target fusion features into the decoupling head. The regression head of the decoupling head outputs the coordinates of the predicted points of individuals, and the classification head of the decoupling head outputs the class confidence score of each predicted point. Based on the class confidence score of the predicted point, it is determined whether the predicted point belongs to the target class. If so, the coordinates and class confidence score of the predicted point are output, thereby obtaining the total number of individuals belonging to the target class. Those skilled in the art should understand that the embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An image object counting method based on image text pairs, characterized by, The method comprises the following steps: inputting an original image into an image encoder to output image encoding; inputting text of a target category to be counted in the original image into a text encoder to output text encoding; inputting the text encoding and the image encoding into a text-image enhancement module to output target fusion features, comprising: inputting the text encoding and the image encoding into a first feature enhancement module of the text-image enhancement module to obtain first enhanced features, comprising: inputting the image encoding as input features of a first local enhanced multi-head self-attention mechanism to obtain first image enhanced features; inputting the text encoding and the first image enhanced features into a first cross-attention mechanism, wherein the text encoding is taken as a query and the first image enhanced features are taken as keys and values to output first text-image fusion features; inputting the first text-image fusion features into a first multi-layer perceptron to obtain the first enhanced features; inputting the first enhanced features into a second feature enhancement module of the text-image enhancement module after passing through a convolution layer and an up-sampling layer to obtain second enhanced features, comprising: inputting the first enhanced features into a first local enhanced multi-head self-attention mechanism of the second feature enhancement module after passing through the convolution layer and the up-sampling layer to obtain second image enhanced features; inputting the text encoding and the second image enhanced features into a second cross-attention mechanism, wherein the text encoding is taken as a query and the second image enhanced features are taken as keys and values to output second text-image fusion features; inputting the second text-image fusion features into a second multi-layer perceptron to obtain the second enhanced features; inputting the first enhanced features and the second enhanced features into a feature fusion module of the text-image enhancement module, and then sequentially passing through two up-sampling sub-modules after adding the first enhanced features and the second enhanced features to obtain the target fusion features; wherein each feature enhancement module comprises a local enhanced multi-head self-attention mechanism, a cross-attention mechanism and a multi-layer perceptron connected in sequence; the local enhanced multi-head self-attention mechanism is a three-branch structure comprising a channel attention branch, a multi-head self-attention branch and a spatial attention branch; inputting the target fusion features into a decoupling head, wherein a regression head of the decoupling head outputs coordinates of predicted points of individuals, and a classification head of the decoupling head outputs category confidence of each predicted point; whether a predicted point belongs to the target category is determined according to the category confidence of the predicted point, and if yes, coordinates and category confidence of the predicted point are outputted to obtain a total number of individuals belonging to the target category.

2. The method of claim 1, wherein, The text encoder comprises a text preprocessing module and a text encoding module connected in sequence; the text preprocessing module is used to reduce noise of input text to output preprocessed text; and the text encoding module is used to receive the preprocessed text to output text encoding. The image encoder comprises an image preprocessing module and an image encoding module; the image preprocessing module is used to scale, randomly flip and normalize input images to output preprocessed images; and the image encoding module is used to receive the preprocessed images to output image encoding.

3. The method of claim 2, wherein, The text encoding module adopts a BERT model.

4. The method of claim 2, wherein, The image encoding module adopts a ViT model.

5. The method of claim 1, wherein, The input feature of the local enhanced multi-head self-attention mechanism is input into a channel attention branch, a multi-head self-attention branch and a spatial attention branch, and channel enhanced features, self-attention enhanced features and spatial enhanced features are obtained. The channel enhanced features, the self-attention enhanced features and the spatial enhanced features are added to obtain output features of the local enhanced multi-head self-attention mechanism.

6. The method of claim 5, wherein, The input feature of the local enhanced multi-head self-attention mechanism is input into a channel attention branch, a multi-head self-attention branch and a spatial attention branch, and channel enhanced features, self-attention enhanced features and spatial enhanced features are obtained. The channel attention branch includes:

7. The method of claim 5, wherein, The channel attention branch includes: The size of the input feature of the local enhanced multi-head self-attention mechanism is where B represents batch size, C represents channel, H represents height, and W represents width. First, the input features of the local enhanced multi-head self-attention mechanism are converted into a three-dimensional sequence with a size of , and then the two dimensions of and are exchanged to convert into a three-dimensional sequence with a size of , which is used as the input features of the multi-head self-attention mechanism; The input feature of the multi-head self-attention mechanism is obtained after the multi-head self-attention mechanism, and the output feature with a size of is obtained. The output feature is converted into a three-dimensional sequence with a size of and two dimensions are exchanged, and the three-dimensional sequence is converted again into a feature map with a size of ; and finally, the three-dimensional sequence is converted into a feature map with a size of , to obtain the self-attention enhanced feature.

8. The method of claim 5, wherein, The multi-head self-attention branch includes: The spatial attention branch includes:

9. An image target counting device based on image-text pairs, characterized in that, The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial attention branch includes: The spatial Each feature enhancement module comprises a local enhancement multi-head self-attention mechanism, a cross-attention mechanism and a multi-layer perception connected in sequence; the local enhancement multi-head self-attention mechanism is a three-branch structure comprising a channel attention branch, a multi-head self-attention branch and a spatial attention branch; The prediction module is configured to input the target fusion feature into a decoupling head, a regression head of the decoupling head outputs coordinates of a prediction point of an individual, and a classification head of the decoupling head outputs a category confidence of each prediction point; whether the prediction point belongs to a target category is determined according to the category confidence of the prediction point, if yes, coordinates and the category confidence of the prediction point are output, and then a total number of individuals belonging to the target category is obtained.

Images