Image target counting method and device based on image text pairs

Abstract

The invention relates to the technical field of image processing, in particular to an image target counting method and device based on an image text pair, and the method comprises the steps: constructing a target counting model which comprises a text encoder, an image encoder, a text image enhancement module and a decoupling head; and inputting the original image and the text of the target category into the target counting model, outputting coordinates of prediction points of a plurality of individuals belonging to the target category in the original image, and further obtaining the total number of the individuals belonging to the target category. According to the method, the local detail information of the image is reserved, the perception ability of the model to the local information is enhanced, and the global information extracted by the multi-head self-attention mechanism is further combined, so that the recognition ability of the model to the target is effectively improved, and the accuracy of target counting is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image object counting method and device based on image-text pairs. Background Art

[0002] Recently, open set models have attracted the attention of object counting researchers because these models have good generalization performance by predicting new and unseen categories.

[0003] Zero-shot learning has been introduced into density map-based counting methods to process image-text data. For example, Contrastive Language-Image Pre-training (CLIP) is an embedded zero-shot learning method that has been applied to zero-shot object counting methods such as CrowdCLIP and CLIP-Count. Both methods extract multimodal information from image-text data through CLIP and can be used to process image-text data with multi-class objects.

[0004] However, since existing methods are all based on density map counting methods, they ignore the key position information of the target in the image and are difficult to effectively utilize the local features of the image, resulting in inaccurate target counting. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem in the prior art that local information of an image cannot be utilized, resulting in inaccurate target counting.

[0006] In order to solve the above technical problems, the present invention provides an image object counting method based on image-text pairs, comprising:

[0007] The original image is input into the image encoder, and the image code is output; the text of the target category to be counted in the original image is input into the text encoder, and the text code is output;

[0008] The text encoding and image encoding are input into the text image enhancement module, and the target fusion features are output, including:

[0009] Inputting the text code and the image code into a first feature enhancement module of a text image enhancement module to obtain a first enhanced feature;

[0010] After the first enhanced feature passes through the convolution layer and the upsampling layer, the first enhanced feature is input into the second feature enhancement module of the text image enhancement module to obtain the second enhanced feature;

[0011] Inputting the first enhanced feature and the second enhanced feature into a feature fusion module of a text image enhancement module to obtain a target fusion feature;

[0012] Each feature enhancement module includes a local enhanced multi-head self-attention mechanism, a cross attention mechanism and a multi-layer perceptron connected in sequence; the local enhanced multi-head self-attention mechanism is a three-branch structure, including a channel attention branch, a multi-head self-attention branch and a spatial attention branch;

[0013] The target fusion features are input into the decoupling head. The regression head of the decoupling head outputs the coordinates of the individual prediction points, and the classification head of the decoupling head outputs the category confidence of each prediction point. According to the category confidence of the prediction point, it is judged whether the prediction point belongs to the target category. If so, the coordinates and category confidence of the prediction point are output to obtain the total number of individuals belonging to the target category.

[0014] Preferably, the text encoder comprises a text preprocessing module and a text encoding module connected in sequence; the text preprocessing module is used to reduce the noise of the input text and output the preprocessed text; the text encoding module is used to receive the preprocessed text and output the text code;

[0015] The image encoder comprises an image preprocessing module and an image encoding module; the image preprocessing module is used to scale, randomly flip and normalize the input image, and output the preprocessed image; the image encoding module is used to receive the preprocessed image and output the image code.

[0016] Preferably, the text encoding module adopts the BERT model.

[0017] Preferably, the image encoding module adopts a ViT model.

[0018] Preferably, the step of inputting the text code and the image code into a text image enhancement module and outputting target fusion features comprises:

[0019] The image code is used as the input feature of the first local enhanced multi-head self-attention mechanism to obtain the first image enhanced feature; the text code and the first image enhanced feature are input into the first cross attention mechanism, where the text code is used as the query and the first image enhanced feature is used as the key and value, and the first text-image fusion feature is output; the first text-image fusion feature is input into the first multi-layer perceptron to obtain the first enhanced feature;

[0020] After passing through the convolution layer and the upsampling layer, the first enhanced feature is input into the first local enhanced multi-head self-attention mechanism of the second feature enhancement module to obtain the second image enhanced feature; the text code and the second image enhanced feature are input into the second cross attention mechanism, where the text code is used as the query and the second image enhanced feature is used as the key and value, and the second text image fusion feature is output; the second text image fusion feature is input into the second multi-layer perceptron to obtain the second enhanced feature;

[0021] The first enhanced feature and the second enhanced feature are input into the feature fusion module, and the first enhanced feature and the second enhanced feature are added and sequentially passed through two upsampling submodules to obtain the target fusion feature.

[0022] Preferably, the input features of the local enhanced multi-head self-attention mechanism are respectively passed through the channel attention branch, the multi-head self-attention branch and the spatial attention branch to obtain channel enhanced features, self-attention enhanced features and spatial enhanced features; the channel enhanced features, self-attention enhanced features and spatial enhanced features are added together to obtain the output features of the local enhanced multi-head self-attention mechanism.

[0023] Preferably, the input features of the local enhanced multi-head self-attention mechanism are passed through the channel attention branch to obtain channel enhanced features, including:

[0024] Channel attention performs maximum pooling and average pooling on the height and width dimensions of the input features respectively to obtain two feature weights; the two feature weights are added together and then dot-multiplied with the input features to obtain channel enhanced features.

[0025] Preferably, the input features of the locally enhanced multi-head self-attention mechanism are passed through the multi-head self-attention branch to obtain self-attention enhanced features, including:

[0026] The size of the input features of the local enhanced multi-head self-attention mechanism is B, C, H, W, where B represents the batch size, C represents the channel, H represents the height, and W represents the width;

[0027] First, the input features of the local enhanced multi-head self-attention mechanism are converted into a three-dimensional sequence of size B, C, H×W, and then the two dimensions of C and H×W are swapped to convert it into a three-dimensional sequence of size B, H×W, C as the input features of the multi-head self-attention mechanism;

[0028] After the input features of the multi-head self-attention mechanism pass through the multi-head self-attention mechanism, the output features of size B,H×W,C are obtained;

[0029] The H×W and C dimensions of the output feature are swapped and converted again into a three-dimensional sequence of size B, H×W, C; finally, the three-dimensional sequence is converted into a feature map of size B, C, H, W to obtain the self-attention enhanced feature.

[0030] Preferably, the input features of the local enhanced multi-head self-attention mechanism are passed through the spatial attention branch to obtain spatial enhanced features, including:

[0031] The spatial attention takes the maximum value and the average value in the channel dimension of the input feature to obtain two feature weights; the two feature weights are added together and then dot-multiplied with the input feature to obtain the spatial enhancement feature.

[0032] The present invention also provides an image object counting device based on image-text pairs, comprising:

[0033] The encoding module is used to input the original image into the image encoder and output the image code; input the text of the target category to be counted in the original image into the text encoder and output the text code;

[0034] The feature fusion module is used to input the text encoding and image encoding into the text image enhancement module and output the target fusion features, including:

[0035] Inputting the text code and the image code into a first feature enhancement module of a text image enhancement module to obtain a first enhanced feature;

[0036] After the first enhanced feature passes through the convolution layer and the upsampling layer, the first enhanced feature is input into the second feature enhancement module of the text image enhancement module to obtain the second enhanced feature;

[0037] Inputting the first enhanced feature and the second enhanced feature into a feature fusion module of a text image enhancement module to obtain a target fusion feature;

[0038] Each feature enhancement module includes a local enhanced multi-head self-attention mechanism, a cross attention mechanism and a multi-layer perceptron connected in sequence; the local enhanced multi-head self-attention mechanism is a three-branch structure, including a channel attention branch, a multi-head self-attention branch and a spatial attention branch;

[0039] The prediction module is used to input the target fusion features into the decoupling head. The regression head of the decoupling head outputs the coordinates of the individual prediction points, and the classification head of the decoupling head outputs the category confidence of each prediction point. According to the category confidence of the prediction point, it is judged whether the prediction point belongs to the target category. If so, the coordinates and category confidence of the prediction point are output to obtain the total number of individuals belonging to the target category.

[0040] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0041] The present invention discloses an image target counting method based on image-text pairs, constructs a target counting model, utilizes a text image enhancement module to perform feature enhancement and feature fusion on input text and input image, and introduces a local enhanced multi-head self-attention mechanism in the text image enhancement module. On the basis of the existing multi-head self-attention mechanism, based on the advantage of convolution being good at capturing local features, two branches, channel attention and spatial attention, are added. By calculating the attention weights at different spatial and channel positions, the local detail information of the image is retained, and the model's perception of local information is enhanced. The global information extracted by the multi-head self-attention mechanism is further combined to effectively improve the model's recognition ability for targets and the accuracy of target counting. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 It is a structural diagram of the target counting model proposed by the present invention;

[0044] Figure 2 It is the structural diagram of the text image enhancement module;

[0045] Figure 3 It is a structural diagram of the local enhanced multi-head self-attention mechanism. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0047] Embodiment 1

[0048] Reference Figure 1 As shown, the present invention provides an image object counting method based on image-text pairs, comprising:

[0049] Build an object counting model, including a text encoder, an image encoder, a text image enhancement module, and a decoupling head;

[0050] The original image and the text of the target category to be counted in the original image are input into the target counting model, and the coordinates of the predicted points of several individuals belonging to the target category in the original image are output, thereby obtaining the total number of individuals belonging to the target category.

[0051] The target counting model is introduced in detail below.

[0052] The text encoder comprises a text preprocessing module and a text encoding module which are connected in sequence.

[0053] The text preprocessing module processes the input text of the target category for model training or inference, and outputs the preprocessed text. First, the text preprocessing module converts all text to lowercase to reduce vocabulary variations. At the same time, the text is segmented into words or subwords and pause words are removed to reduce unnecessary noise. Text preprocessing of the target category can eliminate noise in the text and help improve the stability of the model.

[0054] The text encoding module takes the preprocessed text as input and outputs the text encoding. In order to interact with image data, the text encoding module of this embodiment adopts the BERT (Bidirectional Encoder Representations from Transformers) model. The text encoding module first divides the sentence into word fragments with special marks and adds a unique position vector E to each word fragment. pos1 , so that the model can understand the order of word fragments. Then the word fragments are converted into embedded representations and input into the BERT model for encoding, and the context-related vector representation of each word fragment is output. The context-related vector representations of all word fragments are input into the Transform block for fusion, and the text encoding ε is output. t .

[0055] The image encoder includes an image preprocessing module and an image encoding module.

[0056] The image preprocessing module processes the input original image for model training or reasoning, and outputs the preprocessed image. First, the image preprocessing module scales the original image to ensure that its size meets the input requirements of the model. Then, with a probability of 0.5, the image preprocessing module randomly flips the scaled image to increase the diversity of the data, thereby improving the generalization ability of the model. Finally, the mean and standard deviation of the randomly reversed image are calculated to standardize the pixel values ​​of the image so that it falls within the range expected by the model training, thereby achieving image normalization. Preprocessing the original image helps to eliminate the brightness and contrast differences between different images in the data set, which helps to accelerate the convergence of the model and improve the stability of training.

[0057] The image encoding module takes the preprocessed image as input and outputs the image code. In order to interact with text data, the image encoding module used in the present invention is the ViT (Vision Transformer) model. The image encoding module first divides the preprocessed image N times in both height and width directions to obtain N 2 To save the position information of the image blocks, add a position vector E to each image block. pos2 After each image block is flattened into a one-dimensional vector through a flattening operation, it is input into the stacked Transform block, global information is extracted for each image block, and the image code ε is output. p .

[0058] The input of the text image enhancement module is the text code and image code output by the text encoder and the image encoder respectively, and the output is the target fusion feature.

[0059] Reference Figure 2As shown, the text image enhancement module includes a first feature enhancement module, a second feature enhancement module and a feature fusion module which are connected in sequence.

[0060] The feature enhancement module includes a locally enhanced multi-head self-attention mechanism (Locally enhanced Multi-Head Self-Attention, L-MHSA), a cross attention mechanism and a multi-layer perceptron connected in sequence.

[0061] The text encoding and the image encoding are input into the first feature enhancement module of the text image enhancement module; in the first feature enhancement module, the image encoding is used as the input feature of the first local enhanced multi-head self-attention mechanism to obtain the first image enhancement feature; the text encoding and the first image enhancement feature are input into the first cross attention mechanism, in which the text encoding is used as the query and the first image enhancement feature is used as the key and value, and the first text image fusion feature is output; the first text image fusion feature is input into the first multi-layer perceptron to obtain the first enhanced feature.

[0062] After passing through the convolution layer and the upsampling layer, the first enhanced feature is input into the first local enhanced multi-head self-attention mechanism of the second feature enhancement module to obtain the second image enhanced feature; the text code and the second image enhanced feature are input into the second cross attention mechanism, in which the text code is used as the query and the second image enhanced feature is used as the key and value, and the second text-image fusion feature is output; the second text-image fusion feature is input into the second multi-layer perceptron to obtain the second enhanced feature.

[0063] Reference Figure 3 The local enhanced multi-head self-attention mechanism shown is a three-branch structure, including a channel attention branch, a multi-head self-attention branch and a spatial attention branch, which perform attention calculations on the channel, space and itself of the input features respectively.

[0064] The input features of the local enhanced multi-head self-attention mechanism pass through the channel attention branch, the multi-head self-attention branch and the spatial attention branch respectively to obtain channel enhanced features, self-attention enhanced features and spatial enhanced features; the output features of the local enhanced multi-head self-attention mechanism are obtained by adding the channel enhanced features, self-attention enhanced features and spatial enhanced features.

[0065] Specifically, the dimensions of the input feature X of the local enhanced multi-head self-attention mechanism are B, C, H, W, where B represents the batch size, C represents the channel, H represents the height, and W represents the width.

[0066] In the channel attention branch, maximum pooling and average pooling are performed on the height and width dimensions of the input feature X, respectively, to obtain two feature weights of shape B, C, 1, 1; the two feature weights are added and then dot-multiplied with the input feature X to obtain the channel enhanced feature X c The formula is:

[0067] X c =X*(MaxPool(X)+AvgPool(X))

[0068] Among them, X c is the channel enhancement feature, X is the input feature of the local enhanced multi-head self-attention mechanism, MaxPool and AvgPool are the maximum pooling operation and average pooling operation in the height and width dimensions respectively.

[0069] In the multi-head self-attention branch, the shape of the input feature X is converted into a three-dimensional sequence of size B, C, H×W, and then the C and H×W dimensions are swapped to convert it into a three-dimensional sequence of size B, H×W, C as the input feature of the multi-head self-attention mechanism. After the input feature of the multi-head self-attention mechanism passes through the multi-head self-attention mechanism, the output feature of size B, H×W, C is obtained. The H×W and C dimensions of the output feature are swapped and converted again into a three-dimensional sequence of size B, H×W, C. Finally, the three-dimensional sequence is converted into a feature map of size B, C, H, W to obtain the self-attention enhanced feature X m The formula is:

[0070] X seq =X.flatten(2).view(B,H×W,C)

[0071] X m =MHSA(X seq ).unsqueez(-1).view(B,C,H,W)

[0072] Among them, X is the input feature of the local enhanced multi-head self-attention mechanism, B is the batch size, C is the channel, H is the height, W is the width, flatten is the flattening operation, view is the shape conversion operation, and X seq is the input feature of the multi-head self-attention mechanism; MHSA is the multi-head self-attention mechanism, unsqueez(-1) means adding a dimension of size 1 to the last position of the tensor, X m Enhance features for self-attention.

[0073] In the spatial attention branch, the maximum value and average value are taken on the channel dimension of the input feature X to obtain two feature weights of shape B,1,H,W; the two feature weights are added and then dot-multiplied with the input feature X to obtain the spatial enhanced feature X s The formula is:

[0074] X s =X*(Max(X)+Avg(X))

[0075] Among them, X s To enhance the spatial features, Max and Avg are the maximum and average values ​​in the channel dimension respectively.

[0076] The feature fusion module includes an addition operation, a first upsampling submodule and a second upsampling submodule connected in sequence. Each upsampling submodule includes a convolution layer and an upsampling layer connected in sequence. The first enhanced feature and the second enhanced feature are input into the feature fusion module, and after the first enhanced feature and the second enhanced feature are added, they pass through the first upsampling submodule and the second upsampling submodule in sequence to obtain a target fusion feature.

[0077] The input of the decoupling head is the target fusion feature, and the structure includes a regression head and a classification head. The regression head and the classification head both include three convolutional layers with a convolution kernel size of 3×3 connected in sequence, and the first two convolutional layers are connected to the ReLU activation function. The last convolutional layer of the regression head outputs the coordinates of the individual prediction points, and the last convolutional layer of the classification head outputs the category confidence of each prediction point.

[0078] In this embodiment, the decoupling head generates a prediction point for every two pixels of the target fusion feature, the regression head calculates the offset value of each prediction point, outputs the coordinates of the individual prediction point, and the classification head outputs the category confidence of each prediction point. The model determines whether each prediction point is a positive sample based on the category confidence of the prediction point. If the prediction point is a positive sample, the coordinates and category confidence of the prediction point are output, and finally the total number of individuals belonging to the target category is obtained.

[0079] Embodiment 2

[0080] To verify the effect of the present invention, this embodiment conducts experiments on the target counting dataset FSC147, which consists of 6135 images and 343,818 annotations, wherein the training set contains 3659 images, the validation set contains 1286 images, and the test set contains 1190 images.

[0081] In the image preprocessing module, this embodiment randomly crops the input original image, resizes the image to 224×224, randomly flips the image, and finally normalizes the image with means of 0.485, 0.456, and 0.406 and standard deviations of 0.229, 0.224, and 0.225.

[0082] In the image encoding module, the preprocessed image is evenly divided into 14 2 image blocks.

[0083] The effect of the present invention can be verified by the counting results of the test set part of the FSC147 data set:

[0084] The target counting model proposed in the present invention is compared with the network models such as CLIP-Count on the same data set. The evaluation index is the grid mean absolute error (MAE), which is calculated as follows:

[0085]

[0086] Among them, Num is the total number of samples in the test set, y i represents the total number of individuals marked as the target category on the i-th sample, Represents the total number of individuals of the target category on the i-th sample predicted by the model.

[0087] The experimental results are shown in Table 1.

[0088] Table 1. Comparison of target counting results on the FSC147 dataset

[0089] method Zaro-shotCounting CLIP-Count The present invention MAE 22.09 17.78 15.91

[0090] The experimental results in Table 1 prove that the counting results of the method of the present invention on the FSC147 dataset are greatly improved compared with the existing methods.

[0091] In summary, the image target counting method based on image-text pairs described in the present invention constructs a target counting model, uses a text image enhancement module to perform feature enhancement and feature fusion on the input text and input image, and introduces a local enhanced multi-head self-attention mechanism in the text image enhancement module. On the basis of the existing multi-head self-attention mechanism, based on the advantage of convolution being good at capturing local features, two branches, channel attention and spatial attention, are added. By calculating the attention weights at different spatial and channel positions, the local detail information of the image is retained, and the model's perception of local information is enhanced. Further combined with the global information extracted by the multi-head self-attention mechanism, the model's ability to recognize targets is effectively improved, and the accuracy of target counting is improved.

[0092] Embodiment 3

[0093] Based on the image object counting method based on image-text pairs described in the first embodiment, this embodiment further provides an image object counting device based on image-text pairs, including:

[0094] The encoding module is used to input the original image into the image encoder and output the image code; input the text of the target category to be counted in the original image into the text encoder and output the text code;

[0095] The feature fusion module is used to input the text encoding and image encoding into the text image enhancement module and output the target fusion features, including:

[0096] Inputting the text code and the image code into a first feature enhancement module of a text image enhancement module to obtain a first enhanced feature;

[0097] After the first enhanced feature passes through the convolution layer and the upsampling layer, the first enhanced feature is input into the second feature enhancement module of the text image enhancement module to obtain the second enhanced feature;

[0098] Inputting the first enhanced feature and the second enhanced feature into a feature fusion module of a text image enhancement module to obtain a target fusion feature;

[0099] Each feature enhancement module includes a local enhanced multi-head self-attention mechanism, a cross attention mechanism and a multi-layer perceptron connected in sequence; the local enhanced multi-head self-attention mechanism is a three-branch structure, including a channel attention branch, a multi-head self-attention branch and a spatial attention branch;

[0100] The prediction module is used to input the target fusion feature into the decoupling head, the regression head of the decoupling head outputs the coordinates of the individual prediction points, and the classification head of the decoupling head outputs the category confidence of each prediction point; according to the category confidence of the prediction point, it is judged whether the prediction point belongs to the target category, and if so, the coordinates and category confidence of the prediction point are output, so as to obtain the total number of individuals belonging to the target category. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0104] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for counting image objects based on image-text pairs, characterized in that: include: Input the original image into the image encoder and output the image code; Input the text of the target category to be counted in the original image into the text encoder and output the text encoding; The text encoding and image encoding are input into the text image enhancement module, and the target fusion features are output, including: Input the text code and the image code to the first feature enhancement module of the text image enhancement module, obtaining a first enhanced feature; After the first enhanced feature passes through the convolution layer and the upsampling layer, the first enhanced feature is input into the second feature enhancement module of the text image enhancement module to obtain the second enhanced feature; Inputting the first enhanced feature and the second enhanced feature into a feature fusion module of a text image enhancement module to obtain a target fusion feature; Each feature enhancement module includes a local enhanced multi-head self-attention mechanism, a cross attention mechanism and a multi-layer perceptron connected in sequence; the local enhanced multi-head self-attention mechanism is a three-branch structure, including a channel attention branch, a multi-head self-attention branch and a spatial attention branch; The target fusion features are input into the decoupling head. The regression head of the decoupling head outputs the coordinates of the individual prediction points, and the classification head of the decoupling head outputs the category confidence of each prediction point. According to the category confidence of the prediction point, it is judged whether the prediction point belongs to the target category. If so, the coordinates and category confidence of the prediction point are output to obtain the total number of individuals belonging to the target category.

2. The image object counting method based on image-text pairs according to claim 1 is characterized in that: The text encoder comprises a text preprocessing module and a text encoding module connected in sequence; the text preprocessing module is used to reduce the noise of the input text and output the preprocessed text; the text encoding module is used to receive the preprocessed text and output the text encoding; The image encoder comprises an image preprocessing module and an image encoding module; the image preprocessing module is used to scale, randomly flip and normalize the input image, and output the preprocessed image; the image encoding module is used to receive the preprocessed image and output the image code.

3. The image object counting method based on image-text pairs according to claim 2 is characterized in that: The text encoding module adopts the BERT model.

4. The image object counting method based on image-text pairs according to claim 2 is characterized in that: The image encoding module adopts the ViT model.

5. The image object counting method based on image-text pairs according to claim 1, characterized in that: The text code and the image code are input into the text image enhancement module, and the target fusion feature is output, including: The image code is used as the input feature of the first local enhanced multi-head self-attention mechanism to obtain the first image enhanced feature; the text code and the first image enhanced feature are input into the first cross attention mechanism, where the text code is used as the query and the first image enhanced feature is used as the key and value, and the first text-image fusion feature is output; the first text-image fusion feature is input into the first multi-layer perceptron to obtain the first enhanced feature; After passing through the convolution layer and the upsampling layer, the first enhanced feature is input into the first local enhanced multi-head self-attention mechanism of the second feature enhancement module to obtain the second image enhanced feature; the text code and the second image enhanced feature are input into the second cross attention mechanism, where the text code is used as the query and the second image enhanced feature is used as the key and value, and the second text image fusion feature is output; the second text image fusion feature is input into the second multi-layer perceptron to obtain the second enhanced feature; The first enhanced feature and the second enhanced feature are input into the feature fusion module, and the first enhanced feature and the second enhanced feature are added and sequentially passed through two upsampling submodules to obtain the target fusion feature.

6. The image object counting method based on image-text pairs according to claim 1, characterized in that: The input features of the local enhanced multi-head self-attention mechanism are respectively passed through the channel attention branch, the multi-head self-attention branch and the spatial attention branch to obtain channel enhanced features, self-attention enhanced features and spatial enhanced features; The output features of the local enhanced multi-head self-attention mechanism are obtained by adding the channel enhanced features, self-attention enhanced features and space enhanced features.

7. The image object counting method based on image-text pairs according to claim 6 is characterized in that: The input features of the local enhanced multi-head self-attention mechanism are passed through the channel attention branch to obtain channel enhanced features, including: Channel attention performs maximum pooling and average pooling on the height and width dimensions of the input features respectively to obtain two feature weights; the two feature weights are added together and then dot-multiplied with the input features to obtain channel enhanced features.

8. The image object counting method based on image-text pairs according to claim 6 is characterized in that: The input features of the local enhanced multi-head self-attention mechanism are passed through the multi-head self-attention branch to obtain self-attention enhanced features, including: The size of the input features of the local enhanced multi-head self-attention mechanism is B, C, H, W, where B represents the batch size, C represents the channel, H represents the height, and W represents the width; First, the input features of the local enhanced multi-head self-attention mechanism are converted into a three-dimensional sequence of size B, C, H×W, and then the two dimensions of C and H×W are exchanged to convert it into a three-dimensional sequence of size B, H×W, C as the input features of the multi-head self-attention mechanism; After the input features of the multi-head self-attention mechanism pass through the multi-head self-attention mechanism, the output features of size B,H×W,C are obtained; The H×W and C dimensions of the output feature are swapped and converted again into a three-dimensional sequence of size B, H×W, C; finally, the three-dimensional sequence is converted into a feature map of size B, C, H, W to obtain the self-attention enhanced feature.

9. The image object counting method based on image-text pairs according to claim 6, characterized in that: The input features of the local enhanced multi-head self-attention mechanism are passed through the spatial attention branch to obtain spatial enhanced features, including: The spatial attention takes the maximum value and the average value in the channel dimension of the input feature to obtain two feature weights; the two feature weights are added together and then dot-multiplied with the input feature to obtain the spatial enhanced feature.

10. An image object counting device based on image-text pairs, characterized in that: include: The encoding module is used to input the original image into the image encoder and output the image encoding; Input the text of the target category to be counted in the original image into the text encoder and output the text encoding; The feature fusion module is used to input the text encoding and image encoding into the text image enhancement module and output the target fusion features, including: Inputting the text code and the image code into a first feature enhancement module of a text image enhancement module to obtain a first enhanced feature; After the first enhanced feature passes through the convolution layer and the upsampling layer, the first enhanced feature is input into the second feature enhancement module of the text image enhancement module to obtain the second enhanced feature; Inputting the first enhanced feature and the second enhanced feature into a feature fusion module of a text image enhancement module to obtain a target fusion feature; Among them, each feature enhancement module includes a local enhanced multi-head self-attention mechanism, a cross attention mechanism and a multi-layer perceptron connected in sequence; the local enhanced multi-head self-attention mechanism is a three-branch structure, including a channel attention branch, a multi-head self-attention branch and a spatial attention branch; the prediction module is used to input the target fusion feature into the decoupling head, the regression head of the decoupling head outputs the coordinates of the individual prediction point, and the classification head of the decoupling head outputs the category confidence of each prediction point; according to the category confidence of the prediction point, it is judged whether the prediction point belongs to the target category. If so, the coordinates and category confidence of the prediction point are output, and then the total number of individuals belonging to the target category is obtained.

Images