Object counting method and system based on multimodal and multi-scale cross attention

Abstract

The present invention discloses a target counting method and system based on multimodal and multi-scale cross-attention, comprising: constructing a similarity matrix based on block-level visual embedding vectors and text embedding vectors; and then establishing a contrastive learning loss based on multi-classification cross entropy to enhance the text pairing ability of block-level visual vectors under a visual-language model. At the same time, a text embedding containing type semantic information and quantity semantic information, as well as a visual embedding tensor with the same dimension are constructed. After calculation by a scaled dot product model, an attention score is output, and finally a feature representation of the block-level visual embedding in the text embedding vector space is obtained. A multi-scale visual feature interaction module based on a dual-branch Cross-ViT structure is constructed to achieve access and fusion between visual features of different scales. The present invention enhances the model's feature extraction performance for target details and contextual information of different scales in complex scenes, and provides good visual encoding for the downstream density map regression task of target counting.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a target counting method and system based on multimodal and multi-scale cross-attention. Background Art

[0002] Deep learning-based object counting technology has been widely used in many fields in recent years, especially object counting models based on the Transformer backbone network. In real-world scenarios, object counting tasks are also widely used in many production fields such as agriculture, industry, and biological sciences. For example, estimating the total number of crops in the field, counting the total number of industrial products on factory production lines, and counting wildlife in the wild. With the widespread expansion of application scenarios, object counting tasks have become more challenging, including but not limited to unfamiliar counting object types, complex background interference, and uneven target distribution. These issues limit the counting performance of the algorithm and are currently difficult problems that need to be solved. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems in the prior art of unfamiliar counting object types, complex background interference, and uneven target distribution that affect the performance of target counting algorithms, and to provide a target counting method and system based on multimodal and multi-scale cross-attention.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The object counting method based on multimodal and multi-scale cross attention includes:

[0006] Segment the captured image through a visual encoder to obtain a number of block-level visual features, and then obtain a block-level visual embedding vector; each of the block-level visual embedding vectors corresponds to a text embedding vector;

[0007] Construct a similarity matrix based on the block-level visual embedding vector and text embedding vector;

[0008] Based on the constructed similarity matrix and multi-class cross entropy loss function, a contrast loss is obtained between the aligned block-level visual features and the quantity text embedding, thereby obtaining an optimized fine-tuned visual encoder; the quantity text embedding is to embed the text into a vector related to the quantity;

[0009] Given an image containing several similar specific objects, a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated by fine-tuning the visual encoder and text encoder based on CLIP ViT.

[0010] The visual embedding features and text embedding vectors are input into the text-image cross-interaction network (TICM) based on the cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thus achieving a joint representation of multimodal information.

[0011] The image-image cross interaction module IICM based on the dual-branch Cross-ViT structure fuses multi-scale visual embedding features and outputs the target counting results.

[0012] A further improvement of the present invention is:

[0013] The method constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector, specifically: aligning the block-level visual embedding vector and the text embedding vector, constructing a similarity matrix between the block-level visual embedding vector and the text embedding vector, and using the one-hot true value label as the ground-truth to supervise the normalized similarity vector.

[0014] The similarity matrix between the block-level visual embedding vector and the text embedding vector is constructed, and the normalized similarity vector is supervised by the one-hot true value label as the ground-truth. Specifically, the cosine similarity of the N pairs of true matching visual embeddings and text embeddings is maximized, and the remaining N 2 - cosine similarity of N pairs of mismatched visual embeddings and text embeddings;

[0015] Calculate the cosine similarity between each of the N block-level visual embedding vectors and the N count text embeddings, and use the softmax function to normalize the similarity information to obtain an N×N similarity matrix, specifically:

[0016]

[0017] Among them, P i and T j They represent block-level visual embedding and text embedding vectors respectively, τ represents the temperature coefficient; k represents the index of the text embedding vector.

[0018] Based on the constructed similarity matrix and multi-classification cross entropy loss function, a comparative loss of aligning block-level visual features and number text embeddings is obtained. Specifically, based on the similarity matrix between the block-level visual embedding vector and the text embedding vector and the similarity vector normalized after one-hot true value label supervision, the loss of similarity between the count text embedding vector matched by the block-level visual embedding vector and the true value result and the loss of similarity between the block-level visual embedding vector matched by the count text embedding vector and the true value result are obtained respectively; when the above losses are minimized, an optimal fine-tuned visual encoder is obtained;

[0019] The loss of similarity between the count text embedding vector matched by the block-level visual embedding vector and the ground truth result is specifically:

[0020]

[0021] Among them, block-level visual loss Loss P Measures the difference between the probability distribution predicted by the model and the true value distribution of one-hot encoding, indicating the loss of similarity between multiple count text embedding vectors matched by block-level visual embedding vectors and the true value results;

[0022] Transpose the cosine similarity matrix and calculate the count text loss Loss in the same way T , which measures the loss of counting text embeddings that match multiple block-level visual embeddings;

[0023]

[0024] The count text loss Loss T and block-level visual loss Loss P The average of the two parts is used to obtain the comprehensive loss function Loss of the aligned count text Count-ICL .

[0025] For a given image containing several similar specific objects, a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated through a fine-tuned visual encoder based on CLIP ViT and a text encoder Text Encoder. Specifically, for an H×W×C input image, a set of visual embedding vectors are generated through a CLIP ViT-based visual encoder and a dimensionality flattening operation; the type of the target counting object contained in the image is combined with different quantity texts, and the results are passed through the text encoder to generate a set of text embedding vectors containing type semantic information and quantity semantic information.

[0026] The visual embedding features and text embedding vectors are input into the text-image cross-interaction network TICM based on the cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thereby realizing the joint representation of multimodal information. Specifically,

[0027] The visual embedding vector and text embedding vector are denoted as Ev and Et respectively. After the preprocessing operation of the self-attention layer to capture long-range visual dependencies, they are used as the input variables of the cross-attention mechanism operation. The visual embedding vector Ev and the text embedding vector Et are used as the query vector set Q and the key vector set K respectively. The attention scores of the visual features in the text feature vector space are calculated through the scaled dot product model. The text embedding Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output.

[0028] The attention score of the visual feature in the text feature vector space is calculated as follows: based on the scaled dot product model, the Q matrix obtained by the visual embedding vector Ep and the K matrix obtained by the text embedding Et are transposed and the dot product is calculated to obtain the cosine similarity matrix between the vectors; then divided by the arithmetic square root of the dimension; the result is normalized by the softmax function as the final output attention score; the Q matrix is ​​the query vector set Q; the K matrix is ​​the key vector set K; the visual embedding vector Ep is the visual embedding vector Ev.

[0029] The output block-level visual embedding is a feature representation of the text embedding vector space, specifically: the text embedding vector Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output; the output visual features fuse type semantic information and quantity semantic information to achieve a joint representation of multimodal information.

[0030] The image-image cross interaction module IICM based on the dual-branch Cross-ViT structure fuses multi-scale visual embedding features and outputs the target counting result. Specifically, the visual features of different scales are pre-processed by the self-attention layer to capture long-range visual dependencies and then serve as the input of the multi-scale visual feature cross attention layer; the visual features of different scales are placed at both ends of the dual-branch structure, and the large-scale visual features E are used to embed the features. L The cls token accesses the small-scale visual features E S , and using small-scale visual features E S The cls token accesses large-scale visual features E L , realizing multi-scale visual feature fusion;

[0031] Realize multi-scale visual feature fusion, specifically:

[0032] Respectively express the small-scale visual features E S , and large-scale visual features E L Definition:

[0033]

[0034] The variables with the superscript cls represent the global class token of the visual feature, and the variables with the superscript patch represent the block-level embedding vector of the visual feature. The global class token is used as the query vector Q for attention calculation, and the global class tokens of visual features of different scales are concatenated with the block embedding vectors as the key vector K and value vector V for attention calculation to calculate the attention score and the final attention output.

[0035] Q = [xl cls ] (6)

[0036]

[0037] Based on the above-mentioned cross-attention mechanism, access between visual features of different scales is achieved, the interaction of multi-scale visual embedding vectors is established, and multi-scale feature fusion is realized; the global class token will be spliced ​​with the original block-level embedding vector.

[0038] An object counting system based on multimodal and multi-scale cross-attention, including:

[0039] A segmentation module, which segments the captured image through a visual encoder to obtain a number of block-level visual features, and then obtains a block-level visual embedding vector; each of the block-level visual embedding vectors corresponds to a text embedding vector;

[0040] A construction module, wherein the construction module constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector;

[0041] A first acquisition module, wherein the first acquisition module obtains the contrast loss of aligned block-level visual features and quantity text embedding based on the constructed similarity matrix and the multi-classification cross entropy loss function, thereby obtaining an optimized fine-tuned visual encoder; the quantity text embedding is to embed the text into a vector related to the quantity in the vector;

[0042] The second acquisition module, which, given an image containing several similar specific objects, generates a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information through a fine-tuned visual encoder and text encoder based on CLIP ViT;

[0043] An output module, which inputs the visual embedding features and the text embedding vector into a text-image cross-interaction network (TICM) based on a cross-attention mechanism to obtain a cross-modal mapping from visual modality features to textual modality features, thereby achieving a joint representation of multimodal information;

[0044] The fusion module is based on an image-image cross interaction module (IICM) of a dual-branch Cross-ViT structure, which fuses multi-scale visual embedding features and outputs target counting results.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention constructs a similarity matrix and then establishes a contrastive learning loss based on multi-classification cross entropy to enhance the text pairing ability of block-level visual vectors under the visual-language model. At the same time, a text embedding containing type semantic information and quantity semantic information, as well as a visual embedding tensor with the same dimension are constructed. After calculation by the scaled dot product model, the attention score is output, and finally the feature representation of the block-level visual embedding in the text embedding vector space is obtained. A multi-scale visual feature interaction module based on a dual-branch Cross-ViT structure is constructed to achieve access and fusion between visual features of different scales. The present invention realizes multi-scale feature fusion, enhances the model's feature extraction performance for target details and contextual information of different scales in complex scenes, and provides good visual encoding for the downstream density map regression task of target counting. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 Schematic diagram of the process of the object counting method based on multimodal multi-scale cross attention of the present invention;

[0049] Figure 2 Schematic diagram of the structure of the target counting system based on multimodal multi-scale cross attention of the present invention;

[0050] Figure 3 This is the overall structure diagram of the target counting model based on multimodal and multi-scale cross attention of the present invention;

[0051] Figure 4 This is a network structure diagram of the proxy task of aligning cross-modal features of the present invention;

[0052] Figure 5 This is a network structure diagram of the text-image cross-interaction module of the present invention;

[0053] Figure 6 This is the network structure diagram of the image-image cross interaction module of the present invention DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0056] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0057] In the description of the embodiments of the present invention, it should be noted that if the terms "upper," "lower," "horizontal," "inner," etc. appear, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the inventive product is typically placed when in use. These terms are merely for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0058] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0059] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0060] The present invention is described in further detail below with reference to the accompanying drawings:

[0061] See also Figure 1 The present invention discloses a target counting method based on multimodal multi-scale cross attention, comprising:

[0062] S101, segmenting the captured image through a visual encoder to obtain a number of block-level visual features, and then obtaining a block-level visual embedding vector; each of the block-level visual embedding vectors corresponds to a text embedding vector;

[0063] S102, constructing a similarity matrix based on the block-level visual embedding vector and the text embedding vector;

[0064] The block-level visual embedding vector and the text embedding vector are aligned, and the similarity matrix between the block-level visual embedding vector and the text embedding vector is constructed. The one-hot true value label is used as the ground-truth to supervise the normalized similarity vector.

[0065] Maximize the cosine similarity between the N pairs of true matching visual embeddings and text embeddings, and minimize the remaining N 2 - cosine similarity of N pairs of mismatched visual embeddings and text embeddings;

[0066] Calculate the cosine similarity between each of the N block-level visual embedding vectors and the N count text embeddings, and use the softmax function to normalize the similarity information to obtain an N×N similarity matrix, specifically:

[0067]

[0068] Among them, P i and T j Represents block-level visual embedding and text embedding vectors, τ represents the temperature coefficient; k represents the index of the text embedding vector

[0069] S103, based on the constructed similarity matrix and the multi-classification cross entropy loss function, obtaining the contrast loss of the aligned block-level visual features and the quantity text embedding, thereby obtaining an optimized fine-tuned visual encoder; the quantity text embedding is to embed the text into a vector related to the quantity in the vector;

[0070] Based on the similarity matrix between the block-level visual embedding vector and the text embedding vector and the similarity vector normalized after one-hot ground truth label supervision, the loss of the similarity between the counted text embedding vector matched by the block-level visual embedding vector and the ground truth result and the loss of the similarity between the counted text embedding vector matched by the block-level visual embedding vector and the ground truth result are obtained respectively. When the above losses are minimized, the optimal fine-tuned visual encoder is obtained.

[0071] The loss of similarity between the count text embedding vector matched by the block-level visual embedding vector and the ground truth result is specifically:

[0072]

[0073] Among them, the block-level visual loss LossP measures the difference between the probability distribution predicted by the model and the true value distribution of one-hot encoding, indicating the loss of similarity between multiple count text embedding vectors matched by the block-level visual embedding vector and the true value result;

[0074] Transpose the cosine similarity matrix and calculate the count text loss Loss in the same way T , which measures the loss of counting text embeddings that match multiple block-level visual embeddings;

[0075]

[0076] The count text loss Loss T and block-level visual loss Loss P The average of the two parts is used to obtain the comprehensive loss function Loss of the aligned count text Count-ICL .

[0077] S104: Given an image containing several similar specific objects, a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated by fine-tuning the visual encoder and text encoder based on CLIP ViT.

[0078] For an H×W×C input image, a set of visual embedding vectors is generated after a CLIP ViT-based visual encoder and dimensionality flattening operation. The type of target counting object contained in the image is combined with different quantity texts and passed through a text encoder to generate a set of text embeddings containing type semantic information and quantity semantic information.

[0079] S105: Input the visual embedding features and the text embedding vector into a text-image cross-interaction network (TICM) based on a cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thereby achieving a joint representation of multimodal information.

[0080] The visual embedding vector and text embedding vector are denoted as Ev and Et respectively. After the preprocessing operation of the self-attention layer to capture long-range visual dependencies, they are used as the input variables of the cross-attention mechanism operation. The visual embedding vector Ev and the text embedding vector Et are used as the query vector set Q and the key vector set K respectively. The attention scores of the visual features in the text feature vector space are calculated through the scaled dot product model. The text embedding Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output.

[0081] Calculate the attention score of the visual feature in the text feature vector space, specifically: based on the scaled dot product model, obtain the Q matrix obtained by the visual embedding vector Ep and the K matrix obtained by the text embedding Et, transpose and calculate the dot product between them to obtain the cosine similarity matrix between the vectors; then divide it by the arithmetic square root of the dimension; normalize the result through the softmax function as the final output attention score; the Q matrix is ​​the query vector set Q; the K matrix is ​​the key vector set K; the visual embedding vector Ep is the visual embedding vector Ev.

[0082] Output block-level visual embedding feature representation in the text embedding vector space, specifically:

[0083] The text embedding vector Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output; the output visual feature integrates the type semantic information and quantity semantic information to realize the joint representation of multimodal information.

[0084] S106, based on the dual-branch Cross-ViT structure, the image-image cross interaction module IICM fuses multi-scale visual embedding features and outputs the target counting results.

[0085] After the preprocessing operation of the self-attention layer to capture the long-range visual dependency, the visual features of different scales are used as the input of the multi-scale visual feature cross attention layer. The visual features of different scales are placed at both ends of the dual-branch structure, respectively, with the large-scale visual features E L The cls token accesses the small-scale visual features E S , and using small-scale visual features E S The cls token accesses large-scale visual features E L , realizing multi-scale visual feature fusion.

[0086] Realize multi-scale visual feature fusion, specifically:

[0087] Respectively express the small-scale visual features E S, and large-scale visual features E L Definition:

[0088]

[0089] The variables with the superscript cls represent the global class token of the visual feature, and the variables with the superscript patch represent the patch-level embedding vector of the visual feature. The global class token is used as the query vector Q for attention calculation. The global class token of visual features of different scales is concatenated with the patch embedding vector as the key vector K and value vector V for attention calculation. The attention score and final attention output are calculated.

[0090] Q = [xl cls ] (6)

[0091]

[0092] Based on the above-mentioned cross-attention mechanism, access between visual features of different scales is achieved, the interaction of multi-scale visual embedding vectors is established, and multi-scale feature fusion is realized; the global class token will be spliced ​​with the original block-level embedding vector.

[0093] See also Figure 2 The present invention discloses a target counting system based on multimodal multi-scale cross attention, comprising:

[0094] A segmentation module, which segments the captured image through a visual encoder to obtain a number of block-level visual features, and then obtains a block-level visual embedding vector; each of the block-level visual embedding vectors corresponds to a text embedding vector;

[0095] A construction module, wherein the construction module constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector;

[0096] A first acquisition module, wherein the first acquisition module obtains the contrast loss of aligned block-level visual features and quantity text embedding based on the constructed similarity matrix and the multi-classification cross entropy loss function, thereby obtaining an optimized fine-tuned visual encoder; the quantity text embedding is to embed the text into a vector related to the quantity in the vector;

[0097] The second acquisition module, which, given an image containing several similar specific objects, generates a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information through a fine-tuned visual encoder and text encoder based on CLIP ViT;

[0098] An output module, which inputs the visual embedding features and the text embedding vector into a text-image cross-interaction network (TICM) based on a cross-attention mechanism to obtain a cross-modal mapping from visual modality features to textual modality features, thereby achieving a joint representation of multimodal information;

[0099] The fusion module is based on an image-image cross interaction module (IICM) of a dual-branch Cross-ViT structure, which fuses multi-scale visual embedding features and outputs target counting results.

[0100] This paper designs a proxy task to align cross-modal features, namely: for N 2 For visual-text pairings, the cosine similarity between the N pairs of true matches between the visual embedding and the textual embedding is maximized, while the cosine similarity between the remaining N2-N pairs of false matches between the visual embedding and the textual embedding is minimized. To fine-tune the visual encoder, this task calculates the cosine similarity between N block-level visual embedding vectors and N count text embeddings, constructs a similarity matrix, and then establishes a contrastive learning loss based on multi-class cross-entropy to enhance the text pairing ability of block-level visual embeddings under the visual-language model (VLM).

[0101] Furthermore, in the present invention, a text embedding containing type semantic information and quantity semantic information and a visual embedding tensor with the same dimension are constructed, which are denoted as E t and E v After the preprocessing operation of the self-attention layer to capture long-range visual dependencies, the two are jointly used as input variables for the cross-attention mechanism operation. After calculation by the scaled dot product model, the attention score is output, and finally the feature representation of the block-level visual embedding in the text embedding vector space is obtained. The output feature result is a joint representation of text modality information and visual modality information, which enables the network to more accurately construct the representation of visual features in the counting-related text vector space, enhance the model's ability to locate and perceive details in dense areas under target counting tasks, and improve the model's counting performance.

[0102] Furthermore, in the present invention, a multi-scale visual feature interaction module based on a dual-branch Cross-ViT structure is constructed to achieve access and fusion between visual features of different scales. Among them, the global CLS token of the visual feature is used as the query vector for self-attention calculation to establish interaction between features of different scales in a relatively efficient way. This module makes full use of the ability of the self-attention mechanism of the Transformer model to capture long-distance visual dependencies, combines the mechanism of mutual access between word encodings, and uses the attention mechanism to dynamically focus on those features that are most relevant to specific visual tasks, thereby achieving multi-scale feature fusion and enhancing the model's feature extraction performance for target details and contextual information of different scales in complex scenarios, providing good visual encoding for the downstream density map regression task of target counting.

[0103] Example:

[0104] See also Figure 3 The present invention discloses a target counting method based on multimodal and multi-scale cross-attention, specifically a target counting method based on the CLIP visual-language model. It uses counting text to guide the understanding of the correspondence between text and visual features, focuses the attention of feature extraction on image areas containing dense target objects, and optimizes the recognition and counting accuracy of any specified target object in complex scenes.

[0105] The following steps are involved:

[0106] Step 1: Design a proxy task to align cross-modal features and construct a loss function based on contrastive learning

[0107] First, for any two-dimensional input image, the Transformer-based encoder decomposes it into N one-dimensional visual word encoding vectors, each of which has a corresponding true number value. In the vision-language model, each visual word encoding vector should have the highest similarity match with the count text feature vector corresponding to the true number value of that region. To achieve the pairing of visual and text modality features, a corresponding pre-training task is designed for cross-modal feature alignment.

[0108] See also Figure 4 The proxy task of aligning cross-modal features is designed as follows: each element of the block-level visual embedding feature corresponds to a block of the original input image, and each block corresponds to a corresponding target count truth value. For the density map truth value H(x) of the labeled dataset, the density map truth value with the same resolution as the block-level visual embedding matrix should be obtained through average pooling, and the corresponding count text feature vector should be constructed for each block-level visual embedding vector.

[0109] Furthermore, according to the output density distribution truth map M, a corresponding count truth token sequence is generated for each block-level visual feature. The count truth token sequence is encoded by the text encoder of CLIP, and the output matching count text embedding feature T with a sequence length of N is output. In this way, for each vector element x in the block-level visual embedding feature sequence, i , have corresponding text embedded in T i Corresponding to it.

[0110] Furthermore, the proxy task of aligning cross-modal features is defined as: for N 2 For visual-text pairs, maximize the cosine similarity between the N pairs of true matching visual embeddings and text embeddings, and minimize the remaining N 2 - Cosine similarity of N pairs of mismatched visual embeddings and text embeddings. In this task, the cosine similarity between N block-level visual embedding vectors and N count text embeddings is calculated, and the similarity matrix is ​​constructed by softmax normalization under scaling operation based on temperature coefficient τ (experimentally set to 0.07).

[0111]

[0112] Where Sim is the similarity matrix; P i T j is the inner product between the block-level visual embedding and the text embedding vector; τ is the temperature coefficient.

[0113] Use the one-hot true value label and the normalized similarity vector to perform multi-classification cross entropy calculation to obtain the block-level visual loss Loss P .

[0114]

[0115] Furthermore, the cosine similarity matrix is ​​transposed, and the count text loss based on multi-classification cross entropy is calculated in the same way. T .

[0116]

[0117] Finally, the count text loss Loss T and block-level visual loss Loss P The average of the two parts is used to obtain the comprehensive loss function Loss of the aligned count text Count-ICL , represents the optimization direction of visual encoder fine-tuning, and is used as the supervisory signal for training for several (experimentally set to 30) iterations.

[0118] Step 2: Obtain visual features with joint representation capabilities through a cross-modal multi-scale network based on the cross-attention mechanism

[0119] See also Figure 3 ,The image first passes through the CLIP ViT-based feature extraction network and dimension flattening operation,,generating a set of visual embedding tensors of size 197×512, which contains the global embedding vector and,block-level embedding vector of the image.

[0120] For further information, see Figure 3 In the downstream task of density map regression, the image will pass through the multimodal cross-interaction module and the multi-scale fusion cross-interaction module respectively. Figure 5 First, before passing through the cross attention layer, it is necessary to first undergo alignment calculation and embed the block-level visual feature Z through linear projection v ∈R p2×d Projected to the embedding vector E corresponding to each text t i ∈R n Same channel dimensions.

[0121] The visual embedding tensor of size 197×512 and the text embedding of size 6×512 are obtained, which are denoted as E v and E t The two have the same vector dimensions and, after preprocessing by the self-attention layer to capture long-range visual dependencies, serve together as the query vector set Q and key vector set K for the cross-attention mechanism. Finally, based on a scaled dot product model, a scaling operation by dividing by the arithmetic square root of the dimension, and softmax normalization, the attention score of the visual features in the text feature vector space is calculated.

[0122]

[0123] An inner product operation is performed between the attention score and the corresponding V matrix, outputting a feature representation of the block-level visual embedding in the text embedding vector space. This allows the block-level visual feature to not only obtain visual information based on its own embedding vector but also, through the features of the corresponding count text modality, obtain the "quantity" of text information that has the highest cosine similarity, thereby learning a joint representation that combines visual and textual semantic information.

[0124]

[0125] See also Figure 3 , the previously output visual embedding is scaled and then fed into the text-image cross-interaction module to output visual embeddings of different scales. Figure 6, feature maps of different scales will be input into the image-image cross interaction module. The module is based on a dual-branch Cross-ViT structure and adopts a cross attention mechanism to use large-scale visual features E L cls token to access the small-scale visual features E S , and using small-scale visual features E S cls token to access large-scale visual features E L , to enhance the model's ability to capture local features of images from different scales.

[0126]

[0127] Where: xl cls A global class token for large-scale visual features; is the block-level embedding vector of large-scale visual features; xs cls It is a global class token of small-scale visual features; is the block-level embedding vector of small-scale visual features.

[0128] Furthermore, in the attention calculation, the global class token of the large-scale visual feature is used as the query vector Q of the attention calculation; the global class token of the large-scale visual feature is spliced ​​with the block embedding vector of the small-scale feature as the key vector K and value vector V of the attention calculation.

[0129] Q = [xl cls ] (15)

[0130]

[0131] The scaled dot product model is used to calculate the attention score and the final attention output. The output global CLS token needs to be concatenated with the original block-level embedding vector to restore and integrate the original large-scale visual features at the channel dimension level.

[0132] See also Figure 3 The block-level visual embeddings of different scales are fed into a density map regressor consisting of multiple convolutional layers and upsampling layers to achieve the fusion of multi-scale visual features. The final output is a point-level density map with a resolution of 224×224 as the inference result. Taking into account the scale differences of the input images, the input images of different sizes are cropped according to their actual aspect ratio using sliding windows to obtain the predicted density maps of multiple sliding windows. These density maps are then spliced ​​and fused through a nonlinear weighted average.

[0133] Step 3: Density map regression loss function

[0134] During the training phase, data augmentation was applied. This involves altering and augmenting the training set images using methods such as Gaussian noise, Gaussian blur, and color dithering. Furthermore, horizontal flipping and rotation were used to simulate the perspectives of images from different directions and positions in real-world scenes. The original images from the augmented training set and the ground-truth density map served as input for supervised training of the model network.

[0135] Furthermore, the target counting task of this method adopts the mean square error (MSE) loss function, which is used to calculate the mean of the sum of squares of the differences between the predicted density map and the true density map in the density map regression task.

[0136]

[0137] Where: f is the predicted output density map; y is the true value density map; W is the image width; H is the image height; Loss D is the density map loss.

[0138] Furthermore, for a predicted density map with an output size of W×H and a corresponding true density map, the loss is calculated by taking out all its pixels, calculating the square of the error of each pair of pixels one by one, and then averaging the squared error values ​​of all pixels to obtain the final mean square error at the matrix level.

[0139] Step 4: Use in downstream object counting tasks

[0140] This paper provides a method for object counting based on multimodal and multiscale cross-attention, offering a universal pre-trained model for any type of downstream object counting task. This model can replace the backbone of any downstream object counting task. In experiments, the paper used the CLIP-based Vision Transformer backbone network as the model network for visual feature extraction and generated a prediction density map based on a single-stage regression task.

[0141] Furthermore, the present invention can be applied to downstream target counting tasks of any target type, including but not limited to crowd counting, bird counting, and vehicle counting. When performing inference on a specific type of downstream target counting task, the present invention requires using English nouns of that type as natural language prompts to guide the target counting model to accurately focus on the objects of interest in the input image. For example, in a crowd counting task, "people" should be used as a prompt; in a bird counting task, "birds" should be used as a prompt, and so on.

[0142] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. The object counting method based on multimodal and multi-scale cross attention is characterized by: include: The collected image is segmented through a visual encoder to obtain several block-level visual features, and then a block-level visual embedding vector is obtained; Each block-level visual embedding vector corresponds to a text embedding vector; Construct a similarity matrix based on the block-level visual embedding vector and text embedding vector; Based on the constructed similarity matrix and multi-class cross entropy loss function, we obtain the contrastive loss of aligning block-level visual features with quantitative text embeddings, and then obtain the optimized fine-tuned visual encoder; The quantity text embedding is to embed the text into a vector related to the quantity; Given an image containing several similar specific objects, a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated by fine-tuning the visual encoder and text encoder based on CLIP ViT. The visual embedding features and text embedding vectors are input into the text-image cross-interaction network (TICM) based on the cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thus achieving a joint representation of multimodal information. The image-image cross interaction module IICM based on the dual-branch Cross-ViT structure fuses multi-scale visual embedding features and outputs the target counting results.

2. The object counting method based on multimodal multi-scale cross attention according to claim 1 is characterized in that: The method constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector, specifically: aligning the block-level visual embedding vector and the text embedding vector, constructing a similarity matrix between the block-level visual embedding vector and the text embedding vector, and using the one-hot true value label as the ground-truth to supervise the normalized similarity vector.

3. The object counting method based on multimodal multi-scale cross attention according to claim 2 is characterized in that: The similarity matrix between the block-level visual embedding vector and the text embedding vector is constructed, and the normalized similarity vector is supervised by the one-hot true value label as the ground-truth. Specifically, the maximum N The cosine similarity of the visual embedding and text embedding of the true match is minimized. N 2 - N Cosine similarity between visual and textual embeddings for mismatches; calculate N block-level visual embedding vectors and N Count the cosine similarities between pairwise text embeddings and use softmax The function normalizes the similarity information and obtains N × N The similarity matrix is: （1） in, P i and T j represent block-level visual embedding and text embedding vectors respectively, τ represents the temperature coefficient; Represents the index of the text embedding vector.

4. The object counting method based on multimodal multi-scale cross attention according to claim 3 is characterized in that Based on the constructed similarity matrix and multi-classification cross entropy loss function, a comparative loss of aligning block-level visual features and number text embeddings is obtained. Specifically, based on the similarity matrix between the block-level visual embedding vector and the text embedding vector and the similarity vector normalized after one-hot true value label supervision, the loss of similarity between the count text embedding vector matched by the block-level visual embedding vector and the true value result and the loss of similarity between the block-level visual embedding vector matched by the count text embedding vector and the true value result are obtained respectively; when the above losses are minimized, an optimal fine-tuned visual encoder is obtained; The loss of similarity between the count text embedding vector matched by the block-level visual embedding vector and the ground truth result is specifically: Among them, block-level visual loss Loss P Measures the difference between the probability distribution predicted by the model and the true value distribution of one-hot encoding, indicating the loss of similarity between multiple count text embedding vectors matched by block-level visual embedding vectors and the true value results; Transpose the cosine similarity matrix and calculate the count text loss in the same way Loss T , which measures the loss of counting text embeddings that match multiple block-level visual embeddings; Will count text loss Loss T and block-level visual loss Loss P The average of the two parts is used to obtain the comprehensive loss function of aligned count text .

5. The object counting method based on multimodal multi-scale cross attention according to claim 4 is characterized in that: For a given picture containing several similar specific objects, a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated by fine-tuning the visual encoder and text encoder based on CLIP ViT. Specifically, for The input image is processed through a CLIPViT-based visual encoder and dimension flattening operation to generate a set of visual embedding vectors; the type of target counting object contained in the image is combined with different quantity texts and processed through a text encoder to generate a set of text embeddings containing type semantic information and quantity semantic information.

6. The object counting method based on multimodal multi-scale cross attention according to claim 5 is characterized in that: The visual embedding features and text embedding vectors are input into the text-image cross-interaction network TICM based on the cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thereby realizing the joint representation of multimodal information. Specifically, The visual embedding vector and text embedding vector are denoted as Ev and Et respectively. After the preprocessing operation of the self-attention layer to capture long-range visual dependencies, they are used as the input variables of the cross-attention mechanism operation. The visual embedding vector Ev and the text embedding vector Et are used as the query vector set Q and the key vector set K respectively. The attention scores of the visual features in the text feature vector space are calculated through the scaled dot product model. The text embedding Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output.

7. The object counting method based on multimodal multi-scale cross attention according to claim 6 is characterized in that: The attention score of the visual feature in the text feature vector space is calculated by: based on the scaled dot product model, the Q matrix obtained by the visual embedding vector Ep and the K matrix obtained by the text embedding Et are transposed and the dot product is calculated to obtain the cosine similarity matrix between the vectors; Then divide by the square root of the dimension; The result is normalized by the softmax function and used as the final output attention score; The Q matrix is ​​the query vector set Q; the K matrix is ​​the key vector set K; and the visual embedding vector Ep is the visual embedding vector Ev.

8. The object counting method based on multimodal multi-scale cross attention according to claim 7 is characterized in that: The output block-level visual embedding is a feature representation of the text embedding vector space, specifically: the text embedding vector Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output; the output visual features fuse type semantic information and quantity semantic information to achieve a joint representation of multimodal information.

9. The object counting method based on multimodal multi-scale cross attention according to claim 8, characterized in that: The image-image cross interaction module IICM based on the dual-branch Cross-ViT structure fuses multi-scale visual embedding features and outputs the target counting result. Specifically, the visual features of different scales are pre-processed by the self-attention layer to capture long-range visual dependencies and then serve as the input of the multi-scale visual feature cross attention layer; the visual features of different scales are placed at both ends of the dual-branch structure, and the large-scale visual features are used to embed the features. E L The cls token accesses small-scale visual features E S , and using small-scale visual features E S The cls token accesses large-scale visual features E L , realizing multi-scale visual feature fusion; Realize multi-scale visual feature fusion, specifically: Respectively express the visual features of small scale E S , and large-scale visual features E L Definition: Among them, the superscript cls The variable represents the global class token of the visual feature, with a superscript patch The variable represents the block-level embedding vector of the visual feature; the global class token is used as the query vector Q for attention calculation, and the global class token of visual features of different scales is concatenated with the block embedding vector as the key vector K and value vector V for attention calculation to calculate the attention score and the final attention output; Based on the above-mentioned cross-attention mechanism, access between visual features of different scales is achieved, the interaction of multi-scale visual embedding vectors is established, and multi-scale feature fusion is realized; the global class token will be spliced ​​with the original block-level embedding vector.

10. Object counting system based on multimodal multi-scale cross attention, characterized by: include: A segmentation module, which segments the collected image through a visual encoder to obtain a number of block-level visual features, and then obtains a block-level visual embedding vector; Each block-level visual embedding vector corresponds to a text embedding vector; A construction module, wherein the construction module constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector; a first acquisition module, which obtains the contrast loss of aligned block-level visual features and quantitative text embedding based on the constructed similarity matrix and the multi-classification cross entropy loss function, thereby obtaining an optimized fine-tuned visual encoder; The quantity text embedding is to embed the text into a vector related to the quantity; The second acquisition module, which, given an image containing several similar specific objects, generates a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information through a fine-tuned visual encoder and text encoder based on CLIP ViT; An output module, which inputs the visual embedding features and the text embedding vector into a text-image cross-interaction network (TICM) based on a cross-attention mechanism to obtain a cross-modal mapping from visual modality features to textual modality features, thereby achieving a joint representation of multimodal information; The fusion module is based on an image-image cross interaction module (IICM) of a dual-branch Cross-ViT structure, which fuses multi-scale visual embedding features and outputs target counting results.

Images