Target counting method and system based on multi-modal multi-scale cross attention

Abstract

The invention discloses a multi-modal multi-scale cross attention-based target counting method and system, and the method comprises the steps: constructing a similarity matrix based on a block-level visual embedding vector and a text embedding vector; and further establishing comparison learning loss based on multi-classification cross entropy so as to enhance the text pairing capability of the block-level visual vector under the vision-language model. Meanwhile, text embedding containing type semantic information and quantity semantic information and a visual embedding tensor with the same dimension as the text embedding are constructed, an attention score is output through calculation of a zoom dot product model, and finally feature representation of block-level visual embedding in a text embedding vector space is obtained. A multi-scale visual feature interaction module based on a double-branch Cross-ViT structure is constructed, and access and fusion among visual features of different scales are achieved. According to the method, the feature extraction performance of the model on target details and context information of different scales in a complex scene is enhanced, and good visual coding is provided for a downstream density map regression task of target counting.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a target counting method and system based on multi-modal multi-scale cross-attention. Background Art

[0002] In recent years, the target counting technology based on deep learning has been widely used in many fields, especially the target counting model based on the Transformer backbone network. In actual scenarios, target counting tasks are also widely used in many production fields such as agriculture, industry, and biological sciences, such as estimating the total number of crops in the field, counting the total number of industrial products on factory production lines, and counting the presence of wild animals in the wild. With the wide expansion of application scenarios, target counting tasks have ushered in more challenges, including but not limited to unfamiliar types of counting objects, complex background interference, and uneven target distribution. These problems limit the counting performance of the algorithm and are also the difficulties that need to be solved urgently. Summary of the invention

[0003] The purpose of the present invention is to solve the problem in the prior art that unfamiliar types of counting objects, complex background interference, and uneven target distribution affect the performance of target counting algorithms, and to provide a target counting method and system based on multimodal and multi-scale cross-attention.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The object counting method based on multimodal and multi-scale cross-attention includes:

[0006] Segmenting the captured image through a visual encoder to obtain a number of block-level visual features, and then obtaining a block-level visual embedding vector; each of the block-level visual embedding vectors corresponds to a text embedding vector;

[0007] Construct a similarity matrix based on the block-level visual embedding vector and text embedding vector;

[0008] Based on the constructed similarity matrix and multi-classification cross entropy loss function, the contrast loss of aligned block-level visual features and quantity text embedding is obtained, thereby obtaining an optimized fine-tuned visual encoder; the quantity text embedding is to embed the text into a vector related to the quantity in the vector;

[0009] For a given picture containing several specific objects of the same type, a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated by fine-tuning the visual encoder and text encoder Text Encoder based on CLIP ViT.

[0010] The visual embedding features and text embedding vectors are input into the text-image cross-interaction network TICM based on the cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thus realizing the joint representation of multimodal information.

[0011] The image-image cross interaction module IICM based on the dual-branch Cross-ViT structure fuses multi-scale visual embedding features and outputs the target counting results.

[0012] A further improvement of the present invention is:

[0013] The method constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector, specifically: aligning the block-level visual embedding vector and the text embedding vector, constructing a similarity matrix between the block-level visual embedding vector and the text embedding vector, and using a one-hot true value label as the ground-truth to supervise the normalized similarity vector.

[0014] The similarity matrix between the block-level visual embedding vector and the text embedding vector is constructed, and the normalized similarity vector is supervised by the one-hot true value label as the ground-truth, specifically: the cosine similarity of the N pairs of true matching visual embeddings and text embeddings is maximized, and the remaining N 2 - Cosine similarity of N pairs of mismatched visual embeddings and text embeddings;

[0015] Calculate the cosine similarity between the N block-level visual embedding vectors and the N count text embeddings, and use the softmax function to normalize the similarity information to obtain an N×N similarity matrix, which is:

[0016]

[0017] Among them, P i and T j They represent block-level visual embedding and text embedding vectors respectively, τ represents the temperature coefficient; k represents the index of the text embedding vector.

[0018] Based on the constructed similarity matrix and multi-classification cross entropy loss function, the comparative loss of aligned block-level visual features and quantity text embedding is obtained, specifically: based on the similarity matrix of the block-level visual embedding vector and the text embedding vector and the similarity vector normalized by one-hot true value label supervision, the loss of the similarity of the count text embedding vector matched by the block-level visual embedding vector and the true value result and the loss of the similarity of the block-level visual embedding vector matched by the count text embedding vector and the true value result are obtained respectively; when the above losses are minimized, the optimal fine-tuned visual encoder is obtained;

[0019] The loss of similarity between the count text embedding vector matched by the block-level visual embedding vector and the true value result is specifically:

[0020]

[0021] Among them, block-level visual loss Loss P Measures the difference between the probability distribution predicted by the model and the true value distribution of one-hot encoding, representing the loss of similarity between multiple count text embedding vectors matched by block-level visual embedding vectors and the true value results;

[0022] Transpose the cosine similarity matrix and calculate the count text loss Loss in the same way T , which measures the loss of counting text embedding vectors matched by multiple block-level visual embedding vectors;

[0023]

[0024] The count text loss Loss T and block-level visual loss Loss P The average of the two parts is used to obtain the comprehensive loss function Loss of the aligned count text. Count-ICL .

[0025] For a given picture containing several similar specific objects, a set of rough visual embedding features with the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated through a fine-tuned visual encoder based on CLIP ViT and a text encoder Text Encoder. Specifically, for an H×W×C input image, a set of visual embedding vectors are generated through a CLIP ViT-based visual encoder and a dimensionality flattening operation; the type of the target counting object contained in the image is combined with different quantity texts, and a set of text embeddings containing type semantic information and quantity semantic information are generated through a text encoder.

[0026] The visual embedding features and text embedding vectors are input into the text-image cross-interaction network TICM based on the cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thereby realizing the joint representation of multimodal information, specifically:

[0027] The visual embedding vector and text embedding vector are denoted as Ev and Et respectively, and after the preprocessing operation of the self-attention layer to capture the long-distance visual dependencies, they are used as the input variables of the cross-attention mechanism operation; the visual embedding vector Ev and the text embedding vector Et are used as the query vector set Q and the key vector set K respectively, and the attention scores of the visual features in the text feature vector space are calculated through the scaled dot product model; the text embedding Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output.

[0028] The calculation of the attention score of the visual feature in the text feature vector space is specifically as follows: based on the scaled dot product model, the Q matrix obtained by the visual embedding vector Ep and the K matrix obtained by the text embedding Et are transposed and the dot product is calculated to obtain the cosine similarity matrix between the vectors; then divided by the arithmetic square root of the dimension; the result is normalized by the softmax function as the final output attention score; the Q matrix is ​​the query vector set Q; the K matrix is ​​the key vector set K; the visual embedding vector Ep is the visual embedding vector Ev.

[0029] The output block-level visual embedding feature representation in the text embedding vector space is specifically: the text embedding vector Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the block-level visual embedding feature representation in the text embedding vector space is output; the output visual feature integrates the type semantic information and the quantity semantic information to realize the joint representation of multimodal information.

[0030] The image-image cross-interaction module IICM based on the dual-branch Cross-ViT structure fuses multi-scale visual embedding features and outputs the counting result of the target. Specifically, the visual features of different scales are pre-processed by the self-attention layer to capture the long-distance visual dependency relationship, and then used as the input of the multi-scale visual feature cross-attention layer; the visual features of different scales are placed at the two ends of the dual-branch structure, and the large-scale visual features E and E are used to embed the features. L The cls token accesses the small-scale visual features E S , and using small-scale visual features E S The cls token accesses large-scale visual features E L , realize multi-scale visual feature fusion;

[0031] Realize multi-scale visual feature fusion, specifically:

[0032] Respectively express the small-scale visual features E S , and large-scale visual features E L Definition:

[0033]

[0034] Among them, the variables with superscript cls represent the global class token of the visual feature, and the variables with superscript patch represent the block-level embedding vector of the visual feature; the global class token is used as the query vector Q for attention calculation, and the global class tokens of visual features of different scales are concatenated with the block embedding vector as the key vector K and value vector V for attention calculation, and the attention score and the final attention output are calculated;

[0035] Q = [xl cls ] (6)

[0036]

[0037] Based on the above cross-attention mechanism, access between visual features of different scales is achieved, the interaction of multi-scale visual embedding vectors is established, and multi-scale feature fusion is realized; the global class token will be spliced ​​with the original block-level embedding vector.

[0038] The object counting system based on multimodal and multi-scale cross-attention includes:

[0039] A segmentation module, wherein the segmentation module segments the collected image through a visual encoder to obtain a number of block-level visual features, and then obtains a block-level visual embedding vector; each of the block-level visual embedding vectors corresponds to a text embedding vector;

[0040] A construction module, wherein the construction module constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector;

[0041] A first acquisition module, wherein the first acquisition module acquires the contrast loss of the aligned block-level visual features and the quantity text embedding based on the constructed similarity matrix and the multi-classification cross entropy loss function, thereby obtaining an optimized fine-tuned visual encoder; the quantity text embedding is to embed the text into a vector related to the quantity in the vector;

[0042] A second acquisition module, wherein the second acquisition module generates a set of rough visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information by fine-tuning the visual encoder and the text encoder Text Encoder based on CLIP ViT for a given picture containing a number of similar specific objects;

[0043] An output module, wherein the output module inputs the visual embedding features and the text embedding vector into a text-image cross-interaction network TICM based on a cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thereby realizing a joint representation of multimodal information;

[0044] The fusion module is based on an image-image cross interaction module IICM of a dual-branch Cross-ViT structure, which fuses multi-scale visual embedding features and outputs a target counting result.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention constructs a similarity matrix and then establishes a contrastive learning loss based on multi-classification cross entropy to enhance the text pairing ability of block-level visual vectors under the visual-language model. At the same time, a text embedding containing type semantic information and quantity semantic information and a visual embedding tensor with the same dimension are constructed. After calculation by a scaled dot product model, an attention score is output, and finally the feature representation of the block-level visual embedding in the text embedding vector space is obtained. A multi-scale visual feature interaction module based on a dual-branch Cross-ViT structure is constructed to achieve access and fusion between visual features of different scales. The present invention realizes multi-scale feature fusion, enhances the model's feature extraction performance for target details and contextual information of different scales in complex scenes, and provides good visual encoding for the downstream density map regression task of target counting. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a flow chart of the target counting method based on multi-modal multi-scale cross-attention of the present invention;

[0049] Figure 2 It is a structural schematic diagram of the target counting system based on multi-modal multi-scale cross-attention of the present invention;

[0050] Figure 3 The overall structure diagram of the target counting model based on multi-modal multi-scale cross-attention of the present invention;

[0051] Figure 4 A network structure diagram of the proxy task of aligning cross-modal features of the present invention;

[0052] Figure 5 A network structure diagram of the text-image cross-interaction module of the present invention;

[0053] Figure 6 The network structure diagram of the image-image cross-interaction module of the present invention is DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0057] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0058] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0059] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0060] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0061] See also Figure 1 The present invention discloses a target counting method based on multi-modal multi-scale cross attention, comprising:

[0062] S101, segmenting the collected image through a visual encoder to obtain a number of block-level visual features, and then obtaining a block-level visual embedding vector; each of the block-level visual embedding vectors corresponds to a text embedding vector;

[0063] S102, constructing a similarity matrix based on the block-level visual embedding vector and the text embedding vector;

[0064] The block-level visual embedding vector and the text embedding vector are aligned, and the similarity matrix between the block-level visual embedding vector and the text embedding vector is constructed. The one-hot true value label is used as the ground-truth to supervise the normalized similarity vector.

[0065] Maximize the cosine similarity between the N pairs of true matching visual embeddings and text embeddings, and minimize the remaining N 2 - Cosine similarity of N pairs of mismatched visual embeddings and text embeddings;

[0066] Calculate the cosine similarity between the N block-level visual embedding vectors and the N count text embeddings, and use the softmax function to normalize the similarity information to obtain an N×N similarity matrix, which is:

[0067]

[0068] Among them, P i and T j Represent the block-level visual embedding and text embedding vectors respectively, τ represents the temperature coefficient; k represents the index of the text embedding vector

[0069] S103, based on the constructed similarity matrix and the multi-classification cross entropy loss function, obtaining the contrast loss of the aligned block-level visual features and the quantity text embedding, and then obtaining the optimized fine-tuned visual encoder; the quantity text embedding is to embed the text into a vector related to the quantity in the vector;

[0070] Based on the similarity matrix between the block-level visual embedding vector and the text embedding vector and the similarity vector normalized by one-hot true value label supervision, the loss of the similarity between the counted text embedding vector matched by the block-level visual embedding vector and the true value result and the loss of the similarity between the block-level visual embedding vector matched by the counted text embedding vector and the true value result are obtained respectively; when the above losses are minimized, the optimal fine-tuned visual encoder is obtained;

[0071] The loss of similarity between the count text embedding vector matched by the block-level visual embedding vector and the true value result is specifically:

[0072]

[0073] Among them, the block-level visual loss LossP measures the difference between the probability distribution predicted by the model and the true value distribution of one-hot encoding, indicating the loss of similarity between multiple count text embedding vectors matched by the block-level visual embedding vector and the true value result;

[0074] Transpose the cosine similarity matrix and calculate the count text loss Loss in the same way T , which measures the loss of counting text embedding vectors matched by multiple block-level visual embedding vectors;

[0075]

[0076] The count text loss Loss T and block-level visual loss Loss P The average of the two parts is used to obtain the comprehensive loss function Loss of the aligned count text. Count-ICL .

[0077] S104, for a given picture containing a number of similar specific objects, a set of rough visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated by fine-tuning the visual encoder and the text encoder Text Encoder based on CLIP ViT;

[0078] For an H×W×C input image, a set of visual embedding vectors is generated after a CLIP ViT-based visual encoder and dimension flattening operation. The type of the target counting object contained in the image is combined with different quantity texts and passed through a text encoder to generate a set of text embeddings containing type semantic information and quantity semantic information.

[0079] S105, inputting the visual embedding features and the text embedding vector into a text-image cross-interaction network TICM based on a cross-attention mechanism, obtaining a cross-modal mapping from visual modality features to text modality features, and realizing a joint representation of multi-modal information;

[0080] The visual embedding vector and text embedding vector are denoted as Ev and Et respectively, and after the preprocessing operation of the self-attention layer to capture the long-distance visual dependencies, they are used as the input variables of the cross-attention mechanism operation; the visual embedding vector Ev and the text embedding vector Et are used as the query vector set Q and the key vector set K respectively, and the attention scores of the visual features in the text feature vector space are calculated through the scaled dot product model; the text embedding Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output.

[0081] Calculate the attention score of the visual feature in the text feature vector space, specifically: based on the scaled dot product model, transpose and calculate the dot product between the Q matrix obtained by the visual embedding vector Ep and the K matrix obtained by the text embedding Et to obtain the cosine similarity matrix between the vectors; then divide by the arithmetic square root of the dimension; normalize the result through the softmax function as the final output attention score; the Q matrix is ​​the query vector set Q; the K matrix is ​​the key vector set K; the visual embedding vector Ep is the visual embedding vector Ev.

[0082] Output the feature representation of block-level visual embedding in the text embedding vector space, specifically:

[0083] The text embedding vector Et is taken as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output; the output visual feature integrates the type semantic information and quantity semantic information to realize the joint representation of multimodal information.

[0084] S106, an image-to-image cross interaction module IICM based on a dual-branch Cross-ViT structure, fuses multi-scale visual embedding features and outputs the target counting results.

[0085] After the preprocessing operation of the self-attention layer to capture the long-distance visual dependency, the visual features of different scales are used as the input of the multi-scale visual feature cross-attention layer. The visual features of different scales are placed at the two ends of the dual-branch structure, and the large-scale visual features E L The cls token accesses the small-scale visual features E S , and using small-scale visual features E S The cls token accesses large-scale visual features E L , realizing multi-scale visual feature fusion.

[0086] Realize multi-scale visual feature fusion, specifically:

[0087] Respectively express the small-scale visual features E S, and large-scale visual features E L Definition:

[0088]

[0089] The variables with superscript cls represent the global class token of the visual feature, and the variables with superscript patch represent the block-level embedding vector of the visual feature. The global class token is used as the query vector Q for attention calculation, and the global class tokens of visual features of different scales are concatenated with the block embedding vector as the key vector K and value vector V for attention calculation to calculate the attention score and the final attention output.

[0090] Q = [xl cls ] (6)

[0091]

[0092] Based on the above cross-attention mechanism, access between visual features of different scales is achieved, the interaction of multi-scale visual embedding vectors is established, and multi-scale feature fusion is realized; the global class token will be spliced ​​with the original block-level embedding vector.

[0093] See also Figure 2 The present invention discloses a target counting system based on multi-modal multi-scale cross-attention, comprising:

[0094] A segmentation module, wherein the segmentation module segments the collected image through a visual encoder to obtain a number of block-level visual features, and then obtains a block-level visual embedding vector; each of the block-level visual embedding vectors corresponds to a text embedding vector;

[0095] A construction module, wherein the construction module constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector;

[0096] A first acquisition module, wherein the first acquisition module acquires the contrast loss of the aligned block-level visual features and the quantity text embedding based on the constructed similarity matrix and the multi-classification cross entropy loss function, thereby obtaining an optimized fine-tuned visual encoder; the quantity text embedding is to embed the text into a vector related to the quantity in the vector;

[0097] A second acquisition module, wherein the second acquisition module generates a set of rough visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information by fine-tuning the visual encoder and the text encoder Text Encoder based on CLIP ViT for a given picture containing a number of similar specific objects;

[0098] An output module, wherein the output module inputs the visual embedding features and the text embedding vector into a text-image cross-interaction network TICM based on a cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thereby realizing a joint representation of multimodal information;

[0099] The fusion module is based on an image-image cross interaction module IICM of a dual-branch Cross-ViT structure, which fuses multi-scale visual embedding features and outputs a target counting result.

[0100] The present invention designs a proxy task for aligning cross-modal features, namely: 2 For visual-text pairing, the cosine similarity between the N pairs of true matching visual embeddings and text embeddings is maximized, and the cosine similarity between the remaining N2-N pairs of false matching visual embeddings and text embeddings is minimized. In order to fine-tune the visual encoder, the cosine similarity between the N block-level visual embedding vectors and the N count text embeddings is calculated in this task, and the similarity matrix is ​​constructed, thereby establishing a contrastive learning loss based on multi-classification cross entropy to enhance the text pairing ability of block-level visual vectors under the visual-language model (VLM).

[0101] Furthermore, in the present invention, a text embedding containing type semantic information and quantity semantic information and a visual embedding tensor with the same dimension are constructed, which are denoted as E t and E v After the preprocessing operation of the self-attention layer to capture long-distance visual dependencies, the two are used together as the input variables of the cross-attention mechanism operation. After the calculation of the scaled dot product model, the attention score is output, and finally the feature representation of the block-level visual embedding in the text embedding vector space is obtained. The output feature result is a joint representation of text modality information and visual modality information, which enables the network to more accurately construct the representation of visual features in the counting-related text vector space, enhance the model's ability to locate and perceive details in dense areas under target counting tasks, and improve the model's counting performance.

[0102] Furthermore, in the present invention, a multi-scale visual feature interaction module based on a dual-branch Cross-ViT structure is constructed to achieve access and fusion between visual features of different scales. Among them, the global CLS token of the visual feature is used as a query vector for self-attention calculation to establish interactions between features of different scales in a relatively efficient way. This module makes full use of the Transformer model's self-attention mechanism's ability to capture long-distance visual dependencies, combines the mechanism of mutual access between word encodings, and uses the attention mechanism to dynamically focus on those features that are most relevant to specific visual tasks, achieving multi-scale feature fusion, and enhancing the model's feature extraction performance for target details and contextual information of different scales in complex scenes, providing good visual encoding for downstream density map regression tasks of target counting.

[0103] Example:

[0104] See also Figure 3 The present invention discloses a target counting method based on multimodal and multi-scale cross-attention, specifically a target counting method based on the CLIP visual-language model, which uses counting text to guide the understanding of the correspondence between text and visual features, focuses the attention of feature extraction on image areas containing dense target objects, and optimizes the recognition and counting accuracy of any specified target object in complex scenes.

[0105] The following steps are involved:

[0106] Step 1: Design a proxy task to align cross-modal features and construct a loss function based on contrastive learning

[0107] First, for any two-dimensional input image, the Transformer-based encoder decouples it into N one-dimensional visual word encoding vectors, and each visual word encoding has a corresponding true value of the number that matches it correctly. In the visual-language model, each visual word encoding vector should have the highest similarity match with the count text feature vector corresponding to the true value of the number of regions. In order to achieve the pairing of visual modality features and text modality features, it is necessary to design corresponding pre-training tasks for cross-modal feature alignment.

[0108] See also Figure 4 ,The proxy task of aligning cross-modal features is designed as follows: each element of the block-level visual embedding feature corresponds to a block of the original input image, and each block corresponds to the corresponding target count truth value. For the density map truth value H(x) of the labeled dataset, the density map truth value with the same resolution as the block-level visual embedding matrix should be obtained through average pooling operations, and the corresponding count text feature vector is constructed for each block-level visual embedding vector.

[0109] Furthermore, according to the output density distribution truth map M, a corresponding count truth token sequence is generated for each block-level visual feature. The count truth token sequence is encoded by the text encoder of CLIP, and the count text embedding feature T with a sequence length of N is output. In this way, for each vector element x in the block-level visual embedding feature sequence i , all have corresponding text embedded in T i Corresponding to it.

[0110] Furthermore, the proxy task of aligning cross-modal features is defined as: 2 For visual-text pairs, maximize the cosine similarity between the N pairs of true matching visual embeddings and text embeddings, and minimize the remaining N 2 - Cosine similarity of N pairs of mismatched visual embeddings and text embeddings. In this task, the cosine similarity between N block-level visual embedding vectors and N count text embeddings is calculated, and the similarity matrix is ​​constructed by softmax normalization under scaling operation based on temperature coefficient τ (experimentally set to 0.07).

[0111]

[0112] Where Sim is the similarity matrix; P i T j is the inner product between the block-level visual embedding and the text embedding vector; τ is the temperature coefficient.

[0113] Use the one-hot true value label and the normalized similarity vector to perform multi-classification cross entropy calculation to obtain the block-level visual loss Loss P .

[0114]

[0115] Furthermore, the cosine similarity matrix is ​​transposed, and the count text loss Loss based on multi-classification cross entropy is calculated in the same way. T .

[0116]

[0117] Finally, the count text loss Loss T and block-level visual loss Loss P The average of the two parts is used to obtain the comprehensive loss function Loss of the aligned count text. Count-ICL , represents the optimization direction of the visual encoder fine-tuning, and is used as the supervisory signal for training for several (experimentally set to 30) iterations.

[0118] Step 2: Obtain visual features with joint representation capabilities through a cross-modal multi-scale network based on a cross-attention mechanism

[0119] See also Figure 3 ,The image first passes through the CLIP ViT-based feature extraction network and ,dimensionality flattening operation, generating a set of visual embedding tensors of size ,197×512, which contain the global embedding vector and block-level embedding vector of the image.

[0120] For further information, see Figure 3 In the downstream task of density map regression, the image will pass through the multi-modal cross-interaction module and the multi-scale fusion cross-interaction module respectively. Figure 5 First, before passing through the cross attention layer, it is necessary to first undergo an alignment calculation to embed the block-level visual feature Z through linear projection v ∈R p2×d Project to the embedding vector E corresponding to each text t i ∈R n Same channel dimensions.

[0121] The visual embedding tensor of size 197×512 and the text embedding of size 6×512 are obtained, denoted as E v and E t The two have the same vector dimension and, after the preprocessing operation of the self-attention layer to capture long-range visual dependencies, serve together as the query vector set Q and key vector set K for the cross-attention mechanism operation. Finally, based on the scaled dot product model, the scaling operation of dividing by the arithmetic square root of the dimension and the softmax normalization process, the attention score of the visual feature in the text feature vector space is calculated.

[0122]

[0123] The inner product operation between the attention score and the corresponding V matrix is ​​performed to output the feature representation of the block-level visual embedding in the text embedding vector space. Therefore, the block-level visual feature can not only obtain visual information based on its own embedding vector, but also obtain the "quantity" text information with the highest cosine similarity match through the features of the corresponding counting text modality, and learn a joint representation containing visual information and text semantic information.

[0124]

[0125] See also Figure 3 , the previously output visual embedding is scaled and then input into the text-image cross-interaction module to output visual embeddings of different scales. Figure 6, feature maps of different scales will be input into the image-image cross interaction module. The module is based on a dual-branch Cross-ViT structure and adopts a cross attention mechanism to use large-scale visual features E L The cls token is used to access the small-scale visual features E S , and using small-scale visual features E S cls token to access large-scale visual features E L , to enhance the model's ability to capture local features of images from different scales.

[0126]

[0127] Where: xl cls It is a global class token for large-scale visual features; is the block-level embedding vector of large-scale visual features; xs cls It is a global class token for small-scale visual features; is the block-level embedding vector of small-scale visual features.

[0128] Furthermore, in the attention calculation, the global class token of the large-scale visual feature is used as the query vector Q of the attention calculation; the global class token of the large-scale visual feature is concatenated with the block embedding vector of the small-scale feature as the key vector K and value vector V of the attention calculation.

[0129] Q = [xl cls ] (15)

[0130]

[0131] The scaled dot product model is used to calculate the attention score and the final attention output. The output global CLS token needs to be concatenated with the original block-level embedding vector to restore and integrate the original large-scale visual features at the channel dimension level.

[0132] See also Figure 3 The block-level visual embeddings of different scales are passed through a density map regressor containing multiple convolutional layers and upsampling layers to achieve the fusion of multi-scale visual features, and finally output a point-level density map with a resolution of 224×224 as the inference result. Considering the ratio difference of the input images, the input images of different sizes will be cropped by sliding windows according to their actual length and width ratio to obtain the predicted density maps of multiple sliding windows, and the splicing and fusion of multiple sliding window density maps are achieved through nonlinear weighted averaging.

[0133] Step 3: Density map regression loss function

[0134] Data enhancement was applied during the training phase, i.e., Gaussian noise, Gaussian blur, color jitter, etc. were used to change and expand the images in the training set. In addition, horizontal flipping and rotation were used to simulate the perspectives of images in different directions and positions in real scenes. The original images of the expanded training set and the true values ​​of the density map were used as input for supervised training of the model network.

[0135] Furthermore, the target counting task of this method adopts the mean square error (MSE) loss function, which is used to calculate the mean of the sum of squares of the differences between the predicted density map and the true density map in the density map regression task.

[0136]

[0137] Where: f is the predicted output density map; y is the true value density map; W is the image width; H is the image height; Loss D is the density map loss.

[0138] Furthermore, for a predicted density map with an output size of W×H and a corresponding true density map, the loss is calculated by taking out all its pixels, calculating the square of the error of each pair of pixels one by one, and then averaging the squared error values ​​of all pixels to obtain the final mean square error at the matrix level.

[0139] Step 4: Use in downstream object counting tasks

[0140] The present invention provides a target counting method based on multimodal multi-scale cross-attention, and provides a general pre-training model for any type of downstream target counting tasks, which can replace the backbone with this model on any downstream target counting task. In the experimental process, the present invention uses the CLIP-based Vision Transformer backbone network as the model network for visual feature extraction, and realizes the generation of prediction density maps based on a single-stage regression task.

[0141] Furthermore, the present invention can be applied to downstream target counting tasks of any target type, including but not limited to crowd counting tasks, bird counting tasks, vehicle counting tasks, etc. When performing reasoning tasks in a specific type of downstream target counting task, the present invention needs to use English nouns of the counting type as natural language prompts to guide the target counting model to accurately focus on the objects of interest in the input image. For example: in the crowd counting task, "people" needs to be used as a prompt; in the bird counting task, "birds" needs to be used as a prompt, and so on.

[0142] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The object counting method based on multi-modal multi-scale cross attention is characterized by: include: The collected image is segmented through a visual encoder to obtain several block-level visual features, and then a block-level visual embedding vector is obtained; Each of the block-level visual embedding vectors corresponds to a text embedding vector; Construct a similarity matrix based on the block-level visual embedding vector and text embedding vector; Based on the constructed similarity matrix and multi-classification cross entropy loss function, the contrast loss of aligning block-level visual features and quantitative text embedding is obtained, and then the optimal fine-tuned visual encoder is obtained; The quantity text embedding is to embed the text into a vector related to the quantity; For a given picture containing several specific objects of the same type, a set of coarse visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated by fine-tuning the visual encoder and text encoder Text Encoder based on CLIP ViT. The visual embedding features and text embedding vectors are input into the text-image cross-interaction network TICM based on the cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thus realizing the joint representation of multimodal information. The image-image cross interaction module IICM based on the dual-branch Cross-ViT structure fuses multi-scale visual embedding features and outputs the target counting results.

2. The target counting method based on multimodal multi-scale cross attention according to claim 1 is characterized in that: The method constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector, specifically: aligning the block-level visual embedding vector and the text embedding vector, constructing a similarity matrix between the block-level visual embedding vector and the text embedding vector, and using a one-hot true value label as the ground-truth to supervise the normalized similarity vector.

3. The target counting method based on multimodal multi-scale cross attention according to claim 2 is characterized in that: The similarity matrix between the block-level visual embedding vector and the text embedding vector is constructed, and the normalized similarity vector is supervised by the one-hot true value label as the ground-truth, specifically: the cosine similarity of the N pairs of true matching visual embeddings and text embeddings is maximized, and the remaining N 2 - Cosine similarity of N pairs of mismatched visual embeddings and text embeddings; Calculate the cosine similarity between the N block-level visual embedding vectors and the N count text embeddings, and use the softmax function to normalize the similarity information to obtain an N×N similarity matrix, which is: Among them, P i and T j They represent block-level visual embedding and text embedding vectors respectively, τ represents the temperature coefficient; k represents the index of the text embedding vector.

4. The target counting method based on multimodal multi-scale cross attention according to claim 3 is characterized in that: Based on the constructed similarity matrix and multi-classification cross entropy loss function, the comparative loss of aligned block-level visual features and quantity text embedding is obtained, specifically: based on the similarity matrix of the block-level visual embedding vector and the text embedding vector and the similarity vector normalized by one-hot true value label supervision, the loss of the similarity of the count text embedding vector matched by the block-level visual embedding vector and the true value result and the loss of the similarity of the block-level visual embedding vector matched by the count text embedding vector and the true value result are obtained respectively; when the above losses are minimized, the optimal fine-tuned visual encoder is obtained; The loss of similarity between the count text embedding vector matched by the block-level visual embedding vector and the true value result is specifically: Among them, block-level visual loss Loss P Measures the difference between the probability distribution predicted by the model and the true value distribution of one-hot encoding, representing the loss of similarity between multiple count text embedding vectors matched by block-level visual embedding vectors and the true value results; Transpose the cosine similarity matrix and calculate the count text loss Loss in the same way T , which measures the loss of counting text embedding vectors matched by multiple block-level visual embedding vectors; The count text loss Loss T and block-level visual loss Loss P The average of the two parts is used to obtain the comprehensive loss function Loss of the aligned count text. Count-ICL .

5. The target counting method based on multimodal multi-scale cross attention according to claim 4 is characterized in that: For a given picture containing several similar specific objects, a set of rough visual embedding features with the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information are generated through a fine-tuned visual encoder based on CLIP ViT and a text encoder Text Encoder. Specifically, for an H×W×C input image, a set of visual embedding vectors are generated through a CLIP ViT-based visual encoder and a dimensionality flattening operation; the type of the target counting object contained in the image is combined with different quantity texts, and a set of text embeddings containing type semantic information and quantity semantic information are generated through a text encoder.

6. The object counting method based on multimodal multi-scale cross attention according to claim 5 is characterized in that: The visual embedding features and text embedding vectors are input into the text-image cross-interaction network TICM based on the cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thereby realizing the joint representation of multimodal information, specifically: The visual embedding vector and text embedding vector are denoted as Ev and Et respectively, and after the preprocessing operation of the self-attention layer to capture the long-distance visual dependencies, they are used as the input variables of the cross-attention mechanism operation; the visual embedding vector Ev and the text embedding vector Et are used as the query vector set Q and the key vector set K respectively, and the attention scores of the visual features in the text feature vector space are calculated through the scaled dot product model; the text embedding Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the feature representation of the block-level visual embedding in the text embedding vector space is output.

7. The object counting method based on multimodal multi-scale cross attention according to claim 6 is characterized in that: The attention score of the visual feature in the text feature vector space is calculated by: based on the scaled dot product model, the dot product is calculated by transposing the Q matrix obtained by the visual embedding vector Ep and the K matrix obtained by the text embedding Et to obtain the cosine similarity matrix between the vectors; Then divide by the square root of the dimension; The result is normalized by the softmax function as the final output attention score; The Q matrix is ​​the query vector set Q; the K matrix is ​​the key vector set K; and the visual embedding vector Ep is the visual embedding vector Ev.

8. The object counting method based on multimodal multi-scale cross attention according to claim 7 is characterized in that: The output block-level visual embedding feature representation in the text embedding vector space is specifically: the text embedding vector Et is used as the value vector set V, and based on the inner product operation result between the attention score and the V matrix, the block-level visual embedding feature representation in the text embedding vector space is output; the output visual feature integrates the type semantic information and the quantity semantic information to realize the joint representation of multimodal information.

9. The object counting method based on multimodal multi-scale cross attention according to claim 8 is characterized in that: The image-image cross-interaction module IICM based on the dual-branch Cross-ViT structure fuses multi-scale visual embedding features and outputs the counting result of the target. Specifically, the visual features of different scales are pre-processed by the self-attention layer to capture the long-distance visual dependency relationship, and then used as the input of the multi-scale visual feature cross-attention layer; the visual features of different scales are placed at the two ends of the dual-branch structure, and the large-scale visual features E and E are used to embed the features. L The cls token accesses the small-scale visual features E S , and using small-scale visual features E S The cls token accesses large-scale visual features E L , realize multi-scale visual feature fusion; Realize multi-scale visual feature fusion, specifically: Respectively express the small-scale visual features E S , and large-scale visual features E L Definition: Among them, the variables with superscript cls represent the global class token of the visual feature, and the variables with superscript patch represent the block-level embedding vector of the visual feature; the global class token is used as the query vector Q for attention calculation, and the global class tokens of visual features of different scales are concatenated with the block embedding vector as the key vector K and value vector V for attention calculation, and the attention score and the final attention output are calculated; Q = [xl cls ] (6) Based on the above cross-attention mechanism, access between visual features of different scales is achieved, the interaction of multi-scale visual embedding vectors is established, and multi-scale feature fusion is realized; the global class token will be spliced ​​with the original block-level embedding vector.

10. Object counting system based on multimodal multi-scale cross-attention, characterized by: include: A segmentation module, wherein the segmentation module segments the collected image through a visual encoder to obtain a number of block-level visual features, and then obtains a block-level visual embedding vector; Each of the block-level visual embedding vectors corresponds to a text embedding vector; A construction module, wherein the construction module constructs a similarity matrix based on the block-level visual embedding vector and the text embedding vector; A first acquisition module, wherein the first acquisition module acquires the contrast loss of the aligned block-level visual features and the quantity text embedding based on the constructed similarity matrix and the multi-classification cross entropy loss function, thereby obtaining an optimized fine-tuned visual encoder; The quantity text embedding is to embed the text into a vector related to the quantity; A second acquisition module, wherein the second acquisition module generates a set of rough visual embedding features of the same dimension and a set of text embedding vectors containing type semantic information and quantity semantic information by fine-tuning the visual encoder and the text encoder Text Encoder based on CLIP ViT for a given picture containing a number of similar specific objects; An output module, wherein the output module inputs the visual embedding features and the text embedding vector into a text-image cross-interaction network TICM based on a cross-attention mechanism to obtain a cross-modal mapping from visual modality features to text modality features, thereby realizing a joint representation of multimodal information; The fusion module is based on an image-image cross interaction module IICM of a dual-branch Cross-ViT structure, which fuses multi-scale visual embedding features and outputs a target counting result.

Images