Arbitrary category counting model based on language expression and counting method thereof
Through arbitrary category counting model based on language expression, combined with language encoder and visual encoder, the multimodal integration and cross-attention mechanism are used to solve the problems of low generalization ability and computing resource consumption in the existing technology, and achieve high Performance zero sample counting and widely applicable intelligent universal counting model.
Patent Information
- Application Number
- CN202410948085.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-15
AI Technical Summary
The existing arbitrary category counting model has low generalization capabilities in complex application scenarios, high sample collection costs, and computational resources are consumed and does not have the ability to quickly identify locations, resulting in extremely low generalization of applications.
Using arbitrary category counting model based on language expression, combining language encoder, visual encoder and multimodal integrator, the template feature extraction and similarity learning are guided by natural language to achieve the predicted number of target instances in the original image, using dual Branch backbone network and cross attention mechanism perform feature recalibration and similarity mapping to reduce traversal overhead and improve computing efficiency.
It realizes high-performance zero sample counting, has plug-and-play and wide applicability, reduces computing resource consumption, improves application versatility and counting accuracy, and supports high-level human-computer interaction capabilities.
Smart Images

Figure CN118965058A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a counting model, in particular to an arbitrary category counting model based on language expression and a counting method thereof. Background Art
[0002] At present, the problem of counting specific categories has been widely studied. A series of category-specific counting models already exist in the prior art. Although these models can achieve good counting accuracy in their respective categories, the successful training of their models requires the collection of a large amount of labeled data for different categories. In order to reduce the cost of annotation and increase the applicability of counting models, a series of category-independent counting models are designed. This type of counting model can count instances of objects of any category in the input scene, but it relies on high-quality target category template examples for feature matching. In complex application scenarios, such as underwater biological monitoring systems, the templates of the targets to be counted are usually difficult to obtain, resulting in low generalization ability. Therefore, the existing category-independent counting models have the problems of high sample collection costs and extremely low generalization ability.
[0003] There are also arbitrary category counting models in the existing counting models. Arbitrary category counting models can be specifically divided into three sub-fields: template-dependent counting, template implicit learning counting, and general template learning counting. Template-dependent counting is a counting model based on few-shot learning, which aims to give the counting model higher generalization to reduce the cost of collecting samples of different categories. Although template-dependent counting allows the input of target scene graphs of any category, it requires the user to provide displayed template clues.
[0004] Template implicit learning counting is a completely zero-shot learning in arbitrary category counting models, achieving zero-shot counting of open-world object categories. Although template implicit learning counting has initially achieved automatic learning of template clues, they usually perform poorly when encountering plug-and-play or real-time monitoring scenarios, thus affecting the flexibility of general counting models.
[0005] General template learning count uses a text conditional variational autoencoder model to explicitly predict template blocks, but it uses a relatively time-consuming traversal algorithm to select the optimal block, which is not conducive to the flexible deployment of the model.
[0006] Therefore, existing arbitrary category counting models tend to learn and extract template clues by scanning / traversing the entire input image, lacking rich semantic priors. The whole process consumes computing resources and does not have the ability to respond quickly to location recognition, so it has extremely low application generalization. Summary of the invention
[0007] The purpose of the present invention is to provide an arbitrary category counting model based on language expression and a counting method thereof. The present invention has the characteristics of saving computing resources, quickly identifying positions, and high application generalization.
[0008] The technical solution of the present invention is: an arbitrary category counting model based on language expression, including a referential expression understanding module and a visual count regression module. The referential expression understanding module is used to perform feature labeling embedding, feature extraction, feature fusion and other processing on the input natural language or text description to predict the target template information in the original image; the visual count regression module is used to perform feature extraction and similarity learning based on the target template information and the original image, and finally output the predicted number of target instances in the original image scene.
[0009] In the aforementioned arbitrary category counting model based on language expression, the referential expression understanding module includes a language encoder, a visual encoder and a multimodal integrator; the language encoder is used to embed feature tags according to the input language or text information to obtain high-level language features; the visual encoder is used to obtain high-level visual features under the condition of high-level language features for a noisy black image; the multimodal integrator is used to fuse and globally model high-level language features and high-level visual features to obtain a target template bounding box.
[0010] In the aforementioned arbitrary category counting model based on language expression, the language encoder includes a tokenization unit and an embedding processing unit; the tokenization unit is used to convert each word in the language text into an expression token in the vocabulary; the embedding processing unit is used to extract high-level language features with rich semantic information from the expression tokens.
[0011] In the aforementioned arbitrary category counting model based on language expression, the visual encoder includes a shallow feature extraction unit and a global semantic modeling unit; the shallow feature extraction unit is used to extract shallow representations with local semantic information from the black image and compress them into channel features; the global semantic modeling unit is used to extract corresponding global semantics from the channel features, and combine the local semantic information for spatial prior to obtain high-level visual features.
[0012] In the aforementioned arbitrary category counting model based on language expression, the visual counting regression module includes a dual-branch backbone network, a cross-attention mechanism and a counting sub-network. The dual-branch backbone network includes a sample encoding branch network and an image encoding branch network, which are respectively used for template feature extraction of sample encoding and original scene feature description of image encoding; the cross-attention mechanism is used to realize the re-correction of template features and original scene features through weights according to the semantic information in the sample and the original image, so as to obtain the features and image features of specific classification targets; the counting sub-network is used to perform similarity mapping on the features and image features of specific classification targets to obtain their respective corresponding vectors, and then perform inner product on the vectors, count the data points whose inner product values are not less than the set threshold, and obtain the predicted count value.
[0013] In the aforementioned arbitrary category counting model based on language expression, the dual-branch backbone network also includes a conversion processor for obtaining a higher-level one-dimensional feature with global semantic information of the template feature.
[0014] The above counting method of the arbitrary category counting model based on language expression includes the following steps:
[0015] S1, input the original image and language / text description, perform feature tag embedding, feature extraction, feature fusion and other processing on the input language / text description, and predict the target template information in the original image;
[0016] S2. Perform feature extraction and similarity learning based on the target template information and the original image, and output the predicted number of target instances in the original image scene.
[0017] In the aforementioned counting method, the step S1 is specifically as follows:
[0018] a. Input original image I∈R 3×H×W and language / text description, where H and W represent the height and width of the original image resolution. The language / text description is processed by text tagging and embedding to obtain the language high-level feature F l =E l (T l (re l ))∈R Dl×Nl , where rel represents the actual language described by the target, Dl and Nl represent the length and number of language tags respectively;
[0019] b. Generate a two-dimensional noise black image x from the original image i ∈R 3×H1×Wl , where R represents the pixels of the image, Hl, Wl represent the height and width of the input image resolution, from the two-dimensional noisy black image x i Extract a shallow representation f with local semantic information froms ∈R 2048×Hl / 32×Wl / 32 , and compressed into a compact 256-channel feature f st , from the channel feature f st Extract the corresponding global semantics from the local semantic information and the global semantics, and perform spatial priori on the local semantic information and the global semantics to obtain the high-level visual feature F i =T i (R i (x i ))∈R Di×Ni , where D i =256 represents the length of the channel dimension, N i =H / 32×W / 32 represents the number of marks;
[0020] c. Language high-level features F l and visual high-level features F i Embed the tag [Loc, Fl, Fi] to obtain multimodal information F li , parse [Loc] through the regression head and predict the target template bounding box (x, y, h, w), where x, y represent the horizontal and vertical coordinates of the center of the bounding box in the original image, h, w represent the height and width of the bounding box, and locate and capture the template image block on the original image according to the target template bounding box.
[0021] In the aforementioned counting method, the step S2 is specifically as follows:
[0022] d. According to the bounding box of the template image block and the original image, the template block I is obtained by cropping and resizing the original image. e ∈R 3×128×128 and I I ∈R 3×H / 16×W / 16 , respectively, by sample code R e Get the template feature F e =R e (I e )∈R 256×8×8 , through the image encoding R I Get the original scene feature F I =R I (I I )∈R 256×H / 16×W / 16 ;
[0023] e. Set the template feature F e After processing by the conversion processor, the vector feature is obtained
[0024] f. Recalibrate the features according to weights W1 and W2 to obtain the features of specific classification targets and image features <F I >=F I ×W2;
[0025] g. The characteristics of specific classification targets And image features <F I >Get the corresponding vectors through similarity mapping and U FI , the vector and U FI Perform the inner product and count the data points whose inner product value is not less than the set threshold, that is, obtain the predicted number of target instances in the original image scene.
[0026] In the aforementioned counting method, the weight W1∈R in step S2 1×256 =L1(M1(α1F I , α2F I )).Sigmoid(), where α1 and α2 represent the balance factors of training, M1 represents matrix multiplication, L1 represents linear transformation, and Sigmoid represents the calculation function; weight Where L2 represents linear transformation and Sigmoid represents calculation function.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] The counting model of the present invention is based on a target template learning strategy guided by natural language, is language-centric, combines computer counting algorithms with human language-visual understanding, understands fine-grained natural language expressions with a large number of semantic priors, uses natural language descriptions to guide the precise positioning of templates, infers template clues, extracts, fuses and compares the features of template blocks and original images, predicts accurate target templates, and finally infers the number of instances belonging to unknown categories, achieving high-performance zero-sample counting. The high natural language acceptance promotes the overall performance of zero-sample counting, has high versatility, realizes more generalized and intelligent counting functions, can be used for counting of various categories, has no traversal overhead, and has higher computational efficiency.
[0029] The referential expression understanding module in the counting model uses the pre-trained LLM GPT and the visual-language large model BLIP as the existing large-scale dataset FSC-147, adds fine-grained reference expressions, and achieves fine-grained enhancement of the existing large-scale dataset FSC-147 for arbitrary category counting models. With the help of the prior knowledge contained in the LLM, the counting model of the present invention performs well in perceiving and regressing template information. The regressed template blocks and original images can be seamlessly integrated into the downstream counting network, with high regression accuracy, and high-performance zero-sample counting.
[0030] The counting model of the present invention has plug-and-play characteristics and wide applicability. It integrates plug-and-play referential expression understanding with visual counting tasks and can be easily deployed in any novel counting scenario without further fine-tuning using samples. It achieves seamless connection with modern large-scale language models and thus has advanced human-computer interaction capabilities.
[0031] Therefore, the present invention utilizes the contemporary mainstream pre-trained language-vision model, multimodal integration strategy and a dual-branch counting network based on the cross-attention mechanism to realize a high-performance and efficient intelligent universal counting model. The counting model has plug-and-play properties and wide applicability, and can be easily deployed to any unseen example counting scenario without further fine-tuning. More importantly, it uses a modern large language model to guide the needs of human-computer interaction, has advanced human-computer interaction capabilities, and has high counting accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is an overall flow chart of the counting method of the present invention. DETAILED DESCRIPTION
[0033] The present invention will be further described below in conjunction with the embodiments, but they are not intended to limit the present invention.
[0034] Embodiment 1:
[0035] An arbitrary category counting model based on language expression, including a referential expression understanding module and a visual counting regression module.
[0036] Refers to the expression understanding module, which is used to embed, extract, and fuse the input natural language or text description into features, and predict the target template information in the original image based on the natural language text. For example, a sheep is framed in a farm image as the target for counting.
[0037] The referential expression understanding module includes a language encoder, a visual encoder, and a multimodal integrator based on existing neural machine translation.
[0038] The language encoder is obtained by pre-training the BERT model using mainstream natural languages, and embedding feature tags according to the input language or text information to obtain high-level language features. The language encoder contains 12 layers, and the learning rate and batch size are selected as 0.00005 and 15 respectively based on experience.
[0039] The speech encoder comprises a tokenization unit (T l ) and embedded processing unit (E l ). Among them, the tokenization unit (T l), which is used to convert each word in the language text into an expression token in the vocabulary; the embedding processing unit (E l ) is used to extract high-level language features with rich semantic information from expression tokens.
[0040] The processing of the language encoder is: l =E l (T l (re l ))∈R Dl×Nl , where re l Indicates the natural language text describing the target. Dl and Nl represent the length and number of language text tags, respectively, which are generally set to 768 and 20. l Indicates high-level features of a language.
[0041] The visual encoder is obtained by pre-training the DERT model and is used to convert the noisy black image x i ∈R 3×H1×Wl Under the condition of language high-level features, visual high-level features are obtained. Where R represents the pixels of the image, Hl and Wl represent the height and width of the input image resolution.
[0042] The visual encoder consists of 6 layers.
[0043] The visual encoder includes a shallow feature extraction unit and a global semantic modeling unit. The shallow feature extraction unit is used to use a pre-trained residual network (R i ) from the black image x i Extract a shallow representation f with local semantic information from s ∈R 2048×Hl / 32×Wl / 32 , then f s Compressed into a compact 256-channel feature f through a 1×1 convolution layer st .
[0044] The global semantic modeling unit is used to introduce a 256-wide Transformer layer T with an 8-head attention mechanism. i , and from the channel feature f st The corresponding global semantics are extracted from the image and combined with the local semantic information for spatial prior to obtain high-level visual features.
[0045] The processing of the visual encoder is: i =T i (R i (x i ))∈R Di×Ni , where D i =256 represents the length of the channel dimension, N i =Hl / 32×Wl / 32 represents the number of marks, Fi Represents high-level visual features.
[0046] The multimodal integrator is used to fuse high-level language features and high-level visual features, perform global modeling, and obtain a target template bounding box. The multimodal integrator includes 6 Transformer layers and has a high global semantic representation capability.
[0047] The multimodal integrator is a transformation-based integration module that mixes text and visual embeddings in a hybrid mode. Its work is as follows: In the learnable tag [Loc]∈R 256×1 Perform a concatenation operation on the language and convert the language high-level features F l and visual high-level features F i Embedding [Loc,F l ,F i ], obtain multimodal information F li ∈R 256×(1+Nl+Ni) Then, F li It is sent to a series of 6 Transformer layers to mix and represent multimodal information, thereby enriching the information content of the head tag [Loc]. At the end of the multimodal integrator, a regression head consisting of "three linear projections and a Sigmoid function" is set to pass the black image x i and multimodal information F li , parse [Loc] and predict the target template bounding box (x, y, h, w), where x, y represent the horizontal and vertical coordinates of the center of the bounding box in the original image, h, w represent the height and width of the bounding box, and locate and intercept the template image block on the original image according to the target template bounding box.
[0048] The visual count regression module is used to perform feature extraction and similarity learning based on the target template information and the original image obtained by the referential expression understanding module, and finally output the predicted number of target instances in the original image scene.
[0049] The visual count regression module includes a dual-branch backbone network, a cross-attention mechanism and a counting sub-network.
[0050] Since the receptive fields / semantic information of the template and the original image are different, the dual-branch backbone network includes a sample encoding branch network and an image encoding branch network. These two networks are two pre-trained residual networks, which are used for template feature extraction of sample encoding and original scene feature description of image encoding, respectively.
[0051] The samples and the whole image are combined for counting through different branch networks. The augmentation operation is performed by random erasing and random resizing to prevent possible overfitting problems. The learning rate is set to 1e-5 and the batch size is set to 15. The predicted template block is resized (128×128) and no augmentation technology is used. An end-to-end training strategy is adopted, using the AdamW optimizer with a weight decay of 0.0005.
[0052] The working process of the dual-branch backbone network is: given the bounding box (x, y, h, w) of the template image patch and the original image I∈R 3×H×W , where H and W represent the height and width of the original image resolution. The template block I is obtained by cropping and resizing the original image I. e ∈R 3×128×128 and I I ∈R 3×H / 16×W / 16 , respectively input sample encoding branch network R e and image encoding branch network R I In the example, we obtain the template features F e =R e (I e )∈R 2048×8×8 and the original scene feature F I =R I (I I )∈R 2048×H / 16×W / 16 , due to the inconsistency of the number of channels, a 1×1 convolutional layer is used to reduce and unify the channel dimension, and F e ∈R 256×8×8 , F I ∈R 256 ×H / 16×W / 16 .
[0053] The dual-branch backbone network also includes a fine-grained conversion processor, which is used to obtain higher-level one-dimensional features with global semantic information of the template block. It consists of a sequence of "PatchSpliting→LinearProjection→1×1Conv" operations. The template feature Fe is passed through the conversion processor to calculate the vector feature that can represent the template feature
[0054] The cross attention mechanism is used to realize the re-correction of its own features according to the semantic information in the sample and the original image, so as to achieve the purpose of feature complementation and fine-grained enhancement. And the low-level features F of the original image I Afterwards, due to the potential heterogeneity of the semantic space and the synergy of samples and dense instances in the feature space, a cross-attention mechanism, namely a cross-attention scheme CA(·), is used to recalibrate the vector features that are conditional on each other. and the original scene feature F of the original image I .
[0055] The specific work of the cross attention mechanism is: in order to enrich the sample features Input original scene feature F I ∈R 256 ×H / 16×W / 16 , using "matrix multiplication M1, linear transformation L1 and Sigmoid function" to generate weight W1∈R 1×256 =L1(M1(α1F I , α2F I )).Sigmoid(), where α1 and α2 represent the balance factors for training. At the same time, the input vector feature The weight W2∈R is generated through the process of "linear transformation L2 and Sigmoid function" 1×256 = Finally, the vector features of the template and the low-level features of the original image are re-corrected according to the two sets of weights to obtain the features of the specific classification target. and image features <F I >=F I ×W2, to achieve the purpose of feature complementarity and fine-grained enhancement.
[0056] The counting subnetwork is used to transform the features of specific classification targets And image features <F I >Perform similarity mapping to obtain the corresponding vectors and U FI , then the vector
[0057] and U FI Perform inner product, count the data points whose inner product value is not less than the set threshold, and obtain the predicted count value C of the target instance in the original image scene p .
[0058] In the model training phase, given the predicted position E p (x p ,y p ,h p , w p ) and the labeled true value E g (x g ,y g ,h g , w g ), the mean absolute error L can be calculated as Where n is the batch size, x p / x g and p / y gRespectively represent the horizontal and vertical coordinates of the center of the bounding box in the image, that is, the position. p / h g and w p / w g Represents the height and width of the bounding box respectively.
[0059] Count loss L c It is used to supervise the counting subnetwork to generate accurate instance counts, using L1 distance, and the counting loss is defined as Where N is the batch size, C g Represents the known label count value in the original image, C p Represents the predicted count value obtained by regression calculation. L, L1, L c is a mixed supervision signal. Through this mixed supervision signal, after each training, L, L1, L c The smaller it is, the better the target optimization is.
[0060] The present invention combines computer counting algorithms with human language-visual understanding. In actual operation, for example, when receiving a natural language description of "there is a basket of books on the table", people can easily capture key clues, such as the understanding of the category relationship of "the books in the basket" and the perception of the expected position of "the book in the center of the table". Even if they are not directly told the template information (such as a separate book image), humans can easily use the accumulated prior knowledge to make inferences.
[0061] Embodiment 2: A counting method based on an arbitrary category counting model of language expression, comprising the following steps:
[0062] Model training: Each time, a batch of "image-language" sample pairs is randomly generated from the data set. The language is input into the referential expression understanding module for text feature extraction and template prediction. The predicted target template information and the original image are input into the visual count regression module. After feature extraction, feature re-correction, and regression calculation, the predicted quantity value is finally output. The loss function is calculated using the predicted template box and the true box, the predicted quantity and the true value, and the end-to-end training optimization of the network is achieved based on the loss function.
[0063] Model application: After the model is trained, a language-expression-based counting model of any category that can be used directly is obtained. The target image and its corresponding language description are input into the model. The referential expression understanding module is used to process the text signal first, and the predicted bounding box is output. The template image block is cut out from the original image according to the predicted bounding box, and then the template image block and the original image are input into the visual count regression module to realize the final counting function.
[0064] like Figure 1 As shown, the prediction process of the present invention comprises the following steps:
[0065] S1, input original image I∈R 3×H×W , where H and W represent the height and width of the original image resolution, i.e., a picture of a birthday cake with candles on a black background.
[0066] S2, text encoding: Input language description "There is a birthday cake with candles on a black background", after tokenization unit T l Text tagging and embedding processing unit E l The text embedding process is used to obtain the language high-level feature F l :F l =E l (T l (re l ))∈R Dl×Nl , where re l represents the actual language describing the goal, D l and N l Represents the length and number of language tags respectively, set to 768 and 20.
[0067] S3, the original image is generated into a two-dimensional noisy black image xi∈R with a scale of 640×640, which is mainly composed of a series of zero values 3×H1×Wl , where R represents the pixels of the image, Hl, Wl represent the height and width of the input image resolution, establishing a unified spatial range.
[0068] S4. Visual encoding: From a black image x with 2D noise i In the above example, we extract a shallow representation f with local semantic information. s ∈R 2048×Hl / 32×Wl / 32 , then f s Compressed into a compact 256-channel feature f through a 1×1 convolution layer st , and then introduce a 256-wide Transformer layer T with an 8-head attention mechanism i , from the channel feature f st Extract the corresponding global semantics from the local semantic information and the global semantics, and perform spatial priori on the local semantic information and the global semantics to obtain the high-level visual feature F i =T i (R i (x i ))∈R Di×Ni , where D i =256 represents the length of the channel dimension, N i =H / 32×W / 32 represents the number of marks.
[0069] S5, feature fusion, global modeling:
[0070] In the learnable mark [Loc]∈R 256×1 Perform a concatenation operation on the language and convert the language high-level features F l and visual high-level features F i Embedding [Loc,F l ,F i ], obtain multimodal information F li ∈R 256×(1+Nl+Ni) , then, F li It is sent to a series of 6 Transformer layers to mix and represent multimodal information, thereby enriching the information content of the head tag [Loc]; finally, the regression head composed of "three linear projections and a Sigmoid function" parses [Loc] and predicts the target template bounding box (x, y, h, w), where x and y represent the horizontal and vertical coordinates of the center of the bounding box in the original image, and h and w represent the height and width of the bounding box. The template image block is located and intercepted on the original image according to the target template bounding box.
[0071] S6. According to the bounding box of the template image block and the original image, the template block I is obtained by cropping and resizing the original image. e ∈R 3×128×128 and I I ∈R 3×H / 16×W / 16 , respectively input into the dual-branch backbone network through R e Sample encoding obtains template feature F e =R e (I e )∈R 256×8×8 , through R I Image encoding obtains the original scene feature F I =R I (I I )∈R 256×H / 16×W / 16 .
[0072] S7, the template feature F e The vector feature is obtained by processing the conversion processor consisting of a sequence of "Patch Spliting→Linear Projection→1×1Conv" operations
[0073] S8, cross attention fusion: according to the original scene features F I , generating weights W1∈R 1×256 =L1(M1(α1F I , α2F I )).Sigmoid(), where α1 and α2 represent the balance factors of training, M1 represents matrix multiplication, L1 represents linear transformation, and Sigmoid represents the calculation function; according to the vector features Generate weights Where L2 represents linear transformation; recalibration of features is performed according to two sets of weights to obtain the features of specific classification targets and image features <F I >=F I ×W2.
[0074] S9. The features of specific classification targets And image features <F I >Get the corresponding vectors through similarity mapping and U FI , the vector and U FI Perform inner product and count the data points whose inner product value is not less than the set threshold, that is, obtain the predicted count value C of the target instance in the original image scene p .
[0075] Comparative test:
[0076] The counting model of the present invention is tested on the validation set and test set of FSC-147 or FSC-147-Express together with the existing sample-based counting model (ERC), implicit sample learning (IEL) and generalized sample learning (GEL). The quantitative results are shown in Table 1.
[0077] Among them, FSC-147 refers to the existing large-scale dataset FSC-147 using pre-trained LLM GPT and visual-language large model BLIP
[0078] FSC-147-Express refers to a large-scale dataset FSC-147-Express that adds fine-grained reference expression to the existing large-scale dataset FSC-147.
[0079] Table 1 Quantitative results of different counting models on the validation set and test set of FSC-147 or FSC-147-Express
[0080]
[0081] The MAE and RMSE in Table 1 represent the mean absolute error and the root mean square error, respectively.
[0082] ExpressCount-E is one of the models in which the example perceptron is trained only through language expression and does not rely on any image signal. ExpressCount-EV is an improvement on ExpressCount-E. This variant combines annotated language-image pairs, but uses a frozen visual encoder to enrich spatial semantics. ExpressCount further studies language-oriented semantic priors through the coordinated fine-tuning of language and visual encoders. Different counting models are compared in the table. It can be found that the detection effect of ExpressCount in the last row, that is, the counting model of the present invention, on the validation set and test set is much better than that of the previous related counting models, and the counting accuracy is the highest.
Claims
1. An arbitrary category counting model based on language expression, characterized by: It includes a referential expression understanding module and a visual count regression module. The referential expression understanding module is used to perform feature labeling embedding, feature extraction, feature fusion and other processing on the input natural language or text description to predict the target template information in the original image; the visual count regression module is used to perform feature extraction and similarity learning based on the target template information and the original image, and finally output the predicted number of target instances in the original image scene.
2. The arbitrary category counting model based on language expression according to claim 1, characterized in that: The referential expression understanding module includes a language encoder, a visual encoder and a multimodal integrator; the language encoder is used to embed feature tags according to the input language or text information to obtain high-level language features; the visual encoder is used to obtain high-level visual features under the condition of high-level language features for a noisy black image; the multimodal integrator is used to fuse and globally model high-level language features and high-level visual features to obtain a target template bounding box.
3. The arbitrary category counting model based on language expression according to claim 2, characterized in that: The language encoder includes a tokenization unit and an embedding processing unit; the tokenization unit is used to convert each word in the language text into an expression token in a vocabulary; The embedding processing unit is used to extract high-level language features with rich semantic information from expression tags.
4. The arbitrary category counting model based on language expression according to claim 2, characterized in that: The visual encoder includes a shallow feature extraction unit and a global semantic modeling unit; the shallow feature extraction unit is used to extract shallow representations with local semantic information from the black image and compress them into channel features; the global semantic modeling unit is used to extract corresponding global semantics from the channel features, and combine the local semantic information for spatial prior to obtain high-level visual features.
5. The arbitrary category counting model based on language expression according to claim 1, characterized in that: The visual counting regression module includes a dual-branch backbone network, a cross-attention mechanism and a counting sub-network, wherein the dual-branch backbone network includes a sample encoding branch network and an image encoding branch network, which are respectively used for template feature extraction of sample encoding and original scene feature description of image encoding; The cross attention mechanism is used to achieve re-correction of template features and original scene features through weights according to semantic information in samples and original images, so as to obtain features and image features of specific classification targets; The counting subnetwork is used to perform similarity mapping on the features of a specific classification target and the image features to obtain their respective corresponding vectors, and then perform inner product on the vectors, count the data points whose inner product values are not less than a set threshold, and obtain the predicted count value.
6. The arbitrary category counting model based on language expression according to claim 5, characterized in that: The dual-branch backbone network also includes a conversion processor for obtaining higher-level one-dimensional features of the template features with global semantic information.
7. A counting method based on an arbitrary category counting model of language expression, using the arbitrary category counting model described in any one of claims 1 to 6 above, characterized in that: The steps include: S1, input the original image and language / text description, perform feature tag embedding, feature extraction, feature fusion and other processing on the input language / text description, and predict the target template information in the original image; S2. Perform feature extraction and similarity learning based on the target template information and the original image, and output the predicted number of target instances in the original image scene.
8. The counting method of any category counting model based on language expression according to claim 7, characterized in that: The step S1 is specifically as follows: a. Input original image I∈R 3×H×W and language / text description, where H and W represent the height and width of the original image resolution. The language / text description is processed by text tagging and embedding to obtain the language high-level feature F l =E l (T l (re l ))∈R Dl ×Nl , where rel represents the actual language described by the target, Dl and Nl represent the length and number of language tags respectively; b. Generate a two-dimensional noise black image x from the original image i ∈R 3×H1×Wl , where R represents the pixels of the image, Hl, Wl represent the height and width of the input image resolution, from the two-dimensional noisy black image x i Extract a shallow representation f with local semantic information from s ∈R 2048×Hl / 32×Wl / 32 , and compressed into a compact 256-channel feature f st , from the channel feature f st Extract the corresponding global semantics from the local semantic information and the global semantics, and perform spatial priori on the local semantic information and the global semantics to obtain the high-level visual feature F i =T i (R i (x i ))∈R Di ×Ni , where D i =256 represents the length of the channel dimension, N i =H / 32×W / 32 represents the number of marks; c. Language high-level features F l and visual high-level features F i Embed the tag [Loc, Fl, Fi] to obtain multimodal information F li , parse [Loc] through the regression head and predict the target template bounding box (x, y, h, w), where x, y represent the horizontal and vertical coordinates of the center of the bounding box in the original image, h, w represent the height and width of the bounding box, and locate and capture the template image block on the original image according to the target template bounding box.
9. The counting method of any category counting model based on language expression according to claim 7, characterized in that: The step S2 is specifically as follows: d. According to the bounding box of the template image block and the original image, the template block I is obtained by cropping and resizing the original image. e ∈R 3×128×128 and I I ∈R 3×H / 16×W / 16 , respectively, by sample code R e Get the template feature F e =R e (I e )∈R 256×8×8 , through the image encoding R I Get the original scene feature F I =R I (I I )∈R 256×H / 16×W / 16 ; e. Set the template feature F e After processing by the conversion processor, the vector feature is obtained f. Recalibrate the features according to weights W1 and W2 to obtain the features of specific classification targets and image features <F I >=F I ×W 2; g. The characteristics of specific classification targets And image features <F I >Get the corresponding vectors through similarity mapping and U FI , the vector and U FI Perform the inner product and count the data points whose inner product value is not less than the set threshold, that is, obtain the predicted number of target instances in the original image scene.
10. The counting method of any category counting model based on language expression according to claim 9, characterized in that: The weight W1∈R in step S2 1×256 =L1(M1(α1F I , α2F I )).Sigmoid(), where α1 and α2 represent the balance factors of training, M1 represents matrix multiplication, L1 represents linear transformation, and Sigmoid represents the calculation function; weight Where L2 represents linear transformation and Sigmoid represents calculation function.
Citation Information
Cited By
Target counting method and system based on image shape embedding
CN119624981A
Target counting method and system based on multi-modal multi-scale cross attention
CN119785057A
Target counting model training method based on collective annotation and electronic equipment
CN120147297A
Tower collection method and system based on spatial relationship
CN120997714A
