Multi-element example unified crowd counting method and system based on visual language prompt

Abstract

The invention discloses a multi-element example unified crowd counting method and system based on visual language prompt, and belongs to the technical field of crowd counting. The method comprises the following steps: inputting a target image into an image encoder to capture local detail features and global context information to obtain image embedding, carrying out position encoding on a point input prompt and a box input prompt through a position encoder to obtain position embedding, and carrying out text encoding on a text input prompt through a text encoder to obtain text embedding, the position embedding and the text embedding are input into a prompt embedding module for merging to obtain prompt embedding, the image embedding and the prompt embedding are input into a task adaptive decoder, and the image embedding and the prompt embedding are fused through a bidirectional attention feature fusion module to obtain shared feature representation; and inputting the shared feature representation into a corresponding target prediction head module in a task decoder based on the input prompt to obtain a crowd quantity prediction result. The method improves the accuracy of crowd counting.

Description

Technical Field

[0001] This application belongs to the field of crowd counting technology, and in particular relates to a multi-paradigm unified crowd counting method and system based on visual language cues. Background Technology

[0002] Crowd counting, a core research topic in computer vision, has increasing application value in scenarios such as public safety monitoring, law enforcement, large-scale event management, and emergency response. With the acceleration of urbanization and the normalization of crowd gatherings in public places, using intelligent technology to accurately estimate and analyze crowd density has become a key means to prevent congestion and stampedes, optimize resource allocation, and improve urban management efficiency.

[0003] Existing crowd counting techniques can be broadly categorized into different paradigms based on their implementation methods. Early research often employed direct regression methods, which establish a direct mapping between global features of an image and the total number of people. However, these methods have significant limitations in practical applications due to their inability to provide spatial distribution information and susceptibility to interference from complex backgrounds.

[0004] Existing crowd counting methods require the design and deployment of multiple independent and heterogeneous model frameworks to cope with different crowd counting tasks (such as bounding box detection, point localization, and density map generation). This results in technical problems such as low crowd counting accuracy, system redundancy, excessive consumption of computing resources, cumbersome post-processing steps, and difficulty in flexibly switching between tasks. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a unified crowd counting method and system based on visual language cues and multiple paradigms, which improves the accuracy of crowd counting.

[0006] Firstly, this application provides a method for unified crowd counting based on visual language cues across multiple paradigms, the method comprising: The target image is acquired and input into an image encoder trained by a mask autoencoder to capture local detail features and global contextual information, thus obtaining the image embedding. The input prompts are input into the visual language prompt encoder module. The position encoder encodes the point input prompts and box input prompts to obtain the position embedding. The text encoder encodes the text input prompts to obtain the text embedding. The position embedding and text embedding are input into the prompt embedding module and merged to obtain the prompt embedding. The input prompts include point input prompts, box input prompts and text input prompts. Image embedding and cue embedding are input into the task-adaptive decoder. The image embedding and cue embedding are fused by the bidirectional attention feature fusion module to obtain a shared feature representation. Based on the input cue, the shared feature representation is input into the corresponding target prediction head module in the task decoder to obtain the crowd size prediction result. The task decoder includes a box detection prediction head module, a point localization prediction head module, and a density map generation prediction head module.

[0007] According to one embodiment of this application, the step of inputting the target image into an image encoder trained by a mask autoencoder to capture local detail features and global context information to obtain an image embedding includes: The target image is input into the position encoding module, the resolution of the target image is adjusted and the image is divided into multiple image blocks, and the multiple image blocks are subjected to linear projection to add position encoding to obtain a feature vector sequence. The feature vector sequence is input into the windowed attention mechanism module, and different levels of features of the feature vector sequence are extracted through multiple Transformer layers to obtain multi-level features; Multi-level features are input into the global attention module. The first global attention submodule extracts global context information to obtain context enhancement features. The context enhancement features are input into the second global attention submodule to extract spatial structure features to obtain spatial structure enhancement features. The spatial structure enhancement features are input into the third global attention submodule to extract association features to obtain association enhancement features. The association enhancement features are input into the fourth global attention submodule for channel dimension reduction to obtain image embedding. The image encoder includes a position encoding module, a windowed attention mechanism module, and a global attention module. The global attention module includes a first global attention submodule, a second global attention submodule, a third global attention submodule, and a fourth global attention submodule.

[0008] According to one embodiment of this application, the step of inputting the input prompt into the visual language prompt encoder module, and obtaining the position embedding by performing position encoding on the point input prompt and the box input prompt through the position encoder, includes: The point input prompt is input into the visual language prompt encoder module, the point input prompt is corrected for offset, the offset point coordinates are obtained, the offset point coordinates are converted into point embedding vectors based on the point label, and the position is encoded by the position encoder to obtain the point position embedding; The input prompt is fed into the visual language prompt encoder module, the input prompt is corrected and offset to obtain the offset box coordinates, the offset box coordinates are converted into box embedding vectors based on the box label, and the position is encoded by the position encoder to obtain the box position embedding; The point position embedding and the box position embedding are merged to obtain the position embedding.

[0009] According to one embodiment of this application, the step of encoding the text input prompt using a text encoder to obtain text embedding, and then merging the position embedding and text embedding into a prompt embedding module to obtain prompt embedding, includes: The text input prompt is fed into the text encoder, and the deep semantic features of the text input prompt are extracted by the trained CLIP model to obtain the text feature vector; The text feature vector is input into a linear layer for dimension alignment to obtain the dimension-reduced text embedding. The location embedding and text embedding are input into the prompt embedding module and merged through feature concatenation to obtain a prompt embedding that integrates location and text information.

[0010] According to one embodiment of this application, the step of inputting the image embedding and the cue embedding into the task-adaptive decoder, and fusing the image embedding and the cue embedding through a bidirectional attention feature fusion module to obtain a shared feature representation includes: Image embeddings and cue embeddings are input into a multi-head self-attention module, and the dependency between image embeddings and cue embeddings is captured through cross-modal bidirectional interaction to obtain interaction features; The interactive features are input into the multilayer perceptron module, and the expressive power of the interactive features is enhanced through nonlinear transformation to obtain the enhanced features; The enhanced features are input into the cross-attention mechanism module, and cross-modal fusion is performed using cue embedding as the query and image embedding as the key to obtain a shared feature representation; The bidirectional attention feature fusion module includes a multi-head self-attention module, a multilayer perceptron module, and a cross-attention mechanism module.

[0011] According to one embodiment of this application, the step of inputting the shared feature representation into the corresponding target prediction head module in the task decoder based on input prompts to obtain the crowd size prediction result includes: When the input prompt is a box detection task, the shared feature representation is input into the corresponding box detection prediction head module in the task decoder. Based on the box coordinate prediction head sub-module and the box confidence prediction head module, the box coordinate prediction and confidence filtering are performed to obtain the number of valid boxes. Based on the number of valid boxes, the population number prediction result is obtained. When the input prompt is a point localization task, the shared feature representation is input into the corresponding point localization prediction head module in the task decoder. Based on the point coordinate prediction head sub-module and the point confidence prediction head module, point coordinate prediction and confidence filtering are performed to obtain the number of valid points. Based on the number of valid points, the population size prediction result is obtained. When the input prompt is a density map generation task, the shared feature representation is input into the corresponding density map generation prediction head module in the task decoder to obtain the population size prediction result.

[0012] According to one embodiment of this application, when the input prompt is a density map generation task, inputting the shared feature representation into the corresponding density map generation prediction head module in the task decoder to obtain the crowd size prediction result includes: The shared feature representation is input into the dynamic weight generation module, and the dynamic weights are obtained by nonlinear transformation and dimension mapping through a multilayer perceptron. The shared feature representation and the image features of the target image are input into the upsampling module. The shared feature representation and the image features are upsampled and fused through transposed convolution to obtain the sampled feature map. The dynamic weights and sampled feature maps are weighted and fused using matrix multiplication to obtain the density map; The density map is integrated to obtain the population size prediction result.

[0013] Secondly, this application provides a unified crowd counting system based on visual language cues and multiple paradigms, the system comprising: The acquisition module is used to acquire the target image. It inputs the target image into the image encoder generated by the mask autoencoder to capture local detail features and global context information, and obtains the image embedding. The processing module is used to input input prompts into the visual language prompt encoder module, perform positional encoding on point input prompts and box input prompts through a position encoder to obtain positional embedding, and perform text encoding on text input prompts through a text encoder to obtain text embedding. The positional embedding and text embedding are then input into the prompt embedding module for merging to obtain prompt embedding. The input prompts include point input prompts, box input prompts and text input prompts. The counting module is used to input the image embedding and the cue embedding into the task adaptive decoder. The bidirectional attention feature fusion module fuses the image embedding and the cue embedding to obtain a shared feature representation. Based on the input cue, the shared feature representation is input into the corresponding target prediction head module in the task decoder to obtain the crowd size prediction result. The task decoder includes a box detection prediction head module, a point localization prediction head module, and a density map generation prediction head module.

[0014] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-paradigm unified crowd counting method based on visual language cues as described in the first aspect above.

[0015] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-paradigm unified population counting method based on visual language cues as described in the first aspect above.

[0016] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the multi-paradigm unified crowd counting method based on visual language cues as described in the first aspect.

[0017] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the multi-paradigm unified crowd counting method based on visual language cues as described in the first aspect above.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0019] The present invention provides a multi-paradigm unified crowd counting method based on visual language cues, which has the following advantages over the prior art: (1) By employing an image encoder, a visual language prompt encoder module, and a task adaptive decoder, this invention can collaboratively process and output various types of crowd counting results within a unified framework based on external prompts. It uses natural language text as task instructions and encodes the text instructions and optional spatial location prompts into a unified feature representation through the visual language prompt encoder. This guides the task adaptive decoder to dynamically and non-intrusively select and execute specific counting task examples. It can understand and execute task instructions issued in the form of natural language or spatial coordinates and dynamically generate counting results that meet the requirements of specific examples. While significantly improving computational efficiency and deployment flexibility, it maintains high-precision counting performance.

[0020] (2) This invention receives shared feature representations through a multi-task decoder and combines different task prompt embeddings to realize box detection, point localization and density map generation tasks. It can adaptively adjust the prediction method according to task requirements. It receives information fused from image features and prompt features through a task adaptive decoder and adopts a parallel processing decoding structure. Based on a single fused feature, it can activate the corresponding prediction head according to the task prompt in one forward propagation, efficiently generate the output of the specified example, and improve the accuracy and efficiency of crowd counting.

[0021] (3) By using three counting paradigms to optimize in a shared feature space, the present invention can learn from and supplement each other’s information. The spatial positioning information shared by the box detection and point localization tasks helps to improve the positioning accuracy of each other. The global distribution information generated by the density map can provide contextual constraints for local positioning tasks. By designing a combined loss function, the consistency and accuracy of the total population estimation can be maintained while optimizing the positioning accuracy of each paradigm. It can learn a more comprehensive and robust feature representation, which improves the counting accuracy and stability. Attached Figure Description

[0022] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts of the unified crowd counting method based on visual language prompts provided in the embodiments of this application; Figure 2 This is the second flowchart of the unified crowd counting method based on visual language prompts provided in the embodiments of this application; Figure 3 This is the third flowchart of the unified crowd counting method based on visual language prompts provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of a multi-paradigm unified crowd counting system based on visual language cues provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0025] The following description, in conjunction with the accompanying drawings, details the multi-paradigm unified crowd counting method, system, electronic device, and readable storage medium based on visual language cues provided in this application, through specific embodiments and application scenarios.

[0026] Among them, the multi-paradigm unified crowd counting method based on visual language cues can be applied to the terminal, specifically executed by the hardware or software in the terminal.

[0027] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0028] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0029] The method for unified crowd counting based on visual language cues provided in this application embodiment can be executed by an electronic device or a functional module or entity within an electronic device that can implement the method. Electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The following description uses an electronic device as an example to illustrate the method for unified crowd counting based on visual language cues provided in this application embodiment.

[0030] To improve counting accuracy and obtain richer scene information, existing research has led to three mainstream technical paradigms: bounding box detection, point localization, and density map generation.

[0031] Boundary detection methods originate from general object detection techniques, counting objects by identifying and locating the bounding boxes of each head or human body in an image. This method provides instance-level precise location and size information, offering data support for downstream tasks such as individual behavior analysis. Point localization methods achieve counting and localization by regressing the precise coordinates of the center point of each head. Compared to bounding box detection, its annotation cost is lower, exhibiting good balance in medium-density scenes. Density map generation methods are currently the mainstream technique for handling high-density and severely occluded scenes. It smooths discrete head point annotations using a Gaussian kernel function to generate a continuous density distribution map, and estimates the total number of people by integrating this map.

[0032] As application scenarios become more complex, the aforementioned single-paradigm technical solutions still suffer from the following shortcomings in practical deployment: First, real-world surveillance images often exhibit significant viewpoint distortion effects, resulting in multiple areas with vastly different crowd densities within the same frame. A single counting paradigm struggles to effectively address this uneven density simultaneously. Second, there is an inherent trade-off between the ability of different counting paradigms to handle high-density scenes and the richness of location information they provide. Generally, paradigms capable of handling higher-density scenes provide less location information, and vice versa. Third, forcibly merging multiple independent model frameworks such as bounding box detection, point localization, and density map generation often leads to bloated models, excessively high computational complexity, cumbersome post-processing steps, and low application efficiency. Fourth, while existing density-aware methods have attempted to enable models to perceive regional crowding through attention mechanisms or density mask generation, these methods typically only adapt different model parameters to the same counting paradigm, failing to fundamentally address the inherent limitations of a single paradigm and the inefficiency of multiple models coexisting.

[0033] Figure 1 This is one of the flowcharts illustrating the unified crowd counting method based on visual language cues provided in this application embodiment, such as... Figure 1 As shown, the multi-paradigm unified crowd counting method based on visual language cues includes steps 110, 120, and 130.

[0034] Step 110: Obtain the target image. Input the target image into the image encoder generated by training the mask autoencoder to capture local detail features and global context information to obtain the image embedding. In some embodiments, inputting the target image into an image encoder trained by a mask autoencoder to capture local detail features and global contextual information to obtain an image embedding includes: The target image is input into the position encoding module, the resolution of the target image is adjusted and the image is divided into multiple image blocks, and the multiple image blocks are subjected to linear projection to add position encoding to obtain a feature vector sequence. The feature vector sequence is input into the windowed attention mechanism module, and different levels of features of the feature vector sequence are extracted through multiple Transformer layers to obtain multi-level features; Multi-level features are input into the global attention module. The first global attention submodule extracts global context information to obtain context enhancement features. The context enhancement features are input into the second global attention submodule to extract spatial structure features to obtain spatial structure enhancement features. The spatial structure enhancement features are input into the third global attention submodule to extract association features to obtain association enhancement features. The association enhancement features are input into the fourth global attention submodule for channel dimension reduction to obtain image embedding. The image encoder includes a position encoding module, a windowed attention mechanism module, and a global attention module. The global attention module includes a first global attention submodule, a second global attention submodule, a third global attention submodule, and a fourth global attention submodule.

[0035] Figure 2 This is the second flowchart illustrating the multi-paradigm unified crowd counting method based on visual language cues provided in this application embodiment, as follows: Figure 2 As shown, the image encoder is a visual encoder pre-trained based on a masked autoencoder, with minor optimizations for high-resolution input, providing rich and effective feature representations for subsequent segmentation tasks. By combining a 14×14 windowed attention mechanism with four evenly distributed global attention modules, the image encoder can simultaneously capture local detail features and global contextual information, thus taking into account both the local features and the overall structure of the target image.

[0036] The target image is input into the positional encoding module. First, it undergoes size normalization to obtain image data of uniform specifications: by scaling the image and padding the shorter side, the target image is adjusted to a fixed resolution of 1024×1024. After size normalization, the image is divided into multiple 16×16 image blocks. Each image block is converted into a feature vector through linear projection, and spatial location information is injected by adding positional encoding, ultimately resulting in a directly processable feature vector sequence.

[0037] Furthermore, during feature extraction, deep processing is performed through a series of Transformer layers: the feature vectors are processed using a multi-head self-attention mechanism, and image features are extracted step by step at different levels. After feature processing, the channel dimension of the features is first reduced to D through 1×1 convolution, and then the spatial correlation and expressive power of the features are further optimized through 3×3 convolution, finally outputting a high-quality image embedding with dimensions D×W′×H′.

[0038] Step 120: Input the input prompts into the visual language prompt encoder module. The position encoder performs position encoding on the point input prompts and box input prompts to obtain position embedding. The text encoder performs text encoding on the text input prompts to obtain text embedding. The position embedding and text embedding are input into the prompt embedding module for merging to obtain prompt embedding. The input prompts include point input prompts, box input prompts and text input prompts. In some embodiments, the step of inputting the input prompt into the visual language prompt encoder module, and obtaining the position embedding by performing position encoding on the point input prompt and the box input prompt through the position encoder, includes: The point input prompt is input into the visual language prompt encoder module, the point input prompt is corrected for offset, the offset point coordinates are obtained, the offset point coordinates are converted into point embedding vectors based on the point label, and the position is encoded by the position encoder to obtain the point position embedding; The input prompt is fed into the visual language prompt encoder module, the input prompt is corrected and offset to obtain the offset box coordinates, the offset box coordinates are converted into box embedding vectors based on the box label, and the position is encoded by the position encoder to obtain the box position embedding; The point position embedding and the box position embedding are merged to obtain the position embedding.

[0039] It's easy to understand that the visual language cue encoder module can uniformly encode various input cues, such as points, boxes, and text, into feature representations for use by subsequent decoders. Its core objective is to generate location embeddings and text embeddings to support tasks such as point localization, box detection, and density map generation. The visual language cue encoder includes a location encoder and a text encoder. The location encoder uses a random spatial frequency location encoding method to generate location embeddings for point and box cues, supporting both normalized coordinates and raw coordinate inputs. The generated location embedding contains four learnable embedding vectors, corresponding to positive points, negative points, and the two corner points of the box, used to distinguish different types of point cues. The text encoder uses a pre-trained CLIP model to extract text features and performs dimensionality reduction using linear layers to match the target embedding dimension.

[0040] The position encoder encoding process is mainly divided into point cue encoding and box cue encoding. For point cue encoding, the input is the point coordinates (x, y) and a label (0 for negative points, 1 for positive points, and -1 for invalid points). The formula for processing the point coordinates is as follows: Where c is the correction offset. Let x be the x-coordinate of the initial point. The ordinate of the initial point. The x-coordinate of the point after correction. This represents the ordinate of the point after correction.

[0041] The pixel center is aligned using an offset, and then a corresponding embedding vector is added to each point according to the label. A positional embedding is then generated using a position encoder, with the shape of [shape missing]. Where B is the batch size, N is the number of points, and D is the embedding dimension.

[0042] For the input box prompt encoding, the coordinates of the input box This represents the two corner points of the box, whose coordinates are offset and then converted to corner point format. The calculation formula is as follows: in, Let x be the initial x-coordinate of the first corner point. Let x be the initial x-coordinate of the second corner point. Let be the initial y-coordinate of the first corner point. Let be the initial ordinate of the second corner point, and c be the correction offset. The x-coordinate of the first corner point after correction. The x-coordinate of the second corner point after correction. The corrected ordinate of the first corner point. This is the corrected ordinate of the second corner point.

[0043] A positional embedding is generated using a position encoder. Corresponding embedding vectors are added to the two corner points to generate a sparse embedding with the shape of [shape missing]. .

[0044] In this embodiment, by introducing a visual language prompt encoder module as a medium for task selection, the counting task to be performed can be dynamically and flexibly specified through input prompts without any modification to the structure or parameters. This enhances the flexibility and interactivity of the counting process, allowing for real-time switching of the counting mode based on the crowd density of the scene or specific application requirements, meeting diverse needs and enhancing scene adaptability and practical value.

[0045] In some embodiments, the step of encoding the text input prompt using a text encoder to obtain text embedding, and then merging the position embedding and text embedding into a prompt embedding module to obtain prompt embedding, includes: The text input prompt is fed into the text encoder, and the deep semantic features of the text input prompt are extracted by the trained CLIP model to obtain the text feature vector; The text feature vector is input into a linear layer for dimension alignment to obtain the dimension-reduced text embedding. The location embedding and text embedding are input into the prompt embedding module and merged through feature concatenation to obtain a prompt embedding that integrates location and text information.

[0046] For text prompt encoding, the input is a text prompt (such as Head Point Location). A pre-trained CLIP model is used to extract text features, which are then reduced to the target embedding dimension via a linear layer to generate a sparse embedding with the shape of... .

[0047] The output of the visual language cue encoder is a sparse embedding, and its shape is... Where N_t is the sum of all points, boxes, and text hints, calculated by embedding points, boxes, and text along the hint dimension. The calculation formula is shown below: in, For shared feature representation, For point location embedding, Embed for the box position, Embedded text.

[0048] Text embedding is used to achieve adaptive task selection. It can generate feature representations for performing corresponding counting tasks based on different task prompt texts, combined with location prompts and task embedding. Specific counting tasks include point localization, bounding box detection, and density map generation.

[0049] Optionally, optional configuration of input prompts is supported, and the embedding and splicing process can be automatically adjusted.

[0050] In this embodiment, by using text prompts to dynamically specify task types, it is possible to uniformly process various prompts such as points, boxes, and text without configuring separate coding modules for different tasks, and generate a unified embedded representation to support various crowd counting tasks, thereby enhancing the adaptability in different application scenarios.

[0051] Step 130: Input the image embedding and the cue embedding into the task adaptive decoder. The image embedding and the cue embedding are fused through the bidirectional attention feature fusion module to obtain a shared feature representation. Based on the input cue, the shared feature representation is input into the corresponding target prediction head module in the task decoder to obtain the crowd size prediction result. The task decoder includes a box detection prediction head module, a point localization prediction head module, and a density map generation prediction head module.

[0052] In some embodiments, the step of inputting the image embedding and the cue embedding into the task-adaptive decoder, and fusing the image embedding and the cue embedding through a bidirectional attention feature fusion module to obtain a shared feature representation includes: Image embeddings and cue embeddings are input into a multi-head self-attention module, and the dependency between image embeddings and cue embeddings is captured through cross-modal bidirectional interaction to obtain interaction features; The interactive features are input into the multilayer perceptron module, and the expressive power of the interactive features is enhanced through nonlinear transformation to obtain the enhanced features; The enhanced features are input into the cross-attention mechanism module, and cross-modal fusion is performed using cue embedding as the query and image embedding as the key to obtain a shared feature representation; The bidirectional attention feature fusion module includes a multi-head self-attention module, a multilayer perceptron module, and a cross-attention mechanism module.

[0053] Figure 3 This is the third flowchart of the multi-paradigm unified crowd counting method based on visual language cues provided in the embodiments of this application, as shown below. Figure 3 As shown, the task-adaptive decoding includes a bidirectional attention feature fusion module and a multi-task decoder. By deeply integrating visual features and cue information, it dynamically adapts to the needs of three tasks: detection counting, point localization counting, and density map counting.

[0054] The bidirectional attention feature fusion module facilitates deep interaction between cue embeddings and image embeddings, integrating feature information from different sources to provide high-quality feature representations for point and bounding box detection and density map generation tasks. The module consists of multiple bidirectional attention layers, each including self-attention, a multilayer perceptron, and cross-attention mechanisms for cue embedding to image embedding and image embedding to cue embedding. Self-attention is a core component of the Transformer, capable of capturing global dependencies between input features.

[0055] For example, the cue embedding and image embedding are first input into a self-attention module, where the global dependencies of their internal elements are captured to obtain cue features and image features with enhanced internal structures. Subsequently, these two sets of enhanced features are fed into a bidirectional cross-attention mechanism: first, cross-attention is performed using the cue feature as the query and the image feature as the key and value, enabling the cue information to extract relevant visual context from the image, resulting in a visually enhanced cue embedding; then, reverse cross-attention is performed using the image feature as the query and the enhanced cue embedding from the previous step as the key and value, enabling the image feature to dynamically focus on key regions based on the task cue, resulting in a task-oriented enhanced image embedding. Finally, these two sets of bidirectionally interacting features are input into a multilayer perceptron, where their feature representation capabilities are further refined through nonlinear transformations, ultimately outputting a fused feature of the optimized cue embedding and image embedding from this layer.

[0056] The multi-head self-attention module is applied to both the image encoding and cue embedding modules, enhancing the internal relational representation of the embeddings. The cue embeddings include positional embeddings and text embeddings. The query, key, and value vectors are calculated using the multi-head self-attention mechanism, and a weighted output is finally generated. The calculation formula is shown below: in, These are linear transformation matrices for the query, key, and value, respectively. For position encoding, This is a scaling factor to mitigate the numerical instability caused by high-dimensional features. This is for querying the matrix.

[0057] A multilayer perceptron (MLP) is a fully connected neural network architecture that enhances feature representation through nonlinear transformations. An MLP consists of multiple linear layers and uses ReLU as the activation function to enhance nonlinear representation. The calculation formula is shown below: in, For input, and These are the weight matrix and the bias vector, respectively. For output of the hidden layer, To enhance features.

[0058] The cross-attention mechanism module enables efficient interaction between query embeddings and image embeddings. Compared to traditional unidirectional cross-attention, the bidirectional structure enhances the query embedding's ability to aggregate image features through bidirectional interaction and, conversely, optimizes the image embedding to adapt to task requirements. Its computation process consists of two steps: the first step optimizes the query embedding to capture image features more comprehensively; the second step optimizes the image embedding to improve the robustness of the query embedding. The calculation formula is shown below: in, This indicates the initial query embedding. To query the location encoding of the embedding, K represents the image embedding. and These are the linear transformation matrices for the query, key, and value, respectively. Let K be the dimension of the key vector, and K represent the initial image embedding. Encoding the location of the image embedding. This is the query embedding optimized in the first step.

[0059] Finally, after MLP and normalization operations, the query embedding is further optimized, and feature fusion is completed by embedding the final cue into the cross-attention layer of the image embedding.

[0060] It is worth noting that although the Transformer possesses powerful global modeling capabilities, its computational complexity increases dramatically with increasing input resolution. To reduce computational complexity, a downsampling attention mechanism is introduced into the attention computation. This mechanism takes the original high-resolution image embedding as input, reduces the dimension D of the key-value embedding to obtain a compact downsampling image embedding as output, and uses it for subsequent attention computation. The complexity of the traditional self-attention mechanism is... Where N=HW. The embedding dimension is reduced from D to HW by the downsampling rate r. This reduces the computational complexity to This effectively reduces computational overhead, making it suitable for high-resolution crowd counting tasks.

[0061] In some embodiments, the step of inputting the shared feature representation into the corresponding target prediction head module in the task decoder based on input prompts to obtain the crowd size prediction result includes: When the input prompt is a box detection task, the shared feature representation is input into the corresponding box detection prediction head module in the task decoder. Based on the box coordinate prediction head sub-module and the box confidence prediction head module, the box coordinate prediction and confidence filtering are performed to obtain the number of valid boxes. Based on the number of valid boxes, the population number prediction result is obtained. When the input prompt is a point localization task, the shared feature representation is input into the corresponding point localization prediction head module in the task decoder. Based on the point coordinate prediction head sub-module and the point confidence prediction head module, point coordinate prediction and confidence filtering are performed to obtain the number of valid points. Based on the number of valid points, the population size prediction result is obtained. When the input prompt is a density map generation task, the shared feature representation is input into the corresponding density map generation prediction head module in the task decoder to obtain the population size prediction result.

[0062] The task decoder is the core component of the task-based autonomous decoding, supporting three tasks simultaneously—box detection, point localization, and density map generation—through a single framework. Task decoding extracts shared information from the output of the bidirectional attention feature fusion module and combines it with task prompts to generate diverse prediction results. The multi-task parallel processing capability enhances the model's versatility, providing an efficient implementation scheme for multi-paradigm population counting research.

[0063] Specifically, the multi-task decoder includes output labels, cue embeddings, and a task decoder. The output labels are learnable embedding vectors shared across all tasks, with dimensions consistent with the image embedding dimensions. The cue embeddings provide task-specific cue information, enabling adjustments to the prediction method based on different task requirements. The output labels and cue embeddings are combined to form the query embedding, which, after processing through a bidirectional cross-attention mechanism, generates a unified feature representation. Where B is the batch size. To output the marker size, To provide a hint about the embedding size, the generated shared feature representation R is used as input to the subsequent task decoder, enabling information sharing between tasks and adapting to different tasks based on changes in the hint embedding. The task decoder includes a prediction head module specifically designed for box detection, point localization, and density map generation, achieving accurate predictions through linear layers and an MLP.

[0064] When the counting requirement is a bounding box detection task, the input prompt is Head Box Detection, and the multi-task decoder will perform bounding box detection for prediction. The feature input for the bounding box detection task uses a shared feature representation of the output features in R, denoted as . The bounding box coordinate prediction head submodule includes a bounding box coordinate prediction head and a bounding box confidence prediction head, where the coordinate prediction output is... The calculation formula is as follows: in, This indicates the offset of the box center relative to the input position center. This indicates the predicted height and width of the head frame. The number of prediction boxes, This is the result of the bounding box coordinate prediction.

[0065] Confidence prediction output ,in, Indicates prediction Confidence level for the number of people. During the prediction process, As input, after processing by the MLP, the output is the bounding box coordinates and confidence score. When the predicted bounding box... confidence level If the value exceeds the threshold, the predicted bounding box is considered a valid bounding box. The calculation formula is as follows: in, For valid frames, For confidence level, This is the predicted bounding box.

[0066] The task of counting people detected by bounding boxes is completed by counting the number of valid bounding boxes. Due to confidence gating, complex post-processing procedures are avoided.

[0067] To ensure effective model training, a single-image, multi-cue training strategy is designed. For a single input image, the image encoder performs only one forward computation, and the generated image feature embedding is reused by multiple different cue points and their corresponding task cues, thus significantly reducing the computational overhead during training. Simultaneously, to collaboratively optimize multi-task performance, independent loss functions are designed for different tasks.

[0068] For bounding box detection tasks, the total loss function includes a weighted average of coordinate loss, generalized intersection-union (GUU) loss, and confidence loss. The Hungarian matching algorithm ensures a one-to-one correspondence between predicted and ground truth bounding boxes. The matching cost function is calculated based on GUU, and the formula is shown below: in, and These represent the first and second boxes, respectively. This is the minimum bounding box of the two. This is the intersection-union ratio of the first and second boxes.

[0069] In this embodiment, by adopting a training strategy based on Hungarian matching and confidence threshold screening, sparse and more accurate prediction results can be generated directly, effectively reducing complex post-processing steps. This not only simplifies the entire counting task process but also lowers the deployment and optimization threshold of the algorithm application, making the counting system more robust and easier to use.

[0070] The total loss function for bounding box detection is a weighted average of coordinate loss, GIoU loss, and confidence loss. Specifically: in, These are the weighting coefficients for each type of loss. For coordinate loss function, For GIoU loss, the loss function is... For confidence loss function, Let be the total loss function, ( The model predicts the first The coordinates of the top left corner of each box, ( The model predicts the first The coordinates of the bottom right corner of the box. The coordinates of the top left corner of the actual label box The coordinates of the bottom right corner of the actual label box. The absolute value sign is used to calculate the distance between the predicted value and the true value. For generalized intersection-union ratio functions, The total number of matching box samples used in the loss calculation. The model predicts the first The set of coordinate parameters for each box. For the first A set of coordinate parameters for each real bounding box. The confidence score for the model's prediction represents the probability that the model considers the bounding box to be a human head. For real labels, The balancing weights are used to control the contribution weights of positive and negative samples to the total loss. This is a focusing parameter used to control the degree of attention given to "difficult-to-classify samples".

[0071] When the counting requirement is a point localization task, the input prompt is "Head Point Location," and the task decoder performs point localization to make predictions. The feature input for the point localization task also comes from a shared feature representation R, denoted as... The point prediction head module includes a point coordinate prediction head submodule and a point confidence prediction head submodule, where the coordinate prediction output is... , This represents the offset relative to the center of the input location. Confidence prediction output. ,in, Point The confidence level is based on the number of people. During the prediction process, [the following will be used]: As input, the MLP processes the data and outputs the point coordinates and confidence score. When the confidence score exceeds a threshold, the predicted point is considered a valid point. By counting the number of valid points, the number of valid people around the indicated location is obtained, thus completing the point location crowd counting task.

[0072] For point localization tasks, the total loss function includes coordinate loss and confidence loss. Coordinate loss and total loss function The calculation formula is as follows: in, and These are the weighting coefficients. For coordinate loss function, This is the total loss function.

[0073] In this embodiment, by using three counting paradigms to optimize in a shared feature space, information can be mutually referenced and supplemented. Spatial positioning information shared by the box detection and point localization tasks helps to improve positioning accuracy. Global distribution information generated by density maps can provide contextual constraints for local positioning tasks. By designing a combined loss function, the consistency and accuracy of the total population estimation can be maintained while optimizing the positioning accuracy of each paradigm. More comprehensive and robust feature representations can be learned, improving counting accuracy and stability.

[0074] In some embodiments, when the input prompt is a density map generation task, inputting the shared feature representation into the corresponding density map generation prediction head module in the task decoder to obtain the crowd size prediction result includes: The shared feature representation is input into the dynamic weight generation module, and the dynamic weights are obtained by nonlinear transformation and dimension mapping through a multilayer perceptron. The shared feature representation and the image features of the target image are input into the upsampling module. The shared feature representation and the image features are upsampled and fused through transposed convolution to obtain the sampled feature map. The dynamic weights and sampled feature maps are weighted and fused using matrix multiplication to obtain the density map; The density map is integrated to obtain the population size prediction result.

[0075] When the counting requirement is the density map generation task, the input prompt is Head Density MapPrediction, and the multi-task decoder performs the density map generation task to make predictions. The feature input of the density map generation task includes a shared feature representation. and image features , where C is the same as the number of channels in the image embedding. The dynamic weights are generated using a dynamic weight generation mechanism, while F is processed by an upsampling module. The prediction head module combines the dynamic weight generation mechanism and the upsampling module to ultimately generate a single-channel density map. The input for generating the dynamic convolutional kernel is... It outputs dynamic weights through multi-layer MLP processing. Where D represents the number of channels for dynamic weights. The image feature upsampling module consists of three transposed convolutional layers, with F as the input. The feature map is obtained after upsampling. Density map calculation uses matrix multiplication to dynamically weight the data. Compared with the flattened feature map The calculation is performed using the formula shown below: Where M is the density map, For dynamic weights, This is a sampled feature map.

[0076] The final generated density map In the prediction process, Using image features F as input, prediction is performed through dynamic weight calculation and transposed convolution upsampling. Finally, the effective number of people around the cue location point is obtained by integrating the density map, thus completing the density map generation and crowd counting task.

[0077] For density map generation, the total loss function is a weighted average of the mean squared error loss and the marker-based improved structural similarity loss. Structural similarity loss The total loss function for density map tasks The calculation formula is as follows: in, This represents the total loss for the density map task. For mean square error loss, This is an improved structural similarity loss based on markers, where K is the number of effective markers. For structural similarity functions, To predict the density map, This is the true density map. Mark the coordinates of the point. and These are the weighting coefficients.

[0078] In this embodiment, a shared feature representation is received through a multi-task decoder, and different task prompt embeddings are combined to realize box detection, point localization, and density map generation tasks. The prediction method can be adaptively adjusted according to task requirements. The information fused from image features and prompt features is received through a task adaptive decoder. The parallel processing decoding structure can activate the corresponding prediction head according to the task prompt in one forward propagation based on a single fused feature, and efficiently generate the output of the specified example, thereby improving the accuracy and efficiency of crowd counting.

[0079] The unified crowd counting method based on visual language cues provided in this application employs a shared image encoder, visual language cue encoder, and task-adaptive decoder. Within a unified framework, it collaboratively processes and outputs various types of crowd counting results based on external cues. Natural language text is used as task instructions, and the visual language cue encoder encodes the text instructions and optional spatial location cues into a unified feature representation. This guides the task-adaptive decoder to dynamically and non-intrusively select and execute specific counting task paradigms. It can understand and execute task instructions given in natural language or spatial coordinates, dynamically generating counting results that meet the requirements of specific paradigms. This significantly improves computational efficiency and deployment flexibility while maintaining high-precision counting performance.

[0080] The multi-paradigm unified crowd counting method based on visual language cues provided in this application can be implemented by a multi-paradigm unified crowd counting system based on visual language cues. This application example illustrates the multi-paradigm unified crowd counting system based on visual language cues executing the method, demonstrating the implementation of the system.

[0081] This application also provides a unified crowd counting system based on visual language cues and multiple paradigms, such as... Figure 4 As shown, the unified crowd counting system based on visual language cues and multiple paradigms includes: an acquisition module 410, a processing module 420, and a counting module 430.

[0082] The acquisition module 410 is used to acquire the target image and input the target image into the image encoder generated by the mask autoencoder to capture local detail features and global context information to obtain the image embedding. Processing module 420 is used to input input prompts into visual language prompt encoder module, perform position encoding on point input prompts and box input prompts through position encoder to obtain position embedding, perform text encoding on text input prompts through text encoder to obtain text embedding, and input the position embedding and text embedding into prompt embedding module for merging to obtain prompt embedding, wherein the input prompts include point input prompts, box input prompts and text input prompts; The counting module 430 is used to input the image embedding and the cue embedding into the task adaptive decoder. The image embedding and the cue embedding are fused by the bidirectional attention feature fusion module to obtain a shared feature representation. Based on the input cue, the shared feature representation is input into the corresponding target prediction head module in the task decoder to obtain the crowd number prediction result. The task decoder includes a box detection prediction head module, a point localization prediction head module, and a density map generation prediction head module.

[0083] The unified crowd counting system based on visual language cues provided in this application employs a shared image encoder, visual language cue encoder, and task-adaptive decoder. Within a unified framework, it collaboratively processes and outputs various types of crowd counting results based on external cues. Utilizing natural language text as task instructions, the visual language cue encoder encodes the text instructions and optional spatial location cues into a unified feature representation. This guides the task-adaptive decoder to dynamically and non-intrusively select and execute specific counting task paradigms. It can understand and execute task instructions given in natural language or spatial coordinates, dynamically generating counting results that meet the requirements of specific paradigms. This significantly improves computational efficiency and deployment flexibility while maintaining high-precision counting performance.

[0084] The unified crowd counting system based on visual language cues provided in this application can achieve... Figures 1 to 3 The various processes implemented in the embodiment of the unified crowd counting method based on visual language prompts and multiple paradigms will not be described in detail here to avoid repetition.

[0085] In some embodiments, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements the various processes of the above-described embodiment of the multi-paradigm unified crowd counting method based on visual language prompts and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0086] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0087] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of the unified crowd counting method based on visual language prompts and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0088] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0089] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for unified crowd counting based on visual language cues and multiple paradigms.

[0090] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0091] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the unified crowd counting method based on visual language prompts and multiple paradigms, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0092] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0093] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element. Furthermore, it should be noted that the scope of the methods and systems in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the multi-paradigm unified crowd counting method based on visual language prompts of the various embodiments of this application.

[0095] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0096] In the description of this application, "multiple" means two or more.

[0097] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0098] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0099] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A unified crowd counting method based on visual language cues and multiple paradigms, characterized in that, The method includes: The target image is acquired and input into an image encoder trained by a mask autoencoder to capture local detail features and global contextual information, thus obtaining the image embedding. The input prompts are input into the visual language prompt encoder module. The position encoder encodes the point input prompts and box input prompts to obtain the position embedding. The text encoder encodes the text input prompts to obtain the text embedding. The position embedding and text embedding are input into the prompt embedding module and merged to obtain the prompt embedding. The input prompts include point input prompts, box input prompts and text input prompts. Image embedding and cue embedding are input into the task-adaptive decoder. The image embedding and cue embedding are fused by the bidirectional attention feature fusion module to obtain a shared feature representation. Based on the input cue, the shared feature representation is input into the corresponding target prediction head module in the task decoder to obtain the crowd size prediction result. The task decoder includes a box detection prediction head module, a point localization prediction head module, and a density map generation prediction head module.

2. The method for unified crowd counting based on visual language cues according to claim 1, characterized in that, The step of inputting the target image into an image encoder trained by a mask autoencoder to capture local detail features and global contextual information to obtain an image embedding includes: The target image is input into the position encoding module, the resolution of the target image is adjusted and the image is divided into multiple image blocks, and the multiple image blocks are subjected to linear projection to add position encoding to obtain a feature vector sequence. The feature vector sequence is input into the windowed attention mechanism module, and different levels of features of the feature vector sequence are extracted through multiple Transformer layers to obtain multi-level features; Multi-level features are input into the global attention module. The first global attention submodule extracts global context information to obtain context enhancement features. The context enhancement features are input into the second global attention submodule to extract spatial structure features to obtain spatial structure enhancement features. The spatial structure enhancement features are input into the third global attention submodule to extract association features to obtain association enhancement features. The association enhancement features are input into the fourth global attention submodule for channel dimension reduction to obtain image embedding. The image encoder includes a position encoding module, a windowed attention mechanism module, and a global attention module. The global attention module includes a first global attention submodule, a second global attention submodule, a third global attention submodule, and a fourth global attention submodule.

3. The method for unified crowd counting based on visual language cues according to claim 1, characterized in that, The step of inputting input prompts into the visual language prompt encoder module, and obtaining position embedding by performing position encoding on point input prompts and box input prompts through a position encoder, includes: The point input prompt is input into the visual language prompt encoder module, the point input prompt is corrected for offset, the offset point coordinates are obtained, the offset point coordinates are converted into point embedding vectors based on the point label, and the position is encoded by the position encoder to obtain the point position embedding; The input prompt is fed into the visual language prompt encoder module, the input prompt is corrected and offset to obtain the offset box coordinates, the offset box coordinates are converted into box embedding vectors based on the box label, and the position is encoded by the position encoder to obtain the box position embedding; The point position embedding and the box position embedding are merged to obtain the position embedding.

4. The method for unified crowd counting based on visual language cues according to claim 1, characterized in that, The text input prompt is encoded using a text encoder to obtain text embedding. The position embedding and text embedding are then input into the prompt embedding module for merging to obtain the prompt embedding, including: The text input prompt is fed into the text encoder, and the deep semantic features of the text input prompt are extracted by the trained CLIP model to obtain the text feature vector; The text feature vector is input into a linear layer for dimension alignment to obtain the dimension-reduced text embedding. The location embedding and text embedding are input into the prompt embedding module and merged through feature concatenation to obtain a prompt embedding that integrates location and text information.

5. The method for unified crowd counting based on visual language cues according to claim 1, characterized in that, The process of inputting image embeddings and cue embeddings into a task-adaptive decoder, and fusing the image embeddings and cue embeddings through a bidirectional attention feature fusion module to obtain a shared feature representation includes: Image embeddings and cue embeddings are input into a multi-head self-attention module, and the dependency between image embeddings and cue embeddings is captured through cross-modal bidirectional interaction to obtain interaction features; The interactive features are input into the multilayer perceptron module, and the expressive power of the interactive features is enhanced through nonlinear transformation to obtain the enhanced features; The enhanced features are input into the cross-attention mechanism module, and cross-modal fusion is performed using cue embedding as the query and image embedding as the key to obtain a shared feature representation; The bidirectional attention feature fusion module includes a multi-head self-attention module, a multilayer perceptron module, and a cross-attention mechanism module.

6. The method for unified crowd counting based on visual language cues according to claim 1, characterized in that, The process of inputting the shared feature representation into the corresponding target prediction head module in the task decoder based on input prompts to obtain the crowd size prediction result includes: When the input prompt is a box detection task, the shared feature representation is input into the corresponding box detection prediction head module in the task decoder. Based on the box coordinate prediction head sub-module and the box confidence prediction head module, the box coordinate prediction and confidence filtering are performed to obtain the number of valid boxes. Based on the number of valid boxes, the population number prediction result is obtained. When the input prompt is a point localization task, the shared feature representation is input into the corresponding point localization prediction head module in the task decoder. Based on the point coordinate prediction head sub-module and the point confidence prediction head module, point coordinate prediction and confidence filtering are performed to obtain the number of valid points. Based on the number of valid points, the population size prediction result is obtained. When the input prompt is a density map generation task, the shared feature representation is input into the corresponding density map generation prediction head module in the task decoder to obtain the population size prediction result.

7. The method for unified crowd counting based on visual language cues according to claim 6, characterized in that, When the input prompt is a density map generation task, the shared feature representation is input into the corresponding density map generation prediction head module in the task decoder to obtain the crowd size prediction result, including: The shared feature representation is input into the dynamic weight generation module, and the dynamic weights are obtained by nonlinear transformation and dimension mapping through a multilayer perceptron. The shared feature representation and the image features of the target image are input into the upsampling module. The shared feature representation and the image features are upsampled and fused through transposed convolution to obtain the sampled feature map. The dynamic weights and sampled feature maps are weighted and fused using matrix multiplication to obtain the density map; The density map is integrated to obtain the population size prediction result.

8. A unified crowd counting system based on visual language cues and multiple paradigms, implemented using the unified crowd counting method based on visual language cues according to any one of claims 1 to 7, characterized in that, The system includes: The acquisition module is used to acquire the target image. It inputs the target image into the image encoder generated by the mask autoencoder to capture local detail features and global context information, and obtains the image embedding. The processing module is used to input input prompts into the visual language prompt encoder module, perform positional encoding on point input prompts and box input prompts through a position encoder to obtain positional embedding, and perform text encoding on text input prompts through a text encoder to obtain text embedding. The positional embedding and text embedding are then input into the prompt embedding module for merging to obtain prompt embedding. The input prompts include point input prompts, box input prompts and text input prompts. The counting module is used to input the image embedding and the cue embedding into the task adaptive decoder. The bidirectional attention feature fusion module fuses the image embedding and the cue embedding to obtain a shared feature representation. Based on the input cue, the shared feature representation is input into the corresponding target prediction head module in the task decoder to obtain the crowd size prediction result. The task decoder includes a box detection prediction head module, a point localization prediction head module, and a density map generation prediction head module.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-paradigm unified crowd counting method based on visual language cues as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multi-paradigm unified crowd counting method based on visual language cues as described in any one of claims 1 to 7.

Images