Vision-based intelligent counting method and system

Abstract

The application belongs to the technical field of computer vision, and discloses an intelligent counting method and system based on vision, which comprises the following steps: obtaining a multi-scale visual feature map by visualizing an original image; projecting a semantic embedding vector by querying a text semantic of an open vocabulary; modulating a multi-scale visual feature map channel by using a semantic vector to obtain an initial joint feature map; taking an initial joint feature map pixel grid as a graph node, and constructing a local density perception graph by assigning connection weights between nodes according to semantic correlation, visual similarity and spatial distance between nodes; performing message passing on the local density perception graph, and applying a local density consistency constraint and a cross-modal semantic focusing constraint to obtain an enhanced node feature set; and performing weighted summation on the enhanced node feature set by using an attention pooling function to obtain a global target count value corresponding to the open vocabulary query text. The application significantly reduces counting drift error, improves target recognition purity and reasoning efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a vision-based intelligent counting method and system. Background Technology

[0002] Vision-based target counting is one of the core tasks in the field of computer vision. It aims to automatically count the number of instances of a specific target from images or videos. It is widely used in scenarios such as counting parts on industrial production lines, estimating the yield of agricultural fruits, monitoring traffic flow, counting biological cells, and analyzing crowd density in public places.

[0003] However, in dense target and highly crowded scenarios, existing methods are prone to problems such as severe target occlusion, feature response diffusion, and large counting drift errors. At the same time, traditional density map regression methods cannot perform consistent normalization of node responses within the same dense target cluster, resulting in significant density response deviations within the cluster, which can easily lead to accumulated counting errors.

[0004] Furthermore, existing visual counting schemes lack effective semantic guidance mechanisms for open vocabularies, failing to dynamically suppress background interference and semantically irrelevant regions during message passing and feature aggregation. This leads to semantic drift, resulting in insufficient purity and limited accuracy in the counting results. These issues collectively cause traditional methods to suffer from high annotation costs in complex scenarios, large errors in dense scenes, strong background interference, and non-generalization of open categories, making it difficult to meet the practical needs of high-precision, robust, and generalizable intelligent counting.

[0005] Therefore, how to provide a vision-based intelligent counting method and system is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a vision-based intelligent counting method and system to solve the problems in the prior art.

[0007] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0008] According to a first aspect of the present invention, a vision-based intelligent counting method is provided.

[0009] In one embodiment, a vision-based intelligent counting method includes: Visual extraction is performed on the original images of the scene to be counted to obtain multi-scale visual feature maps; Perform text semantic projection on the obtained open-vocabulary query text of the scene to be counted to obtain semantic embedding vectors; By using a cross-modal attention mechanism, semantic embedding vectors are used to perform channel-level modulation on multi-scale visual feature maps to generate an initial joint feature map; Using each pixel grid in the initial joint feature map as a graph node, and assigning connection weights between nodes based on semantic relevance, visual similarity, and spatial distance, a local density-aware map is constructed. Message passing and state updates between nodes are performed on the local density-aware graph, and local density consistency constraints and cross-modal semantic focusing constraints are applied during message passing to obtain an enhanced set of node features. The global target count value corresponding to the open vocabulary query text is obtained by using the attention pooling function to perform a weighted summation of the final states of all nodes in the enhanced node feature set.

[0010] According to a second aspect of the present invention, a vision-based intelligent counting system is provided.

[0011] In one embodiment, a vision-based intelligent counting system includes: The visual feature extraction module is used to perform visual extraction on the acquired original image of the scene to be counted, and obtain a multi-scale visual feature map. The text semantic projection module is used to perform text semantic projection on the acquired open-vocabulary query text of the scene to be counted, and obtain semantic embedding vectors. The cross-modal feature modulation module is used to perform channel-level modulation of multi-scale visual feature maps using semantic embedding vectors through a cross-modal attention mechanism to generate an initial joint feature map. The local density-aware graph construction module is used to construct a local density-aware graph by taking each pixel grid in the initial joint feature map as a graph node and assigning connection weights between nodes according to semantic relevance, visual similarity and spatial distance between nodes. The dual-constraint message passing module is used to perform message passing and state updates between nodes on the local density-aware graph, and to apply local density consistency constraints and cross-modal semantic focusing constraints during message passing to obtain an enhanced set of node features. The attention pooling counting module is used to perform a weighted summation of the final states of all nodes in the enhanced node feature set using the attention pooling function to obtain the global target count value corresponding to the open vocabulary query text.

[0012] According to a third aspect of the present invention, a computer device is provided.

[0013] In some embodiments, the computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0014] According to a fourth aspect of the present invention, a computer-readable storage medium is provided.

[0015] In one embodiment, a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the above method.

[0016] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: 1. This invention establishes a cross-modal semantic alignment mechanism between open-vocabulary query text and visual features, maps natural language descriptions into semantic embedding vectors and guides visual feature channel-level modulation, enabling a single counting method to understand and adaptively focus on any user-specified category target without needing to retrain the model for new categories, significantly reducing the deployment cost and response cycle of cross-category counting tasks.

[0017] 2. This invention forces the minimization of the density response variance of nodes within the same dense cluster through local density consistency constraints, ensuring smooth and consistent counting contributions within the cluster; and dynamically suppresses the feature responses of semantically irrelevant nodes through cross-modal semantic focus constraints, enabling the message passing process to focus on the specified target region, thus fundamentally solving the core problems of response diffusion and semantic drift in traditional methods under severe occlusion and highly crowded scenarios.

[0018] 3. This invention solves the core problems of high annotation cost, large error in dense scenes, strong background interference, and non-generalization of open categories in traditional visual counting by integrating multi-scale visual extraction, open vocabulary semantic alignment, local density consistency regularization, cross-modal semantic focus suppression, and attention pooling direct counting. It significantly reduces counting drift error and improves target recognition purity and inference efficiency.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0021] Figure 1 This is a flowchart illustrating a vision-based intelligent counting method according to an exemplary embodiment; Figure 2 This is a block diagram illustrating the principle of a vision-based intelligent counting system according to an exemplary embodiment; Figure 3 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0022] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some portions and features of certain embodiments may be included in or replace portions and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims and all available equivalents thereof. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0023] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] Figure 1 An embodiment of the vision-based intelligent counting method of the present invention is shown.

[0026] In this optional embodiment, the vision-based intelligent counting method includes: S101. Visual extraction is performed on the original image of the scene to be counted to obtain a multi-scale visual feature map.

[0027] It should be noted that the original image of the scene to be counted is acquired by an image acquisition device (such as an industrial camera or a mobile terminal camera), and the image may contain a varying number of target objects of different scales. To effectively capture the visual features of targets at different scales in the original image, multi-granularity local region response analysis is performed on the original image. The specific implementation method is as follows: Multi-granularity local region response analysis is performed on the original image to obtain several visual representations of coarse-grained global distribution, medium-grained structural features, and fine-grained local details. Each visual representation is spatially aligned and scale-normalized and mapped to a unified reference system to obtain a standard visual representation. The standard visual representations at the same spatial location are then weighted and fused element by element to generate a multi-scale visual feature map.

[0028] S102. Perform text semantic projection on the obtained open-word query text of the scene to be counted to obtain a semantic embedding vector; It should be explained that, in order to convert the above natural language query text into a numerical representation that can be interactively computed with visual features, text semantic projection is performed on the open vocabulary query text. The specific implementation method is as follows: The system acquires open-word query text for the scenarios to be counted without pre-formatting. It preprocesses the open-word query text, removing irrelevant and redundant information while retaining core semantic words. It then performs text semantic projection on the core semantic words, transforming each core semantic word into a fixed-dimensional vector fragment using semantic mapping rules. These rules are based on semantic relationships between words, mapping words with similar meanings to vector fragments with high similarity. Finally, it fuses the vector fragments corresponding to each core semantic word element-by-element to obtain a semantic embedding vector, whose dimension is consistent with the feature channel dimension of the multi-scale visual feature map.

[0029] S103. Through a cross-modal attention mechanism, the semantic embedding vector is used to perform channel-level modulation on the multi-scale visual feature map to generate an initial joint feature map.

[0030] In this optional embodiment, the initial joint feature map is generated by channel-level modulation of multi-scale visual feature maps using a cross-modal attention mechanism and semantic embedding vectors: The semantic embedding vector and each feature channel of the multi-scale visual feature map are used to calculate the semantic visual matching degree, and the attention response value corresponding to each feature channel is obtained. Specifically, the multi-scale visual feature map contains multiple feature channels, each corresponding to a visual feature; the semantic visual matching degree is used to characterize the degree of fit between the visual feature of a single feature channel and the semantics of the open vocabulary. The higher the degree of fit, the larger the matching degree value, and vice versa; the attention response value directly reflects the correlation between the corresponding feature channel and the open vocabulary target.

[0031] The attention response values ​​corresponding to each feature channel are normalized to generate channel attention weight vectors; Specifically, the attention response values ​​for each feature channel are normalized to eliminate scale differences in the attention response values ​​of different feature channels, ensuring the comparability of the weights of each channel. Among them, feature channels with high semantic fit with open vocabulary have larger weight values ​​in the weight vector, while feature channels that are semantically unrelated have smaller weight values, providing a reasonable basis for subsequent channel-level weighting.

[0032] The channel attention weight vector is used to weight each feature channel of the multi-scale visual feature map to generate an initial joint feature map.

[0033] Specifically, feature channels with large weights (related to semantics) have their features enhanced, while feature channels with small weights (unrelated to semantics) have their features suppressed. After weighting, the weighted features of all feature channels are integrated, aggregating multi-scale visual features and open-vocabulary semantic features to generate an initial joint feature map. This initial joint feature map retains both global and local information from multi-scale visual features and incorporates semantic guidance from open-vocabulary, accurately reflecting the feature distribution of the user-specified target.

[0034] S104. Using each pixel grid in the initial joint feature map as a graph node, and assigning connection weights between nodes according to semantic correlation, visual similarity and spatial distance, a local density-aware map is constructed. In this optional embodiment, a local density-aware map is constructed by using each pixel grid in the initial joint feature map as a graph node, and assigning connection weights between nodes based on semantic relevance, visual similarity, and spatial distance. Each pixel grid in the initial joint feature map is treated as an independent graph node, and each graph node corresponds to the semantic visual joint feature at a pixel position in the initial joint feature map; Specifically, the semantic-visual joint feature is a vector composed of all feature values ​​along the channel dimension at the pixel location in the initial joint feature map. The feature vector simultaneously contains the visual appearance information of the image region corresponding to the pixel location and the semantically guided information after semantic modulation.

[0035] For any two graph nodes, calculate their semantic relevance, visual similarity, and spatial distance respectively; Semantic relevance, visual similarity, and spatial distance are assigned preset weights and fused together to obtain the connection weights between two graph nodes. Specifically, semantic relevance, visual similarity, and spatial distance are each assigned a preset weight coefficient, with the sum of the three preset weights being 1. Each weight coefficient can be adjusted according to the target distribution characteristics and the clarity of the query semantics in the actual application scenario. For example, in scenarios where the target distribution is extremely dense and severely occluded, the weight coefficient of spatial distance can be appropriately increased to strengthen local neighborhood constraints; in scenarios where the query text semantics are clear and the background interference is complex, the weight coefficient of semantic relevance can be appropriately increased to highlight the semantic guidance role. To reduce the storage and computational overhead of the graph structure, this step can calculate the connection weight only for node pairs with a spatial distance greater than a preset spatial threshold, that is, only retain the connection edges between spatially adjacent nodes. For node pairs with excessively large spatial distances, their connection weights are directly set to zero, forming a sparsely connected local density-aware graph.

[0036] A local density-aware graph is constructed based on the node weights of each graph node and the connection weights between all graph nodes.

[0037] Specifically, in a local density-aware map, clusters of tightly connected nodes with high weights often correspond to densely distributed target areas or internal pixels of the same target instance; regions of sparsely connected nodes with low weights often correspond to sparse target areas or background areas.

[0038] In this optional embodiment, for any two graph nodes, calculating their corresponding semantic relevance, visual similarity, and spatial distance includes: The semantic relevance is obtained by performing a normalized inner product operation on the feature vectors corresponding to two graph nodes under the guidance of the semantic embedding vector. Specifically, for any two graph nodes i and j, their corresponding feature vectors are denoted as f. i and f j First, the eigenvector f i and f j Element-wise multiplication with the semantic embedding vector yields the semantically weighted feature vector f. i 'and f j This operation strengthens the semantically relevant dimensional components of the feature vector. Subsequently, for f... i 'and f j Perform a normalized inner product operation, which involves calculating the inner product of the two nodes and then dividing it by the product of their moduli. The result is used as the semantic relevance. This semantic relevance measures the consistency of the feature representation of two graph nodes under semantic guidance: if the two nodes correspond to the target region of the same semantic category, their semantic relevance is high; if the two nodes belong to the target and background regions or different semantic category regions, their semantic relevance is low.

[0039] Calculate the cosine similarity of the feature vectors corresponding to two graph nodes to obtain the visual similarity; Specifically, semantic relevance focuses on consistency guided by semantics, while visual similarity focuses on the directional consistency of feature vectors in the original high-dimensional space. This step directly calculates the feature vector f corresponding to the two graph nodes. i and f j Cosine similarity is used as a visual similarity metric. For example, image regions with similar colors and textures may still have high visual similarity even if their semantic categories are different; however, when used in conjunction with semantic relevance, visual similarity can supplement and verify semantic relevance.

[0040] Based on the mapping relationship between each pixel grid in the initial joint feature map and the pixel coordinates of the original image, the corresponding pixel coordinates of the two graph nodes in the original image are determined, the Euclidean distance between the two pixel coordinates is calculated, and the Euclidean distance is mapped to a negative exponential value to obtain the spatial distance.

[0041] Specifically, the semantic relevance and visual similarity mentioned above are measures in the feature space, without considering the actual spatial positional relationship of graph nodes on the physical imaging plane. To introduce spatial prior constraints on the target distribution, this step calculates the spatial distance between two graph nodes in the original image. Specifically, based on the mapping relationship between each pixel grid in the initial joint feature map and the pixel coordinates of the original image, the corresponding pixel coordinates of the two graph nodes in the original image are determined, and the Euclidean distance between the two pixel coordinates is calculated. Euclidean distance calculation is an existing technique and will not be elaborated here. Since directly using Euclidean distance as a similarity measure has inconsistencies in dimensions and a reverse relationship where the greater the distance, the lower the similarity, this step performs a negative exponential mapping on the Euclidean distance. After the negative exponential mapping, the spatial distance value range is (0,1]. The closer the two nodes are in physical space, the closer their spatial distance value is to 1; the farther the two nodes are in space, the closer their spatial distance value is to 0. This mapping method ensures that the spatial distance is consistent with the semantic relevance and visual similarity in both numerical range and direction of action.

[0042] S105. Perform message passing and state updates between nodes on the local density-aware map, and apply local density consistency constraints and cross-modal semantic focusing constraints during message passing to obtain an enhanced node feature set. In this optional embodiment, applying local density consistency constraints during message passing includes: After each round of message passing, based on the current connection weight and density response value of each graph node in the local density perception graph, the set of connected nodes whose connection weight is greater than the preset connection threshold and whose average density response value exceeds the preset density threshold is identified, forming a high-density connected subgraph. For each high-density connected subgraph, the variance of the density response values ​​of each graph node within the high-density connected subgraph is calculated, and the variance is introduced as a constraint term into the message passing optimization process to constrain the density response values ​​of each graph node within the same high-density connected subgraph to tend to be consistent.

[0043] In this optional embodiment, applying cross-modal semantic focus constraints during message passing includes: In each round of message passing, the semantic relevance between the current feature vector and the semantic embedding vector of each graph node is calculated; Generate semantic gating coefficients for each graph node based on semantic relevance; Semantic gating coefficients are applied to the information weights of the corresponding graph nodes during message aggregation.

[0044] It should be explained that, in order to achieve accurate perception of the target distribution and reasonable allocation of counting contributions on this graph structure, this step performs multiple rounds of message passing and state updates between nodes on the graph, and applies two innovative constraint mechanisms in the process.

[0045] The message passing and state updates in this step can be implemented using the common message passing paradigm in graph neural networks. In each round of message passing, each graph node performs the following operations: 1. Message generation: Node i sends message m to its neighbor node j. i→j The connection weight is determined by the current feature vectors of both parties and the connection weight, and the connection weight makes the message strength positively correlated with the degree of association between the nodes.

[0046] 2. Message Aggregation: Node j collects neighborhood messages and integrates them into aggregate message a using an aggregation function. j The aggregation method can use weighted summation, where the connection weight affects both the message content and the aggregation weight.

[0047] 3. State update: Node j updates its own feature vector based on the aggregated message, which can be achieved using residual connections.

[0048] 4. After multiple rounds of message passing, the feature vectors of each graph node continuously absorb neighborhood information, gradually aggregate local context, and finally form a node state containing rich neighborhood relationships.

[0049] In the message passing process of traditional graph neural networks, the state updates of each node lack explicit constraints on the consistency of density response. This leads to significant differences in the density response values ​​of different nodes within the target dense cluster, resulting in uneven distribution of counting contributions and counting errors after integration. To address this issue, this step applies local density consistency constraints during message passing, as detailed below: 1. After each round of message passing, identify high-density connected subgraphs in the graph. A set of connected nodes that simultaneously meets the following two conditions is identified as a high-density connected subgraph: First, the connection weight between any adjacent nodes in the set is greater than a preset connection threshold, ensuring that the nodes in the subgraph have high semantic, visual, and spatial consistency; Second, the average density response of all nodes in the set exceeds a preset density threshold, ensuring that the corresponding region of the subgraph is indeed a densely distributed target area. The density response value is a scalar value obtained by mapping the feature vector of each graph node through a fully connected layer after the state update in the current round, representing the probability contribution of the node's location to the existence of a target center. The above identification process can be implemented using graph traversal algorithms (such as depth-first search or breadth-first search) combined with threshold determination, which is existing technology and will not be elaborated further here.

[0050] 2. For each identified high-density connected subgraph, calculate the variance of the density response values ​​of each node within that subgraph. A larger variance indicates a more uneven distribution of density responses within the subgraph, and a more unbalanced distribution of counting contributions. To suppress this unevenness, this step introduces the variance as a constraint into the message passing optimization process. During the training phase, this constraint is directly added to the overall loss function, optimizing it together with the counting error loss. During the inference phase, this constraint acts as a regularization term on the node state update process, causing the state update directions of nodes within the high-density subgraph to converge in the next round of message passing. Specifically, this constraint effect can be achieved by adding a Laplace smoothing term to the state update formula: for nodes within a high-density subgraph, their updated feature vectors not only depend on the aggregated message but are also influenced by the mean of the feature vectors of other nodes within the subgraph. In this way, the density response values ​​of each node in the same high-density connected subgraph tend to minimize the variance after multiple iterations, ensuring that the counting contribution of each node in the same dense target cluster is smooth and consistent, and avoiding local counting errors caused by the deviation of the node response within the cluster.

[0051] During message passing, without semantic guidance, information exchange between nodes will occur uniformly across the entire graph. Background nodes irrelevant to the user's query semantics will also participate in message passing, increasing computational overhead and potentially interfering with the state updates of nodes in the target region. To ensure that the message passing process adaptively focuses on the user-specified target region, this step applies a cross-modal semantic focusing constraint, implemented as follows: 1. In each round of message passing, calculate the semantic relevance between the current feature vector and the semantic embedding vector of each graph node. The calculation method for this semantic relevance can be consistent with the calculation method for semantic relevance, i.e., performing a normalized inner product operation on the node's feature vector and semantic embedding vector. The resulting semantic relevance reflects the strength of the node's response to the user's query semantics in the feature space: nodes highly relevant to the query semantics (such as target region pixels) will obtain higher semantic relevance, while nodes unrelated to the query semantics (such as background region pixels) will obtain lower semantic relevance.

[0052] 2. Generate semantic gating coefficients for each graph node based on semantic relevance. Semantic gating coefficients are obtained by thresholding and scaling the semantic relevance. For example, a preset semantic threshold τ is set. For nodes with semantic relevance higher than τ, their semantic gating coefficient is set to 1, fully preserving their contribution to message passing. For nodes with semantic relevance lower than τ, their semantic gating coefficient is set to a decay value less than 1, or directly set to zero, suppressing or blocking their message passing. To ensure the gating process is continuously differentiable, the Sigmoid function can also be used to map the semantic relevance to the (0,1) interval as the semantic gating coefficient.

[0053] 3. By applying semantic gating coefficients to the information weights of corresponding graph nodes during message aggregation, the messages sent by nodes with low semantic relevance are weakened or masked, thus failing to significantly impact the state updates of adjacent nodes. After multiple rounds of message passing, the feature responses of semantically unrelated nodes gradually decay, while the feature responses of semantically relevant nodes are consolidated and enhanced due to the continuous reception of homogeneous information, achieving adaptive semantic focusing in the message passing process.

[0054] S106. Use the attention pooling function to perform a weighted summation of the final states of all nodes in the enhanced node feature set to obtain the global target count value corresponding to the open vocabulary query text.

[0055] In this optional embodiment, the final states of all nodes in the enhanced node feature set are weighted and summed using an attention pooling function to obtain the global target count value corresponding to the open vocabulary query text, including: Calculate the matching score between the final state of each graph node in the enhanced node feature set and the semantic embedding vector; Specifically, the matching score measures the degree to which the final state of each graph node matches the semantics of the user query. A node with a higher degree of matching is more likely to have an image region belonging to the user-specified target category, and its contribution weight in the global count should be correspondingly increased. The matching score can be calculated in any of the following ways: 1. Perform an inner product operation on the node's final state feature vector and the semantic embedding vector, and use the projection length of the two in the vector space as the matching score; 2. First, map the node's final state feature vector and semantic embedding vector to the same comparison space through learnable linear transformations, and then calculate the inner product of the transformed vectors to enhance the expressive power of the matching score.

[0056] The matching scores of each graph node are normalized to generate the attention weights corresponding to each graph node. The density response value corresponding to each graph node is determined based on the final state of each graph node, and the density response value corresponding to each graph node is multiplied by the corresponding attention weight to obtain the weighted density contribution value of each graph node. Specifically, the density response value characterizes the probability contribution of the target center point to the spatial location of the node. In practice, the final state feature vector of the node can be input into a transformation unit consisting of a fully connected mapping layer and a sigmoid activation function, and the output is a continuous value between zero and one as the density response value.

[0057] After obtaining the density response value and attention weight of each graph node, the two are multiplied node by node to obtain the weighted density contribution value of the graph node. The attention weight adjusts the node contribution from the perspective of semantic matching, while the density response value quantifies the node contribution from the perspective of spatial density. The multiplication of the two realizes a comprehensive consideration of semantic guidance and spatial distribution.

[0058] The weighted density contribution values ​​of each graph node are summed to obtain the global target count value.

[0059] Specifically, the weighted density contribution values ​​of all graph nodes are summed, and the sum is the global target count value calculated by this method for the open-vocabulary query text input by the user. Since the target count result is a non-negative integer in a physical sense, the accumulated result can be rounded to the nearest integer in the actual output.

[0060] Furthermore, instead of explicitly calculating the density response value of each node, the final state of all nodes and their corresponding attention weights can be used as input to directly aggregate the global feature vector through a graph-level readout function. This global feature vector is then mapped to a scalar count value through a fully connected layer.

[0061] Through the attention pooling mechanism described above, this step achieves direct end-to-end inference from scattered node states to global count values, avoiding the accumulation of intermediate errors caused by predicting the density map first and then integrating globally in traditional methods, thus making the counting results more accurate and stable.

[0062] Figure 2 An embodiment of the vision-based intelligent counting system of the present invention is shown.

[0063] In this optional embodiment, the vision-based intelligent counting system includes: The visual feature extraction module 201 is used to perform visual extraction on the acquired original image of the scene to be counted to obtain a multi-scale visual feature map. The text semantic projection module 202 is used to perform text semantic projection on the acquired open-vocabulary query text of the scene to be counted, so as to obtain a semantic embedding vector; The cross-modal feature modulation module 203 is used to perform channel-level modulation of multi-scale visual feature maps using semantic embedding vectors through a cross-modal attention mechanism to generate an initial joint feature map; The local density-aware graph construction module 204 is used to construct a local density-aware graph by taking each pixel grid in the initial joint feature map as graph nodes and assigning connection weights between nodes according to semantic relevance, visual similarity and spatial distance between nodes. The dual-constraint message passing module 205 is used to perform message passing and state updates between nodes on the local density-aware graph, and to apply local density consistency constraints and cross-modal semantic focusing constraints during message passing to obtain an enhanced node feature set. The attention pooling counting module 206 is used to perform a weighted summation of the final states of all nodes in the enhanced node feature set using the attention pooling function to obtain the global target count value corresponding to the open vocabulary query text.

[0064] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores static and dynamic information data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0065] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0066] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0067] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0068] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0069] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A vision-based intelligent counting method, characterized in that, The method includes: Visual extraction is performed on the original images of the scene to be counted to obtain multi-scale visual feature maps; The semantic embedding vector is obtained by performing text semantic projection on the open-vocabulary query text of the acquired scenario to be counted. By using a cross-modal attention mechanism, semantic embedding vectors are used to perform channel-level modulation on multi-scale visual feature maps to generate an initial joint feature map; Using each pixel grid in the initial joint feature map as a graph node, and assigning connection weights between nodes based on semantic relevance, visual similarity, and spatial distance, a local density-aware map is constructed. Message passing and state updates between nodes are performed on the local density-aware graph, and local density consistency constraints and cross-modal semantic focusing constraints are applied during message passing to obtain an enhanced set of node features. The global target count value corresponding to the open vocabulary query text is obtained by using the attention pooling function to perform a weighted summation of the final states of all nodes in the enhanced node feature set.

2. The vision-based intelligent counting method according to claim 1, characterized in that, The step of generating an initial joint feature map by using a cross-modal attention mechanism and semantic embedding vectors to perform channel-level modulation on multi-scale visual feature maps includes: The semantic embedding vector and each feature channel of the multi-scale visual feature map are used to calculate the semantic visual matching degree, and the attention response value corresponding to each feature channel is obtained. The attention response values ​​corresponding to each feature channel are normalized to generate channel attention weight vectors; The channel attention weight vector is used to weight each feature channel of the multi-scale visual feature map to generate an initial joint feature map.

3. The vision-based intelligent counting method according to claim 1, characterized in that, The process of constructing a local density-aware map by using each pixel grid in the initial joint feature map as a graph node and assigning connection weights between nodes based on semantic relevance, visual similarity, and spatial distance includes: Each pixel grid in the initial joint feature map is treated as an independent graph node, and each graph node corresponds to the semantic visual joint feature at a pixel position in the initial joint feature map; For any two graph nodes, calculate their semantic relevance, visual similarity, and spatial distance respectively; Semantic relevance, visual similarity, and spatial distance are assigned preset weights and fused together to obtain the connection weights between two graph nodes. A local density-aware graph is constructed based on the node weights of each graph node and the connection weights between all graph nodes.

4. The vision-based intelligent counting method according to claim 3, characterized in that, The calculation of semantic relevance, visual similarity, and spatial distance for any two graph nodes includes: The semantic relevance is obtained by performing a normalized inner product operation on the feature vectors corresponding to two graph nodes under the guidance of the semantic embedding vector. Calculate the cosine similarity of the feature vectors corresponding to two graph nodes to obtain the visual similarity; Based on the mapping relationship between each pixel grid in the initial joint feature map and the pixel coordinates of the original image, the corresponding pixel coordinates of the two graph nodes in the original image are determined, the Euclidean distance between the two pixel coordinates is calculated, and the Euclidean distance is mapped to a negative exponential value to obtain the spatial distance.

5. The vision-based intelligent counting method according to claim 1, characterized in that, The application of local density consistency constraints during message passing includes: After each round of message passing, based on the current connection weight and density response value of each graph node in the local density perception graph, the set of connected nodes whose connection weight is greater than the preset connection threshold and whose average density response value exceeds the preset density threshold is identified, forming a high-density connected subgraph. For each high-density connected subgraph, the variance of the density response values ​​of each graph node within the high-density connected subgraph is calculated, and the variance is introduced as a constraint term into the message passing optimization process to constrain the density response values ​​of each graph node within the same high-density connected subgraph to tend to be consistent.

6. The vision-based intelligent counting method according to claim 1, characterized in that, The application of cross-modal semantic focus constraints during message passing includes: In each round of message passing, the semantic relevance between the current feature vector and the semantic embedding vector of each graph node is calculated; Generate semantic gating coefficients for each graph node based on semantic relevance; Semantic gating coefficients are applied to the information weights of the corresponding graph nodes during message aggregation.

7. The vision-based intelligent counting method according to claim 1, characterized in that, The process of using an attention pooling function to perform a weighted summation of the final states of all nodes in the enhanced node feature set, resulting in the global target count value corresponding to the open-vocabulary query text, includes: Calculate the matching score between the final state of each graph node in the enhanced node feature set and the semantic embedding vector; The matching scores of each graph node are normalized to generate the attention weights corresponding to each graph node. The density response value corresponding to each graph node is determined based on the final state of each graph node, and the density response value corresponding to each graph node is multiplied by the corresponding attention weight to obtain the weighted density contribution value of each graph node. The weighted density contribution values ​​of each graph node are summed to obtain the global target count value.

8. A vision-based intelligent counting system, characterized in that, The system includes: The visual feature extraction module is used to perform visual extraction on the acquired original image of the scene to be counted, and obtain a multi-scale visual feature map. The text semantic projection module is used to perform text semantic projection on the acquired open-vocabulary query text of the scene to be counted, and obtain semantic embedding vectors. The cross-modal feature modulation module is used to perform channel-level modulation of multi-scale visual feature maps using semantic embedding vectors through a cross-modal attention mechanism to generate an initial joint feature map. The local density-aware graph construction module is used to construct a local density-aware graph by taking each pixel grid in the initial joint feature map as a graph node and assigning connection weights between nodes according to semantic relevance, visual similarity and spatial distance between nodes. The dual-constraint message passing module is used to perform message passing and state updates between nodes on the local density-aware graph, and to apply local density consistency constraints and cross-modal semantic focusing constraints during message passing to obtain an enhanced set of node features. The attention pooling counting module is used to perform a weighted summation of the final states of all nodes in the enhanced node feature set using the attention pooling function to obtain the global target count value corresponding to the open vocabulary query text.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images